Gemeinsame Systemgruppe IfI/b-it

You are here: aktuelles » en » lmgpu-info

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
en:lmgpu-info [2023-07-19 13:23] – Thomas Thielen:lmgpu-info [2024-11-29 14:00] (current) – [Shared Storage] Anna Gierlach
Line 1: Line 1:
 ====== The Lamarr GPU Cluster ====== ====== The Lamarr GPU Cluster ======
  
-We are building and expanding a GPU Cluster as part of the [[https://lamarr-institute.org/|Lamarr Institute]]. At the moment, the cluster consists of seven nodes with the following basic configuration:+We are building and expanding a GPU Cluster as part of the [[https://lamarr-institute.org/|Lamarr Institute]]. At the moment, the cluster consists of nine nodes with the following basic configuration:
  
-AMD Epyc 7713 (64 Cores)\\+2 AMD Epyc 7713 Processors (64 Cores * 2 Threads each => 256 Cores per Node, 248 available for jobs)\\
 2TB System-RAM\\ 2TB System-RAM\\
 +28TB local SSD storage\\
 8x A100 SXM4 HGX w/each 80GB GPU-Memory, NVLink (600GB/s) connected\\ 8x A100 SXM4 HGX w/each 80GB GPU-Memory, NVLink (600GB/s) connected\\
 10GbE Network\\ 10GbE Network\\
Line 17: Line 18:
 selective groups of users via standard Posix file permissions/user  selective groups of users via standard Posix file permissions/user 
 groups. groups.
 +
 +All nodes have 28TB local SSD storage available, mounted under /data. 
 +Please remove your data from there after you are finished with your 
 +calculations.
  
 The workload management software/scheduler is SLURM. Please refer to The workload management software/scheduler is SLURM. Please refer to
Line 23: Line 28:
  
 [[https://slurm.schedmd.com/documentation.html]] [[https://slurm.schedmd.com/documentation.html]]
- 
-The SLURM job time limits are set to a maximum of 72h, with a default job time 
-limit of 3h. 
  
 Full container support is enabled. We recommend the usage of enroot as Full container support is enabled. We recommend the usage of enroot as
 container backend for SLURM. If you need a personalized library or OS  container backend for SLURM. If you need a personalized library or OS 
 setup around your software, containers will be your way of choice. setup around your software, containers will be your way of choice.
 +
 +As a general rule, realistic resource allocation for your jobs is **mandatory**. You are only allowed to request the resources from the workload manager that you are actually needing to complete your calculations. Wasting precious machine time and making other users wait in line needlessly will lead to revocation of access to the cluster.
 +
 +===== SLURM partitions =====
 +
 +At this time, there is only one active partition available. It is called "batch",
 +the timelimit for jobs in this partition is set to a maximum of 72h and a 
 +default of 3h.
  
 ===== How can I access the cluster? ===== ===== How can I access the cluster? =====
Line 39: Line 49:
 ===== Example of custom software environments using containers  ===== ===== Example of custom software environments using containers  =====
  
-Obviously you will not get administrative privileges on the cluster to setup your personal software environment. But you can import containers to setup any libraries or frameworks you need to use. We tested the cluster setup using [[https://github.com/NVIDIA/enroot|enroot]] from NVidia, which is integrated into SLURM quite well. Any other container environment supported by pyxis should work, too, but was not tested by us yet. +Obviously you will not get administrative privileges on the cluster to setup your personal software environment. But you can import containers to setup any libraries or frameworks you need to use. We tested the cluster setup using [[https://github.com/NVIDIA/enroot|enroot]] from NVidia, which is integrated into SLURM quite well. Any other container environment supported by [[https://github.com/NVIDIA/pyxis|pyxis]] should work, too, but was not tested by us yet. 
  
 In enroot, you import first import external container images, but those images are immutable. You have to create a container and start it read/write to make persistent changes to it.  In enroot, you import first import external container images, but those images are immutable. You have to create a container and start it read/write to make persistent changes to it. 
Line 77: Line 87:
 the .sqfs file from anywhere to the login node (i.e. via scp/sftp) and  the .sqfs file from anywhere to the login node (i.e. via scp/sftp) and 
 start your container via srun just like shown above. start your container via srun just like shown above.
 +
 +===== Shared Storage  =====
 +
 +If you want to share bigger datasets with other users on the cluster you can request shared storage on the cluster. It will be located under /home/Shared/. Please specify the usernames of all users that should be able to access this data (and any other hepful information for us regarding your request) in an [[gsg+lmgpu@informatik.uni-bonn.de|informal request email]]. We will then create a user group for the specified users, create the directory and set the necessary permissions for you.
 +
 +===== Requesting GPUs =====
 +
 +If you want to allocate a certain number of GPUs for your job, use the parameter '--gres=gpu:COUNT' in your srun/squeue calls, where COUNT is the requested number of GPUs for your job.
 +
 +===== Email Notifications =====
 +
 +If you want to receive email notifications about the status of your jobs, you can specify a recipient email address at job submission, i.e.:
 +
 +sbatch --mail-type=ALL --mail-user=someuser@somedomain.com ...
 +
 +**Please specify a full (with domain!) and valid email address here.**
 +
 +
 +===== A Note about Interactive Jobs =====
 +
 +Please remember that interactive jobs often leave things unnecessarily idle (especially if forgotten and/or unterminated), so please think about your impatient colleagues in the queue behind you and try to avoid interactive jobs whenever possible.
  
 ===== Exchange with other users ===== ===== Exchange with other users =====
Line 83: Line 114:
  
 #lmgpu-tech:matrix.informatik.uni-bonn.de #lmgpu-tech:matrix.informatik.uni-bonn.de
 +