How do I?

User Tools

Site Tools

You are here: start » techstaff » slurm
techstaff:slurm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Next revisionBoth sides next revision
techstaff:slurm [2018/05/04 12:38] – kauffmantechstaff:slurm [2019/10/08 17:24] – [Notice] kauffman
Line 1: Line 1:
 ===== Notice ===== ===== Notice =====
-**2017-08-31**: Configuration change to allow allocation on CPUs and RAM. Please read the 'Default Quota' section under https://howto.cs.uchicago.edu/techstaff:slurm#usage  +**2019-10-08**: New computer nodes added under the partiton ''%%genfast%%''
 ====== Peanut Job Submission Cluster ====== ====== Peanut Job Submission Cluster ======
  
Line 98: Line 97:
 | **debug** | The partition your job will be submitted to if none is specified. The purpose of this partition is to make sure your code is running as it should before submitting a long running job to the general queue. | | **debug** | The partition your job will be submitted to if none is specified. The purpose of this partition is to make sure your code is running as it should before submitting a long running job to the general queue. |
 | **general** | All jobs that have been thoroughly tested can be submitted here. This partition will have access to more nodes and will process most of the jobs. If you need to use the ''%%--exclusive%%'' flag it should be done here.| | **general** | All jobs that have been thoroughly tested can be submitted here. This partition will have access to more nodes and will process most of the jobs. If you need to use the ''%%--exclusive%%'' flag it should be done here.|
-| **gpu** | Contains servers with graphics cards. As of May 2016 there is only one node containing a Tesla M2090. You will be forced to use this server exclusively for now. Please keep your time in interactive mode to a minimum.|+| **pascal** | 2018-05-04: 1x Nvidia GTX1080. You will be forced to use this server exclusively for now. Please keep your time in interactive mode to a minimum.| 
 +| **titan** | 2018-05-04: 4x Nvidia GTX1080Ti. This partition is shared and you MUST use the ''%%--gres%%'' to specify the resources you wish to use. It is also encouraged to specify cpu and memory.|
  
 ====== Job Submission ====== ====== Job Submission ======
Line 111: Line 111:
 | ^ SLURM ^ Example ^ | ^ SLURM ^ Example ^
 ^ Submit a batch serial job | sbatch | sbatch runscript.sh | ^ Submit a batch serial job | sbatch | sbatch runscript.sh |
-^ Run a script interatively | srun | srun --pty -p interact -t 10 --mem 1000 \\ /bin/bash \\ /bin/hostname |+^ Run a script interactively | srun | srun --pty -p interact -t 10 --mem 1000 \\ /bin/bash \\ /bin/hostname |
 ^ Kill a job | scancel | scancel 4585 | ^ Kill a job | scancel | scancel 4585 |
 ^ View status of queues | squeue | squeue -u cnetid | ^ View status of queues | squeue | squeue -u cnetid |
Line 268: Line 268:
 Example when using tensorflow: Example when using tensorflow:
  
-Give the file 'f':   +Given the file ''%%f%%'':   
 <code> <code>
 #!/usr/bin/env python3 #!/usr/bin/env python3
Line 277: Line 277:
 Here we can see that no GPU was allocated to us because we did not specify the ''%%--gres%%'' option Here we can see that no GPU was allocated to us because we did not specify the ''%%--gres%%'' option
 <code> <code>
-  kauffman3@bulldozer:~$ srun -p titan --pty /bin/bash +user@bulldozer:~$ srun -p titan --pty /bin/bash 
-  kauffman3@gpu3:~$ ./f 2>&1 | grep physical_device_desc +user@gpu3:~$ ./f 2>&1 | grep physical_device_desc 
-  kauffman3@gpu3:~$+user@gpu3:~$
 </code> </code>
  
 If we request only 1 GPU. If we request only 1 GPU.
 <code> <code>
-  kauffman3@bulldozer:~$ srun -p titan --pty --gres=gpu:1 /bin/bash +user@bulldozer:~$ srun -p titan --pty --gres=gpu:1 /bin/bash 
-  kauffman3@gpu3:~$ ./f 2>&1 | grep physical_device_desc +user@gpu3:~$ ./f 2>&1 | grep physical_device_desc 
-  physical_device_desc: "device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:19:00.0, compute capability: 6.1"+physical_device_desc: "device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:19:00.0, compute capability: 6.1"
 </code> </code>
  
 If we request 2 GPUs. If we request 2 GPUs.
 <code> <code>
-kauffman3@bulldozer:~$ srun -p titan --pty --gres=gpu:2 /bin/bash +user@bulldozer:~$ srun -p titan --pty --gres=gpu:2 /bin/bash 
-kauffman3@gpu3:~$ ./f 2>&1 | grep physical_device_desc +user@gpu3:~$ ./f 2>&1 | grep physical_device_desc 
-  physical_device_desc: "device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:19:00.0, compute capability: 6.1" +physical_device_desc: "device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:19:00.0, compute capability: 6.1" 
-  physical_device_desc: "device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:1a:00.0, compute capability: 6.1"+physical_device_desc: "device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:1a:00.0, compute capability: 6.1"
 </code> </code>
  
 If we request more GPUs then are available. If we request more GPUs then are available.
 <code> <code>
-  kauffman3@bulldozer:~$ srun -p titan --pty --gres=gpu:5 /bin/bash +kauffman3@bulldozer:~$ srun -p titan --pty --gres=gpu:5 /bin/bash 
-  srun: error: Unable to allocate resources: Requested node configuration is not available+srun: error: Unable to allocate resources: Requested node configuration is not available
 </code> </code>
  
Line 319: Line 319:
  
  
 +==== Checking how many Generic RESources are being consumed ====
  
-===== Paths ===== +Simple use the ''%%-O%%'' option for ''%%squeue%%'' and you can see how many generic resources any particular job is consuming. 
-You will need to add the following to your $PATH and $LD_LIBRARY_PATH.+<code> 
 +$ squeue -O username,nodelist,gres 
 +USER                NODELIST            GRES                 
 +someusername        gpu3                gpu:1                
 +otherusername       gpu3                gpu:3                
 +... 
 +</code> 
 + 
 + 
 +===== Environment Variables ===== 
 + 
 +==== CUDA_HOME, LD_LIBRARY_PATH ==== 
 + 
 +Please make sure you specify $CUDA_HOME and if you want to take advantage of CUDNN libraries you will need to append /usr/local/cuda-x.x/lib64 to the $LD_LIBRARY_PATH environment variable. 
 + 
 +  cuda_version=9.2 
 +  export CUDA_HOME=/usr/local/cuda-${cuda_version} 
 +  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_HOME/lib64 
 + 
 +Currently we support the same versions of CUDA that the latest version of CUDNN supports. This is not written in stone and we can accommodate most other versions if required; just let techstaff know what your needs are. 
 + 
 +==== PATH ==== 
 +You may also need to add the following to your ''%%$PATH%%''
  
   export PATH=$PATH:/usr/local/cuda/bin   export PATH=$PATH:/usr/local/cuda/bin
-  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/usr/local/cuda/lib+ 
 +==== CUDA_VISIBLE_DEVICES ==== 
 +Do not set this variable. It will be set for you by SLURM. 
 + 
 +The variable name is actually misleading; since it does NOT mean the amount of devices, but rather the physical device number assigned by the kernel (e.g. /dev/nvidia2). 
 + 
 +For example: If you requested multiple gpu's from SLURM (--gres=gpu:2), the CUDA_VISIBLE_DEVICES variable should contain two numbers(0-3 in this case) separated by a comma (e.g. 1,3).
  
  
Line 382: Line 411:
 STDERR should be blank. STDERR should be blank.
 ====== More ====== ====== More ======
-If you feel this documentation is lacking in some way please let techstaff know. Email [[techstaff@cs.uchicago.edu]], call (773-702-1031), or stop by our office (Ryerson 154).+If you feel this documentation is lacking in some way please let techstaff know. Email [[techstaff@cs.uchicago.edu]], call (773-702-1031), or stop by our office (Crerar 357).
/var/lib/dokuwiki/data/pages/techstaff/slurm.txt · Last modified: 2021/01/06 16:13 by kauffman
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki