How do I?

User Tools

Site Tools

You are here: start » techstaff » slurm
techstaff:slurm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Next revisionBoth sides next revision
techstaff:slurm [2016/05/09 14:50] – [sbatch] kauffmantechstaff:slurm [2017/08/31 15:13] – [Resources] kauffman
Line 1: Line 1:
-====== DRAFT | Peanut Job Submission Cluster ======+===== Notice ===== 
 +**All is back to normal. Please submit jobs from linux1,2,3. Email techstaff@cs.uchicago.edu if you find that something is amiss.**
  
-We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might have been rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3).+The SLURM cluster will become unavailable starting 2017-08-22 for an upgrade. Normal service should resume on 2017-08-25. Please check back here for status updates. 
 + 
 +**2017-08-22 1800**: Main cluster upgraded. You can try to use it now but I can't guarantee that I won't kill your job tomorrow or Firday. 
 + 
 +**2017-08-23 1345**: GPU servers upgraded and added back to the cluster. They may be missing some software that was not automatable at previous time of installation. Send me an email if you find anything missing. 
 + 
 +We still run systems with Ubuntu 14.04 installed. As of right now these systems cannot submit jobs to the cluster. This is on purpose. The slurm version jump between 14.04 and 16.04 was so huge that this was unavoidable. This means you should prefer to use linux.cs.uchicago.edu or any CS machine that run Ubuntu 16.04. 
 + 
 +**2017-08-24**: Everything seems to be working as expected. Please start using the cluster again. Email techstaff@cs.uchicago.edu if something is wrong/unexpected/broken/etc. 
 +====== Peanut Job Submission Cluster ====== 
 + 
 +We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might be rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3).
  
 For job submission we will be using a piece of software called [[http://slurm.schedmd.com|SLURM]]. Simply put, SLURM is a queue management system and stands for **S**imple **L**inux **U**tility for **R**esource **M**anagement; it was developed at the Lawrence Livermore National Lab. It currently supports some of the largest compute clusters in the world. The best description of SLURM can be found on its homepage: For job submission we will be using a piece of software called [[http://slurm.schedmd.com|SLURM]]. Simply put, SLURM is a queue management system and stands for **S**imple **L**inux **U**tility for **R**esource **M**anagement; it was developed at the Lawrence Livermore National Lab. It currently supports some of the largest compute clusters in the world. The best description of SLURM can be found on its homepage:
Line 28: Line 40:
   * [[http://slurm.schedmd.com/tutorials.html|SLURM tutorial videos]]   * [[http://slurm.schedmd.com/tutorials.html|SLURM tutorial videos]]
   * [[https://computing.llnl.gov/linux/slurm/quickstart.html|LLNL quick start user guide]]   * [[https://computing.llnl.gov/linux/slurm/quickstart.html|LLNL quick start user guide]]
- +  * [[http://research.computing.yale.edu/support/hpc/user-guide/slurm| Yale's User Guide]]
 ===== Infrastructure ===== ===== Infrastructure =====
  
Line 37: Line 48:
   * 64gb RAM   * 64gb RAM
   * 2x 500GB SATA 7200RPM in RAID1   * 2x 500GB SATA 7200RPM in RAID1
- 
-To better manage the cluster we have virtualized the job submission nodes and give them all resources of the hardware. So, the actual resources you can consume on any one node is: 
-  * 14 Cores, 14 threads 
-  * 62GB RAM 
  
 ==== Storage ==== ==== Storage ====
Line 109: Line 116:
 ===== Usage ===== ===== Usage =====
 Below are some common examples. You should consult the [[http://slurm.schedmd.com/documentation.html|documentation]] of SLURM if you need further assistance. Below are some common examples. You should consult the [[http://slurm.schedmd.com/documentation.html|documentation]] of SLURM if you need further assistance.
 +
 +=== Default Quotas ===
 +By default we set a job to be run on one CPU and allocate 100MB of RAM. If you require more than that you should specify what you need. Using the following options will do: ''%%--mem-per-cpu%%'', ''%%--nodes%%'', ''%%--ntasks%%''.
  
 === Exclusive access to a node === === Exclusive access to a node ===
Line 118: Line 128:
 === Sample script === === Sample script ===
 Make sure you create a directory in which to deposit the ''%%STDIN%%'', ''%%STDOUT%%'', ''%%STDERR%%'' files. Make sure you create a directory in which to deposit the ''%%STDIN%%'', ''%%STDOUT%%'', ''%%STDERR%%'' files.
-   mkdir -p $HOME/slurm/slurm_out+   mkdir -p $HOME/slurm/out
  
 <code> <code>
Line 125: Line 135:
 #SBATCH --mail-user=cnetid@cs.uchicago.edu #SBATCH --mail-user=cnetid@cs.uchicago.edu
 #SBATCH --mail-type=ALL #SBATCH --mail-type=ALL
-#SBATCH --output=/home/cnetid/slurm/slurm_out/%j.%N.stdout +#SBATCH --output=/home/cnetid/slurm/out/%j.%N.stdout 
-#SBATCH --error=/home/cnetid/slurm/slurm_out/%j.%N.stderr+#SBATCH --error=/home/cnetid/slurm/out/%j.%N.stderr
 #SBATCH --workdir=/home/cnetid/slurm #SBATCH --workdir=/home/cnetid/slurm
 #SBATCH --partition=debug #SBATCH --partition=debug
Line 132: Line 142:
 #SBATCH --nodes=1 #SBATCH --nodes=1
 #SBATCH --ntasks=1 #SBATCH --ntasks=1
 +#SBATCH --mem-per-cpu=500
 #SBATCH --time=15:00 #SBATCH --time=15:00
  
Line 227: Line 238:
 | error: Unable to allocate resources: More processors requested than permitted | It usually has **nothing** to do with priviledges you may or may not have. Rather, it usually means that you have allocated more processors than one compute node actually has. | | error: Unable to allocate resources: More processors requested than permitted | It usually has **nothing** to do with priviledges you may or may not have. Rather, it usually means that you have allocated more processors than one compute node actually has. |
  
 +====== Using the GPU ======
 +===== Paths =====
 +You will need to add the following to your $PATH and $LD_LIBRARY_PATH.
 +
 +  export PATH=$PATH:/usr/local/cuda/bin
 +  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/usr/local/cuda/lib
 +
 +
 +===== Example =====
 +This sbatch script will get device information from the installed Tesla gpu.
 +<code>
 +#!/bin/bash
 +#
 +#SBATCH --mail-user=cnetid@cs.uchicago.edu
 +#SBATCH --mail-type=ALL
 +#SBATCH --output=/home/cnetid/slurm/slurm_out/%j.%N.stdout
 +#SBATCH --error=/home/cnetid/slurm/slurm_out/%j.%N.stderr
 +#SBATCH --workdir=/home/cnetid/slurm
 +#SBATCH --partition=gpu
 +#SBATCH --job-name=get_tesla_info
 +
 +export PATH=$PATH:/usr/local/cuda/bin
 +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/usr/local/cuda/lib
 +
 +cat << EOF > /tmp/getinfo.cu
 +#include <stdio.h>
 +
 +int main() {
 +  int nDevices;
 +
 +  cudaGetDeviceCount(&nDevices);
 +  for (int i = 0; i < nDevices; i++) {
 +    cudaDeviceProp prop;
 +    cudaGetDeviceProperties(&prop, i);
 +    printf("Device Number: %d\n", i);
 +    printf("  Device name: %s\n", prop.name);
 +    printf("  Memory Clock Rate (KHz): %d\n",
 +           prop.memoryClockRate);
 +    printf("  Memory Bus Width (bits): %d\n",
 +           prop.memoryBusWidth);
 +    printf("  Peak Memory Bandwidth (GB/s): %f\n\n",
 +           2.0*prop.memoryClockRate*(prop.memoryBusWidth/8)/1.0e6);
 +  }
 +}
 +EOF
 +
 +/usr/local/cuda/bin/nvcc /tmp/getinfo.cu -o /tmp/a.out
 +/tmp/a.out
 +rm /tmp/a.out
 +rm /tmp/getinfo.cu
 +</code>
 +==== Output ====
 +STDOUT will look something like this:
 +<code>
 +cnetid@linux1:~$ cat $HOME/slurm/slurm_out/12567.gpu1.stdout 
 +Device Number: 0
 +  Device name: Tesla M2090
 +  Memory Clock Rate (KHz): 1848000
 +  Memory Bus Width (bits): 384
 +  Peak Memory Bandwidth (GB/s): 177.408000
 +</code>
 +STDERR should be blank.
 ====== More ====== ====== More ======
 If you feel this documentation is lacking in some way please let techstaff know. Email [[techstaff@cs.uchicago.edu]], call (773-702-1031), or stop by our office (Ryerson 154). If you feel this documentation is lacking in some way please let techstaff know. Email [[techstaff@cs.uchicago.edu]], call (773-702-1031), or stop by our office (Ryerson 154).
/var/lib/dokuwiki/data/pages/techstaff/slurm.txt · Last modified: 2021/01/06 16:13 by kauffman
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki