techstaff:slurm
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
techstaff:slurm [2017/03/10 13:57] – kauffman | techstaff:slurm [2021/01/06 16:13] (current) – kauffman | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Peanut Job Submission Cluster ====== | ====== Peanut Job Submission Cluster ====== | ||
- | We are currently **alpha** testing and gauging user interest in a cluster of machines that allows for the submission of long running compute jobs. Think of these machines as a dumping ground for discrete computing tasks that might be rude or disruptive to execute on the main (shared) shell servers (i.e., linux1, linux2, linux3). | + | [[slurm|This page has moved.]] |
- | + | ||
- | For job submission we will be using a piece of software called | + | |
- | + | ||
- | "Slurm is an open-source workload manager designed for Linux clusters of all sizes. It provides three key functions. First it allocates exclusive and/or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (typically a parallel job) on a set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work." | + | |
- | + | ||
- | SLURM is similar to most other queue systems in that you write a batch script, then submit it to the queue manager. The queue manager schedules your job to run on the queue (or partition in SLURM parlance) that you designate. Below is an outline of how to submit jobs to SLURM, how SLURM decides when to schedule your job, and how to monitor progress. | + | |
- | + | ||
- | ===== Where to begin ===== | + | |
- | SLURM is a set of command line utilities that can be accessed via the command line from **most** any computer science system you can login to. Using our main shell servers (linux.cs.uchicago.edu) is expected to be our most common use case, so you should start there. | + | |
- | + | ||
- | ssh user@linux.cs.uchicago.edu | + | |
- | + | ||
- | + | ||
- | ===== Documentation ===== | + | |
- | The [[http:// | + | |
- | + | ||
- | A great way to get details on SLURM commands are the manuals that are already on the cluster. For example, if you type the following command: | + | |
- | man sbatch | + | |
- | you will get the manual page for the '' | + | |
- | + | ||
- | ===== Resources ===== | + | |
- | * [[https:// | + | |
- | * [[http:// | + | |
- | * [[http:// | + | |
- | * [[http:// | + | |
- | * [[https:// | + | |
- | + | ||
- | + | ||
- | ===== Infrastructure ===== | + | |
- | + | ||
- | ==== Hardware ==== | + | |
- | Our cluster contains nodes with the following specs: | + | |
- | * 16 Cores (2x 8core 3.1GHz Processors), | + | |
- | * 64gb RAM | + | |
- | * 2x 500GB SATA 7200RPM in RAID1 | + | |
- | + | ||
- | To better manage the cluster we have virtualized the job submission nodes and give them all resources of the hardware. So, the actual resources you can consume on any one node is: | + | |
- | * 14 Cores, 14 threads | + | |
- | * 62GB RAM | + | |
- | + | ||
- | ==== Storage ==== | + | |
- | There is slow scratch space mounted to '' | + | |
- | + | ||
- | * Files older than 90 days will be deleted automatically. | + | |
- | * Scratch space is shared by all users. | + | |
- | + | ||
- | === Access === | + | |
- | Scratch space is only mounted on nodes associated with the cluster. If you want to be able to transfer files to the scratch space you will want to run an [[techstaff: | + | |
- | + | ||
- | - You should only do a file transfer via the debug partition: '' | + | |
- | - Now you can create a directory of your own: '' | + | |
- | + | ||
- | == Example == | + | |
- | + | ||
- | Request interactive shell | + | |
- | < | + | |
- | + | ||
- | Change into my scratch directory: | + | |
- | < | + | |
- | + | ||
- | Get the files I need: | + | |
- | < | + | |
- | user@research2:/ | + | |
- | foo | + | |
- | </ | + | |
- | Check that the file now exists: | + | |
- | < | + | |
- | user@research2:/ | + | |
- | -rw------- 1 user user 105121 Dec 29 2015 foo | + | |
- | </ | + | |
- | + | ||
- | I can now exit my interactive shell. | + | |
- | + | ||
- | == Performance is slow == | + | |
- | This is expected. The maximum speed this server will ever be able to achieve is 1Gb/s because of its single 1G ethernet uplink. If this cluster gains in popularity we plan on upgrading the network and storage server. | + | |
- | ==== Utilization Dashboard ==== | + | |
- | Sometimes it is useful to see how much of the cluster is utilized. You can do that via the following URL: http:// | + | |
- | + | ||
- | ==== Partitions / Queues ==== | + | |
- | To find out what partitions we offer, checkout the [[techstaff: | + | |
- | + | ||
- | As of December, 2015 we have will have at least 2 partitions in our cluster; ' | + | |
- | + | ||
- | ^ Partition Name ^ Description ^ | + | |
- | | **debug** | The partition your job will be submitted to if none is specified. The purpose of this partition is to make sure your code is running as it should before submitting a long running job to the general queue. | | + | |
- | | **general** | All jobs that have been thoroughly tested can be submitted here. This partition will have access to more nodes and will process most of the jobs. If you need to use the '' | + | |
- | | **gpu** | Contains servers with graphics cards. As of May 2016 there is only one node containing a Tesla M2090. You will be forced to use this server exclusively for now. Please keep your time in interactive mode to a minimum.| | + | |
- | + | ||
- | ====== Job Submission ====== | + | |
- | Jobs submitted to the cluster are run from the command line. Almost anything that you can run via the command line on any of our machines in our labs can be run on our job submission server agents. | + | |
- | + | ||
- | The job submission servers run Ubuntu 14.04 with the same software as you will find on our lab computers, but without the X environment. | + | |
- | + | ||
- | You can submit jobs from the departmental computers that you have access to. You will not be able to access the job server agent directly. | + | |
- | + | ||
- | ===== Command Summary ===== | + | |
- | [[http:// | + | |
- | | ^ SLURM ^ Example ^ | + | |
- | ^ Submit a batch serial job | sbatch | sbatch runscript.sh | | + | |
- | ^ Run a script interatively | srun | srun --pty -p interact -t 10 --mem 1000 \\ /bin/bash \\ / | + | |
- | ^ Kill a job | scancel | scancel 4585 | | + | |
- | ^ View status of queues | squeue | squeue -u cnetid | | + | |
- | ^ Check current job by id | sacct | sacct -j 999999 | | + | |
- | + | ||
- | + | ||
- | ===== Usage ===== | + | |
- | Below are some common examples. You should consult the [[http:// | + | |
- | + | ||
- | === Exclusive access to a node === | + | |
- | You will need to add the '' | + | |
- | + | ||
- | ==== sbatch ==== | + | |
- | The sbatch command is used for submitting jobs to the cluster. sbatch accepts a number of options either from the command line, or (more typically) from a batch script. An example of a SLURM batch script is shown below: | + | |
- | + | ||
- | === Sample script === | + | |
- | Make sure you create a directory in which to deposit the '' | + | |
- | mkdir -p $HOME/ | + | |
- | + | ||
- | < | + | |
- | # | + | |
- | # | + | |
- | #SBATCH --mail-user=cnetid@cs.uchicago.edu | + | |
- | #SBATCH --mail-type=ALL | + | |
- | #SBATCH --output=/ | + | |
- | #SBATCH --error=/ | + | |
- | #SBATCH --workdir=/ | + | |
- | #SBATCH --partition=debug | + | |
- | #SBATCH --job-name=check_hostname_of_node | + | |
- | #SBATCH --nodes=1 | + | |
- | #SBATCH --ntasks=1 | + | |
- | #SBATCH --time=15: | + | |
- | + | ||
- | hostname | + | |
- | </ | + | |
- | + | ||
- | If any of the above options are unclear as to what they do please check the man page for '' | + | |
- | man sbatch | + | |
- | + | ||
- | Make sure to replace all instances of the word '' | + | |
- | + | ||
- | === Submitting job script === | + | |
- | Using the above example you will want to place your tested code into a file. ' | + | |
- | < | + | |
- | sbatch hostname.job | + | |
- | </ | + | |
- | + | ||
- | You can then check the status via squeue or see the output in the output directory ' | + | |
- | ==== srun ==== | + | |
- | Used to submit a job to the cluster that doesn' | + | |
- | < | + | |
- | user@host: | + | |
- | research2 | + | |
- | research2 | + | |
- | </ | + | |
- | + | ||
- | '' | + | |
- | < | + | |
- | user@host: | + | |
- | </ | + | |
- | + | ||
- | ==== squeue ==== | + | |
- | This command will show jobs in the queue. | + | |
- | + | ||
- | < | + | |
- | user@host: | + | |
- | JOBID PARTITION | + | |
- | | + | |
- | </ | + | |
- | + | ||
- | ==== scancel ==== | + | |
- | Cancel one of your own jobs. Please read the '' | + | |
- | + | ||
- | < | + | |
- | scancel 29 | + | |
- | </ | + | |
- | + | ||
- | ==== sinfo ==== | + | |
- | View information about SLURM nodes and partitions. | + | |
- | + | ||
- | The following code block shows the what happens when you run the '' | + | |
- | < | + | |
- | user@host: | + | |
- | PARTITION AVAIL TIMELIMIT | + | |
- | debug* | + | |
- | general | + | |
- | </ | + | |
- | + | ||
- | + | ||
- | ====== Monitoring Jobs ====== | + | |
- | + | ||
- | '' | + | |
- | + | ||
- | Running '' | + | |
- | squeue -u cnetid | + | |
- | + | ||
- | or for a particular job id. | + | |
- | squeue -j 7894 | + | |
- | + | ||
- | + | ||
- | ====== Interactive Jobs ====== | + | |
- | + | ||
- | Though batch submission is the best way to take full advantage of the compute power in the job submission cluster, foreground, interactive jobs can also be run. | + | |
- | + | ||
- | An interactive job differs from a batch job in two important aspects: | + | |
- | - The partition to be used is the interact partition | + | |
- | - Jobs should be initiated with the srun command instead of sbatch. | + | |
- | + | ||
- | This command: | + | |
- | srun -p general --pty --cpus-per-task 1 --mem 500 -t 0-06:00 /bin/bash | + | |
- | will start a command line shell ('' | + | |
- | ====== Job Scheduling ====== | + | |
- | + | ||
- | We use a [[http:// | + | |
- | + | ||
- | We also have backfill turned on. This allows for jobs which are smaller to sneak in while a larger higher priority job is waiting for nodes to free up. If your job can run in the amount of time it takes for the other job to get all the nodes it needs, SLURM will schedule you to run during that period. **This means knowing how long your code will run for is very important and must be declared if you wish to leverage this feature. Otherwise the scheduler will just assume you will use the maximum allowed time for the partition when you run.**((https:// | + | |
- | + | ||
- | + | ||
- | ====== Common Issues ====== | + | |
- | + | ||
- | ^Error ^What does it mean? ^ | + | |
- | | JOB < | + | |
- | | Job < | + | |
- | | JOB < | + | |
- | | error: Unable to allocate resources: More processors requested than permitted | It usually has **nothing** to do with priviledges you may or may not have. Rather, it usually means that you have allocated more processors than one compute node actually has. | | + | |
- | + | ||
- | ====== Using the GPU ====== | + | |
- | ===== Paths ===== | + | |
- | You will need to add the following to your $PATH and $LD_LIBRARY_PATH. | + | |
- | + | ||
- | export PATH=$PATH:/ | + | |
- | export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/ | + | |
- | + | ||
- | + | ||
- | ===== Example ===== | + | |
- | This sbatch script will get device information from the installed Tesla gpu. | + | |
- | < | + | |
- | # | + | |
- | # | + | |
- | #SBATCH --mail-user=cnetid@cs.uchicago.edu | + | |
- | #SBATCH --mail-type=ALL | + | |
- | #SBATCH --output=/ | + | |
- | #SBATCH --error=/ | + | |
- | #SBATCH --workdir=/ | + | |
- | #SBATCH --partition=gpu | + | |
- | #SBATCH --job-name=get_tesla_info | + | |
- | + | ||
- | export PATH=$PATH:/ | + | |
- | export LD_LIBRARY_PATH=$LD_LIBRARY_PATH=/ | + | |
- | + | ||
- | cat << EOF > / | + | |
- | #include < | + | |
- | + | ||
- | int main() { | + | |
- | int nDevices; | + | |
- | + | ||
- | cudaGetDeviceCount(& | + | |
- | for (int i = 0; i < nDevices; i++) { | + | |
- | cudaDeviceProp prop; | + | |
- | cudaGetDeviceProperties(& | + | |
- | printf(" | + | |
- | printf(" | + | |
- | printf(" | + | |
- | | + | |
- | printf(" | + | |
- | | + | |
- | printf(" | + | |
- | | + | |
- | } | + | |
- | } | + | |
- | EOF | + | |
- | + | ||
- | / | + | |
- | / | + | |
- | rm / | + | |
- | rm / | + | |
- | </ | + | |
- | ==== Output ==== | + | |
- | STDOUT will look something like this: | + | |
- | < | + | |
- | cnetid@linux1: | + | |
- | Device Number: 0 | + | |
- | Device name: Tesla M2090 | + | |
- | Memory Clock Rate (KHz): 1848000 | + | |
- | Memory Bus Width (bits): 384 | + | |
- | Peak Memory Bandwidth (GB/s): 177.408000 | + | |
- | </ | + | |
- | STDERR should be blank. | + | |
- | ====== More ====== | + | |
- | If you feel this documentation is lacking in some way please let techstaff know. Email [[techstaff@cs.uchicago.edu]], | + |
/var/lib/dokuwiki/data/pages/techstaff/slurm.txt · Last modified: 2021/01/06 16:13 by kauffman