Cluster Job Management - Revision history

Gisely: /* Common SLURM arguments */

2015-09-12T00:26:07Z

Common SLURM arguments

← Older revision		Revision as of 00:18, 12 September 2015
Line 39:		Line 39:
	-e errorfile.txt		-e errorfile.txt

	* Monitoring Jobs		== Monitoring Jobs ==

	~~Additional options can be passed to sbatch to monitor outputs from the running jobs~~

	~~sbatch -o outputfile.txt -e errofile.txt -J jobdescriptor myscript.sh~~

	~~the output of the job will be piped to outputfile.txt and any errors if the job crashes to errofile.txt~~

	* Cluster usage

	Use		Use
Line 53:		Line 45:
	to get a list of pending and running jobs on the cluster. It will show user names jobdescriptor passed to sbatch, runtime and nodes.		to get a list of pending and running jobs on the cluster. It will show user names jobdescriptor passed to sbatch, runtime and nodes.


			== Interactive Sessions ==

	To start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here):		To start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here):

Gisely: /* Basic job submission */

2015-09-12T00:24:30Z

Basic job submission

← Older revision		Revision as of 00:16, 12 September 2015
Line 20:		Line 20:

	#!/bin/bash -l		#!/bin/bash -l
	#SBATCH -~~outputfile~~ myscript.sh		#SBATCH -o myscript.sh
	...		...

	Where ... is the body of script.		Where ... is the body of your script.

			== Common SLURM arguments ==

	the --~~time defines the~~ walltime ~~of the job, which is~~ an upper bound on ~~the estimated runtime~~. ~~The~~ job ~~will be killed after this~~ time ~~is elapsed.~~ --~~mem specifies how much memory~~ the job ~~requires, the default is 1GB per job~~.		It's a good idea to assign your job a name. This helps make your job identifiable when using other command such as squeue (see below). To do this use the -J argument:

			-J YOUR_JOB_NAME

			Good cluster etiquette dictates that you set a walltime (i.e. an upper bound on how long it can run) on your script. This helps SLURM fairly schedule jobs. For example to limit your job to one hour of execution time use

			--time=01:00:00

			In addition capturing standard output (-o) of your process you can also capture the standard error (-e) output (e.g. if running job causes compile errors):

			-e errorfile.txt

	* Monitoring Jobs		* Monitoring Jobs

Gisely: /* Basic job submission */

2015-09-12T00:13:38Z

Basic job submission

← Older revision		Revision as of 00:05, 12 September 2015
Line 13:		Line 13:
	sbatch myscript.sh		sbatch myscript.sh

	where the myscript.sh is a shell script that executes your program.		where the myscript.sh is a shell script that executes your program. Slurm has several parameters that need to specified. For example, you might like to specify an output file for your job. This can done using the -o argument:

	~~Slurm has several parameters that need to specified~~. ~~For example, you might like to specify an output file for your job~~. ~~This can done using the -o argument:~~		sbatch -o outputfile.txt myscript.sh

	~~sbatch -o outputfile~~.~~txt myscript~~.sh		Remembering to include all the arguments you need can get cumbersome. An easier way is to include the slurm parameters in the header of the script that you submit. This done by putting one or more lines starting with #SBATCH near the start of your file. For example, here is simple script that does specifies output file in the script (rather than as command line argument):

	#!/bin/bash -l		#!/bin/bash -l
	#SBATCH -~~p cortex~~		#SBATCH -outputfile myscript.sh
	~~#SBATCH --time=03:30:00~~		...
	~~#SBATCH --mem-per-cpu=2G~~
	~~cd /clusterfs/cortex/scratch/working/dir/for/your/code~~		Where ... is the body of script.
	~~module load matlab/R2013a~~
	~~matlab -nodisplay -nojvm -r "mymatlabfunction( parameters); exit"~~
	~~exit~~

	the --time defines the walltime of the job, which is an upper bound on the estimated runtime. The job will be killed after this time is elapsed. --mem specifies how much memory the job requires, the default is 1GB per job.		the --time defines the walltime of the job, which is an upper bound on the estimated runtime. The job will be killed after this time is elapsed. --mem specifies how much memory the job requires, the default is 1GB per job.

Gisely: /* SLURM usage */

2015-09-11T23:47:21Z

SLURM usage

← Older revision		Revision as of 23:39, 11 September 2015
Line 7:		Line 7:
	Full description of our system by the LBL folks is at http://go.lbl.gov/hpcs-user-svcs/ucb-supercluster/cortex		Full description of our system by the LBL folks is at http://go.lbl.gov/hpcs-user-svcs/ucb-supercluster/cortex

	== ~~SLURM usage~~ ==		== Basic job submission ==

	* Submitting a Job

	From the login node, you can submit jobs to the compute nodes using the syntax		From the login node, you can submit jobs to the compute nodes using the syntax
Line 15:		Line 13:
	sbatch myscript.sh		sbatch myscript.sh

	where the myscript.sh is an shell script ~~containing the call~~ to ~~the executable to be submitted to the cluster~~. ~~Typically~~, for ~~a matlab~~ job~~, it would look like~~		where the myscript.sh is a shell script that executes your program.

			Slurm has several parameters that need to specified. For example, you might like to specify an output file for your job. This can done using the -o argument:

			sbatch -o outputfile.txt myscript.sh

	#!/bin/bash -l		#!/bin/bash -l

Gisely: /* Overview */

2015-09-11T23:37:08Z

Overview

← Older revision		Revision as of 23:29, 11 September 2015
Line 7:		Line 7:
	Full description of our system by the LBL folks is at http://go.lbl.gov/hpcs-user-svcs/ucb-supercluster/cortex		Full description of our system by the LBL folks is at http://go.lbl.gov/hpcs-user-svcs/ucb-supercluster/cortex

	=== SLURM usage ===		== SLURM usage ==

	* Submitting a Job		* Submitting a Job
Line 47:		Line 47:
	srun -u -p cortex -t 2:0:0 --pty bash -i		srun -u -p cortex -t 2:0:0 --pty bash -i

	=== Perceus commands ===		== Perceus commands ==

	The perceus manual is [http://www.warewulf-cluster.org/portal/book/export/html/7 here]		The perceus manual is [http://www.warewulf-cluster.org/portal/book/export/html/7 here]
Line 70:		Line 70:
	* help pages are [http://lrc.lbl.gov/html/guide.html here]		* help pages are [http://lrc.lbl.gov/html/guide.html here]

	=== Finding out the list of occupants on each cluster node ===		== Finding out the list of occupants on each cluster node ==

	* One can find out the list of users using a particular node by ssh into the node, e.g.		* One can find out the list of users using a particular node by ssh into the node, e.g.

Gisely: /* Useful commands */

2015-09-11T23:36:15Z

Useful commands

← Older revision		Revision as of 23:28, 11 September 2015
Line 1:		Line 1:
	== ~~Useful commands~~ ==		==Overview==

			In order to coordinate our cluster usage patterns fairly, our cluster uses a job manager known as SLURM. This page provides some information about common usage patterns for SLURM and cluster etiquette.

	See https://sites.google.com/a/lbl.gov/high-performance-computing-services-group/scheduler/ucb-supercluster-slurm-migration for a detailed FAQ on the SLURM job manager.		See https://sites.google.com/a/lbl.gov/high-performance-computing-services-group/scheduler/ucb-supercluster-slurm-migration for a detailed FAQ on the SLURM job manager.

Gisely: Created page with "== Useful commands == See https://sites.google.com/a/lbl.gov/high-performance-computing-services-group/scheduler/ucb-supercluster-slurm-migration for a detailed FAQ on the SL..."

2015-09-11T23:30:20Z

Created page with "== Useful commands == See https://sites.google.com/a/lbl.gov/high-performance-computing-services-group/scheduler/ucb-supercluster-slurm-migration for a detailed FAQ on the SL..."

New page

== Useful commands ==

See https://sites.google.com/a/lbl.gov/high-performance-computing-services-group/scheduler/ucb-supercluster-slurm-migration for a detailed FAQ on the SLURM job manager.

Full description of our system by the LBL folks is at http://go.lbl.gov/hpcs-user-svcs/ucb-supercluster/cortex

=== SLURM usage ===

* Submitting a Job

From the login node, you can submit jobs to the compute nodes using the syntax

sbatch myscript.sh

where the myscript.sh is an shell script containing the call to the executable to be submitted to the cluster. Typically, for a matlab job, it would look like

#!/bin/bash -l
#SBATCH -p cortex
#SBATCH --time=03:30:00
#SBATCH --mem-per-cpu=2G
cd /clusterfs/cortex/scratch/working/dir/for/your/code
module load matlab/R2013a
matlab -nodisplay -nojvm -r "mymatlabfunction( parameters); exit"
exit

the --time defines the walltime of the job, which is an upper bound on the estimated runtime. The job will be killed after this time is elapsed. --mem specifies how much memory the job requires, the default is 1GB per job.

* Monitoring Jobs

Additional options can be passed to sbatch to monitor outputs from the running jobs

sbatch -o outputfile.txt -e errofile.txt -J jobdescriptor myscript.sh

the output of the job will be piped to outputfile.txt and any errors if the job crashes to errofile.txt

* Cluster usage

Use
squeue
to get a list of pending and running jobs on the cluster. It will show user names jobdescriptor passed to sbatch, runtime and nodes.

To start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here):

srun -u -p cortex -t 2:0:0 --pty bash -i

=== Perceus commands ===

The perceus manual is [http://www.warewulf-cluster.org/portal/book/export/html/7 here]

* listing available cluster nodes:

wwstats
wwnodes

* list cluster usage

wwtop

* to restrict the scope of these commands to cortex cluster, add the following line to your .bashrc

export NODES='*cortex'

* module list
* module avail
* module help

* help pages are [http://lrc.lbl.gov/html/guide.html here]

=== Finding out the list of occupants on each cluster node ===

* One can find out the list of users using a particular node by ssh into the node, e.g.

ssh n0000.cortex

* After logging into the node, type

top

* This is useful if you believe someone is abusing the machine and would like to send him/her a friendly reminder.