Cluster-Software - Revision history

Jesselivezey: /* Using Theano */

2016-10-12T18:00:00Z

Using Theano

← Older revision		Revision as of 17:52, 12 October 2016
Line 69:		Line 69:
	module unload intel		module unload intel

	Theano caches certain compiled functions and these will sometimes cause errors when Theano or ~~CUDAgets~~ updated. If you are experiencing problems with Theano, you can try clearing the cache with		Theano caches certain compiled functions and these will sometimes cause errors when Theano or CUDA gets updated. If you are experiencing problems with Theano, you can try clearing the cache with
	theano-cache clear		theano-cache clear
	and if you still have problems you can delete the .theano folder from your home directory.		and if you still have problems you can delete the .theano folder from your home directory.
Line 81:		Line 81:

	[global]		[global]
	~~root = /global/software/sl-6.x86_64/modules/langs/cuda/6.5/~~
	device = gpu		device = gpu
	floatX = float32		floatX = float32
Line 91:		Line 90:
	==== Using the CPU ====		==== Using the CPU ====

	Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano ~~build against~~ the MKL BLAS library that comes with Anaconda and so your .theanorc might look like		Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano use the MKL BLAS library that comes with Anaconda and so your .theanorc might look like

	[global]		[global]

Jesselivezey: /* CUDA */

2016-10-12T17:59:00Z

CUDA

← Older revision		Revision as of 17:51, 12 October 2016
Line 58:		Line 58:
	#SBATCH --gres=gpu:1		#SBATCH --gres=gpu:1

	options must be used in order to schedule a node with a GPU~~. We have installed the CUDA 6.5 driver and toolkit~~.		options must be used in order to schedule a node with a GPU.

	In order to use CUDA, you have to load the CUDA environment:		In order to use CUDA, you have to load the CUDA environment:
Line 69:		Line 69:
	module unload intel		module unload intel

	Theano caches certain compiled ~~libraries~~ and these will sometimes cause errors when Theano ~~gets~~ updated. If you are experiencing problems with Theano, you can try clearing the cache with		Theano caches certain compiled functions and these will sometimes cause errors when Theano or CUDAgets updated. If you are experiencing problems with Theano, you can try clearing the cache with
	theano-cache clear		theano-cache clear
	and if you still have problems you can delete the .theano folder from your home directory.		and if you still have problems you can delete the .theano folder from your home directory.

Jesselivezey: /* CUDA */

2016-10-12T17:56:05Z

CUDA

← Older revision		Revision as of 17:48, 12 October 2016
Line 52:		Line 52:
	== CUDA ==		== CUDA ==

	CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing~~. We have a separate wiki page to collect information on how to do general-purpose computing on the GPU: [[GPGPU]]~~.		CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing.
	The --constraint={cortex_k40, cortex_fermi} ~~option~~ must be used in order to schedule a node with a GPU.		The
	We have installed the CUDA 6.5 driver and toolkit.
			#SBATCH--constraint={cortex_k40, cortex_fermi}
			#SBATCH --gres=gpu:1

			options must be used in order to schedule a node with a GPU. We have installed the CUDA 6.5 driver and toolkit.

	In order to use CUDA, you have to load the CUDA environment:		In order to use CUDA, you have to load the CUDA environment:

Gisely: /* Obtain GPU lock for Jacket in Matlab */

2015-09-11T23:16:16Z

Obtain GPU lock for Jacket in Matlab

← Older revision		Revision as of 23:08, 11 September 2015
Line 93:		Line 93:
	floatX = float32		floatX = float32
	ldflags = -lmkl_rt		ldflags = -lmkl_rt

	~~=== Obtain GPU lock for Jacket in Matlab ===~~

	~~If you are using Matlab, you can obtain a GPU lock by running~~

	~~addpath('/clusterfs/cortex/software/gpu_lock');~~
	~~addpath('/clusterfs/cortex/software/jacket/engine');~~
	~~gpu_id = obtain_gpu_lock_id();~~
	~~gselect(gpu_id);~~

	~~By default, obtain_gpu_lock() throws an error when all gpu cards are taken.~~
	~~There is another option: obtain_gpu_lock_id(true) will return -1 in case there~~
	~~is no card available and you can then write your own code to deal with that~~
	~~fact.~~

	~~ginfo tells you which gpu card you are using.~~

	~~The following lines should also be in your .bashrc~~

	~~## jacket stuff!~~
	~~module load cuda~~
	~~export LD_LIBRARY_PATH=/clusterfs/cortex/software/jacket/engine/lib64:$LD_LIBRARY_PATH~~

Jesselivezey: /* Anaconda Python Distribution */

2015-09-11T23:11:41Z

Anaconda Python Distribution

← Older revision		Revision as of 23:03, 11 September 2015
Line 30:		Line 30:

	The Anaconda Python 2.7 or 3.4 Distributions can be loaded through		The Anaconda Python 2.7 or 3.4 Distributions can be loaded through
	module load python~~/anaconda2~~/anaconda2		module load python/anaconda2
	or		or
	module load python~~/anaconda3~~/anaconda3		module load python/anaconda3
	respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.		respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.

Gisely: /* Obtain GPU lock in python */

2015-09-11T23:04:35Z

Obtain GPU lock in python

@@ Line 93: / Line 93: @@
    floatX = float32
    ldflags = -lmkl_rt
 === Obtain GPU lock for Jacket in Matlab ===

Gisely: /* Matlab */

2015-09-11T23:00:35Z

Matlab

← Older revision		Revision as of 22:52, 11 September 2015
Line 2:		Line 2:

	== Matlab ==		== Matlab ==

	~~Start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here):~~

	~~srun -u -p cortex -t 2:0:0 --pty bash -i~~

	In order to use matlab, you have to load the matlab environment:		In order to use matlab, you have to load the matlab environment:

Gisely: Created page with "= Software = == Matlab == Start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here): srun -u -p cortex -t 2:0:0 --pty ba..."

2015-09-11T22:53:57Z

Created page with "= Software = == Matlab == Start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here): srun -u -p cortex -t 2:0:0 --pty ba..."

New page

= Software =

== Matlab ==

Start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here):

srun -u -p cortex -t 2:0:0 --pty bash -i

In order to use matlab, you have to load the matlab environment:

module load matlab/R2013a

Once the matlab environment is loaded, you can start a matlab session by running

matlab -nodesktop

An example SLURM script for running matlab code is

#!/bin/bash -l
#SBATCH -p cortex
#SBATCH --time=03:30:00
#SBATCH --mem-per-cpu=2G
module load matlab/R2013a
matlab -nodesktop -r "scriptname. $variable1 $variable2"

The above script takes a matlab job with scriptname = scriptname and accepts two variables $variable1 and $variable2

If you would like to see who is using matlab licenses, enter

lmstat

== Python ==
=== Anaconda Python Distribution ===

The Anaconda Python 2.7 or 3.4 Distributions can be loaded through
module load python/anaconda2/anaconda2
or
module load python/anaconda3/anaconda3
respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.

On the cluster
cd
mkdir .continuum

On the machine where you downloaded the license file
scp file_name username@hpc.brc.berkeley.edu:/global/home/users/username/.continuum/.

=== Local Install of Anaconda Python Distribution ===
If you want to manage your own python distribution the Anaconda Python is a very good distribution. To get it, go the the [http://continuum.io/downloads Continuum downloads] page and select the linux distribution (penguin).
Copy the download link address, and then in a terminal on the cluster run:

wget paste_link_here
This should download a .sh file that can be run with
bash Anaconda-version_info.sh

== CUDA ==

CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing. We have a separate wiki page to collect information on how to do general-purpose computing on the GPU: [[GPGPU]].
The --constraint={cortex_k40, cortex_fermi} option must be used in order to schedule a node with a GPU.
We have installed the CUDA 6.5 driver and toolkit.

In order to use CUDA, you have to load the CUDA environment:

module load cuda

=== Using Theano ===
By default, Theano expects the default compiler to be gcc, so you'll need to unload the intel compiler.

module unload intel

Theano caches certain compiled libraries and these will sometimes cause errors when Theano gets updated. If you are experiencing problems with Theano, you can try clearing the cache with
theano-cache clear
and if you still have problems you can delete the .theano folder from your home directory.

==== Using the GPU ====

You must request a GPU node. The Anaconda Python distribution comes with a version of Theano that should work. If you need new Theano features, the development version of Theano can be obtained from the [https://github.com/Theano/Theano github repository], installed locally, and added to your PYTHONPATH if you are using the preinstalled Python verions. If you have a local python install you can install theano with
python setup.py develop
from the repository folder.
Theano must be configured to use the GPU. General information can be found in the [http://deeplearning.net/software/theano/library/config.html Theano documentation], but a working (June 2015) version is to create a .theanorc file in your HOME directory with the contents:

[global]
root = /global/software/sl-6.x86_64/modules/langs/cuda/6.5/
device = gpu
floatX = float32
force_device=True

[nvcc]
fastmath = True

==== Using the CPU ====

Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano build against the MKL BLAS library that comes with Anaconda and so your .theanorc might look like

[global]
device = cpu
floatX = float32
ldflags = -lmkl_rt

=== Obtain GPU lock in python ===

If you would like to use one of the GPU cards on node n0000 or n0001, please obtain a GPU lock to make sure the card is not in use and that no one else will be using the card.

If you are using Python, you can obtain a GPU lock by running

import gpu_lock
gpu_lock.obtain_lock_id()

The function either returns the number of the card you can use (0 or 1) or -1 if both cards are in use.

=== Obtain GPU lock for Jacket in Matlab ===

If you are using Matlab, you can obtain a GPU lock by running

addpath('/clusterfs/cortex/software/gpu_lock');
addpath('/clusterfs/cortex/software/jacket/engine');
gpu_id = obtain_gpu_lock_id();
gselect(gpu_id);

By default, obtain_gpu_lock() throws an error when all gpu cards are taken.
There is another option: obtain_gpu_lock_id(true) will return -1 in case there
is no card available and you can then write your own code to deal with that
fact.

ginfo tells you which gpu card you are using.

The following lines should also be in your .bashrc

## jacket stuff!
module load cuda
export LD_LIBRARY_PATH=/clusterfs/cortex/software/jacket/engine/lib64:$LD_LIBRARY_PATH