Cluster-Software

Software

Matlab

In order to use matlab, you have to load the matlab environment:

 module load matlab/R2013a

Once the matlab environment is loaded, you can start a matlab session by running

 matlab -nodesktop

An example SLURM script for running matlab code is

 #!/bin/bash -l
 #SBATCH -p cortex
 #SBATCH --time=03:30:00
 #SBATCH --mem-per-cpu=2G
 module load matlab/R2013a
 matlab -nodesktop -r "scriptname. $variable1 $variable2"

The above script takes a matlab job with scriptname = scriptname and accepts two variables $variable1 and $variable2

If you would like to see who is using matlab licenses, enter

 lmstat

Python

Anaconda Python Distribution

The Anaconda Python 2.7 or 3.4 Distributions can be loaded through

 module load python/anaconda2

or

 module load python/anaconda3

respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an academic license from Continuum and copy it to the cluster.

On the cluster

 cd
 mkdir .continuum

On the machine where you downloaded the license file

 scp file_name username@hpc.brc.berkeley.edu:/global/home/users/username/.continuum/.

Local Install of Anaconda Python Distribution

If you want to manage your own python distribution the Anaconda Python is a very good distribution. To get it, go the the Continuum downloads page and select the linux distribution (penguin). Copy the download link address, and then in a terminal on the cluster run:

 wget paste_link_here

This should download a .sh file that can be run with

 bash Anaconda-version_info.sh

CUDA

CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing. We have a separate wiki page to collect information on how to do general-purpose computing on the GPU: GPGPU. The --constraint={cortex_k40, cortex_fermi} option must be used in order to schedule a node with a GPU. We have installed the CUDA 6.5 driver and toolkit.

In order to use CUDA, you have to load the CUDA environment:

 module load cuda

Using Theano

By default, Theano expects the default compiler to be gcc, so you'll need to unload the intel compiler.

 module unload intel

Theano caches certain compiled libraries and these will sometimes cause errors when Theano gets updated. If you are experiencing problems with Theano, you can try clearing the cache with

 theano-cache clear

and if you still have problems you can delete the .theano folder from your home directory.

Using the GPU

You must request a GPU node. The Anaconda Python distribution comes with a version of Theano that should work. If you need new Theano features, the development version of Theano can be obtained from the github repository, installed locally, and added to your PYTHONPATH if you are using the preinstalled Python verions. If you have a local python install you can install theano with

 python setup.py develop

from the repository folder. Theano must be configured to use the GPU. General information can be found in the Theano documentation, but a working (June 2015) version is to create a .theanorc file in your HOME directory with the contents:

 [global]
 root = /global/software/sl-6.x86_64/modules/langs/cuda/6.5/
 device = gpu
 floatX = float32
 force_device=True
 
 [nvcc]
 fastmath = True

Using the CPU

Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano build against the MKL BLAS library that comes with Anaconda and so your .theanorc might look like

 [global]
 device = cpu
 floatX = float32
 ldflags = -lmkl_rt

Obtain GPU lock for Jacket in Matlab

If you are using Matlab, you can obtain a GPU lock by running

 addpath('/clusterfs/cortex/software/gpu_lock');
 addpath('/clusterfs/cortex/software/jacket/engine');
 gpu_id = obtain_gpu_lock_id();
 gselect(gpu_id);

By default, obtain_gpu_lock() throws an error when all gpu cards are taken. There is another option: obtain_gpu_lock_id(true) will return -1 in case there is no card available and you can then write your own code to deal with that fact.

ginfo tells you which gpu card you are using.

The following lines should also be in your .bashrc

 ## jacket stuff!
 module load cuda
 export LD_LIBRARY_PATH=/clusterfs/cortex/software/jacket/engine/lib64:$LD_LIBRARY_PATH