Difference between revisions of "Cluster-Software"

From RedwoodCenter
Jump to navigationJump to search
(Created page with "= Software = == Matlab == Start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here): srun -u -p cortex -t 2:0:0 --pty ba...")
 
 
(6 intermediate revisions by 2 users not shown)
Line 2: Line 2:
  
 
== Matlab ==
 
== Matlab ==
 
Start an interactive session on the cluster (requires specifying the cluster and walltime as is shown here):
 
 
  srun -u -p cortex -t 2:0:0 --pty bash -i
 
  
 
In order to use matlab, you have to load the matlab environment:
 
In order to use matlab, you have to load the matlab environment:
Line 34: Line 30:
  
 
The Anaconda Python 2.7 or 3.4 Distributions can be loaded through
 
The Anaconda Python 2.7 or 3.4 Distributions can be loaded through
   module load python/anaconda2/anaconda2
+
   module load python/anaconda2
 
or
 
or
   module load python/anaconda3/anaconda3
+
   module load python/anaconda3
 
respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.
 
respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.
  
Line 56: Line 52:
 
== CUDA ==
 
== CUDA ==
  
CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing. We have a separate wiki page to collect information on how to do general-purpose computing on the GPU: [[GPGPU]].
+
CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing.
The --constraint={cortex_k40, cortex_fermi} option must be used in order to schedule a node with a GPU.
+
The  
We have installed the CUDA 6.5 driver and toolkit.
+
 
 +
  #SBATCH--constraint={cortex_k40, cortex_fermi}  
 +
  #SBATCH --gres=gpu:1
 +
 
 +
options must be used in order to schedule a node with a GPU.
  
 
In order to use CUDA, you have to load the CUDA environment:
 
In order to use CUDA, you have to load the CUDA environment:
Line 69: Line 69:
 
   module unload intel
 
   module unload intel
  
Theano caches certain compiled libraries and these will sometimes cause errors when Theano gets updated. If you are experiencing problems with Theano, you can try clearing the cache with
+
Theano caches certain compiled functions and these will sometimes cause errors when Theano or CUDA gets updated. If you are experiencing problems with Theano, you can try clearing the cache with
 
   theano-cache clear
 
   theano-cache clear
 
and if you still have problems you can delete the .theano folder from your home directory.
 
and if you still have problems you can delete the .theano folder from your home directory.
Line 81: Line 81:
  
 
   [global]
 
   [global]
  root = /global/software/sl-6.x86_64/modules/langs/cuda/6.5/
 
 
   device = gpu
 
   device = gpu
 
   floatX = float32
 
   floatX = float32
Line 91: Line 90:
 
==== Using the CPU ====
 
==== Using the CPU ====
  
Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano build against the MKL BLAS library that comes with Anaconda and so your .theanorc might look like
+
Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano use the MKL BLAS library that comes with Anaconda and so your .theanorc might look like
  
 
   [global]
 
   [global]
Line 97: Line 96:
 
   floatX = float32
 
   floatX = float32
 
   ldflags = -lmkl_rt
 
   ldflags = -lmkl_rt
 
=== Obtain GPU lock in python ===
 
 
If you would like to use one of the GPU cards on node n0000 or n0001, please obtain a GPU lock to make sure the card is not in use and that no one else will be using the card.
 
 
If you are using Python, you can obtain a GPU lock by running
 
 
  import gpu_lock
 
  gpu_lock.obtain_lock_id()
 
 
The function either returns the number of the card you can use (0 or 1) or -1 if both cards are in use.
 
 
=== Obtain GPU lock for Jacket in Matlab ===
 
 
If you are using Matlab, you can obtain a GPU lock by running
 
 
  addpath('/clusterfs/cortex/software/gpu_lock');
 
  addpath('/clusterfs/cortex/software/jacket/engine');
 
  gpu_id = obtain_gpu_lock_id();
 
  gselect(gpu_id);
 
 
By default, obtain_gpu_lock() throws an error when all gpu cards are taken.
 
There is another option: obtain_gpu_lock_id(true) will return -1 in case there
 
is no card available and you can then write your own code to deal with that
 
fact.
 
 
ginfo tells you which gpu card you are using.
 
 
The following lines should also be in your .bashrc
 
 
  ## jacket stuff!
 
  module load cuda
 
  export LD_LIBRARY_PATH=/clusterfs/cortex/software/jacket/engine/lib64:$LD_LIBRARY_PATH
 

Latest revision as of 17:52, 12 October 2016

Software

Matlab

In order to use matlab, you have to load the matlab environment:

 module load matlab/R2013a

Once the matlab environment is loaded, you can start a matlab session by running

 matlab -nodesktop

An example SLURM script for running matlab code is

 #!/bin/bash -l
 #SBATCH -p cortex
 #SBATCH --time=03:30:00
 #SBATCH --mem-per-cpu=2G
 module load matlab/R2013a
 matlab -nodesktop -r "scriptname. $variable1 $variable2"

The above script takes a matlab job with scriptname = scriptname and accepts two variables $variable1 and $variable2

If you would like to see who is using matlab licenses, enter

 lmstat

Python

Anaconda Python Distribution

The Anaconda Python 2.7 or 3.4 Distributions can be loaded through

 module load python/anaconda2

or

 module load python/anaconda3

respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an academic license from Continuum and copy it to the cluster.

On the cluster

 cd
 mkdir .continuum

On the machine where you downloaded the license file

 scp file_name username@hpc.brc.berkeley.edu:/global/home/users/username/.continuum/.

Local Install of Anaconda Python Distribution

If you want to manage your own python distribution the Anaconda Python is a very good distribution. To get it, go the the Continuum downloads page and select the linux distribution (penguin). Copy the download link address, and then in a terminal on the cluster run:

 wget paste_link_here

This should download a .sh file that can be run with

 bash Anaconda-version_info.sh

CUDA

CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing. The

 #SBATCH--constraint={cortex_k40, cortex_fermi} 
 #SBATCH --gres=gpu:1

options must be used in order to schedule a node with a GPU.

In order to use CUDA, you have to load the CUDA environment:

 module load cuda

Using Theano

By default, Theano expects the default compiler to be gcc, so you'll need to unload the intel compiler.

 module unload intel

Theano caches certain compiled functions and these will sometimes cause errors when Theano or CUDA gets updated. If you are experiencing problems with Theano, you can try clearing the cache with

 theano-cache clear

and if you still have problems you can delete the .theano folder from your home directory.

Using the GPU

You must request a GPU node. The Anaconda Python distribution comes with a version of Theano that should work. If you need new Theano features, the development version of Theano can be obtained from the github repository, installed locally, and added to your PYTHONPATH if you are using the preinstalled Python verions. If you have a local python install you can install theano with

 python setup.py develop

from the repository folder. Theano must be configured to use the GPU. General information can be found in the Theano documentation, but a working (June 2015) version is to create a .theanorc file in your HOME directory with the contents:

 [global]
 device = gpu
 floatX = float32
 force_device=True
 
 [nvcc]
 fastmath = True

Using the CPU

Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano use the MKL BLAS library that comes with Anaconda and so your .theanorc might look like

 [global]
 device = cpu
 floatX = float32
 ldflags = -lmkl_rt