============================
Interactive Session on Savio
============================

# To ssh to a node in savio with cuda, use the following two partitions: savio2_gpu, or savio2_1080ti.
# The two examples below sets time limit of 15 houts, on either savio2_gpu or savio2_1080ti partition, then opens
# bash shell for interactive use.

srun -A fc_biome -p savio2_gpu --gres=gpu:1 --ntasks 2 -t 15:00:00 --pty bash -i
srun -A fc_biome -p savio2_1080ti --gres=gpu:1 --ntasks 2 -t 15:00:00 --pty bash -i

============
Load Modules
============

module load python/3.7
module load cuda    # or module load cuda/10.2

# Check which modules are loaded
module list

# Check which modules are available on the machine
module avail

============================
Install Python cuda packages
============================

# Install pycuda:
python -m pip install --no-cache-dir pycuda

# Install scikit-cuda
pip install scikit-cuda 

# Install cupy
module load cuda/9.0
python -m pip install --no-cache-dir cupy-cuda90

===========================
Install Nvidia Cuda Toolkit
===========================

Best way to install the latest cuda toolkit (currently 11) is through:
https://developer.nvidia.com/cuda-downloads

# Install nvidia drivers
conda install -c nvidia nvcc_linux-64

# Install cuda toolkit on ubuntu (this installs cuda toolkit 9, which does not have many cusparse features)
sudo apt install nvidia-cuda-toolkit
nvcc --version


============
Device Query
============

# Find info about device
deviceQuery

# or
nvidia-smi -q

# or
nvidia-smi


========
Profiler
========

module load cuda/10.2
which nvprof    # should be in /global/software/sl-7.x86_64/modules/langs/cuda/10.2/bin/nvprof

# Run a profiler
nvprof python script.py

# Visual profiler on a remote machine:
	# run the following, which should generate prof.nvvp file
	nvprof -o python script.py

	# Copy nvvp file to a local machine
	scp local_machine:~ prof.nvvp

	# Display profile
	nvvp prof.nvvp