TACC tutorials

[dev-onejun]

This is a simple workflow with an example. Please refer the following references, if you need more advanced information

References

• https://docs.tacc.utexas.edu/hpc/lonestar6
• https://www.youtube.com/watch?v=cV88q7k8MGo&list=PLTP5tdFMXQ36acgNifo23ubaH5a6FSnVF&index=2

[dev-onejun]

1. Login to the Lonestar6 machine

$ ssh {YOUR_TACC_PORTAL_ID}@ls6.tacc.utexas.edu

You might need a portal password or 2EA tokens from 3rd parties, based on your settings on the portal.
A default setting as I recalled is a portal password.

1.5 Prepare your environments

• Install your environments before connecting to GPU sessions. Logging in to GPU sessions is a cost in itself.
• I recommend to use a path under cd $SCRATCH to locate your code and data

References

• https://docs.tacc.utexas.edu/hpc/lonestar6/#table3

[dev-onejun]

IMPORTANT: Commands in Section 2 and 3 consume assigned resources to us. Please use them with a caution.

Choose which node you will use among gpu-a100-small, gpu-a100-dev, gpu-a100, etc;
$\bullet$ gpu-a100-small: 1 A100 GPU
$\bullet$ gpu-a100-dev: 3 A100 GPUs, only can get 2 hours sessions (test purpose, I guess)
$\bullet$ gpu-a100: 3 A100 GPUs

2. Interactive Shell

The Interactive Shell on TACC is similar with using docker exec -it in a container environment. It allows you to interact directly with the compute nodes for testing, debugging, or experimenting with your code before formally submitting a job with sbatch.

Use Cases
$\bullet$ Code Testing: Run your scripts interactively to ensure they work as expected within the TACC environment as well.
$\bullet$ Debugging: Diagnose issues in the code or environment setup.
$\bullet$ Parameter tuning: For instance, determine the optimal 'batch size' for your work.

This section, Step 2, can be skipped if you are already confident that your code executes correctly in the TACC environment. Note that, however, although you already ran your code on our lab machine, it does not guarantee that your code runs well in TACC as well.

Example Command

$ idev -p gpu-a100-small -N 1 -n 1 -t 00:30:00

• idev: a command to launch an interactive session.
• gpu-a100-small: Specifies a node to use
• -N and -n: Use 1
• 00:30:00: Set a time, hh:mm:ss. (30 minutes now)

References

• https://docs.tacc.utexas.edu/software/idev/#works
• https://docs.tacc.utexas.edu/hpc/lonestar6/#table5

[dev-onejun]

3. SBATCH

• It is kind a shell script, but utilized for a reservation of gpu works

#!/bin/bash

#SBATCH -J llmixer          # Job name
#SBATCH -o llmixer.o%j    # Name of stdout output file
#SBATCH -e llmixer.e%j    # Name of stderr error file
#SBATCH -p gpu-a100       # Queue (partition) name
#SBATCH -N 1               # Total # of nodes (must be 1 for serial)
#SBATCH -n 1               # Total # of mpi tasks (should be 1 for serial)
#SBATCH -t 48:00:00        # Run time (hh:mm:ss)
#SBATCH --mail-type=all    # Send email at begin and end of job
#SBATCH --mail-user=wxp7177@mavs.uta.edu

cd $SCRATCH

. miniconda3/bin/activate
conda activate emnlp

cd EMNLP-2025/experiments

pwd
date

torchrun --nproc_per_node=3 train.py --model_type AudioSep --text_tokenizer gpt2 --annotated_co_occur C2 --num_workers 36 --batch_size 18 --mixer llm --workspace workspace/AudioSep-CS6_LLMixer-Mix+Distance
# or if you use 'gpu-a100-small' which has 1 GPU, (same as we use our machine)
# python train.py --model_type AudioSep --text_tokenizer gpt2 --annotated_co_occur C2 --num_workers 36 --batch_size 18 --mixer llm --workspace workspace/AudioSep-CS6_LLMixer-Mix+Distance

• Each #SBATCH line is a configuration of your GPU work
• After configuration, all other things are same as using a shell script

1. To submit a work (reserve a work to run),

$ sbatch llmixer.sh # Let's say the script above is named llmixer.sh

2. You can check the status of your work

$ squeue -u {YOUR_ID}

• PD: waiting to start
• `R': running

3. If you want to cancel your job

$ scancel {ASSIGNED_JOB_NUM}

References

• https://docs.tacc.utexas.edu/hpc/lonestar6/#jobs-other

[dev-onejun]

Please leave a question if you still have doubts about this.