Jumping-Drops

Jumping drops Basilisk simulation with a two-phase workflow: initialization (STL geometry -> dumpInit) and a main MPI-compatible run.

Requirements

• Basilisk (qcc on PATH)
• MPI (optional, for parallel main phase)
• macOS or Linux

Basilisk (Required)

First-time install (or reinstall):

curl -sL https://raw.githubusercontent.com/comphy-lab/basilisk-C/main/reset_install_basilisk-ref-locked.sh | bash -s -- --ref=v2026-01-29 --hard

Subsequent runs (reuses existing basilisk/ if same ref):

curl -sL https://raw.githubusercontent.com/comphy-lab/basilisk-C/main/reset_install_basilisk-ref-locked.sh | bash -s -- --ref=v2026-01-29

Load the environment before running the scripts:

source .project_config

Note: Replace v2026-01-29 with the latest release tag from https://github.com/comphy-lab/basilisk-C/releases.

Quick start

# Full workflow (init + main)
./runSimulation.sh default.params

# Init only (STL -> dumpInit)
./runSimulation.sh --init-only default.params

# Main only (MPI)
./runSimulation.sh --main-only --mpi --cores 8 default.params

# Parameter sweep
./runParameterSweep.sh sweep.params

Overview

This codebase has been refactored from two monolithic files into a modular workflow with a clean separation between initialization and the MPI-compatible main simulation.

Deprecated (do not use)

• simulationCases/JumpingDrops_legacy.c - local initialization with STL
• simulationCases/JumpingDrops_Snellius_legacy.c - HPC version (missing gravity bug)

Current sources

• src-local/jumpingDrops_common.h - shared definitions, tolerances, boundary conditions, and helper functions (refRegion(), adapt(), writingFiles(), logWriting()), with MPI-aware logging via MPI_MODE
• simulationCases/jumpingDrops_init.c - init phase (serial only; uses distance.h and reduced.h) that writes dumpInit
• simulationCases/jumpingDrops_main.c - main simulation (MPI-ready; no distance.h) that restores from dump/dumpInit
• src-local/parse_params.sh - parameter file parsing helpers

File execution summary

File	Purpose	MPI	Input	Output
simulationCases/jumpingDrops_init.c	Create initial condition	No (serial only)	STL file	dumpInit
simulationCases/jumpingDrops_main.c	Run simulation	Yes	dump/dumpInit	snapshots, log

Parameters

• Parameter files are key=value lines.
• Required keys: CaseNo, Oh, Bo, MAXlevel, tmax.
• Sweep files define BASE_CONFIG, CASE_START, CASE_END, and SWEEP_* values.

Parameter file template (required keys)

CaseNo=1000              # 4-digit case number (1000-9999)
Oh=0.001                 # Ohnesorge number
Bo=0.001                 # Bond number
MAXlevel=10              # Maximum refinement level
tmax=10.0                # Maximum simulation time (main phase only)

Example: default.params

# Jumping Drops Simulation Parameters
CaseNo=1000
Oh=0.001
Bo=0.001
MAXlevel=10
tmax=10.0

Script options

runSimulation.sh

--init-only       # Init phase only
--main-only       # Main phase only
--mpi             # Enable MPI
--cores N         # Number of MPI cores (default: 4)
--compile-only    # Compile but don't run
--debug           # Debug mode (-g -DTRASH=1)
--verbose         # Verbose output

runParameterSweep.sh

--skip-init       # Skip init for all cases
--dry-run         # Show combinations, don't run
--mpi             # Enable MPI for all cases
--cores N         # MPI cores per case
--compile-only    # Compile but don't run
--verbose         # Verbose output

Common workflows

Single case development

# First time
./runSimulation.sh case.params

# Subsequent runs (skip init)
./runSimulation.sh --main-only case.params

Parameter sweep

# First sweep (auto-runs init if needed)
./runParameterSweep.sh sweep.params

# Re-run after changes (skip init)
./runParameterSweep.sh --skip-init sweep.params

HPC workflow (Snellius)

Step 1: Local - create initial conditions

for case in 1000 1001 1002; do
    ./runSimulation.sh --init-only "simulationCases/${case}/case.params"
done

Step 2: Local - transfer to HPC

for case in 1000 1001 1002; do
    scp "simulationCases/${case}/dumpInit" \
        "hpc:Drop-Impact/simulationCases/${case}/"
done

Or transfer all at once:

rsync -av --include='**/dumpInit' --include='*/' --exclude='*' \
    simulationCases/ hpc:Drop-Impact/simulationCases/

Step 3: HPC - run sweep

sbatch runSweepSnellius.sbatch

What you need on HPC

Minimum files per case directory:

simulationCases/<CaseNo>/
├── dumpInit              # From local init phase
└── case.params           # Generated by sweep

Auto-created by the sbatch script:

simulationCases/<CaseNo>/
├── jumpingDrops_main.c   # Copied by script
├── jumpingDrops_main     # Compiled executable
└── intermediate/         # Created if needed

Shared header used during compilation:

src-local/jumpingDrops_common.h

Quick troubleshooting

Error	Cause	Fix
"dump file not found"	No initial condition	Run --init-only first
"Cannot restore dump"	Corrupted dump	Re-run init phase
"Source file not found"	Missing files on HPC	Transfer all new .c/.h files
Init phase too slow	High MAXlevel	Expected - only run once

Critical fixes applied

1. Missing gravity: both init and main now set G.y = -Bo
2. Error tolerances consistent: fErr=1e-3, KErr=1e-4, VelErr=1e-2
3. Gas viscosity consistent: mu2 = Mu21*Oh = 1e-3*Oh
4. MPI-aware logging with pid() checks inside logWriting()
5. Shared common header for consistent behavior across phases

Compilation details

Initialization phase (simulationCases/jumpingDrops_init.c)

• Compiler: qcc (serial only)
• Include: distance.h, reduced.h
• Flags: -O2 -Wall -disable-dimensions
• MPI: not supported

Main phase (simulationCases/jumpingDrops_main.c)

• Compiler: qcc (serial) or mpicc + qcc (parallel)
• Include: no distance.h
• Flags: -O2 -Wall -disable-dimensions [-D_MPI=1]
• MPI: supported

Directory structure

├── src-local/
│   ├── jumpingDrops_common.h          # Shared definitions
│   └── parse_params.sh                # Parameter parsing helpers
├── simulationCases/
│   ├── jumpingDrops_init.c            # Initialization
│   ├── jumpingDrops_main.c            # Main simulation
│   ├── JumpingDrops_legacy.c          # Deprecated
│   ├── JumpingDrops_Snellius_legacy.c # Deprecated
│   └── <CaseNo>/                      # Case directories
│       ├── case.params                # Parameter file
│       ├── InitialCondition.stl       # STL geometry (for init)
│       ├── dumpInit                   # Initial condition (from init)
│       ├── dump                       # Current state
│       ├── restart                    # Restart file (if exists)
│       ├── log                        # Simulation log
│       └── intermediate/              # Snapshot files
│           └── snapshot-*.dump
├── runSimulation.sh                   # Single case runner
├── runParameterSweep.sh               # Parameter sweep
├── runSweepSnellius.sbatch            # HPC sweep
├── default.params                     # Example single-case parameters
├── sweep.params                       # Example sweep configuration
└── .project_config.example            # Basilisk environment template

Advantages of modular design

1. Clean separation: initialization vs main simulation
2. MPI compatibility: no distance.h in parallel code
3. Flexibility: reuse dumpInit for multiple runs
4. Consistency: single source of truth for common code
5. Development speed: skip init during main-phase testing

Migration guide

From old local code (JumpingDrops_legacy.c):

# Before
./JumpingDrops Oh Bo MAXlevel

# After
./jumpingDrops_init Oh Bo MAXlevel
./jumpingDrops_main tmax Oh Bo MAXlevel

From old HPC code (JumpingDrops_Snellius_legacy.c):

# Before (missing gravity)
mpirun -np 48 ./JumpingDrops_Snellius_legacy case.params

# After
./runSimulation.sh --init-only case.params
scp dumpInit hpc:case/
srun -n 48 ./jumpingDrops_main tmax Oh Bo MAXlevel

File sizes (typical)

File	Size	Transfer?
InitialCondition.stl	~1-10 MB	No (local only)
dumpInit	~10-100 MB	Yes (to HPC)
snapshot files	~10-100 MB each	No (keep on HPC)

Performance notes

Initialization phase

• Runtime: ~1-2 minutes (depends on MAXlevel and STL complexity)
• Memory: moderate (STL loading + distance field)
• Parallelization: serial only (distance.h limitation)
• Frequency: once per geometry

Main phase

• Runtime: hours to days (depends on tmax, MAXlevel)
• Memory: varies with refinement
• Parallelization: strong MPI scaling
• Frequency: multiple runs with same initial condition

Best practices

1. Run init locally (STL loading not MPI-compatible)
2. Transfer dumpInit to HPC (smaller than STL files)
3. Use --skip-init for sweeps after first run
4. Check logs for parameter values (Oh, Bo, gravity)
5. Use restart files for long runs
6. Keep dumpInit once generated

Testing checklist

Local

• Compile init phase: ./runSimulation.sh --init-only --compile-only
• Compile main phase: ./runSimulation.sh --main-only --compile-only
• Run init phase: ./runSimulation.sh --init-only default.params
• Verify dumpInit created
• Run main phase serial: ./runSimulation.sh --main-only default.params
• Run main phase MPI: ./runSimulation.sh --main-only --mpi --cores 4 default.params
• Check log file for correct Oh, Bo values
• Verify gravity effects in output

HPC

• Transfer dumpInit to HPC
• Test single case compilation on HPC
• Run single case with MPI: srun -n 48 ./jumpingDrops_main tmax Oh Bo MAXlevel
• Submit parameter sweep: sbatch runSweepSnellius.sbatch
• Verify log files show correct parameters
• Check that all cases complete successfully

Future enhancements

• Support different STL geometries per case
• Automatic dumpInit synchronization script
• Parameter validation in shell scripts
• Progress monitoring during long runs
• Automatic restart on HPC node failure
• Post-processing integration

Contact

• Author: Vatsal Sanjay
• Email: vatsalsanjay@gmail.com
• Physics of Fluids