vg index terminated with signal 15 during make_giraffe_indexes (dist index, ~1.4TB memory usage)

[shin0727]

Hi,

I am running cactus-pangenome and encountering a failure during the make_giraffe_indexes step. The job crashes while running vg index to generate the .dist index.

The process exits with signal 15 (SIGTERM) and not SIGKILL. The logged peak memory usage is approximately 1.47 TB.

I would like to understand whether this is expected behavior (e.g. memory underestimation or scheduler-related termination) or if there may be an issue with resource handling in this step.

cactus-pangenome ./jobstorepath ./sequenceFile.tsv \
  --binariesMode local \
  --workDir /Data/jwshin0727/toilwork \
  --outDir ${PREFIX} --outName ${PREFIX} \
  --reference ARS_UCD --haplo \
  --giraffe clip filter \
  --vcf --vcfReference ARS_UCD \
  --viz --odgi \
  --chrom-vg clip filter --chrom-og \
  --gbz clip filter full --gfa clip full \
  --maxMemory 1400G \
  --logFile 260227_3rd.cactus.log

Error log

Running the command:
vg index -t 143 -j clip.Hanwoo_27ea.dist clip.Hanwoo_27ea.gbz -P ARS_UCD

RuntimeError:
Command /usr/bin/time -f "CACTUS-LOGGED-MEMORY-IN-KB: %M" vg index ...
exited 143: stderr=Command terminated by signal 15

CACTUS-LOGGED-MEMORY-IN-KB: 1469793116

Memory information

MemTotal:       1584984504 kB
MemFree:        144034300 kB
MemAvailable:   1434507540 kB
Buffers:         1634528 kB
Cached:         1289645148 kB
SwapCached:       108096 kB
Active:         236804144 kB
Inactive:       1191347396 kB
Active(anon):   134147196 kB
Inactive(anon):  2736100 kB

Questions

Is it expected that vg index -j (dist index) may require >1.3 TB RAM for large pangenomes?
It consists of 55 assemblies for the same species (3G).

and how can i solve this memory issue?

Any guidance on stabilizing the make_giraffe_indexes step would be greatly appreciated.

Thank you.

[glennhickey]

Yes. The preferred way is to use --haplo instead of --giraffe. But if you want to use --giraffe, use --giraffe filter and --filter N where N is roughly 10% of your input haplotypes. If you use --giraffe clip it will run out of memory and crash.

[shin0727]

Hi, thank you for the explanation.

I have a few follow-up questions to better understand the issue.

In my run, the failure occurred during vg index -j while building the .dist index, and the peak memory usage reported by cactus was about 1.47 TB.
Could you explain why the memory usage becomes so large in this step when using --giraffe clip?
Is it mainly due to the size/complexity of the clipped graph used for building the .dist index?

Does the number of threads (-t) used by vg index significantly affect memory usage?
For example, would reducing the thread count help reduce peak RAM usage, or is the graph complexity the dominant factor?

Regarding the cactus option --maxMemory, is setting it like --maxMemory 1400G the correct way to limit memory usage for this step?
Or does this parameter not strictly control the memory used by vg index jobs?

For my downstream analysis, I plan to map short reads to the pangenome graph using Giraffe.
If I skip --giraffe clip (or even --giraffe) during the cactus run to avoid the memory issue, can I generate the Giraffe indexes later from the resulting graph (e.g., GBZ) separately?

Any advice on the recommended workflow for building Giraffe indexes for large pangenomes would also be greatly appreciated.

Thank you very much for your help.

[glennhickey]

1. yes
2. no.
3. no
4. see here for how to index your graph for giraffe.