Usage feedback

[RagnarGrootKoerkamp]

So I'm trying to use AWRY again to include it in an FM-index benchmark, but I must say that it's rather annoying to use and I'm close to giving up... My user journey for trying it on a human genome.

• The input must be from disk, so I have to write my data to a file first.
• Actually, it must be a fasta file, so I have to prepend it with >\n.
• Ah but it must be ASCII, not u8 integer values 0/1/2/3. (ok that's on me though)
• Ah but libsufr writes in $TMPDIR which is /tmp which is tmpfs/RAM for me, and libsufr needs more than the 32GB available there.
• Ok I create a local directory tmp and put export TMPDIR=tmp, but now this causes compile time errors because rust tries creating sufr/libsufr/tmp/... and AWRY/tmp/... instead of my_crate/tmp/....
• So I build without the env var, and run with the env var.
• It then uses ~40GB disk, and 17GB RAM for the most time. That's fine.
• Then, after 11 minutes, I think after suffix array construction is done, RAM suddenly spikes to 32 GB and it OOMs.
• I do have 64 GB of RAM, but it turns out 32GB of that is taken by temporary files that are still lingering in /tmp (so add clarification to github actions to only test on main branch. #6 is still not really fixed apparently?).
• So now I'm going to have to wait 11min again for a new attempt at construction, which may or may not end up failing.

In fact, I already have the BWT on disk, but I don't think there's a way for me to directly give that to AWRY?
That would have saved me a lot of pain here. It's also not quite clear if this option to keep/remove the temporary suffix array actually would reuse it between runs.

[Sawwave]

Hi Ragnar,

Most of these issues come from the Suffix Array library we are using. Specifically:

1. the fact it has to be from disk
2. libsufr writing to /tmp
3. the disk usage

It has also become clear that that library has some issues, and doesn't seem likely to be updated soon. To fix Awry, we would need to replace the library. These are issues that can be addressed, but I left my academic position a year ago- these open source projects have become a side hobby rather than a priority.

I have talked to another developer in the research lab who could take over the project, but it seems like that may not be something that will happen soon. I'll try to open dialog again,

I may find time to work on it in the near future, but I think it's likely this library won't see development in the near future. Instead, you might either look into the C implementation (AwFmIndex) that doesn't rely on libsufr, or maybe just try a different FM-index rust library.

[RagnarGrootKoerkamp]

Yeah, all understandable.
Maybe adding support for libsais-rs is easy and a sufficient workaround. And/or add a field to the constructor to pass in a precomputed BWT? (Which would need a bit of explanation of how the sentinel should be handled.)

In the end I mostly just want to run AWRY so I can compare it against other FM-index implementations, and I don't really care about actually using it. (Nor will I actually use my own implementation, for the record 😅).
But I would to include it for scientific rigor of course.

[RagnarGrootKoerkamp]

For the record: things did end up working eventually with the same results as I get with other FM-index implementations. Note that I only benchmark count, and so this avoids #44.

See last plot here: https://github.com/RagnarGrootKoerkamp/quadrank