Improve memory usage of unaligned dataset joins in an RDataFrame analysis

[TomasDado]

Explain what you would like to see improved and how.

I had a short discussion with @vepadulano regarding one issue we see in RDataFrame.

We run on TTrees that we need to "join" with BuildIndex(). The problem is in our case the number of events we need to match is around 500M, this causes issues because the memory needed to keep the hash map for the matching is huge. This would be manageable on its own, but the problem is that each thread keeps the copy of the map. This results in our case of 40 threads using more than 120 GBs of memory (would need probably much more but this is the limitation of the hardware). As you can see this is pretty restrictive as the solution here is to either:

• Do not use that many threads
• Somehow split the files so you dont need to have a map of 500M entries
• "Just get more RAM"

These are not very compelling options.

We understand that this is probably beyond the scope of TTree and RDF support but this is something that could maybe improve for the RNtuple and RDF? As the current situation with TTrees is not sustainable

ROOT version

Any

Installation method

Any

Operating system

Any

Additional context

No response

[vepadulano]

Hi @TomasDado ,

Thanks for reaching out and reporting our discussion. Indeed, the situation cannot be improved for TTree. Maybe we can do something for RNTuple. For now I will keep this issue open as a reminder but we may close it and keep track of it separately.

[Hashbrownsss]

I traced the problem to the threaded friend/index path in TTreeProcessorMT/TTreeView. When a friend tree has BuildIndex(), the backend clones its TTreeIndex object for each worker thread, and TTreeIndex::Clone() currently deep-copies the huge index arrays. That means the same 500M-entry index is stored once per thread. The clean fix i think would be to make TTreeIndex clones share the underlying index arrays (or otherwise separate shared immutable index data from thread-local reader state), which would remove the O(nThreads) memory blowup for indexed friend joins.

[vepadulano]

Dear @Hashbrownsss ,

Thanks for reaching out! You are right, that's one of the key parts of the problem and I had already described it to @TomasDado outside of this github issue. That's partly implied in my sentence above

Indeed, the situation cannot be improved for TTree. Maybe we can do something for RNTuple

I may have abused the term "cannot". Of course TTree can be improved because it's code anyway. But this type of large effort involving probably new multi-threaded strategies is better if devoted to RNTuple, which is the new storage technology being developed for future production use.

[Hashbrownsss]

Thanks for the clarification! I agree on your wordings. Besides that , in the RNTuple direction, are you looking at implementing something like a native RNTuple join/lookup operator designed for these unaligned datasets or like a partitioned/blocked join strategy in RNTuple so huge keyed joins do not require one giant in-memory hash per thread, or will this issue be coming to a close and track will be kept seperately as you mentioned before?

[vepadulano]

Yes, RNTuple already has a dataset iteration tool called RNTupleProcessor, which will among other things also support iteration over entries belonging to different datasets matched with joins (whether aligned or unaligned). The feature is still experimental, and in particular it does not cover yet the issue described here, but you can take a look at this tutorial to see what is already available today https://root.cern.ch/doc/master//ntpl015__processor__join_8C.html

[Hashbrownsss]

Will do, Thank you!