[disk-buffering] Replace the read-delay synchronisation with atomic rename

[mamazzol]

Component(s)

disk-buffering

Is your feature request related to a problem? Please describe.

The current disk-buffering writer/reader synchronisation is time-based: a file is written for up to maxFileAgeForWriteMillis (default 30 s), and the reader is forbidden from touching it until minFileAgeForReadMillis (default 33 s) has elapsed since creation.
This protects the reader from observing a file before it's ready, but it also means that there is a time window where the file is there but invisible to the reader and there is also a strong coupling and order between those two settings.

Describe the solution you'd like

Move from a time-based to a rename-based sync mechanism, writing to a tmp file and atomic renaming.

Benefits:

• The reader sees buffered data as soon as the writer rolls a file — no extra 3 s+ wait, and the perceived end-to-end latency drops by up to ~33 s in the common case.
• The writer and reader work on disjoint files by construction (different name shapes), which is much simpler to reason about than two interlocking time windows.
• The "in-process only" caveat in the README ("this tool assumes that both the reading and the writing actions are executed within the same application process") is no longer necessary for correctness — atomic rename is a cross-process primitive.
• Configuration becomes orthogonal: write-window and read-window settings no longer constrain each other.

Describe alternatives you've considered

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[breedx-splk]

I like this suggestion. One question that this brings up to me tho is what happens when the process exits or crashes? Are the temp files able to be recovered and resumed/exported?

Right now, with the current approach, I think the files are nearly always able to be picked up when the app restarts. Would this rename approach change that for the worse?