Add atomic KVStore batch writes for restart-consistent state

[joostjager]

Issue #966 tracks a recurring class of crash/replay gaps between LDK-managed state and ldk-node-managed state. LDK persists ChannelManager, ChannelMonitors, and replayable LDK events. ldk-node separately persists PaymentDetails in PaymentStore and user-facing events in EventQueue.

When those streams are written independently, one side of a logical transition can become durable while the other does not. That leads to missing payment records, replay-only reconstruction, duplicate app events, startup backfill questions, and case-by-case reconciliation logic.

I think we should consider adding an atomic batch-write primitive to the ldk-node KV store layer, so related records can be committed as one restart-consistent unit. For example, one batch could include the LDK-derived persistence update, the corresponding PaymentStore update, and the EventQueue update that exposes it to the application.

The goal is not to require SQL, one big serialized blob, or a full LDK persistence redesign. The narrower goal is to let ldk-node write semantically coupled records together when needed, and have either all of them become durable or none of them become durable.

This seems like a natural extension of deferred writing. We already hold work until a persistence boundary; the missing piece is that ldk-node records still land through separate streams. If the underlying store supports atomic batches, the same boundary could include ldk-node state too.

[tnull]

This one should def. live / get discussed further on LDK (first)?

[joostjager]

I’d actually push back on starting this in LDK. We’ve discussed an LDK-level atomic batch write a few times before, and the blocker has always been that it needs a credible story for all supported KVStore implementations, especially FilesystemStore. I don’t think we have consensus on how to offer true multi-key atomicity there.

So I think that if we want to make progress, this should start in ldk-node instead. The consistency problem here is specifically between LDK-managed writes and ldk-node-managed writes for payments and events.

The direction I’d be thinking of is a StagingStore wrapper that ldk-node passes to LDK. That store would collect LDK write/remove operations and keep their futures pending. Then ldk-node would have a coordinator point where it atomically commits the staged LDK writes plus the corresponding ldk-node writes to a real backing store that supports this. Only after that commit succeeds would the staged LDK futures resolve.

Not ideal, but that may be the price of maintaining flexibility.