[kv] Avoid unnecessary RocksDB flushes during Tablet Server shutdown

[platinumhamburg]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Closing a RocksDB-backed KV tablet currently uses RocksDB's default close behavior, which flushes dirty memtables before closing the database.

During Tablet Server shutdown, this happens for every local KV replica. When a Tablet Server hosts many replicas, these flushes can produce substantial disk I/O and make shutdown take a long time.

The flushed SST files do not establish a committed Fluss recovery point:

• If a completed snapshot exists, replica initialization recreates the local KV directory, downloads the snapshot, and replays the changelog after the snapshot offset.
• If no completed snapshot exists, recovery rebuilds the KV state by replaying the changelog from offset 0.
• A close-time RocksDB flush does not commit the corresponding Fluss recovery metadata, such as the snapshot offset and other tablet state.

Therefore, flushing dirty memtables during Tablet Server shutdown provides little recovery benefit while adding shutdown latency proportional to the number of KV replicas and their dirty memtable volume.

However, flush-on-close cannot be disabled globally. Existing direct KvManager.shutdown() and local-reopen behavior relies on RocksDB preserving the local state.

Solution

Introduce an explicit KV close mode to distinguish the two lifecycle requirements:

• PRESERVE_LOCAL_STATE: retain the existing RocksDB flush-on-close behavior.
• DISCARD_UNPERSISTED_STATE: allow unpersisted memtable state to be discarded because recovery will rebuild it from a completed snapshot and the changelog.

Use the modes as follows:

• Keep the no-argument RocksDBKv.close(), KvTablet.close(), and KvManager.shutdown() methods in PRESERVE_LOCAL_STATE mode for compatibility.
• Use DISCARD_UNPERSISTED_STATE when shutting down the KV manager as part of Tablet Server shutdown.
• Use DISCARD_UNPERSISTED_STATE when dropping a tablet because its local directory is deleted afterward.
• Immediately before closing RocksDB in discard mode, dynamically set avoid_flush_during_shutdown=true.
• Keep the global RocksDB DB options unchanged.
• Ensure all RocksDB native resources are still released if setting the dynamic option fails.

This keeps existing local-reopen semantics while avoiding shutdown-time flushes only where the local unpersisted state is not a recovery source.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!