[spark] Reduce driver memory usage when deserializing compact commit messages

[lilei1128]

Purpose

Release serialized commit message byte arrays as they are deserialized to avoid
retaining both serialized and deserialized messages in driver memory.

Tests

No

[zhongyujiang]

LGTM. We applied the same fix in our internal fork, which reduced a lot of memory consumption during commits involving large tables.

[zhongyujiang]

This is actually releasing serialized bytes, not commit messages.

[lilei1128]

Good catch! The deserialized commit messages are retained, while only the references to the serialized byte arrays are released. I’ll rename the method to deserializeCommitMessagesAndReleaseSerializedBytes to make this explicit.

[lilei1128]

LGTM. We applied the same fix in our internal fork, which reduced a lot of memory consumption during commits involving large tables.

Thanks for taking a look!
Exactly, I ran into an issue where compacting a large table with tons of small files under tight resource limits was failing/OOMing. Glad to know this fix works well for your workloads as well!