Track logical quicklist memory incrementally via lpBytes + compressed size

[liorsve]

Summary

Add incremental memory tracking to quicklist via a single new field tracked_data_bytes on the quicklist struct. Tracks the sum of logical entry sizes across all nodes — node->sz (= lpBytes) for uncompressed nodes, sizeof(quicklistLZF) + lzf->sz for compressed nodes. No per-node fields added, no listpack changes, zero zmalloc_size calls.

Structural overhead (sizeof(quicklist) + ql->len * sizeof(quicklistNode)) is O(1)-computable from existing fields and not included in tracked_data_bytes.

Design

Based on the "lpBytes + compressed size tracking" approach. Four choke points maintain the running total:

1. __quicklistInsertNode — adds entry size when a node enters the list
2. __quicklistDelNode — subtracts entry size when a node leaves
3. quicklistNodeUpdateSz macro — tracks lpBytes delta on in-place listpack mutations
4. __quicklistCompressNode / __quicklistDecompressNode — swaps between node->sz and sizeof(quicklistLZF) + lzf->sz when compression state changes

Helper macros

• quicklistNodeEntryBytes(node) — returns logical entry size accounting for compression, used by insert/delete hooks
• quicklistNodeUpdateSz(ql, node) — updates node->sz from lpBytes and tracks delta; accepts NULL for ql when node is not yet linked
• quicklistNodeSetSz(ql, node, new_sz) — direct sz assignment with delta tracking, for the plain node replace path

Changes from the design plan

Aspect	Plan	Implementation	Why
Insert hook	+= new_node->sz	+= quicklistNodeEntryBytes(new_node)	quicklistDup inserts already-compressed nodes — node->sz is uncompressed size, but entry is an LZF buffer. Must check encoding.
Unlinked nodes in quicklistNodeUpdateSz	No NULL guard — always passes ql	if (ql) guard, pass NULL for unlinked nodes	Without guard, unlinked nodes get their size added by the macro AND by the insert hook — double-counting.
_quicklistSplitNode	Not in touchpoint list	Added quicklist * param	Original (linked) node needs tracked delta from quicklistNodeUpdateSz(quicklist, node). New (unlinked) node needs quicklistNodeUpdateSz(NULL, new_node).
Empty-list insert (line 970)	Not mentioned	Added quicklistNodeUpdateSz(NULL, new_node) before linking	new_node->sz was 0 from quicklistCreateNode(), never updated. Insert hook would add 0.
Decompress failure	Subtract first, rollback on failure	Tracking placed after success check — no rollback needed	Simpler: tracked_data_bytes only changes on the success path.

Accuracy

Tracks logical sizes, not jemalloc allocation sizes. The gap vs objectComputeSize (which uses zmalloc_size) is jemalloc size-class rounding — consistently 1-10% for typical lists, with larger relative gaps only on very small lists where a single allocation's rounding dominates:

Test	objectComputeSize	tracked total	gap	gap %
BasicPushHeadTail	1520	1437	83	5.5%
PushCreatesNewNodes	1504	1458	46	3.1%
PopHeadAndTail	944	898	46	4.9%
DeleteEntireNode	424	403	21	5.0%
CompressDecompress	12216	12039	177	1.4%
InsertNonFullNode	424	405	19	4.5%
InsertTriggersSplit	384	346	38	9.9%
InsertNextNeighbor	248	229	19	7.7%
InsertPrevNeighbor	248	227	21	8.5%
ReplaceEntryListpack	360	317	43	11.9%
ReplaceEntryPlainNode	512	481	31	6.1%
DelRange	752	686	66	8.8%
Rotate	520	485	35	6.7%
Dup	1640	1385	255	15.5%
DupWithCompression	11448	11096	352	3.1%
NodeMerge	288	259	29	10.1%
PlainNodeInsert	1000	980	20	2.0%
InsertEmptyList	128	107	21	16.4%
InterleavedCompressOps	22704	21980	724	3.2%
AppendListpackAndPlainNode	216	195	21	9.7%
DelEntryViaIterator	472	434	38	8.1%
ReplaceAtIndex	616	589	27	4.4%
SameSizeReplace	224	202	22	9.8%
ReplacePlainWithSmall	128	108	20	15.6%
LargeElementInsertWithSplit	504	480	24	4.8%
DeepCompressDepth	48080	47267	813	1.7%
Fuzzer (2000 ops)	5224	4779	445	8.5%

Tracked value is always ≤ computed. Gap shrinks with list size (DeepCompressDepth: 1.7%, InterleavedCompressOps: 3.2%).

Files changed

• src/quicklist.h — added size_t tracked_data_bytes to quicklist struct
• src/quicklist.c — 4 tracking choke points, 3 helper macros, quicklist * threaded through compress/decompress/split functions and 5 macros
• src/unit/test_quicklist_tracking.cpp — 28 correctness tests (new file)
• src/unit/test_quicklist_tracking_vs_compute.cpp — 27 comparison tests (new file)

Tests

test_quicklist_tracking.cpp — Correctness tests. Walks the list after each operation and verifies tracked_data_bytes == Σ quicklistNodeEntryBytes(node) for all nodes. 28/28 pass. Covers:

• Push/pop head and tail (into existing and new nodes)
• Delete entire node, delete range, delete via iterator
• Compression/decompression (depth 1 and 3)
• Insert before/after (non-full, neighbor, split, empty list)
• Replace (listpack, plain→large, plain→small conversion, same-size)
• Node merge, rotate, dup (with and without compression)
• AppendListpack/AppendPlainNode (RDB load paths)
• Large element insert with split
• Fuzzer: 2000 weighted random operations (60% adds, 25% removes, 15% replaces/rotates) with correctness check every 100 ops

test_quicklist_tracking_vs_compute.cpp — Comparison tests. Same tests comparing sizeof(quicklist) + ql->len * sizeof(quicklistNode) + ql->tracked_data_bytes against objectComputeSize. All 27 fail with the expected jemalloc rounding gap (see table above). These tests document the accuracy trade-off and serve as a regression baseline — they will pass if upgraded to lp_malloc_size-based tracking in the future.