Skip to content

Combined: hashtable + rax memory tracking with vset integration#9

Draft
liorsve wants to merge 11 commits into
unstablefrom
combined-hashtable-rax-tracking
Draft

Combined: hashtable + rax memory tracking with vset integration#9
liorsve wants to merge 11 commits into
unstablefrom
combined-hashtable-rax-tracking

Conversation

@liorsve
Copy link
Copy Markdown
Owner

@liorsve liorsve commented Apr 7, 2026

Summary

Combined branch providing O(1) logical size reporting for hash objects via hashTypeLogicalSize(). Integrates hashtable entry tracking with rax/stream/vset tracking into a single branch.

How this branch was built

This branch is based on hashtable-logical-size-tracking (PR #5) with all commits from rax-data-tracking-caller (PR #6) cherry-picked on top. That said, the other two POC branches for rax tracking can be used just the same. Then the vset integration was added as a new commit to wire everything together. So the new integration code is only commit 7d89748

What this branch adds on top

hashTypeLogicalSize() — O(1) total hash memory

New function in object.c that combines all tracked counters into a single O(1) read:

size_t hashTypeLogicalSize(robj *o) {
    // listpack encoding: lpBytes(data)
    // hashtable encoding:
    //   hashtableMemUsage(ht)          — hashtable container: sizeof(hashtable)
    //                                    + metadata + sizeof(bucket) * num_buckets
    //   + hashtableTrackedDataBytes(ht) — entry data: per entry field SDS + value SDS
    //                                     + optional expiry + optional stringref pointer
    //   + vsetLogicalSize(vset)         — vset container: rax nodes + inner bucket
    //                                     containers (pVector headers + pointer arrays,
    //                                     hashtable bucket arrays). Does NOT include
    //                                     entry data — entries are owned by the hash's
    //                                     hashtable data (covered in hashtableTrackedDataBytes), 
    //                                     vset just holds pointers to them.
}

For hashtable encoding, this replaces the O(n) sampled walk in objectComputeSize with an O(1) read.

vsetLogicalSize() — O(1) vset container size

New function in vset.c. Returns the logical size of the vset's own container memory:

  • NONE/SINGLE: 0
  • VECTOR: sizeof(pVector) + pvLen(pv) * sizeof(void *)
  • RAX: sizeof(vsetRaxState) + raxLogicalSize(r) + tracked_data_bytes

vsetComputeLogicalSize() — O(n) walk for test verification

New function in vset.c. Walks all vset structures independently of tracked counters. Used only in tests to verify tracked values are correct.

Updated test assertions

All HASH tracking tests now verify both data tracking AND total logical size (including vset) at every assertion point via ASSERT_HASH_LOGICAL_SIZE(o, ht), which checks:

  • ASSERT_TRACKED_DATA_CORRECThashtableTrackedDataBytes vs independent entry walk
  • ASSERT_TRACKED_OVERHEAD_CORRECThashtableMemUsage + vsetLogicalSize vs independent overhead walk using vsetComputeLogicalSize

3 new vset-specific tests

  • VsetSingleExpiry — 1 entry with expiry, SINGLE encoding (zero container overhead). Verifies vsetLogicalSize == 0.
  • VsetVectorExpiry — 50 entries with expiry, VECTOR encoding. Verifies per-entry growth and shrink on delete.
  • VsetRaxExpiry — 200 entries with spread expiries, RAX encoding. Observes SINGLE→VECTOR→RAX transitions via size values. Verifies plain entries don't affect vset. Verifies value updates don't change vset container. Full add→update→delete lifecycle.

liorsve added 10 commits April 7, 2026 07:13
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
@liorsve liorsve force-pushed the combined-hashtable-rax-tracking branch 2 times, most recently from 7b0d930 to 7d89748 Compare April 7, 2026 13:04
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant