Skip to content

Add logical size tracking for SET and HASH hashtable encoding#5

Draft
liorsve wants to merge 2 commits intounstablefrom
hashtable-logical-size-tracking
Draft

Add logical size tracking for SET and HASH hashtable encoding#5
liorsve wants to merge 2 commits intounstablefrom
hashtable-logical-size-tracking

Conversation

@liorsve
Copy link
Copy Markdown
Owner

@liorsve liorsve commented Mar 25, 2026

Summary

Adds incremental logical size tracking for hashtable-encoded SET and HASH data types. The hashtable struct maintains a tracked_data_bytes field that is automatically updated on every insert/remove, giving O(1) access to the total logical entry size without walking the table.

Hashtable changes (hashtable.c/hashtable.h)

  • Added entryGetSize callback to hashtableType — returns logical size of an entry
  • Added tracked_data_bytes field to struct hashtable, maintained automatically on insert(), hashtableInsertAtPosition(), hashtablePop(), hashtableTwoPhasePopDelete(), hashtableEmpty()
  • Added hashtableTrackedDataBytes() accessor and hashtableAdjustTrackedDataBytes() for in-place entry mutations

SET tracking (server.c)

  • Added sdsEntryGetSize callback: sdsHdrSize(type) + sdslen(s) + 1 — pure arithmetic, no zmalloc calls
  • Wired into setHashtableType
  • Fully automatic — zero changes to t_set.c

HASH tracking (entry.c, server.c, t_hash.c)

  • Added entryGetLogicalSize() in entry.c covering all 4 entry types (TYPE_5 embedded, TYPE_8 embedded, separate SDS value, stringRef) — no zmalloc calls
  • Wired hashEntryGetSize callback into hashHashtableType and hashWithVolatileItemsHashtableType
  • 3 in-place mutation sites in t_hash.c need manual delta tracking via hashtableAdjustTrackedDataBytes():
    • entryUpdate in hashTypeSet (value change)
    • entryUpdateAsStringRef in hashTypeUpdateAsStringRef (stringRef path)
    • entrySetExpiry in hashTypeSetExpire and hashTypePersist (expiry toggle)

Design

  • Logical size, not allocation size: tracks semantic sizes from struct fields, never calls zmalloc_size
  • Data separated from metadata: tracked_data_bytes = sum of entry logical sizes. Metadata = hashtableMemUsage() (struct + buckets), computed O(1) from existing fields

Tests

  • test_set_tracking.cpp — 22 tests covering all mutation paths + 2 non-deterministic fuzzers
  • test_hash_tracking.cpp — 17 tests covering all mutation paths including expiry toggle, stringRef, conversion, and a non-deterministic fuzzer

@liorsve liorsve force-pushed the hashtable-logical-size-tracking branch 4 times, most recently from 76e6af1 to 6814655 Compare March 29, 2026 11:56
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
Signed-off-by: Lior Sventitzky <liorsve@amazon.com>
@liorsve liorsve force-pushed the hashtable-logical-size-tracking branch from 402214e to 51d834e Compare April 7, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant