vector-store: fix index size counter desync#447
Conversation
usearch's remove() returns Ok(usize) where the value is the number of entries actually removed (0 if the key was not found). The previous wrapper converted this to Ok(()) unconditionally, causing size.fetch_sub(1) to be called even when nothing was removed from the index. This manifests when an item is first seen during the range scan with an invalid vector (e.g. wrong dimensionality), causing the usearch add() to return Err. The item is still recorded in primary_ids and vector_timestamps with ETValue::Some — intentionally, to advance the timestamp for CDC deduplication. When a later valid update arrives via CDC, the Entry::Occupied path emits RemoveBeforeAddVector (because vector_already_exists=true) followed by AddVector. The remove hits a key that was never in usearch, returns Ok(0), but the size counter was decremented anyway. The subsequent successful add only brought it back to the pre-update value, so the count never advanced. Fix: UsearchIndex::remove() now returns Ok(bool) and size is decremented only when an entry was actually removed.
|
Initially it was part of #392 (as some tests added there would fail). Answering remarks from that review:
Initially I wrote that "usearch add() fail silently." I used wrong wording. Usearch returns error. And it is handled and generates warn. I meant that such vector in general is ignored (not indexed) and considered it a correct behavior.
It is not an error. And it would trigger warn, when we actually fix the invalid entry. @QuerthDP |
3 participants
usearch's remove() returns Ok(usize) where the value is the number of entries actually removed (0 if the key was not found). The previous wrapper converted this to Ok(()) unconditionally, causing size.fetch_sub(1) to be called even when nothing was removed from the index.
This manifests when an item is first seen during the range scan with an invalid vector (e.g. wrong dimensionality), causing the usearch add() to return Err. The item is still recorded in primary_ids and vector_timestamps with ETValue::Some — intentionally, to advance the timestamp for CDC deduplication. When a later valid update arrives via CDC, the Entry::Occupied path emits RemoveBeforeAddVector (because vector_already_exists=true) followed by AddVector. The remove hits a key that was never in usearch, returns Ok(0), but the size counter was decremented anyway. The subsequent successful add only brought it back to the pre-update value, so the count never advanced.
Fix: UsearchIndex::remove() now returns Ok(bool) and size is decremented only when an entry was actually removed.