[VQueues] Optimize sys_vqueue_meta point lookups#5048
Draft
AhmedSoliman wants to merge 4 commits into
Draft
Conversation
This was referenced Jul 15, 2026
Test Results 8 files ± 0 8 suites ±0 2m 49s ⏱️ - 2m 13s Results for commit 9f81529. ± Comparison against base commit 47c66b3. This pull request removes 18 tests.♻️ This comment has been updated with latest results. |
NOT IN (...) and != predicates enumerate excluded values, but the partition-key extractor and the entry-id/invocation-id filters ignored InList::negated and built lookup sets or ranges from those excluded IDs. Reject negated IN lists in those extractors so DataFusion keeps them as normal filters instead of treating them as positive lookup inputs. Cover the sys_invocation_status regression with excluded and retained invocation IDs on different partition keys, and keep sys_vqueue_entry_status coverage for the same NOT IN behavior.
… reads Keep equality and IN predicates as exact ID selections in the DataFusion filter layer instead of collapsing large sets into scan ranges. This lets point-lookup tables preserve the user's requested IDs while range-scan tables can still convert the same selection back to bounds. Teach the partitioned table provider to group exact point reads by Restate partition when the scanner can re-fetch the exact IDs itself. Use that for sys_invocation_status by adding an InvocationIdSet scan path backed by bounded RocksDB multi-get batches. Split partition-store inclusive key-range scans into single-prefix and total-order variants, and use total-order scans when invocation-status, journal, or journal-events ID bounds span multiple partition keys. This preserves correctness for range-scan consumers of exact ID selections. Adapt the worker cleaner to the fallible stream shape used by the new invocation-status scan path, and add scan-mode coverage for the new total-order range variant.
Make sys_vqueue_entry_status understand exact entry_id predicates so equality and IN filters can be planned as targeted reads rather than broad partition scans. Entry IDs are parsed back to their partition keys for routing and displayed consistently from the stored EntryId representation. Add the partition-store support needed to read selected entry-status records through bounded batched multi-get calls while preserving range scans for non-point predicates. Entry-id range scans now use total-order iteration when the bounds span multiple partition keys. Only advertise partition_key ordering for sys_vqueue_entry_status, because the exposed entry_id string ordering does not match the raw on-disk entry-id ordering. Cover both the DataFusion point-query path and multi-get batching behavior, including a NOT IN regression so excluded IDs are not treated as an exact lookup set.
Use the generic exact ID selection machinery for sys_vqueue_meta so id equality and IN predicates route directly to the owning Restate partitions. Add a targeted vqueue-meta read path backed by bounded RocksDB multi-get batches, while keeping range scans for scope and non-point predicates. Vqueue-meta ID range scans now use total-order iteration when the bounds span multiple partition keys. Add DataFusion coverage for vqueue-meta point queries and NOT IN handling so negated lists remain normal filters instead of lookup inputs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use the generic exact ID selection machinery for sys_vqueue_meta so id equality and IN predicates route directly to the owning Restate partitions.
Add a targeted vqueue-meta read path backed by bounded RocksDB multi-get batches, while keeping range scans for scope and non-point predicates. Vqueue-meta ID range scans now use total-order iteration when the bounds span multiple partition keys.
Add DataFusion coverage for vqueue-meta point queries and NOT IN handling so negated lists remain normal filters instead of lookup inputs.
Stack created with Sapling. Best reviewed with ReviewStack.