[VQueues] Optimize sys_vqueue_entry_status point lookups#5047
Draft
AhmedSoliman wants to merge 3 commits into
Draft
[VQueues] Optimize sys_vqueue_entry_status point lookups#5047AhmedSoliman wants to merge 3 commits into
AhmedSoliman wants to merge 3 commits into
Conversation
This was referenced Jul 15, 2026
NOT IN (...) and != predicates enumerate excluded values, but the partition-key extractor and the entry-id/invocation-id filters ignored InList::negated and built lookup sets or ranges from those excluded IDs. Reject negated IN lists in those extractors so DataFusion keeps them as normal filters instead of treating them as positive lookup inputs. Cover the sys_invocation_status regression with excluded and retained invocation IDs on different partition keys, and keep sys_vqueue_entry_status coverage for the same NOT IN behavior.
… reads Keep equality and IN predicates as exact ID selections in the DataFusion filter layer instead of collapsing large sets into scan ranges. This lets point-lookup tables preserve the user's requested IDs while range-scan tables can still convert the same selection back to bounds. Teach the partitioned table provider to group exact point reads by Restate partition when the scanner can re-fetch the exact IDs itself. Use that for sys_invocation_status by adding an InvocationIdSet scan path backed by bounded RocksDB multi-get batches. Split partition-store inclusive key-range scans into single-prefix and total-order variants, and use total-order scans when invocation-status, journal, or journal-events ID bounds span multiple partition keys. This preserves correctness for range-scan consumers of exact ID selections. Adapt the worker cleaner to the fallible stream shape used by the new invocation-status scan path, and add scan-mode coverage for the new total-order range variant.
Make sys_vqueue_entry_status understand exact entry_id predicates so equality and IN filters can be planned as targeted reads rather than broad partition scans. Entry IDs are parsed back to their partition keys for routing and displayed consistently from the stored EntryId representation. Add the partition-store support needed to read selected entry-status records through bounded batched multi-get calls while preserving range scans for non-point predicates. Entry-id range scans now use total-order iteration when the bounds span multiple partition keys. Only advertise partition_key ordering for sys_vqueue_entry_status, because the exposed entry_id string ordering does not match the raw on-disk entry-id ordering. Cover both the DataFusion point-query path and multi-get batching behavior, including a NOT IN regression so excluded IDs are not treated as an exact lookup set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make sys_vqueue_entry_status understand exact entry_id predicates so equality and IN filters can be planned as targeted reads rather than broad partition scans. Entry IDs are parsed back to their partition keys for routing and displayed consistently from the stored EntryId representation.
Add the partition-store support needed to read selected entry-status records through bounded batched multi-get calls while preserving range scans for non-point predicates. Entry-id range scans now use total-order iteration when the bounds span multiple partition keys.
Only advertise partition_key ordering for sys_vqueue_entry_status, because the exposed entry_id string ordering does not match the raw on-disk entry-id ordering.
Cover both the DataFusion point-query path and multi-get batching behavior, including a NOT IN regression so excluded IDs are not treated as an exact lookup set.
Stack created with Sapling. Best reviewed with ReviewStack.