[InvocationStatus] Add exact ID selection and batched multi-get point reads#5046
[InvocationStatus] Add exact ID selection and batched multi-get point reads#5046AhmedSoliman wants to merge 2 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5754ad5895
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Some(selection) if selection.keys.len() > 4096 => { | ||
| itertools::Either::Left(Some((partition_id, partition)).into_iter()) | ||
| } |
There was a problem hiding this comment.
For sys_invocation_status, with_grouped_invocation_id("id") enables PerPartition fanout, but this guard runs before the PerPartition arm, so an id IN (...) with more than 4096 distinct partition keys schedules every live partition instead of only partitions containing requested IDs. On partitions with no matching ID, InvocationIdFilter::new returns no ID selection and the status table converts that to a PartitionKey scan, causing unnecessary full partition scans; the per-key safety limit should not disable the grouped path because it creates at most one scan per Restate partition.
Useful? React with 👍 / 👎.
NOT IN (...) and != predicates enumerate excluded values, but the partition-key extractor and the entry-id/invocation-id filters ignored InList::negated and built lookup sets or ranges from those excluded IDs. Reject negated IN lists in those extractors so DataFusion keeps them as normal filters instead of treating them as positive lookup inputs. Cover the sys_invocation_status regression with excluded and retained invocation IDs on different partition keys, and keep sys_vqueue_entry_status coverage for the same NOT IN behavior.
… reads Keep equality and IN predicates as exact ID selections in the DataFusion filter layer instead of collapsing large sets into scan ranges. This lets point-lookup tables preserve the user's requested IDs while range-scan tables can still convert the same selection back to bounds. Teach the partitioned table provider to group exact point reads by Restate partition when the scanner can re-fetch the exact IDs itself. Use that for sys_invocation_status by adding an InvocationIdSet scan path backed by bounded RocksDB multi-get batches. Split partition-store inclusive key-range scans into single-prefix and total-order variants, and use total-order scans when invocation-status, journal, or journal-events ID bounds span multiple partition keys. This preserves correctness for range-scan consumers of exact ID selections. Adapt the worker cleaner to the fallible stream shape used by the new invocation-status scan path, and add scan-mode coverage for the new total-order range variant.
1 participant
Keep equality and IN predicates as exact ID selections in the DataFusion filter layer instead of collapsing large sets into scan ranges. This lets point-lookup tables preserve the user's requested IDs while range-scan tables can still convert the same selection back to bounds.
Teach the partitioned table provider to group exact point reads by Restate partition when the scanner can re-fetch the exact IDs itself. Use that for sys_invocation_status by adding an InvocationIdSet scan path backed by bounded RocksDB multi-get batches.
Split partition-store inclusive key-range scans into single-prefix and total-order variants, and use total-order scans when invocation-status, journal, or journal-events ID bounds span multiple partition keys. This preserves correctness for range-scan consumers of exact ID selections.
Adapt the worker cleaner to the fallible stream shape used by the new invocation-status scan path, and add scan-mode coverage for the new total-order range variant.
Stack created with Sapling. Best reviewed with ReviewStack.