Skip to content

Return bulk export result with history - #16

Open
sagargg wants to merge 4 commits into
feature/support_cloudflare_r2from
fix/datastore_bulk_export_history
Open

Return bulk export result with history#16
sagargg wants to merge 4 commits into
feature/support_cloudflare_r2from
fix/datastore_bulk_export_history

Conversation

@sagargg

@sagargg sagargg commented May 15, 2024

Copy link
Copy Markdown
Member
  1. Create a lookup table _query_history_lookup in BigQuery to store query history.
  2. Store encoded queries and table last modified timestamps in the query history lookup table.
  3. Create a history entry if the query does not exist in the lookup table. If it exists, return the result from the history.

If the query history exists, the current response time is ~6 seconds, but returning directly from BigQuery takes an additional 5 seconds due to the extra logic we added.

@sagargg
sagargg requested a review from shubham-mahajan May 15, 2024 10:37
@sagargg
sagargg changed the base branch from master to feature/support_cloudflare_r2 May 22, 2024 10:38

@shubham-mahajan shubham-mahajan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sagargg
sagargg force-pushed the fix/datastore_bulk_export_history branch 2 times, most recently from 047ab90 to 468bd00 Compare May 22, 2024 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants