[Prototype] Bigtable

[Kbhat1]

Describe your changes and provide context

Testing performed to validate your change

---

Note

Medium Risk
Adds an optional fallback path for pruned Get/Has state-store reads to query external Scylla/Cassandra or Bigtable, plus new ingestion tooling and cloud/db dependencies; misconfiguration or backend issues could affect historical query behavior.

Overview
Adds prototype historical-state offload support so SS point reads (Get/Has) for versions below local retention can fall back to an external store (Scylla/Cassandra or Bigtable), wrapped via historical.FallbackStateStore and enabled through new [state-store] TOML/flag config (mutually exclusive backends).

Introduces a new Kafka-based offload consumer (sei-db/state_db/ss/offload/consumer) with configurable sinks for Scylla and Bigtable, including schema/examples and retry/batching/backpressure behavior, and exports offload.NewSASLMechanism so external consumers share the producer auth path.

Extends SS config structs/templates/tests for the new historical-offload settings and adds new dependencies (cloud.google.com/go/bigtable, github.com/gocql/gocql, golang.org/x/oauth2).

^{Reviewed by Cursor Bugbot for commit 66f8c9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	May 27, 2026, 8:53 PM

[codecov]

Codecov Report

❌ Patch coverage is 41.47683% with 745 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.14%. Comparing base (654d40b) to head (571a943).
⚠️ Report is 68 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/offload/historical/bigtable.go	38.52%	214 Missing and 19 partials ⚠️
sei-db/state_db/ss/offload/consumer/consumer.go	0.00%	177 Missing ⚠️
sei-db/state_db/ss/offload/historical/scylla.go	49.29%	71 Missing and 1 partial ⚠️
sei-db/state_db/ss/offload/consumer/scylla.go	66.85%	48 Missing and 11 partials ⚠️
sei-db/state_db/ss/offload/historical/store.go	53.76%	34 Missing and 9 partials ⚠️
sei-db/state_db/ss/store.go	22.64%	39 Missing and 2 partials ⚠️
sei-db/state_db/ss/offload/consumer/config.go	39.28%	29 Missing and 5 partials ⚠️
sei-db/state_db/ss/offload/consumer/bigtable.go	69.72%	27 Missing and 6 partials ⚠️
sei-db/state_db/ss/offload/consumer/kafka.go	44.82%	31 Missing and 1 partial ⚠️
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3437      +/-   ##
==========================================
- Coverage   59.25%   59.14%   -0.11%     
==========================================
  Files        2110     2120      +10     
  Lines      174181   176516    +2335     
==========================================
+ Hits       103210   104408    +1198     
- Misses      62044    63101    +1057     
- Partials     8927     9007      +80     

Flag	Coverage Δ
sei-chain-pr	51.45% <100.00%> (?)
sei-db	70.41% <ø> (ø)
sei-db-state-db-pr	41.40% <41.24%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/seidb.go	79.04% <100.00%> (+7.04%)	⬆️
sei-db/config/ss_config.go	100.00% <ø> (ø)
sei-db/state_db/ss/offload/kafka.go	59.37% <100.00%> (ø)
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00% <0.00%> (ø)
sei-db/state_db/ss/offload/consumer/kafka.go	44.82% <44.82%> (ø)
sei-db/state_db/ss/offload/consumer/bigtable.go	69.72% <69.72%> (ø)
sei-db/state_db/ss/offload/consumer/config.go	39.28% <39.28%> (ø)
sei-db/state_db/ss/store.go	24.07% <22.64%> (-75.93%)	⬇️
sei-db/state_db/ss/offload/historical/store.go	53.76% <53.76%> (ø)
sei-db/state_db/ss/offload/consumer/scylla.go	66.85% <66.85%> (ø)
... and 3 more

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

High Risk
Changes how historical state point reads are resolved when local SS is pruned, including external DB auth and MVCC lookup correctness; mistakes could return wrong or missing state for old heights.

Overview
Adds a prototype historical state offload path so pruned MVCC versions can still be served from external storage while local SS stays the hot path for writes, pruning, and iterators.

Node reads: NewStateStore can wrap the composite SS in FallbackStateStore, which sends Get/Has to Scylla/Cassandra or Bigtable when the requested version is below local GetEarliestVersion(); iterators and commits stay on primary SS. Only one fallback backend is allowed at a time.

Ingest: A Kafka consumer (historical-scylla-consumer) decodes changelog entries, batches by partition, and writes mutation rows before version markers into Scylla (CQL schema) or Bigtable (salted row keys + inverted height). NewSASLMechanism is exported for shared Kafka auth with the producer.

Config: New [state-store] fields and TOML for Scylla and Bigtable offload; app/seidb wires Scylla flags into parseSSConfigs. Dependencies add gocql and cloud.google.com/go/bigtable.

Documented limits: no offload iterators, no cross-row ingest transactions (idempotent replay via write ordering).

^{Reviewed by Cursor Bugbot for commit 571a943. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ecdf5cc. Configure here.}