feat(data-api): retire D1 chain-data write path (#4772)#4908
Merged
Conversation
Drops the D1-side write/prune/rollup infrastructure for blocks, extrinsics, account_events, neurons, and neuron_daily now that #4771's Postgres write path (neurons-sync) and the Postgres-native indexer-rs pipeline (blocks/extrinsics/account_events/chain_events) are the real serving tiers, with tryPostgresTier(...) ?? d1Fallback() already wired across every read route. Deletes the realtime streamer scripts (fetch/stream/backfill-events.py) and their workflow/docs, the internal ingest handlers (handleEventIngest/handleBlockIngest/ handleNeuronBackfill), the D1 staged-batch loaders for those four tables, and each table's prune/rollup/archive cron logic -- restructuring EVENTS_LOAD_CRON, HEALTH_PRUNE_CRON, and NEURON_HISTORY_ROLLUP_CRON to drop only what they retired. Fixes a latent bug the restructuring surfaced: HEALTH_PRUNE_CRON's unrelated pruneHealthHistory prune was gated on the now-removed rollupAccountEventsDaily succeeding, which would have silently wedged it forever once D1's account_events table is gone. account_position_daily, account_events_daily's independent Postgres rollup, and the ~40 dead D1-fallback branches in entities.mjs are explicitly out of scope and left as-is.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
metagraphed-data-api | a6197b6 | Jul 11 2026, 05:03 PM |
Checks off sequencing item 8 (D1 write/prune/ingest code retired, PR #4908) and adds item 11 documenting the indexer-rs backfill DO NOTHING -> DO UPDATE conflict-resolution fix found while auditing the eventual archive-gated backfill for correctness. Updates Links/resources for the files this PR deleted.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
This was referenced Jul 11, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4908 +/- ##
==========================================
+ Coverage 97.88% 97.94% +0.05%
==========================================
Files 162 162
Lines 20030 19425 -605
Branches 7527 7349 -178
==========================================
- Hits 19606 19025 -581
+ Misses 59 53 -6
+ Partials 365 347 -18
🚀 New features to boost your workflow:
|
gittensory-orb
Bot
added
the
gittensor:bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
blocks,extrinsics,account_events,neurons, andneuron_dailynow that Build the metagraph/neuron pipeline's missing Postgres write path (neurons + neuron_daily) #4771's Postgres write path (neurons-sync) and the Postgres-native indexer-rs pipeline (blocks/extrinsics/account_events/chain_events) are the real serving tiers, withtryPostgresTier(...) ?? d1Fallback()already wired across every read route.fetch-events.py,stream-events.py,backfill-events.py) and their workflow/docs, the internal ingest handlers (handleEventIngest/handleBlockIngest/handleNeuronBackfill), the D1 staged-batch loaders for those four tables, and each table's prune/rollup/archive cron logic.EVENTS_LOAD_CRON,HEALTH_PRUNE_CRON, andNEURON_HISTORY_ROLLUP_CRONto drop only what they retired, keepingsubnet_hyperparams/account_identitystaging andaccount_position_daily's rollup/prune untouched.HEALTH_PRUNE_CRON's unrelatedpruneHealthHistoryprune was gated on the now-removedrollupAccountEventsDailysucceeding, which would have silently wedged it forever once D1'saccount_eventstable is gone.metagraphed-indexer-rsrepo (already merged/deployed separately): the backfillflush()usedON CONFLICT DO NOTHINGforextrinsics/account_events/chain_eventsinstead ofDO UPDATE, which would have silently preserved stale rows on the eventual archive-gated backfill re-run.Explicitly out of scope, left as-is:
account_position_daily(has a Postgres write mirror but no read route yet),account_events_daily's independent Postgres-side rollup (already had its own GitHub Actions workflow, never depended on the removed D1 rollup), and the ~40 dead D1-fallback branches inworkers/request-handlers/entities.mjs(cosmetic, provably safe viad1All's graceful degradation — tracked as follow-up).Closes #4772
Test plan
npx vitest run— 255 files / 7517 tests passingnpm run lint— cleannpx prettier --checkon all touched files — cleangit diff --check— cleannpm run test:coverage(unsharded) — 99.5% stmts / 97.92% branch overall; manually verified zero uncovered lines within this diff's actual added/changed lines (the diff is almost entirely deletions)npm run scan:public-safety— cleannpm run validate— clean (129 subnets, 1553 surfaces)node scripts/validate-workflows.mjs— 24 workflows validated