`AccountTree`: Establish re-synchronization between sqlite and rocksdb storage

[drahnr]

Once we land the AccountTreeWithHistory<AccountTree<RocksDb>> implementation, we will have two distinct places for fetching persisted accounts data - the LargeSmt<_> and the SQLite database. Now whenever we get into a situation where we have a discrepancy between we need a path to resolve.

Proposal

• drop the AccountTreeWithHistory and re-build from the SQL database recent history
• do something smarter™

[bobbinth]

drop the AccountTreeWithHistory and re-build from the SQL database recent history

I think that's probably the way to go. We can probably do this manually too.

[Mirko-von-Leipzig]

Now whenever we get into a situation where we have a discrepancy between we need a path to resolve.

How would this occur? When would one be checking both, and might notice?

[bobbinth]

One way it could happen is if there is some data corruption in the underlying storage of the account tree (i.e., RocksDB). Depending on the nature of the failure, we'd detect it in the following ways:

• Requests to the account tree may start failing. We'd need to pause the node, rebuild the RocksDB, and then start the node again.
• The tree starts returning invalid data. This will cause block production to fail (since the roots of the trees won't match any more). Then, we'd also need to pause the node, rebuild the RocksDB, and then start the node again.

[bobbinth]

We kind of already have this, right?

[Mirko-von-Leipzig]

We currently have:

• Check on startup
• Check on applying block
  • However see this comment chain. This isn't fatal.

A quick fix is to give apply_block a shutdown signal so it can pass the error to the outer scope and we can abort from there. Note that one might be tempted to use axum's builtin with_graceful_shutdown, however that returns Ok(()) which won't pass the error. So we want to have a separate channel.