feat: add optimized accountsdb crate

Cargo.toml

@@ -1,5 +1,6 @@
 [workspace]
 members = [
+    "accountsdb",
     "nucleus",
     "programs/magic-root-interface",
     "programs/magic-root-program",
@@ -22,6 +23,7 @@ rust-version = "1.94.1"
 version = "0.1.0"
 
 [workspace.dependencies]
+accountsdb = { path = "accountsdb" }
 magic-root-interface = { path = "programs/magic-root-interface" }
 magic-root-program = { path = "programs/magic-root-program" }
 nucleus = { path = "nucleus", package = "magicblock-engine-nucleus" }
@@ -35,18 +37,28 @@ arc-swap = "1.9.1"
 assert_matches = "1.5.0"
 base64 = "0.22.1"
 bincode = "1.3.3"
+bitcode = "0.6.9"
 bitflags = "2.11.1"
 blake3 = "1.8.5"
+bytemuck = { version = "1.25", features = ["derive", "extern_crate_std"] }
 cfg-if = "1.0.4"
+clonetree = "0.0.2"
 criterion = "0.7.0"
 derive_more = "2.1.1"
 env_logger = "0.11.8"
 futures = { version = "0.3.32", default-features = false }
+heed = { version = "0.22.1", default-features = false }
 itertools = "0.14.0"
 log = "0.4.29"
+memmap2 = "0.9.10"
+num_cpus = "1.17.0"
+oneshot = "0.2.1"
+parking_lot = "0.12.5"
+prometheus = { version = "0.14.0", default-features = false }
 qualifier_attr = "0.2.2"
 rand = "0.9.2"
 rustix = { version = "1.1.4" }
+scc = "3.8.4"
 serde = "1.0.228"
 serde_bytes = "0.11.19"
 tar = "0.4.45"
@@ -56,6 +68,7 @@ tokio = "1.52.1"
 tokio-util = "0.7.18"
 tracing = "0.1.44"
 tracing-subscriber = { version = "0.3.23", features = ["env-filter", "fmt"] }
+twox-hash = { version = "2.1.2", default-features = false }
 wincode = "0.5.1"
 zstd = { version = "0.13.3", default-features = false }
 

README.md

@@ -0,0 +1,327 @@
+# MagicBlock Engine
+
+A durable Solana execution engine for ephemeral rollups.
+
+MagicBlock Engine runs Solana transactions, owns account state, records execution
+results, and gives callers a small async API for accounts, transactions, blocks,
+and subscriptions.
+
+At the top of the stack is `engine::Engine`. A caller starts it once, then uses
+the handle to mutate accounts, submit transactions, simulate execution, stream
+updates, and query retained ledger state.
+
+```rust
+use std::{num::NonZeroU64, path::PathBuf, sync::Arc, time::Duration};
+
+use engine::Engine;
+use keeper::builder::{AccountsDBParams, BlockstoreParams, KeeperBuilder, LedgerParams};
+use nucleus::shutdown::ShutdownManager;
+use solana_keypair::Keypair;
+use solana_sysvar::rent::Rent;
+
+async fn open_engine(home: PathBuf) -> engine::Result<Engine> {
+    let mut shutdown = ShutdownManager::default();
+    let authority = Arc::new(Keypair::new());
+
+    let builder = KeeperBuilder {
+        authority,
+        accountsdb: AccountsDBParams {
+            directory: home.join("accountsdb"),
+        },
+        ledger: LedgerParams {
+            directory: home.join("ledger"),
+            size_limit: 256 * 1024 * 1024 * 1024,
+        },
+        blockstore: BlockstoreParams {
+            blocktime: Duration::from_millis(400),
+            superblock: NonZeroU64::new(16).unwrap(),
+        },
+        builtins: Default::default(),
+        programs: Default::default(),
+        accounts: Default::default(),
+        rent: Rent::default(),
+    };
+
+    Engine::new(builder, None, &mut shutdown).await
+}
+```
+
+## Cooperative Shutdown
+
+The caller owns the shutdown lifecycle. `Engine::new` receives a
+`ShutdownManager` so the engine can register its background services with one
+ordered coordinator, but the embedding service keeps the manager and drives it
+from the outside.
+
+`shutdown.wait().await` returns when the process receives SIGTERM or Ctrl-C, an
+engine service requests shutdown internally, or a registered service terminates
+early. After that, the host should stop accepting outside work, release or flush
+its engine handle, and then call `shutdown.terminate().await` to drain the
+engine's services by tier.
+
+```rust
+use engine::Engine;
+use nucleus::shutdown::ShutdownManager;
+use tokio::sync::oneshot;
+
+async fn run_service(
+    engine: Engine,
+    mut shutdown: ShutdownManager,
+    stop_ingress: oneshot::Sender<()>,
+) -> engine::Result<()> {
+    shutdown.wait().await;
+
+    let _ = stop_ingress.send(());
+    engine.flush()?;
+    drop(engine);
+
+    shutdown.terminate().await;
+    Ok(())
+}
+```
+
+Shutdown is cooperative and ordered. The pacemaker is cancelled first so no new
+block boundaries are produced. The sequencer is cancelled next and drains
+in-flight work before joining its executors. Ledger, simulator, subscription
+cleanup, and other backing workers are cancelled last. Each tier gets a bounded
+window to report why it stopped before the manager moves on.
+
+Inside engine-owned services, registration is explicit: the manager hands the
+service a `ShutdownHandle`, the service waits on its tier token, and it reports
+why it stopped.
+
+```rust
+use nucleus::shutdown::{Service, ShutdownManager, ShutdownReason};
+
+fn spawn_cleanup(shutdown: &mut ShutdownManager) {
+    let mut handle = shutdown.handle(Service::SubscriptionsCleanup);
+
+    tokio::spawn(async move {
+        handle.signalled().await;
+        // Drop subscriptions, release timers, or flush local cleanup state here.
+        handle.terminate(ShutdownReason::Signalled);
+    });
+}
+```
+
+External services integrate at the supervisor boundary today: stop their ingress
+or worker tasks before calling `terminate()`, rather than reusing engine-owned
+`Service` labels as external names. If an external service needs to be registered
+into the same ordered drain later, add that service explicitly to the shutdown
+API when the integration owns a real ordering requirement.
+
+## Account Model
+
+The engine is organized around one question: who owns an account right now?
+
+Mutable accounts are under the rollup's exclusive control. The engine is the
+authoritative owner, so the account is persisted on disk and survives restart.
+Read-only accounts are owned by chain or service state. The engine can always
+re-fetch them, so they live in volatile memory.
+
+That distinction is dynamic. A transaction can delegate an account into the
+rollup, hand it back, or create state that only exists inside the rollup. The
+storage layer routes those transitions without leaving stale copies behind.
+
+## Ergonomic Account Operations
+
+Account mutation is privileged, so callers do not hand-write storage mutations.
+`Engine::account(pubkey)` builds MagicRoot instruction sequences, signs them with
+the engine authority, submits them, and waits for the committed result.
+
+```rust
+use engine::Engine;
+use solana_account::{AccountBuilder, AccountFieldPatch, AccountMode, OwnedAccount, ReadableAccount};
+use solana_pubkey::Pubkey;
+
+async fn account_flow(engine: &Engine, owner: Pubkey) -> engine::Result<()> {
+    let pubkey = Pubkey::new_unique();
+
+    let account = AccountBuilder::default()
+        .lamports(1_000_000)
+        .owner(owner)
+        .mode(AccountMode::Delegated)
+        .data(b"hello engine".to_vec())
+        .build();
+
+    engine.account(pubkey).create(account, None).await?;
+
+    let current = engine.accounts().get(&pubkey)?.expect("account exists");
+    assert_eq!(current.data(), b"hello engine");
+
+    engine
+        .account(pubkey)
+        .patch(vec![
+            AccountFieldPatch::Lamports(2_000_000),
+            AccountFieldPatch::DataAt {
+                offset: 6,
+                data: b"rollup".to_vec(),
+            },
+        ])
+        .await?;
+
+    let replacement = AccountBuilder::default()
+        .lamports(3_000_000)
+        .owner(owner)
+        .mode(AccountMode::Ephemeral)
+        .data(b"local-only state".to_vec())
+        .build();
+
+    engine.account(pubkey).update(replacement).await?;
+    engine.account(pubkey).delete().await?;
+
+    Ok(())
+}
+```
+
+Callers that need chain-owned accounts can coordinate cache misses
+with `engine.accounts().ensure()`. Stored accounts are skipped. The first caller
+to see a missing account owns the load and calls `commit()` after storing it;
+concurrent callers wait on the same pubkey.
+
+```rust
+use std::future::Future;
+
+use engine::Engine;
+use keeper::{MissingAccount, error::KeeperError};
+use solana_account::AccountSharedData;
+use solana_pubkey::Pubkey;
+
+async fn ensure_accounts<F, Fut>(
+    engine: &Engine,
+    pubkeys: &[Pubkey],
+    mut fetch_account: F,
+) -> engine::Result<()>
+where
+    F: FnMut(Pubkey) -> Fut,
+    Fut: Future<Output = engine::Result<Option<AccountSharedData>>>,
+{
+    let accounts = engine.accounts();
+
+    for missing in accounts.ensure(pubkeys) {
+        match missing {
+            MissingAccount::Load(load) => {
+                if let Some(account) = fetch_account(load.pubkey).await? {
+                    engine.account(load.key).create(account, None).await?;
+                    load.commit();
+                }
+            }
+            MissingAccount::Wait(wait) => {
+                wait.wait().await;
+            }
+        }
+    }
+
+    Ok(())
+}
+```
+
+## Transactions
+
+The transaction API accepts the shape callers already have. A slice of
+instructions becomes a signed transaction using the engine authority and latest
+blockhash. A `Message` works the same way. A pre-sanitized `TransactionView` can
+also be submitted directly.
+
+```rust
+use engine::Engine;
+use solana_instruction::{AccountMeta, Instruction};
+use solana_pubkey::Pubkey;
+
+async fn run_instruction(engine: &Engine, program_id: Pubkey, account: Pubkey) -> engine::Result<()> {
+    let ix = Instruction {
+        program_id,
+        accounts: vec![AccountMeta::new(account, false)],
+        data: vec![1, 2, 3, 4],
+    };
+
+    engine.transaction(&[ix])?.execute().await?.map_err(Into::into)
+}
+```
+
+Execution commits account changes and records a status receipt. Simulation runs
+against current state on owned account copies and returns the execution record
+without committing anything. Scheduling queues work without waiting for the
+result.
+
+```rust
+use engine::Engine;
+use solana_instruction::Instruction;
+
+async fn transaction_modes(engine: &Engine, instructions: &[Instruction]) -> engine::Result<()> {
+    let simulated = engine.transaction(instructions)?.simulate().await?;
+    if simulated.is_ok() {
+        engine.transaction(instructions)?.schedule().await?;
+    }
+
+    Ok(())
+}
+```
+
+## Live Subscriptions
+
+Reads and subscriptions come from the keeper namespaces exposed by `Engine`.
+They are plain broadcast streams for state that is already being committed:
+accounts, program-owned accounts, transaction statuses, logs, processed
+transactions, and blocks.
+
+```rust
+use engine::Engine;
+use keeper::TransactionView;
+use solana_account::ReadableAccount;
+use solana_pubkey::Pubkey;
+
+async fn watch(engine: &Engine, account: Pubkey, program: Pubkey) {
+    let accounts = engine.accounts().subscribe(account).await;
+    let program_accounts = engine.accounts().subscribe_program(program).await;
+    let logs = engine.transactions().subscribe_logs(account).await;
+    let processed = engine.transactions().subscribe_processed();
+    let blocks = engine.blocks().subscribe();
+
+}
+
+```
+
+## Startup And Recovery
+
+A normal restart opens persisted state. The engine does not replay the full
+ledger when accountsdb is already current.
+
+Replay is the corruption-recovery path. If accountsdb opens corrupt, keeper
+restores the newest archived snapshot from retained superblocks. That can leave
+accountsdb behind the ledger tip, so engine replay re-executes retained ledger
+entries from the restored slot forward. At each sealed superblock it quiesces the
+replay sequencer and compares the reconstructed account checksum with the sealed
+checksum. A mismatch is `ReplayError::StateMismatch`.
+
+Volatile state can be dropped at runtime because it mirrors chain-owned accounts.
+Persisted rollup-exclusive state is left intact.
+
+## Crates
+
+The workspace is layered bottom to top. Each crate knows only about the layers
+below it.
+
+- `nucleus` holds shared primitives, ledger schema, heed helpers, runtime
+  barriers, and cooperative shutdown.
+- `solana/*` contains the Solana forks reshaped around the engine's account
+  representation and execution model.
+- `accountsdb` routes accounts between persisted and volatile storage and owns
+  snapshots, backup, and defrag.
+- `ledger` records transactions, block boundaries, and execution metadata in
+  self-contained superblocks.
+- `keeper` opens durable state, owns read-side caches, seeds startup state, and
+  fans out subscriptions.
+- `processor` schedules non-conflicting transactions across SVM executors and
+  commits results through keeper.
+- `engine` is the consumer-facing handle that wires keeper, processor, replay,
+  pacemaking, account mutation, and transaction submission together.
+
+## Transaction Flow
+
+Inbound transactions reach the processor, which resolves conflicts by account
+locks and fans non-conflicting work across its executor pool. Executors run the
+transaction through SVM, reading accounts through keeper/accountsdb. Results are
+appended to the ledger, dirty accounts are written back, and subscriptions are
+published. When a superblock seals, keeper snapshots accountsdb and archives it
+beside the sealed ledger segment.