sync to blockchain time

[ryardley]

As the protocol becomes more complex we are having timing issues within out end to end tests creep in.

we need to fastforward in sync with blockchain time.

We should do this by:

Making all "waits" dependent on blocks received from whatever chain we are listening to.

This means listening to each block and processing the block extracting the timestamp and considering that the timestamp to react to within the protocol.

Eg

test('full e2e with blockchain time travel', async ({ page }) => {
  // 1. Mock browser time
  await page.clock.install({ time: Date.now() });
  
  await page.goto('http://localhost:3000');
  
  // 2. Advance blockchain time (via test RPC)
  await page.evaluate(async () => {
    await window.ethereum.request({
      method: 'evm_increaseTime',
      params: [3600]
    });
    await window.ethereum.request({ method: 'evm_mine' });
  });
  
  // 3. Advance browser time to match
  await page.clock.fastForward(3600000); // Should probably not be required
  
  // Now both client and blockchain are synchronized in the future
  await page.click('#check-status');
  await expect(page.locator('#result')).toContainText('ready');
});

We should ensure all components synchronize with block advancement to make end to end tests run more quickly.

[ryardley]

This is related to #839

[ryardley]

Investigating this today:

This will create further listeners per chain.

What we really want is something like this for efficiency:

// each block
{
  jsonrpc: "2.0",
  id: 1,
  method: "eth_subscribe",
  params: ["newHeads"]
}

// all contract events
{
  jsonrpc: "2.0",
  id: 2,
  method: "eth_subscribe",
  params: ["logs", {
    address: ["0xcontract1", "0xcontract2"],
    topics: [["0xevent1topic", "0xevent2topic"]]
  }]
}

This would mean that per chain we only have two subscriptions instead of once per contract and then a block subscription.

Ciphernodes / Rust

Fixing up Committe Finalizer delay.

CommitteeFinalizer will trigger EnclaveEvent::DispatchLater on the EventBus this event.

enum BlockOrTimestamp {
  // Block(u64), 
  Timestamp(u64)
}

struct DispatchLater {
  when: BlockOrTimestamp,
  chain_id:ChainId,
  cancel_id:CorrelationId,
  payload: EnclaveEvent
}

The alarm will dispatch the given EnclaveEvent on the EventBus once the chain has reached the required timestamp on the given chain_id.

For optimization either here or later we should consider refactoring and consolidating crates/evm subscriptions to do something like the following:

• Wrap the Provider in a EvmInterface that handles all streaming duties.
• We can probably ditch the HistoricalEventsCoordinator as the EvmInterface will be centralized for all contracts and can handle this functionality. It should still dispatch an event to indicate that it has finished polling for historical events.
• EvmInterface communicates EvmEnclaveEvent::Log messages to EvmReader actor which are then extracted using the configured extractors

---
config:
  look: neo
---
flowchart TB
    n1["EvmInterface: Wrap the provider and handle subscription to the provider and forwards `Log` to evm reader"] L_n1_d_0@-.- d["EvmInterface"]
    n3["EvmReader: Handles event extraction from Log messages. Configured with relevant `extractor`"] L_n3_h_0@-.- f
    n4["EvmAlarmClock: Can respond to EnclaveEvents requesting notifications by triggering named alarm events"] L_n3_h_0@-.- n2
    a["EnclaveSolReader"] --> f["EvmReader"]
    c["CiphernodeRegistryReader"] --> g["EvmReader"]
    b["BindingRegistryReader"] L_b_h_0@--> h["EvmReader"]
    f L_f_d_0@<-.-> d
    g L_g_d_0@<-.-> d
    h L_h_d_0@<-.-> d
    d <-.-> e["Provider { chain_id:1 }"]
    n2["EvmAlarmClock"] L_n2_d_0@<-.-> d
    n1@{ shape: card}
    n2@{ shape: rect}
    n3@{ shape: card}
     d:::Peach
     a:::Peach
     f:::Peach
     c:::Peach
     g:::Peach
     b:::Peach
     h:::Peach
     e:::Aqua
     n2:::Peach
    classDef Ash stroke-width:1px, stroke-dasharray:none, stroke:#999999, fill:#EEEEEE, color:#000000
    classDef Rose stroke-width:1px, stroke-dasharray:none, stroke:#FF5978, fill:#FFDFE5, color:#8E2236
    classDef Aqua stroke-width:1px, stroke-dasharray:none, stroke:#46EDC8, fill:#DEFFF8, color:#378E7A
    classDef Peach stroke-width:1px, stroke-dasharray:none, stroke:#FBB35A, fill:#FFEFDB, color:#8F632D
    style n1 stroke-width:1px,stroke-dasharray: 1
    L_n1_d_0@{ curve: linear } 
    L_b_h_0@{ curve: linear } 
    L_f_d_0@{ curve: linear } 
    L_g_d_0@{ curve: linear } 
    L_h_d_0@{ curve: linear } 
    L_n2_d_0@{ curve: linear } 
    L_n3_h_0@{ curve: linear }

Loading

We SHOULD NOT refactor straight to this form. Firstly our line limit wont allow it. Secondly we should set up the EvmAlarmClock first so we get the functionality under test and then refactor the other readers to use the EvmInterface as an optimization step.

CRISP Server / Rust [DONE]

In CRISP we currently do something like this:

if !wait_duration.is_zero() {
  sleep(wait_duration).await;
}

What we need is to have a method on indexer for waiting.

What we need is something like this:

indexer.dispatch_later(timestamp, async move |b| {...});

This should forward to a BlockListener that takes an Arc<Provider> and can execute async closures on each block.

We then need a struct that stores closures and returns a Vec of all expired ones when passed a timestamp. I am thinking it should be called ThresholdQueue it should be generic to use on both async handlers as well as events.

We can then use a BinaryHeap<Reverse> where Cmp is implemented for the timestamp. This should keep everything in min first order meaning that we can peek at the value and get the lowest timestamp every block.

type AsyncCallback = Box<dyn FnOnce() -> Pin<Box<dyn Future<Output = ()> + Send>> + Send>;

struct ThresholdItem<T> {
    threshold: u64,
    item: T,
}

pub struct ThresholdQueue<T> {
    heap: BinaryHeap<Reverse<ThresholdItem<T>>>,
}

pub fn insert(&mut self, item:T);
pub fn take_until_including(&mut self, threshold: u64) -> Vec<T>; // can now do whatever to `Vec<T>`

We can then implement PartialOrd Ord and all the business that BinaryHeap needs for ThresholdItem using the threshold. threshold being probably timestamp but it might also blockheight.

This pattern should be usable in EvmAlarmClock to store events.

Typescript / Default Template

We should follow the same sort of structure as for CRISP.

graph TD
  a[EnclaveSdk]
  a --> b[EventListener]
  a --> c[BlockListener]

Loading

BlockListener stores async closures that process blocks

We can use a class that stores closures in a ThresholdQueue similar to Rust but it is probably fine to use a sorted vector as there is not Heap in the standard library of JS

TODO

• Ciphernodes
• CRISP Rust feat: connect crisp to blockchain time [skip-line-limit] #1052
• TypeScript (basically copy feat: connect crisp to blockchain time [skip-line-limit] #1052 but for typescript SDK)

[ryardley]

This will supercede #989

[ryardley]

Obviously this is going to be slow multiphase rollout. We have #1052 as the current resting place for this which refactors CRISP to use blockchain time. There might be an argument or requirement to look at this whilst setting up event sourcing for the ciphernodes but it probably can happen afterwards.