feat: event-time commits with @recorded: audit-axis time travel#1429
feat: event-time commits with @recorded: audit-axis time travel#1429bplatz wants to merge 1 commit into
Conversation
Commits now distinguish two time axes: - Event time (Commit.time, existing field): user-suppliable per transaction via opts.eventTime / CommitOpts::with_timestamp. @iso: time travel resolves against it, so historical data replayed as ordinary transactions gets real wall-clock time travel. Enforced monotonically non-decreasing along the chain and rejected beyond now + 5min skew (an immutable future stamp would permanently wedge the timeline). The default stamp is clamped to the head's event time, fixing pre-existing silent @iso: mis-resolution when the system clock stepped backwards between commits. - Recorded time (db:receivedAt, new system txn-meta entry): wall-clock time the commit was recorded. Emitted from the first caller-supplied event time onward (sticky dual-stamp mode), so plain ledgers stay byte-identical. The new @recorded: selector (and TimeSpec::AtRecorded) time-travels the audit axis, falling back to the event axis for history before the flip point. Implementation notes: - HeadTemporal (event ms + receivedAt ms) rides LedgerState, captured during the load-time novelty walk and after each commit; resolved lazily via ensure_head_temporal (one storage read, write path only) using the branch-aware store - a flat store 404s on branched ledgers whose head commit lives in an ancestor namespace. - receivedAt reuses the txn_meta envelope channel (no wire-format change; old readers decode new commits). The indexer resolver's reserved-namespace debug_assert now exempts system-injected entries; it already mis-fired on f:identity in debug builds. - apply_single_commit gains soft guards mirroring the build-path validation (event-time monotonicity + dual-stamp continuity). - fluree-memory git replay now backdates the schema transaction to the first git commit and clamps non-monotonic git author dates. - No format changes anywhere: no lower bound on event time for a fresh ledger (pre-1970 works), no migration, no cost to ledgers that never use the feature.
078af0b to
929b024
Compare
defbd6b to
4d4430c
Compare
aaj3f
left a comment
There was a problem hiding this comment.
This is really neat! It's something we've always dismissed as separate from our time-axis concerns but I quite like this
| // Guard: event time must be monotonically non-decreasing along the | ||
| // chain. `datetime_to_t` (`@iso:` resolution) relies on this ordering; | ||
| // a violation would silently mis-resolve wall-clock time travel. | ||
| // Soft guard: only enforced when both sides are known — legacy commits | ||
| // without parseable times fall through rather than wedging the ledger. | ||
| let commit_temporal = HeadTemporal::from_commit(&commit); | ||
| if let (Some(prev), Some(new)) = (self.head_temporal, commit_temporal) { | ||
| if new.event_time_ms < prev.event_time_ms { | ||
| return Err(LedgerError::InvalidData(format!( | ||
| "Cannot apply commit at t={commit_t}: event time {} is earlier than \ | ||
| the current head's event time {} (event time must be monotonically \ | ||
| non-decreasing)", | ||
| new.event_time_ms, prev.event_time_ms | ||
| ))); | ||
| } | ||
| if prev.dual_stamp() && !new.dual_stamp() { | ||
| return Err(LedgerError::InvalidData(format!( | ||
| "Cannot apply commit at t={commit_t}: ledger is in dual-stamp mode \ | ||
| (head commit carries db:receivedAt) but this commit does not; \ | ||
| @recorded: resolution requires every post-flip commit to dual-stamp" | ||
| ))); | ||
| } | ||
| } |
There was a problem hiding this comment.
The apply_single_commit event-time monotonicity guard hard-errors on the replay/catch-up path, where it is applied to commits that are already durable in the chain. The comment calls this a "soft guard … rather than wedging the ledger," but it only falls through for unparseable times — a legacy commit with a parseable but non-monotonic event time (real NTP step-back, or leadership handoff between raft nodes with skewed clocks — exactly the pre-existing scenario the PR description says exists) will return Err. apply_single_commit is the incremental catch-up path (ledger_manager.rs:1971), so a pre-PR ledger whose chain has any backwards clock step between two committed events will now fail incremental catch-up on upgrade. (A full reload via load_novelty/ bulk_apply_commits has no such guard, so behavior is also inconsistent between the two load paths, and the failure may repeat until a reload is forced.) The build path (resolve_commit_times) is the correct place to reject a new user commit; the replay path should tolerate history it can no longer change. Suggest warn-and-continue on apply:
let commit_temporal = HeadTemporal::from_commit(&commit);
if let (Some(prev), Some(new)) = (self.head_temporal, commit_temporal) {
if new.event_time_ms < prev.event_time_ms {
// Already-durable history: a pre-existing non-monotonic chain
// (clock skew / pre-PR commits) must still load. The build path
// rejects *new* violations; here we only surface the anomaly.
tracing::warn!(
commit_t, new = new.event_time_ms, prev = prev.event_time_ms,
"applying commit whose event time predates the head; @iso: \
resolution may be approximate across this point"
);
}
if prev.dual_stamp() && !new.dual_stamp() {
tracing::warn!(commit_t, "post-flip commit missing db:receivedAt");
}
}(If you want to keep the dual-stamp arm strict, note legacy ledgers can never be dual-stamped, so it only bites forward-built data and is lower risk than the monotonicity arm.)
| let received_at_ms = if dual_stamp { | ||
| let ms = match received { | ||
| Some(ts) => iso_to_epoch_ms_opt(&ts).ok_or_else(|| { | ||
| TransactError::InvalidEventTime(format!( | ||
| "receivedAt '{ts}' is not a valid RFC 3339 timestamp" | ||
| )) | ||
| })?, | ||
| None => now_ms, | ||
| }; | ||
| let prev = head.and_then(|h| h.received_time_ms); | ||
| Some(prev.map_or(ms, |p| ms.max(p))) | ||
| } else { | ||
| None | ||
| }; |
There was a problem hiding this comment.
The audit axis has no future-skew guard, unlike the event axis. resolve_commit_times validates that a caller-supplied event time is <= now + MAX_EVENT_TIME_FUTURE_SKEW_MS, but a caller-supplied received_at (via CommitOpts::with_received_at, reachable from the Rust API / CLI replay hooks) is accepted with any value. The PR's own argument — "commits are immutable and [the axis] is monotonic, so a future stamp would permanently pin the ledger's timeline ahead of reality" — applies verbatim to db:receivedAt. The HTTP route is safe (it always passes Utc::now()), so this is defense-in-depth for the programmatic path. Suggest applying the same future-bound check to the resolved received_at_ms:
let received_at_ms = if dual_stamp {
let ms = match received { /* … */ };
if ms > now_ms + MAX_EVENT_TIME_FUTURE_SKEW_MS {
return Err(TransactError::InvalidEventTime(format!(
"receivedAt is in the future; the audit axis is immutable and monotonic"
)));
}
let prev = head.and_then(|h| h.received_time_ms);
Some(prev.map_or(ms, |p| ms.max(p)))
} else { None };| let earliest_flakes = | ||
| probe_timestamp_axis(snapshot, overlay, recv_predicate.clone(), current_t, None).await?; | ||
|
|
||
| let Some(earliest) = earliest_flakes.first() else { | ||
| // Never dual-stamped: the recorded axis is identical to event time. | ||
| tracing::debug!("recorded_to_t: no db:receivedAt flakes; falling back to event axis"); | ||
| return datetime_to_t(snapshot, overlay, target_epoch_ms, current_t).await; | ||
| }; |
There was a problem hiding this comment.
@recorded: on a ledger that never dual-stamped does three single-flake POST probes (the empty receivedAt earliest-probe here, then datetime_to_t's two event-axis probes) versus @iso:'s two. Each probe is an O(log n) indexed single-flake read, so the absolute cost is small, but since the docs promise @recorded: is "identical to @iso: cost," it's worth a note. No change required; flag only if @recorded: is expected on hot query paths over plain ledgers.
| Once a ledger uses a caller-supplied event time, each commit records a second, | ||
| system-controlled timestamp: **`db:receivedAt`** — the wall-clock time the | ||
| commit was actually recorded. This begins at the first backdated commit and | ||
| continues on every commit thereafter (sticky), so the audit trail has no gaps. | ||
| Ledgers that never supply event times carry no extra metadata at all. |
There was a problem hiding this comment.
"Once a ledger uses a caller-supplied event time, each commit records a second … db:receivedAt" is accurate for the HTTP path (the transact.rs route pairs eventTime with with_received_at(now)), but not for the Rust API: CommitOpts::with_timestamp alone does not flip dual-stamp mode (resolve_commit_times gates it on received.is_some() || head.dual_stamp()). fluree-memory git replay backdates via with_timestamp only, so those ledgers are event-axis-only and never carry receivedAt. Consider clarifying that the audit axis is enabled by the server pairing the two stamps (or by with_received_at directly), not by any event-time use.
Summary
Adds a second, independent time axis to commits so ledgers can support audit-axis time travel without disturbing existing event-time semantics.
Commits now distinguish two times:
Event time (
Commit.time, existing field) — user-suppliable per transaction viaopts.eventTime/CommitOpts::with_timestamp.@iso:time travel resolves against it, so historical data replayed as ordinary transactions gets real wall-clock time travel. Enforced monotonically non-decreasing along the chain and rejected beyondnow + 5minskew (an immutable future stamp would permanently wedge the timeline). The default stamp is clamped to the head's event time, fixing pre-existing silent@iso:mis-resolution when the system clock stepped backwards between commits.Recorded time (
db:receivedAt, new system txn-meta entry) — wall-clock time the commit was recorded. Emitted from the first caller-supplied event time onward (sticky dual-stamp mode), so plain ledgers stay byte-identical. The new@recorded:selector (andTimeSpec::AtRecorded) time-travels the audit axis, falling back to the event axis for history before the flip point.Implementation notes
HeadTemporal(event ms + receivedAt ms) ridesLedgerState, captured during the load-time novelty walk and after each commit; resolved lazily viaensure_head_temporal(one storage read, write path only) using the branch-aware store — a flat store 404s on branched ledgers whose head commit lives in an ancestor namespace.receivedAtreuses thetxn_metaenvelope channel (no wire-format change; old readers decode new commits).debug_assertnow exempts system-injected entries; it already mis-fired onf:identityin debug builds.apply_single_commitgains soft guards mirroring the build-path validation (event-time monotonicity + dual-stamp continuity).fluree-memorygit replay now backdates the schema transaction to the first git commit and clamps non-monotonic git author dates.Testing
fluree-db-api/tests/it_event_time.rs(new) plus history-query coverage.cargo check --workspace --all-features --all-targets— clean.Notes
Stacked on
feature/remote-mounts(#1428); review/merge that first. Docs updated:docs/concepts/time-travel.md,docs/api/endpoints.md.