Dataset observability and coreset selection for robotics imitation learning.
Calibra tells you what is wrong with your robot demonstrations — and removes the redundant ones — before you waste GPU time training on bad data.
Source-available and local-first (BSL 1.1) — free for research and internal use; converts to Apache 2.0 on 2030-06-30. See LICENSE and LICENSING.md. Not OSI open source: reselling Calibra as a hosted/managed service requires a commercial license.
pip install calibra-robotics
calibra compare hf://lerobot/my_dataset aloha
calibra certify /data/my_demos --reference aloha --policy diffusion
calibra prune /data/100k_episodes --keep 0.3 --out coreset.json
calibra retarget /data/isaac_lab.h5 --out retargeted/
# New in v0.6.0
calibra predict /data/my_demos.h5 --policy gr00t # predict training success before training
calibra predict /data/my_demos.h5 --record-outcome 0.82 # close the loop after training
calibra card /data/my_demos.h5 --push # push quality card to HuggingFace Hub
calibra watch /data/session/ --remediate # real-time operator feedback
calibra watch --stream --remediate # pipe metrics from collect script
calibra calibrate # re-fit weights from training historyRobot learning labs collect thousands of demonstration episodes. Naively training on all of them:
- Silently trains on bad data — jerk spikes, dropped frames, communication lag, and stuck actuators all look like valid training signal to your policy.
- Wastes compute on redundancy — in a 10,000-episode dataset, 60–80% of episodes are near-duplicates. GPU cost scales with volume, not uniqueness.
- Produces undiagnosable failures — when a policy stalls or flails, you have no idea whether the cause is the architecture, the training recipe, or the data itself.
Calibra solves the data side.
| Command | Description |
|---|---|
calibra (default) |
Full diagnostic audit report |
calibra compare |
Evidence-backed cross-dataset comparison |
calibra certify |
Structured pass/fail certification |
calibra prune |
Two-stage coreset selection |
calibra corrupt |
Inject synthetic corruptions to validate metric sensitivity |
calibra retarget |
Convert absolute EEF actions to relative delta actions |
calibra predict |
Predict training outcome before spending GPU time |
calibra card |
Generate a HuggingFace dataset quality card |
calibra watch |
Real-time teleoperation quality monitor |
calibra score |
Composite 0–100 quality score across four dimensions |
calibra sim2real |
Quantify sim-to-real distribution gap |
calibra transfer |
Cross-embodiment compatibility scoring |
calibra cure |
Automatic data remediation (smoothing, resampling, trimming) |
calibra serve |
Local REST API server and web dashboard |
calibra /data/robot_demos.h5
calibra lerobot/pusht --policy diffusion
calibra /data/demo.h5 --policy act --json
calibra /data/robot_demos.h5 --html-out report.html # save visual HTML dashboardRuns four analyzers over every episode and flags anomalies with bootstrap confidence intervals and per-episode outlier detection.
calibra compare /data/my_demos pusht
calibra compare hf://lerobot/my_dataset aloha
calibra compare /data/robot.h5 aloha --format hdf5 --gripper-dims 6,13━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
calibra compare — my_dataset vs. aloha
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Reference: lerobot/aloha_mobile_cabinet (position-command · 14D · 85 episodes)
Yours: my_dataset (120 episodes)
────────────────────────────────────────────────────────
VELOCITY DISCONTINUITY RATE
Yours: 12.1%
aloha 1.3%
Delta: +10.8% ▲
Significantly rougher than aloha_mobile_cabinet.
If using position commands: investigate control noise or
abrupt operator corrections.
Confidence: HIGH · [HIGH · n=2 (aloha_sim, aloha_mobile)]
────────────────────────────────────────────────────────
JERK SPIKE RATE
Yours: 8.4%
aloha 0.7%
Delta: +7.7% ▲
Higher spike rate than reference. Check for dropped
frames, bad episode boundaries, or bimodal speed profiles.
Confidence: MODERATE · [LOW-MODERATE · n=1 (aloha_sim)]
────────────────────────────────────────────────────────
RECOMMENDED ACTIONS
────────────────────────────────────────────────────────
Prune episode(s) 14, 22, 41 — jerk outliers detected by MAD analysis.
Velocity discontinuity rate is 12.1% (above 4% position-control
threshold). Investigate command packet drops, hardware communication
lag, or abrupt operator corrections.
────────────────────────────────────────────────────────
Every interpretation is backed by a falsifiable claim in calibra/claims/ with an evidence count, confidence rating, and a stated falsification condition. Calibra never guesses.
calibra certify /data/my_demos
calibra certify /data/my_demos --reference aloha --policy diffusion --strict
calibra certify hf://lerobot/my_dataset --json # for CI pipelines━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA CERTIFICATION REPORT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Dataset : my_demos
Episodes : 120
Steps : 180000
Policy : diffusion
Reference: aloha
──────────────────────────────────────────────────────────
⚠ PROVISIONALLY CERTIFIED
Warnings:
• ldlj: Mean LDLJ = -12.4 (threshold: >-10). Action trajectories
contain significant jerk.
──────────────────────────────────────────────────────────
REMEDIATION CHECKLIST
──────────────────────────────────────────────────────────
1. [WARNING] ldlj: High jerk in demonstration data forces the policy
to learn discontinuous action transitions. Consider applying action
smoothing (e.g. Savitzky-Golay) before training.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Exit codes: 0 = CERTIFIED, 1 = PROVISIONALLY CERTIFIED (warnings), 2 = NOT CERTIFIED (critical failures). Wire into CI with --json for machine-readable output.
calibra prune /data/100k_episodes --keep 0.3 --out coreset.json
calibra prune /data/my_ds --keep 0.5 --quality-only
calibra prune /data/my_ds --keep 0.25 --max-spike-rate 0.03 --max-vel-disc-rate 0.08━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA PRUNING SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Original episodes : 1000
Quality failures : 87 (removed in Stage 1)
Diversity pruned : 613 (removed in Stage 2)
Coreset size : 300 (30.0% of original)
Method : quality_filter + greedy_max_coverage
────────────────────────────────────────────────────────
To use: filter your dataset to the episode IDs in keep_episode_ids.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Two-stage pipeline:
- Stage 1 — Quality filter: removes episodes that fail kinematic/temporal thresholds (jerk spike rate, velocity discontinuity, dropout, LDLJ, minimum length).
- Stage 2 — Greedy max-coverage: from the quality-passing pool, selects the K most behaviorally diverse episodes using farthest-point sampling on action-space statistics. O(N × K) — handles ~50k episodes without approximation.
Use --entropy-weight 0.4 (or --policy gr00t) to bias selection toward high-entropy (informationally rich) episodes, which improves GR00T fine-tuning outcomes. Alternatively, use --strategy influence to select episodes based on estimated learning value (combining action novelty, task contact representation, and Shannon entropy).
Output coreset.json contains keep_episode_ids, quality_fail_ids, diversity_pruned_ids, and per-episode quality and diversity scores.
calibra corrupt lerobot/pusht --drop-frames 0.10
calibra corrupt /data/robot.h5 --inject-spikes 0.05
calibra corrupt lerobot/pusht --add-jitter-ms 50 --drop-frames 0.08━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
calibra corrupt — pusht
Corruptions: drop_frames=10.0%
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Metric Original Corrupted Δ React
──────────────────────────────────────────────────────────────────
Timestamp dropout rate 0.0% 9.4% +9.4% 🔴
Timestamp jitter CV 3.0e-06 8.1e-06 +5.1e-06 🟡
Jerk spike rate 4.9% 5.2% +0.3% —
Velocity discontinuity 16.7% 16.9% +0.2% —
Inject synthetic corruptions into a known-good dataset to verify that your metrics actually respond to the defects they claim to detect.
calibra retarget /data/isaac_lab_demos.h5 --out /data/retargeted/
calibra retarget /data/demos.h5 --pad --out retargeted/
calibra retarget /data/demos.h5 --obs-key-pos robot0_eef_pos \
--obs-key-quat robot0_eef_quat━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
calibra retarget — isaac_lab_demos
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Episodes converted : 500
Episodes skipped : 0
Output directory : /data/retargeted/
Action shape : (T−1, 6) [dx, dy, dz, droll, dpitch, dyaw]
Rotation units : radians (intrinsic XYZ)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NVIDIA GR00T N1.7+ uses a Relative End-Effector (EEF) action space. Isaac Lab and robomimic HDF5 datasets record actions in absolute world-frame coordinates. retarget converts absolute 7-DoF poses [x, y, z, qx, qy, qz, qw] into 6-DoF local-frame deltas [dx, dy, dz, droll, dpitch, dyaw] — one .npz per episode.
Use --pad to append a zero row so output shape is (T, 6) instead of (T−1, 6) when your policy requires fixed-length sequences.
calibra predict /data/my_demos.h5
calibra predict lerobot/my_dataset --policy diffusion --reference aloha
calibra predict /data/my_demos.h5 --policy gr00t --record-outcome 0.82━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA TRAINING OUTCOME PREDICTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Dataset : my_demos · Episodes: 120 · Policy: gr00t
🟢 Predicted Success: 81% [range 71%–91%] — GOOD
──────────────────────────────────────────────────────────
⚠️ -8.0pt ldlj
Mean LDLJ = -12.4. High jerk forces discontinuous action transitions.
──────────────────────────────────────────────────────────
NEXT STEPS
✓ Data quality is sufficient. Proceed with training.
After training, close the loop:
calibra predict <dataset> --record-outcome <actual_success_rate>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
--record-outcome RATE stores the observed training success rate alongside the diagnostic fingerprint in ~/.calibra/outcomes.jsonl. Future predictions on similar datasets blend the heuristic score with these empirical observations via inverse-distance weighting. Run calibra calibrate after 10+ outcomes to re-fit the prediction weights from your lab's actual training history.
calibra card /data/my_demos.h5
calibra card lerobot/my_dataset --policy diffusion --out quality_card.md
calibra card /data/my_demos.h5 --push # push directly to HuggingFace Hub READMEGenerates a structured Markdown quality card with certification badge, per-metric status table, and predicted training outcome. Embed it in your dataset's HuggingFace Hub README so other researchers can see data quality at a glance.
calibra watch /data/collection_session/
calibra watch /data/session/ --remediate # print fix instructions on failure
calibra watch /data/session/ --log-file session.jsonl
# Stream mode: pipe metrics from your collection script
python collect_demos.py | calibra watch --stream --remediate━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA WATCH — real-time data quality monitor
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Remediation advice: ON
Watching: /data/collection_session/
✅ [ 1] ep_001.h5 PASS — all metrics OK
✅ [ 2] ep_002.h5 PASS — all metrics OK
❌ [ 3] ep_003.h5 FAIL — jerk_spike_rate = 0.087
↳ RE-RECORD: Move more smoothly — avoid abrupt stops and direction changes.
✅ [ 4] ep_004.h5 PASS — all metrics OK
--remediate prints a specific operator instruction on every FAIL/WARN: what caused the failure and exactly how to fix the motion. Operators get feedback within seconds of saving an episode instead of discovering problems during training hours later.
--stream reads JSON metric lines from stdin, enabling integration with teleoperation software without filesystem round-trips. See examples/lerobot_watch_integration.py for a drop-in integration snippet.
calibra score /data/robot_demos.h5
calibra score lerobot/my_dataset --policy diffusion
calibra score /data/my_ds --reference aloha --json
calibra score hf://lerobot/pusht_image --badge # print markdown badge for dataset cards━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA SCORE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Dataset : my_demos
Episodes : 120 · Steps: 180000
────────────────────────────────────────────────────────────
🟢 78.0 / 100 — Good
────────────────────────────────────────────────────────────
Temporal Stability 22.00/25 [█████████████████░░░] 88%
jitter_cv: 0.038
dropout_rate: 0.003
Control Smoothness 26.00/35 [██████████████░░░░░░] 74%
ldlj: -10.6
spike_rate: 0.021
vel_disc_rate: 0.027
Coverage / Diversity 19.00/25 [███████████████░░░░░] 76%
action_entropy_bits_per_dim: 2.9
Task Structure 11.00/15 [██████████████░░░░░░] 73%
trajectory_diversity: 0.31
short_episode_fraction: 0.04
0 critical flags · 3 warnings
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Aggregates all four diagnostic dimensions into a single 0–100 number: Temporal Stability (25 pts), Control Smoothness (35 pts), Coverage/Diversity (25 pts), and Task Structure (15 pts). Score categories: 90–100 Excellent, 75–89 Good, 60–74 Fair, 40–59 Poor, 0–39 Critical. Use --badge to generate a shields.io markdown badge for HuggingFace dataset cards. Exit codes: 0 = Good or better (≥75), 1 = Fair or Poor (40–74), 2 = Critical (<40).
calibra sim2real /data/sim_demos.h5 /data/real_demos.h5
calibra sim2real lerobot/sim_dataset /data/real.h5 --policy pi0
calibra sim2real /data/sim.h5 /data/real.h5 --json━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA SIM-TO-REAL GAP ANALYSIS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sim dataset : isaac_lab_pick (500 eps)
Real dataset : real_pick (120 eps)
────────────────────────────────────────────────────────────
🟡 Overall Transfer Risk: MEDIUM
📊 Pre-training Alignment Index (PAI): 71.3%
────────────────────────────────────────────────────────────
🟢 Ldlj Gap [LOW]
Sim: -6.2 Real: -8.1 Δ = 1.9
→ Real motions are smoother than sim.
🟡 Action Kl Divergence [MEDIUM]
Value: 0.73
→ KL(sim||real) = 0.730. Significant action distribution mismatch.
🟢 Sim Coverage Of Real [LOW]
Value: 0.81
→ Sim covers 81% of the real action space. Good coverage.
🟢 Control Frequency Gap [LOW]
Sim: 50.0 Real: 50.0 Δ = 0.0
→ Sim runs at 50 Hz, real at 50 Hz. Frequency match is good.
────────────────────────────────────────────────────────────
RECOMMENDATIONS
────────────────────────────────────────────────────────────
• Consider collecting a small real dataset (50–200 episodes) for
fine-tuning or domain randomisation in sim.
• Use `calibra prune` to select the sim episodes closest to the
real distribution before training.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Measures the distribution gap between a simulation and real-robot dataset across action-space KL divergence, trajectory smoothness delta, coverage overlap, transition dynamics, and control frequency mismatch. Reports an overall transfer risk level (LOW / MEDIUM / HIGH / CRITICAL) and a Pre-training Alignment Index (PAI, 0–100%) summarising how well the sim distribution covers real-world conditions. Exit codes: 0 = LOW or MEDIUM, 1 = HIGH, 2 = CRITICAL.
calibra transfer /data/source_robot.h5 /data/target_robot.h5
calibra transfer lerobot/aloha_mobile_cabinet lerobot/svla_so100_pickplace
calibra transfer /data/source.h5 /data/target.h5 --json━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA CROSS-EMBODIMENT TRANSFER SCORE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Source : aloha_mobile_cabinet (85 eps)
Target : franka_pick (60 eps)
────────────────────────────────────────────────────────────
🟡 Transfer Compatibility: ADAPT
────────────────────────────────────────────────────────────
🟡 Action Dimensionality [ADAPT]
Source has 14D actions, target has 7D. Subset retargeting
(drop extra dims) may work — use `calibra retarget` to convert.
✅ Control Frequency [DIRECT]
Control frequencies are similar (50 Hz vs 50 Hz).
✅ Trajectory Smoothness [DIRECT]
Similar smoothness profiles (ΔLDLJ = 1.80).
✅ Episode Length [DIRECT]
Similar episode lengths (410 vs 390 steps).
🟡 Action Range Overlap [ADAPT]
Source covers 63% of target action range. Some target actions
have no source demonstrations.
────────────────────────────────────────────────────────────
RECOMMENDATIONS
────────────────────────────────────────────────────────────
• Normalise action spaces before mixing source and target data.
• Use `calibra retarget` if action dims differ.
• Consider weighting source data lower (e.g. 0.3×) than target data.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Scores the compatibility of reusing source-robot demonstrations to train a policy for a different target robot across five dimensions: action dimensionality, control frequency, trajectory smoothness, episode length, and action range overlap. Levels: DIRECT (mix freely), ADAPT (normalise or retarget first), DIFFICULT (targeted domain adaptation required), INCOMPATIBLE (structural mismatch). Exit codes: 0 = DIRECT or ADAPT, 1 = DIFFICULT, 2 = INCOMPATIBLE.
calibra cure /data/robot_demos.h5 --out cured/
calibra cure /data/demos.h5 --remedy smooth,trim --out cured/
calibra cure lerobot/pusht --hz 10 --out cured/ --format lerobot━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
calibra cure — my_demos
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Episodes cured : 120
Output directory : /data/cured/
Manifest written : /data/cured/cure_manifest.json
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Automatically applies kinematic and temporal fixes to every episode and writes cleaned per-episode .npz files. The default remedy pipeline is smooth,interpolate,trim: Savitzky-Golay filtering removes jerk spikes, uniform resampling resolves packet drops and timing jitter, and dead-time trimming cuts leading/trailing static segments. Use --remedy to apply a subset, --hz to pin the output control frequency, and --trim-threshold to tune the motion-detection sensitivity. A cure_manifest.json records original and cured step counts and Hz for every episode.
calibra serve # start on localhost:7842
calibra serve --port 8000
calibra serve --host 0.0.0.0━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CALIBRA SERVE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Web dashboard : http://localhost:7842
REST API : http://localhost:7842/api/v1
Press Ctrl+C to stop.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Starts a local HTTP server that exposes all Calibra diagnostics as a REST API and serves the visual web dashboard at http://localhost:7842. Useful for programmatic access from scripts, CI pipeline integrations, and browsing dataset metrics in a browser without the terminal. Use --host 0.0.0.0 to expose the server on all network interfaces.
Calibra is a dataset observability framework for diagnosing, predicting, and curating robotics training data. Its diversity-based selection component is competitive with established coreset methods (K-Center, Facility Location) rather than a claimed improvement over them, and its offline metrics provide measurable, moderate pre-training predictive signal. The results below are reported at the strength the evidence supports — seeded, paired comparisons are the headline; single-seed runs are labelled exploratory.
To validate Calibra's real-world impact, we ran full policy training and evaluation loops on an RTX 2080 GPU under two critical validation setups.
Trains a Behavior Cloning (BC) policy on a curated 30% Calibra coreset vs. the full raw dataset and a random 30% baseline.
| Curation Condition | Training Steps | Avg Coverage | Success Rate ( |
Optimization Steps Saved | CUDA Train Time |
|---|---|---|---|---|---|
| Full Raw Dataset (100%) | 150,000 | 21.9% | 2.0% | Base (0.0% saved) | 186.7s |
| Calibra 30% Coreset | 6,300 | 23.3% | 8.0% | 95.8% fewer steps | 7.8s |
| Random 30% Baseline | 45,000 | 23.8% | 6.0% | 69.5% fewer steps | 56.9s |
⚠️ Reported as exploratory evidence, not a general result. This is a single-seed run and absolute success rates are low (2–8%). A 10-seed variance sweep (experiments/check_pusht_variance.py) shows the clean coreset averages 14.8% ± 6.3%, so single-run gaps at this scale are within training-seed noise. The seeded, paired-t ablation below (5 seeds, 3 datasets) is the result to rely on.On the 95.8% figure: the coreset used 6,300 optimization steps vs. 150,000 for full-data under this benchmark's episode-scaled step schedule — i.e. 95.8% fewer optimization steps and 95.8% lower measured train time under this protocol. This does not imply that retaining 30% of episodes universally cuts training cost by 95.8%; the saving depends on the training-budget protocol.
To reproduce:
python experiments/pusht_real_benchmark.pyValidates that Calibra can intercept dataset failures before wasting GPU compute. Tested across 15 PushT dataset variants corrupted with controlled noise (frame drops, joystick spikes, noisy episodes) at varying severity levels.
-
Predictive Correlation (L6, within-PushT corruption severity): Spearman
$\rho = 0.6749$ ($p = 0.0057$ ) between offline scores and downstream success across the 15 corruption-severity variants of a single task. This measures sensitivity to corruption severity — it is a different question from the cross-dataset ranking correlation reported below. - Failure Prediction Accuracy (L4): 73.3% (11/15 conditions correctly classified as PASS/FAIL prior to training).
- Root-Cause Accuracy (L4): 88.9% (8/9 single-fault modes correctly identified, pinpointing teleoperation spikes and packet loss).
To reproduce:
python experiments/failure_prevention_benchmark.py --save-fig --out-json results_l4l6.jsonOffline predicted success probabilities (calibra predict) achieve a Spearman rank correlation (ρ) of 0.5971 (p = 0.0146, statistically significant) with actual downstream policy success rates across seven verified dataset conditions (ALOHA sim × 4, PushT image, Mobile ALOHA × 2). This is a cross-dataset ranking correlation and is distinct from the within-PushT corruption-severity correlation (ρ = 0.6749) reported above. DROID-100 and BridgeData V2 are excluded from this correlation due to a control-mode mismatch (velocity-command datasets have structurally high vel_disc that the current rubric over-penalises).
Calibra ships reference profiles for 16 datasets — a separate count from the seven datasets used in this correlation.
A five-condition ablation isolates which part of Calibra's pipeline contributes to the improvement. All experiments use BC-MLP policies, 5 random seeds for the baseline, 200–300 epochs, RTX 2080. Gains are relative to the random-k baseline at the same episode budget.
Without contact-aware quality filter (baseline):
| Condition | ALOHA mobile (clean, 14D) | DROID-100 (noisy, 7D) | PushT real (contact, 8D) |
|---|---|---|---|
| Random k (5 seeds) | 0.0209 ± 0.003 | 1.995 ± 0.177 | 0.222 ± 0.080 |
| Quality-filter only | +8.8% | −5.8% (hurts) | −29.2% (hurts) |
| Diversity-only | +13.0% | +16.9% | +48.4% |
| Calibra full pipeline | +22.6% | +16.9% | −30.5% (hurts) |
With contact-aware quality filter (contact_aware=True, default):
| Condition | ALOHA mobile | DROID-100 | PushT real |
|---|---|---|---|
| Quality-filter only | +8.8% (unchanged) | −5.8% (unchanged) | +32.0% |
| Diversity-only | +13.0% (unchanged) | +16.9% (unchanged) | +47.4% (unchanged) |
| Calibra full pipeline | +22.6% (unchanged) | +16.9% (unchanged) | +39.5% |
The contact-aware filter detects that real PushT's velocity discontinuities are contact-driven (vel_disc/spike ratio = 24.4) and relaxes the vel_disc threshold by 3×. This transforms a −30.5% failure into a +39.5% gain while leaving ALOHA and DROID completely unchanged (their ratios are 1.9 and 1.6 — well below the 3.0 threshold).
Quality filtering failure modes (without contact-aware fix):
-
Heterogeneous datasets (DROID) — quality-only collapses coverage of rare robot morphologies. Removing the noisiest episodes inadvertently removes the only representatives of certain robot types.
-
Contact-rich tasks (real PushT) — velocity discontinuities during contact events (block-hits, pushes) are classified as noise. The quality filter removes the most contact-rich demonstrations, which have the highest learning signal.
The contact-aware fix addresses failure mode 2 by using the vel_disc/spike ratio as a contact detector. Failure mode 1 (morphology collapse) remains an open problem — see calibra/strategy.py for the regime-dependent configuration that mitigates it via relaxed thresholds.
The ablation above compares Calibra against random selection. To test it against the coreset / data-selection literature, we add three published baselines — K-Center greedy (Gonzalez farthest-point sampling), Herding (mean-matching), and Facility Location (greedy submodular coverage) — all run on the same per-episode behavioral features as Calibra's diversity stage, so the only variable is the selection algorithm. Every condition is trained over 5 shared seeds with per-seed MSEs paired across conditions, enabling a paired t-test. Keep fraction 30%, BC-MLP, RTX 2080.
Per-dataset improvement vs. random (5-seed mean):
| Method (keep 30%) | ALOHA mobile | DROID-100 | PushT real |
|---|---|---|---|
| Full dataset (100%) | +41.4% | +12.0% | +12.8% |
| K-Center greedy | +21.4% | +3.9% | +46.8% |
| Herding | −17.0% | +2.5% | −19.7% |
| Facility Location | +17.3% | +3.4% | +43.9% |
| Quality-filter only | +5.5% | +7.0% | +36.2% |
| Diversity-only | +17.8% | +24.3% | +46.4% |
| Calibra full | +18.7% | +16.3% | +38.5% |
Aggregate across the 3 datasets (experiments/aggregate_ablation.py):
| Method | Mean rank ↓ | Mean vs-random |
|---|---|---|
| Diversity-only | 2.00 | +29.5% |
| K-Center greedy | 2.00 | +24.0% |
| Calibra full | 2.67 | +24.5% |
| Facility Location | 4.00 | +21.5% |
| Quality-filter only | 4.33 | +16.2% |
| Random | 6.33 | 0.0% |
| Herding | 6.67 | −11.4% |
| Oracle (best method per dataset) | 1.00 | +30.8% |
Calibra full — Win/Tie/Loss vs. each method (paired t-test, α = 0.05):
| Opponent | W-T-L | Notes |
|---|---|---|
| Random | 2-1-0 | never loses |
| K-Center greedy | 1-1-1 | tie on ALOHA (p=0.37), win DROID, loss PushT |
| Herding | 3-0-0 | dominates |
| Facility Location | 1-1-1 | wins DROID, loses PushT |
| Quality-filter only | 2-1-0 | quality+diversity ≥ quality alone |
| Diversity-only | 0-1-2 | loses to its own diversity stage on DROID & PushT |
Takeaways:
- Calibra's diversity (coverage) selection is the strongest method — best mean improvement (+29.5%), tied-best mean rank (2.00), and more consistent than K-Center (worst rank 3 vs. K-Center's rank-4 on DROID). It never significantly loses to any published baseline.
- The quality filter is regime-dependent, not a universal default. On these clean teleop datasets it drags the full pipeline below diversity-only (significant on DROID and PushT). It pays off only when real corruption is present (see the failure-prediction and
corruptbenchmarks). The recommended default is diversity-only, with quality-filtering gated on detected corruption. - Adaptive-across-methods headroom is small — the oracle (+30.8%) beats the best fixed method (diversity-only, +29.5%) by only 1.3 points, so no complex regime-adaptive selector is warranted on this evidence.
To reproduce:
python experiments/ablation_benchmark.py --dataset lerobot/aloha_mobile_cabinet --seeds 5 --json results/ablation_aloha.json
python experiments/ablation_benchmark.py --dataset lerobot/droid_100 --seeds 5 --json results/ablation_droid.json
python experiments/ablation_benchmark.py --dataset lerobot/columbia_cairlab_pusht_real --seeds 5 --json results/ablation_pusht_real.json
python experiments/aggregate_ablation.py results/ablation_*.jsonThe ablation above uses a BC-MLP learner. To test whether the findings are an artifact of that one architecture, we re-ran the entire benchmark — identical datasets, splits, seeds, and byte-identical coreset selection — swapping only the downstream learner across three families:
- BC-MLP — deterministic 3-layer regressor (the original ablation).
- ACT — Action Chunking Transformer: a state-conditioned CVAE with a
transformer encoder–decoder that predicts a 16-step action chunk (masked L1 + KL).
experiments/act_ablation_benchmark.py. - Diffusion Policy — a state-conditioned DDPM that denoises a 16-step action
chunk.
experiments/diffusion_ablation_benchmark.py.
All at 5 seeds, keep 30%, RTX 2080. Every coreset is identical across the three; only the policy changes.
Scope. These are state-based policies scored on offline first-action prediction error (the action each policy executes), not image-conditioned policies evaluated by simulator rollout. They test whether the selection ranking transfers across learners, not task-level success rate. Diffusion is a stochastic sampler, so it is evaluated by single-sample first-action MSE and is only compared relative to Random / by rank within the diffusion learner — absolute MSE is never compared across policy families.
Mean improvement over Random (5 seeds, keep 30%, 3 datasets; per-method mean rank in parentheses):
| Method (keep 30%) | BC-MLP | ACT | Diffusion |
|---|---|---|---|
| Diversity-only | +29.5% (2.0) | +26.5% (2.0) | +11.9% (2.7) |
| Calibra full | +24.5% (2.7) | +23.7% (3.0) | +13.8% (1.3) |
| K-Center greedy | +24.0% (2.0) | +23.1% (2.7) | +10.1% (4.0) |
| Facility Location | +21.5% (4.0) | +18.4% (3.7) | +8.7% (4.3) |
| Quality-filter only | +16.2% (4.3) | +17.4% (4.7) | +11.1% (3.3) |
| Random | 0.0% (6.3) | 0.0% (5.7) | 0.0% (5.7) |
| Herding | −11.4% (6.7) | −7.5% (6.3) | −5.5% (6.7) |
Rank agreement of the method ordering (Spearman ρ): BC↔ACT 1.00, BC↔Diffusion 0.86, ACT↔Diffusion 0.86 (Kendall τ = 1.00 / 0.71 / 0.71). Changing the learner from an MLP to a transformer (BC→ACT) did not change the method ordering at all; changing to a generative policy (→Diffusion) altered some details but preserved most of the ranking.
What holds across all three, and the one thing that shifts:
- The core ranking is architecture-robust (ρ ≥ 0.86). Coverage-based selection (Diversity-only / Calibra full) consistently outperformed Random across all three evaluated policy architectures; K-Center and Facility Location are mid-pack; and Herding was worst under every learner. Calibra full was never significantly worse than the best published coreset baseline (K-Center) under any learner.
- Diversity-only (coverage) selection is the best-or-tied-best method under all three learners. In the single-sample diffusion run above, Calibra full's quality filter appeared to lift it to the top rank — but two robustness checks (below) overturn that. Under both equal compute and multi-sample-averaged evaluation, Diversity-only is again the top method under diffusion and Calibra full significantly loses to it on DROID and PushT (p<0.001) — the same quality-filter drag seen on BC-MLP and ACT. So the honest conclusion is that the quality filter is a drag on clean data under all three architectures, and the apparent diffusion exception did not survive robustness checks.
- Practical takeaway: diversity/coverage selection is the safe default across all three architectures; quality filtering pays off only under detected corruption (see the failure-prediction and
corruptbenchmarks), not on clean teleop data.
Robustness checks on the diffusion learner (5 key conditions, 5 seeds). Because diffusion is a stochastic sampler evaluated by single-sample first-action MSE, we ran two controls — reported alongside the single-sample numbers, not replacing them:
| Method | Single-sample (default) | Equal compute (5000 steps) | Multi-sample (10× avg) |
|---|---|---|---|
| Diversity-only | +11.9% (rank 2.7) | +23.4% (1.67) | +22.9% (1.67) |
| Calibra full | +13.8% (1.33) | +19.3% (2.33) | +21.2% (2.00) |
| K-Center greedy | +10.1% (4.0) | +22.0% (2.00) | +20.2% (2.33) |
- Equal compute (
--max-steps 5000, identical optimizer steps for every condition): Full-dataset's default-protocol advantage was largely a compute artifact — its vs-random gap collapses from +59/+14/+36% to +2.6/−6.4/+3.6% (ALOHA/DROID/PushT) once steps are equalized, while the coreset selection ranking is unchanged. The coreset advantage is not a compute artifact. - Multi-sample (
--eval-samples 10, averaging sampler noise): the method ranking is stable and matches the equal-compute ranking — confirming the single-sample diffusion result's ranking was not merely sampler noise, even though the specific "Calibra full first" ordering was.
To reproduce:
# ACT
python experiments/act_ablation_benchmark.py --dataset lerobot/aloha_mobile_cabinet --seeds 5 --json results/act_ablation_aloha.json
python experiments/act_ablation_benchmark.py --dataset lerobot/droid_100 --seeds 5 --json results/act_ablation_droid.json
python experiments/act_ablation_benchmark.py --dataset lerobot/columbia_cairlab_pusht_real --seeds 5 --json results/act_ablation_pusht_real.json
python experiments/aggregate_ablation.py results/act_ablation_*.json
# Diffusion Policy
python experiments/diffusion_ablation_benchmark.py --dataset lerobot/aloha_mobile_cabinet --seeds 5 --json results/diffusion_ablation_aloha.json
python experiments/diffusion_ablation_benchmark.py --dataset lerobot/droid_100 --seeds 5 --json results/diffusion_ablation_droid.json
python experiments/diffusion_ablation_benchmark.py --dataset lerobot/columbia_cairlab_pusht_real --seeds 5 --json results/diffusion_ablation_pusht_real.json
python experiments/aggregate_ablation.py results/diffusion_ablation_*.json
# Diffusion robustness checks (per dataset; --conditions restricts to the 5 key methods)
# equal compute: add --max-steps 5000
# multi-sample: add --eval-samples 10
python experiments/diffusion_ablation_benchmark.py --dataset lerobot/droid_100 --seeds 5 \
--max-steps 5000 --conditions "Full dataset,K-Center greedy,Diversity-only,Calibra full" \
--json results/diffusion_equalstep_droid.json
# NOTE: aggregate_ablation.py takes explicit paths — shell globs like results/*.json are not
# expanded by PowerShell, so pass filenames (or run from bash).Calibra's diagnostic metrics predict which selection regime applies before any training:
from calibra.pipeline import Pipeline
from calibra.strategy import diagnose_regime
report = Pipeline().run(batch)
diagnosis = diagnose_regime(report)
print(diagnosis.regime) # SelectionRegime.LOW_NOISE / MODERATE_NOISE / HIGH_NOISE
print(diagnosis.explanation) # human-readable mechanism description
selector = CoresetSelector(keep_fraction=0.3, **diagnosis.recommended_config)Regime classification across tested datasets:
| Dataset | vel_disc/spike ratio | Diagnosed regime | Calibra full (before fix) | Calibra full (after fix) |
|---|---|---|---|---|
| ALOHA mobile | 1.9 | LOW NOISE | +22.6% | +22.6% (unchanged) |
| DROID-100 | 1.6 | MODERATE NOISE | +16.9% | +16.9% (unchanged) |
| PushT real | 24.4 | HIGH NOISE (contact) | −30.5% | +39.5% |
The vel_disc/spike ratio correctly identifies PushT real as contact-driven and triggers the 3× threshold relaxation. See experiments/regime_space.py for the visualization.
Note: These results represent a hypothesis supported by 3 datasets, not an established law. The pattern is consistent and reproducible but requires validation across more policy families and embodiments before being treated as a general principle.
Calibra's advantage over random selection is stable across the full data-fraction range (10%–70%) on clean and heterogeneous datasets. On contact-rich tasks, diversity-only selection (without quality filtering) should be used instead of the full pipeline.
ALOHA mobile — Calibra full vs. random at each keep fraction:
| Keep | k | Calibra vs. Random | Calibra vs. Full |
|---|---|---|---|
| 10% | 7 | +50.0% | −101% |
| 20% | 14 | +35.5% | −65% |
| 30% | 20 | +29.2% | −46% |
| 50% | 34 | +7.8% | −29% |
| 70% | 48 | +7.2% | −17% |
Advantage over random is largest at small budgets (+50% at 10%) and narrows as budget grows — on clean structured datasets, more data continues to help, so the coreset closes but does not eliminate the full-data gap.
DROID-100 — Calibra full vs. random:
| Keep | Calibra vs. Random | Calibra vs. Full |
|---|---|---|
| 10% | +20.9% | +16.8% |
| 20% | +15.2% | +10.5% |
| 30% | +10.1% | +11.8% |
| 50% | +7.2% | +11.3% |
| 70% | +4.5% | +7.3% |
On DROID, Calibra with 10% of episodes outperforms the full dataset by 16.8%.
PushT real (contact-aware) — Calibra full vs. random:
| Keep | k | Calibra vs. Random | Calibra vs. Full |
|---|---|---|---|
| 10% | 11 | +56.6% | +41.7% |
| 20% | 22 | +49.1% | +38.4% |
| 30% | 33 | +42.5% | +31.5% |
| 50% | 54 | +32.2% | +17.6% |
| 70% | 76 | +15.0% | +10.4% |
On contact-rich tasks like PushT real, a Calibra coreset pruned to only 10% of the dataset size outperforms training on the full unpruned dataset by 41.7% while saving 90% of GPU compute.
To reproduce all ablations:
pip install "calibra-robotics[lerobot]" torch
python experiments/ablation_benchmark.py --dataset lerobot/aloha_mobile_cabinet --n-epochs 300 --seeds 5 --curve --save-fig
python experiments/ablation_benchmark.py --dataset lerobot/droid_100 --n-epochs 300 --seeds 5 --curve --save-fig
python experiments/ablation_benchmark.py --dataset lerobot/columbia_cairlab_pusht_real --n-epochs 200 --seeds 5 --curve --save-fig
python experiments/regime_space.py --savePyPI package name:
calibra-robotics(thecalibraname on PyPI is an unrelated package)
# Core (numpy + pydantic only — no format adapters)
pip install calibra-robotics
# With LeRobot / HuggingFace Hub support (recommended)
pip install 'calibra-robotics[lerobot]' # Parquet, DuckDB, Hub IDs
# Other format adapters
pip install 'calibra-robotics[hdf5]' # HDF5 (Isaac Lab, Robomimic)
pip install 'calibra-robotics[rlds]' # RLDS / TensorFlow Datasets
pip install 'calibra-robotics[mcap]' # MCAP / ROS2 bags
# Everything
pip install 'calibra-robotics[all]'from calibra.metrics import (
compute_velocity_discontinuity_rate,
compute_jerk_spike_rate,
compute_ldlj,
compute_action_entropy,
compute_jitter_cv,
)
# actions / states: np.ndarray of shape (T, D)
disc = compute_velocity_discontinuity_rate(actions, states, dt=0.02)
jerk = compute_jerk_spike_rate(states, dt=0.02, sigma_limit=5.0)
ldlj = compute_ldlj(positions, dt=0.02)
entopy = compute_action_entropy(actions)
cv = compute_jitter_cv(timestamps)from calibra.core import LazyDatasetReader
with LazyDatasetReader("/data/lerobot/aloha_mobile") as reader:
print(reader.fps, reader.episode_count())
# Query only the columns you need — images never leave the Parquet pages
table = reader.query_proprioception_tensors(
["observation.state", "action"], episode_idx=0
)
actions = table["action"].to_pylist()from calibra.ingestion.registry import load
from calibra.pipeline import Pipeline
batch = load("lerobot/pusht") # Hub ID, local path, or hf:// URI
report = Pipeline().run(batch, policy_family="diffusion")
print(report.summary())from calibra.pruning import CoresetSelector
selector = CoresetSelector(
keep_fraction=0.3,
max_spike_rate=0.05,
max_vel_disc_rate=0.15,
)
result = selector.select(batch, report)
print(result.summary())
# result.keep_episode_ids → filter your Parquet shardsfrom calibra.core import SchemaNormalizer
# YAML config for your robot's naming convention
n = SchemaNormalizer(config_path="my_robot/mappings.yaml")
normalized = n.normalize({"my_robot/joints/q": arr, "my_robot/ee": arr2})Dataset (Parquet / HDF5 / RLDS / MCAP / Hub ID / hf:// URI)
│
▼ Format adapters — metadata-first, DuckDB lazy scan skips images
EpisodeBatch — normalised internal representation
│
▼ Four analyzers (parallelisable)
DiagnosticReport — flags + 95% bootstrap CIs + per-episode arrays
│
├──▶ audit — terminal summary + MAD outlier table
├──▶ compare — evidence-backed cross-dataset comparison
├──▶ certify — CERTIFIED / PROVISIONALLY CERTIFIED / NOT CERTIFIED
└──▶ prune — quality filter + greedy max-coverage coreset
| Analyzer | Metrics computed |
|---|---|
TemporalAnalyzer |
timestamp jitter CV, dropout rate, camera lag std, action-obs misalignment |
ControlSmoothnessAnalyzer |
LDLJ, jerk spike rate, velocity discontinuity rate, action-state divergence |
CoverageEntropyAnalyzer |
action entropy (bits/dim), state entropy, PCA top-2 variance, episode length distribution |
TaskStructureAnalyzer |
contact density, grasp events per episode, trajectory diversity score, short episode fraction |
All metrics report 95% bootstrap confidence intervals computed over episodes (not steps), avoiding artificially narrow intervals from correlated within-episode samples.
Every interpretation in calibra compare output is backed by a falsifiable claim in calibra/claims/. Each claim tracks:
- assertion — what the metric is expected to show for a given dataset class
- evidence — which datasets have been profiled and whether they support the claim
- confidence — derived from evidence count: NOT VALIDATED → LOW → MEDIUM → HIGH → STRONG
- falsification condition — exactly what data would invalidate the claim
- pending tests — the highest-value next dataset to profile
See docs/claims.md for the full registry.
Ratio rule: the number of reference profiles must be ≥ the number of active claims.
Enforce in CI with: python scripts/generate_claims_doc.py --check
Three empirical baselines are shipped:
| Reference | Control | Freq | DOF | Episodes | Hardware |
|---|---|---|---|---|---|
pusht |
velocity | 10 Hz | 2 | 206 | sim |
aloha_sim |
position | 50 Hz | 14 | 50 | sim |
aloha_mobile_cabinet |
position | 50 Hz | 14 | 85 | ✓ real |
aloha_mobile_shrimp |
position | 50 Hz | 14 | 100 | ✓ real |
aloha_sim_insertion_scripted |
position | 50 Hz | 14 | 50 | sim |
aloha_sim_transfer_cube_scripted |
position | 50 Hz | 14 | 50 | sim |
aloha_sim_transfer_cube_human |
position | 50 Hz | 14 | 50 | sim |
aloha_static_battery |
position | 50 Hz | 14 | — | ✓ real |
aloha_static_candy |
position | 50 Hz | 14 | — | ✓ real |
aloha_static_coffee |
position | 50 Hz | 14 | — | ✓ real |
aloha_static_cups_open |
position | 50 Hz | 14 | — | ✓ real |
pusht_image |
velocity | 10 Hz | 2 | — | sim |
droid_100 |
position | 15 Hz | 7 | 100 | ✓ real |
svla_so100_pickplace |
position | 15 Hz | 6 | 50 | ✓ real |
svla_so100_stacking |
position | 15 Hz | 6 | 56 | ✓ real |
bridgedata_v2 |
velocity | 5 Hz | 7 | 50415 | ✓ real |
You can profile additional datasets locally using scripts/profile_dataset.py.
| Format | Extra install | Notes |
|---|---|---|
| LeRobot v2 (Parquet shards) | calibra[lerobot] |
DuckDB lazy scan — image columns never enter RAM |
| LeRobot v1 (HF datasets) | calibra[lerobot] |
HuggingFace datasets + pandas groupby |
| HuggingFace Hub | calibra[lerobot] |
lerobot/pusht, hf://lerobot/pusht |
| HDF5 (Isaac Lab, Robomimic) | calibra[hdf5] |
Convention A + B |
| RLDS / TF Datasets | calibra[rlds] |
tensorflow-datasets |
| MCAP / ROS2 bags | calibra[mcap] |
mcap + mcap-ros2-support |
Calibra is not open to external pull requests (PRs) or contributions at this time.
calibra/
├── core/ # Public API: LazyDatasetReader, SchemaNormalizer, mappings.yaml
├── metrics/ # Standalone pure-numpy functions (no pipeline needed)
├── analyzers/ # Pipeline analyzers: temporal, smoothness, coverage, task_structure
├── ingestion/ # Format adapters (lerobot, hdf5, rlds, mcap) + registry
├── comparison/ # DatasetComparator, EpisodeCurator
├── world_model/ # Lightweight (non-torch) world-model surprise scoring + curation
├── models/ # RobotJEPA — trained world model (optional, requires torch)
├── schema/ # EpisodeBatch, DiagnosticReport, normalization layer
├── claims/ # Falsifiable claim registry (JSON + SPEC.md)
├── knowledge_base/ # claims.yaml (auto-generated — edit the source JSON files)
├── references/ # Profiled reference datasets (JSON)
└── interpretations/ # Metric interpretation docs (Markdown)
scripts/
├── profile_dataset.py # Profile any dataset → references/<name>.json
└── generate_claims_doc.py # Regenerate docs/claims.md + CI ratio check
docs/
└── claims.md # Auto-generated from calibra/claims/ — do not edit
git clone https://github.com/omerTT/Calibra
pip install -e '.[all,dev]'
pytest # 596 tests
ruff check . # zero errors expectedCalibra supports two data curation philosophies. The right one depends on your policy architecture. Both use the same tool — the same quality metrics, the same CLI — but apply different selection criteria in Stage 2 of calibra prune.
# For diffusion policy, ACT, GR00T fine-tuning
calibra prune /data/demos --keep 0.3 --strategy diversity
calibra watch /data/session/ --remediateSelection goal: remove corrupted episodes, keep behaviorally diverse ones.
Metrics that matter: jerk spike rate, dropout, LDLJ, velocity discontinuity.
Stage 2 runs greedy max-coverage over the quality-passing pool — maximising behavioural spread across the action-space. A policy trained on a diverse coreset generalises better than one trained on near-duplicate demonstrations.
# For JEPA-based world models (I-JEPA, V-JEPA successors, latent MPC)
calibra prune /data/demos --keep 0.3 --strategy world-model
calibra watch /data/session/ --world-modelSelection goal: select episodes that maximise what the world model doesn't yet know — highest latent prediction error relative to the current model state.
Works out of the box — no extra install required.
Stage 2 scores each quality-passing episode by latent prediction error and keeps the highest-surprise fraction. Two backends compute that score, chosen automatically:
- Lightweight baseline (default, core install): a closed-form encoder (PCA / random projection) and a closed-form linear next-latent predictor — no gradient descent, no GPU, fit in milliseconds. See
calibra/world_model/surprise.py. - Trained JEPA (advanced,
pip install torch): a small MLP encoder/predictor trained with a VICReg objective (calibra/models/robot_jepa.py) for higher-fidelity surprise scores on larger datasets.
calibra prune --strategy world-model and calibra watch --world-model use the trained JEPA when torch is installed and fall back to the lightweight baseline otherwise — same CLI flags, same output shape, no code changes required either way. The result is a dataset that pushes the world model toward unexplored dynamics rather than reinforcing what it already knows.
WORLD-MODEL CURATION SUMMARY
Original episodes: 1000
Quality failures: 87
High-surprise kept: 300
Low-surprise pruned: 613
Top novel episodes:
ep_104 surprise=0.82 reason="unusual contact dynamics"
ep_511 surprise=0.77 reason="rare state-space excursion"
ep_208 surprise=0.74 reason="long-horizon recovery"
| World-model surprise | Kinematic quality | Interpretation | Action |
|---|---|---|---|
| HIGH | FAIL (high jerk) | Corrupted episode | Prune |
| HIGH | PASS (smooth) | Genuinely novel dynamics | Keep |
| LOW | any | Redundant / well-covered | Prune |
The table is the key insight: surprise alone is not sufficient. A noisy actuator or a packet-drop produces high prediction error but teaches the world model nothing useful. Stage 1 quality filtering — identical in both workflows — is what separates genuinely novel episodes from corrupted ones before surprise scoring ever runs.
Clean data is a prerequisite for both paradigms. A JEPA trained on jittery, dropout-heavy data learns corrupted latent representations that degrade downstream planning and latent MPC rollouts. The quality filtering stage (Stage 1 of calibra prune) is therefore identical for both — only Stage 2 (greedy diversity selection vs. surprise-maximisation) changes.
calibra sim2real reports a world-model transfer gap metric alongside the standard distribution-gap analysis, quantifying how much of the latent dynamics your sim fails to cover before you commit to training.
- Using diffusion policy, ACT, or fine-tuning a GR00T/Octo checkpoint → use
--strategy diversity - Training a JEPA world model from scratch, or maximising the information content of a small dataset → use
--strategy world-model - Collecting new data in real-time →
calibra watch --world-modeltells you which episodes are genuinely novel vs. redundant as you collect, so operators can prioritise effort on configurations the model hasn't seen
- Not a dataset score ("your dataset is 7.4/10") — Calibra surfaces specific, falsifiable anomalies
- Not an AI assistant — it runs deterministic mathematical estimators, not a language model
- Not a cloud service — it runs entirely locally against your files
- Not a replacement for domain expertise — it tells you what to look at; you decide what to do
Business Source License 1.1 — free for internal use, open-source under Apache 2.0 on 2030-06-30. Commercial hosting requires a license: omertahtoko@gmail.com