Feature/prediction - #2
Merged
Merged
Conversation
- New configs/evaluation.json with all gate thresholds - EvaluateDetectorFlow now uses eval_config instead of Parameters - Increased max_rate_diff from 10% to 25% (crypto volatility) - Remove hardcoded defaults from src/eval.py in favor of config 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Agent that: - Critiques the business value of current ML task - Identifies gaps in evaluation methodology - Proposes evolution paths toward practical value - Provides implementation guidance for improvements Uses opus model for deeper analysis and web search for industry context. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Raw timestamps in title="" attributes are intentional for hover tooltips. The check now uses inner_text() to only validate visible text, not HTML attributes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Data Engineering: - src/features.py: Temporal feature engineering (lag, rolling stats) - src/predictions.py: Prediction logging for outcome validation - BuildDatasetFlow: No longer deduplicates! Keeps all temporal data - Adds lag features (1h, 6h, 24h) - Adds rolling statistics (mean, std over 6h, 24h) - Time-based train/holdout split Prediction Logging: - TrainFlow now logs all predictions with timestamps - Stored in predictions/dt=YYYY-MM-DD/hour=HH/ - Enables future outcome validation (did flagged coins actually crash?) This transforms the model from "static anomaly detector" to "crash early warning system" that can be validated against actual price movements. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
New validation checks: - Flow versions shown (Train/185989, Promote/185991 format) - No Internal Server Error on page - Model version count >= 1 - Champion badge via locator (not just text search) These checks ensure models are actually registered and displaying correctly, not just that the page renders. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Closes the feedback loop by checking if flagged anomalies actually crashed: - Loads predictions from 24h ago (logged by TrainFlow) - Fetches current prices and computes actual price changes - Computes crash_recall, anomaly_precision, false_alarm_rate - Updates lifecycle-runner.md with new data pipeline diagram 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixes CI deployment failure - the deploy script needs eval_config in flow_configs to pass the config path correctly when running from flow directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace Asset API tags with Metaflow's native run tagging for champion
tracking. This enables server-side queries via Flow.runs("champion").
Changes:
- Remove ModelStatus enum (candidate/evaluated/champion states)
- Add get_champion_run_id() and set_champion_run() functions
- Simplify register_model() to not require status parameter
- Update register_evaluation() to link results to model versions
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- TrainDetectorFlow: add data lineage tracking (snapshot range, count) - EvaluateDetectorFlow: flexible model version selection via parameters - PromoteModelFlow: use Metaflow run tags instead of Asset status 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Models page: show champion status, version comparison - Data page: improved snapshot visualization - Overview: updated metrics display 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Streamline documentation with clearer quick start, project structure, and champion management explanation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add missing alias property that evaluate flow references. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
EvaluateDetectorFlow now loads eval_holdout asset by default instead of fetching fresh data. This provides: - Reproducible evaluations (same holdout set across runs) - Proper train/eval separation - Full lineage from eval results back to BuildDatasetFlow Config vs Parameter override pattern: - eval_config sets defaults at deployment time - Parameters (eval_data_source, eval_data_version) can override at runtime Usage: # Default: use holdout set python flow.py run # Override to fresh data at runtime python flow.py run --eval_data_source fresh 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Updated image tag for better responsiveness in README.
Document how to fill trigger forms for: - BuildDatasetFlow (holdout_hours, max_history_hours, etc.) - EvaluateDetectorFlow (candidate_version, compare_to, eval_data_source) - PromoteModelFlow (version, alias, model_name) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The header was checking evaluation.decision but get_latest_evaluation() returns evaluation.passed. Changed to use correct field. Also add trigger form parameter tables to README. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Metaflow runs are namespaced by project_branch tag, not global.
The dashboard was querying Flow('TrainDetectorFlow').runs() without
filtering, showing models from all branches.
Changes:
- get_model_versions() now filters by project_branch:{branch} tag
- get_champion_run_id() now accepts branch param for scoped lookup
- model_registry() passes branch to both functions
- overview.html: fix evaluation.decision -> evaluation.passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The dashboard was only seeing runs in the default user namespace.
Argo runs exist in a different namespace but share the same project.
Fix: Use namespace(f"project:{PROJECT}") to see all runs (dev + argo)
for the project, then filter by project_branch tag for branch-specific
queries.
Key insight from Metaflow docs:
- namespace('user:X') = only local dev runs by that user
- namespace('project:X') = all runs for project (dev + argo + prod)
- Filter by project_branch:{branch} tag for branch-specific runs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Cards page:
- Fix branch scoping using namespace(f"project:{PROJECT}")
- Filter runs by project_branch tag to show only relevant cards
- Improve dropdown display with run IDs
Models page:
- Add loading spinner for Compare button
- Fix lineage comparison to show Dataset Version (not timestamp)
- Add hover tooltip for full IDs
- Improve Outcome Validation explanation (3-step process)
Compare API:
- Add project namespace for cross-runtime run access
Base styles:
- Add .spinner and .spinner-lg CSS animations
- Add .loading-overlay for async operations
- Add .btn.loading state with inline spinner
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The promote flow failed when trying to promote Argo runs because: 1. Argo run IDs (like argo-foo-bar-xyz) weren't recognized as run IDs 2. Metaflow Client API needed project namespace to find Argo runs Changes: - flows/promote/flow.py: Detect argo- prefix as run ID format - flows/promote/flow.py: Use project namespace when loading runs - src/registry.py: set_champion_run() now accepts project param and uses project namespace to find runs across all runtimes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Allow comparing multiple runs of the same flow (e.g., latest vs latest-1 training runs) in addition to cross-flow comparison. This helps track how score distributions, anomaly rates, and detected coins change over time. - Collect up to 5 recent runs per flow for intra-flow comparison - Add comparison mode toggle (cross-flow vs intra-flow) - Add flow selector and version dropdowns for intra-flow mode - Pass flow_runs data to template for JavaScript-driven UI 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The Overview page was using Asset API (latest-1) to find the champion, which doesn't reflect the actual promoted champion. Now uses the same get_champion_run_id() function as the Models page to correctly find runs tagged with 'champion'. - get_pipeline_status() now uses get_champion_run_id(branch) - overview() now fetches champion details from the tagged training run - Template handles missing silhouette_score gracefully (only in eval flow) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Updated image source in README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.