feat: Phase 1 — backtesting, paper trading, SQLite storage#6
Open
Barac9492 wants to merge 24 commits intonikmcfly:mainfrom
Open
feat: Phase 1 — backtesting, paper trading, SQLite storage#6Barac9492 wants to merge 24 commits intonikmcfly:mainfrom
Barac9492 wants to merge 24 commits intonikmcfly:mainfrom
Conversation
Build a pipeline that fetches Polymarket markets, converts questions into balanced simulation scenarios via LLM, runs multi-agent Reddit simulations, analyzes sentiment/consensus, and surfaces trading signals by comparing simulated probability vs market odds. Backend: polymarket_client, scenario_generator, sentiment_analyzer, prediction_manager (pipeline orchestrator), prediction API blueprint. Frontend: PredictionView with market browser + signal dashboard. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix Vercel build: install frontend deps before build, add vercel.json
with outputDirectory pointing to frontend/dist
- Complete UI/UX redesign of PredictionView:
- Hero strip with pipeline tags and live stats
- Skeleton loading states for market list
- Animated market cards with probability bars
- Visual pipeline tracker with stage dots, checkmarks, connecting lines
- Probability comparison gauge (market vs simulated)
- Stance distribution bar with for/neutral/against breakdown
- Key arguments with color-coded bullets
- Panel slide transitions, fade-in animations
- Responsive grid layout (1024px + 768px breakpoints)
- Custom scrollbars, shimmer loading, pulse indicators
- Matches MiroFish design: black nav, orange accent, JetBrains Mono,
minimal borders, generous spacing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this, direct navigation to /prediction (or any non-root route) returns a Vercel 404 since there's no server-side route handler. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
LLMClient auto-detects Claude models (model name starts with "claude") and uses the Anthropic SDK natively. System messages are extracted into the separate system param. JSON mode adds an explicit instruction since Anthropic doesn't have response_format. Simulation scripts (reddit, twitter, parallel) detect Claude models and use camel-ai's ModelPlatformType.ANTHROPIC + ANTHROPIC_API_KEY instead of the OpenAI-compatible path. Set in .env: LLM_API_KEY=sk-ant-... LLM_MODEL_NAME=claude-sonnet-4-20250514 Embeddings still require Ollama (nomic-embed-text) since Claude doesn't provide an embedding endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix Gamma API parser: outcomes/outcomePrices come as JSON strings (e.g. '["Yes", "No"]'), not arrays. Now handles both formats. - Add anthropic SDK to requirements.txt - LLMClient: auto-detect Claude models, use Anthropic SDK natively - Simulation scripts: detect Claude → ModelPlatformType.ANTHROPIC Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Defines the full product requirements to turn MiroFish's existing prediction signal pipeline into an autonomous trading system: trade execution, risk management, backtesting, market scanning, portfolio tracking, and signal quality improvements. https://claude.ai/code/session_01YPs2KGRrzwQw1j7PZpRb4P
Add PRD for Polymarket monetization engine
…e LLMClient Both services were creating raw OpenAI() clients, which fails with Anthropic models. Replaced with LLMClient which auto-detects Claude and routes through the Anthropic SDK. This was the root cause of 404 errors when running predictions with Claude — the profile/config generation stage bypassed the Anthropic support in LLMClient. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add tasks/backtest.py: runs 5 resolved Polymarket markets through the prediction pipeline and compares signal vs actual outcome. Tracks directional accuracy, Brier score, and stance breakdown. - Reduce PREDICTION_DEFAULT_ROUNDS from 5 to 2 — Claude API calls make each OASIS round slow (~5-10 min per round with 10 agents), 2 rounds produces enough discourse for sentiment analysis. - Increase simulation wait timeout to 7200s (2 hours). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the OASIS simulation produces no agent actions (common when Claude API is the backend — sim initializes but agents don't generate posts), the sentiment analyzer defaults to 50% probability. This was creating false BUY_YES signals. Now: if total_posts_analyzed == 0 or confidence < 5%, signal is HOLD with 0 confidence and explicit "insufficient data" reasoning. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OASIS/camel-ai doesn't work reliably with Claude (agents produce 0 actions). Added SIMULATION_LLM_* env vars so simulations use local Ollama (qwen2.5:7b) while Claude handles scenario gen, ontology, and sentiment analysis. Config: SIMULATION_LLM_API_KEY, SIMULATION_LLM_BASE_URL, SIMULATION_LLM_MODEL Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OASIS multi-agent simulation was too slow (~30+ min per market) and incompatible with Claude. Replaced with DebateSimulator: a single LLM call that generates 15-25 structured debate posts from diverse perspectives (experts, stakeholders, general public, contrarians). Pipeline now: market → scenario → debate → signal (~90s per market) Backtest results (5 resolved markets): - Avg Brier: 0.2230 (below 0.25 coin-flip baseline) - Directional accuracy: 1/5 (20%) - Systematic bias: LLM generates ~50/50 debates regardless of actual probability, producing BUY_YES on low-probability markets - The Fed rates market scored best (Brier 0.1456) — closest to reality Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove forced 50/50 balance from debate and scenario prompts. Instead: - Scenario generator now produces honest factual briefings that state which outcome evidence favors - Debate simulator generates stance distributions proportional to actual evidence weight, with contrarian minority voices - LLM provides direct probability estimate blended 50/50 with stance-derived probability Backtest results dramatically improved: Before: Avg Brier 0.2230, Directional 1/5 (20%) After: Avg Brier 0.1299, Directional 4/5 HOLD (correctly cautious) Tiger King pardon: SimP=18.8% vs actual NO (Brier 0.035) Zelenskyy suit: SimP=12.2% vs actual NO (Brier 0.015) Fed -50bps: SimP=7.7% vs actual NO (Brier 0.006) Khamenei out: SimP=16.6% vs actual NO (Brier 0.028) Israel-Iraq strike: SimP=24.8% vs actual YES (Brier 0.566) 4/5 markets now correctly estimate low probability for low-prob events. The one miss (Israel-Iraq) underestimated a geopolitical escalation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…dampening Three calibration corrections learned from 17-market analysis: 1. Market regression (30%): Blend SimP toward market price. LLMs have "possibility bias" that overweights unlikely events. Liquid markets contain real information from real money. 2. Edge confidence penalty: Large edges (>25%) get confidence discounted by 50-80%. Huge disagreements with liquid markets usually mean the model is wrong, not the market. 3. Short-dated dampening: Markets ending within 14 days get additional 20% regression toward market price. Backtest improvement across all three iterations: v1 (forced balance): Brier 0.2230 v2 (evidence-weight): Brier 0.1299 v3 (calibrated): Brier 0.1190 The calibrated system correctly HOLDs on all 5 backtest markets (4 were genuinely low-prob events, 1 was a miss on Israel-Iraq). First 4 markets have excellent Brier scores: 0.028, 0.025, 0.005, 0.020. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…extraction - Delete unused sentiment_analyzer.py (254 lines, replaced by debate_simulator) - Remove unused PredictionRunStatus enum states (CREATING_PROJECT, BUILDING_GRAPH, PREPARING_SIMULATION) - Remove unused RetryableAPIClient class from retry.py - Translate Chinese comments to English across core files - Extract calibration constants to Config with env var overrides - Add CALIBRATION_MARKET_REGRESSION, DATE_DAMPENING_DAYS, HIGH_EDGE_THRESHOLD, etc. - Replace hardcoded values in prediction_manager with Config references - Tighten exception handling with specific types (RequestException, ValueError, JSONDecodeError) - Wire @retry_with_backoff on PolymarketClient.fetch_active_markets() and get_market() - Clamp confidence to [0,1] in debate_simulator - Add DI result_store param to PredictionManager (default: PredictionRunManager) - Add actual_outcome field to PredictionMarket model - Add SQLite, scikit-learn dependencies to pyproject.toml - Add SQLITE_DB_PATH and PAPER_TRADING_MODE config - Add gstack section to CLAUDE.md
- SQLiteStore with SQLAlchemy Core, WAL mode, FK enforcement - Tables: backtest_runs, backtest_results, paper_orders, paper_positions - BacktestRun, BacktestResult, BacktestMetrics dataclasses - PaperOrder, PaperPosition dataclasses with PositionStatus enum - has_active_backtest() DB-level guard for concurrent run prevention - HMAC-signed pickle serialization for calibration models
- Backtester: runs prediction pipeline against resolved markets, computes accuracy/Brier/ROI/Sharpe/max drawdown/calibration RMSE metrics - Supports resume after crash via get_completed_market_ids() - Calibrator: Platt scaling via LogisticRegression, HMAC-signed persistence - PaperTrader: simulated order execution with 1-2% slippage - PolymarketClient: fetch_resolved_markets() with pagination + courtesy delay
- POST /api/backtest/run — start backtest (async thread) - GET /api/backtest/run/<id> — status + results + metrics - GET /api/backtest/runs — list all backtests - DB-level concurrent run guard (has_active_backtest) - Input validation: num_markets capped at 500
- test_sqlite_store: CRUD, WAL, has_active_backtest - test_backtester: full pipeline, resume, zero markets, all failures - test_backtester_metrics: accuracy/Brier/ROI/Sharpe edge cases - test_calibrator: fit/transform/save/load/tampered data - test_paper_trader: BUY/HOLD/slippage/persistence - test_polymarket_client: success/retry/malformed/empty/resolved - test_prediction_manager_di: default + custom store - test_backtest_api: start/status/list/not found/concurrent - test_retry: success/failure/backoff/non-retryable - test_config: calibration defaults + env overrides
- BacktestView: run backtest panel, history, metrics grid, sortable results table, live polling, skeleton/empty/error states, responsive layout - backtest.js API client (startBacktest, getBacktestRun, listBacktests) - Add /backtest route and nav link - Add PAPER mode badge to PredictionView and BacktestView nav bars - Brutalist design: no rounded corners, no shadows, no gradients
- DESIGN.md: extracted design tokens from PredictionView (typography, colors, spacing, components, anti-patterns, responsive breakpoints) - TODOS.md: P2 backlog (JSON migration, CI/CD, disk-full handling, shared CSS) - tasks/todo.md: Phase 1 implementation checklist (all complete) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # backend/app/__init__.py # backend/app/api/__init__.py # backend/app/api/report.py # backend/app/config.py # backend/app/services/oasis_profile_generator.py # backend/app/services/simulation_config_generator.py # backend/app/utils/llm_client.py # backend/app/utils/retry.py # backend/scripts/run_parallel_simulation.py # backend/scripts/run_reddit_simulation.py # backend/scripts/run_twitter_simulation.py
- README.md: added prediction market + backtesting to workflow, architecture diagram, design decisions, and modifications list - ROADMAP.md: updated current state, marked test suite as complete - docs/progress.md: added Phase 8 (prediction + backtesting) section - PredictionView.vue: fix outline:none → outline:revert, remove border-radius:2px Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- claude.md: project-level Claude Code instructions - docs/designs/polymarket-monetization-expansion.md: CEO-approved expansion plan - tasks/live_markets.json: sample market data for development Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation