Intruder eval rework + autointerp provider rate limits#465
Open
ocg-goodfire wants to merge 6 commits intodevfrom
Open
Intruder eval rework + autointerp provider rate limits#465ocg-goodfire wants to merge 6 commits intodevfrom
ocg-goodfire wants to merge 6 commits intodevfrom
Conversation
- Format scripts/export_blog_data.py and scripts/export_component_data.py - Suppress sklearn import errors in geometric_interaction/statistical_analysis.py (sklearn is not in the project dependencies) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Intruder eval improvements: - Streaming trial generation (lazy iterator, not pre-built list) - Lightweight DensityIndex (stores key+density, not full ComponentData) - XML prompt format with raw + annotated views - JSON response with reasoning field - Save prompts to intruder_prompts DB table - New spd-intruder SLURM CLI Autointerp/graph_interp: - Rate-limit config (max_concurrent, max_requests_per_minute) moved from global config to per-provider settings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ocg-goodfire
force-pushed
the
intruder-rework
branch
from
728d6c7 to
45a28a9
Compare
Resolve conflicts in export scripts: take dev's canonical_to_concrete key iteration (HEAD had .values() bug) and reasoning field additions. Also fix unnecessary pyright ignore comments in statistical_analysis.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Existing DBs migrated in-place. intruder_prompts is already in _SCHEMA for new DBs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ported from notebooks/2026-03-27-10-40_coherence_vs_density.py into a proper CLI script with JSON config for specifying models/groups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip coherence/violin plots, use ember for VPD and sandstone for others. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spd-intruderSLURM CLISplit out from #463 (clustering-core) — independent changes.
Test plan
🤖 Generated with Claude Code