Skip to content

Feature/global ci fn#344

Draft
danbraunai-goodfire wants to merge 1 commit intodevfrom
feature/global-ci-fn
Draft

Feature/global ci fn#344
danbraunai-goodfire wants to merge 1 commit intodevfrom
feature/global-ci-fn

Conversation

@danbraunai-goodfire
Copy link
Copy Markdown
Collaborator

@danbraunai-goodfire danbraunai-goodfire commented Jan 20, 2026

Copied over from #336 with the new dev base branch.

Description

Add global CI function mode as an alternative to the existing layerwise CI functions.

Key changes:

  • Add discriminated union config pattern (LayerwiseCiConfig | GlobalCiConfig) for CI function configuration
  • Implement GlobalSharedMLPCiFn that concatenates activations across all layers, processes through a shared MLP, and splits outputs back to per-layer CI values
  • Add unified wrapper classes (LayerwiseCiFnWrapper, GlobalCiFnWrapper) that provide a consistent interface regardless of CI mode
  • Update all experiment configs to new ci_config format
  • Add comprehensive tests for both CI modes and checkpoint compatibility validation
  • Update canonical runs in registry with new format

Related Issue

N/A

Motivation and Context

The existing layerwise CI functions learn separate importance functions per layer, which limits parameter sharing and may miss cross-layer patterns. Global CI provides an alternative where a single network sees all layer activations simultaneously, enabling:

  • Better parameter efficiency for models with many layers
  • Potential to learn cross-layer relationships in component importance
  • Simpler architecture for experiments where layer-specific CI isn't needed

How Has This Been Tested?

  • All tests pass (380 passed locally, CI green)
  • New tests added for:
    • Global CI function creation and forward pass
    • Checkpoint compatibility validation between CI modes
    • State dict save/load for both wrapper types
    • Integration with full ComponentModel pipeline
  • New canonical runs created with updated config format

Does this PR introduce a breaking change?

Yes. Checkpoints from before this PR are incompatible due to:

  • Config format change from flat ci_fn_type/ci_fn_hidden_dims to nested ci_config discriminated union
  • State dict key prefix change (now under ci_fn._ci_fns or ci_fn._global_ci_fn)

New canonical runs have been created to replace the old ones.

New Canonical Runs

Experiment Run ID WandB Link
tms_5-2 s-38e1a3e2 https://wandb.ai/goodfire/spd/runs/s-38e1a3e2
tms_5-2-id s-a1c0e9e2 https://wandb.ai/goodfire/spd/runs/s-a1c0e9e2
tms_40-10 s-7387fc20 https://wandb.ai/goodfire/spd/runs/s-7387fc20
tms_40-10-id s-2a2b5a57 https://wandb.ai/goodfire/spd/runs/s-2a2b5a57
resid_mlp1 s-62fce8c4 https://wandb.ai/goodfire/spd/runs/s-62fce8c4
resid_mlp2 s-a9ad193d https://wandb.ai/goodfire/spd/runs/s-a9ad193d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants