Adds grid stratification, acceptability model, and full layers interop (0.8.0)#9
Merged
Merged
Conversation
….8.0) Adds N-dimensional, mixed-variable grid stratification to bead.lists: a BinningSpec tagged union (quantile, equal-width, threshold, standard-deviation, and categorical binners), assign_grid_cells / assign_grid_cells_by_uuid / grid_shape / flatten_cell / unflatten_cell, a GridStratificationConstraint, and a stratum-aware QuantileBalancer.balance_by_cell with a matching ListPartitioner path. The one-dimensional assign_quantiles is now the quantile specialization of the single engine. Adds AcceptabilityScorer (bead.items.scoring), which scores a forced-choice item by a trained ForcedChoiceModel and returns the predicted preference margin. Extends bead.interop.layers with lenses for every mappable bead model: a FilledTemplate to a resource.Filling, an AnnotationRecord to a judgment.Judgment (grouped per annotator into a judgment.JudgmentSet), a ListConstraint to a judgment.ListConstraint, and an ExperimentList to a resource.Collection of kind stimulus-pool with one membership per item and its list constraints. All round-trip exactly. A Participant is intentionally not mapped: a persona.Persona is an annotator role, not a concrete participant, whose identity is an AgentRef. Rebuilds the gallery/eng/argument_structure example: an acceptability model trained on MegaAcceptability (per-annotator Likert to 2AFC pairing with rater random effects) scores items, the 2AFC pairs are stratified across a grid of the acceptability margin, the language-model score, and the pair type, and the active-learning loop is seeded with the pretrained model. Every artifact (lexicons, templates, filled templates, cross-product items, 2AFC pairs, the training corpus, and experiment lists) is persisted through the layers schema and the bead lairs codec. Bumps the version to 0.8.0. See CHANGELOG for the full entry.
Running the loop surfaced two blockers in run_pipeline.py. First, the model
validates each item against the template's task_spec.options and reads those
option names from rendered_elements, but the protocol template leaves options
unset (FORCED_CHOICE uses per-item options). build_item_template now sets
("option_a", "option_b") on the model-facing template so the 2AFC pairs
validate. Second, the loop trains without participant ids, which a
random-intercepts model rejects; load_acceptability_model now switches the
seeded model to fixed effects (the pretrained encoder carries over) so the loop
fine-tunes at the population level.
Verified end to end with bert-base-uncased on CPU: initial training
(ForcedChoiceModel.train with random intercepts), AcceptabilityScorer scoring
via the population fallback, and a full loop.run iteration with the loaded model.
…bility model Running real training surfaced several issues, fixed here. ForcedChoiceModel reported train_accuracy as 0.0 because it read accuracy from the shared mixed-effects trainer's evaluate(), whose prediction step only labels cloze batches, so for forced choice it returns no usable accuracy or loss. Train and dev metrics are now computed from the model's own predict path (which applies the fitted random effects), so they reflect the deployed model. compute_binary_metrics and compute_multiclass_metrics no longer pass zero_division to evaluate's f1, which this version rejects. The model now supports early stopping on a held-out dev set via a callback that scores the dev set through the predict path at each epoch end, stops after early_stopping_patience epochs without dev cross-entropy improvement, and restores the best encoder and head. A new early_stopping_patience config field controls it. The eng gallery builds the dev set by holding out whole sentences: prepare_megaacceptability.py splits unique sentences (and all their ratings) into train and dev so the dev set contains only sentences never seen during training, and writes both; train_acceptability_model.py loads the dev set and passes it for early stopping. The gallery JSONL loaders now use Item.model_validate_json, since Item(**json.loads(line)) no longer coerces string UUIDs under didactic. Verified end to end on bert-base-uncased: training learns (loss drops, train and dev accuracy above chance), early stopping engages, and the existing forced-choice, categorical, and binary model tests pass.
…ame pairs MegaAcceptability lists are structured so no verb or frame repeats within a list, so the only within-rater controlled contrast is same-frame (different verbs, one frame), which a rater sees once per list across their lists. Capping at 200 pairs per annotator discarded most of these, so max_pairs_per_annotator now defaults to null (no cap). On the full dataset this raises the derived same-frame training set from roughly 145k pairs to roughly 760k (minus ties and the held-out dev split).
…and CUDA Switching the acceptability model to device: mps surfaced a device mismatch: _do_predict_proba adds the per-participant random-effects intercept, a CPU tensor, to the encoder logits on the accelerator, raising "Expected all tensors to be on the same device". The random-effects bias (and the random-slopes head) are now moved to the logits / embedding device, which is a no-op on CPU. The early-stopping callback methods now return None and mutate control in place, matching the TrainerCallback signature. The eng gallery config sets acceptability_model.device to mps, since the uncapped same-frame training set (~760k pairs) is impractical on CPU. Verified: BERT trains and scores on MPS, and the forced-choice model tests still pass on CPU.
layers represents a data producer as an AgentRef (identity), separate from the persona (role/framework) and tool (software), and documents JudgmentSet.features as the home for annotator demographics, session metadata, and payment info. Map a bead Participant onto that: PARTICIPANT_AGENT lenses Participant <-> AgentRef losslessly, participant_features renders the study fields as a FeatureMap, and records_to_judgment_set takes an optional participant so a judgment set carries the annotator's identity and study fields natively.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Releases bead 0.8.0. Three interlocking pieces plus the training fixes surfaced while verifying them:
bead.lists): aBinningSpectagged union (quantile, equal-width, threshold, standard-deviation, categorical),assign_grid_cells/assign_grid_cells_by_uuid/grid_shape/flatten_cell/unflatten_cell, aGridStratificationConstraint, and a stratum-awareQuantileBalancer.balance_by_cellwith a matchingListPartitionerpath. The one-dimensionalassign_quantilesis now the quantile specialization of this single engine.AcceptabilityScorer(bead.items.scoring) scores a forced-choice item by a trainedForcedChoiceModel. Theeng/argument_structuregallery trains that model on MegaAcceptability (per-annotator Likert to 2AFC, with a by-sentence held-out dev set and early stopping), stratifies 2AFC pairs across an acceptability x language-model grid, and seeds the active-learning loop with the pretrained model.FILLED_TEMPLATE_FILLING), judgments (ANNOTATION_RECORD_JUDGMENT+JudgmentSet), list constraints (LIST_CONSTRAINT), and experiment lists (EXPERIMENT_LIST_LAYERS), all exact round-trips. Every gallery artifact (lexicons, templates, filled templates, cross-product items, 2AFC pairs, training corpus, experiment lists) is persisted through the layers schema.evaluate's F1 no longer receives an unsupportedzero_division; and per-participant random-effects tensors are placed on the model device, so training and prediction run on MPS and CUDA, not only CPU.A bead
Participantis intentionally not mapped to a layerspersona.Persona(Persona is an annotator role, not a concrete participant);judgment.AgreementReporthas no bead model to lens.Motivation
Bring
gallery/engcurrent with the 0.7.0 layers interop, add an acceptability-judgment model (trained on MegaAcceptability) to drive item stratification, and generalize stratification from a single quantile dimension to an arbitrary grid of continuous and discrete variables.Type of Change
Checklist
uv run ruff check .anduv run ruff format .uv run pyrightwith no errorsuv run pytest tests/)Testing
pyright bead/: 0 errors;ruff check+ruff format --checkclean acrossbead/.tests/lists/test_grid_stratification.py),AcceptabilityScorer(tests/items/test_acceptability_scorer.py), and the layers lenses (tests/interop/test_layers_resource.py,test_layers_judgment.py,test_layers_list.py), all exact round-trips.bert-base-uncasedon CPU andbert-tinyon MPS: the fit learns (loss drops, train/dev accuracy above chance), dev-set early stopping engages and restores the best weights, and the scorer and active-learning loop both run on the trained model.Screenshots (if applicable)
N/A