Skip to content

Extract Dataset and use api instead of state#98

Merged
delner merged 3 commits intomainfrom
refactor/api_for_eval_datasets
Feb 4, 2026
Merged

Extract Dataset and use api instead of state#98
delner merged 3 commits intomainfrom
refactor/api_for_eval_datasets

Conversation

@delner
Copy link
Collaborator

@delner delner commented Feb 4, 2026

This pull requests makes two refactors to Evals and Datasets.

Issue: Dataset is internal to Eval

Right now, datasets are entirely internal to Eval and inaccessible for other purposes.

Idea: Extract it to a first-class API

dataset = Braintrust::Dataset.new(name: "my-dataset", project: "my-project")
dataset.each { |record| puts record[:input] }

Issue: Use of state for Eval.run

Right now, Eval uses state as an argument to run:

Braintrust::Eval.run(
  project: "my-project",
  dataset: "my-dataset",
  task: ->(input) { classify(input) },
  scorers: [...],
  state: state # <-- Uses Braintrust::State
)

This seems undesirable, given that State is meant to be an internal construct, rather than a public one... if we make changes to state management, it could cause a breaking change.

Idea: Encapsulate within api

Rather than expose state: as an argument, expose api: instead. This simplifies the logic, and substitutes an internal construct for a more meaningfully public one.

@delner delner requested review from clutchski and realark February 4, 2026 23:21
@delner delner self-assigned this Feb 4, 2026
@delner delner added the enhancement New feature or request label Feb 4, 2026
@delner delner merged commit 286b39c into main Feb 4, 2026
7 checks passed
@delner delner deleted the refactor/api_for_eval_datasets branch February 4, 2026 23:47
@clutchski
Copy link
Contributor

I'm not sure about this one. Will comment tmw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants