Skip to content

Title: Step 3 (correctness evaluator setup) may be ordered before the dataset run it depends on #27

Description

@annabellscha

Body:## Summary
In "Step 3 — Set up the correctness evaluator in Langfuse", users configure the
Correctness evaluator and target dataset runs, but the step that actually runs
the dataset comes later in the workshop. So the evaluator setup screen has no
data to map or validate against.

What happens

Following Step 3, the "Set up evaluator" screen shows:

  • Results table (filter Dataset = dad-it-support-workshop) says "No results."
  • Evaluation Prompt Preview says "No trace data found, please adjust filters or
    switch to not show preview."

With no dataset run yet, there's nothing to map the template variables against
or confirm the evaluator is wired correctly.

Expected

Either:

  1. Reorder so the dataset run happens before setting up the evaluator, or
  2. Clarify in Step 3 that empty data is expected here — the evaluator runs
    automatically on new/incoming experiments once the dataset is run in the
    later step ("Run on new experiments" is enabled), so the empty preview is
    normal.

Reported by

Workshop attendee (Si Shuo Yang) via Slack #langfuse-workshop-feedback.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions