Skip to content

Include sample_input in generate(), fix(), and interpretation prompts #25

@AK11105

Description

@AK11105

What problem does this solve?

sample_input is collected during deploy but only used for validation. The LLM generates load() and predict() without knowing the concrete input type — it relies entirely on inferred hints. The fix loop receives only a traceback and the broken code, with no information about what input caused the failure.

This means the LLM is generating and fixing blind. Including sample_input is the single cheapest improvement to codegen quality.

Proposed solution

Include sample_input in all three LLM calls:

generate() prompt — append after the metadata block:

Sample input: [1.2, 0.4, 3.1]
predict() must handle this exact input type.

fix() prompt — append metadata summary and sample input so the LLM knows what input caused the failure, not just the traceback:

Artifact metadata: {"framework": "sklearn", "class_name": "RandomForestClassifier", ...}
Sample input used during validation: [1.2, 0.4, 3.1]

LLM interpretation prompt (A3) — already included by design; listed here for completeness.

Alternatives considered

Relying on input_hint from the inspector alone. input_hint is a string description; sample_input is a concrete value. "great movie" tells the LLM more about input handling than "raw text string".

Area

CLI (deploy / fix)

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions