Skip to content

sample_input passed as raw string to validate_pipeline, causing numeric models to always fail #18

@AK11105

Description

@AK11105

What happened?

sample_input is collected as a raw string and passed directly to pipeline.run(sample_input). A model expecting a numeric array receives a string and raises a type error. The validation loop treats this as a codegen failure and burns all three LLM retries on code that was never wrong.

Example: user enters [1.2, 0.4, 3.1]. pipeline.run("[1.2, 0.4, 3.1]") passes a string to an sklearn model expecting a numpy array. Every attempt fails. The scaffold is written. The codegen was correct the whole time.

Steps to reproduce

  1. Deploy any sklearn numeric model: inference-engine deploy ./model.pkl --sample-input "[1.2, 0.4, 3.1]"
  2. Observe all three validation attempts fail with a type error
  3. Note the generated load() and predict() are syntactically correct — the input type was the problem

Expected behavior

sample_input is parsed with json.loads before being passed to validate_pipeline. A JSON array becomes a Python list; a plain string stays a string. The raw string is preserved in DeployAnswers for the curl example in the writer.

def _parse_sample_input(raw: str) -> Any:
    try:
        return json.loads(raw)
    except (json.JSONDecodeError, ValueError):
        return raw

Apply in deploy.py and fix.py before every call to validate_pipeline.

Environment

  • Area: app/cli/core/prompts.py, app/cli/commands/deploy.py, app/cli/commands/fix.py
  • Phase: 8 (fix before Phase 9 begins)
  • Priority: High

Relevant logs or error output

Attempt 1/3 failed: TypeError: could not convert string to float: '[1.2, 0.4, 3.1]'
Attempt 2/3 failed: TypeError: could not convert string to float: '[1.2, 0.4, 3.1]'
Attempt 3/3 failed: TypeError: could not convert string to float: '[1.2, 0.4, 3.1]'

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions