feat: improve validation error messages for invalid YAML specs

[aicontentcreate2023-star]

Fixes #24

Summary

Enhance validation error messages to be more descriptive and actionable for developers working with Open Agent Spec YAML files.

Problem

When users provide invalid YAML specs, the current error messages are minimal and don't provide enough context to quickly fix the issue. For example:

• agent.name must be a string (doesn't show what type was actually provided)
• task X references non-existent tool Y (doesn't list available tools)
• No examples of correct format

This leads to:

• Frustration for new users
• Time wasted guessing the correct format
• Back-and-forth debugging

Solution

Enhanced all validation functions to provide:

1. Specific field path (e.g., agent.name, tools[2].id)
2. Actual type received (e.g., got int, got missing)
3. Expected type/format (e.g., must be a string)
4. Example of correct format (YAML snippets)
5. Available options (e.g., list of available tool IDs)

Changes

oas_cli/validators.py

_validate_version()

• Show actual type when wrong type provided
• Include example: open_agent_spec: "1.0"

_validate_agent()

• Check for completely missing agent section
• Show examples for name and role fields
• Display actual type when wrong

_validate_behavioural_contract()

• Improved error messages for version and description
• Better messages for optional fields (behavioural_flags, policy, etc.)

_validate_tools()

• Show array indices in errors: tools[2].id
• Display actual type vs expected
• Provide examples for each required field
• Explain what each field should contain

_validate_tasks()

• Check if task definition is a dict
• List available tools when reference fails
  • Before: task X references non-existent tool 'Y'
  • After: task X references non-existent tool 'Y'. Available tools: 'tool1', 'tool2'. Check your 'tools' section.
• Show examples for input/output schemas
• Better messages for multi-step tasks

tests/test_validation_errors.py (NEW)

Comprehensive test suite verifying:

• Missing fields show helpful messages with examples
• Type mismatches show actual vs expected types
• Array errors include item indices
• Tool references show available options
• Consistent error message format across all validators

Before/After Examples

Example 1: Wrong Type for open_agent_spec

Before:

ValueError: open_agent_spec version must be specified as a string

After:

ValueError: Field 'open_agent_spec' must be a string, got int. Example: open_agent_spec: "1.0"

Example 2: Missing Agent Section

Before:

ValueError: agent.name must be a string

After:

ValueError: Missing required section 'agent'. Add:
  agent:
    name: "your-agent-name"
    role: "agent description"

Example 3: Invalid Tool Reference

Before:

ValueError: task my_task references non-existent tool 'web_search'

After:

ValueError: tasks.my_task references non-existent tool 'web_search'. Available tools: 'file_reader', 'calculator'. Check your 'tools' section.

Example 4: Wrong Type in Array

Before:

ValueError: tool 1.id must be a string

After:

ValueError: tools[1].id must be a string, got int. Provide a unique identifier, e.g., id: "web_search"

Benefits

✅ Faster debugging — users immediately know what's wrong and how to fix it
✅ Better onboarding — new users can self-serve without reading docs extensively
✅ Reduced support burden — fewer questions about "what's wrong with my YAML?"
✅ Consistent format — all errors follow the same pattern
✅ Actionable guidance — every error includes an example or suggestion

Testing

# Run the new test suite
pytest tests/test_validation_errors.py -v

# Test with intentionally broken YAML
echo 'open_agent_spec: 123' > test.yaml
oas validate test.yaml  # Should show improved error

Code Quality

• ✅ 280+ lines of new tests
• ✅ Maintains backward compatibility (still raises ValueError)
• ✅ No breaking changes to public API
• ✅ Follows existing code style

Future Work

This pattern can be extended to:

• JSON Schema validation errors
• Runtime validation errors
• CLI argument validation

[vercel]

Someone is attempting to deploy a commit to the Andrew's projects Team on Vercel.

A member of the Team first needs to authorize it.