Skip to content

feat: add provider-independent quiz output validation rules#415

Open
Xenon010101 wants to merge 2 commits into
MRIARC-08:mainfrom
Xenon010101:fix/quiz-output-validation-rules
Open

feat: add provider-independent quiz output validation rules#415
Xenon010101 wants to merge 2 commits into
MRIARC-08:mainfrom
Xenon010101:fix/quiz-output-validation-rules

Conversation

@Xenon010101

@Xenon010101 Xenon010101 commented Jun 22, 2026

Copy link
Copy Markdown

Closes #374

Summary

Implements provider-independent quiz output validation for the AI backend, covering all acceptance criteria from the issue.

What's included

Pydantic schemas (\schemas/quiz.py)

  • \QuestionType\ (MCQ, SUBJECTIVE) and \Difficulty\ (EASY, MEDIUM, HARD) enums matching Prisma schema
  • \GeneratedQuestion\ with field-level constraints (min_length, ge, le)
  • \ValidationErrorDetail\ with field path, message, error code, and repair hint
  • \QuizValidationResult, \QuestionValidationResult\ for structured output
  • \QuizGenerationInput\ and \QuizGenerationOutput\ for end-to-end workflows

Validation rules

  • MCQ: minimum 2 options, unique labels and values, exactly one correct answer
  • Correct-answer membership verification
  • Explanation required (min 10 chars)
  • Marks must be >= 0.5
  • Cross-question duplicate/near-duplicate detection via difflib.SequenceMatcher
  • Unsupported question types are rejected with a clear error
  • Field-level errors with
    epair_hint\ for controlled repair

Tests (26 cases)

  • Valid MCQ and subjective fixtures pass
  • Invalid correct answers (none, multiple) fail
  • Duplicate option labels/values fail
  • Missing/empty/short explanation fails
  • Zero and negative marks fail (Pydantic field-level)
  • Source citation accepted
  • Exact and near-duplicate questions detected
  • Distinct questions not flagged
  • Field-level error structure verified (field, message, code)
  • Empty list, subjective without options, boundary marks, medium difficulty

@Xenon010101
feat: add provider-independent quiz output validation rules
d18e4dc 
- Pydantic schemas for MCQ and SUBJECTIVE question types
- Field validation: option count, unique options, correct-answer membership
- Presence validation for explanations, marks boundary checks
- Cross-question duplicate detection via difflib similarity
- Rejection of unsupported question types
- Field-level ValidationErrorDetail with repair hints
- 26 parameterized test cases covering valid and invalid fixtures
@vercel

vercel Bot commented Jun 22, 2026

Copy link
Copy Markdown

@Xenon010101 is attempting to deploy a commit to the Adarsh's projects Team on Vercel.

A member of the Team first needs to authorize it.

@Xenon010101
feat: add OpenAPI-based contract tests between FastAPI and Next.js
36bdc2b 
- Python export script (scripts/export-openapi.py) generates FastAPI
  OpenAPI schema without DB, model downloads, or Groq
- Golden spec (openapi.golden.json) as committed baseline
- TypeScript contract tests (src/__tests__/contract.test.ts) validate:
  - All expected path/method combinations exist
  - Response schemas and status codes match golden spec
  - Auth headers (X-Internal-API-Key) are correctly applied
  - Error response schemas (HTTPValidationError, ValidationError)
  - Enum values match TypeScript expectations
  - Breaking path/field/nullability changes are detected
- CI job exports spec from FastAPI and runs contract tests
- One-command regeneration: pnpm contract:update-golden
- Documentation in services/ai/README.md
MRIARC-08
MRIARC-08 requested changes Jun 25, 2026
View reviewed changes

@MRIARC-08 MRIARC-08 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Xenon010101 The Python quiz validation part looks solid, but this PR also includes the shared contract-test scaffolding, and that part fails the Node checks.

Local checks:

  • From services/ai, .venv/bin/python -m pytest passes: 31 tests.
  • From services/ai, .venv/bin/python -m ruff check . passes.
  • pnpm exec tsc --noEmit fails in src/__tests__/contract.test.ts: missing beforeAll import, OpenApiSpec.security is not typed, spec is used outside its scope, and pathItem is unknown.
  • pnpm exec eslint src/__tests__/contract.test.ts fails on explicit any types and an unused path variable.
  • pnpm test src/__tests__/contract.test.ts fails from a clean checkout because services/ai/openapi.json is missing, then the idempotency suite references spec out of scope.

Please either fix the contract-test scaffolding here or split this PR so the Python quiz validation can be reviewed independently from the OpenAPI contract test work.

@MRIARC-08 MRIARC-08 added gssoc gssoc level:intermediate GSSoC difficulty: intermediate type:testing Testing related contribution area:ai-backend Python AI service, embeddings, retrieval, and LLM infrastructure labels Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MRIARC-08 MRIARC-08 MRIARC-08 requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

area:ai-backend Python AI service, embeddings, retrieval, and LLM infrastructure gssoc gssoc level:intermediate GSSoC difficulty: intermediate type:testing Testing related contribution

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[AI BACKEND] Add provider-independent quiz output validation rules

2 participants

@Xenon010101 @MRIARC-08