ACME Presentation Validator

Deterministic ACME PowerPoint guideline validation with Docker-contained rendering, AI review, and a reviewer-facing web workflow.

Approach

I treated this as a review workflow system, not a prompt-only brand checker.

The app parses PPTX structure and OOXML, renders slides inside Docker with LibreOffice, runs deterministic ACME checks where code can prove the issue, groups raw findings into low-noise reviewer-facing issues, and uses AI only as a verifier/ranker for judgement-heavy candidates. Human reviewers can then accept, reject, or leave issues unresolved with notes, and export the final review state.

PPTX upload
-> deterministic PPTX/OOXML extraction
-> Docker-contained LibreOffice render
-> static ACME rule engine
-> raw deterministic findings
-> grouped reviewer issues
-> selected LLM/VLM candidate manifest
-> AI judge decisions
-> final reviewer visibility
-> persisted reviewer decisions and exports

The annotated deck is used for calibration and evaluation, not as hardcoded logic. Guideline items are split into deterministic rules and AI judgement rubric items in the project specs and ACME configuration.

Guideline Artifacts

For this case study, the ACME guideline has been converted into two static executable policy artifacts:

• src/case_study/config/acme/guideline-principles.json for deterministic rules.
• src/case_study/config/acme/vlm-judgement-rubric.md for LLM/VLM judgement items.

The runtime follows these files; it does not ask a model to reinterpret the raw guideline Markdown for every uploaded deck. This keeps validation reproducible and lets the AI layer focus on selected, judgement-heavy candidates.

To support another guideline, create the same two artifacts for that guideline:

1. Convert objective, programmatically checkable rules into JSON entries with stable rule_id, source_section, scope, enforcement, expected, tolerance, severity, confidence, evidence_required, and message_template fields.

2. Convert judgement-heavy items into rubric blocks using:

   ## RULE-ID - Short title
   Judge type: llm
   Applies to: target scope
   Source section: exact guideline section
   Question: narrow judgement question
   Fail conditions:
   - ...
   Pass conditions:
   - ...
   Evidence expectations:
   - ...

   Use llm, vlm, or llm_vlm as the judge type depending on whether the item needs text-only reasoning, rendered slide vision, or both.

3. Wire those artifacts into the configuration/rule registry and run calibration on representative decks.

The same extraction, reviewer issue grouping, AI decision, review-state, and export workflow can then follow the new guideline. If the new rules need facts that are not currently extracted from PPTX or rendered slides, add that evidence to the extractor/rule layer first.

What To Try

1. Start the app with Docker and open http://localhost:8765.
2. Upload one of the reference PPTX decks.
3. Watch the deterministic/render/AI progress log.
4. Review the default-visible issues first, then inspect collapsed or hidden issues when useful.
5. Accept/reject issues, add reviewer notes, refresh the page, and confirm state is preserved.
6. Export JSON, Markdown, or PDF reviewer summaries.

Detailed Docs

• Features
• Web UI Usage
• CLI Usage
• Configuration
• Judge Adapters
• Deployment

Noise Control

The raw validation layer preserves every deterministic finding for audit/debug evidence. The UI does not show that raw list directly. Instead, findings flow through a reviewer-facing layer that groups repeated formatting issues, downgrades high-noise visual/layout candidates, and keeps critical client-facing risks prominent.

AI review does not generate a fresh issue list. It receives only grouped candidate issues with slide images, deterministic evidence, and relevant rubric text. Its job is to confirm, downgrade, hide, or mark a candidate as needs_human_review. A confirm keeps the issue's existing visibility rather than automatically promoting it, so model review cannot flood the default issue bucket.

The AI judge backend is swappable through the runtime-checkable AIJudgeAdapter protocol. The shipped implementation uses Pydantic AI, but another framework such as LangChain, LiteLLM, or a direct provider SDK can be used by implementing judge(request) -> JudgeDecision; see Judge Adapters.

Docker Usage

Build the runtime image:

docker compose build validator

Run the web app:

docker compose up app

Then open http://localhost:8765, upload a PPTX, and review the generated report. The upload pipeline runs PPTX extraction, LibreOffice rendering, deterministic checks, AI judge review, final report generation, and reviewer state persistence. The app stores uploaded sessions under .deck-validation-traces/.

The app upload path requires model credentials for the Pydantic AI/OpenRouter judge. Copy .env.example to .env, then fill in OPENROUTER_API_KEY; docker-compose.yml loads that file as optional runtime environment for both the app and validator services. Deterministic CLI smoke commands can run without .env. Add CASE_STUDY_APP_PASSWORD to password-protect the web app with HTTP Basic Auth. CASE_STUDY_APP_USERNAME is optional and defaults to case-study.

OPENROUTER_API_KEY=...
CASE_STUDY_APP_USERNAME=reviewer
CASE_STUDY_APP_PASSWORD=change-me-before-deploy

If CASE_STUDY_APP_PASSWORD is not set, the app stays open for local development.

Run a model-free deterministic validation smoke for a deck:

docker compose run --rm validator validate "references/Plutos-Assessment.pptx" --out .tmp/docker-validation/plutos --summary

The container includes Python, LibreOffice, required fonts, PDF rasterization, and the application code. Host machines do not need LibreOffice or Python for normal use.

When --out is provided, validation also writes:

• validation-report.md: human-readable English feedback grouped by slide
• validation-report-thin.md: compact reviewer-facing English summary
• validation-report.json: machine-readable validation payload

The JSON payload preserves every raw deterministic finding for audit/debug evidence and also includes reviewer_issues, a grouped and prioritized issue layer for human review. When AI review has been applied, the final report also includes the AI decision layer and final reviewer visibility. The thin report uses that reviewer-facing layer, so repeated table, bullet, and layout findings are collapsed instead of shown as raw one-by-one flags.

If model decisions have been produced separately, apply them without mutating the raw findings or original reviewer issue layer:

docker compose run --rm validator validate "references/Plutos-Assessment.pptx" --out .tmp/docker-validation/plutos --ai-decisions .tmp/judge/plutos/judge-decisions.jsonl --thin

The applied report adds ai_review.applied_reviewer_issues and computes final reviewer visibility for confirm, downgrade, hide, and needs_human_review decisions.

Use --thin to print the compact Markdown report to stdout:

docker compose run --rm validator validate "references/Plutos-Assessment.pptx" --out .tmp/docker-validation/plutos --thin

Both Markdown reports start with validation pass/fail status and average validity coverage.

Run annotated-deck calibration with static extraction only:

docker compose run --rm validator calibrate "references/Goofy-Assessment-ANNOTATED.pptx" --summary

Include render-derived deterministic findings in calibration:

docker compose run --rm validator calibrate "references/Goofy-Assessment-ANNOTATED.pptx" --full-validation --out .tmp/docker-calibration/goofy --summary

Local Development

Local development uses uv:

uv sync
uv run pytest --cov=case_study --cov-report=term-missing --cov-fail-under=100 -q

The AI judge batch runner is available for explicit local model execution after a validation report has been generated:

uv run case-study judge --validation-report .tmp/docker-validation/plutos/validation-report.json --output-dir .tmp/judge/plutos --deck-name plutos --concurrency 4 --force

The runner uses the protocol-based judge adapter surface and the Pydantic AI adapter by default.

Known Limitations

• PDF export uses a swappable Markdown renderer. The current implementation uses markdown-pdf for output quality; if AGPL licensing is not acceptable for a production deployment, the exporter can be replaced without changing validation data, reviewer state, or the web workflow.
• If an AI provider is unavailable, the current upload path fails the run rather than publishing a partial AI-reviewed report. The deterministic report and reviewer grouping layers are independent of model execution, and a production hardening step would keep successful AI decisions while marking failed candidates as collapsed or needs-review.
• The Needs review bucket is supported by the schema and UI, but it can be empty when the selected model chooses only confirm, downgrade, or hide.

License

The source code in this repository is licensed under the GNU Affero General Public License v3.0 only (AGPL-3.0-only).

Case Study Materials

This repository is a redacted public version of a system originally built for a hiring case study. Company-identifying information has been removed, and the included presentation decks use fictional mock data.

This project is not affiliated with or endorsed by the company for which the original case study was prepared.