Interactive web application for evaluating and testing the Section 508 Compliance Analysis pipeline. Upload solicitation PDFs and watch the full AI analysis run in real time with complete LLM prompt/response transparency.
This is the iterative improvement workbench for the 508 AI pipeline. Use it to:
- Upload solicitation documents and run the full analysis pipeline
- See every LLM prompt and response in real time (SSE streaming)
- Compare results across different AI providers (LM Studio, OpenAI, AWS Bedrock)
- Tune temperature, max tokens, and other parameters via the Settings panel
- Download full analysis results as JSON
cd 508_eval_platform
# Create virtual environment (first time only)
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# Copy and configure environment
cp .env.example .env
# Edit .env with your provider settings
# Start the server
./run.shOpen http://localhost:8080 in your browser.
| Provider | Config | Notes |
|---|---|---|
| LM Studio | USE_LMSTUDIO=true |
Local inference, port 1236 |
| OpenAI | USE_OPENAI=true + API key |
GPT-4o default |
| AWS Bedrock | Default (neither flag set) | Requires AWS credentials |
508_eval_platform/
├── eval_ui/ # FastAPI web app
│ ├── api.py # API endpoints + SSE streaming
│ ├── runner.py # Pipeline orchestrator
│ ├── instrumentation.py # LLM call capture
│ ├── models.py # Pydantic data models
│ └── static/ # Frontend (HTML + JS)
├── src/
│ ├── config_loader.py # Environment config
│ └── core/
│ ├── processors.py # Main analysis pipeline
│ ├── vector_matching.py # FAISS vector search
│ ├── file_utils.py # PDF/text extraction
│ ├── bedrock_client.py # AWS Bedrock client
│ ├── lmstudio_client.py # LM Studio client
│ └── openai_client.py # OpenAI client
├── data/standards/ # 508 standards text
├── section_508_db/ # FAISS vector index
├── requirements.txt
├── run.sh
├── .env.example
└── README.md