ML Knowledge & Evaluation Platform for Model Development

A citation-backed RAG knowledge platform over ML papers, model cards, API docs, experiment notes, and evaluation reports. It lets ML engineers ask questions and get answers grounded in retrieved sources, and ships an MLflow-backed evaluation harness (retrieval metrics + factual scoring + regression gate) to track model/index behavior across updates.

Stack: RAG · FastAPI · LangChain · FAISS · GPT-4 · MLflow

---

Highlights

• Citation-backed Q&A — every answer must support its claims with [n] markers that resolve back to the exact source chunk (doc id, title, source type, snippet, score).
• Robust ingestion — load → normalize → deduplicate (exact content-hash + near-dup Jaccard shingling) → chunk (overlapping, traceable chunk ids).
• FAISS retrieval — cosine similarity over normalized embeddings, persisted to disk.
• MLflow-backed evaluation — recall@k, precision@k, MRR, hit-rate, factual accuracy, and citation rate logged per run so behavior is comparable across versions.
• Regression gate — a pytest suite fails the build if factual accuracy drops below the configured threshold (default 0.90).
• Runs offline — with no API key it uses a deterministic local backend, so retrieval, evaluation, and CI work without network access. Set OPENAI_API_KEY to switch to GPT-4 + OpenAI embeddings automatically.

Architecture

                 ┌─────────────────────────────────────────────┐
   corpus/  ──▶  │  Ingestion: load → dedup → chunk             │
 (papers,        └─────────────────────────────────────────────┘
  model cards,                     │ chunks
  api docs,                        ▼
  exp notes,     ┌─────────────────────────────────────────────┐
  eval reports)  │  FAISS Vector Index (persisted)             │
                 └─────────────────────────────────────────────┘
                                   │ top-k + scores
                                   ▼
 user query ──▶  ┌─────────────────────────────────────────────┐
                 │  RAG Pipeline: cited prompt → GPT-4 →        │
                 │  parse [n] markers → resolve citations       │
                 └─────────────────────────────────────────────┘
                                   │ answer + citations
                                   ▼
                 ┌─────────────────────────────────────────────┐
                 │  Evaluation harness → MLflow (metrics)       │
                 │  retrieval metrics · factual accuracy · gate │
                 └─────────────────────────────────────────────┘

Project layout

app/
  config.py        # env-driven settings (offline/OpenAI auto-switch)
  backends.py      # embeddings + LLM backends (OpenAI / deterministic offline)
  ingestion.py     # load, deduplicate, chunk
  vectorstore.py   # FAISS index: build/search/save/load
  pipeline.py      # retrieval → cited prompt → generation → citations
  evaluation.py    # retrieval metrics, factual scoring, MLflow logging
  api.py           # FastAPI app
data/corpus/       # sample ML corpus (papers, cards, api docs, notes, reports)
eval/eval_dataset.json   # curated evaluation queries + gold doc ids + keywords
scripts/           # ingest_corpus.py, run_eval.py
tests/             # ingestion, pipeline, and regression tests
main.py            # `python main.py` / `uvicorn main:app`

Quick start

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env        # optional: add OPENAI_API_KEY for GPT-4, else runs offline

python -m scripts.ingest_corpus   # build the FAISS index
python main.py                    # serve API at http://localhost:8000/docs

Ask a question

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "How does LoRA reduce trainable parameters for fine-tuning?"}'

Response includes the answer plus a citations array, each mapping a [n] marker to its source document.

Testing in the Swagger UI

With the API running (python main.py), open http://localhost:8000/docs for an interactive UI. To test a question:

1. Expand the POST /query endpoint and click "Try it out".

2. Paste a request body and click "Execute":

   {
     "query": "How does LoRA reduce trainable parameters for fine-tuning?",
     "top_k": 4
   }

3. Read the Response body — the answer plus a citations array (each [n] marker maps to a doc_id, title, source_type, snippet, and similarity score).

Other endpoints to try the same way:

• POST /evaluate — runs the full evaluation suite and returns the metrics + passed_gate (set log_to_mlflow=false to skip creating an MLflow run).
• POST /ingest — rebuilds the FAISS index from data/corpus/ after you add documents.
• GET /health — service and index status.

Sample queries that map cleanly to the bundled corpus:

• What objective is BERT pretrained with?
• What is the rate limit on the predict endpoint?
• What chunk size and overlap maximized retrieval recall?
• What F1 does sentiment-classifier-v3 achieve and what are its limitations?

Evaluation & regression testing

python -m scripts.run_eval        # runs the curated set, logs metrics to MLflow
mlflow ui                         # inspect runs at http://localhost:5000
pytest -q                         # regression gate: fails if accuracy < ACCURACY_GATE

Each MLflow run logs params (models, chunking, retrieval_k, backend) and metrics (factual_accuracy, citation_rate, recall_at_k, precision_at_k, mrr, hit_rate, avg_latency_ms, passed_gate) so you can compare model/index updates over time.

API endpoints

Endpoint	Method	Description
/	GET	Service info
/query	POST	Citation-backed Q&A
/ingest	POST	Rebuild the FAISS index from the corpus
/evaluate	POST	Run the evaluation suite (MLflow-logged)
/health	GET	Service + index status

Configuration

All settings come from environment variables (see .env.example). Key ones: OPENAI_API_KEY, LLM_MODEL, EMBEDDING_MODEL, CHUNK_SIZE, CHUNK_OVERLAP, NEAR_DUP_THRESHOLD, RETRIEVAL_K, MLFLOW_TRACKING_URI, ACCURACY_GATE, OFFLINE.

Extending the corpus

Drop .json, .jsonl, .md, or .txt files into data/corpus/ (JSON records use id, title, source_type, content). The included sample is a representative subset; the pipeline scales to a much larger corpus (500+ documents) — re-run python -m scripts.ingest_corpus to rebuild the index.

Security

• Never commit .env or API keys. .gitignore excludes .env; use .env.example as a template. If a key was ever committed, rotate it immediately.