Documentation-Agent

Production-style AI workflow for generating and serving SOPs (Standard Operating Procedures) from email threads.

The system supports:

• INGEST mode: turn email thread updates into validated, versioned SOPs.
• QUERY mode: answer user questions strictly grounded in the approved SOP content.

It is implemented as a LangGraph multi-agent workflow with policy checks, quality checks, retries, and persistent storage in Supabase plus semantic retrieval via ChromaDB.

Table of Contents

• 1. Problem Statement
• 2. Objectives and Scope
• 3. High-Level Design (HLD)
• 4. Low-Level Design (LLD)
• 5. Data Model
• 6. Configuration
• 7. Setup and Bootstrap
• 8. Commands
• 9. Runtime Outputs and Status Contracts
• 10. Testing
• 11. Troubleshooting
• 12. Security Notes
• 13. Project Structure

1. Problem Statement

Teams often maintain process knowledge in long email threads. This creates three operational problems:

1. SOP drift: process changes are discussed in email but not reflected in official documentation.
2. Retrieval friction: users ask process questions and receive inconsistent answers from memory or outdated docs.
3. Governance risk: generated answers may leak sensitive content or include hallucinated, unapproved instructions.

Documentation-Agent addresses these by converting thread deltas into validated SOP versions and serving policy-filtered, SOP-grounded answers.

2. Objectives and Scope

In Scope

• Incremental SOP generation from thread delta (new, not-yet-ingested messages).
• SOP versioning and persistence in Supabase.
• Query answering based on stored active SOP version(s).
• Policy, quality, and evaluator gates with retry loops.
• Gmail API ingestion path with OAuth 2.0 and token caching.
• Local vector search using ChromaDB and Vertex embeddings (fallback available).

Out of Scope

• Multi-tenant authz/authn and user identity management.
• Human approval workflow UI.
• Real-time streaming ingestion.
• Distributed task queue or horizontal orchestration.

3. High-Level Design (HLD)

3.1 System Architecture

flowchart TD
            CLI[main.py CLI] --> MODE{Mode}

            MODE -->|INGEST| IR[Email Retriever]
            MODE -->|QUERY| SR[SOP Retriever]

            IR --> TP[Thread Parser]
            TP --> TD[Thread Delta Extractor]
            TD --> MC[Merge Email Context]
            MC --> SG[SOP Generator]

            SG --> PI[Policy Checker - Ingest]
            SG --> QI[Quality Checker - Ingest]
            PI --> EI[Evaluator - Ingest]
            QI --> EI

            EI -->|pass| SV[SOP Versioner]
            EI -->|retry| RP1[Retry Prep]
            RP1 --> SG
            EI -->|fail| IF[Ingest Failed]

            SV --> DS[Database Store]
            DS --> SUP[(Supabase)]
            DS --> CH[(ChromaDB)]

            SR --> PQ[Policy Checker - Query]
            PQ -->|allowed| QA[Query Agent]
            PQ -->|denied| QD[Query Denied]
            QA --> QQ[Quality Checker - Query]
            QQ --> EQ[Evaluator - Query]
            EQ -->|pass| QS[Query Success]
            EQ -->|retry| RP2[Retry Prep]
            RP2 --> QA
            EQ -->|fail| QF[Query Failed]

Loading

3.2 Major Components

• Orchestration: LangGraph state machine in graph/sop_graph.py.
• Agents: generation, quality, policy, evaluator, retrieval, versioning.
• Persistence: Supabase tables (sops, sop_versions, thread_messages).
• Retrieval: Chroma persistent collection with Vertex embeddings (text-embedding-004) and SHA256 fallback.
• LLM backend: Vertex AI gemini-1.5-pro (configurable).
• Integration: Gmail OAuth + Gmail REST API for thread retrieval.

3.3 Core Data Flow

INGEST path:

1. Retrieve thread from Gmail or local sample snapshot.
2. Normalize and sort messages.
3. Compute delta by filtering already-ingested message_ids.
4. Merge existing active SOP payload with delta context.
5. Generate SOP draft.
6. Run policy and quality checks in parallel.
7. Evaluate combined result; retry generator if needed.
8. On pass, version and persist SOP, store ingested message IDs, upsert vector doc.

QUERY path:

1. Resolve SOP context by explicit --sop-id or vector search.
2. Run policy check on query text.
3. Generate answer from retrieved SOP only.
4. Validate grounding and basic quality.
5. Retry answer generation if needed.
6. Return success, denied, or failed status payload.

4. Low-Level Design (LLD)

4.1 Graph State Contract

Key state fields used by the graph:

• Routing/control: mode, retry_count, feedback.
• Ingest context: thread_id, messages, thread_delta, merged_context, generated_sop.
• Query context: query, retrieved_sop, retrieval_matches, query_answer.
• Validation: policy_result, quality_result, evaluation_result.
• Persistence/result: version_result, final_response.
• Observability: timeline (state events written at key stages).

4.2 Node-Level Behavior

INGEST nodes:

• email_retriever: retrieves thread, parses/normalizes, seeds SOP ID if absent.
• thread_delta_extractor: stateless membership-based delta on message_id.
• merge_email_context: deterministic context merge (existing_sop, thread_delta).
• sop_generator_agent: Vertex LLM JSON generation with deterministic fallback.
• policy_checker_ingest: deny-keyword based policy guard.
• quality_checker_ingest: step presence + lexical grounding against delta.
• evaluator_ingest: schema validation + required fields + merged feedback.
• ingest_retry_prep: increments retries and propagates feedback.
• sop_versioner: computes change summary and writes next active version.
• database_store: upserts message tracking and vector document.

QUERY nodes:

• sop_retriever: fetch active payload by sop_id or semantic match.
• policy_checker_query: blocks disallowed query content.
• query_agent: Vertex answer constrained by prompt, with fallback answer.
• quality_checker_query: computes grounding ratio against SOP blob.
• evaluator_query: enforces non-empty output + quality gating.
• query_retry_prep: retries response generation with feedback.

4.3 Retry Strategy

• Max retries from env var MAX_AGENT_RETRIES (default 3).
• Ingest retry loop: sop_generator_agent -> checks -> evaluator -> retry.
• Query retry loop: query_agent -> quality -> evaluator -> retry.
• Terminal failure returns structured final_response with accumulated feedback.

4.4 Validation and Safety Layers

Validation stack in order:

1. Policy check (keyword denylist).
2. Quality check (grounding/actionability heuristics).
3. Evaluator schema checks (SOPOutput for ingest).
4. Retry/fail routing.

4.5 Namespace/Import Design

This repository uses a namespace shim so imports follow email_sop_agent.<module> even though folders are top-level. This behavior is established in email_sop_agent/__init__.py and test root conftest.py.

5. Data Model

Supabase schema (database/schema.sql):

sops

• id (text, PK)
• thread_id (text)
• domain (text)
• title (text)
• current_version (integer)
• updated_at (timestamptz)

sop_versions

• id (identity PK)
• sop_id (FK -> sops.id)
• version (int, unique per SOP)
• status (active or deprecated)
• payload (jsonb)
• change_summary (text)
• created_at, updated_at

thread_messages

• message_id (text, PK)
• thread_id (text)
• sop_id (FK -> sops.id)
• ingested_at (timestamptz)

6. Configuration

6.1 .env Example

Copy .env.example to .env, then fill values:

cp .env.example .env

Recommended .env template:

# Supabase
SUPABASE_URL=https://your-project-ref.supabase.co
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

# Gmail OAuth (required only for --gmail ingestion)
GOOGLE_CLIENT_ID=your-google-oauth-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=your-google-oauth-client-secret

# Vertex AI
SERVICE_ACCOUNT_PATH=service-account.json
VERTEX_MODEL=gemini-1.5-pro
VERTEX_LOCATION=us-central1

# Optional runtime tuning
CHROMA_PATH=.chroma
SAMPLE_THREADS_FILE=tests/sample_email_threads.json
MAX_AGENT_RETRIES=3

Notes:

• For backend writes with RLS enabled, prefer SUPABASE_SERVICE_ROLE_KEY.
• SUPABASE_ANON_KEY is a fallback and may not have write permissions.
• SERVICE_ACCOUNT_PATH is read by both LLM and embedding clients.

6.2 service-account.json (service JSON) Example

Create a service account JSON file at the path specified by SERVICE_ACCOUNT_PATH.

{
      "type": "service_account",
      "project_id": "your-gcp-project-id",
      "private_key_id": "replace-with-key-id",
      "private_key": "-----BEGIN PRIVATE KEY-----\\nREPLACE_WITH_PRIVATE_KEY\\n-----END PRIVATE KEY-----\\n",
      "client_email": "documentation-agent@your-gcp-project-id.iam.gserviceaccount.com",
      "client_id": "123456789012345678901",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/documentation-agent%40your-gcp-project-id.iam.gserviceaccount.com"
}

Required IAM role for this service account:

• roles/aiplatform.user

6.3 Gmail OAuth Requirements

If using Gmail ingestion:

• Create OAuth 2.0 client credentials in Google Cloud Console.
• Add GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET.
• First run caches tokens in .gmail_tokens.json.

7. Setup and Bootstrap

7.1 Prerequisites

• Python 3.11+
• Supabase project
• Google Cloud project with Vertex AI API enabled
• Service account JSON for Vertex AI
• Optional Gmail OAuth credentials

7.2 Install Dependencies

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

7.3 Initialize Database Schema

Run SQL in database/schema.sql inside Supabase SQL Editor.

8. Commands

Run all commands from repository root.

8.1 Ingest from Local Sample Thread

python main.py --mode INGEST --thread-id THREAD-ENG-ONBOARDING-01

8.2 Ingest from Gmail by Thread ID

python main.py --mode INGEST --gmail --thread-id 196a2f0examplethreadid

8.3 Ingest from Gmail by Search Phrase

python main.py --mode INGEST --gmail --thread-id "please apply"

If no --access-token is passed, CLI prompts for token or OAuth code flow.

8.4 Query by Explicit SOP ID

python main.py --mode QUERY --sop-id SOP-THREAD-ENG-ONBOARDING-01 --query "What are the onboarding steps?"

8.5 Query Without SOP ID (semantic resolve)

python main.py --mode QUERY --query "How do we onboard a new engineer account?"

8.6 Gmail OAuth Redirect URI Override

python main.py --mode INGEST --gmail --thread-id "security review" --redirect-uri "http://localhost:8080/callback"

9. Runtime Outputs and Status Contracts

All CLI runs print JSON.

INGEST success:

{
      "status": "ingest_success",
      "sop_id": "SOP-THREAD-ENG-ONBOARDING-01",
      "version": 2,
      "thread_delta_count": 1
}

INGEST failure:

{
      "status": "ingest_failed",
      "feedback": [
            "Generated SOP has no actionable steps."
      ]
}

QUERY success:

{
      "status": "query_success",
      "sop_id": "SOP-THREAD-ENG-ONBOARDING-01",
      "answer": "1. Create account ..."
}

QUERY denied:

{
      "status": "query_denied",
      "reason": "Blocked by POL-001 for keyword 'token'."
}

QUERY failed:

{
      "status": "query_failed",
      "feedback": [
            "Answer contains content not sufficiently grounded in retrieved SOP."
      ],
      "partial_answer": "..."
}

10. Testing

Run full suite:

python -m pytest tests -v

Run selected suites:

python -m pytest tests/test_schema.py -v
python -m pytest tests/test_gmail_oauth.py -v
python -m pytest tests/test_vector_db.py -v

11. Troubleshooting

Supabase init error

Symptom:

• ValueError: SUPABASE_URL and a Supabase key must be set...

Fix:

• Set SUPABASE_URL and either SUPABASE_SERVICE_ROLE_KEY or SUPABASE_ANON_KEY.

Vertex client initialization fails

Symptom:

• LLM or embedding calls fail.

Fix:

• Confirm SERVICE_ACCOUNT_PATH points to valid JSON.
• Verify project_id exists in service account file.
• Ensure Vertex AI API is enabled and IAM role is granted.

Gmail auth prompt repeats

Symptom:

• Re-auth required each run.

Fix:

• Ensure .gmail_tokens.json is writable.
• Confirm refresh token was issued (prompt=consent, access_type=offline).

Query returns ungrounded/failed

Possible causes:

• SOP not ingested yet.
• Retrieved SOP context does not contain answer.

Fix:

• Run INGEST first for relevant thread.
• Use explicit --sop-id for deterministic retrieval.

12. Security Notes

• Never commit .env, service-account.json, or .gmail_tokens.json.
• Rotate Google and Supabase credentials if exposed.
• Treat generated outputs as untrusted until policy and quality gates pass.
• POL-001 blocks secret-like terms (password, secret, api key, token) to reduce leakage risk.

13. Project Structure

Documentation-Agent/
      main.py
      conftest.py
      README.md
      requirements.txt
      .env.example
      service-account.json                     # local-only credential file (do not commit secrets)
      agents/
            email_retriever_agent.py
            evaluator_agent.py
            policy_checker_agent.py
            quality_checker_agent.py
            query_agent.py
            sop_generator_agent.py
            sop_versioner_agent.py
            thread_parser_agent.py
      config/
            settings.py
      database/
            db.py
            repository.py
            schema.sql
      graph/
            router_node.py
            sop_graph.py
      prompts/
            evaluator_prompt.py
            query_prompt.py
            sop_generator_prompt.py
      schemas/
            sop.py
      services/
            gmail_oauth.py
            gmail_service.py
            ingestion_pipeline.py
            query_pipeline.py
            vertex_llm.py
      tools/
            merge_email_context.py
            policy_retriever.py
            pydantic_validator.py
            sop_retriever.py
            sop_versioner.py
            state_memory_writer.py
            thread_delta_extractor.py
      vectorstore/
            policy_loader.py
            vector_db.py
      tests/
            test_*.py

---