AccessAware RAG

AccessAware RAG is a prototype Retrieval-Augmented Generation system focused on one core security idea:

A RAG system should only retrieve and send context to the language model that the server-side authorization principal is allowed to access.

The project demonstrates secure document ingestion, classification-aware retrieval, grounded answer generation, source citations, retrieval tuning, and privacy-conscious audit logging.

Current Status

Current checkpoint:

Working tree: clean at checkpoint
Current HEAD: 4130b63 Add upload provider policy enforcement test
Unit tests: 58 passed
Smoke tests: passed at prior checkpoint
Rich-demo evals: passed at prior checkpoint
Current auth boundary: TrustedPrincipal
Provider policy mode: off | observe | enforce
Configured AI provider mode: external | enterprise | local
Core invariant: authorization before retrieval, ranking, context construction, and model generation
Provider-policy invariant: configured provider eligibility is evaluated before upload-time embedding and before RAG answer generation

This is a local development prototype. It currently supports:

• FastAPI backend
• PostgreSQL with pgvector
• OpenAI embeddings
• OpenAI answer generation
• Plain text, Word, and text-based PDF document upload
• Text extraction for .txt, .docx, and text-extractable .pdf files
• Document chunking and embedding with upload-time provider-policy gating
• Token counts on uploaded chunks
• Source metadata on uploaded documents
• Classification metadata on documents
• Provider/data-egress policy metadata on documents: contains_phi, external_ai_approved, requires_local_processing
• Server-side TrustedPrincipal boundary for endpoint and retrieval authorization
• User clearance-based retrieval filtering plus group/document grant enforcement
• Grounded RAG answers with source metadata
• Vector search endpoint
• RAG answer endpoint
• Document listing endpoint
• Document deletion endpoint for confidential users
• Audit logging for document upload, provider-policy blocks, and question activity
• Neighbor chunk expansion and simple local reranking
• Hybrid retrieval first slice using authorized keyword candidates
• PowerShell smoke test script for the core local demo flow
• Confidential-only audit event listing endpoint
• Provider/model metadata in safe audit records
• Tested provider-policy decision service for external, enterprise, and local provider modes
• Provider-policy observe/enforce controls for /ask/rag answer generation
• Rich demo corpus and retrieval eval script, including unauthorized high-similarity access checks
• Architecture and security documentation under docs/

The system is intentionally small enough to understand, but realistic enough to demonstrate important security and retrieval design choices.

Why This Project Exists

Basic RAG systems often retrieve the most semantically relevant chunks without checking whether the user is allowed to see those chunks.

That creates a security problem:

User asks question
-> vector search finds relevant restricted document
-> restricted content is sent to the model
-> model may expose sensitive information

AccessAware RAG changes the retrieval flow:

User asks question
-> resolve demo/local identity when enabled
-> build server-side TrustedPrincipal
-> derive allowed classifications and group memberships from TrustedPrincipal
-> filter documents/chunks by classification and grants
-> rank only authorized chunks
-> build model context from authorized chunks only
-> return answer with source metadata
-> write safe audit event

The model should never receive unauthorized document context. Semantic similarity can influence ranking only after authorization has already narrowed the candidate set.

Current Classification Model

The prototype uses four classification levels:

public
internal
restricted
confidential

Users have a clearance level. A user can access documents at or below their clearance.

User Clearance	Allowed Classifications
public	public
internal	public, internal
restricted	public, internal, restricted
confidential	public, internal, restricted, confidential

The prototype now combines classification clearance with group/document grants.

Classification controls the maximum sensitivity level a user may access. Document grants add need-to-know control for specific documents.

Current grant rule:

no grants on document = open within classification
one or more grants = user must have a direct user grant or group grant

Retrieval requires both:

classification allowed
AND
document grant/relationship allowed when grants exist

In this prototype, the confidential user also acts as a demo admin/view-all user. In production, clearance, admin rights, and break-glass access should be separated.

Supported File Types

Current ingestion support:

File type	Status	Notes
.txt	Supported	Plain text extraction
.docx	Supported	Word paragraph extraction
.pdf	Supported with caveat	Text-based PDFs only

Important PDF limitation:

PDF support currently means text-extractable PDFs. Scanned/image-only PDFs require OCR and remain future scope.

Tech Stack

• Python
• FastAPI
• SQLAlchemy
• Alembic
• PostgreSQL
• pgvector
• Docker Compose
• OpenAI embeddings
• OpenAI Responses API
• PowerShell-friendly local development workflow

Dependency management uses uv.

This project intentionally does not use requirements.txt.

Documentation

Additional architecture and security notes live under docs/.

Useful starting points:

• docs/ARCHITECTURE.md
• docs/SECURITY.md
• docs/DEMO.md
• docs/ROADMAP.md
• docs/TROUBLESHOOTING.md

These docs are intentionally compact. Older reader-facing fragments were consolidated into the current documentation package.

Core Data Model

The current backend includes models for:

• User
• Document
• DocumentChunk
• AuditEvent
• Group
• UserGroup
• DocumentAccessGrant

Documents store metadata such as:

• filename
• content type
• source
• classification level
• provider/data-egress policy flags:
  • contains_phi
  • external_ai_approved
  • requires_local_processing
• SHA-256 hash

Chunks store:

• document ID
• chunk index
• chunk text
• embedding vector
• token count

Audit events store safe operational metadata about important actions.

Groups, user-group memberships, and document access grants support need-to-know authorization on top of classification clearance.

Main API Endpoints

Health

GET /health

Basic API health check.

Document Upload

POST /documents/upload

Uploads a supported document file, evaluates embedding provider-policy eligibility from request metadata, then extracts text, chunks it, embeds each chunk, stores document metadata, and writes an audit event when allowed. In enforce mode, provider-policy-blocked uploads are rejected before file content is read, extracted, chunked, embedded, or stored.

Currently supported formats:

• .txt
• .docx
• text-based .pdf

Scanned/image-only PDFs are not supported yet. They require OCR and are future scope.

Current upload metadata includes:

• classification_level
• source
• contains_phi
• external_ai_approved
• requires_local_processing

Example source values used in local testing:

demo
upload
rich-demo
test-seed

Document Listing

GET /documents

Lists documents the current TrustedPrincipal is authorized to see.

Supports optional filtering by:

• source
• classification_level

Document Delete

DELETE /documents/{document_id}

Deletes a document by ID.

In this prototype, deletion requires a confidential user.

Vector Ask

POST /ask/vector

Returns authorized vector search results with source metadata.

RAG Ask

POST /ask/rag

Retrieves authorized context, evaluates provider/data-egress policy for answer generation, generates a grounded answer when allowed, returns source metadata, and writes an audit event. In enforce mode, provider-policy-blocked answer generation returns a safe fallback without calling the answer model.

Audit Events

GET /audit-events

Lists recent audit events for confidential/admin review.

In this prototype, audit-event listing is restricted to confidential users. Lower-clearance users receive an access-denied response and the denied attempt is also audited.

Prototype Authentication

The current prototype uses an X-User-Email request header for local/demo identity, but that header is accepted only when demo auth is explicitly enabled.

Expected local/demo settings:

AUTH_MODE=demo
DEMO_AUTH_ENABLED=true

Example:

-H "X-User-Email: restricted.user@example.com"

This is a development shortcut, not production authentication. The request identity is resolved into a server-side TrustedPrincipal, and endpoints/retrieval should make authorization decisions from that principal rather than from the raw header.

Future work should replace demo identity with real authentication, likely JWT/OIDC-based auth with trusted claims used to resolve identity, clearance, groups, and grants into the same TrustedPrincipal shape.

Retrieval Design

Current retrieval flow:

1. Clean and validate the question.
2. Resolve the request identity into a server-side TrustedPrincipal.
3. Derive allowed classifications and group memberships from TrustedPrincipal.
4. Build an authorized base query using classification and grant filters.
5. Embed the question.
6. Select authorized vector seed chunks.
7. Add authorized keyword candidates.
8. Expand seed chunks to nearby chunks from the same authorized document.
9. Deduplicate expanded chunks.
10. Apply simple local reranking to authorized candidates only.
11. Return the final authorized context chunks.

Current retrieval settings:

seed_limit=3
final_context_limit=6

Neighbor expansion was added after a realistic failure case:

Question:
What is the deductible for Northwind Standard?

Problem:
The answer existed in a nearby chunk, but initial top-3 vector retrieval missed it.

Fix:
Neighbor expansion included the nearby chunk, and reranking promoted it.

Result:
The system correctly answered:
$2,000 per person, per year.

The internal-user access test still refused to expose the restricted Northwind Standard answer, confirming that retrieval-quality improvements did not bypass authorization.

Reranking

The current reranker is simple and local. It boosts chunks based on:

• question term matches
• filename term matches
• exact phrase/entity matches
• dollar amounts for cost-related questions
• deductible mentions for deductible questions

This does not call another model. It is intended as a lightweight prototype reranker.

Future work may include stronger hybrid search, BM25-style keyword search, adaptive retrieval, or dedicated reranking models.

Demo Corpus

The current primary demo corpus uses eight synthetic documents with source=rich-demo:

Document	Classification	Audience / grant	Key fact / purpose
Public_FAQ.txt	public	no grant	Plan-specific costs are not public.
Employee_Handbook.txt	internal	benefits-reviewers	Mentions PerksPlus but not the exact reimbursement amount.
PerksPlus_Benefits.txt	internal	benefits-reviewers	PerksPlus reimbursement is $600/year.
Northwind_Standard_Benefits.txt	restricted	northwind-reviewers	Standard deductible is $2,000.
Northwind_Plus_Benefits.txt	restricted	northwind-reviewers	Plus deductible is $1,000.
Claims_Data_Handling_Procedure.txt	restricted	northwind-reviewers	Regulated data handling and authorization-before-retrieval language.
Vendor_AI_Assessment.txt	restricted	security-reviewers	AI/vendor governance; semantic similarity is not authorization.
Executive_Risk_Memo.txt	confidential	confidential prototype admin only	Northwind renewal risk is High.

Useful demo questions:

What is the PerksPlus reimbursement amount?

What is the deductible for Northwind Standard?

Compare the deductible for Northwind Standard and Northwind Plus.

What is the renewal risk in the Executive Risk Memo?

Expected behavior:

• Confidential user can list all eight rich-demo documents.
• Internal user can access allowed internal benefits material, but not restricted Northwind details.
• Restricted user can answer Northwind Standard and Plus deductible questions.
• Public user receives a grounded fallback for plan-specific costs.
• Lower-clearance users cannot retrieve restricted/confidential sources or answer with their protected facts.
• Lower-clearance users cannot retrieve or answer from the confidential Executive Risk Memo.

Example RAG Request

$body = @{
  question = "What is the deductible for Northwind Standard?"
  source = "rich-demo"
} | ConvertTo-Json

Invoke-RestMethod `
  -Method Post `
  -Uri "http://localhost:8000/ask/rag" `
  -Headers @{ "X-User-Email" = "restricted.user@example.com" } `
  -ContentType "application/json" `
  -Body $body | ConvertTo-Json -Depth 6

Expected behavior:

• Restricted user can answer from restricted Northwind documents.
• Internal user cannot access restricted Northwind detail documents.
• Unauthorized users receive a grounded fallback instead of restricted content.

Smoke Tests

The repo includes a PowerShell smoke test script:

.\scripts\smoke-test.ps1

On Windows, the local execution policy may block unsigned PowerShell scripts. Use a one-time execution policy bypass:

powershell -ExecutionPolicy Bypass -File .\scripts\smoke-test.ps1

The smoke test currently verifies:

• API health endpoint responds.
• Confidential user can list the expected demo documents.
• Internal user can answer the PerksPlus demo question.
• Restricted user can answer the Northwind Standard deductible question.
• Internal user cannot access restricted Northwind detail sources.
• Uploaded documents store source and token_count.
• Word document upload and RAG retrieval work.
• PDF upload, RAG retrieval, and restricted access enforcement work.

The repo also includes scripts/eval-rich-demo.ps1 for richer retrieval and access-control checks against the rich-demo corpus, including unauthorized high-similarity questions where the best semantic match must still be blocked by authorization.

Recent local smoke-test checkpoint:

Smoke tests passed.

Audit Logging

The system writes audit events for important operations, including:

• document listing
• document upload
• document delete
• cleanup attempts
• vector questions
• RAG questions
• access denied events
• audit-event listing access and denied attempts

Audit logs intentionally avoid storing sensitive content.

The audit log should not store:

• full user questions
• answer text
• retrieved chunk content
• embeddings
• full document content

Audit metadata can include safe operational fields such as:

• event type
• route
• user email
• user ID
• result count
• source filter
• source document IDs
• source classifications
• top chunk IDs
• provider/model metadata

Provider/model metadata is included in safe audit records so the system can answer operational questions such as:

Which model or provider processed this request?
Which embedding model and answer model were configured?
What retrieval limit and access-control mode were used?

Provider-policy metadata is also included in RAG audit records:

provider_policy_mode=observe
ai_provider_mode=external
provider_policy_operation=answer_generation
provider_policy_allowed=false
provider_policy_reasons=confidential_external_ai_blocked
provider_policy_document_ids=27

This means the system can audit or enforce when authorized context is disallowed for the configured AI provider.

Audit events can be listed through GET /audit-events by confidential users for local review.

PHI and Provider Boundary

Access control answers:

Who is allowed to retrieve this content?

Provider policy answers a different question:

Where is authorized content allowed to be processed?

This project now has a provider/data-egress policy foundation:

• document-level fields:
  • contains_phi
  • external_ai_approved
  • requires_local_processing
• provider modes:
  • external
  • enterprise
  • local
• operation types:
  • embedding
  • answer_generation
  • reranking
• tested provider-policy decision service
• upload/list/detail support for provider-policy metadata
• upload-time provider-policy enforcement for embedding before file read/text extraction/chunking/storage
• observe/enforce provider-policy controls during /ask/rag answer generation

Current provider-policy behavior:

document upload
  -> evaluate provider policy from declared metadata for operation=embedding
  -> in observe mode, continue upload and audit the decision
  -> in enforce mode, block disallowed content before file read, extraction, chunking, embedding, or storage

RAG answer generation
  -> retrieve authorized context
  -> evaluate provider policy for distinct source documents with operation=answer_generation
  -> in observe mode, answer normally and audit the decision
  -> in enforce mode, return a safe provider-policy fallback without calling the answer model

Example manual verification:

confidential user asks about Executive_Risk_Memo
  -> authorization allows retrieval
  -> provider policy identifies external answer generation as blocked
  -> audit records confidential_external_ai_blocked
  -> observe mode answers; enforce mode safely blocks answer generation

Current production limitation:

Provider policy now gates upload-time embedding and RAG answer generation, but the project still does not include a real local/private model route, redaction/de-identification pipeline, or production compliance validation.

Target production rule:

public/internal content may use an external provider only if configuration permits
restricted/confidential/PHI content should require an approved enterprise/private/local provider, redaction/de-identification, retrieval-only handling, or a policy-blocked response

Future production-oriented controls should include:

• local/private embedding and answer-generation providers
• approved enterprise provider configuration
• classification-based model routing
• reranker provider policy enforcement
• redaction or de-identification before chunking and embedding when required
• post-retrieval scrubbing before prompt construction when required

Current Limitations

This is not production-ready yet.

Known limitations:

• Local demo identity starts from a gated X-User-Email header, then resolves into a server-side TrustedPrincipal. JWT/OIDC integration is planned for production-style authentication. D- Clearance, admin/view-all behavior, and break-glass access are not separated yet.
• Group/document grants work, but need more hardening around direct user grants, denied edge cases, and production-style admin behavior.
• Supported ingestion currently includes .txt, .docx, and text-based .pdf.
• Scanned/image-only PDFs are not supported yet because OCR is intentionally deferred.
• Chunking is still character-based with paragraph/sentence preference, not fully token-based.
• Hybrid keyword retrieval is still an early Python-side implementation.
• No background ingestion worker yet.
• No React UI yet.
• Smoke tests and rich-demo evals cover important local flows, including unauthorized high-similarity access checks, but broader automated test coverage is still limited.
• No deployment hardening yet.
• Provider policy now enforces upload-time embedding and RAG answer generation, but no provider routing/local-private model path exists yet.
• No PHI/private-model routing or redaction/de-identification pipeline yet.
• No observability/OpenTelemetry yet.

Planned Improvements

Near-term roadmap:

1. Add route-level and integration-style tests for provider-policy enforce mode on RAG answers and upload-time embedding.
2. Add local/private provider route or explicit retrieval-only mode for blocked sensitive content.
3. Harden ReBAC/document grants with direct user grant tests, denied edge cases, and clearer production admin behavior.
4. Add a reranker abstraction so future local/API rerankers can be plugged in cleanly.
5. Expand retrieval eval coverage for broad summaries, ambiguous source names, and same-question/different-user behavior.
6. Add React demo UI.
7. Add PHI/private-model routing and redaction/de-identification controls.
8. Add token-aware chunking with paragraph/sentence boundary awareness.
9. Add OCR support for scanned/image-only PDFs if needed.
10. Add observability/OpenTelemetry.

Retrieval and chunking roadmap:

• Keep current character-based chunking for now.
• Improve eval coverage before major retrieval changes.
• Add adaptive authorized retrieval so the system can expand candidates when filters remove too many results.
• Wrap the heuristic reranker behind an abstraction so future local/model rerankers can be added cleanly.
• Move from fixed character chunks to sentence-aware or paragraph-aware chunking.
• Later move toward token-aware chunking.
• Consider target chunk sizes around 512-1,024 tokens with 10-20% overlap.
• Add stronger hybrid vector + keyword retrieval.
• Improve reranking.

File ingestion roadmap:

• Preserve tables and section structure where possible.
• Consider background processing for large files.
• Track ingestion status.
• Add OCR/scanned-PDF support later if needed.

Access-control roadmap:

• Harden direct user grant behavior.
• Add more deny/allow tests for group grants and mixed classification/grant cases.
• Separate confidential clearance from admin/view-all behavior.
• Define production-style admin and break-glass rules.
• Support owner/group/role-based access beyond the current prototype grants.
• Keep authorization filtering before vector ranking and before LLM calls.

UI roadmap:

• Add a simple React UI.
• Include user selector for demo users.
• Show document list.
• Upload .txt, .docx, and .pdf documents.
• Select classification.
• Ask RAG questions.
• Show grounded answer.
• Show source cards.
• Show classification badges.
• Show allowed classifications.
• Possibly show audit trail for confidential/admin users.

Current Project Story

AccessAware RAG is a learning-focused but realistic backend prototype for secure RAG.

The current system demonstrates:

server-side TrustedPrincipal authorization boundary
authorized retrieval before generation
classification-aware and grant-aware source filtering
grounded answers
safe fallback behavior
source citations
privacy-conscious auditability
provider/data-egress policy auditing and enforcement
realistic document ingestion
retrieval-quality improvement without weakening access control

The main engineering lesson so far:

Secure RAG is not only about the prompt. The authorization boundary has to exist before retrieval results are sent to the model, and provider/data-egress policy has to decide where authorized content may be processed.