Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
42 lines (34 loc) · 1.91 KB

ARCHITECTURE.md

File metadata and controls

42 lines (34 loc) · 1.91 KB
Raw

System Architecture

Distill is built as a modular system designed for high-throughput prompt compression. The architecture is split between a high-performance Python backend and a modern React frontend for visualization.

High-Level Flow

  1. Ingestion: The system receives a raw text prompt or a list of context strings.
  2. Preprocessing: Text is chunked based on maximum sequence lengths and special "anchor" tokens (e.g., newlines).
  3. Inference: A BERT-based Token Classification model evaluates the "Information Density" of every token.
  4. Compression Pipeline:
    • Context Filtering: Entire irrelevant chunks are discarded.
    • Token Pruning: Individual tokens within chunks are removed based on percentile thresholds.
  5. Reconstruction: The remaining tokens are re-joined into a coherent (though "lossy") prompt.
  6. Delivery: The compressed prompt is sent to a target LLM (like GPT-4), or returned via API.

Component Map

Backend (/backend)

  • Core Engine: The distill package contains the logic for inference and pruning.
  • API Layer: FastAPI provides endpoints for remote compression services.
  • Models: Local storage for the fine-tuned BERT weights and configuration.

Frontend (/frontend)

  • Dashboard: A React application that visualizes compression ratios, latency improvements, and cost savings.
  • Demo: An interactive interface to test the compression algorithm in real-time.

Data Flow Diagram

flowchart TD
    User[User Input] --> API[API / CLI]
    API --> Chunking[Chunking Logic]
    Tiktoken[Tiktoken] --> Chunking
    Chunking --> BERT[BERT Inference]
    Weights[Local Weights] --> BERT
    BERT --> Filtering[Two-Tier Filtering]
    Filtering --> Context[Context Filter]
    Filtering --> Token[Token Filter]
    Context --> Assembly[Prompt Assembly]
    Token --> Assembly
    Assembly --> Output[Target LLM / Output]
Loading