🌌 N-Grain AI

A Premium Production-Ready NLP Autocomplete System

Intelligent next-word prediction powered by statistical 1-to-6 gram language models, Add-k smoothing, and a stunning glassmorphism frontend.

---

🌟 Overview

N-Grain AI is an advanced Natural Language Processing system that predicts the next word a user types. The system is split into two robust components:

1. Python / FastAPI Backend: Processes raw datasets (like Twitter's Sentiment140), builds n-gram statistical language models (unigram through 6-gram), calculates K-smoothing, evaluates perplexity, and exposes high-performance REST APIs.
2. Next.js / React Frontend: A stunning, dark-mode first UI modeled after premium AI SaaS platforms. It includes a live Playground for testing autocomplete, a fully animated data Pipeline visualization, and an Analytics Dashboard displaying real-time metrics.

---

🚀 Quick Start Guide

Follow these instructions to get both the backend engine and frontend client running on your local machine.

1. Backend Setup (FastAPI)

The backend loads the dataset, trains the NLP models, and exposes the API.

# Clone the repository and navigate to the project directory
cd f:\Work\Auto-Complete-System-NLP-

# Install required Python packages (assuming you have a virtual environment)
pip install fastapi uvicorn pandas nltk

# Run the FastAPI server with hot-reload
python -m uvicorn app.main:app --reload

The API will be available at http://127.0.0.1:8000

2. Frontend Setup (Next.js)

The frontend connects to the backend to provide the interactive user interface.

# Open a NEW terminal and navigate to the frontend directory
cd frontend

# Install Node.js dependencies
npm install

# Start the Next.js development server
npm run dev

The Web App will be available at http://localhost:3000

---

🧠 System Architecture

graph TD
    subgraph Frontend [Next.js 15 Frontend]
        UI[User Interface] --> Playground[Playground]
        UI --> Analytics[Dashboard]
        UI --> PipelineUI[Pipeline UI]
    end

    subgraph Backend [FastAPI Backend]
        Router[API Router] --> AC(Autocomplete Service)
        Router --> PER(Perplexity Calculator)
        Router --> MAT(Matrix Generator)
        Router --> STAT(Analytics Aggregator)
        
        AC --> NGM[N-Gram Models]
        PER --> NGM
        MAT --> NGM
        STAT --> NGM
        
        NGM --> PREP(Preprocessing Service)
        PREP --> CSV[(Twitter CSV Dataset)]
    end

    Playground <-->|POST /autocomplete| Router
    Analytics <-->|GET /stats & /ngrams| Router

Loading

---

📂 Project Structure

Auto-Complete-System-NLP-/
├── app/                        # 🧠 Python FastAPI Backend
│   ├── api/                    # API Routing and Endpoints
│   ├── core/                   # System Configuration
│   ├── models/                 # Core NLP Logic & Mathematics
│   ├── schemas/                # Pydantic Data Validation Models
│   ├── services/               # Orchestration & Data Pipeline Logic
│   ├── utils/                  # Helper Utilities
│   ├── main.py                 # FastAPI Application Entrypoint
│   └── training...csv          # Source Dataset
│
└── frontend/                   # 🎨 Next.js React Frontend
    ├── src/
    │   ├── app/                # Next.js App Router (Pages & Layouts)
    │   ├── components/         # React UI Components
    │   ├── hooks/              # Custom React Hooks
    │   ├── lib/                # Utilities (Tailwind merge, etc.)
    │   ├── services/           # Axios API Client
    │   ├── styles/             # Global CSS & Glassmorphism Theme
    │   └── types/              # TypeScript Interfaces
    ├── public/                 # Static Assets
    ├── tailwind.config.ts      # Tailwind Configuration
    └── package.json            # Node.js Dependencies

---

🔬 Deep Dive: Backend Files

The app/ directory is the brain of N-Grain AI. Here is exactly what every file does:

1. Core Entrypoint

• app/main.py

  Initializes the FastAPI application, configures CORS middleware (allowing the frontend to communicate with it), and registers all the API routers.

2. API Endpoints (app/api/v1/endpoints/)

• autocomplete.py: Exposes POST /autocomplete. Takes user input text and an optional prefix, and returns a sorted list of the top word suggestions and their probabilities.
• perplexity.py: Exposes POST /perplexity. Takes a sentence and calculates how "confused" the language model is by it (lower perplexity = better prediction).
• matrices.py: Exposes endpoints for GET /count_matrix and GET /probability_matrix to peek into the underlying bigram transition probabilities.
• analytics.py: Calculates and exposes real-time statistics (GET /stats) like vocabulary size, train/test split samples, unknown word counts, and average perplexity. Also exposes (GET /ngrams) to extract N-Gram frequency arrays for the frontend charts.

3. Services (app/services/)

• preprocessing_service.py

  The data ingestion pipeline. It loads the CSV using Pandas, splits text into sentences, tokenizes using NLTK, performs an 80/20 train/test split, calculates word frequencies, and replaces rare Out-Of-Vocabulary (OOV) words with an <unk> token based on a strict threshold.

• autocomplete_service.py

  The orchestration layer. It bridges the preprocessing with the mathematical models. It initializes and holds the 1-to-6 gram models in memory. It contains the core logic for merging suggestions from higher-order models down to lower-order models (backoff strategy) and sorting them by confidence.

4. NLP Core Models (app/models/)

• ngram_model.py

  The mathematical heart of the project. Contains pure Python logic to:

  • count_n_grams(): Extract n-gram tuples from tokenized sentences.
  • estimate_probability(): Applies Add-k smoothing to calculate $P(word|context)$.
  • suggest_words(): Uses the probabilities to find the most likely next word.
  • calculate_perplexity(): Evaluates the geometric mean of the inverse probabilities.

5. Utilities & Configuration (app/utils/ & app/core/)

• utils/text_cleaner.py: Contains regex logic to sanitize raw tweets (removing URLs, mentions, hashtags, and special characters).
• core/config.py: Centralized configuration defining the path to the CSV dataset (DATASET_PATH) and the vocabulary threshold limit.
• schemas/request_schema.py: Pydantic models to ensure the API receives valid, strongly-typed JSON bodies from the frontend.

---

🎨 Deep Dive: Frontend Architecture

The frontend is a Next.js 15 application employing the App Router, constructed with TypeScript, Tailwind CSS, Framer Motion (for animations), Recharts (for data visualization), and customized shadcn/ui components.

Key Pages

1. Landing Page (/): A highly polished, scroll-animated introduction to the system featuring a mock system architecture diagram and live metrics.
2. Playground (/playground): A specialized IDE-like interface for testing the model. As you type in the center console, the application debounces API calls to the backend, rendering real-time suggestions, probabilities, and confidence bars.
3. Analytics Dashboard (/analytics): Connects to the analytics.py backend router to render dynamic charts. It visualizes N-Gram distributions, word occurrences, model perplexity curves, and confidence distributions using Recharts.
4. Pipeline Visualization (/pipeline): An interactive, vertical timeline breaking down all 11 stages of the NLP data pipeline (from CSV parsing to final Autocomplete delivery).

Design System

The UI utilizes a custom CSS variables system (src/styles/globals.css) specifically designed to achieve a Glassmorphism aesthetic. This includes .glass-card utilities, vibrant purple/cyan gradients, soft inset shadows, and frosted backdrop blurs (backdrop-blur-xl) that make the application feel like a premium SaaS product.

---

📡 API Documentation

If you wish to interact with the backend programmatically, you can hit the following endpoints at http://127.0.0.1:8000/api/v1/:

Endpoint	Method	Body / Params	Description
/autocomplete	POST	{"text": "hello", "start_with": "w"}	Returns the top 5 next word predictions based on the 1-6 gram models.
/perplexity	POST	{"text": "hello world"}	Calculates the perplexity score of the provided sentence.
/stats	GET	None	Returns real-time metrics (Vocab Size, Train/Test sample counts, Unk words).
/ngrams	GET	None	Returns frequency arrays for n-gram distributions to power frontend charts.
/performance	GET	None	Returns comparative model performance (accuracy, coverage, perplexity).

(A complete, interactive Swagger/OpenAPI documentation is automatically generated by FastAPI and available at http://127.0.0.1:8000/docs)

---

👨‍💻 Complete User Guide

N-Grain AI features an intuitive, modern interface. You can navigate between pages using the top glassmorphism Navbar or by pressing Ctrl + K (or Cmd + K) anywhere to open the global Command Palette.

1. Landing Page (/)

The entry point of the application. It provides a high-level overview of the system architecture, the technology stack, and real-time live metrics (such as the exact vocabulary size loaded from your CSV dataset).

2. The AI Playground (/playground)

This is the core interactive interface where you can test the NLP models in real-time. It is divided into three distinct panels:

• Left Panel (Input & History)
  • Input Area: Type your base sentence here.
  • Prefix Filter: Optionally force the engine to only suggest words starting with a specific letter or prefix.
  • Recent Prompts: Quickly click past inputs to re-test them.
• Center Panel (Live Editor)
  • As you type or click "Suggest", the engine runs the backoff strategy.
  • The top 5 word suggestions are displayed inline as interactive ghost-text bars with stunning gradient confidence indicators.
  • Click on any suggestion to instantly append it to your input.
• Right Panel (Engine Controls)
  • Model Selection: Toggle specific n-gram models on or off (e.g., disable 6-gram to see how the engine relies on lower-order context).
  • Smoothing Slider: Dynamically adjust the Add-K smoothing parameter to see how probability distributes when encountering rare or unknown words.
  • Top Prediction: A dedicated, highlighted card showing the absolute highest-confidence prediction and its exact mathematical probability.

3. Analytics Dashboard (/analytics)

A comprehensive view of the dataset and the mathematical health of the language models. Here is what each diagram represents:

• Top Stat Cards: Live counts of your training/testing samples, your total vocabulary size, the number of words marked as <unk> (Unknown), and the overall system Perplexity.
• N-Gram Frequency Chart (Bar Chart): Visualizes the exponential growth of possible combinations. It plots the raw count of unique unigrams, bigrams, trigrams, up to 6-grams generated from your dataset.
• Word Occurrence Histogram (Bar Chart): Displays the top 15 most frequently used words in your dataset (excluding <unk> and padding tokens).
• Perplexity Comparison (Line Chart): Illustrates how model "confusion" decreases. Typically, as you move from Unigram to 6-gram, the perplexity score drops significantly, meaning the model is more certain of its predictions.
• Model Performance Comparison (Bar Chart): Visualizes the classic NLP trade-off between Accuracy and Coverage. High-order models (like 6-grams) are highly accurate but have low coverage (they often can't find a matching 6-word history). Unigrams have 100% coverage but low contextual accuracy.
• Bigram Probability Heatmap: A color-coded matrix showing the transition probability $P(w_i|w_{i-1})$ from one word context to the next.
• Confidence Distribution (Area Chart): Plots the frequency of the engine's confidence scores across predictions, showing if the model generally leans towards high-certainty or low-certainty guesses.

4. Interactive Pipeline (/pipeline)

If you want to understand how the models were built, visit this page. It is a vertical, interactive timeline breaking down all 11 stages of the NLP data pipeline.

• Scroll through the stages chronologically (from CSV Ingestion to Tokenization to Smoothing).
• Click on any individual node to expand a detailed panel explaining the exact Python logic and mathematics executed during that step.

---

Built with ❤️ using Python and TypeScript