A completely private, offline-capable AI chat interface where the language model runs entirely in your browser via WebGPU. Nothing leaves your device — no servers, no API keys, no data transmission.
🔗 Live Demo: webgpu-llm-chat.vercel.app 📦 Repository: github.com/Kahaan83/webgpu-llm-chat
Most "AI chat" apps are thin wrappers around a cloud API: every message is a network round-trip, every token costs money, and every conversation passes through a third-party server.
Private AI Chat removes all three constraints. Using Transformers.js v3 and the WebGPU API, the model downloads once, caches in the browser, and runs every subsequent generation locally on the user's own GPU. The result is a ChatGPT-style interface that works fully offline, costs nothing per message, and never sends a single token to a server.
This project was built as a capstone to demonstrate practical skills in edge AI inference, browser-based machine learning, and performance-conscious frontend architecture.
Choose from three quantized, ONNX-optimized instruct models depending on your priorities:
| Model | Size | Best For |
|---|---|---|
| Llama 3.2 1B Instruct | ~1.2 GB | Best quality on WebGPU (q4f16) |
| Qwen 2.5 0.5B Instruct | ~500 MB | Fastest first-token latency, great for demos |
| SmolLM2 1.7B Instruct | ~1 GB | Balanced quality/speed on WASM |
- Auto (WebGPU) — uses the GPU when available for fast, hardware-accelerated inference
- Force WASM (CPU) — runs entirely on CPU, useful for testing or unsupported GPUs
- Standard (q4f16) — default; smallest memory footprint, fastest inference
- Precise (q4 FP32) — higher numerical precision; recommended if Standard mode produces gibberish or freezes, particularly on integrated Intel/AMD graphics chips
On first load, each required file (model.onnx, model_data, tokenizer.json, config.json, etc.) is downloaded with per-file progress bars and live MB/GB counters. Files are cached after the first download — every subsequent load is instant. Downloads can be cancelled mid-progress.
- ChatGPT-style streaming responses, rendered token-by-token as they're generated
- Markdown and fenced code-block rendering with language identifiers
- Quick-start prompt suggestions on a fresh chat
- Persistent chat sidebar with recent conversation history
- Live status indicators: active model, backend (WebGPU/WASM), precision mode, and an "On-device · Zero data sent" badge
A full configuration panel lets users tune model behavior in real time:
| Parameter | Range | Purpose |
|---|---|---|
| System Prompt | free text | Injected at the top of context every turn — controls tone, constraints, and role |
| Temperature | 0.0 – 2.0 | Controls randomness; lower = focused/deterministic, higher = creative/diverse |
| Top P (Nucleus) | 0.1 – 1.0 | Nucleus sampling — restricts vocabulary to the top cumulative-probability mass |
| Max New Tokens | 64 – 2048 | Caps response length per turn; higher values use more GPU memory |
| Repetition Penalty | 1.0 – 1.5 | Discourages repeated words/phrases and prevents infinite looping output (1.1–1.15 recommended) |
All settings apply instantly via "Apply & Close" and can be reset to sensible defaults.
| Layer | Technology |
|---|---|
| Build tool / Framework | Vite + React + TypeScript |
| Styling | Tailwind CSS |
| AI Inference Engine | Transformers.js v3 (device: "webgpu", WASM fallback) |
| Models | Llama-3.2-1B-Instruct, Qwen2.5-0.5B-Instruct, SmolLM2-1.7B-Instruct (ONNX, quantized) |
| Concurrency | Web Workers |
| Deployment | Vercel |
┌─────────────────────────┐ postMessage ┌──────────────────────────┐
│ Main Thread │ ◄─────────────────────────► │ Web Worker │
│ │ │ │
│ • React UI (chat window) │ { type: "load", model, │ • Loads pipeline() │
│ • Config panel │ dtype, device } │ • Detects WebGPU/WASM │
│ • Chat history (sidebar) │ { type: "generate", │ • Applies generation │
│ │ messages, genConfig } │ parameters │
│ │ ◄── { type: "progress" } │ • Streams tokens back │
│ │ ◄── { type: "stream" } │ via TextStreamer │
│ │ ◄── { type: "complete" } │ │
└─────────────────────────┘ └──────────────────────────┘
Web Worker isolation — Model loading and token generation are computationally heavy. Running them off the main thread keeps the UI (scrolling, typing, config panel) responsive even mid-generation.
Singleton pipeline per model — Once a model is loaded, it stays in memory and is reused across messages within the session rather than reloaded per turn.
WebGPU with WASM fallback and dual precision modes — WebGPU isn't universally supported, and even where it is, some integrated GPUs (notably Intel/AMD iGPUs) can produce incoherent output under q4f16. The app exposes this directly to the user: Auto (WebGPU) → Force WASM, and Standard (q4f16) → Precise (q4 FP32), so generation quality issues can be resolved without touching code.
Streaming via TextStreamer — Each generated token chunk is pushed to the UI immediately rather than waiting for the full response — critical for perceived responsiveness, since on-device models are slower than cloud APIs.
Tunable generation parameters — Temperature, Top-P, Max New Tokens, and Repetition Penalty are all exposed in the UI rather than hardcoded. Repetition Penalty in particular addresses a common failure mode of small quantized models: looping/repeating output, which is mitigated with a default of 1.1–1.15.
- Node.js 18+
- A WebGPU-capable browser (Chrome/Edge 113+) for GPU acceleration. Other browsers fall back to WASM automatically — generation will be slower but functional.
git clone https://github.com/Kahaan83/webgpu-llm-chat.git
cd webgpu-llm-chat
npm install
npm run devOpen the local dev URL shown in your terminal. On first load, select a model and click Load Model — this downloads and caches the model weights (~500 MB – 1.2 GB depending on selection). Subsequent loads are instant.
npm run build
npm run preview- If generation produces gibberish or the browser freezes, switch WebGPU Precision Mode to Precise (q4 FP32) — this is especially common on integrated Intel/AMD graphics chips.
- If responses repeat the same phrase in a loop, increase Repetition Penalty to 1.15–1.2.
- For the fastest demo experience, select Qwen 2.5 0.5B Instruct — smallest download, fastest first token.
- For the best overall quality on a discrete GPU, use Llama 3.2 1B Instruct with Standard (q4f16).
- Browser support: Full GPU acceleration requires WebGPU (Chrome/Edge most reliable). Firefox and Safari support is improving but inconsistent.
- Initial load size: Even quantized models are multi-hundred-MB downloads. One-time cost, cached afterward via the browser's Cache API.
- Model capability: Small on-device models (≤2B parameters) are fast and private but noticeably less capable than large cloud-hosted models — a deliberate tradeoff for privacy and zero cost.
- Hardware variance: Output quality on WebGPU can vary by GPU vendor; the Precise (q4 FP32) mode exists specifically to mitigate this on integrated graphics.
- Export/import chat history
- Multi-conversation search
- PWA support for true offline installability
- Additional model options (Phi-3, Gemma)
Kahaan Nirav Shah B.Tech Computer Science & Engineering, Ahmedabad University
This project is licensed under the MIT License — see the LICENSE file for details.