A local AI agent with 16K context on 8GB RAM. Runs entirely offline on Apple Silicon — no cloud, no API keys, no subscriptions.

Built by combining 12 research papers into one practical system.

$ python consilium.py

  CONSILIUM

You: What is 56567 + 76678?
Consilium (3.2s): 133245

You: /rlm
  RLM: ON — agent mode (code + search)

[RLM] You: Search: what is TurboQuant by Google?
Consilium (12s, 3 calls): TurboQuant is Google's KV cache compression
  algorithm achieving 6x memory reduction with zero quality loss...

[RLM] You: Is 997 prime? Verify it.
Consilium (5s): Yes — verified by testing all divisors up to 31.

• 29 tok/s generation on Apple Silicon via MLX
• 4-bit weight quantization (2.2 GB model, 97% quality of FP16)
• 2-bit TurboQuant KV cache (8x compression, 16K context on 8GB)
• NoWait inference — skips unnecessary thinking tokens for 4-8x faster responses

• Model writes Python code to solve problems — exact math, real web search
• Sandboxed execution with file-bridge IPC (no shared memory needed)
• Smart routing: math/search goes through code execution, everything else is direct
• Scores 80/80 on our benchmark vs standard mode's 50/80

• Autonomous research loop: searches arxiv, generates experiments, runs them locally
• Config patching: findings modify engine behavior (temperature, token limits, prompts)
• Knowledge base persists across sessions — the system learns over time

• Self-modifying code agent inspired by Meta's HyperAgents
• Writes code, tests it, improves it, improves HOW it improves (meta-learning)
• State persists to disk — picks up where it left off

• Graceful OOM recovery: catches Metal GPU memory errors, retries with 40% fewer tokens
• Never crashes the CLI — always returns a response or helpful error message
• Keyboard interrupt during generation cancels cleanly without losing session

• macOS with Apple Silicon (M1/M2/M3/M4)
• 8 GB RAM minimum
• Python 3.10+
• 3 GB free disk space

git clone https://github.com/YOUR_USERNAME/consilium-ai.git
cd consilium-ai

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Download model (2.2 GB, one-time)
python setup.py

python consilium.py

Qwen3.5-4B fine-tuned on 14,000 Claude 4.6 Opus reasoning traces by Jackrong. The model uses a hybrid DeltaNet + Transformer architecture where 24 of 32 layers use linear attention (constant memory), and only 8 layers use full attention with KV cache.

Quantized to 4-bit using MLX — Apple's framework optimized for unified memory on Apple Silicon.

Each of the 4 billion parameters is compressed from 16 bits to 4 bits:

FP16:  4B params x 2 bytes = 8 GB (doesn't fit on 8GB)
4-bit: 4B params x 0.5 bytes = 2 GB (fits with room to spare)

Group quantization with group_size=64 preserves ~97% of original quality. Each group of 64 weights shares a scale factor and zero-point, with individual weights mapped to one of 16 levels (4 bits).

Based on Google's TurboQuant (ICLR 2026). During generation, the model stores Key and Value tensors for all previous tokens. Without compression, this grows to hundreds of MB and causes OOM.

TurboQuant compresses KV cache from 16 bits to 2 bits per value:

1. Hadamard Transform — rotates KV vectors to normalize distribution
2. Lloyd-Max Quantization — maps to 4 optimal centroids (2-bit)
3. Attention Sinks — first 64 tokens stay in FP16 (system prompt)

Result: 8x compression, 16K context on 8GB RAM with near-zero quality loss.

We use turboquant_mlx which monkey-patches MLX's KV cache class automatically.

Based on NoWait (2025). The Claude-distilled model generates hundreds of <think> tokens before answering. We inject </think> at the end of the prompt, forcing the model to skip thinking and answer directly.

Before: 2000 thinking tokens + 200 answer tokens = 40 seconds
After:  0 thinking tokens + 200 answer tokens = 7 seconds

27-51% fewer tokens with no quality loss (in some cases, accuracy improves).

Based on PrimeIntellect's RLM research (2026). Instead of answering directly, the model writes Python code to solve problems.

Standard: "56567 + 76678" → model does column math → might get it wrong
RLM:      "56567 + 76678" → Python: 133245 → always correct

The code runs in a sandboxed subprocess. When the sandbox needs to call the model (e.g., llm("Is this correct?")), it uses a file bridge:

Sandbox writes:  /tmp/rlm_xyz/request.json
Main process:    Reads request → calls MLX model → writes response
Sandbox reads:   /tmp/rlm_xyz/response.txt

No shared memory, no IPC complexity. Works on any system.

Not everything benefits from code execution. The router decides automatically:

Each /improve cycle:

1. Search — finds new techniques on arxiv via DuckDuckGo
2. Pick — selects the most promising technique (avoids repeats)
3. Experiment — generates Python code to test it
4. Run — executes on your machine, measures results
5. Evaluate — scores the finding (1-10)
6. Patch — if score is high, generates a config patch:

{"temperature_math": 0.1, "system_prompt_suffix": "Verify all calculations."}

The engine reads data/learned_config.json on every request. Findings accumulate over time — the system gets measurably better.

Apple Silicon has limited unified memory. When the Metal GPU runs out:

1. Catch the Insufficient Memory error
2. Clear Metal GPU cache + garbage collect
3. Reduce max_tokens by 40%
4. Retry (up to 3 attempts)
5. If all retries fail, return helpful error message instead of crashing

Tested on MacBook Air M2 with 8GB RAM.

16K context achieved via 2-bit TurboQuant KV cache compression on 8GB RAM.

Sustained: 29 tokens/sec
Peak memory: 2.5 GB
1,500 tokens in 52 seconds

User input
    |
consilium.py (CLI)
    |
    +-- /rlm toggle --> recursive_lm.py
    |                     |-- Smart routing (math/search vs direct)
    |                     |-- Python sandbox execution
    |                     |-- File bridge for llm() calls
    |                     +-- Web search via DuckDuckGo
    |
    +-- fast_mlx.py (inference engine)
          |-- MLX 4-bit model loading
          |-- TurboQuant 2-bit KV cache (automatic)
          |-- NoWait </think> injection
          |-- Learned config from self-improvement
          |-- OOM recovery with retry
          |-- Thread-safe generation lock
          |
          +-- Qwen3.5-4B Claude Opus Distilled (MLX)

consilium-ai/
|-- consilium.py          # CLI entry point
|-- setup.py              # Model downloader
|-- requirements.txt      # Dependencies
|-- LICENSE               # MIT
|-- engine/
|   |-- fast_mlx.py       # MLX inference + TurboQuant + NoWait + OOM recovery
|   |-- recursive_lm.py   # RLM agent with file bridge
|   |-- length_predict.py  # Output length estimation
|   |-- hyper_agent.py    # Self-modifying code agent
|   |-- self_improve.py   # Autonomous research loop
|   +-- turbo_kv.py       # TurboQuant compression math
|-- tests/
|   +-- test_engine.py    # 13 tests
|-- data/
|   +-- .gitkeep          # Runtime data (gitignored)
+-- docs/
    |-- consilium_book.md  # Full technical guide
    +-- consilium_book.pdf # PDF version

Every optimization is based on peer-reviewed research:

This project was built in a single session, starting from "can you compress a model for 4GB RAM?" and ending with a 16K-context AI agent with self-improvement capabilities.

1. Model compression — researched GGUF, bitsandbytes, MLX quantization. Settled on MLX 4-bit for Apple Silicon speed.

2. Model selection — tested DeepSeek-R1 + Qwen2.5 merge (worked but had arithmetic errors), then switched to Jackrong's Claude Opus distilled Qwen3.5-4B (better quality, properly fine-tuned).

3. KV cache compression — implemented TurboQuant from the arXiv paper, tested 3-bit and 2-bit, landed on 2-bit (8x compression, 16K context, quality preserved).

4. NoWait inference — discovered that injecting </think> makes Claude-distilled models skip internal monologue. One line of code, 4-8x faster responses.

5. RLM agent — built a recursive system where the model writes Python code in a sandbox. Invented a file-bridge IPC mechanism for the sandbox to call back into the model.

6. Self-improvement — created a loop that searches arxiv, generates experiments, runs them locally, and patches the engine config with findings.

7. Polish — added OOM recovery, smart RLM routing, error handling, and a professional CLI.

• Speed is hardware-limited. On 100 GB/s memory bandwidth, a 4B model generates at ~29 tok/s. No software trick changes this.
• Context is compression-limited. 2-bit TurboQuant KV gave us 16K context where others get 4-8K on the same hardware.
• Small models need tools. A 4B model makes arithmetic errors. Giving it a Python sandbox (RLM) fixes this completely.
• Simpler prompts work better on small models. The CodePlan-style verbose prompts hurt performance. Short, direct instructions win.
• Speculative decoding hurts small models. Tested with 0.8B draft model — overhead exceeded savings. Only helps with 32B+ models.

• Jackrong — Claude Opus reasoning distillation into Qwen3.5
• helgklaizar — TurboQuant MLX implementation
• Apple MLX team — ML framework for Apple Silicon
• Alibaba Qwen team — Qwen3.5 base architecture
• PrimeIntellect — Recursive Language Model research
• Meta FAIR — HyperAgents research
• Google Research — TurboQuant algorithm

MIT — see LICENSE.

PRs welcome. Run tests before submitting:

python -m pytest tests/ -v