FlashSpec v0.1.4

FlashSpec v0.1.4

This release commits the first real hardware results for FlashSpec,
measured on Google Colab (Tesla T4) using TinyLlama-1.1B-Chat at 4-bit
NF4 quantization. All notebooks now contain actual execution outputs,
not placeholders.

---

What was measured

Inference throughput — TinyLlama-1.1B-Chat · NF4 · Tesla T4 · γ=4

Metric	Value
Throughput	44.2 tok/s
Token acceptance rate (α)	0.75
p50 step latency	22.1 ms

Full result: benchmarks/results/flashspec_ucb_tiny_llama.json

---

Bandit regret — UCB1 vs Thompson vs Oracle (T=10,000, K=3 arms)

Selector	Cumulative regret at T=10,000	vs theory bound (~526)
UCB1	100.2	5.2× under bound ✓
Thompson	18.9	27.8× under bound ✓
Oracle	0.0	—

Both UCB1 and Thompson empirically satisfy the O(√(KT log T)) bound.

Source: notebooks/02_bandit_analysis.ipynb

---

Triton kernel profiling — Tesla T4

Shape	Reference (ms)	Triton (ms)	Speedup
B=1 γ=4 V=32k	0.151	0.785	0.2×
B=8 γ=4 V=32k	0.717	0.459	1.6×
B=1 γ=8 V=32k	0.231	1.248	0.2×
B=32 γ=4 V=32k	0.091	0.222	0.4×

The Triton kernel is slower than the pure-PyTorch reference at batch=1
on T4. This is expected: T4's HBM bandwidth (320 GB/s) is 10× lower than
H100 SXM5 (3.35 TB/s), and Triton's kernel-launch overhead dominates at
small batch sizes. The kernel shows a 1.6× speedup at batch=8, which is
where it becomes memory-bandwidth-bound. H100 profiling is next.

Source: notebooks/03_kernel_profiling.ipynb

---

Gamma sweep (γ ∈ {1, 2, 4, 8, 16}) — acceptance rates genuine; throughput reflects sampling kernel only

γ	α (mean)
1	0.720
2	0.680
4	0.690
8	0.748
16	0.714

Full data: benchmarks/results/gamma_sweep.csv

---

What this release does not contain

To be clear about what has and has not been measured:

• The Llama-3-8B/70B throughput numbers in the README (142 tok/s, 2.31×
  speedup vs AR) are design targets, not measured results. They are
  marked ⊛ throughout. H100 runs with full model weights are the next step.
• The vanilla AR baseline was not measured in this run, so speedup ratios
  cannot be computed yet.
• The draft-size sweep uses simulated (toy) logprobs, not a real model.

---

Installation

# Cross-platform (Windows, macOS, Linux) — pure-PyTorch reference kernels:
pip install flashspec==0.1.4

# Linux + CUDA — GPU-accelerated Triton kernels:
pip install flashspec[gpu]==0.1.4

Users on Windows or macOS who had pip install flashspec fail in v0.1.0–v0.1.2
(due to triton>=3.0.0 being an unsatisfiable dependency on those platforms)
should upgrade to this release. See the
v0.1.3 notes
for the full explanation of that fix.

---

Full changelog

See CHANGELOG.md for the complete history.

v0.1.3 - Triton Dependency Fix & Public API Update

v0.1.3 - Triton Dependency Fix & Public API Update

🚀 Summary

This release resolves a critical installation issue on non‑Linux platforms caused by Triton being a hard dependency. It also improves the developer experience by exposing TRITON_AVAILABLE at the package root and updating documentation.

🔧 Changes

• Dependency fix:

  • Removed triton>=3.0.0 from core dependencies.
  • Added optional extra gpu with platform marker (Linux only).
  • Result: pip install flashspec works everywhere; pip install flashspec[gpu] installs Triton on Linux.

• Graceful degradation:

  • Triton kernels wrapped in try/except.
  • Stub functions raise clear ImportError only when called.
  • Added TRITON_AVAILABLE: bool flag.

• Public API:

  • TRITON_AVAILABLE exported from flashspec/__init__.py.
  • Users can check runtime capability directly.

• Tests:

  • Added CPU‑only tests verifying import success, stub behavior, and error messages when Triton is absent.

• CI workflows:

  • Updated pip install steps to request [dev,gpu] / [bench,gpu] where Triton kernels are tested.
  • Ensures GPU tests run only on Linux.

• Docs & metadata:

  • README.md quickstart updated with platform‑specific install notes.
  • CHANGELOG.md documents the fix under [0.1.3].
  • CITATION.cff version bumped to 0.1.3.

⚠️ Important

• Versions 0.1.0–0.1.2 are yanked on PyPI due to broken Triton dependency.
• Please upgrade to 0.1.3 immediately.

📦 Installation

• Cross‑platform (no GPU):

  pip install flashspec

• Linux + CUDA (GPU kernels):

  pip install flashspec[gpu]

FlashSpec v0.1.1 - Audit Compliance Fixes

Highlights

• Fixed type-ignore annotations in verifier.py to satisfy audit script
• Added missing BibTeX entry (mattral2025flashspec) for paper compliance
• All documentation validated against spec - zero failures

Verification

• 84/84 required files present
• 31 Python files syntax-clean
• All tests passing locally

Next Steps

• Run GPU benchmarks on H100/A100
• Generate paper figures and compile PDF
• Submit to arXiv/JOSS

FlashSpec v0.1.0 - Initial PyPI Release

Highlights

• First public release of FlashSpec on PyPI
• Includes adaptive speculative decoding engine
• Triton-optimised verification kernels
• Online bandit draft selection (UCB1, Thompson, Oracle)
• Full test suite passing locally

Installation

pip install flashspec