Skip to content

feat(inference): perplexity CLI line-oriented JSON event output#200

Merged
ohdearquant merged 2 commits into
mainfrom
pr/eng-4-eval-ppl-events
Jun 22, 2026
Merged

feat(inference): perplexity CLI line-oriented JSON event output#200
ohdearquant merged 2 commits into
mainfrom
pr/eng-4-eval-ppl-events

Conversation

@ohdearquant

@ohdearquant ohdearquant commented Jun 22, 2026

Copy link
Copy Markdown
Owner

What

eval_perplexity emits a line-oriented @@lattice perplexity event the app can parse.

Why

The Models screen's perplexity readout consumes structured stdout. This adds a single JSON event line ("ev":"perplexity", with the score) alongside the human-readable output. Auto-discovered binary, so it is already covered by cargo build --workspace.

Files

  • crates/inference/src/bin/eval_perplexity.rs (+45)

Verification

cargo build --release -p lattice-inference --bin eval_perplexity --features f16,metal-gpu clean. Built green in the integrated-tree gate.

Bench

Binary-only, additive output. make bench-compare's comparator errored assembling the delta (known two-worktree fragility; base benches ran clean) — bench-neutral by construction, as the changed code is off every Criterion-measured path.

Series

Part of the PR #193 engine-slice (finest split). All engine code lands on main; the macOS app surfaces a subset (Models + Chat) for v0.0.1.

@github-actions

github-actions Bot commented Jun 22, 2026

Copy link
Copy Markdown

E2E Parity Report

PASS: all 3 prompts match within first 3 tokens

Prompt Agreement First Diff HF tok/s Lattice tok/s Verdict
The capital of France is 3/15 pos 3 0.2 1.6 PASS
In the year 2024, artificial intelligence 10/15 pos 9 0.3 1.6 PASS
`def fibonacci(n):
if n <= 1:
    return n
return` | 15/15 | none | 0.3 | 1.0 | PASS |

The capital of France is

  • HF: Paris.
    The capital of France is Paris.
    The capital of France
  • Lattice: Paris.
    A: Yes, the capital of France is Paris.

In the year 2024, artificial intelligence

  • HF: (AI) has become a significant part of the global economy. It is
  • Lattice: (AI) has become a significant part of our daily lives. From personal

def fibonacci(n): if n <= 1: return n return

  • HF: fibonacci(n-1) + fibonacci(n-2)

print(fib

  • Lattice: fibonacci(n-1) + fibonacci(n-2)

print(fib

@ohdearquant ohdearquant enabled auto-merge (squash) June 22, 2026 23:21
@ohdearquant ohdearquant merged commit 758b3e6 into main Jun 22, 2026
10 checks passed
@ohdearquant ohdearquant deleted the pr/eng-4-eval-ppl-events branch June 22, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant