feat(graph): let the caller name the machinery, and stop asking the question twice#529
Merged
Merged
Conversation
`publish.mjs --from` folds the agent cells and rewrites the document, and the `index` block — the cold index build time, one cell per tool per repository — was not carried across. It went out of `graph.json` with the republished grid, and the two charts that read it, `graph-index-build-time` and `graph-time-to-answer`, stopped being generated at all: the site kept serving the stale SVGs left on disk from the last build, which is worse than serving nothing, because nothing is visible. The 32 index cells are restored from the document as it stood before the republication. They are unchanged measurements — the dump path did not move in this release — and the chart count goes back to 74. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XmPDmgakdzQQP9it9HZMen
…e model to restate the question Three things were out of date at once. **The `index` axis was gone from the published document.** `publish.mjs --from` folds the agent cells and rewrites `graph.json`, and it did not carry the `index` block across, so the republished grid dropped the 32 cold-index cells with it. The two charts that read them — `graph-index-build-time` and `graph-time-to-answer` — then stopped being generated at all, and the site kept serving the stale SVGs left on disk, which is worse than serving nothing because nothing is visible. Restored; the chart count goes back to 74. **Every code snippet still told the model to restate the question.** The tool's schema says the opposite, and says why: the tour ranks its seeds against the words in the query, so a rewrite ranks a different answer. The README, the design doc and the blog all carried `restate the code question`, which is the line the server itself stopped saying. They now read as the schema does. **The benchmark section had no charts, and the time axis was unreadable.** Each `###` now carries the chart that makes its claim — shared question, project questions, and the wait before either of them. The time-to-answer chart labels every bar `index / answer` in the order you wait for them, in seconds throughout rather than seconds beside minutes, with the two shades named in the legend; and the repositories no longer run together into one wall, because a group of five bars was 75px tall in a 76px row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XmPDmgakdzQQP9it9HZMen
…p publish dropping the index Two holes, and the second one hid the first. **A graph-arm run that never opens the MCP is not a graph-arm measurement.** `validateArmSample` was an empty function in both drivers. So when GPT-5.6 was asked how RxJS carries a value from `subscribe` through the operators, opened with "I'll trace the subscription path through the repository's implementation", and ran ten PowerShell commands without naming the MCP once — the server mounted, the other seven repositories of the same sweep calling it twice each, the same tool line in the prompt — that run went into the table as a 39% saving with five shell commands against it. Re-run, the same cell called the graph twice, opened no file, and spent 72,174 tokens against the 153,954 it had spent shelling: 39% becomes 72%, and the axis 80% becomes 82%. The retry loop already re-runs a sample it marks `ok: false`. Now a tool arm with zero graph calls is marked that way, in both drivers, so the harness re-runs it instead of publishing a model that did not look as if it were a measurement of the tool. With that cell corrected, `@ttsc/graph` answers **all 64 cells** — four models, two prompt families, eight repositories — with **zero file reads and zero shell commands**, and no cell that skipped the tool. **And `publish.mjs` deleted the `index` axis every time it ran.** It rebuilt the document from `structural` and `agent` alone, so folding agent cells silently dropped the 32 cold-index cells, and the two charts that read them stopped being generated at all — the site kept serving the SVGs left on disk from the last build, which is worse than serving none, because a missing chart is visible and a stale one is not. Every block the document carries now survives a publish that does not write it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XmPDmgakdzQQP9it9HZMen
… cells
`IProps.question` and `ITtscGraphTour.IRequest.query` asked for the same
string two lines apart. A schema that asks twice gets it once: GPT-5.6 sent
`{ "type": "tour" }` with no query in 31 of 32 cells, the validator rejected
the call, and every Codex tour paid a full extra round trip. The duplicate
also lost what it asked for -- `question` keeps the user's backticks and
dotted names, which the mention resolver reads to find the symbol the user
named, and a model paraphrasing into `query` drops them.
The tour now ranks against `props.question`. Every Codex cell is down to a
single call and zero validation failures (was 8 of 8 failing on terra/common).
Re-measured all 64 ttsc-graph cells on the current client:
axis before after
sonnet / common 93% 93%
sonnet / dedicated 82% 83%
opus / common 93% 93%
opus / dedicated 92% 90%
terra / common 80% 89%
terra / dedicated 82% 85%
sol / common 90% 94%
sol / dedicated 91% 93%
…udicate A tour ranks its seeds on the question's words, and a codebase names many things alike: "dependency tracking" matches Vue's devtools hook `onRenderTracked` as readily as `track`, the function that records the dependency, and "request" matches NestJS's message listener as readily as its HTTP router. A tour that lands on the wrong machinery is not a wrong answer -- it is a tour the model then spends calls correcting, and it says so: "the tour surfaced the microservices listener path, not the HTTP router path", then two more traces. The tour request now carries `reinterpretations`: a list of symbol names, never a sentence, naming the machinery the caller expects the answer to be made of. Each name resolves the way a handle does. The ones the graph holds take half the entrypoints; the other half stays with what the graph finds central, untouched by the names, so a caller cannot lose the tour on a bad guess or crowd out the symbol it did not think to name. A name the graph does not know -- or knows several of -- is dropped, so prose costs nothing, and `[]` is the right answer when the question names no machinery at all. Two things the schema could not do on its own. A required field the instruction never mentions is not sent: GPT-5.6 submits the smallest object that could be a request, so eight of eight tours failed validation and were re-sent. And a field whose description opens with prose gets prose -- the same model sent one sentence to a `string[]` until the description opened with "a list of symbol names, not a sentence". A model reads the description, not the type. All 64 cells re-measured, 0 file reads, 0 shell: axis before after sonnet / common 93% 92% sonnet / dedicated 83% 87% opus / common 93% 93% opus / dedicated 90% 91% terra / common 89% 87% terra / dedicated 85% 90% sol / common 94% 96% sol / dedicated 93% 95% Sonnet's Vue tour goes from seven calls to one, Opus's NestJS tour from three to one, and every Codex cell answers in a single call with no validation failures.
samchon
force-pushed
the
fix/graph-benchmark-index-block
branch
from
74917e6 to
fc02332
Compare
samchon
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
The tour asked for the question twice.
IProps.questionandITtscGraphTour.IRequest.querywere the same string, two lines apart, and a schema that asks twice gets it once: GPT-5.6 sent{ "type": "tour" }with no query in 31 of 32 cells, the validator rejected it, and every Codex tour paid a full extra round trip. The tour now ranks againstprops.question.The tour landed on the wrong machinery, and the model paid for it. A tour ranks its seeds on words, and a codebase names many things alike: "dependency tracking" matches Vue's devtools hook
onRenderTrackedas readily astrack, the function that records the dependency, and "request" matches NestJS's message listener as readily as its HTTP router. The model notices — "the tour surfaced the microservices listener path, not the HTTP router path" — and then spends two more traces fetching the router by hand.The tour request now carries
reinterpretations: a list of symbol names, never a sentence, naming the machinery the caller expects the answer to be made of. Each name resolves the way a handle does. The ones the graph holds take half the entrypoints; the other half stays with what the graph finds central, untouched by the names. A name the graph does not know — or knows several of — is dropped, so a wrong guess is free, prose costs nothing, and[]is the right answer when the question names no machinery.Also in this PR: the benchmark harness now rejects a tool arm that never called its tool,
publish.mjsstops dropping the index block, and the charts sit under the claims they support in the README, the design guide, and the blog.Results
All 64 agent cells re-measured on today's clients. 0 file reads, 0 shell commands, and every Codex cell answers in a single call with no validation failures.
Sonnet's Vue tour goes from seven calls to one, Opus's NestJS tour from three to one.
Deferred
Whether the tour should ask the caller for symbol names at all is open. The tour is the request you make when you do not yet know them, and the defect it works around is in the ranking:
runtimeEntryScoregrants +70 for an English verb in the name, so Vue'sonRenderTracked— a devtools hook with one execution edge — outrankstrack, which has twelve. Fixing that is question-only, needs no priors from the model, and may replace this field.Test plan
pnpm -F @ttsc/graph buildandtsc --noEmitclean.tests/test-graphend-to-end MCP case updated for the new tour request.