Skip to content

fix(skills): align search guidance with hybrid retrieval and teach answer-card phrasing - #29

Open
vraspar wants to merge 1 commit into
mainfrom
feat/skill-hybrid-harmony
Open

fix(skills): align search guidance with hybrid retrieval and teach answer-card phrasing#29
vraspar wants to merge 1 commit into
mainfrom
feat/skill-hybrid-harmony

Conversation

@vraspar

@vraspar vraspar commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Retrieval went hybrid in prod (lexical + dense, RRF-fused, calibration hybrid-v1), but the vendored skills still taught the old model. tenjin-search told agents "search is lexical, not semantic: it matches words, not meaning", which pushes exactly the wrong behavior: compressing a question into keywords throws away the signal the dense leg runs on. This is the CLI-side half of guidance parity with the server docs in BackTrackCo/tenjin#464.

What changed:

  • skills/tenjin-search/SKILL.md: lookup matches wording and meaning; send the complete question as one natural-language sentence, do not compress to keywords. Privacy hygiene (strip private identifiers, internal service names, secrets, do not search if it cannot be generalized) and the MISS-is-fine rule are untouched.
  • Demand-to-supply loop: the candidate-parking example now passes --question "<the question you looked up>". Verified in src/commands/candidate.ts (stores it in the candidate meta, capped at the server's 200-char bound) and src/commands/publish.ts (prefills it into the card's questionsAnswered as a fallback, still losing to an explicit --question or frontmatter). So the exact phrasing a searcher used becomes a match target on the published card.
  • skills/tenjin-publish/SKILL.md: the answer-card draft rule was one vague sentence. It now carries the same phrasing guidance as the server docs: 5 to 10 questionsAnswered entries at 200 characters max, varied in register (natural symptom sentence, terse keyword line or verbatim error string, why/how question), same-register rephrasings add nothing, tasksSupported is for tasks and not questions, and scope is embedded too so write it dense and factual.
  • README.md: the lookup paragraph said "derive the smallest public phrasing of your task", which is the same compress-to-keywords advice. Now: strip the private parts, then send what is left as one complete sentence.
  • Patch changeset, since skills ship inside the package.

On the vendored zero-install skill (skills/tenjin/SKILL.md): it is byte-identical to live tenjin.blog/skills.md, so pnpm sync:skill produces zero diff, and it already carries the Resource card section, the lookup and submit_feedback MCP tools, and preview in the discovery search description. Commit b3e91d1 (#25) resynced it on 2026-07-24. One stale claim rides along: line 103 still describes lookup matching as "honest lexical, not a semantic score", so this package ships one skill saying lookup matches wording and meaning and another saying it does not. That line cannot be corrected in this repo without breaking skill-drift CI, since the file is a mirror. It gets fixed upstream under BackTrackCo/tenjin#464, and this repo picks it up on the next pnpm sync:skill after that ships. src/fixtures/openapi.fixture.json has the same shape of residual: it is a vendored server contract snapshot consumed by src/contract.test.ts and still pins calibration to a const lexical-v1, and it refreshes on the next contract sync, not here.

On why the earlier drift sat unnoticed: skill-drift.yml is configured correctly (daily schedule, workflow_dispatch, and a PR trigger on skills/** + scripts/sync-skill.mjs) and it did catch the drift. Scheduled runs on main failed on 2026-07-20 and 2026-07-21. The gap is alerting, not detection: a scheduled failure on main gates no PR, opens no issue, and only emails the workflow file's last committer, so main stayed red for days. Adding a failure-path step that opens a tracking issue needs issues: write on the job, so it is left for a follow-up rather than escalating workflow permissions in a docs PR.

No lookup to search renaming here, to stay out of BackTrackCo/tenjin#463's way.

Gates: typecheck, lint, prettier check, and the full vitest suite (709 passed, 9 skipped) all green.

🤖 Generated with Claude Code

… phrasing

Retrieval is hybrid (lexical + dense, RRF-fused), so the tenjin-search skill's
"lexical, not semantic" line was wrong guidance: it pushed agents to compress a
question into keywords, which is exactly what discards the meaning signal. It
now says lookup matches wording and meaning and to send the whole question as
one natural-language sentence. Privacy hygiene and the MISS-is-fine rule are
unchanged.

Closes the demand-to-supply loop: the candidate-parking example passes
--question, which publish prefills into the answer card's questionsAnswered, so
a searcher's phrasing becomes a match target on the published card.

tenjin-publish gets the card-phrasing rules the server docs carry: 5 to 10
questionsAnswered entries at 200 characters, varied register, tasks kept out of
questionsAnswered, and a dense factual scope because scope is embedded too. The
README lookup paragraph matches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant