Skip to content

Experiments to diagnose /research endpoint topic-relevance bug - #5

Draft
SynthLuvr with Copilot wants to merge 1 commit into
mainfrom
copilot/debug-research-responses
Draft

Experiments to diagnose /research endpoint topic-relevance bug#5
SynthLuvr with Copilot wants to merge 1 commit into
mainfrom
copilot/debug-research-responses

Conversation

Copilot AI commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

The /research endpoint returns content unrelated to the requested topic. Manual probing confirmed the code is calling the endpoint correctly (GET /research?topic=<value> with X-API-Key) — the bug is in the endpoint itself.

Experiments (research.experiment.ts)

Four vitest tests against the live API to characterize the failure mode:

  • Consistency — same topic called 5×; measures what fraction of responses are on-topic
  • Specificity — three distinct topics; checks whether responses differ per topic (endpoint actually uses topic param vs. ignoring it)
  • Keyword check — full response printed for solar energy; asserts at least one expected keyword is present
  • Empty rate — tracks how often the endpoint returns an empty/near-empty response across 5 calls

Wiring

  • package.jsontest:experiments script runs only the experiment file
  • ci.yml — new step runs test:experiments with ELYOS_API_KEY and continue-on-error: true; results are visible in check logs regardless of pass/fail

Co-authored-by: SynthLuvr <131367121+SynthLuvr@users.noreply.github.com>
Copilot AI changed the title Add /research endpoint experiments to investigate topic-relevance bug Experiments to diagnose /research endpoint topic-relevance bug Jun 12, 2026
Copilot AI requested a review from SynthLuvr June 12, 2026 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants