[feature][ai-layer] Add connector builder LLM eval definitions#55
Closed
sarthak-kapoor-glean wants to merge 7 commits into
Closed
[feature][ai-layer] Add connector builder LLM eval definitions#55sarthak-kapoor-glean wants to merge 7 commits into
sarthak-kapoor-glean wants to merge 7 commits into
Conversation
Add the core connector-builder skill, API exploration skill, and local validation/generation CLI so agents can prepare cited .glean artifacts before writing connector code. Co-authored-by: Cursor <cursoragent@cursor.com>
Make API exploration the first connector-builder phase, add pull and push layer skills, and capture load, auth, and Glean endpoint planning before generation. Co-authored-by: Cursor <cursoragent@cursor.com>
Move artifact creation and code generation back into the skills, keep .glean scoped to each connector folder, and validate confirmed plans plus test and production auth before implementation. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the end-to-end workflow in the top-level connector-builder skill, require an explicit SDK usage choice, and leave supporting skills focused on their local responsibilities. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove datasource-specific examples, clarify SDK usage choices, and require generated connectors to use the SDK pull recipe base classes. Co-authored-by: Cursor <cursoragent@cursor.com>
Add Scio-style trigger and quality eval prompts plus a rubric for credential-free connector-builder skill behavior evaluation. Co-authored-by: Cursor <cursoragent@cursor.com>
Make synthetic evals self-contained with inline API docs and require the Webex eval to fetch public docs or OpenAPI before producing validation-ready artifacts. Co-authored-by: Cursor <cursoragent@cursor.com>
|
can you tell more about the use of this and output? |
Base automatically changed from
feature/connector-builder-core-api-exploration
to
feature/v0-workstream
July 1, 2026 08:23
Author
|
This is not needed. We will plan to have evals in some other way without exposing in SDK user facing code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The connector-builder skills need LLM-backed eval definitions so we can test whether agents actually follow the workflow, not just whether local artifact validation works.
Solution
Validation
python -m json.tool skills/connector-builder/evals/evals.json >/tmp/connector_builder_evals.jsonpython -m json.tool skills/connector-builder/evals/trigger_eval_set.json >/tmp/connector_builder_trigger_evals.jsonnpm run test:pluginsNotes
.github/pull_request_template.md; this body follows the required problem/solution/validation structure manually.Made with Cursor