Skip to content

feat(workflows): Templates gallery + PR code-review automation#75

Draft
xBalbinus wants to merge 27 commits into
mainfrom
feat/workflow-templates
Draft

feat(workflows): Templates gallery + PR code-review automation#75
xBalbinus wants to merge 27 commits into
mainfrom
feat/workflow-templates

Conversation

@xBalbinus

@xBalbinus xBalbinus commented Jul 1, 2026

Copy link
Copy Markdown

Templates gallery + PR code-review automation

A Zapier-style Templates gallery on top of the dag/v1 workflow interpreter (#43), plus the first template: an automated PR code-review workflow driven by the org's GitHub App. Installing a template creates a real, published, runnable workflow via the canonical create → publish → trigger path — no bespoke execution engine — and the code-review template can be armed on a repo with one click, no manual webhook setup.

What's included

  • Templates gallery — plugin-owned templates (each plugin ships the templates for the actions it owns); a source-agnostic install service that mirrors the HTTP create/publish path. GET /api/templates filters by the org_plugins switch, so disabling a plugin removes its templates.
  • Code-review automation — a github-app trigger type: one App delivery fans out to every matching repo trigger. Claude reviews the PR diff on three axes — intent vs. changeset, conventions, correctness — and posts a single review comment, leading with an intent check.
  • Org + user settings — org codeReviewEnabled / enforced / mentionOnly plus per-owner prefs, precedence modeled on the action-policy system (admin GitHub settings + Settings → Agent).
  • Install prompt — arming an uncovered repo prompts to install the Valet GitHub App; credential:'app' posts as the org bot, never as a person.

How it works

/webhooks/github receives App deliveries; dispatchGithubAppReviews reads the org policy once (an org OFF is absolute), then per armed trigger evaluates resolveCodeReviewGate(org, owner, reason). Precedence is an org ceiling — the owner may only loosen, never tighten past the org or override an org OFF (the same shape as resolveEffectiveActionPolicy). decideReview applies a Greptile-style policy: skip drafts/pushes, review on open/reopen/ready, re-review only on an @<bot> mention. Dispatch is best-effort — one trigger failure never blocks the webhook 200.

The review posts under the App-installation (bot) token: credential:'app' resolves the bot token only and fails with an actionable "install the App" message if it's unavailable — it does not silently fall back to the caller's personal token. github.inspect_pull_request gains an opt-in includePatch (backward-compatible) so the reviewer sees the actual diff.

Two run paths: an armed App trigger (automatic, every PR) and manual "Run now" with { owner, repo, pullNumber } via the gate's isEmpty arm. A manual per-repo webhook remains as a demoted fallback.

Testing

Full local end-to-end: install via API and gallery UI, then both trigger paths through all nodes to a real generated review (which correctly caught injected bugs). Unit/integration coverage for the credential resolver (app vs. default), the code-review gate matrix, dispatch-level no-dispatch paths (draft/push, generic @valet, wrong repo, disabled trigger, org gates), and template install (structural validity, publish, trigger creation, repeat-install collision). tsc clean across worker/shared/client; worker 1250, client 174 vitest green.

Follow-ups

  • Non-policy per-template tunables (review prompt, model) deferred — currently template-fixed; they don't map onto the policy resolver, so they'd be plain settings.
  • Rolling the code-review gate into the action_policies framework proper (org rows + user allow-rows) is a natural next step — the settings were shaped to migrate cleanly.
  • Real GitHub App install is required for post-as-bot; up to that step the mechanism is verified end-to-end.

@xBalbinus xBalbinus marked this pull request as draft July 1, 2026 23:39
xBalbinus added 2 commits July 2, 2026 13:02
Adds a Zapier-style template gallery on top of the dag/v1 workflow
interpreter: a worker-side template registry, an install API, and a
client gallery. The flagship template reviews a pull request — fetches
the diff, reviews the changed lines with an LLM, then posts a review
comment back on the PR.

- shared: WorkflowTemplateSummary / InstallTemplateResponse types.
- worker: template registry + install service (createWorkflow ->
  saveDraft -> publishDraft -> webhook trigger). GET /api/templates and
  POST /api/templates/:id/install. Install is atomic — a failure after
  the workflow row is created rolls it back, and the webhook path/name
  are suffixed with a fresh uuid so repeat installs don't collide.
- plugin-github: opt-in includePatch on github.inspect_pull_request so a
  reviewer can see the actual diff. Backward-compatible — patch is only
  present when includePatch is true.
- client: /automation/templates gallery with "Use this template", a
  one-time webhook secret reveal, and a "Run now" dialog (manual run).
- test: asserts every template definition passes the publish validator.
- app-logo chain per card (trigger -> ... -> action), reusing the app's
  brand logos via getServiceIcon; the LLM step gets a Claude mark
- verb-first, plain-language titles ("Review pull requests and post a
  comment") plus short descriptions
- search box + category filter pills over the gallery
- "Try it" opens a guided setup dialog: numbered steps, then
  "Use this template" -> run-now inputs + one-time webhook reveal
- template metadata gains apps[] + steps[] to drive the card + setup UI
@xBalbinus xBalbinus changed the base branch from spec/cloudflare-workflow-interpreter to main July 2, 2026 17:03
@xBalbinus xBalbinus force-pushed the feat/workflow-templates branch from ee329bd to 48e2046 Compare July 2, 2026 17:04
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

Preview deployment: https://pr-75.dev-valet-turnkey-client.pages.dev

xBalbinus added 6 commits July 3, 2026 11:12
Per Conner's suggestion: templates now live in the plugin that owns their
actions, and the integration registry aggregates them — exactly like
actions/triggers. Enabling/disabling a plugin adds/removes its templates.

- sdk: WorkflowTemplate type + optional `templates` on IntegrationPackage
- plugin-github: src/templates.ts (githubTemplates, the code-review template)
  threaded into the package export next to the github.* actions it uses
- registry: listTemplates() flattens every plugin's contributed templates
- worker: workflow-templates.ts reads from the registry (install path unchanged)
- GET /api/templates gates on org_plugins — a plugin disabled org-wide drops
  its templates from the gallery

No codegen change needed: generate-plugin-registry imports each plugin's default
export, so the new `templates` field rides along automatically.
…t publish

'anthropic/claude-sonnet-4-6' is rejected by parseModelId (expects
'anthropic:<model>'), so every code-review template install 500'd with
llm_model_id_invalid. Caught in live local testing; the template
validation test didn't catch it because it runs without a configured
provider key.
The landing hides the tab strip (full-bleed hero), and its card grid
predates the gallery — so first click on Automation offered no path to
Templates. Add a Templates NavCard (live count) as the first card and
retitle the section Inspect -> Explore since Templates is a starting
point, not an inspection target.
- Install route translates WorkflowVersionError to 400 (503+Retry-After
  for publish_contention), mirroring the /api/workflows publish route —
  a template whose model/provider isn't configured is a caller-fixable
  error, not a 500. Route headers use the /** METHOD /path */ style.
- 'Run now' inputs derive from the trigger node's dataSchema
  (templateRunInputs) instead of a duplicate WorkflowTemplateInput list
  on the template — WorkflowInputDefinition gains advisory label/
  placeholder; wire shape and client untouched.
- plugin-github templates.ts moves into src/actions/ with the rest of
  the ./actions export surface; WorkflowTemplate.apps doc now covers
  non-service brand tokens ('claude').
- Rollback failure during install logs a console.warn instead of a
  silent swallow.
- Tests: publish-time env validation over every template (the gate that
  would have caught the slash-form model id), plus installWorkflowTemplate
  coverage against a real migrated SQLite — happy path, env-gate rollback,
  repeat-install collision, unknown id.
SearchInput + Skeleton + cn() instead of hand-rolled equivalents; an
error state distinct from the empty state (a failed /api/templates no
longer renders as 'No matching templates'); clipboard copy wrapped in
try/catch with a fallback toast (webhook-token-reveal idiom); reuse
useRunWorkflow (restoring clientRequestId dedup + Runs invalidation)
instead of a duplicate hook; templateKeys gains the list() leaf.
dag/v1 stores node.action as the full service-prefixed id
('github.inspect_pull_request'), so composing service + action rendered
'github.github.inspect_pull_request' in trace cards, canvas subtitles,
approval prompts, and the disabled-action error. Pre-existing, surfaced
by this PR's template; guarded with startsWith so hand-typed unprefixed
custom actions keep the composed form.
@xBalbinus xBalbinus changed the title feat(workflows): Templates gallery + PR code-review automation (stacked on #43) feat(workflows): Templates gallery + PR code-review automation Jul 3, 2026
@xBalbinus xBalbinus force-pushed the feat/workflow-templates branch from c20945f to ada311e Compare July 3, 2026 15:37
@xBalbinus

Copy link
Copy Markdown
Author

Solid draft overall — architecture is coherent and the intent is delivered. A few issues worth addressing before merge.

Intent check: The diff delivers everything described: WorkflowTemplate types in the SDK/shared, the GitHub code-review template, install service with rollback, two new API routes, and the gallery UI with search/filter/setup dialog. The double-prefix bug fix in trace cards, canvas subtitles, and approval prompts is a genuine pre-existing problem correctly addressed here.

Findings:

  • packages/worker/src/routes/templates.ts — unauthenticated install endpoint. GET /api/templates and POST /api/templates/:id/install don't verify c.get('user') before use. If the auth middleware is applied globally this is fine, but the install handler dereferences user.id without a guard. Compare to how other routes in routes/workflows.ts handle this — add an explicit 401 check or confirm middleware is unconditional.

  • packages/worker/src/routes/templates.ts — webhook URL construction is fragile. c.req.url.startsWith('https') matches the full request URL (e.g. https://example.com/api/templates/code-review/install), so this works, but it also matches any URL whose path happens to start with https (contrived but worth noting). More importantly, if the worker is behind a proxy sending X-Forwarded-Proto, the req.url may still be http://. Consider reading X-Forwarded-Proto or using a configured BASE_URL env var, consistent with how other routes construct external URLs.

  • packages/plugin-github/src/actions/templates.tsapps array repeats 'github' with 'claude' in the middle, but AppGlyph in the client only checks for service === 'claude' || service === 'anthropic'. If a future template uses openai or another LLM brand token, getServiceIcon('openai') will be called — verify getServiceIcon handles unknown service ids gracefully (returns a fallback) or document the supported token set.

  • packages/worker/src/services/workflow-templates.tstemplateRunInputs silently drops fields whose type is not 'string' or 'number' (e.g. boolean, array). The code-review template only uses those two types, so it's fine for now, but the filter with no warning makes it easy to ship a template whose fields quietly vanish from the UI. At minimum, add a comment; ideally log a warning in dev.

  • packages/client/src/routes/automation/templates/index.tsx — dialog state is not reset on close. installedWorkflowId, webhook, and values are all in TemplateSetupDialog state, but they survive if the user dismisses the dialog and reopens it for the same template (since TemplateCard mounts the dialog inside the same component instance). The second open will show the "Run now" form for an already-installed workflow. Add a React.useEffect that resets state when open goes false, or lift/clear state on close.

  • packages/shared/src/types/workflow-templates.tsWorkflowTemplateSummary.inputs is always present (never optional), but TemplateSetupDialog guards template.inputs.length > 0 before rendering the "Run now" section — that's fine. However, WorkflowTemplateInput.required is boolean | undefined; the client's missingRequired check (i.required && ...) correctly treats undefined as falsy, but it would be cleaner to normalise this to boolean in templateRunInputs.

…tions prompt

The code-review template now targets a REPO, not one PR: the webhook maps
the GitHub event `action`, and an `if` gate only runs the review on events
that introduce or change code (opened/reopened/synchronize/ready_for_review).
Noise events (closed, labeled, assigned, …) short-circuit before the LLM
call — verified live: a 'closed' delivery completes in ~40ms with fetch/
review/post skipped, an 'opened' delivery runs the full pipeline. Manual
'Run now' still works via the gate's isEmpty arm.

The review prompt now judges on three axes: intent-vs-changeset (does the
diff do what the title/body claim?), conventions (consistency with patterns
visible in the surrounding diff + sibling files), and correctness. It leads
with an explicit Intent check.

`action` is a new hidden trigger input — part of the invocation contract so
trigger-data validation accepts it on webhooks, but WorkflowInputDefinition
gains an advisory `hidden` flag so templateRunInputs keeps it out of the
'Run now' form.
xBalbinus added 4 commits July 3, 2026 13:46
The App manifest registered deliveries to /api/webhooks/github, which is
behind the /api/* auth middleware (no webhook exclusion) — so every
App-delivered event 401'd. The handler is mounted at /webhooks/github
(before the auth middleware). One-line fix; already-created Apps still
need their webhook URL hand-edited in GitHub settings.
Tool nodes gain an opt-in credential:'app' — resolves the org GitHub
App installation (bot) token instead of the workflow owner's personal
one, degrading gracefully to the owner's token where no App is installed.
The code-review template's post node uses it, so reviews post as the bot
when the App is set up. create_comment now appends the app-authored
attribution footer under a bot token (per the AI-labeling requirement).
A new 'github-app' trigger (scoped to owner/repo) fires from the org
GitHub App event stream — no per-workflow webhook to configure. A
pull_request delivery fans out to every matching enabled trigger,
mapping the payload into trigger.data via the trigger's variableMapping,
deduped per delivery id, filtered by subscribed event. Migration 0024
adds 'github-app' to the triggers.type CHECK (table recreate).
The setup dialog now leads with 'Arm it for a repository': pick a
connected repo, and if the Valet App covers its owner, one click creates
a github-app trigger (POST /api/templates/:id/enable-app) — reviews
every PR, posted as the bot, no webhook paste. The manual webhook is
demoted to a collapsible fallback. Coverage + ownership validated
server-side; dialog state now resets on close.
…ence

The runForm hint (WorkflowTemplate + WorkflowTemplateSummary) was added
to the client/worker consumers but its type declarations were never
committed — the branch typechecked only against a locally-built dist.
Commit the source so CI (which builds from source) stays green.
The zero-param vi.fn made the mock's call tuple empty, so indexing [1]
failed the workspace typecheck (tsc --build, stricter than the worker's
tsc --noEmit). Type the mock params and cast the asserted input.
Re-arming a repo already armed for the user hit the per-user unique
trigger-name index (idx_triggers_user_name) and leaked a raw D1
constraint error as a 500 'Internal server error'. enableTemplateGithubApp
now checks for an existing github-app trigger with the same name first
and returns it with alreadyArmed:true instead of creating a duplicate.
The dialog shows 'Already armed' in that case.
…dialog

Drop the 'Template added' toast that fired on install (it popped the
moment the arming section appeared). Rename the user-facing 'Arm it for
a repository' / 'Armed for …' copy to 'Install …', plus the internal
section component/state, per product wording.
…he button

Picking a repo that already has a github-app trigger now shows the '✓
Installed' confirmation instead of the 'Enable via GitHub App' button, so
you can't double-install. Derived from the triggers list (client Trigger
type + config gain the github-app variant); combined with the just-
installed local state so the confirmation persists after enabling.
xBalbinus added 2 commits July 3, 2026 14:48
… failure

tryApp is now a total function — a throw from installation-token minting
(e.g. a misconfigured App / undecodable JWT) returns ok:false instead of
propagating, so credentialMode:'app' degrades to the caller's own token
rather than failing the workflow's post step.
…nly, @valet re-review)

decideReview gates App-driven reviews: never review a draft PR; post the
initial review on opened/reopened/ready_for_review; do NOT re-review on a
push (synchronize); re-review only when a PR comment @-mentions Valet
(with a bot-comment loop guard). github-app triggers now subscribe to
pull_request AND issue_comment, and the webhook route dispatches both.
The dispatch sends clean {owner,repo,pullNumber} (no action) since the
decision is made up front — the template gate passes via its isEmpty arm.
@xBalbinus

Copy link
Copy Markdown
Author

Solid PR overall — the architecture is clean, the atomicity story for install is well-handled, and the test coverage is meaningfully hermetic. A few real issues worth acting on:

  • Webhook URL construction is wrong in routes/templates.ts: c.req.url.startsWith('https') matches the full URL string (e.g. https://...), so http requests will never be detected — this always resolves to https in production (fine) but silently breaks in local dev. Use new URL(c.req.url).protocol instead, or inherit c.req.url directly for the origin. (packages/worker/src/routes/templates.ts, ~line 84)

  • issue_comment dispatch fires for all repos, not just repos with a matching trigger: dispatchGithubAppReviews looks up findGithubAppTriggersForRepo(env.DB, owner, repo) where owner/repo come from payload.repository. For issue_comment events, payload.repository is present in GitHub webhook deliveries, so this is actually fine — but decideReview extracts owner/repo from repository and returns them, while dispatchGithubAppReviews re-extracts them from decision rather than from the payload. Verify these are always consistent; if a future event type sets them differently the dispatch could silently mis-route. Minor, but worth a comment. (packages/worker/src/services/webhooks.ts)

  • GithubAppInstallSection checks alreadyInstalled against triggersData which may be stale: after handleEnable succeeds, justInstalled correctly flips the UI, but if the user closes and reopens the dialog (resetting justInstalled to false), the stale triggersData cache (not yet invalidated — useEnableTemplateApp.onSuccess invalidates triggerKeys.lists() but the component reads from the already-cached query) can make the "Enable" button reappear. useEnableTemplateApp should also invalidate or the component should keep justInstalled across the reset. (packages/client/src/routes/automation/templates/index.tsx ~line 226, and packages/client/src/api/templates.ts useEnableTemplateApp.onSuccess)

  • Silent fullName split on / is brittle: e.target.value.split('/') assumes repo names contain no slashes; GitHub org/repo names never do, but a comment would make this contract explicit. More importantly, if fullName is somehow missing a / (e.g. a bare org name leaks into the list), r is undefined and you silently set repo: ''. A ?? '' guard is already there but the invariant should be asserted or the option filtered. (packages/client/src/routes/automation/templates/index.tsx ~line 297)

…ush-review policy

Review prompt now optimizes for signal: report only issues that matter,
say so briefly when a PR is solid, and — critically — do not claim
something is missing (auth, a guard, a test) just because it is absent
from the diff (it likely lives elsewhere in the repo). Also drop
'synchronize' from the gate so the raw-webhook path matches decideReview:
review on open/reopen/ready, re-review only on @valet.
…iew, not 'valet'

The generic word 'valet' resolves to a real, unrelated GitHub user
(github.com/valet), so '@valet' in a comment pinged a stranger and wasn't
the bot. decideReview now matches the installed App's own handle —
@<slug> or the collision-proof @<slug>[bot] login (e.g. @valet-turnkey)
— resolved from the App config; with no configured slug it never matches.
The code-review template's @-mention re-review reacts to issue_comment
webhooks, but the manifest only requested push + pull_request — so a
freshly-created App never received comment events. Add issue_comment to
the default manifest events.
…allback

Per review: an automation that posts as the bot must never silently fall
back to a person's token. credentialMode='app' now resolves the App
installation (anonymous/bot) credential and, if unavailable, fails with
an actionable 'ask an admin to install the App' error instead of using
the caller's personal token. Default (non-app) mode is unchanged.
…action policies

Add admin (org) and per-user controls for the AI code-review automation, with
an org-ceiling + user-may-only-loosen precedence that mirrors
resolveEffectiveActionPolicy — the org value is a hard ceiling and a user may
only make review quieter for their own repos, never override an org OFF.

Policy-shaped controls (resolved by resolveCodeReviewGate):
- org: codeReviewEnabled (master, OFF is absolute), codeReviewEnforced (locks
  the user knobs, the userGrantBehavior:'blocked' analog), codeReviewMentionOnly
  (org-wide quiet mode) — GitHubServiceMetadata, via PUT /api/admin/github/settings.
- user: codeReviewEnabled (owner opt-out) + codeReviewMentionOnly (owner quiet
  mode) — users.code_review_* (migration 0025), via PATCH /api/auth/me.

Enforcement in dispatchGithubAppReviews: org gate short-circuits up front; the
per-owner gate reads the arming user's prefs (skipped entirely when the org
enforces). trigger.user_id is the repo owner who armed the automation.

UI: org toggles in the admin GitHub settings panel; a user "Code review" section
under Settings > Agent.

Prompt/model tuning are deliberately left as a future non-policy seam (they have
no order relation, so they don't fit a mode/deny/allow resolver).

Tests: resolveCodeReviewGate truth tables + dispatch org/owner gate coverage.
…ered repo

When a picked repo's owner has no App installation, offer a direct
'Install the Valet App on <owner>' button (github.com/apps/<slug>/installations/new)
instead of a dead-end 'ask an admin' message — installing the App is part of
arming the automation. Falls back to a plain one-line notice when the App
isn't configured at all.
Self-review pass to trim overengineering and match repo conventions; no
behavior change (worker 1250/1250, client 174/174, tsc clean).

- Fix comments/docstrings that contradicted the code: the github resolver
  docstring + tryApp comment and the template post-node comment all claimed
  credential:'app' falls back to the user token (it never does); the template
  header listed 'synchronize' as a gate arm (the gate deliberately omits it).
- Remove dead isBotToken ternary in github.create_comment — every sibling
  appends attributionSuffix(ctx) unconditionally (it's '' without attribution).
- Delete the up-front structural validation in installWorkflowTemplate — it
  duplicated the publishDraft gate byte-for-byte and the rollback already
  covers the impossible bad-template case.
- Drop the never-rendered WorkflowTemplateInput.description field.
- Collapse the 12-line precedence comment on GitHubServiceMetadata (the rule
  lives on resolveCodeReviewGate) and relocate the orphaned dispatch JSDoc.
- DRY: qualifyActionId() helper (lib/action-id.ts) for the service-prefix rule
  inlined in tool.ts + approval-view.ts; export toolCallName() from the editor
  model instead of a second copy in trace-node-card; a shared RepoSelect for
  the arm + Run-now forms.
@xBalbinus xBalbinus marked this pull request as ready for review July 3, 2026 21:47
@xBalbinus xBalbinus requested a review from a team July 3, 2026 21:47
@xBalbinus xBalbinus marked this pull request as draft July 6, 2026 17:13
@xBalbinus xBalbinus marked this pull request as ready for review July 6, 2026 17:19
@xBalbinus xBalbinus marked this pull request as draft July 6, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant