From 9faac131d0b7eb8db42e6447cc7330392107174c Mon Sep 17 00:00:00 2001 From: Nathan Rodd Date: Sun, 26 Apr 2026 16:29:43 -0400 Subject: [PATCH 1/4] fix: updating skill --- skills/onboard/SKILL.md | 59 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/skills/onboard/SKILL.md b/skills/onboard/SKILL.md index 95860169..2d0028e2 100644 --- a/skills/onboard/SKILL.md +++ b/skills/onboard/SKILL.md @@ -33,6 +33,27 @@ Brief description of what this step does. The horizontal rules and `##` heading create visual separation from tool output above and below. +## Workflow Telemetry + +At each milestone listed below, call the `workflow-event` tool to log an analytics event. This data powers onboarding success dashboards. Always pass `workflow="onboard"`. + +**All `workflow-event` calls are conditional on the user's consent (see Telemetry Consent below). If the user declined, skip every telemetry call silently.** + +**Rules for metadata fields:** +- Use the exact field names shown at each step. Values must be short strings. +- Use enums for all classification fields. Valid enums are listed below. +- `outcome`: `success` | `skipped` | `partial` | `fail` (per-step; see each step for valid values) +- `flow_type`: `user_described` | `feeling_lucky` +- `sightmap_quality`: `high` (10+ components with memories) | `medium` (5-10 components or few memories) | `low` (fewer than 5 components) +- `use_case`: `e-commerce` | `saas-dashboard` | `marketing-site` | `docs-site` | `social-platform` | `developer-tool` | `internal-tool` | `other`. Infer from package.json, site content, or the user's description. +- Only include fields that are known at that step. Don't guess ahead. +- The `metadata` parameter must be a valid JSON string. + +**Outcome classification for the `complete` step:** +- `success`: all steps completed, reproduction showed improvement +- `partial`: workflow finished but a step had degraded results (e.g., reproduction used more tokens than blind) +- `fail`: a step failed and the workflow couldn't finish + ## Start Greet the user: @@ -45,6 +66,30 @@ Greet the user: Let's get started." +### Telemetry Consent + +Before logging any analytics, ask the user for consent using the `AskUserQuestion` tool: + +```json +{ + "questions": [{ + "question": "Would you mind sharing anonymous usage data to help improve Subtext? This includes step outcomes, interaction counts, and timing — no code or personal information.", + "header": "Usage data", + "multiSelect": false, + "options": [ + { "label": "Yes, share anonymously", "description": "Help improve Subtext with anonymous step outcomes, timing, and interaction counts." }, + { "label": "No thanks", "description": "Skip all analytics. Nothing will be sent." } + ] + }] +} +``` + +Store the result as `telemetry_enabled` (true/false). If the user selects **Yes, share anonymously**, set `telemetry_enabled = true`. If they select **No thanks** or **Other**, set `telemetry_enabled = false`. + +**All `workflow-event` calls throughout this workflow are conditional on `telemetry_enabled = true`. If the user declined, skip every telemetry call silently.** + +**Telemetry (if enabled):** Call `workflow-event` with step=`"start"` and metadata containing `harness` (e.g. `claude-code`, `cursor`, `codex`) and `model` (the active model ID, e.g. `claude-sonnet-4-6`). + ## Step 1: Plugin Setup Print: @@ -63,6 +108,8 @@ Checking that the Subtext plugin and MCP servers are installed and connected. If not complete → invoke `subtext:setup-plugin` If complete → "Plugin's already set up. Moving on." +**Telemetry (if enabled):** Call `workflow-event` with step=`"plugin_setup"` and metadata containing `outcome` (`success` | `skipped`). + ## Step 2: First Session (Blind Exploration) ### Dev server gate @@ -108,6 +155,8 @@ After the subagent completes, tell the user: Now let me analyze what I found." +**Telemetry (if enabled):** Call `workflow-event` with step=`"first_session"` and metadata containing `outcome` (`success` | `fail`), `interaction_count`, `tokens`, `duration_ms`, and `flow_type` (`user_described` | `feeling_lucky`). + ## Step 3: Session Review Print: @@ -127,6 +176,8 @@ Invoke `subtext:session-analysis-workflow` with the captured session URL. After the review, explain: "That's what Subtext sees when it analyzes a session. It maps interactions to your source code, identifies friction, and spots issues — all from a single session. The agent's own difficulty notes are part of the timeline." +**Telemetry (if enabled):** Call `workflow-event` with step=`"session_review"` and metadata containing `outcome` (`success` | `fail`) and `issue_count`. + ## Step 4: Sightmap Bootstrap Print: @@ -176,6 +227,8 @@ Example: - "Audience toggle switches ALL copy between 'For Builders' and 'For Agents' perspectives" ``` +**Telemetry (if enabled):** Call `workflow-event` with step=`"sightmap_bootstrap"` and metadata containing `outcome` (`success` | `skipped`), `component_count`, and `memory_count`. + ## Step 5: Informed Reproduction (Pass 2) Print: @@ -194,8 +247,6 @@ Reproducing the same flow — this time with sightmap enrichment. - The sightmap is uploaded automatically after `review-open`/`live-connect` (see `subtext:shared`) - The same app URL -**Capture from subagent result:** interaction count, tokens, duration_ms. - **Capture from subagent result:** session URL, viewer URL, interaction count, tokens, duration_ms. After the subagent completes: "Done! Reproduced the flow in {interaction_count} interactions. You can review the replay with agent comments here: @@ -204,6 +255,8 @@ After the subagent completes: "Done! Reproduced the flow in {interaction_count} Now let's see how the two passes compare." +**Telemetry (if enabled):** Call `workflow-event` with step=`"reproduction"` and metadata containing `outcome` (`success` | `fail`), `interaction_count`, `tokens`, and `duration_ms`. + ## Step 6: Results Print: @@ -246,6 +299,8 @@ Present both viewer URLs so the user can review each session with agent comments - **Pass 1 (Blind):** {pass1_viewer_url} - **Pass 2 (Sightmap):** {pass2_viewer_url} +**Telemetry (if enabled):** Call `workflow-event` with step=`"complete"` and metadata containing `outcome` (`success` | `partial` | `fail` — see classification rules in Workflow Telemetry section), `interaction_delta_pct`, `token_delta_pct`, `time_delta_pct`, `sightmap_quality` (`high` | `medium` | `low`), and `use_case` (inferred from the app). + ## Recap "You're all set! Here's what we accomplished: From c3f8f7cc0c01809b119761c6b69b774a1fde92a2 Mon Sep 17 00:00:00 2001 From: Nathan Rodd Date: Mon, 27 Apr 2026 15:58:06 -0400 Subject: [PATCH 2/4] merging main --- skills/onboard/SKILL.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/skills/onboard/SKILL.md b/skills/onboard/SKILL.md index 2d0028e2..103f10a1 100644 --- a/skills/onboard/SKILL.md +++ b/skills/onboard/SKILL.md @@ -42,7 +42,7 @@ At each milestone listed below, call the `workflow-event` tool to log an analyti **Rules for metadata fields:** - Use the exact field names shown at each step. Values must be short strings. - Use enums for all classification fields. Valid enums are listed below. -- `outcome`: `success` | `skipped` | `partial` | `fail` (per-step; see each step for valid values) +- `outcome` (top-level param, not inside metadata): `success` | `skipped` | `partial` | `fail` (per-step; see each step for valid values) - `flow_type`: `user_described` | `feeling_lucky` - `sightmap_quality`: `high` (10+ components with memories) | `medium` (5-10 components or few memories) | `low` (fewer than 5 components) - `use_case`: `e-commerce` | `saas-dashboard` | `marketing-site` | `docs-site` | `social-platform` | `developer-tool` | `internal-tool` | `other`. Infer from package.json, site content, or the user's description. @@ -108,7 +108,7 @@ Checking that the Subtext plugin and MCP servers are installed and connected. If not complete → invoke `subtext:setup-plugin` If complete → "Plugin's already set up. Moving on." -**Telemetry (if enabled):** Call `workflow-event` with step=`"plugin_setup"` and metadata containing `outcome` (`success` | `skipped`). +**Telemetry (if enabled):** Call `workflow-event` with step=`"plugin_setup"` and outcome=`"success"` or `"skipped"`. ## Step 2: First Session (Blind Exploration) @@ -155,7 +155,7 @@ After the subagent completes, tell the user: Now let me analyze what I found." -**Telemetry (if enabled):** Call `workflow-event` with step=`"first_session"` and metadata containing `outcome` (`success` | `fail`), `interaction_count`, `tokens`, `duration_ms`, and `flow_type` (`user_described` | `feeling_lucky`). +**Telemetry (if enabled):** Call `workflow-event` with step=`"first_session"`, outcome=`"success"` or `"fail"`, and metadata containing `interaction_count`, `tokens`, `duration_ms`, and `flow_type` (`user_described` | `feeling_lucky`). ## Step 3: Session Review @@ -176,7 +176,7 @@ Invoke `subtext:session-analysis-workflow` with the captured session URL. After the review, explain: "That's what Subtext sees when it analyzes a session. It maps interactions to your source code, identifies friction, and spots issues — all from a single session. The agent's own difficulty notes are part of the timeline." -**Telemetry (if enabled):** Call `workflow-event` with step=`"session_review"` and metadata containing `outcome` (`success` | `fail`) and `issue_count`. +**Telemetry (if enabled):** Call `workflow-event` with step=`"session_review"`, outcome=`"success"` or `"fail"`, and metadata containing `issue_count`. ## Step 4: Sightmap Bootstrap @@ -227,7 +227,7 @@ Example: - "Audience toggle switches ALL copy between 'For Builders' and 'For Agents' perspectives" ``` -**Telemetry (if enabled):** Call `workflow-event` with step=`"sightmap_bootstrap"` and metadata containing `outcome` (`success` | `skipped`), `component_count`, and `memory_count`. +**Telemetry (if enabled):** Call `workflow-event` with step=`"sightmap_bootstrap"`, outcome=`"success"` or `"skipped"`, and metadata containing `component_count` and `memory_count`. ## Step 5: Informed Reproduction (Pass 2) @@ -255,7 +255,7 @@ After the subagent completes: "Done! Reproduced the flow in {interaction_count} Now let's see how the two passes compare." -**Telemetry (if enabled):** Call `workflow-event` with step=`"reproduction"` and metadata containing `outcome` (`success` | `fail`), `interaction_count`, `tokens`, and `duration_ms`. +**Telemetry (if enabled):** Call `workflow-event` with step=`"reproduction"`, outcome=`"success"` or `"fail"`, and metadata containing `interaction_count`, `tokens`, and `duration_ms`. ## Step 6: Results @@ -299,7 +299,7 @@ Present both viewer URLs so the user can review each session with agent comments - **Pass 1 (Blind):** {pass1_viewer_url} - **Pass 2 (Sightmap):** {pass2_viewer_url} -**Telemetry (if enabled):** Call `workflow-event` with step=`"complete"` and metadata containing `outcome` (`success` | `partial` | `fail` — see classification rules in Workflow Telemetry section), `interaction_delta_pct`, `token_delta_pct`, `time_delta_pct`, `sightmap_quality` (`high` | `medium` | `low`), and `use_case` (inferred from the app). +**Telemetry (if enabled):** Call `workflow-event` with step=`"complete"`, outcome=`"success"`, `"partial"`, or `"fail"` (see classification rules in Workflow Telemetry section), and metadata containing `interaction_delta_pct`, `token_delta_pct`, `time_delta_pct`, `sightmap_quality` (`high` | `medium` | `low`), and `use_case` (inferred from the app). ## Recap From e935ed7f7601a04af6b570fcbfe7502042ec248a Mon Sep 17 00:00:00 2001 From: Nathan Rodd Date: Tue, 28 Apr 2026 10:31:23 -0400 Subject: [PATCH 3/4] fix: updating skill wording --- skills/onboard/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/skills/onboard/SKILL.md b/skills/onboard/SKILL.md index 103f10a1..7f768d88 100644 --- a/skills/onboard/SKILL.md +++ b/skills/onboard/SKILL.md @@ -37,7 +37,7 @@ The horizontal rules and `##` heading create visual separation from tool output At each milestone listed below, call the `workflow-event` tool to log an analytics event. This data powers onboarding success dashboards. Always pass `workflow="onboard"`. -**All `workflow-event` calls are conditional on the user's consent (see Telemetry Consent below). If the user declined, skip every telemetry call silently.** +**All `workflow-event` calls are conditional on the user's consent (see Telemetry Consent below). If the user declined, YOU MUST skip every telemetry call silently.** **Rules for metadata fields:** - Use the exact field names shown at each step. Values must be short strings. @@ -73,11 +73,11 @@ Before logging any analytics, ask the user for consent using the `AskUserQuestio ```json { "questions": [{ - "question": "Would you mind sharing anonymous usage data to help improve Subtext? This includes step outcomes, interaction counts, and timing — no code or personal information.", + "question": "Would you mind sharing usage data to help improve Subtext? This includes step outcomes, interaction counts, and timing — no code or personal information.", "header": "Usage data", "multiSelect": false, "options": [ - { "label": "Yes, share anonymously", "description": "Help improve Subtext with anonymous step outcomes, timing, and interaction counts." }, + { "label": "Yes, share", "description": "Help improve Subtext with step outcomes, timing, and interaction counts." }, { "label": "No thanks", "description": "Skip all analytics. Nothing will be sent." } ] }] From 028517735b9d1f6e04ae8addc17f6b3e61ccabc3 Mon Sep 17 00:00:00 2001 From: Nathan Rodd Date: Thu, 30 Apr 2026 14:07:53 -0400 Subject: [PATCH 4/4] updating skill for new onbaording --- skills/onboard/SKILL.md | 51 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/skills/onboard/SKILL.md b/skills/onboard/SKILL.md index 32b18555..a6aa9ee0 100644 --- a/skills/onboard/SKILL.md +++ b/skills/onboard/SKILL.md @@ -19,6 +19,27 @@ The goal: walk a new user through one real, useful Subtext run end-to-end. They Do **not** announce a "plugin setup" step. Trust that the plugin is installed — the user just ran a slash command from it. If the first MCP call below fails (server unreachable, auth missing), invoke `subtext:setup-plugin`, then retry the call. Otherwise stay silent about plumbing. +## Workflow Telemetry + +At each milestone listed below, call the `workflow-event` tool to log an analytics event. This data powers onboarding success dashboards. Always pass `workflow="onboard"`. + +**All `workflow-event` calls are conditional on the user's consent (see Telemetry Consent below). If the user declined, YOU MUST skip every telemetry call silently.** + +**Tracking time and tokens:** Record a timestamp at the start of each step. At the end of each step, compute `duration_ms` (wall-clock milliseconds) and `tokens` (total input + output tokens consumed during the step). Include both in the step's `workflow-event` metadata. + +**Rules for metadata fields:** +- Use the exact field names shown at each step. Values must be short strings. +- Use enums for all classification fields. Valid enums are listed below. +- `outcome` (top-level param, not inside metadata): `success` | `skipped` | `partial` | `fail` (per-step; see each step for valid values) +- `use_case`: `e-commerce` | `saas-dashboard` | `marketing-site` | `docs-site` | `social-platform` | `developer-tool` | `internal-tool` | `other`. Infer from package.json, site content, or the user's description. +- Only include fields that are known at that step. Don't guess ahead. +- The `metadata` parameter must be a valid JSON string. + +**Outcome classification for the `complete` step:** +- `success`: all steps completed successfully +- `partial`: workflow finished but a step had degraded results +- `fail`: a step failed and the workflow couldn't finish + ## Greeting Open with two short paragraphs — no banner, no checklist: @@ -29,6 +50,28 @@ Open with two short paragraphs — no banner, no checklist: > > We're going to make one small visible change to your running app together. You'll watch it happen live in a trace you can replay later, see before/after screenshots, and end up with a starter sightmap your future agents can read. Should take about five minutes." +### Telemetry Consent + +Before logging any analytics, ask the user for consent using the `AskUserQuestion` tool: + +```json +{ + "questions": [{ + "question": "Would you mind sharing usage data to help improve Subtext? This includes step outcomes, interaction counts, and timing — no code or personal information.", + "header": "Usage data", + "multiSelect": false, + "options": [ + { "label": "Yes, share", "description": "Help improve Subtext with step outcomes, timing, and interaction counts." }, + { "label": "No thanks", "description": "Skip all analytics. Nothing will be sent." } + ] + }] +} +``` + +Store the result as `telemetry_enabled` (true/false). If the user selects **Yes, share**, set `telemetry_enabled = true`. If they select **No thanks** or **Other**, set `telemetry_enabled = false`. + +**Telemetry (if enabled):** Call `workflow-event` with step=`"start"` and metadata containing `harness` (e.g. `claude-code`, `cursor`, `codex`) and `model` (the active model ID, e.g. `claude-sonnet-4-6, GPT-5.5`). + ## Step 1 — Connect to your local dev server Print: @@ -70,6 +113,8 @@ Tell the user briefly: > "I'm connected. Open that link in another window — you'll watch live as I work. The same link stays valid as a replayable recording after we finish, so you can come back to it." +**Telemetry (if enabled):** Call `workflow-event` with step=`"connect"`, outcome=`"success"` or `"fail"`, and metadata containing `duration_ms` and `tokens`. + ## Step 2 — Make a small change Print: @@ -108,6 +153,8 @@ When proof's loop completes, recap to the user in a single message: If the user says no, ask whether to revert, retry, or stop. If yes, continue to Step 3. +**Telemetry (if enabled):** Call `workflow-event` with step=`"proof"`, outcome=`"success"` or `"fail"`, and metadata containing `interaction_count` (number of `live-interact` calls made during the proof loop), `duration_ms`, and `tokens`. + ## Step 3 — Bootstrap a sightmap from what we just learned Print: @@ -135,6 +182,8 @@ After writing, show the user the file and a brief commit pointer: > "Wrote `.sightmap/components.yaml`. Commit it alongside your code change — it travels with the repo, and every future agent that reads it gets a head start." +**Telemetry (if enabled):** Call `workflow-event` with step=`"sightmap_bootstrap"`, outcome=`"success"` or `"skipped"`, and metadata containing `component_count`, `memory_count`, `duration_ms`, and `tokens`. + ## Wrap-up Recap and point to next steps: @@ -149,3 +198,5 @@ Recap and point to next steps: > - **`/proof`** — use this any time you make a UI change. Same before/after evidence loop, no onboarding wrapper. > - **`/review`** — paste any session URL to get a structured summary, with optional reproduction steps on request. > - **Learn more about sightmap** — read `skills/sightmap/SKILL.md` to teach agents about more of your app's surface (views, requests, scoped components, memory entries)." + +**Telemetry (if enabled):** Call `workflow-event` with step=`"complete"`, outcome=`"success"`, `"partial"`, or `"fail"` (see classification rules in Workflow Telemetry section), and metadata containing `total_duration_ms` (wall-clock time for the entire onboarding), `total_tokens` (sum across all steps), `sightmap_quality` (`high` | `medium` | `low`), and `use_case` (inferred from the app).