feat(ai): correlate AI usage with delivery metrics#19
Conversation
Add a Grafana dashboard joining ai_activities with project_pr_metrics on calendar week and developer identity, and fix the account resolution it depends on. No schema change — both tables already exist. All four AI plugins resolved developers with an unordered First(WHERE email = ?). One person often owns several rows in `accounts` (gitextractor mints one keyed by commit email alongside the source plugin's), so the match was arbitrary — and picking the gitextractor twin attributed AI activity to an account with no PRs. gh-copilot queried `WHERE name = ?` on a table with no `name` column, so it never resolved anybody. Replaced with a shared identityhelper.AccountResolver that prefers org mappings, then PR count, then lowest id. Dashboard SQL also handles: two ai_activities row grains (aggregate columns, never COUNT(*)), pr_cycle_time in minutes not seconds, and developers owning multiple accounts (delivery summed across all).
The AI↔Delivery dashboard resolved developer identity at query time in every panel, re-running the same ai_activities → accounts → users → user_accounts → pull_requests → project_mapping join on each refresh. Add a dbt project (dbt/ai_delivery) that materialises the two joins that only change per sync: - mart_ai_person_accounts: every account belonging to each AI user - mart_ai_project_members: AI users who authored a PR in a project Rewrite the five data panels (Weekly, Developer, Project, Project×Week, Project×Provider) to read these marts instead of the inline CTEs. Output is byte-identical to the previous query-time version (verified against seeded data). Metric aggregation stays query-time to keep time-picker flexibility. Identity-health panels keep querying raw tables by design. No core DevLake changes: marts run via the dbt plugin (blueprint afterPlan), so they refresh after accounts/PRs/project_mapping populate, with zero edits under core/models or migration registries.
| if id == "" { | ||
| continue | ||
| } | ||
| org := r.orgMapped[id] |
There was a problem hiding this comment.
One thing I'm not sure holds here: orgMapped[id] is true whenever the account is mapped to any user, not specifically to the identity we're resolving. So on the email path, if byEmail[key] turns up an account that a human happened to map to a different person, it still gets the tier-1 boost and can beat the account that actually authored the PRs. I think the org preference should only apply to candidates that came in via orgByEmail[key] — e.g. pass that set into pick instead of reading the global map. The current tests all have the org-mapped account also be the correct one, so they'd pass either way and wouldn't catch this.
| if userName == "" { | ||
| return "" | ||
| } | ||
| return r.pick(r.byUserName[normalize(userName)]) |
There was a problem hiding this comment.
Two things on the login path. It never consults the org mapping at all, so gh-copilot — the one integration that couldn't resolve anybody before — still has no human override for a login that doesn't line up with user_name. And it runs the candidates through the same pick, which will hand the tier-1 org boost to a username match that's mapped to some unrelated user (same over-broad check as the email path). An orgByUserName index would cover the first; scoping the org check to the current identity covers the second.
| } | ||
|
|
||
| var accounts []crossdomain.Account | ||
| if err := db.All(&accounts); err != nil { |
There was a problem hiding this comment.
Small one: db.All(&accounts) reads every column of every account row when we only ever touch id/email/user_name — a dal.Select("id, email, user_name") trims the payload noticeably on a large install. Related: each of the six converters builds its own resolver, so this full accounts scan plus the pull_requests GROUP BY runs six times per pipeline. If it's easy to stash one instance in shared task data, the subtasks could reuse it.
|
|
||
| // newTestResolver builds a resolver without a database so the selection rules | ||
| // can be exercised directly. | ||
| func newTestResolver() *AccountResolver { |
There was a problem hiding this comment.
These all hand-populate the struct, which exercises the selection rules nicely but leaves NewAccountResolver — the actual DB-loading path — with no coverage. That's the exact spot the old WHERE name = ? bug lived, so one test that runs the loader against a mock dal and checks the HasTable fallback and the prCounts query populate the maps would close the loop.
| "datasource": "mysql", | ||
| "format": "table", | ||
| "rawQuery": true, | ||
| "rawSql": "WITH member_emails AS (\n SELECT DISTINCT LOWER(ai.user_email) AS email\n FROM ai_activities ai\n LEFT JOIN accounts acc ON LOWER(acc.email) = LOWER(ai.user_email)\n LEFT JOIN users u ON LOWER(u.email) = LOWER(ai.user_email)\n LEFT JOIN user_accounts ua ON ua.user_id = u.id\n JOIN pull_requests pr\n ON pr.author_id IN (ai.account_id, acc.id, ua.account_id)\n JOIN project_mapping pm\n ON pm.row_id = pr.base_repo_id AND pm.`table` = 'repos'\n WHERE pm.project_name IN ($project)\n)\nSELECT COUNT(DISTINCT user_email) AS \"AI Active Users\"\nFROM ai_activities\nWHERE $__timeFilter(date)\n AND provider IN ($provider)\n AND user_email <> ''\n AND LOWER(user_email) IN (SELECT email FROM member_emails)\n", |
There was a problem hiding this comment.
After the mart refactor the dashboard ends up defining "project member" two different ways: these overview panels (1–3) and the weekly timeseries still use the inline member_emails CTE with pr.author_id IN (ai.account_id, acc.id, ua.account_id), while 11/20/40–42 read mart_ai_project_members. The mart resolves identity through mart_ai_person_accounts and the CTE doesn't, so the two can produce different member sets — the headline "AI Active Users" here could disagree with what the fact tables show. Pointing these at the mart too would keep them consistent and drop the per-refresh join.
| "datasource": "mysql", | ||
| "format": "table", | ||
| "rawQuery": true, | ||
| "rawSql": "WITH member_emails AS (\n SELECT DISTINCT LOWER(ai.user_email) AS email\n FROM ai_activities ai\n LEFT JOIN accounts acc ON LOWER(acc.email) = LOWER(ai.user_email)\n LEFT JOIN users u ON LOWER(u.email) = LOWER(ai.user_email)\n LEFT JOIN user_accounts ua ON ua.user_id = u.id\n JOIN pull_requests pr\n ON pr.author_id IN (ai.account_id, acc.id, ua.account_id)\n JOIN project_mapping pm\n ON pm.row_id = pr.base_repo_id AND pm.`table` = 'repos'\n WHERE pm.project_name IN ($project)\n),\nai AS (\n SELECT DATE_SUB(DATE(date), INTERVAL WEEKDAY(date) DAY) AS week_start,\n COUNT(DISTINCT user_email) AS ai_users,\n SUM(lines_added) AS ai_lines,\n SUM(num_sessions) AS ai_sessions,\n SUM(estimated_cost_usd) AS ai_cost\n FROM ai_activities\n WHERE $__timeFilter(date)\n AND provider IN ($provider)\n AND user_email <> ''\n AND LOWER(user_email) IN (SELECT email FROM member_emails)\n GROUP BY 1\n),\ndelivery AS (\n SELECT DATE_SUB(DATE(pr_merged_date), INTERVAL WEEKDAY(pr_merged_date) DAY) AS week_start,\n COUNT(*) AS prs_merged,\n AVG(pr_cycle_time) / 60.0 AS avg_cycle_hours\n FROM project_pr_metrics\n WHERE $__timeFilter(pr_merged_date)\n AND project_name IN ($project)\n GROUP BY 1\n)\nSELECT ai.week_start AS time,\n ai.ai_users AS \"AI Active Users\",\n d.prs_merged AS \"PRs Merged\",\n ROUND(d.avg_cycle_hours, 1) AS \"Avg PR Cycle Time (h)\"\nFROM ai\nINNER JOIN delivery d ON d.week_start = ai.week_start\nORDER BY 1", |
There was a problem hiding this comment.
Since the point of this panel is correlating the two series, the INNER JOIN on week quietly drops any week that had AI usage but no merged PRs (and the reverse) — which is exactly the low-delivery signal you'd want visible. Same in the fact table just below. A small week spine with LEFT JOINs and COALESCE(..., 0) would keep those weeks on the chart.
| }, | ||
| { | ||
| "datasource": "mysql", | ||
| "description": "AI usage next to that same person's delivery. Blank PR columns mean identity did not match, NOT that the developer opened no PRs \u2014 check 'Identity' and the gaps table below. 'Accounts Merged' is how many `accounts` rows were treated as this person; PRs are summed across all of them, so someone active on two platforms is not undercounted.", |
There was a problem hiding this comment.
Couple of notes on this one. ai_agg is already filtered to member_emails, and membership requires having authored a PR, so every row that survives has at least one account — which means the 'UNMATCHED' branch in identity can't actually fire, and the description's "blank PR columns mean identity did not match" won't happen for that reason. A blank now only comes from the project/time filters, so the wording is a little misleading. Also prs_opened/prs_merged arrive through a LEFT JOIN and render blank rather than 0 — a COALESCE would make "active in AI, nothing merged" read as a real zero instead of looking like missing data.
| pm.project_name AS project_name, | ||
| pa.email AS email | ||
| FROM {{ ref('mart_ai_person_accounts') }} pa | ||
| JOIN {{ source('devlake', 'pull_requests') }} pr |
There was a problem hiding this comment.
This defines a project member as "authored a PR in the project's repos," resolved through mart_ai_person_accounts. The overview and weekly panels in the dashboard still define membership inline with a different join, so the two can hand back different member sets for the same project. Having the dashboard lean on this mart everywhere would give a single source of truth for who counts as a member.
| ## How it runs | ||
|
|
||
| Executed by DevLake's **dbt plugin**. Wire it into a project's blueprint | ||
| `afterPlan` so it runs **after** github/org/dora populate accounts, PRs and |
There was a problem hiding this comment.
Might be worth stating the operational dependency more loudly somewhere the dashboard user will see it: panels 11, 20 and 40–42 return nothing until this dbt project has run, and it only runs if someone wires it into a blueprint's afterPlan — nothing triggers it automatically. On a fresh deployment that leaves a half-populated dashboard (inline-CTE panels have data, mart panels are blank), which is a confusing state to land in and debug. A note on the dashboard itself, or a graceful fallback, would save someone that.
Extend the AI-delivery dashboard from velocity to quality via three new dbt marts (membership-proxy, zero core edits, rebase-safe) and four panels: - mart_project_bugs_weekly: bugs opened/resolved per project-week, read from the domain layer (issues/board_issues) so it is tracker-agnostic (Jira, GitHub Issues, GitLab, TAPD, Zentao, PagerDuty, ...). - mart_project_churn_weekly: quality-variance engine — churn/rework split by whether the commit author used AI that week (adopter vs non-adopter). - mart_project_sonar_debt: technical debt sourced solely from SonarQube (SQALE, code smells, reliability), per-project snapshot. Panels: Bug Correlation (table + dual-axis trend), Quality Variance, Technical Debt (SonarQube) vs AI Usage. Every panel states membership-scope-not-attribution. Co-Authored-By: Saqib Kamran <saqib.kamran@arbisoft.com>
Surface SUM(input_tokens)/SUM(output_tokens) from ai_activities as two new overview stat tiles and as columns in the weekly, developer, project, project×week, and project×provider fact tables. Purely additive Grafana SQL; same membership scoping as existing AI metrics. Co-Authored-By: Saqib Kamran <saqib.kamran@arbisoft.com>
3 participants
pr-type/bug-fix,pr-type/feature-development, etc.Summary
AI adoption and DORA delivery metrics were only visible in separate dashboards with no relational join, so we could not ask "in weeks where AI usage rose, what happened to delivery?" in a single query.
This adds a Grafana dashboard that joins
ai_activitieswithproject_pr_metricson calendar week, developer identity, and project, and fixes the account resolution that the developer join depends on. It also precomputes the identity/membership joins as dbt marts so the resolution runs once per sync instead of on every panel refresh.No schema change, no core DevLake change. Both domain tables already exist and four plugins already write to
ai_activities, so the dashboard is presentation only — no migrations, no domain-model changes, nothing undercore/models/. The dbt marts run via the existing dbt plugin, so they add no code to core either.Bug fixes included
All four AI plugins resolved a developer's account with an unordered
First(WHERE email = ?). One person routinely owns several rows inaccounts— gitextractor mints one keyed by the commit author email alongside the source plugin'sgithub:GithubAccount:*row — so the match was arbitrary. When it picked the gitextractor twin, which authors no pull requests, AI activity was attributed to an account with no delivery history: a wrong number rather than a missing one.gh-copilotwas worse. It queriedWHERE name = ?, butaccountshas nonamecolumn (onlyuser_name/full_name). Every call errored, the error was swallowed byreturn "", and it never resolved anybody.Both are replaced by
helpers/identityhelper.AccountResolver, which gathers every candidate account and prefers, in order:users→user_accounts)It also aggregates PR counts in SQL rather than in Go, so a large install does not materialise one row per pull request, and loads its lookup tables once per subtask instead of once per row.
Behaviours the dashboard SQL has to account for
ai_activitiesis written at two grains under identicalprovider/type— cursor writes both a daily summary and per-event rows. On the test dataset that is 292 rows for 95 real user-days. The metric columns are disjoint, so panels aggregate columns and neverCOUNT(*).pr_cycle_timeis in minutes, not seconds (change_lead_time_calculator.go:math.Ceil(span.Minutes())). Reading it as seconds would be wrong by 60×.ai_activitieshas no repo or project column, so AI totals are never split across projects — splitting would be invented precision.Project-wise rollup (membership scope)
Adds project-scoped panels on top of the week/developer join, honest about what today's schema supports:
ai_activitiesis the unified domain table every AI plugin writes into; the existing$providerfilter still applies.The two Identity-health panels stay global — they are hygiene diagnostics, not project metrics.
Performance: precomputed identity & membership marts (dbt)
Identity resolution was re-run at query time in every panel (
ai_activities → accounts → users → user_accounts → pull_requests → project_mapping), on every refresh, by every viewer — wasteful at scale for a result that only changes per sync.A new dbt project (
dbt/ai_delivery/) materialises the two static joins:mart_ai_person_accounts— every DevLake account belonging to each AI user's email (so delivery sums across all of a person's accounts).mart_ai_project_members— AI users who authored a PR in each project's repos.The five data panels now read these marts instead of the inline CTEs. Metric aggregation stays query-time to keep full time-picker flexibility.
Conflict-free by design. The marts run via the existing dbt plugin, wired into a blueprint's
afterPlanso they refresh after github/org/dora populateaccounts/pull_requests/project_mapping— with zero edits undercore/models/, no migration registry, and nodomaininfo.go. Keeps the fork cleanly rebasable.Does this close any open issues?
N/A
Other Information
Testing
Dashboard is at
/grafana/d/ai_delivery_correlation/. Verified against a real project (12 PRs, 8 DORA metric rows, 10 accounts) with synthetic AI usage, plus a second synthetic project to exercise cross-project membership.go build ./...clean,go vetclean on all changed packagesidentityhelper, covering each resolution rule plus the gitextractor-duplicate regressiondbt runclean); the rewritten panels produce byte-identical output to the previous query-time version, verified by diffing result sets (including the Developer table that uses the identity mart)Rebase surface
backend/helpers/identityhelper/, the dashboard JSON, and thedbt/ai_delivery/project are new files and cannot conflict. The six plugin converters are ~7 added lines each, all of the same shape (one import, one resolver construction, one call site), so reapplying after an upstream conflict is mechanical.