Enhance Copilot Review Analyst skill: v3 newsletter report + adoption-signal & fairness methodology, Fixes AB#3696013 - #449
Conversation
…-signal & fairness methodology, Fixes AB#3696013 Report / rendering: - v3 newsletter Outlook report: 1200px hybrid-fluid shell, edge-to-edge newsletter masthead (inset hero + 1120px VML), big-number KPI hero grid, single-row meta-chips, Methodology section - Headline now leads with adoption % (Helpful & adopted), not precision - Document Outlook rendering learnings in report-formatting.md: single-row chip table (not inline-block), mso-line-height-rule:exactly for prose, edge-to-edge frame (whiten bgcolor AND inline CSS), inset masthead + VML width, Graph-draft delivery with Outlook COM fallback Classification / methodology: - Declined verdict + Copilot precision (Helpful / (Helpful + Incorrect)); Declined is neutral - Force-push confound validation before scoring precision - Engineer-intention nuance: correct-but-declined feedback is not a Copilot error - New adoption signals: reaction + apply-suggestion detection, merged-only PR filter, Phase 0 preflight coverage audit, review-coverage split, self-consistency gate, PR triage table Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4fbe65f-9d71-4531-b63e-b5a12b21fba4
github-actions
Bot
changed the title
|
✅ Work item link check complete. Description contains link AB#3696013 to an Azure Boards work item. |
1 similar comment
|
✅ Work item link check complete. Description contains link AB#3696013 to an Azure Boards work item. |
…-signal & fairness methodology, Fixes AB#3696013 Report / rendering: - v3 newsletter Outlook report: 1200px hybrid-fluid shell, edge-to-edge newsletter masthead (inset hero + 1120px VML), big-number KPI hero grid, single-row meta-chips, Methodology section - Headline now leads with adoption % (Helpful & adopted), not precision - Document Outlook rendering learnings in report-formatting.md: single-row chip table (not inline-block), mso-line-height-rule:exactly for prose, edge-to-edge frame (whiten bgcolor AND inline CSS), inset masthead + VML width, Graph-draft delivery with Outlook COM fallback Classification / methodology: - Declined verdict + Copilot precision (Helpful / (Helpful + Incorrect)); Declined is neutral - Force-push confound validation before scoring precision - Engineer-intention nuance: correct-but-declined feedback is not a Copilot error - New adoption signals: reaction + apply-suggestion detection, merged-only PR filter, Phase 0 preflight coverage audit, review-coverage split, self-consistency gate, PR triage table Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4fbe65f-9d71-4531-b63e-b5a12b21fba4
…-microsoft-copilot-review-analysis
…-signal & fairness methodology, Fixes AB#3696013 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4fbe65f-9d71-4531-b63e-b5a12b21fba4
…-microsoft-copilot-review-analysis
Removes the AuthClientAndroidPipelines and telemetry-platform ignore entries that were inadvertently carried over from earlier branch work; they are not part of the copilot-review-analyst skill changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4fbe65f-9d71-4531-b63e-b5a12b21fba4
|
PR Body mentioned a .gitignore addition, but i'm not seeing it in the files, is it stale? |
| notHelpful = [ordered]@{ count = $confirmedNotHelpful; pct = $notHelpfulPct } | ||
| notHelpfulIncorrect = [ordered]@{ count = $confirmedNotHelpfulIncorrect; pct = $notHelpfulIncorrectPct } | ||
| declined = [ordered]@{ count = $declinedCount; pct = $declinedPct } | ||
| unresolved = [ordered]@{ count = $unresolved; pct = $unresolvedPct } |
There was a problem hiding this comment.
unknown verdict is dropped from the snapshot - buckets won't reconcile to total.
$unknownCount is computed but never added to $snapshot. The emitted buckets are helpful + notHelpfulIncorrect + declined + unresolved, so any unknown comment makes them sum to less than total.
unknown does occur: in final-classification.ps1, a replied comment absent from ReplyVerdictsFile/OverridesFile stays "unknown", and that script's self-consistency gate treats unknown as a canonical bucket — so omitting it here makes the history inconsistent with the classification output.
There was a problem hiding this comment.
Good catch — fixed in 2a03435. $unknownCount was computed but never written to the snapshot, so any comment that stayed unknown made the additive buckets under-sum vs total and left history.json inconsistent with final-classification.ps1's canonical taxonomy.
The snapshot now emits an unknown = { count, pct } bucket (after unresolved), prints an Unknown line in the console summary, and I added a self-consistency assertion that warns if helpful + incorrect + declined + unresolved + unknown doesn't reconcile to total. (notHelpful stays a derived combined field, intentionally excluded from that sum.) Verified with a synthetic run containing an unknown — buckets reconcile 5/5, no warning.
append-history.ps1 computed $unknownCount but never wrote it to the
snapshot, so whenever any comment stayed 'unknown' the additive buckets
(helpful + incorrect + declined + unresolved) summed to less than total.
That made history.json inconsistent with final-classification.ps1, which
treats 'unknown' as a canonical bucket in its sum-to-total gate.
- Add unknown = { count, pct } to the emitted snapshot (after unresolved)
- Compute $unknownPct alongside the other bucket percentages
- Print an Unknown line in the console summary for parity
- Add a self-consistency assertion that warns if the five canonical
additive buckets do not reconcile to total
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d4fbe65f-9d71-4531-b63e-b5a12b21fba4
|
❌ Work item link check failed. Description does not contain AB#{ID}. Click here to Learn more. |
|
@fadidurah you're right, it was stale. The |
3 participants
Summary
Bakes this cycle's learnings into the copilot-review-analyst skill (
.github/skills/copilot-review-analyst). Two themes: a redesigned v3 newsletter report + hardened Outlook rendering, and a stronger, fairer classification methodology.Fixes AB#3696013
Report / rendering
assets/…-Outlook.html): 1200px hybrid-fluid shell, edge-to-edge newsletter masthead (inset hero + 1120px VML width), big-number KPI hero grid, single-row meta-chips, and a Methodology section. Committed as the anonymized sample, single-quoted (escaping-free).references/report-formatting.md§ v3 Newsletter Rendering:inline-blocktables stack vertically in classic Outlook.line-heightplusmso-line-height-rule:exactlyor the Word engine collapses leading.bgcolorand inline CSS (Word engine readsbgcolor).Classification / methodology
Helpful / (Helpful + Incorrect); Declined is neutral and never counted against Copilot.Notes
.md/.htmlassets remain structural references (illustrative prior-run data — computed fresh each run).