[Labelling Health] Labelling Health Report — 2026-05-26

[mnkiefer]

Summary

Status: Regressing

Auto-labelling continues to run daily without failures, but the correction feedback loop has fully stalled: the Labelling Correction Collector has not executed in over 30 days, no new correction signals have been filed in that period, and 333 open signals carrying 851 total correction events remain unresolved — with the oldest approaching 7 weeks of age.

---

Key Metrics

Metric	Value
Discussions reviewed — last 7 days	Unknown — no auto-labelling summary issues were found; the Label Discussions workflow ran 7× successfully but produced no parseable daily issue
Label changes applied — last 7 days	Unknown — same cause; best estimate is that changes occurred but volume is unavailable
Change rate (changed / reviewed) — last 7 days	Not calculable
Correction-collector runs — last 7 days	0 (also 0 in last 30 days)
Open correction signals	333
New correction signals — last 7 days	0
New correction signals — last 30 days	0
Oldest open correction signal	47 days (issue #47, created 2026-04-08)

Note on reviewed/changed counts: The auto_labelling_summaries array was empty — either daily summary issues were not created, were not discoverable by the data collector, or were not parsed. The absence of this data is itself a signal worth investigating.

---

Correction Pressure

All 333 open signals predate the 30-day window (latest was created 2026-04-18). The Labelling Correction Collector has not run at all in the last 30 days, so these represent an entirely static, unprocessed backlog rather than ongoing inflow.

Top categories by open signal count:

Category	Open Signals
Copilot Conversations	110
GitHub Education	71
Other Feature Feedback, Questions, & Ideas	52
Apps, API and Webhooks	21
Mobile	14
Discussions	13
Actions	10

Top labels by correction event (latest_label on open signals):

Label	Count
Copilot	56
Copilot in GitHub	42
GitHub Education	38
bug	37
Other Features and Feedback	20
Profile	10
Mobile	10
Apps API and Webhooks	10

Event type split: 197 labeled corrections vs 136 unlabeled corrections — meaning roughly 59 % of signals indicate a label was added by a trusted actor (auto-labeller missed it), and 41 % indicate a label was removed (auto-labeller over-applied it).

Pressure is concentrated in two clusters:

1. Copilot / Copilot in GitHub confusion — 110 signals in the Copilot Conversations category, with Copilot (56) and Copilot in GitHub (42) as the most-corrected labels. This likely reflects an ambiguous boundary in .github/instructions/community-discussion-labeling.md between the two labels.
2. GitHub Education classification — 71 signals, with GitHub Education (38) and bug (37) as top corrections; the bug pressure in an Education context suggests systematic mislabelling of support/eligibility questions as bugs.

Highest-correction individual signals (multi-event)

Issue	Corrections	Category	Latest Label
#114	19	Other Feature Feedback, Questions, & Ideas	Code Search and Navigation
#69	17	GitHub Education	question
#63	17	Other Feature Feedback, Questions, & Ideas	Profile
#55	17	GitHub Education	GitHub Education Benefits
#51	17	Other Feature Feedback, Questions, & Ideas	Code Search and Navigation
#109	16	Copilot Conversations	Copilot
#75	15	GitHub Education	GitHub Education
#118	14	Apps, API and Webhooks	Apps API and Webhooks
#77	14	Copilot Conversations	Copilot
#73	14	Other Feature Feedback, Questions, & Ideas	Copilot

Signals #114, #51, #63 (Code Search / Profile applied to general feedback) and #109, #77, #73 (Copilot in Copilot Conversations) represent the highest-density instruction ambiguity per discussion.

---

Open Instruction Debt

The correction backlog is static and growing in age. No signals were resolved (only 2 of 335 total are closed), and no new signals are being collected. The backlog contains 851 total correction events across 333 open signals — an average of ~2.6 corrections per discussion.

• 4 open parent intake issues (#45, #46, #49, #50), all titled "Batch 02", all created 2026-04-08 and never updated.
• 1 closed parent (#26, Batch 01, closed 2026-04-03) — the only batch that was ever resolved.
• No Batch 03 or later exists, suggesting the intake pipeline stopped after Batch 02 was created.
• The oldest open signal (#47) is 47 days old. At current resolution rate (≈0 signals/day), the backlog will not self-resolve.

The backlog appears stale rather than actively growing — the collector stopped, so no new signals are being ingested, but the existing debt is also not being worked down.

Open correction signal sample (first 20 by issue number)

Issue	Title	Created	Corrections	Category
#47	Discussion #186: How do you automate workflows...	2026-04-08	—	(oldest)
#49	Batch 02 parent	2026-04-08	—	intake
#50	Batch 02 parent	2026-04-08	—	intake
#51	Discussion #...: Code Search	2026-04-08	17	Other Feature Feedback
#55	Discussion #...: GitHub Education Benefits	2026-04-08	17	GitHub Education
#63	Discussion #...: Profile	2026-04-08	17	Other Feature Feedback
#69	Discussion #...: question	2026-04-08	17	GitHub Education
#73	Discussion #...: Copilot	2026-04-08	14	Other Feature Feedback
#75	Discussion #...: GitHub Education	2026-04-08	15	GitHub Education
#77	Discussion #...: Copilot	2026-04-08	14	Copilot Conversations
#109	Discussion #...: Copilot	2026-04-08	16	Copilot Conversations
#114	Discussion #...: Code Search and Navigation	2026-04-08	19	Other Feature Feedback
#118	Discussion #...: Apps API and Webhooks	2026-04-08	14	Apps, API and Webhooks
#400	Discussion #266: UX for large-scale repos	2026-04-17	1	Other Feature Feedback
#404	Discussion #279: Weekly limit reached	2026-04-18	1	A Welcome to GitHub

---

Recommendations

1. Investigate and restart the Labelling Correction Collector. It has not run in 30+ days. Check the workflow's schedule, trigger conditions, and any failing pre-conditions. Without it running, no new trusted correction signals can enter the pipeline and the backlog cannot grow — but neither can it be properly characterized.

2. Triage the Copilot / Copilot in GitHub ambiguity in .github/instructions/community-discussion-labeling.md. With 110 signals in Copilot Conversations and Copilot + Copilot in GitHub as the top two corrected labels (98 combined), the labelling rule for distinguishing these two is either missing or unclear. Adding an explicit decision rule with examples would reduce correction pressure in this category.

3. Review the GitHub Education + bug mislabelling pattern. 71 Education signals and 37 bug-label corrections suggest eligibility/support questions are being tagged as bugs. Tighten the bug label criteria in the instructions to explicitly exclude program-eligibility and access questions.

4. Close or batch-triage the 4 open "Batch 02" parent intake issues (#45, #46, #49, #50). They have not been updated since creation on 2026-04-08. Either process the child signals under each parent or close the parents with a tracking note if the Batch 02 approach is superseded. Leaving them open inflates the apparent backlog and obscures the real resolution rate.

---

References

Recent workflow run references

Workflow	Last 7 days	Last 30 days	Notes
Label Discussions	7 runs, all success	30 runs	Running normally
Labelling Correction Collector	0 runs	0 runs	⚠️ Not executing
Labelling Correction Feedback	6 runs, all skipped	28 runs	Skipping due to no new signals

• §82 — Label Discussions (latest, 2026-05-25, success)
• §420 — Labelling Correction Feedback (latest, 2026-05-25, skipped)
• §419 — Labelling Correction Feedback (2026-05-24, skipped)

---

Generated automatically on 2026-05-26. Data window: last 7 days (2026-05-19 → 2026-05-26), previous 7 days (2026-05-12 → 2026-05-19), last 30 days (2026-04-26 → 2026-05-26).

Generated by Labelling Health Report · ● 357.5K · ◷

• expires on Jun 25, 2026, 3:38 AM UTC

[mnkiefer]

This issue is being closed as outdated. A newer issue has been created: #444

View newer issue

---

This action was performed automatically by the Labelling Health Report workflow.