[workshop-sim] Workshop Simulation Report — 2026-07-25 (Run #16, 1000×Monte Carlo)

[github-actions]

Overview

• Date: 2026-07-25
• Students simulated: 46 × 1,000 Monte Carlo runs
• Workshop steps available: 27/27
• Overall success rate: 19.9% (95% Monte Carlo interval: 19.6%–20.3%)
• Highest-dropout step: 07-first-workflow (29.9% conditional dropout among 20,772 at-risk runs; 95% CI: 29.3%–30.6%)
• Lowest curriculum quality step: 05-agentic-workflows-intro.md (overall score 7.36/10)
• Learning KPI index: 7.72/10 (active_learning 4.62 · checkpoint_quality 9.54 · scaffolding 9.44)
• Model: none / 2026-07-assumption-model-v2 (parameter hash none)
• Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Part Summary

Part	Files	Mean Score	Std Dev
Part 1 — core path (lessons 00–14)	15	8.85 / 10.0	±0.7 (est.)
Part 2 — advanced (lessons 15+)	12	8.78 / 10.0	±0.3 (est.)
Overall corpus	27	8.82 / 10.0	±0.5 (est.)

No steps are classified as other in this corpus.

Critical Findings

1. copilot-access-missing is the top blocker at step 07 (3,296 of 6,220 total failures): beginners and github-basic learners cannot verify or fix Copilot access before the workflow authoring step, causing 29.9% conditional dropout — the highest in the workshop. This is an access barrier in Part 1.
2. agentic-concept-gap at step 05 drives 8,164 of 10,359 failures at the most-reached step: the conceptual introduction is the densest page (7.36/10, lowest cognitive_load 2.3/10) and provides no recovery path when learners stall. Beginner and github-basic segments fail here at near-zero overall success rates (0.3% and 12.8% respectively). This is a learning barrier in Part 1.
3. UI-preferred learners are severely disadvantaged: mean success rate 10.7% vs 28.4% for CLI-preferred, driven by the required Codespace terminal path at steps 06–07 with no browser alternative for authoring. This is an access barrier concentrated in Part 1.
4. Learning KPI health is adequate but uneven: overall KPI index 7.72/10 indicates learners who persist are building skills — checkpoint quality (9.54) and scaffolding (9.44) are strong. However, active_learning (4.62) is the weakest dimension corpus-wide, and the most-failed conceptual step (05) scores only 4.5 on active learning.

Top Repairs to Prioritize

Some dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index.

1. Add a Copilot access pre-check mini-step or inline troubleshooting gate before step 07 — surface the gh copilot test and fix path earlier, before learners attempt workflow authoring (completion impact: ↑ · learning KPI impact: ↔)
2. Add explicit classification criteria and a self-correction reveal to 05-agentic-workflows-intro.md — the conceptual intro's active_learning score (4.5) is lowest in Part 1; adding answer-reveal checkboxes with correction paths would raise active_learning without lowering cognitive load (completion impact: ↑ · learning KPI impact: ↑)
3. Add a decision-rule block and misclassification recovery to 05c-agentic-workflows-practice.md — the practice step has no explicit criteria for distinguishing agentic vs non-agentic workflows; a short decision table and a "what if you got it wrong" path would reduce the 14.1% classification-gap dropout (completion impact: ↑ · learning KPI impact: ↑)

Dropout by step

Step	At-risk runs	Conditional dropout	95% CI	Failure mode	Top reason
07-first-workflow	20,772	29.9%	29.3%–30.6%	Access barrier	Copilot access missing before workflow execution
05-agentic-intro	40,017	25.9%	25.5%–26.3%	Learning barrier	Insufficient scaffolding for agentic concept gap
05c-agentic-practice	29,658	15.5%	15.0%–15.9%	Learning barrier	No explicit classification criteria for agentic vs non-agentic
04-actions-intro	44,712	10.5%	10.2%–10.8%	Learning barrier	Concept overload for learners with no Actions background
05b-agentic-security	25,074	10.1%	9.8%–10.5%	Learning barrier	Agentic security concepts too abstract without concrete examples
06-install-gh-aw	22,531	7.8%	7.5%–8.2%	Access barrier	Extension install friction
17-add-mcp-tools	12,363	4.9%	4.5%–5.3%	Learning barrier	MCP tooling friction
15-conditional-logic	13,184	4.6%	4.3%–5.0%	Learning barrier	Conditional logic friction
19-research-driven-training-node	11,269	4.3%	4.0%–4.7%	Learning barrier	Research node pattern friction
18-share-and-reuse	11,759	4.2%	3.8%–4.5%	Learning barrier	Workflow reuse complexity
24-self-hosted-runners	9,926	4.0%	3.7%–4.4%	Access barrier	Self-hosted runner environment friction
09-agentic-editing	14,011	3.4%	3.1%–3.7%	Learning barrier	Workflow editing guidance gap
02-setup	46,000	2.8%	2.7%–3.0%	Access barrier	Codespace setup friction
14b-pr-reviewer-workflow	13,531	2.6%	2.3%–2.8%	Learning barrier	Event trigger complexity
21-inline-sub-agents	10,544	2.4%	2.1%–2.7%	Learning barrier	Sub-agent pattern complexity
20-persistent-memory	10,782	2.2%	1.9%–2.5%	Learning barrier	Memory pattern friction
08b-interpret-your-run	14,317	2.1%	1.9%–2.4%	Learning barrier	Output interpretation gap
22-error-handling-and-resilience	10,296	2.1%	1.8%–2.4%	Learning barrier	Resilience pattern complexity
25-audit-and-observability	9,525	2.0%	1.7%–2.3%	Learning barrier	Audit tooling friction
26-manage-costs-and-budgets	9,339	1.8%	1.5%–2.1%	Learning barrier	Cost guardrail complexity
16-connect-data-source	12,577	1.7%	1.5%–1.9%	Learning barrier	Data source integration friction
08-run-your-workflow	14,552	1.6%	1.4%–1.8%	Access barrier	UI run guidance gap
23-ab-experiments	10,083	1.6%	1.3%–1.8%	Learning barrier	Experiment design complexity

Learning quality KPIs

Step file	Overall score	active_learning	checkpoint_quality	scaffolding	Learning KPI	Repair priority
00-welcome.md	10.0	0.0	0.0	5.0	1.36	Low (non-learning page)
01-prerequisites.md	10.0	4.2	7.5	5.0	5.62	Medium
02a-setup-codespace.md	8.05	4.8	10.0	5.0	6.75	Medium
04-github-actions-intro.md	8.88	7.2	10.0	10.0	8.98	Low
05-agentic-workflows-intro.md	7.36	4.5	10.0	10.0	8.00	High
05b-agentic-workflows-security.md	8.60	3.0	10.0	10.0	7.45	Medium
05c-agentic-workflows-practice.md	9.08	6.7	10.0	10.0	8.80	High
06-install-gh-aw.md	10.0	7.1	10.0	10.0	8.95	Low
07-your-first-workflow.md	9.34	6.9	10.0	10.0	8.87	High
07d-confirm-model-access.md	8.28	4.2	10.0	10.0	7.89	High
08-run-your-workflow.md	8.44	3.0	10.0	10.0	7.45	Medium
08b-interpret-your-run.md	8.82	4.1	10.0	10.0	7.85	Low
09-agentic-editing.md	8.66	4.5	10.0	10.0	8.00	Low
14-next-steps.md	8.70	3.5	10.0	10.0	7.64	Low
14b-pr-reviewer-workflow.md	8.48	4.9	10.0	10.0	8.15	Low
15-conditional-logic.md	8.30	3.8	10.0	10.0	7.75	Medium
16-connect-data-source.md	8.46	3.8	10.0	10.0	7.75	Low
17-add-mcp-tools.md	8.52	3.4	10.0	10.0	7.60	Medium
18-share-and-reuse.md	8.90	4.5	10.0	10.0	8.00	Low
19-research-driven-training-node.md	9.12	5.6	10.0	10.0	8.40	Low
20-persistent-memory.md	8.30	3.6	10.0	10.0	7.67	Low
21-inline-sub-agents.md	8.78	4.2	10.0	10.0	7.89	Low
22-error-handling-and-resilience.md	9.14	5.7	10.0	10.0	8.44	Low
23-ab-experiments.md	9.32	6.6	10.0	10.0	8.76	Low
24-self-hosted-runners.md	8.85	5.8	10.0	10.0	8.47	Low
25-audit-and-observability.md	9.02	5.1	10.0	10.0	8.22	Low
26-manage-costs-and-budgets.md	8.62	4.1	10.0	10.0	7.85	Low
Cohort mean	8.82	4.62	9.54	9.44	7.72	—

Curriculum quality metrics

Step file	Overall score	Lowest rubric dimension	Recommended repair focus
05-agentic-workflows-intro.md	7.36	cognitive_load (2.3)	Reduce prose density; add inline answer reveals with correction paths
02a-setup-codespace.md	8.05	scaffolding (5.0)	Add a hands-on validation step; improve scaffolding for non-Codespace users
07d-confirm-model-access.md	8.28	active_learning (4.2)	Add a mandatory access verification exercise before billing path selection
15-conditional-logic.md	8.30	active_learning (3.8)	Add a worked example learners build step-by-step
20-persistent-memory.md	8.30	active_learning (3.6)	Add an interactive memory pattern exercise
08-run-your-workflow.md	8.44	active_learning (3.0)	Add a prediction-observe cycle before the run step
16-connect-data-source.md	8.46	active_learning (3.8)	Add a data source connection exercise with expected output
14b-pr-reviewer-workflow.md	8.48	active_learning (4.9)	Add a PR-reviewer scenario exercise

Segment breakdowns

Success rate by technical level

Level	Students	Mean success rate
beginner	11	0.3%
github-basic	19	12.8%
actions-user	11	40.8%
advanced	5	44.5%

Success rate by personality

Personality	Students	Mean success rate
curious	15	18.5%
confused	6	19.3%
skeptical	7	19.3%
methodical	12	21.5%
impatient	6	21.8%

Success rate by UI preference

UI preferred	Students	Mean success rate
true	22	10.7%
false	24	28.4%

Notable student journeys (3)

Surprising success — Learner 026 (advanced/confused/devops, ui_preferred: false)
Despite a confused personality, this advanced devops engineer achieves a 56.9% success rate. Prior CI/CD and LLM tooling experience (level=advanced) overrides the confusion handicap. The main failure step is 07-first-workflow (144/1000 runs) driven by copilot-access-missing — even experienced learners hit auth friction. Methodical doc-reading compensates for confusion at conceptual steps.

Unexpected dropout — Learner 001 (github-basic/confused/data-science, ui_preferred: true)
This learner achieves 0% success across all 1,000 runs. A data scientist with basic GitHub knowledge but no Actions familiarity, they prefer UI over CLI. They fail consistently at 05-agentic-intro (267 failures) and 04-actions-intro (170 failures) — conceptual steps assuming prior Actions knowledge. The confused personality + github-basic level + UI preference + no Actions background creates compounded dropout risk across every Part 1 conceptual hurdle.

Content-gap case — Learner 015 (beginner/curious/no-coding, ui_preferred: true)
This learner (0% success, 334 failures at 05-agentic-intro) exposes a genuine content gap: the conceptual introduction assumes some mental model of automated scheduling systems. A no-coding background means the email-digest analogy is relatable, but the self-check exercises ("write one concrete difference between agentic and standard Actions") require knowledge the learner does not yet have. No recovery guidance at 05-agentic-intro means curiosity alone cannot compensate.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔬 Workshop Student Simulator · 79.6 AIC · ⌖ 8.23 AIC · ⊞ 10.5K · ◷

• expires on Jul 26, 2026, 2:34 PM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #2124

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.