[experiments] Daily Experiment Report — 2026-06-13

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-13

39 experiments across 36 workflows. 6 reached min_samples; 2 ABANDON (no detectable effect, p ≥ 0.05); 37 EXTEND. Outcome stats available for smoke-copilot (29 correlated runs).

⚡ Quick Stats

Metric	Value
Active experiments	39
Total tracked runs	~1,095
Reached min_samples	6
Statistically tested	2 (smoke-copilot)
Imbalanced distributions	4
✅ PROMOTE · ❌ ABANDON · 🟡 EXTEND	0 · 2 · 37

---

🟢 Ready for Analysis

dailyastrostylelitemarkdownspellcheck / prompt_style 🟡

Variants: 41 runs · ✅ balanced (p=0.8759) · min_samples=20

Variant	n	Succ%	Mean dur	p-value	Progress
detailed	20	N/A	N/A	(ref)	████████ 20/20 (100%)
concise	21	N/A	N/A	N/A	████████ 21/20 (100%)

🟡 EXTEND — All min_samples reached; awaiting outcome correlation for final determination

smoke-antigravity / sub_agent_strategy 🟡

Variants: 84 runs · ✅ balanced (p=0.8273) · min_samples=30

Variant	n	Succ%	Mean dur	p-value	Progress
single_agent	41	N/A	N/A	(ref)	████████ 41/30 (100%)
sub_agents	43	N/A	N/A	N/A	████████ 43/30 (100%)

🟡 EXTEND — All min_samples reached; awaiting outcome correlation for final determination

smoke-gemini / sub_agent_strategy 🟡

Variants: 113 runs · ⚠️ IMBALANCED (p=0.0064) · min_samples=30

Variant	n	Succ%	Mean dur	p-value	Progress
single_agent	42	N/A	N/A	(ref)	████████ 42/30 (100%)
sub_agents	71	N/A	N/A	N/A	████████ 71/30 (100%)

🟡 EXTEND — Imbalanced (p_balance=0.0064) — gather more data before outcome test

smokepi / sub_agent_decomposition 🟡

Variants: 76 runs · ✅ balanced (p=0.3588) · min_samples=20

Variant	n	Succ%	Mean dur	p-value	Progress
single_agent	34	N/A	N/A	(ref)	████████ 34/20 (100%)
parallel_sub_agents	42	N/A	N/A	N/A	████████ 42/20 (100%)

🟡 EXTEND — All min_samples reached; awaiting outcome correlation for final determination

smokecopilot / caveman ❌

Variants: 150 runs · ✅ balanced (p=1.0000) · min_samples=20

Variant	n	Succ%	Mean dur	p-value	Progress
no	75	33.3%	904s	(ref)	████████ 75/20 (100%)
yes	75	21.4%	921s	0.4735	████████ 75/20 (100%)

❌ ABANDON — p=0.4735 ≥ 0.05, all variants ≥ min_samples — no detectable effect

smokecopilot / subagent_model ❌

Variants: 130 runs · ✅ balanced (p=1.0000) · min_samples=20

Variant	n	Succ%	Mean dur	p-value	Progress
large	65	26.7%	849s	(ref)	████████ 65/20 (100%)
small	65	28.6%	980s	0.9087	████████ 65/20 (100%)

❌ ABANDON — p=0.9087 ≥ 0.05, all variants ≥ min_samples — no detectable effect

---

🟡 Collecting Data

37 experiments still gathering samples. Top 10 by progress:

Experiment	Workflow	Lowest n / min	%	Balanced
prompt_style	dailycommunityattribut	19/20	95%	✅
caveman	smokecopilotaoaiapikey	16/20	80%	✅
subagent_model	smokecopilotaoaiapikey	16/20	80%	✅
output_format	daily-code-metrics	13/20	65%	✅
prompt_compression	agent-performance-anal	9/14	64%	✅
output_format	deep-report	9/15	60%	✅
tone_style	typist	6/10	60%	✅
output_format	daily-issues-report	17/30	56%	✅
reasoning_depth	daily-security-red-tea	14/30	46%	✅
output_format	daily-compiler-quality	9/20	45%	✅

View all 33 collecting experiments

Experiment	Workflow	Total	Low n/min	Bal
prompt_style	dailycommunityattrib	39	19/20	✅
caveman	smokecopilotaoaiapik	32	16/20	✅
subagent_model	smokecopilotaoaiapik	32	16/20	✅
output_format	daily-code-metrics	27	13/20	✅
prompt_compression	agent-performance-an	24	9/14	✅
output_format	deep-report	29	9/15	✅
tone_style	typist	14	6/10	✅
output_format	daily-issues-report	37	17/30	✅
reasoning_depth	daily-security-red-t	32	14/30	✅
output_format	daily-compiler-quali	28	9/20	✅
sub_agent_strategy	agent-persona-explor	24	6/14	⚠️
prompt_style	ci-coach	21	8/20	✅
prompt_style	daily-news	32	12/30	✅
semgrep_output_format	daily-semgrep-scan	27	9/30	✅
tone_variant	aw-failure-investiga	50	14/50	✅
tool_verbosity	gpclean	20	8/30	✅
detail_level	daily-architecture-d	2	2/10	✅
reasoning_depth	daily-fact	22	6/30	⚠️
caveman	smokecopilotaoaientr	7	3/20	✅
subagent_model	smokecopilotaoaientr	7	3/20	✅
model_size	daily-cache-strategy	4	2/20	✅
model_size	daily-caveman-optimi	6	2/20	✅
model_size	daily-doc-healer	6	2/20	✅
model_size	daily-doc-updater	5	2/20	✅
model_size	daily-function-namer	6	2/20	✅
caveman_mode	dataflow-pr-discussi	1	1/10	✅
summary_detail	dependabot-campaign	6	3/30	✅
output_format	copilot-agent-analys	5	2/30	✅
prompt_style	issue-arborist	4	2/30	✅
sub_agent_strategy	daily-agentrx-trace-	9	1/20	⚠️
log_fetch_strategy	daily-safe-output-op	2	1/20	✅
timeout_setting	dailysubagentoptimiz	4	1/20	✅
prompt_style	dependabot-go-checke	7	1/30	✅

---

State data from experiments/* git branches · Outcome data: smoke-copilot.yml (29 runs correlated)
Run: 27462107007

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 1.8K AIC · ⌖ 32 AIC · ⊞ 23.2K · ◷

• expires on Jun 16, 2026, 1:13 AM UTC-08:00