[experiments] Daily Experiment Report — 2026-06-23

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-23

Analyzed 41 experiments across 38 workflows. 10 reached min_samples (READY); 31 are collecting. 6 show assignment imbalance (χ2 p<0.05).

⚠️ Outcome metrics (success rate, duration) require authenticated GitHub API — unavailable this run. Analysis covers variant distribution and balance only.

⚡ Quick Stats

Metric	Value
Experiments	41
Workflows	38
🟢 READY for analysis	10
🟡 EXTEND (collecting)	31
⚠️ Imbalanced (p<0.05)	6

---

🟢 Ready for Analysis (10 experiments)

caveman · smoke-copilot

🟢 READY · no vs yes · 201 total runs · min_samples=20 · ✅ balanced

📊 View chart & stats

Variant	n	Progress
no	100	████████ 100/20 (100%)
yes	101	████████ 101/20 (100%)

subagent_model · smoke-copilot

🟢 READY · large vs small · 181 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
large	91	████████ 91/20 (100%)
small	90	████████ 90/20 (100%)

sub_agent_strategy · smoke-gemini

🟢 READY · single_agent vs sub_agents · 157 total runs · min_samples=30 · ⚠️ IMBALANCED p=0.0053

📊 View chart & stats

Variant	n	Progress
single_agent	61	████████ 61/30 (100%)
sub_agents	96	████████ 96/30 (100%)

sub_agent_strategy · smoke-antigravity

🟢 READY · single_agent vs sub_agents · 128 total runs · min_samples=30 · ✅ balanced

Variant	n	Progress
single_agent	62	████████ 62/30 (100%)
sub_agents	66	████████ 66/30 (100%)

caveman · smoke-copilot-aoai-apikey

🟢 READY · no vs yes · 76 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
no	38	████████ 38/20 (100%)
yes	38	████████ 38/20 (100%)

subagent_model · smoke-copilot-aoai-apikey

🟢 READY · large vs small · 76 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
large	38	████████ 38/20 (100%)
small	38	████████ 38/20 (100%)

sub_agent_decomposition · smoke-pi

🟢 READY · parallel_sub_agents vs single_agent · 76 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
parallel_sub_agents	42	████████ 42/20 (100%)
single_agent	34	████████ 34/20 (100%)

prompt_style · daily-community-attribution

🟢 READY · concise vs verbose · 49 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
concise	25	████████ 25/20 (100%)
verbose	24	████████ 24/20 (100%)

caveman · smoke-copilot-aoai-entra

🟢 READY · no vs yes · 48 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
no	24	████████ 24/20 (100%)
yes	24	████████ 24/20 (100%)

subagent_model · smoke-copilot-aoai-entra

🟢 READY · large vs small · 48 total runs · min_samples=20 · ✅ balanced

Variant	n	Progress
large	24	████████ 24/20 (100%)
small	24	████████ 24/20 (100%)

---

🟡 Collecting Data (31 experiments)

🏁 Nearly Ready (≥70%)

Experiment	Workflow	Runs	Lowest Variant Progress	Issue
prompt_compression	agent-performance-analyzer	33	███████░ 12/14 (86%)	#33280
output_format	daily-code-metrics	37	███████░ 17/20 (85%)	#1
prompt_style	daily-astrostylelite-markdown-spellcheck	51	██████░░ 24/30 (80%)	—
tone_style	typist	20	██████░░ 8/10 (80%)	#34032
sub_agent_strategy	agent-persona-explorer	34	██████░░ 11/14 (79%)	—
output_format	daily-compiler-quality	38	██████░░ 15/20 (75%)	#32390
output_format	daily-issues-report	48	██████░░ 22/30 (73%)	#30573
output_format	deep-report	35	██████░░ 11/15 (73%)	—

📋 Full Experiments Table (all 41)

Experiment	Workflow	Runs	min_samples	Balance	Status
caveman	smoke-copilot	201	20	✅	🟢 READY_FOR_ANALYSIS
subagent_model	smoke-copilot	181	20	✅	🟢 READY_FOR_ANALYSIS
sub_agent_strategy	smoke-gemini	157	30	⚠️ p=0.0053	🟢 READY_FOR_ANALYSIS
sub_agent_strategy	smoke-antigravity	128	30	✅	🟢 READY_FOR_ANALYSIS
tone_variant	aw-failure-investigator	89	50	✅	🟡 EXTEND #36105
caveman	smoke-copilot-aoai-apikey	76	20	✅	🟢 READY_FOR_ANALYSIS
subagent_model	smoke-copilot-aoai-apikey	76	20	✅	🟢 READY_FOR_ANALYSIS
sub_agent_decomposition	smoke-pi	76	20	✅	🟢 READY_FOR_ANALYSIS
prompt_style	daily-astrostylelite-markdown-spellcheck	51	30	✅	🟡 EXTEND
prompt_style	daily-community-attribution	49	20	✅	🟢 READY_FOR_ANALYSIS
output_format	daily-issues-report	48	30	✅	🟡 EXTEND #30573
caveman	smoke-copilot-aoai-entra	48	20	✅	🟢 READY_FOR_ANALYSIS
subagent_model	smoke-copilot-aoai-entra	48	20	✅	🟢 READY_FOR_ANALYSIS
prompt_style	daily-news	43	30	✅	🟡 EXTEND #31190
reasoning_depth	daily-security-red-team	42	30	✅	🟡 EXTEND #31673
output_format	daily-compiler-quality	38	20	✅	🟡 EXTEND #32390
output_format	daily-code-metrics	37	20	✅	🟡 EXTEND #1
semgrep_output_format	daily-semgrep-scan	37	30	✅	🟡 EXTEND #32795
sub_agent_strategy	agent-persona-explorer	34	14	⚠️ p=0.0374	🟡 EXTEND
output_format	deep-report	35	15	✅	🟡 EXTEND
prompt_compression	agent-performance-analyzer	33	14	✅	🟡 EXTEND #33280
tool_verbosity	gpclean	30	30	✅	🟡 EXTEND
reasoning_depth	daily-fact	28	30	✅	🟡 EXTEND #31324
prompt_style	ci-coach	27	20	✅	🟡 EXTEND #32335
tone_style	typist	20	10	✅	🟡 EXTEND #34032
model_size	daily-caveman-optimizer	19	20	⚠️ p=0.0	🟡 EXTEND
model_size	daily-doc-healer	19	20	⚠️ p=0.0	🟡 EXTEND
model_size	daily-doc-updater	19	20	⚠️ p=0.0	🟡 EXTEND
model_size	daily-cache-strategy-analyzer	18	20	⚠️ p=0.0	🟡 EXTEND
sub_agent_strategy	daily-agentrx-trace-optimizer	16	20	✅	🟡 EXTEND
output_format	copilot-agent-analysis	15	30	✅	🟡 EXTEND
summary_detail	dependabotcampaign	12	20	✅	🟡 EXTEND
prompt_style	dependabot-go-checker	11	30	✅	🟡 EXTEND
log_fetch_strategy	daily-safe-output-optimizer	9	20	✅	🟡 EXTEND
sub_agent_strategy	architecture-guardian	6	30	✅	🟡 EXTEND #39062
detail_level	daily-architecture-diagram	4	10	✅	🟡 EXTEND #31926
timeout_setting	dailysubagentoptimizer	4	20	✅	🟡 EXTEND
prompt_style	issue-arborist	4	30	✅	🟡 EXTEND #30015
model_size	daily-function-namer	3	20	✅	🟡 EXTEND
caveman_mode	dataflow-pr-discussion-dataset	2	10	✅	🟡 EXTEND #37102
prefetch_strategy	weekly-blog-post-writer	2	10	✅	🟡 EXTEND #38590

Analysis window: all runs in experiment state branches · Balance p<0.05 (χ2)
§28014244690

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 206.5 AIC · ⌖ 24.5 AIC · ⊞ 11.2K · ◷

• expires on Jun 26, 2026, 1:20 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #41203.