ADR 007 — Few-shot Route Retired, CIO Remains Zero-shot

Date: 2026-05-27 Status: Accepted Related: 004-v1-fallback-retirement · agents/cio.py (docstring records v2 not integrated) · experiments/archive_few_shot_failed/

Background

To make the CIO less overly conservative, an attempt was made to inject few-shot demos into the CIO using DSPy (learning "when to act" from a trainset labeled by the historical oracle). Four versions were implemented:

Version	Trainset	Result
v1	66 samples, single regime 2024-05~11	Verdict collapse + 6 structural issues including symbol loss, never integrated
v2 / v2.2	5565 samples, cross-regime 2024-05~2026-05, 5 verdicts	MIPROv2 optimization = +0.0pp (baseline 0.67 near ceiling); random demos reduced REGIME mention in reasoning from 64% to 1.8%
v3	v2 relabeled by path_c alpha-oracle	Collapsed to 3 tiers (BUY 52% / ACC,TRIM=0%); sandbox holdout (2025-11~2026-02, 200 samples): optimized −1pp vs zero-shot, −5pp vs random demos, alpha underperformed buy-and-hold. Conclusion: retain zero-shot
v4	Phase 1.5 (2023-24 pure bull market) relabeled by path_c	Collapse worse: BUY 59.5% / ACC,TRIM=0%. Holdout not run — see root cause below, mechanism already confirmed

Root Cause (Not a Data Issue, but an Oracle Structural Flaw)

The path_c oracle = "label the verdict maximizing portfolio terminal value over forward-30d", with alloc mapping BUY=+1.0 / ACCUMULATE=+0.5 / HOLD=0 / TRIM=−0.1 / SELL=−0.3.

• When fwd>0: BUY(1.0) terminal value is always ≥ ACCUMULATE(0.5) → ACCUMULATE is mathematically never the per-sample optimum.
• When fwd<0: Among reductions, SELL(−0.3) reduces more than TRIM(−0.1), yielding higher terminal value, and at fwd=0 / asset_pct boundaries, HOLD is the tiebreaker → TRIM is also never optimal.
• ⇒ The oracle only outputs BUY / HOLD / SELL three tiers, regardless of input data. Phase 1.5 being a pure bull market merely pushed BUY share from 52% to 59.5%.

Another independent harm of few-shot (measured in v2): random/optimized demos suppress CIO's REGIME reasoning (mention 64%→1.8%), degrading reasoning quality even if hit rate doesn't drop.

Decision

1. CIO remains zero-shot (agents/cio.py:build_cio_prompt uses pure load_skill("cio")). The verdict hit rate ceiling is empirically ~67%, which few-shot cannot break through.
2. Address conservatism via Dreaming insight injection (caution/reliable patterns, see 03-dreaming), abandoning the few-shot approach. Dreaming's missed-opportunity warnings (e.g., "HOLD in a volatile market for 30d has a high miss rate") are the correct lever to reduce excessive HOLD.
3. No further investment in few-shot / DSPy oracle retraining, unless the structural flaw in path_c is resolved first (requires redesigning the alloc model so ACCUMULATE/TRIM can be optimal in certain states, or switching to a non-terminal-value oracle objective). Until then, this path is considered proven invalid.

Evidence Trail

• v1–v3 artifacts + sandbox A/B: experiments/dspy_optimized_v*.json, experiments/sandbox/sandbox_v2_results.json (v3 holdout three-arm comparison).
• v4 artifacts: experiments/archive_few_shot_failed/ (trainset + stats, holdout not compiled).