17 · Closed-Loop Backtest & Committee Skill Conclusion

2026-06-27. Reproduction package + raw data: experiments/closed-loop-skill-test/. Methodology premise: adr/022-backtest-memory-contamination-and-holdout-discipline.md. ⚠ This page was revised once on 2026-06-27 – the initial conclusion was based on an execution bug (see below), now corrected.

One-Sentence Conclusion

The committee's active trading underperformed buy-and-hold on this (2025-2026 bull market) holdout period (post-fix baseline alpha ≈ −8%) – suffering from both "under-allocation" (cash sitting idle for long periods) and "reducing positions during a bull market" (selling too early), a double blow. Currently, it is not an alpha machine, but this is a starting point for optimization, not a death sentence: the root causes are clear (under-allocation + erratic position reduction) and fixable. Never use backtest alpha for marketing.

⚠ Bug in the Initial Conclusion (Must Remember)

The initial version stated "the committee de-risks (TRIM 16, lower MaxDD = defensive skill)" – wrong. The alloc_cny <= 0 guard in core/paper_trade_simulator.py swallowed all TRIM (negative alloc) as HOLD → SELL actual execution = 0, the committee never actually reduced positions. That "low MaxDD" was just the cash buffer from slow position building, not market timing. Fixed (qty_cny = abs(alloc_cny) + regression test tests/test_paper_trade_simulator_trim.py).

Before vs. After Fix (Baseline Empty Stub)

	Before Fix (buggy)	After Fix (TRIM actually executed)
BUY / SELL Executions	2 / 0	12 / 6
Total Return / Sharpe	+41.4% / 1.63	+21.8% / 1.03
alpha vs. Same-Asset Buy-Hold	+11.57% (illusion)	−8.06%

The pre-fix +11.57% was a double illusion: ① TRIM was swallowed → never sold; ② Cash was spent by day 1-2, coincidentally avoiding the later underperforming NDQ. With both illusions removed, the truth is active trading underperformed holding by 8 percentage points. (The closed-loop version has been re-run, see "Closed-Loop CI" below – the old −2.58% was based on swallowed TRIMs, now invalid.)

Root Causes (Post-Fix Attribution)

1. Under-allocation / Cash Drag: 140 HOLD + ACCUMULATE orders were too small (only 1200-6300 invested from a 100k principal) → bullets took 9 months to deploy, while most gains occurred within this window.
2. Reducing Positions in a Bull Market = Selling Too Early: With TRIM actually executed, selling during an uptrend lowered returns.

Overfitting-Avoidance Optimization Roadmap (Post-Adversarial Review)

• R0 Fix Execution Bug – ✅ Completed.
• R1 Reward Anchor Cash → Same-Asset Buy-Hold (core/backtest_reward.py): The optimizer currently rewards "sitting in cash as a safe haven"; changing the anchor prevents parameter tuning from drifting towards a defensive posture.
• R3 (Vol-Target Leg Only) Deterministic Risk Budget Sizing (before cio_parse.py clamp): Distribute capital according to volatility, directly addressing under-allocation/cash drag; most robust OOS literature, no directional betting. – ❌ Verified as failed, do not promote (see "Closed-Loop CI" below).
• Do Not Touch: R2 Momentum "No Position Reduction" Hard Gate (core = "always fully invested in bull market" = pseudo-skill, and it was a no-op in the original data), R5 Crash Threshold (tuning against known crashes = snooping), R6 Cross-Sectional Momentum (only 3 assets = tilt towards gold).
• Iron Rule: Each change must have ≥5 re-runs reporting CI on the same window + must outperform naive buy-hold / naive DCA, otherwise it's just beta repackaging.

Closed-Loop CI + R3-vol Verdict (2026-06-27, DeepSeek-v4-flash, n=5+5)

Giving the committee real rolling positions (fixing ADR-022 T3 empty stub), ±R3-vol each re-run 5 times:

Configuration	alpha vs. Same-Asset Buy-Hold
Closed-Loop R0-only (sees real positions)	−6.04% ± 1.70
Closed-Loop + R3-vol (vol-target sizing)	−4.88% ± 1.63
Difference (vol − no-vol)	+1.16% (t≈1.10, p≈0.30, not significant)
Control: Naive DCA / Buy-Hold	−1.15% / 0%

Verdict: R3-vol fails the iron rule (must significantly outperform naive DCA −1.15%), do not promote to live. ① The +1.16% gain is not significant at n=5 (confidence intervals fully overlap); ② Even the best run (−3.44%) loses to −1.15%; ③ Total TRIM/SELL across 10 runs = 2 – Even with real positions, the committee barely reduces positions, showing no timing skill (ADR-022 T3 reconfirmed). Both interventions (empty stub → real positions: −8.06% → −6.04%; +vol-target: → −4.88%) only "reduce losses", failing to change the fundamental fact that the active committee underperforms the passive benchmark in this upward window.

Implications for Product / Methodology

• Positioning: Externally, communicate "transparent, disciplined risk management + auditable decision-making", not outperformance/alpha. Consistent with the honest README tone.
• Backtest Discipline: Testing skill requires closed-loop (feeding real positions) + execution layer sign contract (negative alloc = position reduction, take magnitude); empty stubs + swallowed TRIMs simultaneously create illusions.
• Comparison with Professional Institutions: CMB Private Banking actively downgraded gold in 2025Q1 and maintained full allocation to US stocks during the same period – showing differentiation; our committee was consistently bullish quarter by quarter, more blunt.

Unresolved (Future Work)

• The window is almost entirely an upward market → no stage for position reduction skill; a final verdict requires a post-cutoff window containing a real crash (currently unavailable).
• R1 (reward anchor) verification requires Optuna retraining (not done); R3-vol has been judged a failure. Remaining roadmap items (see above) have not been individually CI-tested.
• n=5 Monte Carlo: The +1.16% vol gain is not significant; increasing n might establish a threshold, but the absolute alpha is already confirmed negative, making this a low priority.