12 — Verification (Empirical Validation / "Scientific Evidence")
This chapter is not about architecture; it's about proving the tool works with real data. It's for those willing to read deeply, for those questioning "Is an LLM committee better than doing nothing?", and for my future self to review whether the design still holds.
Why This Chapter Exists
When readers first see "4 LLMs debating to give investment advice," they have two reactions:
- Sounds cool → wants to fork and use it
- Sounds like pseudoscience → leaves
This chapter is for the second group — it takes all claims of "we think this is useful" and subjects them to falsification/verification through backtesting data. It includes both positive results and negative results (equally important).
Claim 1: The v0 baseline is a 100% HOLD parrot
Empirical Test (2026-05-11, 6-month window 2024-05-15~11-27, NDQ.AX + GC=F, 33 dates × 2 assets = 66 verdicts):
| Metric | Empirical Value |
|---|---|
| HOLD Percentage | 100% (66/66) |
| Monthly Verdict Diversity | 1 type / month |
| Confidence Variance | ≈ 0 (fixed at 0.65±0.05) |
| Total Return | 0% (account 100% cash throughout) |
| vs Yu'ebao | -0.12% (can't even beat doing nothing) |
| Reward Signal | -0.0006 (noise level) |
→ Claim Confirmed: Unoptimized prompts cause the LLM to degenerate into a parrot.
Claim 2: Adding the "Cash Opportunity Cost" rule to the CIO prompt unblocks the system
Change (commit 2b69c83): Added a hard rule to the CIO prompt:
CONCENTRATION_PCT < 20% (position < 20%, bullets ≥ 80%):
- HOLD is not allowed
- Default to at least ACCUMULATE, alloc = dry_powder × 5~10%
- Exemption: Macro=risk_off AND Risk=high_risk (both AND)v0 → v1 Comparison (same 6-month window):
| Metric | v0 baseline | v1 |
|---|---|---|
| HOLD Percentage | 100% | 44% |
| BUY+ACCUMULATE | 0% | 56% |
| Total Return | 0% | +15.55% |
| Annualized Return | 0% | +25.39% |
| Sharpe | 0 | 1.58 |
| Max Drawdown | 0% | 9.0% |
| Reward | -0.0006 | +0.3978 |
| vs Yu'ebao | -0.12% | +14.72% |
| vs CSI 300 | 0% | +15.55% |
| vs Equal-Weight Buy-Hold | 0% | -1.65% ⚠️ |
→ Claim Confirmed: A single prompt change moved the account from holding cash to +25% in six months.
⚠️ Caveat: Underperformed equal-weight buy-and-hold by -1.65% — the committee performs worse than a "buy and hold 25% in four assets and forget it" strategy in a bull market. This is a limitation of the current v1 prompt, left for future experiments.
Claim 3: The v1 prompt's directional accuracy is not high, but it "avoids big mistakes"
Empirical Test: v1 verdict vs. actual price change 7 days later (66 samples, scripts/build_dspy_trainset.py):
| Hit Category | Proportion |
|---|---|
| Direction Correct (+1) | 47.0% (31/66) |
| Neutral (0) | 39.4% (26/66) |
| Direction Wrong (-1) | 13.6% (9/66) |
| Strategy Hit Rate | +33.3% |
Breakdown by verdict type:
ACCUMULATE: 37 instances, only 38% direction correct (LLM is not good at 7d timing)HOLD: 29 instances, 59% direction correct (better at identifying sideways markets)
→ Claim Confirmed: The LLM is not good at timing but is good at avoiding big mistakes. The +25% return over six months is the result of a low reverse-error rate (only 13.6%) multiplied by cumulative compounding, not high hit rate.
Claim 4: Optuna 30-trial found the ceiling
Empirical Test (30 trials × 5-dimensional parameter space, ~60min wall clock with 3-worker parallel):
| Rank | Reward | Annualized Return | Sharpe | DD% |
|---|---|---|---|---|
| 🥇 #20 | 0.4223 | 34.4% | 1.37 | 15.4 |
| 🥈 #4 | 0.4223 | 34.4% | 1.37 | 15.4 |
| 🥉 #9 | 0.4205 | 29.2% | 1.56 | 10.9 |
| 🔚 #25 | 0.2621 | 20.6% | 1.28 | 10.1 |
Reward Statistics: mean=0.395, median=0.417, stdev=0.037 (only 1.5%).
→ Claim Confirmed: Reward 0.42 is the ceiling for the current prompt architecture. Bayesian search converged; hyperparameters cannot overcome the upper limit determined by prompt content.
Unexpected Finding: Of the 5 parameters, 4 are placebos (max_rounds=1 vs 3 same reward; alloc_aggressiveness 0.06 vs 0.25 almost same reward; regime threshold changes don't explain variance). Only alloc_aggressiveness is truly effective (controls SKIP rate, affects 1-2pp reward).
Claim 5: Parameters found by Optuna do not overfit (hold-out validation)
Empirical Test: Best parameters from the train set (2024-05-13 ~ 11-15), run once on the hold-out set (2024-11-18 ~ 12-31, 6 weeks):
| Metric | Train (26w) | Hold-out (6w) |
|---|---|---|
| Reward | 0.4223 | 0.2880 |
| Annualized Return | +29.2% | +17.4% |
| Sharpe | 1.56 | 1.59 |
| Max DD | 10.9% | 2.6% |
| vs Yu'ebao | +14.0% | +1.9% |
| vs CSI 300 | +14.7% | +2.1% |
| vs Equal-Weight | +2.7% ✓ | -1.9% ⚠️ |
Judgment Criteria: holdout_reward > 0.5 × train_reward → 0.288 > 0.211 ✅
→ Claim Confirmed: Parameters generalize OK, not in-sample overfit. The strategy became more conservative during the hold-out period (NDQ was oscillating) (BUY 2 times vs. 12 times in train), but risk control was actually better (Max DD shrank by 75%).
Claim 6: DSPy automated prompt tuning is more accurate than manual writing
Empirical Test (scripts/rl_optimize_prompts.py):
| Run | Train | Dev | Candidates | Bootstrap | Baseline | Optimized | Δ |
|---|---|---|---|---|---|---|---|
| Smoke | 10 | 5 | 3 | 2 | 0.600 | 0.700 | +10pp |
| Full | 46 | 20 | 10 | 4 | 0.725 | 0.825 | +10pp |
Both runs showed a stable +10pp improvement (dev set single-step verdict prediction accuracy 60→70% / 72→83%).
→ Claim Confirmed: DSPy automated selection of few-shot examples is more accurate than manually written prompts.
⚠️ Caveat: A +10pp improvement on the dev set does not directly translate to a backtest reward breakthrough of 0.42. It requires manually injecting the 10 best demos from experiments/dspy_optimized_v1_full.json into agents/cio.py and re-running the walk-forward. This step was not done.
Claim 7: WealthContextOfficer solves the "low cash = high risk" misjudgment
Problem (2026-05-12 user live account report):
"The agent kept telling me I had too little cash and to sell for cash. But in reality, I have family backup cash, this is just pocket money."
Root Cause: The Risk Officer prompt has DRY_POWDER_CNY < 1000 → high_risk, with no concept of "off-portfolio safety net."
Empirical E2E Test (same portfolio context: cash ¥500 + NDQ heavy position 99.9%):
| Dimension | A (No backup) | B (¥4M backup) |
|---|---|---|
| Risk SIGNAL | high_risk | concerned |
| Risk STRENGTH | 10 | 8 |
| Suggested Action | Immediately reduce position by 60% | Increase position limit ≤¥50 (10% bullets) |
| ONE_LINER | "Available bullets only ¥500 no replenishment capability" | "Although family backup eliminates liquidity risk, concentration is still high" |
Key Semantics (commit pending / agents/wealth_context_officer.py):
INVESTABLE_CASH_CNY = portfolio cash(never adds backup)BACKUP_BUFFER_CNY = off-portfolio emergency fund(only for risk mitigation, not investable)- The ¥4M family backup only changes the SIGNAL from high_risk to concerned, does not allow the CIO to shout "BUY ¥1M"
→ Claim Confirmed: Missing user context is a real problem, and adding an independent role can fix it precisely — the LLM's reasoning logic was correct; what was wrong was the premise it saw.
Negative Results (Equally Important)
Recorded here to prevent "post-hoc selective narrative":
The committee underperforms equal-weight buy-and-hold (v1 -1.65%, hold-out -1.9%) → The 4-role debate performs worse than "do nothing, buy 25% in four assets and forget it" in a one-sided bull market → Suggests we should have the ability to "identify a bull market and just buy and hold mindlessly"
Multi-round debate is a placebo (max_rounds=1 vs 3 almost same reward) → Previous effort on the cross-challenge mechanism might be over-engineering → Or the Round 2 prompt makes agents inclined to "maintain the original verdict"
alloc_aggressiveness 0.06 vs 0.25 almost same reward → This hyperparameter mainly affects the "SKIP rate," not the strategy's essence → The real alpha comes from when the LLM calls ACCUMULATE, not how much it buys each time
ACCUMULATE accuracy is only 38% → LLMs are not good at 7-day timing → The long-term accumulation still yields +25% because of "not making reverse-direction mistakes" (only 13.6% wrong), not because of "market timing ability"
TradingAgents-style analyst subagents provide no directional increment (2026-06-11, test_ta experiment, pre-registered Gate 0/3 passed, n=244/analyst) → fundamental 50.3% (real COT data also ≈ coin flip), news 57.4%, sentiment 64.7% — all below the "guess the majority direction blindly" baseline of ~73% → sentiment beat its own input data's mechanical mapping (57.5%): The LLM's value is in interpretation, not directional voting, supporting the "deterministic fact block + CIO veto power" route → 2026-06-12 retest reinforcement (wiki 16 §5b): 2022 bear market window × 2 vendors 3 models × ensemble 3 combinations, only fundamental-2022-flash single cell passed the line (1/3 across models, single evidence is not conclusive); the "stronger model" hypothesis was falsified in reverse (reasoner performed worse in the 2024 window) → Full text 16-ta-analysts-experiment.md, decision adr/009
Gold VIX defense leg "semantic error" hypothesis — acceptance conclusion flipped by a single window unit bug (2026-06-13 drift audit) (
scripts/validate_gold_defense.py) → Hypothesis: Gold is biphasic (liquidity squeeze sell-off first, crisis buying later), blocking gold buys during high VIX is a stock semantic misapplication (2020-03 case: -8.6% squeeze followed by +30.7% in 90d) → Exploratory full sample (2002-2026, n≈5963) strongly supports: VIX≥85th percentile bucket median across all windows is consistently right-skewed → 2026-06-12 initial judgment FAIL (60d breakeven probability marginal in two eras) — later found to be a window unit bug: usedshift(-w)= w trading days (≈1.4w calendar days), inconsistent with the production path distribution's calendar day unit. → 2026-06-13 drift audit corrected the unit (calendar days, consistent with forward_return single source) and flipped to PASS: fit/OOS × 30d/60d all four cells satisfied (median right-skewed + breakeven probability ≤ unconditional), but fit-60d margin is only 1pp (40% vs 41%), thin. → Production defense behavior was not changed based on this — whether to exempt the gold VIX leg is an independent, gated decision requiring counterfactual accounting (interventions.jsonl) money-level evidence, left for another discussion. This entry only records the fact that "the acceptance conclusion flipped after the unit correction." → Meta-lesson: The validation script's forward-return must use the same unit as production (calendar days), otherwise a ±40% horizontal axis error can flip PASS/FAIL. This is the root cause thatforward_returnsingle trusted source (core/regime_probability.py) aims to solve.Trend timing overlay can't withstand trading — Q2's "gold MA200 signal" is beta (2026-06-28,
experiments/signal-eval/trend_dca.py) → Background: signal-eval Q2 found "gold price above MA200 90d forward median +3.43% vs below −0.64%, p_holm=0.016", once considered the "only significant signal." → Made it into a tradable long/flat timing strategy, compared against a matched average exposure passive long position (isolating beta), both legs deducting costs (GC=F 0.38%/side), DSR deflated via MA grid search:- Gold post-2000 timing terminal value 3.07 vs buy-and-hold 15.10 vs matched passive 7.50 (only 1/5 of buy-and-hold)
- Sharpe +0.36 vs passive +0.68; maxDD −57% vs buy-and-hold −44% → "Avoiding drops" empirically failed, drawdown was deeper
- 10 variants (MA{100..300}×buf{0,0.01}) × 3 costs × 3 assets (Gold/A-shares/Nasdaq) none passed DSR>0.95 → DCA input tilt (dual sleeve "invest less today / invest more tomorrow", core never sells): same total amount tilted by MA200 trend vs. average input, terminal value multiples (bottom-fish/average/trend-follow) Gold 4.85/4.91/4.98, A-shares 1.62/1.55/1.49, Nasdaq 3.20/3.18/3.15 — both directions only move ±1–8%, no consistent winner (≈wash), tilting input amount by trend is not a return lever. → Implication: Q2's statistical significance was beta (being above MA200 = being in better days), not timing ability; MA200 lag → exits after the drop (locking in losses), re-enters after the rally (missing the move). Single-asset trend = noise, not contradicting time-series momentum literature (which relies on ~50 market cross-asset diversification for stability); this 3-asset basket cannot create a diversified trend portfolio. Independent implementation cross-validation numbers match digit-for-digit. → Full text experiments/signal-eval/README.md "Trend Timing Gate" section.
How to Reproduce These Data
Each claim corresponds to a reproducible pipeline:
# Claims 1-2 (Diagnosis + v1 change)
python -m scripts.backtest_runner --workspace /tmp/diag \
--start 2024-05-13 --end 2024-11-15 --assets NDQ.AX,GC=F
# Claim 3 (Trainset hit rate)
python -m scripts.build_dspy_trainset \
--workspace /tmp/diag --output /tmp/trainset.json
# Claim 4 (Optuna 30-trial)
python -m scripts.rl_train --workspace /tmp/optuna --n-trials 30 ...
# Claim 5 (Hold-out)
python -m scripts.holdout_validate --optuna-workspace /tmp/optuna ...
# Claim 6 (DSPy)
python -m scripts.rl_optimize_prompts \
--trainset experiments/dspy_trainset_v1_2024_05_to_11.json \
--output /tmp/dspy_out.json --n-candidates 10See docs/training_report.md for the full process (~1200 lines of raw data), and 11-rl-training.md for methodology explanation.
Limitations / What Cannot Be Claimed
- Small sample size: 66 verdicts × 2 assets × 6 months. Not enough to say "100% applicable to all market conditions."
- LLM training data lookahead bias: DeepSeek's training cutoff is ~2024-06-30; backtests after that contain bias.
- Paper trade simulator ≠ real trading: Does not account for slippage, transaction costs, or liquidity constraints.
- Not verified with live trading: All conclusions are from simulations. Actual results when connected to a user's live account may differ.
Maintenance Rules
When adding a new claim, you must include:
- Empirical data (not vibes)
- Reproduction command
- Negative results published together (avoiding survivorship bias)
- Do not delete caveats to make it "look good"