16 — TradingAgents-Style Analyst Experiment (test_ta, 2026-06-11)

Question: Is there any benefit to adding 3 TradingAgents-style analyst subagents (fundamental, sentiment, news) to the committee? What is the upper bound? The experiment code is on the test_ta branch (not merged to main). This page is the single source of truth for the conclusion. Conclusion First: Pre-registration Gate: 0/3 analysts passed. Integration phase not triggered per pre-registration.The existing architecture of "deterministic fact blocks into reasoning, no voting/report-type LLM voice" is supported by the experiment. Related: 12-verification.md, adr/009.

0. Why This Design (Lesson from the Overturned v1)

The initial design (directly running 6 committee draws for A/B/B1 comparison) was overturned by five review comments, the core being the power bottleneck: 122 decision days × 35-45% single-run verdict flip noise → high probability of spending ¥90+12h to buy an "undetectable" result; additionally, the new LLM voice could not be post-hoc denoised using the ablation path of "same verdict, different strategy". The other four comments: news analyst has no 2024 archive (data leakage risk), fundamental analysis without historical PE is just Quant in disguise, B1 replacement arm has bivariate confounding, budget gate might be hit.

v2 was restructured into phases, with expensive parts conditionally triggered:

• Phase A Signal Existence (high power, main source of conclusion): Each analyst independently produces reports with a mandatory STANCE for 244 (decision day × asset) pairs. Deterministic scoring vs. forward returns. n=244/analyst, SE≈3.2% — an order of magnitude more sensitive than committee-level A/B, and cheaper.
• Phase B Integration A/B (low power): Only runs if someone passes Phase A. Only integrates those who passed.
• Phase C Upper Bound (dose-response): Only runs if Phase B is positive. One arm, one stop.

1. Data Label Table (Phase 0 Feasibility Spike)

Analyst	Data Source	Label	Description
Sentiment	^VIX full history rolling 2y percentile + asset ATR mutation ratio (production spec)	Real	Point-in-time clean
Fundamental - Gold	CFTC COT Gold Futures Non-Commercial Weekly Report (deacot yearly archive, free)	Real	as_of + 3-day publication lag aligned; same-day multi-contract uses dominant (largest OI)
Fundamental - NDX	OHLC valuation proxies (2y percentile/MA deviation/distance from ATH) + ^TNX	Degraded	Free historical index PE does not exist (macrotrends 403)
News	GDELT 2.0 DOC API historical archive, headlines 7 days before decision day	Real·Degraded	Headlines only, no body text; multilingual mixed; local IP heavily rate-limited (~75% 429), collected via merged queries + multi-round checkpointed retries

Rejected sources: WGC central bank gold purchases/ETF flows xlsx (login wall); macrotrends index PE (403).

2. Pre-registration Judgment Criteria (Hardcoded Before Full Run)

Phase A Gate (scripts/eval_ta_signal.py@test_ta):

• Primary metric: 30d directional hit rate (only samples where stance≠neutral), Wilson 95% CI
• Analyst "has signal" = CI lower bound > max(① Subset directional base rate, ③ Input data mechanical mapping point estimate)
  • ① Calculated on the subset where the analyst expressed a stance: hit rate of "always guessing the majority direction" (same denominator, paired)
  • ③ Sentiment = VIX percentile mechanical binning (≥0.85→bearish/≤0.35→bullish); Fundamental-Gold = sign of COT net position 4-week change; Fundamental-NDX = 2y percentile ≥0.8→bearish/≤0.2→bullish; News has no ③, only compared to ①
  • Cannot beat "mindlessly guessing the majority direction" or the mechanical mapping of its own input → not considered LLM added value
• 0 pass → Do not run Phase B; ≥1 pass → Phase B only integrates those who passed
• (The "② Production probability table direction" baseline mentioned in the plan, which included full-sample forward-looking data, was discarded and not counted)

Implementation details: One LLM call per (decision day × asset × analyst) (no debate, no committee); Decision days = exactly the same 122 weekly frequency days as historical A/B/C ablation (2024-01-01 → 2026-04-27); Prompt adapted from TauricResearch/TradingAgents (Apache-2.0) analyst system messages; Same model and temperature (0.2) as the production committee; Zero forward-looking (sequence truncation / COT lag alignment / news window [asof−7d, asof]).

3. Phase A Results (2026-06-11, 732 Reports, 0 Parse Failures)

Analyst	Stance/Total	30d Hit Rate [95%CI]	60d Hit Rate	① Base Rate	③ Mechanical Mapping	Agreement Rate with ③	Gate
fundamental	147/244 (60%)	50.3% [42.4, 58.3]	52.4%	70.7%	49.8% (n=221)	56%	FAIL
├ GC=F	95	50.5%
└ NDQ.AX	52	50.0%
news	148/244 (61%)	57.4% [49.4, 65.1]	58.3%	73.0%	— (no ③, compare to ①)	—	FAIL
├ GC=F	84	58.3%
└ NDQ.AX	64	56.2%
sentiment	156/244 (64%)	64.7% [57.0, 71.8]	65.3%	73.1%	57.5% (n=106)	49%	FAIL
├ GC=F	82	61.0%
└ NDQ.AX	74	68.9%

Cost: 972k tokens (≈¥10-20), no committee re-run.

4. Gate Judgment and Attribution Matrix

Gate: 0/3 passed → Phase B/C not triggered per pre-registration.

Attribution per analyst ("Is it our problem or the experiment design's problem?"):

Analyst	Attribution	Basis
fundamental	Data itself has no 30d directional signal (Gold, real data) + Experiment design degradation (NDX)	Gold: Under real COT data, LLM 50.5% and mechanical mapping 49.8% both ≈ coin flip — it's not that the LLM reads poorly, it's that this data has no information content for 30d direction in this window. NDX: Valuation proxy overlaps with Quant dimensions (known design degradation), 50.0%
news	Mixed: Partly experiment design (headlines only, multilingual noise, coarse filtering), partly fundamental (headline information already priced in)	57.4% significantly above coin flip but far below 73% base rate; the two explanations are indistinguishable from this data — a truly fair test requires a forward experiment (see §6)
sentiment	Not a data/implementation issue, it's the goal of "directional prediction increment" itself	Real data + LLM indeed performed non-trivial interpretation (beat its own mechanical mapping by +7.2pp, agreement rate with mapping only 49%, often made contrarian judgments like turning bullish at the panic bottom on 2024-08-05) — but still lost to "mindlessly bullish" in a one-sided window. Consistent with the system's existing conclusion: LLM marginal value at the directional prediction level is extremely low (verdict closed-book ≈57%)

Implications for Architecture:

1. The existing path (deterministic fact blocks into reasoning + probability tables anchor direction + LLM as translator/interpreter) is supported: When scoring independently, LLM voice cannot beat the base rate; entering the noisier committee debate makes it even less likely to produce directional increment.
2. Sentiment "beats mechanical mapping but loses to base rate" precisely illustrates that the LLM's value lies in interpretation (contrarian judgment, context integration) rather than directional voting — this is the design assumption of fact blocks + CIO veto power.
3. Answer to "subagent upper bound": Under this data/window, the upper bound in terms of tradable directional increment is 0. Dose-response (+1/+3/+7) was not tested due to gate failure — if an analyst passes the gate in a future window, it can be retested incrementally according to Phase C.

Post-hoc Decomposition (post-hoc, not pre-registered — used only for attribution, not for changing the judgment)

Broken down by stance direction (30d, n = number of stances):

Analyst	bearish Hit Rate	bearish Median 30d Return	bullish Hit Rate
sentiment	16% (n=19)	+2.8%	72% (n=137)
news	21% (n=39)	+2.3%	71% (n=109)
fundamental	31% (n=80)	+1.3%	73% (n=67)

Two facts that nail the conclusion: ① bullish hit rate 71-73% ≈ base rate 73% — bullish stances are bull market parroting, with no timing content; ② bearish stances in this window are counter-indicators (market continues to rise after calling a top). The mechanism for "sentiment beats mechanical mapping" is thus clarified: it's not separable interpretive wisdom, but that it calls bearish less often than the naive panic mapping — in a window with frequent V-shaped recoveries, this happens to make fewer mistakes. This cross-validates the system's existing finding (real cost of preventing buys during COVID-like V-shapes). Performance of bearish stances in a true bear market window was not tested.

5. Power and Limitations (Honest Statement)

• n=244/analyst, SE≈3.2%; 30d window overlap → independent samples less than nominal value
• One-sided bull market window: ~73% of 30d windows for the two assets in 2024-26 were positive; "beating the base rate" is a high bar — this is the correct bar for "is it useful", but conclusions for bear market windows are not covered; sentiment might differ in choppy/bear markets → Retested, see §8 (2022 bear market window + cross-model retest matrix)
• News is headlines only, fundamental-NDX is a proxy — the applicability of these two FAILs is limited by this
• Single model, single temperature; different model families might differ (not tested) → Retested, see §8
• Phase B (integration P&L) was not run — "no gain from integration" is inferred from "signal does not exist", not directly measured

5b. Retest Matrix (2026-06-12, Driven by User Challenge: "It can't be 0")

Retested for the three blind spots in §5 (single window/single model/no ensemble). Pre-registration extension criteria (hardcoded before judgment): cross-model 2/3 pass = signal exists but model-sensitive; 1/3 = single evidence insufficient, maintain original judgment. (Precise spec: 2 vendors, 3 models — MiMo v2.5-pro is the same as production; reasoner/chat are both DeepSeek with different post-training. GDELT 2022 cache 102/104 keys, missing 2 keys for 2022-01-03, graceful degradation.)

30d Hit Rate vs Gate (CI lower bound > max(① Base Rate, ③ Mechanical Mapping))

Window × Model	fundamental	sentiment	news
2024-26 × mimo-flash (Original Experiment)	50.3% FAIL	64.7% FAIL	57.4% FAIL
2024-26 × deepseek-reasoner	28.8% FAIL	58.2% FAIL	59.3% FAIL
2022 Bear Market × mimo-flash	74.3% PASS	48.3% FAIL	50.9% FAIL
2022 Bear Market × deepseek-reasoner	60.6% FAIL	48.0% FAIL	50.0% FAIL
2022 Bear Market × deepseek-chat (Adjudication)	57.8% FAIL	—	—

"Add all three at once" retest (combined analyst, 2026-06-12 user follow-up): One LLM reads all three data packets simultaneously and then expresses a stance (③ baseline = sentiment/fundamental mechanical mapping majority vote):

Window × mimo-flash	combined 30d Hit Rate	Same Subset Base Rate	vs Best Individual	Judgment
2024-26	48.7% [41.6,55.8]	73.3%	Worse than all three individuals (50.3/57.4/64.7)	FAIL
2022 Bear Market	66.7% [55.9,76.0]	61.7%	Worse than fundamental alone (74.3%)	FAIL

Mechanism: Combined context does not produce emergence, only amplifies risk narratives — 2024 combined called bearish 98 times (individual analysts 19-80 times) with only 27% hit rate; 2022 diluted the only somewhat useful signal from fundamental (bullish leg 85%→57%). "No signal individually ≠ might work together" is positively falsified: together is worse.

Judgment: fundamental-2022 single model passed, cross-model 1/3 → single evidence insufficient. ADR-009 maintained.

Mechanism Findings (More Useful Than the Matrix Itself)

1. "Stronger model" hypothesis ruled out and reversed: reasoner performed worse than flash in the 2024 window (fundamental 28.8% — 107 bearish calls out of 118 stances, reasoning ability used to tell itself a bear market story). The mechanism of bullish parroting base rate / bearish counter-indicator holds across models.
2. What flash-2022's PASS looks like: Both legs won (bearish 68% / bullish 85%), but the winning part was precisely the 40% of judgments where it deviated from mechanical mapping; reasoner/ds-chat had higher agreement rates with mapping (74%/54%), and their hit rates converged to the mapping itself (61.3%) ≈ turning themselves into that if-else. Whether flash's deviation is true interpretation or a window-style coincidence, the 1/3 replication rate favors the latter.
3. Ensemble is also a dead end (2024 window, 3 combinations all FAIL): Majority vote 60.0% vs base rate 69.6%; unanimous vote 80.0% vs 90.0% (unanimous agreement occurred when the market trend was most obvious); any bearish veto 43.2% (worst). Consensus strength and accuracy are non-monotonic (3/3 bullish→88.9% up, 2/3→57.5%, 1/3→68.9%) — members have no additive independent information.
4. Sentiment fails symmetrically across windows: In 2024, its bearish stance was a counter-indicator; in 2022, its bullish stance was a counter-indicator (38% hit rate, "panic = buy the dip" catching falling knives all the way). Whichever leg goes against the regime trend becomes a counter-indicator — sentiment has no stable cross-window signal.
5. News bear market performance narrows down: 2022 both models 50.9%/50.0% ≈ coin flip (2024 was 57-59% vs base rate 73) — across 2 windows × 2 models, consistently pointing to "headline information already priced in". The "partly fundamental" attribution in §4 is upgraded to the primary explanation; the forward experiment (§6) remains the fair test.
6. Residual signal worth noting: ③ Mechanical mapping itself (COT 4-week change sign + valuation percentile) reached 61.3% in 2022 vs base rate 51.4% (n=62, CI includes base rate, not significant). All three models converged towards it. If implemented as a fact block (deterministic, zero LLM voting, regime-conditionally triggered), it's a candidate for a low-cost forward experiment — consistent with the "data on the table, LLM doesn't vote" architecture.

Cost: All retests ~1.6M tokens (reasoner×2 + flash retest + ds-chat adjudication), about ¥30-50.

6. Next Steps: News Analyst Forward Experiment Queue

In the backtest, news only had headlines; a fair test of news is forward-looking: The event layer (starting 2026-05) is accumulating real news + stance. After fixing the #26 index → proxy mapping, NDQ.AX can also hit index events. After accumulating 3-6 months, a live A/B test can be run (same day dual run, with/without event brief), and results will be backfilled to this page.

7. Reproduction

Frozen, de-PII'd reproduction package on main: experiments/ta-analysts/ (data + scripts + baselines + SCHEMA, 2540 reports can directly use eval_ta_signal.py to recalculate the Gate without re-running the LLM). Data role conventions are in experiments/README.md.

To re-run from scratch (requires live experiment code, still on the test_ta branch):

git checkout test_ta   # Live experiment code (do not switch branches while committee is running)
uv run python scripts/ta_data.py cot          # CFTC COT cache
uv run python scripts/ta_data.py fetch_news   # GDELT fetch (rate-limited, several hours)
uv run python scripts/ta_phase_a.py           # 732 LLM calls (checkpointed resume)
uv run python scripts/eval_ta_signal.py       # Pre-registration Gate judgment

Phase B integration code is also on the same branch (INVEST_TA_ANALYSTS flag, default off = bitwise equivalent to main, with equivalence tests). It can be directly reused if the gate is passed.