ADR-022: Backtest Memory Leak via Absolute Price Fingerprints, Not Dates – Contamination/Holdout Bucket Discipline

Date: 2026-06-25 Status: accepted Extends: ADR-010 (param-management), ADR-020 (concentration lens default OFF), ADR-021 (currency-adaptive path-profile); paired with scripts/backtest_committee.py + scripts/holdout_validate.py + tests/test_backtest_leak.py.

Methodological Ruling (Empirically Verified)

The memory leakage channel in openInvest backtests is not date strings – the prompt contains zero dates, and the backtest actively removes dates from all inputs. The contamination comes from absolute price / macro point fingerprints: Current Price / MA120 / MA250 / ^TNX / ^VIX / DXY and the unnormalized ma20 / ma120 in the regime INPUTS line, fed verbatim into the prompt. A MiMo (mimo-v2.5-pro) trained up to the end of 2024, upon receiving the fingerprint GC=F + price 1582 + VIX 13.68 + DXY 99, can infer "this is roughly early 2020" and thus unlock the post-period from its memory – it already knows how gold prices moved after that.

Therefore:

• date <= 2024-12-31 = Contamination bucket. Only for behavioral consistency / regression scans, never report performance.
• date > 2024-12-31 + 95d buffer = Only clean holdout, reportable for predictions / performance.

Key point: The data timestamp itself is clean (root-level get_history_df cut is solid, see §4); the contamination is only in the model weights' memory. "Backtest reading future data" is a separate issue (which we have prevented); this refers to "the model weights have seen this historical period." Do not conflate the two.

---

Body (8 Points + Evidence)

1. Leakage Channel is Absolute Price / Macro Point Fingerprints, Not Dates

Fingerprint fields fed verbatim into the prompt:

• utils/exchange_fee.py:330 – Current Price: {current_price:.4f} (absolute price, 4 decimal places)
• utils/exchange_fee.py:348 – MA120 (Trend): {ma_120:.4f}
• utils/exchange_fee.py:350 – MA250 (Base): {ma_250:.4f}
• utils/exchange_fee.py:387 – US 10Y Treasury Yield (^TNX): {tnx_last:.2f}%
• utils/exchange_fee.py:390 – CBOE Volatility Index (^VIX): {vix_last:.2f}
• utils/exchange_fee.py:399 – US Dollar Index (DXY, DX-Y.NYB): {dxy_last:.2f}
• core/regime.py:155-161 – inputs_used contains unnormalized absolute values for ma20 / ma120, entering the transcript via format_regime_brief's INPUTS line.

This set of absolute quantities collectively forms an era fingerprint. Any LLM whose training corpus covers that market history can reverse-engineer the approximate era from (symbol, price, MA, VIX, TNX, DXY) → unlock the subsequent market. "No dates in the prompt, so no leakage" is a falsified argument; do not repeat it. No matter how cleanly dates are removed, price fingerprints still leak.

2. Date Removal is Ineffective; Normalization Only Degrades, Does Not Solve

• Date removal is ineffective: There are no dates to remove – the backtest hasn't fed dates for a long time. "Re-deleting" something that doesn't exist yields zero benefit.
• Normalization only degrades, does not solve: Even if Current Price / MA were replaced with relative quantities (price/MA250 - 1, etc.), it would only reduce fingerprint resolution, not eliminate it. The reason is that the fingerprint is simultaneously the discipline payload for the committee – utils/exchange_fee.py:391's VIX > 20 indicates fear; VIX < 15 indicates complacency directly uses the absolute value of VIX to determine fear/complacency levels; normalizing away the absolute value would break this discipline. The fingerprint and the decision signal are the same data, with a hard ceiling: the degree to which you can sanitize = the degree to which you can retain decision-making power.
• Corollary: Even if an --anonymize normalization switch were added, the result would be a different strategy (a strategy with a neutered signal), and its performance cannot be promoted to represent this strategy's performance. Anonymization is a research probe, not a substitute for a holdout.

3. Two-Bucket Rule is Ironclad; Prohibit Merging Buckets for Reporting

Implementation is already in place in scripts/backtest_committee.py:

• LLM_TRAINING_CUTOFF = "2024-12-31" (:294) – As of 2026-06-25, MiMo self-reports training up to the end of 2024.
• FORWARD_BUFFER_DAYS = 95 (:295) – Sufficient 90d forward window for verdict_review to score using actual future returns.
• Holdout filter: d > LLM_TRAINING_CUTOFF and d <= hold_end (:300).

Ruling:

• date <= 2024-12-31 = Contamination bucket → Consistency / regression scans, never report performance.
• 2024-12-31 < date <= today - 95d = Clean holdout (--holdout) → Only credible predictions / performance.
• --allow-lookahead (:256) is for upper-bound estimation only; its output is not clean performance.

Prohibit merging the two buckets into a single reported number. Merging = using the model's memory of "known future" to dilute clean samples, inflating performance.

4. Data Timestamp is Clean; Contamination is in Model Weights; Leak Prevention Tests Cannot Catch Memory Leakage

The root-level market data reading path is already truncated by _patch_tools_to_date, guarded by tests/test_backtest_leak.py: test_root_store_cut_covers_pathprofile_and_fx (:33) asserts MarketStore.get_history_df for each symbol has df.index.max() <= cutoff; test_wrapper_get_history_data_cut (:25) guards the ef.get_history_data wrapper layer. This set of tests must be retained – it prevents "backtest reading future OHLC" (data timestamp leakage), and this layer is indeed clean.

However, it structurally cannot catch the memory leakage described in this ADR: it only verifies the end timestamp (df.index.max() <= cutoff), governing "up to which date the input data is cut"; it cannot govern "the past values themselves can reverse-engineer the era." test_backtest_leak all green ≠ backtest performance is clean. Two orthogonal types of "cleanliness": data cleanliness (guarded by tests) vs. memory cleanliness (guarded only by holdout).

5. Two Orthogonal Holdouts; Do Not Impersonate Each Other

• Parameter holdout: scripts/holdout_validate.py, window 2024-11-18 → 2024-12-31 (:24), uses Optuna best params to validate compute_strategy_reward (:79) on this period. It prevents parameter overfitting. However, this entire period is <= 2024-12-31 → entirely within the contamination bucket → it cannot measure performance, only "whether parameters remain stable on an adjacent window."
• Memory holdout: --holdout (post-cutoff), prevents memory leakage, measures performance.

The current repository only has the former, and historically it has been mistakenly reported as performance. The two are orthogonal; neither can substitute for the other: passing the parameter holdout ≠ performance is credible; passing the memory holdout ≠ parameters are not overfit.

6. Backtest Committee ≠ Live; Verdict Distribution Cannot Be Extrapolated to Live

The backtest does not know a user's actual historical positions on any given day. scripts/backtest_committee.py:207-211 hard-codes a neutral portfolio (Assuming user portfolio is neutral (no extreme concentration)). Consequence: The Risk role's concentration gate (ADR-020, opt-in concentration lens) never triggers in backtests – the denominator is always "neutral," and no asset can be calculated as having extreme concentration. Therefore, the verdict distribution (BUY/HOLD/TRIM ratio) obtained from backtests is a distribution missing the concentration dimension and cannot be extrapolated to live – live users have real concentration and real multi-leg positions, leading to TRIM paths that simply don't exist in backtests.

7. Benchmark Must Be Same-Asset Buy-and-Hold, Not Yu'ebao/Cash

core/strategy_metrics.py:119's vs_benchmark already implements the correct formula: alpha_pct (strategy total return − benchmark total return) + beat_days_pct (percentage of days strategy daily return exceeds benchmark daily return), with the benchmark curve being passed in (benchmark_curve). When reporting performance, the benchmark must be same-asset buy-and-hold, not Yu'ebao / cash / risk-free rate.

A deeper survivorship constraint: The backtest's asset pool consists of assets that are still alive and trading today. Therefore, claims like "openInvest can select good assets / can generate alpha" are invalid (stock-picking ability is contaminated by survivorship bias). The only valid claim is: "Given a specific asset, can openInvest's timing outperform that asset's buy-and-hold" – i.e., alpha must be anchored to the same asset, evaluating only timing, not stock selection.

8. Calibration Parameters Do Not Enter the Backtest Prompt, Hence Their Mild Snooping Does Not Contaminate the Backtest Holdout

Calibration text for path-profile / downside metrics is generated by build_reentry_reference (core/regime_probability.py:498). Its only production calls are in the orchestrator layer: core/runner/session.py:198 and core/runner/coordinator.py:162. However, scripts/backtest_committee.py bypasses the orchestrator, directly calling the primitive run_committee (:217), never passing through those two call points → the calibration reference text (path / defense text) never enters the backtest prompt.

Corollary: Even if these calibration parameters have mild snooping in their own fitting (tuned using data close to the sample), they will not contaminate the backtest holdout – because the backtest cannot see them. The two evaluation surfaces are isolated: the snooping risk of calibration parameters is governed by §5's parameter holdout and does not infect the memory holdout performance numbers from §3.

---

Validity Threat Table (10 Items)

#	Threat	Severity	Description / Defense
T1	Memory Leakage (absolute price fingerprints reverse-engineer era, unlocking future market)	High	Only --holdout (post-cutoff + 95d buffer) can prevent; contamination bucket never reports performance. Subject of this ADR.
T2	Survivorship Bias (asset pool only contains assets alive today)	High	Prohibit claims of "can select assets / generate alpha"; only allow claims of "timing on a given asset."
T3	Backtest Committee ≠ Live (portfolio hard-coded neutral → concentration gate never triggers)	High	Verdict distribution lacks concentration dimension, cannot be extrapolated to live. backtest_committee.py:207-211.
T4	Hit Rate Without Base Rate Correction (bare hit-rate not adjusted for base rate)	Medium-High	Reporting hit rate must simultaneously report same-asset buy-and-hold base rate; bare hit rate is uninterpretable.
T5	Reward Anchored to Cash, Not Buy-Hold (compute_strategy_reward treats cash as zero return)	Medium	build_dspy_trainset_v3.py:26 "cash is not a return"; reward treats empty positions as 0, effectively anchoring to cash → biases towards "dare to be fully invested." For performance reporting, use vs_benchmark same-asset alpha.
T6	Parameter Overfitting + Two Holdouts Impersonating Each Other	Medium	Parameter holdout (holdout_validate.py, 2024-11~12, entire period contaminated) ≠ memory holdout; was historically mistaken for performance. Orthogonal, do not substitute.
T7	No Costs + Same-Day Close Lookahead (backtest does not deduct fees/slippage, and uses same-day close price for both decision and execution)	Medium-Low	Upper-bound estimate is optimistic; when reporting, note "costs not accounted for, execution at close price."
T8	Single Sample Point Estimate (one backtest run yields one number, no confidence interval)	Medium	Report intervals / multiple seeds; do not treat a single point as a conclusion.
T9	Path Calibration OOS Mild Snooping (calibration parameters mildly use nearby data during fitting)	Low	Isolated: calibration text does not enter backtest prompt (§8), does not contaminate memory holdout performance; snooping risk governed by parameter holdout.
T10	Warmup / auto_adjust (insufficient indicator warmup period + yfinance auto_adjust adjustment drift)	Low	Ensure sufficient warmup; maintain consistent adjustment methodology, does not affect relative returns.

---

Consequences

• Reporting backtest performance only recognizes numbers from the --holdout bucket; numbers from the contamination bucket are labeled "consistency scan, not performance" and are never consolidated.
• tests/test_backtest_leak.py continues to guard against data timestamp leakage, but documentation must clarify it does not guard against memory leakage – do not use its green status as a performance shield.
• Any alpha claim must be anchored to same-asset buy-and-hold (vs_benchmark), with the statement "evaluating timing, not stock selection."
• Calibration parameters (path / defense) must maintain the current status of not entering the backtest prompt (going through the orchestrator, which the backtest bypasses); this is the prerequisite for §8 isolation. If anyone later adds reentry / prob_hint to the backtest, §8 isolation breaks, and this ADR must be updated accordingly.