ADR 008 — caution switches to lift-based scoring; formally dormant under current data

Date: 2026-05-27 (Second update 2026-05-27: Adopted lift-based scoring + post-cutoff validation) Status: Accepted Related: 007-few-shot-retirement · 03-dreaming · jobs/dreaming.py:_score · jobs/verdict_review.py

Context

The caution insight for Dreaming ("Should have acted but HOLD missed the upside", used to counter-conservatively suppress CIO's HOLD tendency) Previously used absolute missed_up_rate scoring: score = missed_up×0.7 + sample×0.3. Problem: In a unidirectional upward regime, HOLD inevitably misses the upside (the market simply goes up), making missed_up naturally high → treating a base rate illusion as a strong signal. The Phase 1.5 (pure bull market) NDQ HOLD/range_bound 30d with 61% missed upside is a classic false caution (avoided_down=0, no downside risk whatsoever).

Key Evidence (Two Batches of Data + Deterministic Base Rates)

Candidate	regime	missed_up	base_up	base_down	lift=missed_up−base_up
Phase1.5 false caution	NDQ range_bound 30d	0.61	0.75	0.00	−0.14
post-cutoff "true" caution	NDQ downtrend 30d	0.58	0.66	0.04	−0.08
post-cutoff	NDQ range_bound 30d	0.30	0.29	0.44	+0.02

• Both cautions are actually base rate illusions: Phase1.5 is "bull market always goes up", post-cutoff downtrend is "71% V-shaped rebound after a sharp drop"——base_down≈0, the "missed upside" is an inevitable consequence of the rebound, not a fault of HOLD.
• The deepest finding: At times with real downside risk (range_bound/uptrend, base_down 0.24–0.44), HOLD is actually protective (reliable). "Should have acted but HOLD missed the upside" simply does not hold in clean data.

Decision 1: Adopt lift-based caution scoring (a principled correction, not reward hacking)

The caution branch in jobs/dreaming.py:_score is changed to:

base_up / base_down = raw 30d up/down base rates for that (asset, regime, window) across all verdicts
if base_down < CAUTION_MIN_BASE_DOWN(0.15):  return 0   # regime has no real downside → missed upside is a base rate illusion
lift = missed_up_rate − base_up
if lift <= 0:                                 return 0   # HOLD did not miss more upside than the base rate → not a true signal
score = min(lift / CAUTION_LIFT_FULL(0.20), 1) × 0.7 + sample × 0.3

Data flow: verdict_review adds directions to each review (raw market direction up/down/flat per window, verdict-independent, using the same atr flat band as HOLD) → dreaming.rem_sleep aggregates by (asset,regime,window) into base_up/base_down attached to candidates → _score calculates lift. Reliable scoring remains unchanged (still hit_rate based).

Touchstone (proving the formula change is not to favor a specific candidate):

• Phase1.5 false caution (missed_up0.61/base_up0.75/base_down0.00) → score 0, rejected ✓
• post-cutoff "true" caution (NDQ downtrend, base_down0.04) → score 0, also rejected ✓ (no favoritism)
• Hypothetical true caution (missed_up0.70/base_up0.50/base_down0.30, lift+0.20) → score 1.0, accepted ✓

→ Rejects both base rate artifacts, accepts only "real downside + HOLD missed more upside than the base rate"——equal treatment, principled.

Decision 2: Under all current data, caution is formally dormant (signal does not exist, not a high threshold issue)

Across all data from Phase 1.5 (bull market) + post-cutoff (V-shaped sharp drop), 0 caution candidates pass the threshold—— It's not that the 0.8 threshold is too high, but that there are truly no qualified cautions in this data (no "real downside + HOLD systematically missed subsequent upside"). This is the correct dormant state, not a mechanism failure. Counter-conservatism continues to rely on CIO's CONCENTRATION_PCT opportunity cost rules

• already landed reliable insights. No caution is activated.

Decision 3: The bottleneck for unlocking caution is not the formula (already in place), but the data

To solidify caution, data from regimes with sustained declines / non-V-shaped downturns is needed——where HOLD fails to avoid losses in a real sustained decline and also misses the subsequent upside (base_down>0 and lift>0). Clean (post-cutoff) data can only come from a future slow bear market; The 2022 full-year decline from 2018-2023 is precisely this kind of base_down>0 sustained downturn, but it falls within DeepSeek's training data (contaminated). Until such data is available, caution remains dormant.

Decision 4 (A broader conclusion than caution): The "system is overly conservative" hypothesis lacks evidence

The recent pursuit of "the committee is overly conservative, should have acted but HOLD"——lacks demonstrable evidence in clean data:

• When there is real downside risk (base_down 0.24–0.44), HOLD is correct (protective, high avoided_down).
• The "missed upside" after a sharp drop is a V-shaped rebound base rate (base_down≈0), not a judgment error.
• Under lift-based scoring, no caution shows a positive lift.

→ "Overly conservative" as a falsifiable flaw does not hold in the current data. This is recorded to prevent future pursuit of a defect that does not exist in the data. If this hypothesis is ever to be revisited, first prove that in regimes with base_down>0, HOLD's missed_up is significantly > base_up (i.e., it truly missed out on money to be made), then discuss.

Boundaries

• Modified jobs/dreaming.py (_score + base rates), jobs/verdict_review.py (directions field), and corresponding tests. No caution activated, no live deep_sleep run, no live deployment, no commit.
• Post-cutoff experimental data is in memory/.backtest/; Phase 1.5 archive is in memory/.backtest_archive_phase15_clean_20260527/; 3 post-cutoff reliable insights pending review are in memory/insights_postcutoff_pending_review/ (not in live).