ADR-011 — HOLD Oracle Semantics: How Wide Should the hold_wrong Threshold Be

Date: 2026-05-27 Status: Proposed Related: 010-param-management · core/config/defaults.yaml oracle_accuracy

Background

The Oracle evaluates the ex-post accuracy of committee verdicts. The judgment for HOLD is:

# core/backtest_reward.py
if v == "HOLD":
    if abs(r) <= hold_neutral:   return 1   # hit
    if abs(r) >= hold_wrong:     return -1  # wrong
    return 0                                 # neutral

Current values: hold_neutral=3.0, hold_wrong=8.0.

Data (276 verdict_review.jsonl entries, 119 HOLD):

• hit (|ret| <= 3%): 35 entries (29.4%)
• neutral (3% < |ret| < 8%): 52 entries (44.0%)
• wrong (|ret| >= 8%): 32 entries (26.6%)

Problem: hold_wrong=8 means "a 30-day gain/loss >= 8% after a HOLD" is considered wrong. But is a 9% gain after a "HOLD" truly a mistake?

Decision

Semantic Definition of HOLD

HOLD = "Do not act", not "The market will not move."

When the committee says HOLD, it means "Based on current information, the cost of trading to rebalance is not worth it." It does not predict market direction—it evaluates whether the risk/reward of "moving" exceeds transaction friction.

Therefore:

• A 9% gain after HOLD → This is "missing out," not a "wrong call." HOLD did not say "the market shouldn't rise."
• A 9% loss after HOLD → This is "failure to avoid a decline," closer to a "wrong call."
• No movement after HOLD → This is the ideal scenario.

But There's a Boundary: HOLD Cannot Be Completely Exempt

If the market drops 15% after a HOLD, saying "I just didn't act" is shirking responsibility—the committee has an obligation to recommend TRIM/SELL when risk signals are clear, rather than doing nothing.

Therefore, hold_wrong needs to differentiate direction:

• Rise too much (> hold_wrong) → Not wrong: Missing out is regrettable, but not an error. The semantics of HOLD are "do not move"; a market rise does not violate this.
• Fall too much (< -hold_wrong) → Wrong: The committee has a risk control duty. A HOLD during a significant drop constitutes a failure.

New Threshold Scheme

if v == "HOLD":
    if abs(r) <= hold_neutral:       return 1   # hit: market didn't move, HOLD correct
    if r <= -hold_wrong:             return -1  # wrong: significant drop, HOLD failed
    return 0                                    # neutral: other cases (including large rise)

Key change: Only judge wrong on the downside; all upside cases are neutral.

Candidate thresholds (to be verified via sweep, not decided in this ADR):

• hold_wrong=8 (current): ret <= -8% = wrong
• hold_wrong=10: ret <= -10% = wrong
• hold_wrong=12: ret <= -12% = wrong

Sensitivity analysis (current data):

hold_wrong	Wrong Count	Wrong%	Description
8 (current, bidirectional)	32	26.9%	Includes rises >= 8% as wrong
8 (downside only)	~18	~15%	Removes "false wrongs" from large rises
10 (downside only)	~12	~10%
12 (downside only)	~7	~6%

Note: Exact numbers require code changes and re-running; the table above is an estimate.

Why Not Change Directly

1. Changing the oracle function semantics (from bidirectional to unidirectional wrong) will affect reward calculation, and consequently Dreaming's insight learning. Need to confirm that callers of compute_strategy_reward and verdict_oracle_accuracy are not adversely affected.
2. The specific threshold value needs sweep verification (ADR-010 Rule 2: train+holdout dual metrics).
3. This change is independent of other oracle thresholds (e.g., accumulate_negative) and does not need to be bundled.

Evidence

• Of the 119 HOLD entries, 32 have |ret| >= 8%: roughly half are rises >= 8% and half are drops >= 8%.
• Judging a rise >= 8% as wrong penalizes a HOLD for "not missing out," which is semantically unreasonable.
• ACCUMULATE wrong analysis shows: the ACCUMULATE wrong rate during downtrends is only 8%, indicating the committee's judgment quality is good in downtrends—HOLD should have a higher standard during significant drops.

Execution

1. Modify the HOLD branch in core/backtest_reward.py:verdict_oracle_accuracy(): Only return -1 when r <= -hold_wrong; return 0 when r >= +hold_wrong.
2. Update tests/test_backtest_reward_v2.py to cover the new semantics.
3. Re-run sweep to verify the hold_wrong threshold selection.
4. Update defaults.yaml (if the threshold changes).

Boundaries

• This ADR does not change hold_neutral (|ret| <= 3% = hit, this semantics is uncontroversial).
• This ADR does not change the oracle semantics for other verdicts (the directional semantics for BUY/ACCUMULATE/TRIM/SELL are clear).
• This ADR does not decide the specific numerical value of hold_wrong—it only decides the semantic direction (unilateral vs. bilateral).