ADR-010 — Parameter Management Discipline

Date: 2026-05-27 Status: Proposed Related: 007-few-shot-retirement · 008-caution-insight-deferred · 13-param-tuning-feasibility

Context

openInvest has 50+ hardcoded parameters, divided into three categories:

1. Tunable Parameters (tunable.py): Follow the YAML/CLI/env injection chain; the sweep runner can freely search them.
2. Locked Parameters (locked.py): Code reads values directly from source; the injection chain does not touch them at all. Changes require a new ADR.
3. Engineering Parameters (not in config): Retries, timeouts, tool rounds. Located in locked.py but changes yield no benefit.

Previously, there was no unified parameter management discipline. Parameters were scattered across 6 files. What was tuned, why it was tuned, and whether it was validated all relied on memory. locked.py enforces discipline through code structure, not comments.

Decision

Rule 1: What is Tunable vs. What is Not

• Parameters in core/config/tunable.py: Tunable. The sweep runner can freely search them.
• Parameters in core/config/locked.py: Not tunable. Code reads values from locked.py; the injection chain (YAML/CLI/env) physically cannot override them. Changes require a new ADR explaining why the original design commitment is being overturned.

2026-06-17 (ADR-017): After the env layer, the injection chain adds a persistent API override layer (memory/.state/config_overrides.json, highest priority). This allows a whitelist subset (API_SETTABLE) of tunable parameters to be configured at runtime via GET/PUT /api/config + skill config — "configuration goes through the API, env degrades to bootstrap defaults." The whitelist only contains user-safe behavioral switches; internal sweep thresholds are not exposed (upholding this ADR's Rule 4). See 017-config-via-api. locked.py remains inaccessible to both env and API.

Rule 2: Sweeps Must Report Both Train and Holdout Metrics

Any parameter sweep result must report metrics for both the train set and the holdout set. Sweep results reporting only train reward are invalid — they cannot be merged into defaults.yaml.

Overfitting Determination:

• P&L-based: Train → holdout reward shrinkage > 50% = overfitting
• Pure Arithmetic: The optimal train threshold's crash recall on holdout < 80% = overfitting

Rule 3: Changes to locked.py Require a New ADR

To change any parameter in locked.py, you must:

1. Open a new ADR explaining why the original design commitment is no longer valid.
2. Provide data evidence (not intuition/feeling).
3. Only modify locked.py after the ADR is Accepted.

Rule 4: Merging Sweep Results into defaults.yaml Requires OOS Validation

To write optimal parameters found by a sweep back into defaults.yaml, the following must be satisfied:

• P&L-based: Reward on the holdout set > 50% of the reward on the train set.
• P&L-based: Sharpe ratio on the holdout set > 0.5.
• Pure Arithmetic: Crash recall on the holdout set ≥ 80%.
• The optimal parameter values on train and holdout differ by < 30% (parameter stability).

Rule 5: Prompt Changes Take Priority Over Parameter Changes

training_report has already proven that "prompt content is the true ceiling; hyperparameters cannot break through it." If a sweep finds no significant improvement, prompt changes should be considered first.

Explicit Scope: "Prompt changes" in this rule refer to modifying SKILL.md or prompt text under agents/*/. It does not include few-shot demos (ADR 007 has already nailed down CIO zero-shot; the few-shot path has been proven infeasible). Restarting few-shot experiments under the guise of "prompt changes" is not permitted.

Rule 6: Pure Arithmetic Sweeps Must Use Pre-Defined Ground Truth

The objective function for pure arithmetic sweeps (e.g., regime classification accuracy) must be evaluated against a pre-committed ground truth event list. Events cannot be added post-hoc.

Ground truth files are placed in docs/wiki/sweep_ground_truth/ and must be committed before the sweep runs (enforced by git timestamp). They cannot be modified after the sweep — to add events, create a new file.

Rationale: Not using an LLM for a pure arithmetic sweep does not mean there is no room for reward hacking. If ground truth is defined post-hoc, it is equivalent to overfitting thresholds on the training set.

Rule 7: P&L Mode Sweeps Must Perform Trial Sanity Checks

After each P&L trial completes, a sanity check must be performed:

• UNCLEAR rate > 30% → outlier
• Single verdict proportion > 90% → outlier (verdict collapse)
• Annualized return deviation from median > 3x MAD → outlier

Outlier trials are marked and re-run once. If still an outlier, they are excluded from the aggregation. The sweep results report must note the number of outliers.

Rationale: Phase 1.5 testing showed a ~7% Quant malfunction rate; 13 trials likely contain 1-2 outliers.

Evidence

• Optuna 30-trial: All 5 parameters were placebo (reward stdev/mean = 9.4%).
• training_report: v1 prompt change reward 0 → 0.4; Optuna could only search up to 0.42.
• ADR 007: Few-shot v1-v4 all failed; CIO remains zero-shot.
• ADR 008: Caution lift-based touchstone validation.

Boundaries

• This ADR does not constrain parameters for DSPy / prompt engineering (that path is governed by ADR 007).
• This ADR does not constrain trading friction parameters (determined by bank/broker fee schedules, not calibration targets).