11 — RL Training / Backtest / Parameter Search

Want to optimize your committee's "skill"? This chapter explains how we validate + optimize, this is definitely not real ML training.

See also: 13 — Parameter Tuning Feasibility Diagnosis — Complete list of 50+ hardcoded constants + coupling analysis + Do-Not-Tune list + Optuna review + sweep strategy

TL;DR: What We Do Is Not "Training"

Intuitively, "training AI" means GPU + gradient descent + model weight updates. openInvest has none of this:

❌ No model weight updates
❌ No backprop / gradient
❌ No GPU requirement
❌ No embedding models
❌ No fine-tuning

What we do is "parameter search" + "prompt engineering":

Treat the LLM (DeepSeek) as an immutable black-box function
Use Optuna to automatically try different hyperparameters (regime thresholds, allocation ratios, etc.) to find the set that makes the backtest account most profitable
Use prompt modifications to improve the LLM's behavior itself (this is what truly affects the reward)

Concept Map

        ┌──────────────────────────────┐
        │  LLM (DeepSeek-v4-flash)     │   ←   Immutable black box
        │  ├─ Macro Strategist         │
        │  ├─ Quant Analyst            │
        │  ├─ Risk Officer             │
        │  └─ CIO                      │
        └────────────┬─────────────────┘
                     ↓ verdicts (BUY/HOLD/...)
        ┌──────────────────────────────┐
        │  Paper Trade Simulator        │   ←   Pure Python simulator
        │  ¥100k start → decide weekly  │
        │  Simulate trades based on LLM verdict│
        └────────────┬─────────────────┘
                     ↓ Account curve (daily values)
        ┌──────────────────────────────┐
        │  Reward Function              │   ←   Pure math
        │  = annualized - 0.5×drawdown + alpha│
        │    + 0.2×max(0, Sharpe-1)    │
        └────────────┬─────────────────┘
                     ↓ single number (e.g. 0.42)
        ┌──────────────────────────────┐
        │  Optuna (TPE Sampler)         │   ←   Bayesian optimization (CPU only)
        │  Learns from history (params, reward)│
        │  what parameters to try next  │
        └────────────┬─────────────────┘
                     ↑ params: regime_threshold=4.5, alloc_pct=0.2...
                     └─────→ Back to LLM call, loop

Nothing is "trained". Optuna does Bayesian sampling on the CPU; all "training time" is repeatedly calling the DeepSeek API.

Why This Is Necessary

Problem: Is the openInvest LLM committee effective? Unknown. Looking at live trading directly would take months to determine.

Solution: Use historical data for backtesting:

Assume you started on 2024-05-13 with ¥100k
Let the LLM see data up to that date and give a verdict
Simulate buying/selling based on the verdict
Mark-to-market along the timeline until 2024-11-15
Calculate total account value vs. Yu'ebao / CSI 300 / equal-weight buy-and-hold

If it outperforms the benchmark → the committee is useful; if it underperforms → better to do nothing.

Advanced: Use Optuna to automatically tune parameters to see if you can make more profit.

4-Stage Pipeline

Stage 1: Lookahead Prevention

Problem: When backtesting on 2024-05-13, the LLM must not see data after 2024-05-14, otherwise it's like a "prophet".

Countermeasure: 3 places must be fixed

utils/exchange_fee.py:get_history_data() add as_of_date parameter, DB cache filters by date cutoff
core/committee/persist.py:_capture_macro_context() use decision_date instead of datetime.now()
scripts/backtest_committee.py add --allow-lookahead flag, default rejects decision_date > 2024-06-30 (DeepSeek training data cutoff; backtests after this date contain LLM's own lookahead bias)

Test: tests/test_backtest_no_lookahead.py

Stage 2: Data Isolation (Workspace)

Problem: Backtesting writes many verdict files, which must never contaminate real position data.

Countermeasure: scripts/backtest_runner.py monkey-patches data paths before importing the invest module:

python

import core.memory_store as _ms
_ms.MEMORY_ROOT = workspace / "memory"   # Redirect to /tmp/xxx/memory/
import db.trades_db as _trades
_trades.DB_PATH = str(workspace / "db" / "trades.db")

Core Guarantee: All backtest artifacts are in the isolated /tmp/xxx/ workspace; the real memory/ / db/ remain untouched.

Stage 3: Paper Trade Simulation (Core)

User Insight: "Verdict accuracy" is not important; what matters is "if I follow the AI every time, will my account profit or lose?"

Implementation: core/paper_trade_simulator.py

python

sim = PaperTradeSimulator(start_date="2024-05-13", initial_cash_cny=100_000)
for decision_date in walk_forward_dates:
    verdict = run_committee(decision_date, "NDQ.AX")  # 4-role debate
    sim.execute_verdict(decision_date, "NDQ.AX", verdict)  # BUY → deduct cash + add shares

# Mark-to-market along the timeline
for date in trading_days:
    sim.account.daily_values.append((date, sim.mark_to_market(date)))

Compare against 4 benchmarks (scripts/run_walk_forward.py:_build_benchmark_curves):

Yu'ebao (annualized 1.3%)
CSI 300 (510300.SS buy-and-hold)
Pure cash (no operation)
Equal-weight portfolio (25% each of 4 assets, buy-and-hold)

Reward function (core/backtest_reward.py):

reward = annualized_return
         - 0.5 × max_drawdown
         + 0.5 × alpha_vs_yuebao
         + 0.2 × max(0, Sharpe - 1)

Design intent: Reward profit, penalize drawdown, give bonus for outperforming the passive benchmark, add points for high Sharpe.

Stage 4: Optuna Automatic Parameter Tuning

Want to see / modify training parameters? Check experiments/train_config.py —— Single source of truth, listing all hyperparameters + ranges + meanings + empirical effectiveness.
INVEST_* env vars are internal implementation details: They pass parameters across Optuna trial boundaries via env vars. These are not in .env.example; regular users / fork users do not need to configure them—— the code defaults to v0 behavior (no clamp / 1 round). Only when rl_train.py runs does Optuna temporarily override them; the env vars disappear when the process ends.

5 hyperparameters (detailed descriptions in train_config.py):

Parameter (CLI/Optuna)	Internal env var	Range	Meaning	Empirically Effective
`alloc_aggressiveness`	`INVEST_ALLOC_AGGRESSIVENESS`	0.05 ~ 0.30	Upper limit of single allocation as % of ¥100k baseline	✅ Truly effective
`regime_uptrend`	(monkey-patch THRESHOLDS)	3.0 ~ 6.0	MA20/MA120 spread threshold	⚠️ Weak
`regime_atr`	(monkey-patch THRESHOLDS)	2.5 ~ 5.0	ATR crash threshold	⚠️ Weak
`max_rounds`	`INVEST_MAX_DEBATE_ROUNDS`	1 ~ 3	Cross-challenge rounds	❌ Placebo
`cio_confidence_cap`	`INVEST_CIO_CONFIDENCE_CAP`	0.7 ~ 0.95	CIO confidence cap	❌ Placebo

How Optuna searches: TPE Sampler runs 30 backtests, each trying a set of parameters, learning from the (params, reward) history which direction to try next.

SQLite study: workspace/optuna_study.db stores all trial states, supporting checkpoint resume + multi-process parallelism.

Parallel test: 3 workers sharing the study DB, wall clock 5h → 1h (3x speedup, no DeepSeek rate limiting).

Actual Results (First Round, 2026-05-11)

v0 baseline → v1 prompt → Optuna

Stage	Reward	Annualized	HOLD %	Key
v0 baseline	-0.0006	0%	100%	100% HOLD parrot
v1 prompt (cash opportunity cost)	+0.398	+25.4%	44%	Unblocked the key
Optuna best (trial #20)	+0.422	+34.4%	-	Ceiling found in 30 trials
Hold-out 6w validation	+0.288	+17.4%	-	Generalization OK, Sharpe 1.59

Full data in docs/training_report.md.

What Changed in the v1 Prompt

Added a "cash opportunity cost" hard rule to the CIO prompt (agents/cio.py):

CONCENTRATION_PCT < 20% (position < 20%, dry powder ≥ 80%):
- HOLD is not allowed
- Default to at least ACCUMULATE, alloc = dry_powder × 5~10%
- Exemption: Macro=risk_off AND Risk=high_risk (both AND)

Financial logic: Waiting for a pullback ≠ waiting with zero position; it means keeping 90% dry powder to wait for a lower entry.

Negative Results (Equally Important)

Reward landscape is almost flat (30 trial stdev 0.037 vs mean 0.40) → hyperparameters are not the main driver of reward
max_rounds=1 vs 3 reward is almost identical → multi-round debate is a placebo (hypothesis: the LLM's second round just rephrases the first)
alloc_aggressiveness 0.06 vs 0.25 reward is also similar → large vs small orders are essentially the same strategy
Reward 0.42 = ceiling of the current prompt architecture——Bayesian search converged; adding 100 more trials won't break through

Conclusion: The real bottleneck is prompt content (which examples, what kind of wording), not hyperparameters.

How to Run It Yourself

Run a Diagnosis (Check if the current prompt is a parrot)

bash

cd ~/openInvest
set -a && . ./.env && set +a
rm -rf /tmp/my_diagnose

# Use backtest_runner wrapper (auto monkey-patches data paths + preheats 10y yfinance)
.venv/bin/python -m scripts.backtest_runner \
  --workspace /tmp/my_diagnose \
  --start 2024-05-13 --end 2024-11-15 \
  --assets NDQ.AX,GC=F

# Analyze verdict distribution
ls /tmp/my_diagnose/memory/.backtest/ | wc -l
grep -h '\*\*Verdict\*\*:' /tmp/my_diagnose/memory/.backtest/*/*.md | sort | uniq -c

Run Optuna Parameter Tuning

bash

.venv/bin/python -m scripts.rl_train \
  --workspace /tmp/my_optuna \
  --train-start 2024-05-13 --train-end 2024-11-15 \
  --assets NDQ.AX,GC=F \
  --n-trials 30

# View best params
.venv/bin/python -c "
import optuna
s = optuna.load_study(study_name='invest_committee', storage='sqlite:////tmp/my_optuna/optuna_study.db')
print(s.best_params)
print('reward:', s.best_value)
"

Run Hold-out Validation (Prevent Overfitting)

bash

.venv/bin/python -m scripts.holdout_validate \
  --optuna-workspace /tmp/my_optuna \
  --holdout-workspace /tmp/my_holdout \
  --start 2024-11-18 --end 2024-12-31 \
  --assets NDQ.AX,GC=F

Parallel Acceleration (3-worker)

bash

# Terminal A
.venv/bin/python -m scripts.rl_train --workspace /tmp/my_optuna --n-trials 10 ...

# Terminal B (start simultaneously)
.venv/bin/python -m scripts.rl_train --workspace /tmp/my_optuna --n-trials 10 ...

# Terminal C
.venv/bin/python -m scripts.rl_train --workspace /tmp/my_optuna --n-trials 10 ...

Optuna SQLite study coordinates automatically; no duplicate trials.

Key File Reference

# Training / Tuning Entry Points
scripts/rl_train.py            ← Optuna main entry, TPE bayesian search
scripts/backtest_runner.py     ← Workspace isolation wrapper (mandatory wrapper)
scripts/run_walk_forward.py    ← Walk-forward simulation runner
scripts/holdout_validate.py    ← Hold-out validation (prevent overfitting)
scripts/build_dspy_trainset.py ← Prepare trainset for DSPy (verdict + 7d return)

# Core Simulator / Metrics
core/paper_trade_simulator.py  ← PaperAccount + execute_verdict
core/strategy_metrics.py       ← Annualized / Sharpe / Sortino / vs benchmark
core/backtest_reward.py        ← Reward function (single number)
core/committee/cio_parse.py    ← parse_cio_memo includes INVEST_* env clamp

# Experiment Archive
experiments/optuna_final_summary.json           ← 30-trial data
experiments/dspy_trainset_v1_2024_05_to_11.json ← DSPy candidate trainset
experiments/prompt_variants/                    ← Prompt v0/v1 archive
docs/training_report.md                         ← Complete first-round training report

Safety Guarantees

✅ Never contaminates real positions: All training workspaces are in /tmp/, monitored by core.memory_store.MEMORY_ROOT ✅ Does not connect to real trading: PaperTradeSimulator is pure Python in-memory, does not call any broker ✅ Prevents lookahead: Stage 1 fixed 3 data lookahead vulnerabilities + enforced LLM training cutoff rejection ✅ Reproducible: Each Optuna trial uses a fixed seed; trainsets are saved to disk and can be re-run

DSPy Stage (Scaffold Implemented, Smoke Test Shows +10pp)

When Optuna has pushed "hyperparameter tuning" to its limit (0.42 is the ceiling of the current prompt architecture), the next breakthrough is optimizing the prompt content itself——this is exactly what DSPy does.

DSPy is a tool from Stanford NLP, designed for "reverse optimization" of LLM programs——given a trainset

metric, it automatically selects the best few-shot examples and prompt templates.

Implementation Status (2026-05-11):

✅ DSPy 3.2.1 installed
✅ experiments/dspy_trainset_v1_*.json 66 samples ready
✅ scripts/rl_optimize_prompts.py scaffold complete
✅ Smoke test passed: baseline 0.600 → optimized 0.700 (+10pp)
✅ Full run complete (10 candidates × 46 train × 20 dev, 66 min): baseline 0.725 → optimized 0.825 (+10pp / +13.8% relative)
- archive: experiments/dspy_optimized_v1_full.json (contains 10 best demos)
- summary: experiments/dspy_optimized_v1_full.summary.json
⏳ Injecting optimized few-shot examples back into agents/cio.py pending manual merge → re-run walk-forward validation to see if reward breaks 0.42

Running DSPy

bash

# Smoke (~5 min ~¥0.5, verify pipeline)
python -m scripts.rl_optimize_prompts \
  --trainset experiments/dspy_trainset_v1_2024_05_to_11.json \
  --output experiments/dspy_optimized_v1.json \
  --smoke-test

# Full run (~30 min ~¥3)
python -m scripts.rl_optimize_prompts \
  --trainset experiments/dspy_trainset_v1_2024_05_to_11.json \
  --output experiments/dspy_optimized_v1.json \
  --n-candidates 10 --n-bootstrap 4

Expected output: Break reward 0.42 → 0.5+ (smoke showed dev score already +10pp; full run may be even better).

Current Scaffold Limitations

The current DSPy module only wraps "market context → verdict" single-step prediction, not redoing the 4-role cross-challenge. For deeper optimization (letting DSPy learn how each role reasons independently + how to synthesize), you would need to refactor agents/sdk_agent.py into a dspy.Module, which would take 1-2 days of work.

The current minimal viable version already produces a +10pp improvement, so a major refactor may not be worthwhile.

Prompt Organization: SKILL.md Pattern (Deployed)

Referencing the prompt organization pattern from OpenClaw and NousResearch/hermes-agent, already migrated to SKILL.md.

Directory Structure

agents/
├── skills/                          ← Prompt bodies (markdown, decoupled from code)
│   ├── cio/SKILL.md
│   ├── macro_strategist/SKILL.md
│   ├── quant/
│   │   ├── SKILL.md                 # round_label="opening"
│   │   └── SKILL_rebuttal.md        # round_label="rebuttal"
│   ├── risk_officer/
│   │   ├── SKILL.md
│   │   └── SKILL_rebuttal.md
│   ├── wealth_context_officer/SKILL.md
│   └── README.md
├── skills_loader.py                 ← load_skill() + placeholder rendering (zero dependencies)
├── cio.py                           ← Thin wrapper: build_cio_prompt() = load_skill("cio", ...)
├── macro_strategist.py
├── quant.py
├── risk_officer.py
└── wealth_context_officer.py

SKILL.md Format

markdown

---
name: cio
description: Synthesizes Macro/Quant/Risk outputs to give the final verdict
role: cio
---

You are the Chief Investment Officer (CIO), having just heard ... {{asset_name}} ({{asset_symbol}}) ...

Runtime placeholders ( / ) are replaced by skills_loader.load_skill().

Workflow for Changing a Prompt

Directly edit the corresponding agents/skills/<role>/SKILL.md
Do not touch .py (unless adding a new role / new placeholder)
Run pytest tests/test_skills_loader.py to confirm frontmatter parsing is correct
Commit only the .md diff —— completely separated from code changes

Comparison with OpenClaw / Hermes

Dimension	Current invest	OpenClaw / Hermes-Agent
Organization pattern	✅ SKILL.md + frontmatter	✅ SKILL.md + frontmatter
Dynamic fields	✅ placeholders (zero dependencies)	⚠️ jinja2 / custom implementations
DSPy/GEPA integration	⏳ Pending integration (`SkillModule(dspy.Module)`)	✅ Already exists (`evolution/skills/skill_module.py`)
Change prompt without touching code	✅	✅

Next Step (Plan Stage 5)

Referencing NousResearch/hermes-agent-self-evolution:evolution/skills/skill_module.py, wrap SKILL.md as a dspy.Module, so that the DSPy optimization in scripts/rl_optimize_prompts.pydirectly reads and writes .md files (currently, best demos must be manually converted to .py for injection).

ADR Index

These decisions will be written to adr/ in the future:

ADR-008: Why use Optuna instead of PPO/DPO (LLM is discrete and non-differentiable)
ADR-009: Paper trade vs verdict-level evaluation (strategy-level has 10x higher signal-to-noise ratio)
ADR-010: Training data cutoff chosen as 2024-06-30 (DeepSeek's own training data)
ADR-011: Temporarily not switching to SKILL.md pattern (see "Prompt Organization" trade-offs above)

11 — RL Training / Backtest / Parameter Search ​

TL;DR: What We Do Is Not "Training" ​

Concept Map ​

Why This Is Necessary ​

4-Stage Pipeline ​

Stage 1: Lookahead Prevention ​

Stage 2: Data Isolation (Workspace) ​

Stage 3: Paper Trade Simulation (Core) ​

Stage 4: Optuna Automatic Parameter Tuning ​

Actual Results (First Round, 2026-05-11) ​

v0 baseline → v1 prompt → Optuna ​

What Changed in the v1 Prompt ​

Negative Results (Equally Important) ​

How to Run It Yourself ​

Run a Diagnosis (Check if the current prompt is a parrot) ​

Run Optuna Parameter Tuning ​

Run Hold-out Validation (Prevent Overfitting) ​

Parallel Acceleration (3-worker) ​

Key File Reference ​

Safety Guarantees ​

DSPy Stage (Scaffold Implemented, Smoke Test Shows +10pp) ​

Running DSPy ​

Current Scaffold Limitations ​

Prompt Organization: SKILL.md Pattern (Deployed) ​

Directory Structure ​

SKILL.md Format ​

Workflow for Changing a Prompt ​

Comparison with OpenClaw / Hermes ​

Next Step (Plan Stage 5) ​

ADR Index ​

11 — RL Training / Backtest / Parameter Search

TL;DR: What We Do Is Not "Training"

Concept Map

Why This Is Necessary

4-Stage Pipeline

Stage 1: Lookahead Prevention

Stage 2: Data Isolation (Workspace)

Stage 3: Paper Trade Simulation (Core)

Stage 4: Optuna Automatic Parameter Tuning

Actual Results (First Round, 2026-05-11)

v0 baseline → v1 prompt → Optuna

What Changed in the v1 Prompt

Negative Results (Equally Important)

How to Run It Yourself

Run a Diagnosis (Check if the current prompt is a parrot)

Run Optuna Parameter Tuning

Run Hold-out Validation (Prevent Overfitting)

Parallel Acceleration (3-worker)

Key File Reference

Safety Guarantees

DSPy Stage (Scaffold Implemented, Smoke Test Shows +10pp)

Running DSPy

Current Scaffold Limitations

Prompt Organization: SKILL.md Pattern (Deployed)

Directory Structure

SKILL.md Format

Workflow for Changing a Prompt

Comparison with OpenClaw / Hermes

Next Step (Plan Stage 5)

ADR Index