Architecture Overview

The first chapter for developers to build a mental model. After reading, you should be able to answer: "Where does data come from → How many steps does it go through → How are decisions formed → Where is it written to."

← Back to Wiki Index · Next Chapter: 4-Role Committee →

---

In a Nutshell

openInvest is a 4-layer pipeline: External Trigger → LLM Decision Core → Business State Machine → Persistence. Each layer can be independently replaced, and neither knows the implementation details of the other.

┌─────────────────────────────────────────────────────────────┐
│  CONNECTORS (External Triggers, Multi-Consumer Pattern)     │
│  napcat_bot.py · web_api.py · skills/invest/scripts/run.sh  │
└────────────────┬────────────────────────────────────────────┘
                 │  Calls business functions (not directly to LLM)
                 ▼
┌─────────────────────────────────────────────────────────────┐
│  AGENTS (LLM Role Definitions + Prompts)                     │
│  macro · quant · risk · cio  ← All orchestrated by core.committee │
└────────────────┬────────────────────────────────────────────┘
                 │  Coordinator-Worker orchestration, information separation
                 ▼
┌─────────────────────────────────────────────────────────────┐
│  CORE (Business Core + Concurrency Safety + Pydantic Validation) │
│  committee · portfolio_manager · memory_store · regime      │
└────────────────┬────────────────────────────────────────────┘
                 │  fcntl lock + atomic write
                 ▼
┌─────────────────────────────────────────────────────────────┐
│  MEMORY (Frontmatter Markdown Persistence)                   │
│  portfolio.md · strategy.md · daily/<date>/<sym>.md         │
└─────────────────────────────────────────────────────────────┘

---

1. Connectors Layer (Multi-Consumer)

The entry point for the external world into invest. Currently 3 connectors, functionally equivalent, each responsible for its own protocol:

Connector	Trigger Protocol	Suitable For
connectors/napcat_bot.py	QQ WebSocket Private Chat Commands (/balance /deposit 100)	Quick Mobile Operations
connectors/web_api.py	HTTP REST + SSE	Web GUI / Programmatic Integration
skills/invest/scripts/run.sh (CLI)	Claude Code Skill	Let Claude act as the coordinator itself

Key Constraint: Connectors must only perform protocol conversion; all business logic must be forwarded to core/. Violating this leads to behavioral drift between connectors (lesson learned: early napcat modifying the portfolio dict directly caused Web writes to fail).

See the README in each connector subdirectory for details:

• connectors/README.md
• skills/README.md

---

2. Agents Layer (LLM Roles)

Four independent LLM sessions, information isolated:

Role	Can See	Cannot See	Output
Macro Strategist	VIX / TNX / USDCNY / Global Macro	User Holdings, Technical Indicators	SIGNAL + STRENGTH + SCORE
Quant Analyst	Technical Indicators (RSI/MA/Percentile) + REGIME Probability Profile + Valuation/Sentiment Fact Blocks	User Holdings	Bullish/Bearish + Signal Strength
Risk Officer	Position Concentration / Unrealized P&L Buffer / Dreaming Insights	Technical Indicators	Risk Level + Concentration Score
CIO	Full Transcript of the Three	—	BUY/ACCUMULATE/HOLD/TRIM/SELL + confidence

Cross-Challenge Multi-Round Debate (v3):

• Round 1: Quant + Risk independent statements (parallel)
• Round 2..N: See each other's previous output, adjust themselves (max 4 rounds)
• Convergence Detection: SIGNAL + STRENGTH stable for two rounds → early exit
• CIO synthesizes the verdict after the transcript is complete

See 02-agents.md for details.

---

3. Core Layer (Business Core)

3.1 Orchestrator: core/committee/ Package (debate.py:run_committee as Entry Point)

def run_committee(asset, market_data, macro_view, portfolio_summary,
                  *, max_debate_rounds=4, progress_callback=None):
    # Round 1: Quant + Risk parallel (information isolation)
    # Round 2..N: cross-challenge with convergence detection
    # CIO: synthesize verdict from transcript

progress_callback allows the Web SSE endpoint to push stage progress in real-time ("round_1_done" / "round_2_done" / "cio_done").

3.2 State Machine: core/portfolio_manager.py

with pm.with_portfolio_tx() as p:
    cash = dict(p.get("cash") or {})
    cash["CNY"] = cash.get("CNY", 0) + amount
    p["cash"] = cash
# On exit from `with`: automatically validate → render body → atomic write

All write operations must go through this RMW closure — ensuring fcntl file lock + Pydantic schema validation + atomic write. On exception, the entire tx is not persisted (commit-on-success).

3.3 Data Validation: core/schemas.py

Pydantic v2 strict schemas, mandatory before writing:

• cash must be dict[str, NonNegativeFloat]
• holdings[].units >= 0
• holdings[].symbol must be non-empty
• Any violation → ValidationError, cannot write to memory

3.4 REGIME Classification and Probability Profile: core/regime.py + core/regime_probability.py

6 regime classes (uptrend/downtrend/range_bound/crash/recovery/unknown), deterministic algorithm, no LLM involvement. As of 2026-05-31, regime is no longer translated into directional locks (ablation proved directional lock layer = 1 lever + 2 decorations), but instead fed to Quant as a neutral probability profile:

uptrend / downtrend / range_bound / recovery
            → Probability profile of OHLC 30d forward return for that (asset, regime)
              (median / probability of breaking below current price / sample count),
              direction judgment returned to LLM + data
crash       → Force neutral (executability constraint, not directional prediction)
unknown     → Maintain original plan (conservative default due to lack of data)

Deterministic directional intervention goes through `parse_cio_memo` post-processing:
Flash Crash Defense (VIX Sentinel OR ATR Volatility Mutation Ratio → Buy-side downgrade)
and explicit risk profiles (`risk_profile=aggressive` uptrend leverage, default off).
The probability table also provides 30/60/90 multi-window distributions + 4 path shapes
(see [15-path-probability.md](15-path-probability.md)).

See 02-agents.md#regime-classification-and-neutral-probability-profile for details.

---

4. Memory Layer (Markdown Persistence)

No relational database; uses frontmatter Markdown:

---
schema_version: 2
cash:
  CNY: 50000
  AUD: 1000
holdings:
  - symbol: NDQ.AX
    units: 50
    avg_cost: 38.50
    cost_currency: AUD
---

# Current Holdings
- CNY Cash: ¥50,000
- NDQ.AX: 50 shares @ A$38.50

Why this approach:

• Frontmatter for code read/write (atomic)
• Body for LLM to read directly (no secondary formatting needed)
• Same file, Python and LLM always see the same thing
• git diff friendly, backup is just cp -a

See 05-data-model.md and memory_layout.md for details.

---

5. Data Flow (One Complete Committee Run)

Using POST /api/committee/run as an example:

1. web_api.committee_run()
     │
     ├─ Create task_id, write .committee/<task_id>/status.json (queued)
     ├─ asyncio.create_task(_run_committee_task)
     └─ Return task_id to frontend immediately
2. _run_committee_task (async, runs in background)
     │
     ├─ Pull strategy.target_assets → symbols
     ├─ run_macro_view(macro_data) ← Shared across assets
     ├─ on_progress({phase: "macro_done"})
     │
     ├─ for sym in symbols (ThreadPoolExecutor parallel):
     │     │
     │     └─ run_committee_for_symbol(sym, ...)
     │           │
     │           ├─ get_history_data(sym, "2y")
     │           ├─ compute_metrics(df) → regime
     │           ├─ regime_brief = format_regime_brief(metrics)
     │           │
     │           ├─ Round 1: _parallel_ask([quant, risk])
     │           ├─ on_progress({phase: "round_1_done", round: 1, ...})
     │           │
     │           ├─ Round 2..N: cross_challenge with convergence detection
     │           ├─ on_progress({phase: "round_N_done", ...})
     │           │
     │           ├─ CIO ask synthesizing transcript
     │           ├─ on_progress({phase: "cio_done"})
     │           │
     │           └─ Persist to daily/<date>/<sym>.md
     │
     └─ Write .committee/<task_id>/status.json (done + result)

3. Frontend subscribes to SSE GET /api/committee/live/{task_id}
     │
     └─ Each on_progress writes status.json → SSE stream pushes event

Key Points:

• Frontend gets task_id within 1.5s, HTTP is not blocked
• Background ThreadPool enables true parallelism for multiple assets (v2 was serial)
• Writes status.json at each stage, SSE endpoint watches file mtime
• 25s keepalive prevents CF Access 5min idle timeout

---

6. Concurrency Model

Four concurrency scenarios:

Scenario	Risk	Mitigation
napcat deposit + scheduler deduction running simultaneously	TOCTOU lost updates	MemoryStore.transaction() single lock RMW
Multi-thread ThreadPool calling LLM simultaneously	(Read-only, no conflict)	—
Web API write + napcat write simultaneously	TOCTOU	Same as above, fcntl is a process-level lock
Multiple committee tasks running simultaneously	status.json overwrites	One independent dir per task_id

Concurrency stress test (tests/test_memory_store.py):

50 threads concurrently cash["CNY"] += 1   → Final delta = 50.0  (0 lost updates)
20 rounds scheduler deduction + napcat deposit race → Exact delta = -37880  (0 lost updates)

See 05-data-model.md#concurrency-safety for details.

---

7. Extension Points

Each layer is designed to be independently replaceable:

• Add a connector (e.g., Telegram bot): Create a new file following the napcat_bot.py pattern, do not touch core
• Add an agent role (e.g., ESG Analyst): Add a prompt file in agents/, register in core/committee/debate.py:run_committee
• Change LLM provider (DeepSeek → OpenAI): Modify the client init in agents/agent.py, prompts remain unchanged
• Change persistence (Markdown → SQLite): Implement the MemoryStore interface, core remains unchanged

See 07-extending.md for details.

---

Next Steps

→ 02-agents.md to see what the 4 LLM roles actually do + how cross-challenge works

→ 05-data-model.md to see v2 schema evolution + concurrency safety details

→ 07-extending.md to see how to add new features