Data Model

v2 schema (2026-05+): cash dict + holdings list, any yfinance symbol plug-and-play. This chapter explains schema evolution, migration path, and concurrent RMW closure.

← 04-execution-paths · Wiki Index · 06-api →

---

1. Why Markdown + frontmatter

Abandoned:

• ❌ Single file user_profile.json: requires full load + full write-back each time, not diff-friendly
• ❌ SQLite: backup requires export, diff is unreadable, git ignore for the entire .db file is too coarse-grained

Chosen: frontmatter Markdown, one file per concept:

---
schema_version: 2
cash:
  CNY: 50000
  AUD: 1000
holdings:
  - symbol: NDQ.AX
    units: 50
    avg_cost: 38.50
---
# Current Holdings
- CNY Cash: ¥50,000
- NDQ.AX: 50 shares @ A$38.50

Benefits:

• frontmatter for code reading/writing (atomic write)
• body for LLM to read directly (no secondary formatting needed)
• The same file is always consistent for both Python and LLM
• git diff friendly (although memory/ is git ignored, it's human-readable during backup)
• Backup = cp -a memory/ memory.bak/

---

2. v1 → v2 Evolution

v1 (Before 2026-04, Deprecated)

---
cash_cny: 50000
aud_cash: 1000
ndq_shares: 50
ndq_avg_cost_aud_per_share: 38.50
gold_grams: 124
gold_avg_cost_cny_per_gram: 1008.79
---

Problems:

• Adding a new asset (e.g., Apple stock AAPL) requires code changes: add aapl_shares field to portfolio.md, add getter to PortfolioManager, add rendering logic to render_body, add /aapl_set command to NapCat—at least 4 places
• Adding a new currency (e.g., USD/EUR) is similar
• More fields lead to exponentially increasing schema maintenance costs

v2 (2026-05+)

---
schema_version: 2
cash:
  CNY: 50000
  AUD: 1000
  USD: 0
holdings:
  - symbol: NDQ.AX
    kind: etf
    units: 50
    unit_label: shares
    avg_cost: 38.50
    cost_currency: AUD
    channel: CommSec
    display_name: BetaShares Nasdaq 100 ETF
    proxy_kind: direct
  - symbol: GC=F
    kind: metal
    units: 124.0
    unit_label: grams
    avg_cost: 1008.79
    cost_currency: CNY
    channel: Zheshang Accumulated Gold
    yfinance_proxy: GC=F
    proxy_kind: gold_cny_per_gram
    sell_fee_pct: 0.0038
---

Core Abstraction:

• cash: dict[currency_code, amount]
• holdings: list[holding], each holding is self-describing (symbol / kind / units / unit_label / cost / channel)

Benefits:

• Adding a new asset = user clicks 1 button in the GUI "Add Asset" (no code changes needed)
• Adding a new currency = cash["USD"] = 100 (no code changes needed)
• Any yfinance symbol (NDQ.AX / AAPL / 005827.SS / BTC-USD) can be stored

See adr/003-v2-data-model.md for the decision rationale.

---

3. v1 Automatic Fallback (Painless Migration)

PortfolioManager automatically handles fallback at read-time:

@property
def cash(self) -> Dict[str, float]:
    cash = dict(self.portfolio.get("cash") or {})
    if not cash:
        # v1 fallback: aggregate from flat fields
        cny = self.portfolio.get("cash_cny")
        aud = self.portfolio.get("aud_cash")
        if cny is not None: cash["CNY"] = float(cny)
        if aud is not None: cash["AUD"] = float(aud)
    return cash

@property
def holdings(self) -> HoldingsView:
    raw = list(self.portfolio.get("holdings") or [])
    if not raw:
        # v1 fallback: construct from ndq_shares / gold_grams
        ndq = float(self.portfolio.get("ndq_shares", 0) or 0)
        gold = float(self.portfolio.get("gold_grams", 0) or 0)
        if ndq > 0: raw.append({...})
        if gold > 0: raw.append({...})
    return HoldingsView(raw)

On Write: with_portfolio_tx() cleans up all v1 fields upon exit + marks schema_version: 2.

→ Effect: A v1 portfolio.md is automatically upgraded to v2 the first time it is written. Zero migration script downtime.

→ But it is recommended to actively run the migration: python scripts/migrate_portfolio_to_holdings.py for a one-time upgrade (keeps a backup).

---

4. Holding Field Details

- symbol: NDQ.AX            # Required, yfinance format ticker
  kind: etf                 # Required: stock|etf|metal|crypto|bond|fund|other
  units: 50                 # Required, holding quantity
  unit_label: shares        # Required, shares / grams / oz / units
  avg_cost: 38.50           # Required (fill 0 for tracking-only holdings)
  cost_currency: AUD        # Required, 3-5 letter ISO code
  channel: CommSec          # Required, channel (CommSec / Zheshang Accumulated Gold / Robinhood)
  display_name: ...         # Optional, for UI display
  proxy_kind: direct        # Optional, market data calculation method
  yfinance_proxy: GC=F      # Optional, used when the market data ticker differs from the symbol
  is_tracking_only: false   # Optional, observation only, no PnL calculation
  sell_fee_pct: 0.0038      # Optional, sell fee rate

What kind Affects

kind	Effect
stock / etf	Quote directly uses yfinance Close price
metal	Uses yfinance_proxy + FX reverse calculation (common for gold: GC=F + USDCNY → CNY/g)
crypto	Uses BTC-USD / ETH-USD etc.
bond / fund	Same as stock, but different GUI identification
other	Fallback type

What is proxy_kind

A market data proxy protocol. Used when cost_currency differs from the yfinance quote currency.

Example: The average cost for Zheshang Accumulated Gold is ¥/gram, but yfinance GC=F quotes in USD/oz. Setting proxy_kind: gold_cny_per_gram lets utils/gold_price.py automatically:

1. Fetch GC=F → USD/oz
2. Fetch USDCNY=X → exchange rate
3. ÷ 31.1035 (oz to grams)
4. × Zheshang spread (from strategy.price_offset_pct)
5. = CNY/gram quote

proxy_kind Values:

• direct: The symbol is the yfinance ticker (NDQ.AX / AAPL)
• gold_cny_per_gram: The reverse calculation chain described above
• Future expansion: crypto_cny (via USDT intermediary), etc.

is_tracking_only

Checking the "Tracking Only" checkbox in the GUI → is_tracking_only: true. Effects:

• units can be 0
• Not included in total assets / PnL
• Not passed to the Committee Risk Officer prompt (does not affect concentration calculations)
• Displayed in a separate group on the Dashboard

→ The user's entry point for "I want to watch this stock" without pretending to hold it.

---

5. Concurrency Safety

5.1 Write Operations Must Go Through with_portfolio_tx()

from core.portfolio_manager import PortfolioManager

pm = PortfolioManager()
with pm.with_portfolio_tx() as p:
    cash = dict(p.get("cash") or {})
    cash["CNY"] = cash.get("CNY", 0) + 100
    p["cash"] = cash
# On with exit:
# 1. Clean up v1 old fields
# 2. p["schema_version"] = 2
# 3. validate_portfolio(p.metadata)  ← Pydantic strong validation
# 4. p.set_body(_render_portfolio_body_v2(p))
# 5. atomic write (tmp → fsync → os.replace)

On Exception: raise within the with block → entire tx is rolled back, no disk write (commit-on-success).

5.2 fcntl File Lock (Cross-Process)

core/memory_store.py:_file_lock:

@contextmanager
def _file_lock(path: Path):
    lock_path = path.with_suffix(path.suffix + ".lock")
    with open(lock_path, "w") as fp:
        fcntl.flock(fp.fileno(), fcntl.LOCK_EX)
        try:
            yield
        finally:
            fcntl.flock(fp.fileno(), fcntl.LOCK_UN)

→ The same file can be written by the napcat_bot process + invest-web process + scheduler process simultaneously, without losing updates.

5.3 Atomic Write (Prevents Process Kill Issues)

def _atomic_write_text(path: Path, text: str):
    tmp = path.with_suffix(path.suffix + f".tmp.{os.getpid()}")
    with open(tmp, "w") as f:
        f.write(text)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)  # POSIX atomic operation

→ If the process is killed with kill -9 while writing portfolio.md, the worst case is that the new content is not written, but the old content is completely preserved. It will never be half-written.

5.4 Concurrency Stress Test

tests/test_memory_store.py:

50 threads concurrently cash["CNY"] += 1   → Final delta = 50.0   (0 lost updates)
20 rounds of scheduler deduction + napcat deposit race  → Delta precise = -37880  (0 lost updates)

---

6. memory/ Directory Overview

memory/
├── MEMORY.md                  # Index (read by both humans and agents)
├── DREAMS.md                  # Narrative dream diary written by Dreaming (enabled in v3)
├── user.md                    # User identity / risk preference / monthly salary
├── strategy.md                # Investment strategy / target_assets
├── portfolio.md               # Current holdings (v2: cash dict + holdings list)
├── portfolio_history.jsonl    # Transaction history (append-only)
├── llm_usage.jsonl            # LLM call telemetry (v3 transparency)
├── insights/                  # Long-term insights from Deep Sleep passing threshold gates
│   └── *.md
├── daily/                     # Daily committee resolutions (v3 multi-asset = multi-file)
│   └── YYYY-MM-DD/<SYMBOL>.md
├── .committee/                # Committee asynchronous task status
│   └── <task_id>/
│       ├── status.json
│       └── tool_calls.jsonl
├── .runs/                     # (Reserved) run/session data model
├── .dreams/                   # Dreaming subsystem private
│   ├── short-term-recall.json
│   ├── candidates.json
│   └── events.jsonl
└── .state/                    # Simple KV
    └── processed_emails.json

See memory_layout.md for details.

---

7. Historical Stress Tests + Hardening Log

Failure Mode	Fix	Source
portfolio.md half-written when process killed, state corrupted	Atomic write three-step	core/memory_store.py:_atomic_write_text
napcat deposit + scheduler deduction TOCTOU	Single lock RMW + transaction	core/memory_store.py:transaction
Schema field drift (user modified portfolio.md with illegal fields)	Pydantic v2 strong validation + render_body uses template	core/schemas.py
Multi-connector implementation drift (napcat modifies dict directly)	Force use of PortfolioManager interface	core/portfolio_manager.py

---

Next Steps

→ 02-agents.md to see how holdings data enters the Risk Officer prompt

→ 07-extending.md#adding-new-assets to see how to add new assets via GUI / programmatically

→ adr/003-v2-data-model.md to see the v1 → v2 decision details