TODO

Documentation Index

Fetch the complete documentation index at: https://docs.novosky.app/llms.txt

Use this file to discover all available pages before exploring further.

​

Current state — Phase 17.3 (active)

​

OOS performance

​

In progress — immediate blockers

• VM cron (15.2): Wire weekly_optimize.py on trading VM
• Phase 15.5: Broker safety audit documented — see Broker safety audit

​

Phase 15 — Production transition

​

15.0 Broker-agnostic refactor

• config.json no longer requires "broker" key; all scripts load spread/leverage/swap dynamically from MT5 API /account and /symbols
• backtest.py — config-faithful backtester; use --max-lot/--cent-account for preset-style comparisons
• scripts/sweep.py — unified sweep replacing sweep_signal.py + sweep_pos_model.py; --target signal|pos|both
• scripts/check_broker_limits.py — fixed main guard; added --symbol flag

​

15.1 Position model validation

• 38d OOS: PF=4.71 vs baseline 4.27 (+10.3%), MaxDD=4.7%, 1 ML_Exit
• Sweep of 13 configs via scripts/sweep.py --target pos
• Thresholds updated: exit=0.80, min_prob_diff=0.25, min_bars_held=4

​

15.2 Automated retrain pipeline

• scripts/weekly_optimize.py — 13-phase autonomous pipeline (SHAP → tune → retrain → sweep → evaluate → push → commit → notify)
• Score improvement gate: 2% minimum to keep new models
• Incremental warmstart training already enabled by default (ml/train.py, ml/ensemble_trainer.py)

0 2 * * 0  cd /path/NOVOSKY && .venv/bin/python scripts/weekly_optimize.py >> logs/weekly.log 2>&1

​

15.3 Cloud monitoring

• Degradation detection is handled by the weekly walk-forward OOS gate in weekly_optimize.py (3 × 4-week folds, WR ≥ 55%, PF ≥ 1.8)
• Live alerts flow through trading/telegram_commands.py and scripts/notify.py

​

15.4 Telegram bot commands

​

15.5 Broker safety audit

• Run audit against live RoboForex server: python scripts/check_broker_limits.py — PASS (2026-05-10: all limits confirmed, spread=14.59, stops_level=0, lot 0.01–100, contract_size=1.0)
• Run audit against IC Markets when account is provisioned
• Run audit against FundingPips when account is provisioned

​

15.6 Weekly validation cadence

​

Phase 16 — Risk guards & validation

​

16.1 Enable daily loss guard

# bot.py guard logic (active):
_loss_pct = config.get("max_daily_loss_pct")  # 3.0
_equity = _api_balance_to_usd(_acct.equity)
max_daily_loss = _equity * (_loss_pct / 100.0)
if daily_pl <= -max_daily_loss:
    send_telegram_message("🚨 [DAILY_LOSS_GUARD] ...")
    time.sleep(3600)

• Confirmed trading/bot.py:check_max_loss_profit() used fixed USD — upgraded to equity-relative
• Added max_daily_loss_pct: 3.0 to config.json; removed max_daily_loss: 99999
• Bot computes limit = equity * pct / 100 dynamically each cycle via _get_account().equity
• Dry-run: python trading.py --dry — guard logs [DAILY_LOSS_GUARD] on breach
• Daily reset uses LOCAL_TZ_OFFSET (+7 WIB) — resets at WIB midnight ✅

​

16.2 Equity curve filter

• equity_curve_filter config block added (enabled: false, lookback_trades: 10, max_drawdown_pct: 5.0)
• _recent_trade_pnls persisted in state.json and restored on restart
• Entry guard fires [EC_FILTER] when net loss over last N trades ≥ threshold % of equity
• Enable after validating OOS backtest with the filter active

​

16.3 Extend walk-forward OOS gate

• Phase 7b added to scripts/weekly_optimize.py between phase 7 and phase 8
• 3 non-overlapping 4-week OOS backtest folds run after every retrain
• Gate: all folds WR ≥ 55%, all folds PF ≥ 1.8, median PF ≥ 2.0
• Failure rolls back the snapshot immediately and skips push/commit
• Fold results appended to logs/wf_gate.log

​

16.4 Enable Kelly lot sizing

• Enable: ml_active_management.kelly_lot_sizing.enabled = true — already enabled
• OOS sweep (3 runs via --mode kelly): disabled Score=0.19 MaxDD=16.1% | half=0.5 Score=0.30 MaxDD=7.5% | full=1.0 Score=0.31 MaxDD=8.7%
• Kelly must stay on — disabled falls back to raw static sizing, MaxDD spikes to 16.1%
• Keeping max_kelly_fraction: 0.5 (half-Kelly): full-Kelly wins Score by 0.01 but costs 1.2% extra MaxDD — not worth it at $500 balance. Re-evaluate at higher equity.

​

16.5 Broker-Agnostic Multi-Account Architecture

{
  "accounts": {
    "exness_main": {
      "broker_id": "exness",
      "terminal_key": "terminal-one",
      "account_type": "cent",
      "currency_multiplier": 0.01,
      "risk_pct": 3.0,
      "max_lot": 5.0,
      "state_file": "state_exness_main.json"
    },
    "icmarkets_main": {
      "broker_id": "icmarkets",
      "terminal_key": "terminal-two",
      "account_type": "standard",
      "currency_multiplier": 1.0,
      "risk_pct": 2.0,
      "max_lot": 1.0,
      "state_file": "state_icmarkets_main.json"
    }
  }
}

# Normalize close price across brokers to remove spread bias
def normalize_ohlcv(df: pd.DataFrame, broker_id: str) -> pd.DataFrame:
    spread = BROKER_SPREADS[broker_id]          # e.g., 14.59 USD for RoboForex
    df["close_norm"] = df["close"] - spread / 2  # midpoint price
    return df

# Merge: outer join on timestamp, ffill broker gaps (< 2 bars)
merged = pd.merge(df_vt, df_ic, on="timestamp", how="outer", suffixes=("_vt", "_ic"))
merged["close"] = merged[["close_vt", "close_ic"]].mean(axis=1)  # average midpoints
merged.ffill(limit=2, inplace=True)  # fill short gaps; drop if both missing
merged.dropna(subset=["close"], inplace=True)

// ecosystem.config.js
module.exports = {
  apps: [
    { name: "novosky-roboforex",  script: "trading.py", args: "--account exness_main", interpreter: ".venv/bin/python" },
    { name: "novosky-ic",  script: "trading.py", args: "--account icmarkets_main", interpreter: ".venv/bin/python" },
    { name: "ws-server",   script: "trading/ws_server.py", interpreter: ".venv/bin/python" },
  ]
};

• Create config/accounts.json with schema above; add RoboForex entry only for now
• Add --account <account_id> flag to trading.py argument parser; load account config at startup; override risk_pct, max_lot, state_file, and API base URL from account config
• Update trading/bot.py: replace hardcoded state.json reference with config["state_file"]; replace API_BASE_URL with terminals[account["terminal_key"]]["url"]
• Update ml/train.py: add --brokers exness,icmarkets flag; when multiple brokers specified, fetch, normalize, and merge their OHLCV datasets; retrain on merged set
• Unit test multi-broker merge: assert merged DataFrame has no duplicate timestamps, close prices within 0.5% of each broker’s midpoint
• Update ecosystem.config.js with multi-process template (disabled by default — uncomment when IC Markets account is provisioned)
• config/terminals.json port + domain mapping already implemented — RoboForex on port 6542 / terminal-rf1.novosky.app
• Draft Hyperliquid adapter spec in docs/architecture/hyperliquid.mdx: REST endpoints for order placement (POST /exchange), position query (POST /info with type: clearinghouseState), and funding rate feed (/info with type: fundingHistory)

​

Phase 21 — Dynamic SL/TP & position model upgrades

​

21.1 Enable and validate ml_sltp (confidence-scaled TP/SL at entry)

conf_norm = clamp((confidence − threshold) / (1.0 − threshold), 0, 1)
effective_SL = base_sl_atr_mult × ATR × (1 − conf_norm × confidence_sl_adjust)
effective_TP = base_tp_atr_mult × ATR × (1 + conf_norm × confidence_tp_adjust)

• Enable: set ml_sltp.enabled = true in config.json — already enabled
• OOS backtest result: WR=48.8%, PF=1.72, MaxDD=9.0%, Score=0.26 (37d OOS, post-retrain)
• Score degraded vs disabled baseline (Score=0.30, MaxDD=7.5%) — swept confidence_sl_adjust × confidence_tp_adjust (20 combos via --mode ml_sltp)
• Best enabled combo sladj0.4_tpadj0.5 still lost on Score (0.29) and MaxDD (9.1%) — disabled per todo rule. Re-evaluate after more live trades improve confidence signal quality.
• min_tp floor check already present in the ml_sltp path (bot.py:4104–4105)
• Removed dead base_sl_atr_mult / base_tp_atr_mult config keys — SLTP regression model supplies the base multipliers, not static config

​

21.2 Trailing stop + lower min_bars_held

• Enable: set ml_trailing_stop.enabled = true, base_trail_atr_mult = 1.2, min_profit_atr = 0.8
• Run OOS backtest and compare Score vs baseline
• OOS result: WR=46.6%, PF=1.45, MaxDD=12.7%, Score≈0.19 — worse than baseline (Score=0.30, MaxDD=7.5%). Conflicts with partial_close breakeven SL — trailing stop cuts winners after BE move. Disabled. Saved params in config.json._note.
• Do not enable simultaneously with ml_sltp testing — change one variable at a time

• Set position_optimization.min_bars_held = 2 in config.json
• Run python scripts/sweep.py --target pos (pre-retrain): Score 0.30→0.34, best was exit0.60/bars2. Applied temporarily.
• Post clean-retrain sweep (2026-04-26): optimal config reverted to exit_threshold=0.80, min_prob_diff=0.25, min_bars_held=4 — Score=0.66, PF=2.69, DD=6.4%. After --no-warmstart, model EXIT signals more reliable at higher threshold. Applied to config.json + ml_config.json.

​

21.3 M1 intra-candle feature augmentation for position model

• Add compute_m1_fast_features(closes, opens, highs, lows, volumes) to ml/feature_engineering.py — standalone function, 5 scalars, handles short windows gracefully
• Update ml/position_labeling.py: add M1_FEATURES constant; generate_position_samples() accepts df_m1=None; at each sample timestamp, aligns 25 M1 bars via np.searchsorted (no lookahead leak — uses bars at or before M15 timestamp)
• New constant M1_FEATURES = ['m1_price_accel_5', 'm1_vol_ratio_5', 'm1_rsi_9', 'm1_atr_3', 'm1_body_pct'] — separate from POSITION_STATE_FEATURES (keeps 4), total position features 63→68
• ml/position_predictor.py: detects M1 from position_metadata.json (n_features >= 68); get_position_action() gains m1_features=None param; neutral values substituted if model expects M1 but none provided (backward-compatible)
• ml/position_trainer.py: generate_training_data(df_m1=None) — passes M1 DataFrame to labeling; updates all_feature_names and metadata when M1 features included
• ml/train.py: Step 4 fetches M1 bars (same date range as M15, from API); passes df_m1 to pos_trainer.generate_training_data(); patches model_compat.json with M1 feature list
• trading/bot.py: in check_ml_active_management(), fetches 30 M1 bars once per cycle (only when pos_predictor._has_m1_features=True); passes computed features to get_position_action()
• Retrain attempted: M1 API returned empty — MT5 HTTP server must be running during --refresh to populate M1 cache. Position model retrained on 63 features (same as before). No regression in val accuracy (XGB=0.753, LGB=0.743).
• M1 cache solution implemented: datasets/training_data_btcusd_m1.csv (same pattern as M15 CSV). --refresh with API running fetches and saves it; subsequent trains load from cache automatically. Path configurable via ml_config.json paths.m1_training_data.
• OOS backtest post-retrain (2026-04-26): WR=57.5%, PF=2.86, MaxDD=7.5%, Score=0.60 — doubled baseline score (was 0.30). SL hits: 24 (was 66–151). ML exits: 89/233 (38%). Re-sweep after retrain identified better config (0.80/0.25/bars4 → Score=0.66); applied.
• M1 retrain (when API running): python train_ml_model.py --ensemble --position --refresh --no-warmstart
  • --refresh fetches live M1 bars from MT5 API and saves to datasets/training_data_btcusd_m1.csv
  • --no-warmstart forces a clean fit (warmstart would load old 63-feature weights into 68-feature model)
  • Confirm MT5 HTTP server is running before starting: curl http://localhost:6542/health
• OOS sweep after M1 retrain: python scripts/sweep.py --target pos — sweeps exit_threshold × min_prob_diff × min_bars_held grid; apply best config to config.json
• Gate — current best is PF=2.86 (Score=0.60), so PF ≥ 4.0 is aspirational not immediate:
  • Accept retrain if: OOS PF ≥ current PF (2.86) AND EXIT precision ≥ 85% AND MaxDD ≤ current + 2%
  • Aspirational target: PF ≥ 4.0 once M1 cache is fully populated (more M1 bars = better features)
  • If retrain regresses (PF < 2.86): revert via ml/hf_hub.py --pull and investigate which M1 feature adds noise

​

21.4 Catastrophic SL + position model as primary exit

1. Phase 21.1 (ml_sltp) validated OOS — removed (ml_sltp is disabled; position-as-primary-exit does not depend on confidence-scaled TP/SL)
2. Position model EXIT precision ≥ 87% on OOS (current: 85%) — this is the primary readiness gate
3. ≥ 200 live dry-run trades with ml_active_management.enabled = true confirming EXIT fires at correct rate (target: ML_EXIT ≥ 60% of closes)
4. Telemetry already in place: each close logged as SL_HIT, TP_HIT, or ML_EXIT in ml_performance.csv ✅ (implemented)

"ml_active_management": {
  "position_as_primary_exit": {
    "enabled": false,
    "catastrophic_sl_atr_mult": 3.0,
    "emergency_tp_atr_mult": 5.0
  }
}

• position_as_primary_exit config block added to config.json (enabled: false)
• bot.py wired: if enabled, overrides effective_sl = ATR × catastrophic_sl_atr_mult, effective_tp = ATR × emergency_tp_atr_mult, logs [PRIMARY_EXIT_MODE]
• SL_HIT / TP_HIT / ML_EXIT telemetry added to ml_performance.csv via _close_type() helper OOS test methodology for wide-SL:

# Step 1: Measure wide-SL alone (TP still on) — understand the RR change
python backtest.py --oos-only --no-chart \
    --override "dynamic_sltp.sl_atr_multiplier=3.0"

# Step 2: Full mode — wide SL + no TP (position model drives exits)
python backtest.py --oos-only --no-chart \
    --override "position_as_primary_exit.enabled=true" \
    --override "dynamic_sltp.sl_atr_multiplier=3.0" \
    --override "dynamic_sltp.tp_atr_multiplier=99.0"  # effectively no TP

# Accept only if: WR ≥ 55%, MaxDD ≤ 20%, ML_EXIT rate ≥ 60% in simulation

• Run Step 1 OOS (wide-SL alone): log result to logs/wide_sl_test.log; accept if Score ≥ current − 2.0 (some score loss from wider SL is expected)
• Run Step 2 OOS (position-as-primary): accept only if all gate conditions above are met
• Dry-run 100 trades: tail pm2 logs novosky --lines 200 --nostream after each close; count ML_EXIT vs SL_HIT vs TP_HIT; confirm ML_EXIT ≥ 60%
• Only enable on live after all preconditions are met; announce in Telegram: [PRIMARY_EXIT_MODE] Enabled — SL=3.0×ATR, no TP

​

21.5 Risk model scope — architectural constraint

• Add a Scope: section to ml/risk_predictor.py module docstring stating the model outputs lot-sizing multipliers only

​

Phase 17 — Feature engineering

​

17.1 On-chain & derivatives features

# ml/data_sources/binance.py
import requests, pandas as pd

def fetch_funding_rate(symbol="BTCUSDT", limit=500) -> pd.DataFrame:
    url = "https://fapi.binance.com/fapi/v1/fundingRate"
    r = requests.get(url, params={"symbol": symbol, "limit": limit}, timeout=10)
    df = pd.DataFrame(r.json())
    df["timestamp"] = pd.to_datetime(df["fundingTime"], unit="ms", utc=True)
    df["funding_rate"] = df["fundingRate"].astype(float)
    # Normalize: clamp to [-0.003, 0.003] (99th percentile range), then scale to [-1, 1]
    CLIP = 0.003
    df["funding_rate_norm"] = df["funding_rate"].clip(-CLIP, CLIP) / CLIP
    return df[["timestamp", "funding_rate_norm"]].set_index("timestamp")

# Forward-fill to M15 grid:
def merge_funding_to_m15(df_m15: pd.DataFrame, df_funding: pd.DataFrame) -> pd.DataFrame:
    df_m15 = df_m15.join(df_funding, how="left")
    df_m15["funding_rate_norm"] = df_m15["funding_rate_norm"].ffill().fillna(0.0)
    return df_m15

def fetch_open_interest(symbol="BTCUSDT") -> pd.DataFrame:
    url = "https://fapi.binance.com/futures/data/openInterestHist"
    r = requests.get(url, params={"symbol": symbol, "period": "15m", "limit": 500}, timeout=10)
    df = pd.DataFrame(r.json())
    df["timestamp"] = pd.to_datetime(df["timestamp"], unit="ms", utc=True)
    df["oi"] = df["sumOpenInterest"].astype(float)
    # 4h rolling pct change = 16 × 15min bars
    df["oi_change"] = df["oi"].pct_change(16).clip(-1, 1).fillna(0.0)
    return df[["timestamp", "oi_change"]].set_index("timestamp")

def fetch_fear_greed() -> pd.DataFrame:
    r = requests.get("https://api.alternative.me/fng/?limit=365", timeout=10)
    df = pd.DataFrame(r.json()["data"])
    df["timestamp"] = pd.to_datetime(df["timestamp"].astype(int), unit="s", utc=True)
    df["fear_greed"] = df["value"].astype(float) / 100.0  # normalize 0–1
    return df[["timestamp", "fear_greed"]].set_index("timestamp")

• Create ml/data_sources/binance.py with fetch_funding_rate, fetch_open_interest, fetch_fear_greed — each returns a UTC-indexed DataFrame
• Add to ml/train.py data loading block: after df_m15 is built, call all three fetchers and merge via left-join + ffill — same pattern as M1 features
• Add feature names to model_compat.json["features"] in this order (append at end): funding_rate_norm, oi_change, fear_greed
• Run python scripts/retrain.py --ensemble --no-warmstart with the 3 new features; run ml/shap_analysis.py; remove fear_greed from model_compat.json if its mean absolute SHAP < 0.001
• Cache fetched data to datasets/funding_rate.csv, datasets/oi_change.csv, datasets/fear_greed.csv — same pattern as training_data_btcusd_m1.csv
• OOS gate: ensemble with on-chain features must have Score ≥ current + 0.5; if not, revert — on-chain features add API call latency so they must pay for themselves

​

17.2 Market regime features

• volatility_regime — Compute ATR(14) percentile rank over a rolling 500-bar window; encode as continuous 0–1 (not bucketed) to avoid artificial boundaries
• w1_ema_bias — Resample M15 to W1 (504 bars); compute (close − EMA(10)) / close; forward-fill
• w1_rsi_norm — RSI(14) on W1 bars, normalized to 0–1; forward-fill

​

17.3 OHLCV data redundancy pipeline

​

17.4 Smart Money Concepts (SMC) features

pip install smartmoneyconcepts

​

Features to add

​

Implementation

# ml/feature_engineering.py

def add_smc_features(df: pd.DataFrame, swing_length: int = 10) -> pd.DataFrame:
    """Add SMC-derived features. No forward-looking columns used."""
    from smartmoneyconcepts import smc

    swing_hl = smc.swing_highs_lows(df, swing_length=swing_length)

    # Order Blocks — formation data only, strip MitigatedIndex
    ob = smc.ob(df, swing_hl)[["OB", "Top", "Bottom", "OBVolume"]]

    # Fair Value Gaps — formation data only, strip MitigatedIndex
    fvg = smc.fvg(df)[["FVG", "Top", "Bottom"]]

    # BOS / CHoCH — formation data only, strip BrokenIndex
    bos_choch = smc.bos_choch(df, swing_hl)[["BOS", "CHOCH", "Level"]]

    # Liquidity levels — formation data only
    liq = smc.liquidity(df, swing_hl)[["Liquidity", "Level"]]

    atr = df["atr_14"] if "atr_14" in df.columns else df["high"].combine(df["low"], max) - df["low"]

    close = df["close"]

    # --- OB features ---
    bull_ob_mask = ob["OB"] == 1
    bear_ob_mask = ob["OB"] == -1

    df["ob_bullish_present"] = 0
    df["ob_bearish_present"] = 0
    df["ob_bullish_dist"] = float("nan")
    df["ob_bearish_dist"] = float("nan")
    df["ob_volume_ratio"] = float("nan")

    vol_mean = df["tick_volume"].rolling(50, min_periods=1).mean()

    for i in range(len(df)):
        c = close.iloc[i]
        a = atr.iloc[i]
        if pd.isna(a) or a == 0:
            continue

        past_bull = ob[bull_ob_mask].iloc[:i]
        past_bear = ob[bear_ob_mask].iloc[:i]

        if not past_bull.empty:
            dists = (c - past_bull["Top"]) / a
            near = dists[(dists >= -3) & (dists <= 3)]
            if not near.empty:
                idx_min = near.abs().idxmin()
                df.at[df.index[i], "ob_bullish_present"] = 1
                df.at[df.index[i], "ob_bullish_dist"] = float(near[idx_min])
                df.at[df.index[i], "ob_volume_ratio"] = (
                    ob.at[idx_min, "OBVolume"] / vol_mean.iloc[i]
                    if vol_mean.iloc[i] > 0 else 0.0
                )

        if not past_bear.empty:
            dists = (past_bear["Bottom"] - c) / a
            near = dists[(dists >= -3) & (dists <= 3)]
            if not near.empty:
                idx_min = near.abs().idxmin()
                df.at[df.index[i], "ob_bearish_present"] = 1
                df.at[df.index[i], "ob_bearish_dist"] = float(near[idx_min])

    # --- FVG features ---
    bull_fvg = fvg[fvg["FVG"] == 1]
    bear_fvg = fvg[fvg["FVG"] == -1]

    df["fvg_bull_above"] = 0
    df["fvg_bear_below"] = 0
    df["fvg_bull_dist"] = float("nan")
    df["fvg_bear_dist"] = float("nan")

    for i in range(len(df)):
        c = close.iloc[i]
        a = atr.iloc[i]
        if pd.isna(a) or a == 0:
            continue
        past_bull_fvg = bull_fvg.iloc[:i]
        past_bear_fvg = bear_fvg.iloc[:i]
        if not past_bull_fvg.empty:
            above = past_bull_fvg[past_bull_fvg["Bottom"] > c]
            if not above.empty:
                nearest = (above["Bottom"] - c).idxmin()
                df.at[df.index[i], "fvg_bull_above"] = 1
                df.at[df.index[i], "fvg_bull_dist"] = (above.at[nearest, "Bottom"] - c) / a
        if not past_bear_fvg.empty:
            below = past_bear_fvg[past_bear_fvg["Top"] < c]
            if not below.empty:
                nearest = (c - below["Top"]).idxmin()
                df.at[df.index[i], "fvg_bear_below"] = 1
                df.at[df.index[i], "fvg_bear_dist"] = (c - below.at[nearest, "Top"]) / a

    # --- BOS / CHoCH features ---
    df["recent_bos"] = (
        bos_choch["BOS"].rolling(8, min_periods=1).apply(lambda x: int(x.notna().any()))
    )
    df["recent_choch"] = (
        bos_choch["CHOCH"].rolling(8, min_periods=1).apply(lambda x: int(x.notna().any()))
    )
    df["structure_bias"] = df["recent_bos"].astype(int) - df["recent_choch"].astype(int)

    # --- Liquidity features ---
    liq_levels = liq[liq["Liquidity"].notna()]["Level"]

    df["liq_above_dist"] = float("nan")
    df["liq_below_dist"] = float("nan")

    for i in range(len(df)):
        c = close.iloc[i]
        a = atr.iloc[i]
        if pd.isna(a) or a == 0:
            continue
        past_liq = liq_levels.iloc[:i]
        if past_liq.empty:
            continue
        above = past_liq[past_liq > c]
        below = past_liq[past_liq < c]
        if not above.empty:
            df.at[df.index[i], "liq_above_dist"] = (above.min() - c) / a
        if not below.empty:
            df.at[df.index[i], "liq_below_dist"] = (c - below.max()) / a

    df[["ob_bullish_dist", "ob_bearish_dist", "ob_volume_ratio",
        "fvg_bull_dist", "fvg_bear_dist",
        "liq_above_dist", "liq_below_dist"]] = (
        df[["ob_bullish_dist", "ob_bearish_dist", "ob_volume_ratio",
            "fvg_bull_dist", "fvg_bear_dist",
            "liq_above_dist", "liq_below_dist"]]
        .clip(-10, 10)
        .fillna(0.0)
    )

    return df

​

Feature names to add to model_compat.json

"ob_bullish_present", "ob_bearish_present",
"ob_bullish_dist", "ob_bearish_dist", "ob_volume_ratio",
"fvg_bull_above", "fvg_bear_below",
"fvg_bull_dist", "fvg_bear_dist",
"recent_bos", "recent_choch", "structure_bias",
"liq_above_dist", "liq_below_dist"

​

Integration tasks

• Add smartmoneyconcepts to requirements.txt
• Implement add_smc_features(df) in ml/feature_engineering.py — no MitigatedIndex / BrokenIndex columns may appear in the feature matrix
• Add the 14 feature names to model_compat.json["features"] (append at end)
• Run python scripts/retrain.py --ensemble --no-warmstart with all 14 features
• Run ml/shap_analysis.py — inspect beeswarm plot; drop any feature with mean absolute SHAP < 0.001 after retrain
• If more than 4 features are pruned by SHAP, split into two groups (OB+FVG first, BOS+liquidity second) and add one group at a time
• OOS gate: SMC ensemble Score must be ≥ current Score + 0.5 to ship; if not, revert model_compat.json and remove the features
• Performance note: validate that add_smc_features completes in < 60s on the full training dataset; if slower, vectorise before merging to main

​

Expected impact

• OB / FVG features — highest potential. Price returning to an OB or being drawn toward an FVG is a well-documented BTC pattern. Should show non-zero SHAP especially on ob_bullish_dist and fvg_bull_above.
• BOS / CHoCH features — structural regime context. structure_bias directly encodes trend vs reversal, complementing existing h4_ema_bias and d1_trend.
• Liquidity features — encodes stop-hunt dynamics. Less certain; BTC liquidity grabs are real but noisier at M15 than on higher timeframes.

​

Phase 18 — Model architecture

​

18.1 Stacked meta-learner

from sklearn.model_selection import TimeSeriesSplit  # NOT StratifiedKFold

def generate_oof_stacks(models: list, X: np.ndarray, y: np.ndarray,
                        n_splits: int = 5) -> tuple[np.ndarray, np.ndarray]:
    """
    Returns X_meta (N_oof, K*3), y_meta (N_oof,).
    Uses walk-forward folds: each fold trains on [0..t], predicts [t..t+step].
    First fold's training samples are discarded (no OOF predictions for them).
    """
    tscv = TimeSeriesSplit(n_splits=n_splits)
    K = len(models)         # number of base models (3 currently, 5+ in Phase 22)
    X_meta = np.zeros((len(X), K * 3))
    mask = np.zeros(len(X), dtype=bool)

    for train_idx, val_idx in tscv.split(X):
        for k, model in enumerate(models):
            m_clone = clone(model).fit(X[train_idx], y[train_idx])
            probs = m_clone.predict_proba(X[val_idx])  # (n_val, 3)
            X_meta[val_idx, k*3:(k+1)*3] = probs
        mask[val_idx] = True

    return X_meta[mask], y[mask]   # drop rows with no OOF prediction (first fold train set)

from sklearn.isotonic import IsotonicRegression

# After training meta-learner, calibrate on a held-out fold (never on test set)
meta_probs_cal = meta_model.predict_proba(X_cal_meta)  # cal = held-out fold
calibrators = []
for c in range(3):
    ir = IsotonicRegression(out_of_bounds="clip")
    ir.fit(meta_probs_cal[:, c], (y_cal == c).astype(float))
    calibrators.append(ir)

def calibrate(raw_probs: np.ndarray) -> np.ndarray:
    cal = np.column_stack([calibrators[c].transform(raw_probs[:, c]) for c in range(3)])
    cal /= cal.sum(axis=1, keepdims=True)  # renormalize rows
    return cal

• Implement generate_oof_stacks in ml/meta_learner.py using TimeSeriesSplit(n_splits=5) — not StratifiedKFold
• Start with LogisticRegression(C=0.1) meta-learner (Option A); switch to LGB after Phase 22 adds neural models
• Add --meta flag to scripts/retrain.py that triggers OOF generation + meta-learner training after base model training
• Apply isotonic calibration on a chronological held-out fold (last 10% of training data); save calibrators to models/meta_calibrators.pkl
• OOS gate — recalibrated targets (current best is WR=67.6%, PF=2.83, Score=35.13):
  • Keep if OOS Score ≥ current Score + 1.0
  • Keep if OOS WR ≥ 60% (not 75% — that was unreachable; WR > 75% would indicate overfit, not improvement)
  • Keep if OOS MaxDD ≤ current MaxDD + 2%
  • If gate fails: revert to majority-vote, log result to logs/meta_learner_eval.log, do not re-attempt until base models are retrained

​

18.2 Calibrated probability outputs

• calibrate_models() added to ml/ensemble_trainer.py — applies _IsotonicCalibratedClassifier (sklearn-version-agnostic wrapper) to each base model after training; saves *_calibrated.pkl alongside raw pkl
• _load_model() in ml/ensemble_predictor.py updated: calibrated pkl takes priority over ONNX (calibration cannot be applied to ONNX sessions), ONNX is second, raw pkl is fallback
• Verified calibration on val set via ml/calibration_check.py: all three models PASS (RF MAE=0.012, XGB MAE=0.018, LGB MAE=0.014, ensemble MAE=0.030 — well within 0.05 threshold)
• Completed before enabling ml_sltp (Phase 21.1) — calibrated probabilities ready

​

18.3 Regime-switching model

# Run this before deciding to build regime-switching models
import pandas as pd
from backtest import run_backtest

df_oos = pd.read_csv("datasets/oos_signals.csv")  # generated by backtest.py

# Split by regime
trending = df_oos[df_oos["adx_14"] > 0.25]
ranging  = df_oos[df_oos["adx_14"] <= 0.25]

# Check performance gap: if both regimes have WR > 55%, single model is fine
for name, subset in [("trending", trending), ("ranging", ranging)]:
    wr = (subset["pnl"] > 0).mean()
    pf = subset[subset["pnl"]>0]["pnl"].sum() / abs(subset[subset["pnl"]<0]["pnl"].sum())
    print(f"{name}: n={len(subset)}, WR={wr:.1%}, PF={pf:.2f}")

# Uses 3 consecutive bars of new regime before switching model
TRENDING_THRESHOLD = 0.28   # Enter trending above this
RANGING_THRESHOLD  = 0.22   # Enter ranging below this (gap = hysteresis band)

def get_active_model(adx_history: deque, current_model: str) -> str:
    recent = list(adx_history)[-3:]   # last 3 bars
    if all(a > TRENDING_THRESHOLD for a in recent):  return "trending"
    if all(a < RANGING_THRESHOLD  for a in recent):  return "ranging"
    return current_model   # stay in current regime — hysteresis

• Trending-regime Score ≥ 17.0 (better than full-model 35.13 / 2 = 17.6 is the theoretical minimum for half the trades; 17.0 is conservative given trending is the easier regime)
• Ranging-regime Score ≥ 10.0 (ranging is harder; acceptable if at least net-positive)
• Combined (blended) Score ≥ current 35.13 + 2.0 (must beat single model or not worth the complexity)

• Run regime diagnostic above on the latest OOS dataset before doing anything else; log results to logs/regime_diagnostic.txt
• If diagnostic shows no regime gap (both WR > 55%): skip this phase, mark as deferred
• If gap found: segment datasets/training_data_btcusd.csv by adx_14; verify ≥ 20k rows per segment
• Train two ensembles: python scripts/retrain.py --ensemble --regime trending and --regime ranging; each saves to models/signal_trending/ and models/signal_ranging/
• Add RegimeSwitchPredictor to ml/ensemble_predictor.py: maintains adx_history = deque(maxlen=3); calls get_active_model() before each inference; loads both model sets into memory at startup
• OOS backtest: run backtest.py --regime-switch which routes each bar to the correct model; compare blended Score to single-model Score
• Gate as above; if blended Score < current + 2.0: abandon regime switching and document

​

18.4 Multi-instrument expansion

models/
  BTCUSD/              # existing
    signal_rf.onnx
    signal_xgb.onnx
    signal_lgb.onnx
    position_rf.onnx
    ...
    ensemble_scaler.pkl
    model_compat.json
  XAUUSD/              # new
    signal_rf.onnx
    ...
    ensemble_scaler.pkl   # SEPARATE scaler — gold ATR is 100× smaller than BTC
    model_compat.json     # may differ in feature list (e.g. no dist_to_round_number for gold)

datasets/
  training_data_btcusd.csv
  training_data_xauusd.csv   # new — fetched from MT5 XAUUSD M15
  training_data_eurusd.csv   # new

ml_config.json       # BTCUSD defaults
ml_config_xauusd.json  # gold-specific overrides
ml_config_eurusd.json  # forex-specific overrides

// ml_config_xauusd.json
{
  "labeling": {
    "atr_sl_multiplier": 0.8,
    "atr_tp_multiplier": 1.2,
    "min_atr": 0.5,         // gold: $0.50 minimum ATR (not $15 like BTC)
    "lookahead_candles": 48  // same
  },
  "training": {
    "min_samples": 50000     // gold has fewer liquid M15 bars; lower threshold
  }
}

# backtest.py — pip_value must be symbol-aware
PIP_VALUE = {
    "BTCUSD": 100.0,   # USC/lot/point for cent account
    "XAUUSD": 100.0,   # $1/lot/point × 100 for cent
    "EURUSD": 10.0,    # $10/pip/lot standard; 0.1/pip/lot for micro
}

• Add --symbol flag to ml/train.py; when set, load ml_config_{symbol.lower()}.json instead of default; save models to models/{SYMBOL}/
• Add --symbol flag to backtest.py; load correct scaler and model directory; use symbol-specific pip_value in P&L calculation
• Create ml_config_xauusd.json with XAUUSD-specific min_atr, labeling params; keep all other params as BTC defaults initially
• Fetch 2 years of XAUUSD M15 from MT5 API: python ml/train.py --symbol XAUUSD --refresh — saves to datasets/training_data_xauusd.csv
• Train XAUUSD model stack: python scripts/retrain.py --symbol XAUUSD --ensemble --position --no-warmstart
• OOS gate for XAUUSD: PF > 2.0 on 60-day OOS window; MaxDD < 15%
• EURUSD: defer until XAUUSD is live-validated; same pipeline applies

​

Phase 19 — Infrastructure & reliability

​

19.1 Live trade dashboard

• Build scripts/dashboard_live.py:
  import streamlit as st
  from supabase import create_client
  import pandas as pd
  
  st.set_page_config(page_title="NOVOSKY Live", layout="wide", page_icon="📈")
  sb = create_client(os.getenv("SUPABASE_URL"), os.getenv("SUPABASE_KEY"))
  
  @st.cache_data(ttl=30)
  def get_equity():
      r = sb.table("account_snapshots").select("equity,created_at").order("created_at", desc=True).limit(500).execute()
      return pd.DataFrame(r.data)
  
  col1, col2 = st.columns([2, 1])
  with col1:
      st.line_chart(get_equity().set_index("created_at")["equity"])
  with col2:
      st.metric("Latest Equity", f"${get_equity().iloc[0]['equity']:.0f}")
  
• Views: equity area chart (500 snapshots), open position P&L, last 20 signals table with confidence color-coding
• Deploy via PM2 port 8501: streamlit run scripts/dashboard_live.py --server.port 8501 --server.headless true; expose under terminal-rf1.novosky.app/monitor via Caddy

​

19.2 API failover

• _api_fail_count and _api_paused globals track consecutive MT5 API failures in trading/bot.py
• After 3 consecutive failures: Telegram alert [API UNREACHABLE] fired, _api_paused = True blocks new entries
• Resumes automatically when API responds; _api_fail_count cleared, _api_paused = False
• Guard wraps the _get_rates() call in the main loop

​

19.3 Graceful shutdown improvements

• _shutdown_requested flag replaces sys.exit(0) in the SIGTERM handler
• Main loop top: if flag is set, polls open positions and exits cleanly once count reaches 0
• Entry gate blocks new trades while shutdown is pending
• Logs [SHUTDOWN] with position count on clean exit

​

19.4 Config hot-reload

# trading/bot.py — add to main loop top
_config_mtime: float = 0.0
_SAFE_HOT_KEYS = {"risk_percent","max_daily_loss_pct","min_confidence","adx_regime_filter","circuit_breaker"}

def _maybe_reload_config() -> None:
    global _config_mtime, config
    current_mtime = os.path.getmtime("config.json")
    if current_mtime <= _config_mtime:
        return
    new_cfg = json.loads(open("config.json").read())
    changed = {}
    for key in _SAFE_HOT_KEYS:
        if new_cfg.get(key) != config.get(key):
            changed[key] = {"old": config.get(key), "new": new_cfg.get(key)}
            config[key] = new_cfg[key]
    if changed:
        logger.info(f"[CONFIG RELOAD] {changed}")
        _notify_telegram(f"⚙️ Config reloaded: {list(changed.keys())}")
    _config_mtime = current_mtime

• Add _maybe_reload_config() call at top of main loop (before signal gate) — call every cycle (M15 cadence means 15-min max lag is acceptable; no need for a background thread)
• Define _SAFE_HOT_KEYS set as shown above; never iterate all config keys (would silently apply unsafe changes)
• Log changed keys with old/new values (not just key names) — makes audit trail useful
• Telegram notification on reload: send list of changed keys so operator knows the change took effect
• Unit test: write a temp config.json with modified risk_percent, call _maybe_reload_config(), assert config["risk_percent"] updated and _config_mtime advanced

​

Phase 22 — Advanced Ensemble Architecture (XGBoost · RF · FT-Transformer · TFT · LSTM)

Total Error = Bias² + Variance − 2 × Covariance(model_i, model_j)

pip install torch>=2.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install skl2onnx>=1.16 onnxconverter-common>=1.13
pip install pytorch-forecasting>=1.0  # TFT reference implementation
pip install mapie>=0.8               # Conformal prediction / quantile intervals
pip install xgboost>=2.0             # DART mode requires >=1.7; 2.x preferred

​

22.1 XGBoost — DART mode + monotonic constraints + Optuna search space

ŷ_i = Σ_{k ∉ D} f_k(x_i)           # prediction without dropped trees
new tree f_{t+1} fits residuals of ŷ_i
prediction after round: ŷ_i + f_{t+1}(x_i) scaled by 1/|D|+1

if c_j = +1:  for all splits on feature j, left_child_value ≤ right_child_value
if c_j = -1:  left_child_value ≥ right_child_value
if c_j =  0:  unconstrained

"interaction_constraints": [
  [0, 1, 2, 3, 4],          // Group 0: momentum (RSI, MACD, rsi_slope, etc.)
  [5, 6, 7, 8, 9],          // Group 1: volatility (ATR, BB, ADX)
  [10, 11, 12, 13, 14, 15], // Group 2: structure (EMA, price_vs_ema, d1_trend)
  [16, 17, 18, 19, 20]      // Group 3: session/time (flags, sin/cos encodings)
]

• Switch booster in ml_config.json → xgb_params:
  "booster": "dart",
  "rate_drop": 0.10,
  "skip_drop": 0.50,
  "normalize_type": "tree",
  "learning_rate": 0.05,
  "n_estimators": 400,
  "max_delta_step": 1
  
• In ml/train.py around the XGBClassifier constructor, build monotone_constraints tuple from model_compat.json["features"] order — map feature names to constraint values:
  MONOTONE_MAP = {
      "atr_14": 1, "adx_14": 1, "bb_width": 1,
      "volume_ratio": 1, "atr_percentile": 1,
  }  # all others default to 0
  constraints = tuple(MONOTONE_MAP.get(f, 0) for f in feature_names)
  
• Build interaction_constraints list from 4 feature cluster groups; attach to XGB params before fit
• Add DART-specific Optuna search space in ml/tune.py (new branch under if booster == "dart"):
  "rate_drop":  trial.suggest_float("rate_drop", 0.05, 0.30),
  "skip_drop":  trial.suggest_float("skip_drop", 0.30, 0.70),
  "normalize_type": trial.suggest_categorical("normalize_type", ["tree", "forest"]),
  "max_depth":  trial.suggest_int("max_depth", 4, 8),   # shallower than gbtree
  "subsample":  trial.suggest_float("subsample", 0.6, 0.9),
  "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 0.8),
  
• DART disables early_stopping_rounds — use fixed n_estimators=400; remove any early_stopping_rounds from the DART fit call in ml/train.py
• Run python scripts/retrain.py --ensemble --no-warmstart to force a clean DART retrain
• OOS gate: keep if Score ≥ current GBTREE baseline − 0.5 (accept slight score trade-off for lower variance)

​

22.2 Random Forest — ExtraTrees + Quantile intervals for Kelly

For each candidate feature j:
  threshold_j ~ Uniform(min(X[:,j]), max(X[:,j]))  # random, not optimal
Pick j* = argmin Gini over random (j, threshold_j) pairs

S(x) = ∪_{t=1}^{T} { y_i : x_i ∈ leaf_t(x) }   # all leaf-matched labels
P_q(x) = q-th percentile of S(x)

raw_kelly = (p_win − p_loss) / win_loss_ratio      # standard Kelly
interval_width = P_95(x) − P_05(x)                 # normalized 0–1
adjusted_kelly = raw_kelly × (1 − interval_width)  # tighter interval → larger position
effective_risk  = base_risk_pct × min(adjusted_kelly, max_kelly_fraction)

• Add ExtraTreesClassifier to ml/ensemble_trainer.py — insert after the RF definition:
  from sklearn.ensemble import ExtraTreesClassifier
  extra_trees = ExtraTreesClassifier(
      n_estimators=350,
      max_depth=None,          # full depth; randomness controls variance
      min_samples_split=8,
      max_features='sqrt',
      bootstrap=False,         # ExtraTrees convention
      class_weight='balanced',
      random_state=42,
      n_jobs=-1,
  )
  
• Export ExtraTrees to ONNX using the same skl2onnx pipeline in ml/onnx_export.py; output shape [1, 3] float32 probabilities
• Add "extra_trees" key to model_compat.json["models"] list; update ensemble_predictor.py load path
• Create ml/quantile_predictor.py:
  • Class QuantileRFPredictor wraps a trained RandomForestClassifier
  • Method predict_interval(X_row) → iterates all tree leaves matched by X_row, pools their training labels, returns (p05, p50, p95) for each class
  • Inference is O(T × leaf_size) — keep T ≤ 200 for <50 ms latency at M15 frequency
• Wire into trading/bot.py Kelly sizing block (currently around line 1220): fetch interval_width from QuantileRFPredictor; apply adjusted Kelly formula above; log both raw and adjusted Kelly to ml_performance.csv
• Add "quantile_rf": {"enabled": false, "p_low": 0.05, "p_high": 0.95} config block to config.json
• Optuna search space additions in ml/tune.py for ExtraTrees:
  "et_n_estimators": trial.suggest_int("et_n_estimators", 200, 500),
  "et_min_samples_split": trial.suggest_int("et_min_samples_split", 4, 16),
  "et_max_features": trial.suggest_categorical("et_max_features", ["sqrt", "log2", 0.5]),
  
• OOS sweep: compare 3-model majority-vote vs 4-model (RF+XGB+LGB+ExtraTrees) majority-vote; require Score ≥ current + 0.3

​

22.3 FT-Transformer (Feature Tokenizer + Transformer)

token_j = W_j × x_j + b_j + e_j      # W_j ∈ ℝ^d, b_j ∈ ℝ^d, e_j = feature-index embedding

Attention(Q, K, V) = softmax( QK^T / √d_k ) V
Q = K = V = token_sequence × W_{q,k,v}

Input dtype:  float32, shape (batch, 59) — same StandardScaler as RF/XGB
Output dtype: float32, shape (batch, 3) — raw logits (apply softmax at inference)
Training set: same X_train, y_train from ml/train.py
Min samples for stable attention: ~50k (you have ~135k ✓)

• Create ml/models/ft_transformer.py — pure torch.nn.Module:
  class FTTransformer(nn.Module):
      def __init__(self, n_features=59, d_model=64, n_heads=8, n_layers=6,
                   ffn_dim=256, dropout=0.1):
          # Feature Tokenizer: one Linear(1 → d_model) per feature
          self.tokenizers = nn.ModuleList([nn.Linear(1, d_model) for _ in range(n_features)])
          # Feature-index positional embedding (not time-positional)
          self.feat_index_emb = nn.Embedding(n_features, d_model)
          # CLS token
          self.cls_token = nn.Parameter(torch.randn(1, 1, d_model))
          # Transformer encoder
          encoder_layer = nn.TransformerEncoderLayer(
              d_model, n_heads, ffn_dim, dropout, batch_first=True, norm_first=True
          )
          self.transformer = nn.TransformerEncoder(encoder_layer, n_layers)
          # Head: LayerNorm → Linear → 3 classes
          self.head = nn.Sequential(nn.LayerNorm(d_model), nn.Linear(d_model, 3))
  
  Forward pass: tokenize each feature → add index embeddings → prepend CLS → transformer → CLS output → head
• Create ml/trainers/ft_transformer_trainer.py:
  • Convert X_train (numpy float64) to torch.float32 tensor
  • class_weights = compute_sample_weight('balanced', y_train) → torch.FloatTensor
  • loss = F.cross_entropy(logits, y_batch, weight=class_weights_batch)
  • Optimizer: AdamW(lr=5e-4, weight_decay=1e-5, betas=(0.9, 0.999))
  • Scheduler: CosineAnnealingLR(T_max=150, eta_min=1e-6) — cosine decay ensures smooth convergence
  • Batch size: 256; max epochs: 150; early stop patience: 15 on val cross-entropy
  • Save best checkpoint to models/ft_transformer.pt (state_dict only, not full model)
• Add Optuna hyperparameter search for FT-Transformer in ml/tune.py:
  "ft_d_model":  trial.suggest_categorical("ft_d_model", [32, 64, 128]),
  "ft_n_heads":  trial.suggest_categorical("ft_n_heads", [4, 8]),
  "ft_n_layers": trial.suggest_int("ft_n_layers", 3, 8),
  "ft_ffn_dim":  trial.suggest_categorical("ft_ffn_dim", [128, 256, 512]),
  "ft_dropout":  trial.suggest_float("ft_dropout", 0.05, 0.30),
  "ft_lr":       trial.suggest_float("ft_lr", 1e-4, 1e-3, log=True),
  "ft_wd":       trial.suggest_float("ft_wd", 1e-6, 1e-3, log=True),
  
• ONNX export — add to ml/onnx_export.py:
  dummy = torch.randn(1, 59)
  torch.onnx.export(
      model, dummy, "models/ft_transformer.onnx",
      input_names=["features"], output_names=["logits"],
      dynamic_axes={"features": {0: "batch"}, "logits": {0: "batch"}},
      opset_version=17,
  )
  
  Edge case: if n_heads does not divide d_model evenly, ONNX export fails — validate d_model % n_heads == 0 before export.
• Apply isotonic calibration via existing calibrate_models() in ml/ensemble_trainer.py — wrap FT-Transformer inference in a sklearn-compatible predict_proba(X) adapter class
• Add to ml/ensemble_predictor.py model loading: check models/ft_transformer.onnx → fall back to models/ft_transformer_calibrated.pkl → fall back to models/ft_transformer.pt
• OOS gate: FT-Transformer solo OOS WR ≥ 50%, solo PF ≥ 1.5; ensemble with FT-Transformer Score ≥ baseline + 0.5

​

22.4 Temporal Fusion Transformer (TFT) — sequence model

Static features (5) ──→ Static Covariate Encoder (GRN)
                              │
                              ├──→ context_h (init LSTM hidden state)
                              └──→ context_e (enrichment context)

Dynamic features (48 × 54) ──→ Variable Selection Network (VSN)
    │  VSN uses a GRN per feature + softmax over features → weighted sum
    │  Output: (batch, 48, d_model)  — only informative features survive
    │
    ├──→ LSTM Encoder (2-layer, hidden=128)
    │        Outputs: (batch, 48, 128) encoder states
    │
    └──→ LSTM Decoder (2-layer, hidden=128) — initialized from static context_h
             Outputs: (batch, 48, 128) decoder states
             │
             ├──→ Gated Residual Network (GRN) with static context_e
             │
             └──→ Multi-Head Attention (num_heads=4, causal mask off for classification)
                      │
                      └──→ GRN → Layer Norm → (batch, 128)
                                       │
                                       └──→ Linear(128, 3) → BUY/SELL/HOLD logits

GRN(x, c=None):
  η₂ = ELU( W₂ × [x; c] + b₂ )      # c = optional context vector
  η₁ = W₁ × η₂ + b₁
  gate = sigmoid( W_gate × [x; c] + b_gate )
  output = LayerNorm( gate ⊙ η₁ + (1−gate) ⊙ x )  # gated skip connection

VSN for timestep t with features x^(j)_t, j=1..54:
  ξ^(j)_t = GRN_j( x^(j)_t, static_context )   # per-feature processing
  v_t = softmax( W_vs × [ξ^(1)_t; ...; ξ^(54)_t] + b_vs )  # feature weights
  x̃_t = Σ_j v^(j)_t × ξ^(j)_t                              # weighted combination

# ml/data/sequence_dataset.py
class SequenceDataset(Dataset):
    def __init__(self, X: np.ndarray, y: np.ndarray, seq_len: int = 48,
                 static_indices: list[int] = None):
        # X: (N, 59) float32, y: (N,) int64
        # static_indices: positions of the 5 static features in X
        self.X = torch.from_numpy(X.astype(np.float32))
        self.y = torch.from_numpy(y.astype(np.int64))
        self.seq_len = seq_len
        self.static_idx = static_indices or [16, 17, 18, 19, 20]  # session/time features
        self.dyn_idx = [i for i in range(X.shape[1]) if i not in self.static_idx]

    def __len__(self):
        return len(self.X) - self.seq_len   # valid start indices

    def __getitem__(self, idx):
        seq = self.X[idx : idx + self.seq_len]           # (48, 59)
        x_dyn = seq[:, self.dyn_idx]                     # (48, 54)
        x_static = self.X[idx + self.seq_len - 1, self.static_idx]  # (5,) — current bar
        label = self.y[idx + self.seq_len - 1]           # label at bar t
        return x_dyn, x_static, label

optimizer = AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = ReduceLROnPlateau(optimizer, patience=10, factor=0.5, min_lr=1e-5)
# Gradient clipping — mandatory for LSTM in TFT
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# Batch: 64 (sequence data needs smaller batches for gradient stability)
# Loss: CrossEntropyLoss with class weights

• Create ml/models/tft.py — implement GRN, VSN, TFT classes as described above; keep d_model=128, n_heads=4, seq_len=48, n_static=5, n_dynamic=54
• Create ml/data/sequence_dataset.py — SequenceDataset as specified; unit test: assert no label from future bars is in the window (check __getitem__ with known data)
• Create ml/trainers/tft_trainer.py:
  • Build DataLoader(SequenceDataset(...), batch_size=64, shuffle=False) — do NOT shuffle (time-series ordering)
  • Training loop: forward → loss → backward → clip_grad → step → scheduler.step(val_loss)
  • Save best model (val loss) to models/tft.pt; also save final-epoch attention weights tensor to models/tft_attention_cache.npy (shape (n_val, 48, 54)) for dashboard initialization
• Add TFT Optuna search space in ml/tune.py:
  "tft_d_model":    trial.suggest_categorical("tft_d_model", [64, 128]),
  "tft_n_heads":    trial.suggest_categorical("tft_n_heads", [4, 8]),
  "tft_n_grn":      trial.suggest_int("tft_n_grn", 1, 3),   # GRN depth
  "tft_dropout":    trial.suggest_float("tft_dropout", 0.05, 0.25),
  "tft_seq_len":    trial.suggest_categorical("tft_seq_len", [24, 48, 96]),
  "tft_lr":         trial.suggest_float("tft_lr", 5e-4, 5e-3, log=True),
  
• ONNX export: TFT has two inputs — export with input_names=["x_dynamic", "x_static"], shapes [batch, 48, 54] and [batch, 5]; opset 17; verify with onnxruntime.InferenceSession
• In ml/ensemble_predictor.py: maintain a rolling buffer _seq_buffer: deque(maxlen=48) populated after each build_features() call; pass last 48 rows to TFT at inference
• Edge case: if _seq_buffer has < 48 entries (bot just started), zero-pad the head and still run inference — TFT will output lower-confidence results until buffer fills
• OOS gate: TFT solo WR ≥ 50%; ensemble Score ≥ current + 0.5

​

22.5 LSTM with Bahdanau attention + TCN alternative

e_t = v^T × tanh( W_a × h_t + U_a × h_T )    # alignment score at step t
α_t = exp(e_t) / Σ_{t'} exp(e_{t'})           # softmax over T steps
c   = Σ_t α_t × h_t                            # context vector (weighted sum)
ŷ   = W_out × [c ; h_T] + b_out               # concat context + final hidden

Receptive field = 1 + Σ_{i=0}^{n-1} (k−1) × 2^i = 1 + (k−1) × (2^n − 1)
For k=5, n=4: RF = 1 + 4 × 15 = 61 bars  (covers 61 M15 bars = ~15h)
For k=5, n=5: RF = 1 + 4 × 31 = 125 bars (covers 125 M15 bars = ~31h)

Input: (batch, seq_len=48, n_features=59)
Transpose: (batch, 59, 48)  — Conv1d expects (batch, channels, length)

Layer 0: Conv1d(59→128, k=5, dilation=1, padding=4, causal)  → (batch, 128, 48)
Layer 1: Conv1d(128→128, k=5, dilation=2, padding=8, causal) → (batch, 128, 48)
Layer 2: Conv1d(128→128, k=5, dilation=4, padding=16, causal)→ (batch, 128, 48)
Layer 3: Conv1d(128→128, k=5, dilation=8, padding=32, causal)→ (batch, 128, 48)

Each layer: Conv1d → WeightNorm → ReLU → Dropout(0.1) + residual projection if channels change

Take last time step: (batch, 128)
Linear(128, 64) → ReLU → Dropout(0.2) → Linear(64, 3)

def causal_conv1d(x, conv, dilation):
    padding = (conv.kernel_size[0] - 1) * dilation
    x = F.pad(x, (padding, 0))   # left-pad only
    return conv(x)

• Create ml/models/lstm_attention.py:
  • Bidirectional LSTM: nn.LSTM(input_size=59, hidden_size=128, num_layers=2, bidirectional=True, dropout=0.2, batch_first=True) — output dim = 256
  • Attention: implement Bahdanau equations above with da=64; W_a=Linear(256, 64), U_a=Linear(256, 64), v=Linear(64, 1, bias=False)
  • Context vector: c ∈ ℝ^{256}; concat with h_T[-1] (last step, bidirectional) → Linear(512, 3)
  • Inference mode switch: set self.training_mode flag; when False, run only forward direction of LSTM (unidirectional causal)
• Create ml/models/tcn.py:
  • 4 CausalConv1d layers with dilations [1, 2, 4, 8], kernel_size=5, out_channels=128
  • WeightNorm on each Conv1d (improves training stability vs BatchNorm on small batches)
  • Residual connections: if in_channels ≠ out_channels, add Conv1d(in, out, 1) projection
  • Final layer: last timestep output → Linear(128, 3)
• Train both on SequenceDataset(seq_len=48) using same tft_trainer.py loop (swap model); record OOS Score for each; keep whichever is higher (TCN likely within 0.5% but 4× faster)
• Optuna search space for LSTM:
  "lstm_hidden":    trial.suggest_categorical("lstm_hidden", [64, 128, 256]),
  "lstm_layers":    trial.suggest_int("lstm_layers", 1, 3),
  "lstm_dropout":   trial.suggest_float("lstm_dropout", 0.1, 0.4),
  "attn_dim":       trial.suggest_categorical("attn_dim", [32, 64, 128]),
  "lstm_lr":        trial.suggest_float("lstm_lr", 5e-4, 5e-3, log=True),
  
• Optuna search space for TCN:
  "tcn_channels":   trial.suggest_categorical("tcn_channels", [64, 128, 256]),
  "tcn_n_layers":   trial.suggest_int("tcn_n_layers", 3, 6),
  "tcn_kernel":     trial.suggest_categorical("tcn_kernel", [3, 5, 7]),
  "tcn_dropout":    trial.suggest_float("tcn_dropout", 0.05, 0.30),
  
• ONNX export — LSTM: use torch.onnx.export with opset 17; set do_constant_folding=True; verify hidden state output is not exported (classification output only); test inference latency with onnxruntime on a single row — must be < 20 ms on CPU
• At inference in ensemble_predictor.py: feed same _seq_buffer used by TFT; if buffer < 48, pad with zeros (same strategy as TFT)
• Attach alpha_weights (shape (48,)) to the return value of get_signal() for dashboard streaming

​

22.6 Stacked meta-learner + regime-adaptive weighting

Fold 1:  Train base models on months 1-6   → predict months 7-8   → collect OOF_1
Fold 2:  Train base models on months 1-8   → predict months 9-10  → collect OOF_2
Fold 3:  Train base models on months 1-10  → predict months 11-12 → collect OOF_3
...

• K = number of base models (5: RF, XGB, LGB, ExtraTrees, FT-Transformer/LSTM/TFT)
• C = number of classes (3: BUY, SELL, HOLD)
• N_train = training samples with OOF predictions (unavoidably loses first fold’s samples)

L_meta = CrossEntropy( LGB_meta(X_meta), y_true )

from sklearn.isotonic import IsotonicRegression
for c in range(3):
    ir_c = IsotonicRegression(out_of_bounds='clip')
    ir_c.fit(meta_probs_val[:, c], (y_val == c).astype(float))
    calibrated_probs[:, c] = ir_c.transform(meta_probs_test[:, c])
# Renormalize rows to sum to 1
calibrated_probs /= calibrated_probs.sum(axis=1, keepdims=True)

conf_threshold = 0.55
if max(meta_probs) >= conf_threshold:
    final_probs = meta_probs              # trust the meta-learner
else:
    w = regime_weights[current_regime]   # per-regime weight dict
    final_probs = Σ_k w_k × base_probs_k
    final_probs /= final_probs.sum()      # renormalize

def detect_regime(features: dict) -> str:
    adx = features["adx_14"]
    atr_pct = features["atr_percentile"]
    if adx > 0.25 and atr_pct < 0.75:  return "STRONG_TREND"
    if adx < 0.20 and atr_pct < 0.40:  return "RANGING"
    if atr_pct >= 0.75:                 return "VOLATILE"
    return "CHOPPY"

• Create ml/meta_learner.py with:
  • generate_oof_stacks(base_models, X, y, n_splits=5) → returns X_meta (N, 15), y_meta (N,) using TimeSeriesSplit
  • Algorithm: for each fold, retrain all 5 base models on train split, predict predict_proba on val split, store in X_meta[val_idx]
  • Save to models/oof_stacks.npy
  • train_meta_learner(X_meta, y_meta) → fits LGBMClassifier(n_estimators=100, max_depth=3, learning_rate=0.1, num_leaves=15, min_child_samples=20), saves to models/meta_learner.pkl
  • calibrate_meta(meta_model, X_cal, y_cal) → fits 3 IsotonicRegression objects, saves to models/meta_calibrators.pkl
• Create ml/regime_router.py:
  • detect_regime(features: dict) → str — 4-state classification using ADX + ATR percentile as above
  • REGIME_WEIGHTS dict with per-regime model weights (initial values: tune via scripts/sweep.py --target regime)
  • route(meta_probs, base_probs_dict, features) → np.ndarray — implements the fallback formula above
• Update ml/ensemble_predictor.py:
  • Load meta_learner.pkl and meta_calibrators.pkl in __init__ (alongside existing model loading)
  • In get_signal(): collect all base model predict_proba outputs → stack into X_meta_row (1, 15) → run meta_learner.predict_proba → calibrate → pass to RegimeRouter.route()
• Update scripts/weekly_optimize.py — add Phase 13b: after base model retraining, regenerate OOF stacks and retrain meta-learner (takes ~5 min extra; acceptable in weekly job)
• Optuna search space for meta-learner itself:
  "meta_n_est":       trial.suggest_int("meta_n_est", 50, 200),
  "meta_max_depth":   trial.suggest_int("meta_max_depth", 2, 5),
  "meta_lr":          trial.suggest_float("meta_lr", 0.05, 0.30),
  "meta_num_leaves":  trial.suggest_int("meta_num_leaves", 7, 31),
  "meta_conf_thresh": trial.suggest_float("meta_conf_thresh", 0.50, 0.65),
  
• OOS gate: 5-model meta-learner must beat current 3-model majority-vote by ≥ 1.0 Score AND ≥ 2% WR; if not, revert to majority-vote and document result in logs/meta_learner_eval.log

​

Phase 23 — JARVIS Live Trading Dashboard

npm install framer-motion@11        # animation engine
npm install lightweight-charts@4    # professional candlestick chart
npm install recharts@2              # bar/area charts
npm install echarts@5 echarts-for-react@3   # heatmap
npm install @react-three/fiber@8 @react-three/drei@9 three@0.165  # 3D surface (stretch)
npm install @supabase/supabase-js@2  # already installed
npm install zustand@4               # lightweight global state (signal stream store)

fastapi>=0.111
uvicorn[standard]>=0.29
websockets>=12.0
shap>=0.45          # live SHAP values per signal

​

23.1 WebSocket signal stream (backend)

interface SignalEvent {
  ts: string;              // ISO 8601 UTC timestamp
  prediction: "BUY" | "SELL" | "HOLD";
  confidence: number;      // max(probs) after calibration
  prob_diff: number;       // probs[0] - probs[1] (margin of victory)
  probs: [number, number, number];  // [BUY, SELL, HOLD]
  model_votes: {
    rf: "BUY" | "SELL" | "HOLD";
    xgb: "BUY" | "SELL" | "HOLD";
    lgb: "BUY" | "SELL" | "HOLD";
    ft_transformer?: "BUY" | "SELL" | "HOLD";  // optional until Phase 22.3 ships
    lstm?: "BUY" | "SELL" | "HOLD";
  };
  model_confidences: Record<string, number>;  // per-model max prob
  top_shap: Array<{ name: string; value: number }>;  // top 10, signed
  attention_weights?: number[];   // 48 floats from TFT/LSTM (nullable)
  regime: "STRONG_TREND" | "RANGING" | "VOLATILE" | "CHOPPY";
  adx: number;
  atr_percentile: number;
  equity: number;            // current account equity (raw USC)
  open_position: {
    ticket: number;
    direction: "BUY" | "SELL";
    entry: number;
    sl: number;
    tp: number;
    unrealized_pnl: number;
    bars_held: number;
  } | null;
  ohlcv: {  // current completed M15 bar
    time: number;  // Unix timestamp
    open: number; high: number; low: number; close: number; volume: number;
  };
}

• Create trading/ws_server.py — FastAPI app on port 8765:
  app = FastAPI()
  _signal_queue: asyncio.Queue[SignalEvent] = asyncio.Queue(maxsize=10)
  
  @app.websocket("/ws/signal")
  async def stream(ws: WebSocket, token: str = Query(...)):
      if token != os.getenv("DASHBOARD_WS_SECRET"):
          await ws.close(code=4001); return
      await ws.accept()
      try:
          while True:
              event = await _signal_queue.get()
              await ws.send_json(dataclasses.asdict(event))
      except WebSocketDisconnect:
          pass
  
• In trading/bot.py after _get_signal() returns (around line 3200): push SignalEvent to _signal_queue via asyncio.get_event_loop().call_soon_threadsafe(_signal_queue.put_nowait, event) — bot runs in a thread, ws_server in asyncio event loop
• Run ws_server.py via uvicorn in a background thread started at bot startup; add DASHBOARD_WS_SECRET to .env
• Add to ecosystem.config.js: second PM2 process ws_server using uvicorn trading.ws_server:app --port 8765
• Caddy config: add reverse_proxy /ws/signal localhost:8765 under terminal-rf1.novosky.app block
• SHAP computation: after each build_features() call, compute shap.TreeExplainer(lgb_model).shap_values(X_current)[signal_class] — takes ~20 ms on CPU; acceptable at M15 frequency; include top 10 by abs value in top_shap

​

23.2 Core dashboard layout (Next.js)

app/dashboard/
  page.tsx             # root page — imports all panels
  layout.tsx           # standalone dark layout (no site nav/footer)
  loading.tsx          # skeleton loader while WS connects
components/dashboard/
  ConfidenceMeter.tsx
  ModelVotingPanel.tsx
  ModelConfidenceBars.tsx
  RegimeIndicator.tsx
  FeatureImportance.tsx
  TradeFlowPipeline.tsx
  CandlestickChart.tsx
  AttentionHeatmap.tsx
  EquityPanel.tsx
  EquitySurface3D.tsx  # stretch
hooks/
  useSignalStream.ts   # WebSocket client + store
  useEquityHistory.ts  # Supabase historical equity query
stores/
  signalStore.ts       # Zustand store for latest signal state

export function useSignalStream(url: string) {
  const setSignal = useSignalStore(s => s.setSignal);
  const wsRef = useRef<WebSocket | null>(null);
  const reconnectDelay = useRef(1000);

  const connect = useCallback(() => {
    const ws = new WebSocket(`${url}?token=${process.env.NEXT_PUBLIC_WS_SECRET}`);
    ws.onmessage = (e) => {
      setSignal(JSON.parse(e.data));
      reconnectDelay.current = 1000;  // reset backoff on success
    };
    ws.onclose = () => {
      setTimeout(connect, reconnectDelay.current);
      reconnectDelay.current = Math.min(reconnectDelay.current * 2, 30_000);
    };
    wsRef.current = ws;
  }, [url, setSignal]);

  useEffect(() => { connect(); return () => wsRef.current?.close(); }, [connect]);
}

• Create app/dashboard/layout.tsx with className="min-h-screen bg-slate-950 text-slate-100 font-mono" — separate from main site layout; no nav bar
• app/dashboard/page.tsx: CSS Grid layout — grid-cols-[40%_60%] on desktop, single column on mobile; gap-4; all panels inside <Suspense> boundaries
• stores/signalStore.ts: Zustand store with fields signal: SignalEvent | null, history: SignalEvent[] (last 200), connected: boolean; setSignal appends to history and updates latest
• Throttle store updates: wrap setSignal with a 250 ms debounce (4 Hz max re-render rate)
• Connection status pill: top-right corner, 8px dot — animate-pulse green when connected; amber when reconnecting; static red when disconnected for > 10 s

​

23.3 Animated confidence meter

function polarToCartesian(cx, cy, r, angleDeg) {
  const rad = (angleDeg - 90) * Math.PI / 180;
  return { x: cx + r * Math.cos(rad), y: cy + r * Math.sin(rad) };
}

function arcPath(cx, cy, r, startDeg, endDeg) {
  const s = polarToCartesian(cx, cy, r, startDeg);
  const e = polarToCartesian(cx, cy, r, endDeg);
  const large = endDeg - startDeg > 180 ? 1 : 0;
  return `M ${s.x} ${s.y} A ${r} ${r} 0 ${large} 1 ${e.x} ${e.y}`;
}
// Usage: arc from -135° to (-135° + confidence × 270°) → covers 270° total sweep

• ConfidenceMeter.tsx: SVG-based radial arc; background arc (dark stroke) + foreground arc animated with motion.path and animate={{ pathLength: confidence }} (Framer Motion SVG animation); center text shows percentage
• Spring config: transition={{ type: "spring", stiffness: 120, damping: 20 }} on pathLength change — avoids linear snap, feels organic
• On new signal: trigger outer ring pulse using useAnimate:
  const [scope, animate] = useAnimate();
  useEffect(() => {
    if (signal) animate(scope.current, { scale: [1, 1.4, 1], opacity: [1, 0.3, 1] },
                        { duration: 0.6, repeat: 2 });
  }, [signal?.ts]);
  
• Color: derive from signal.prediction — emerald-400 (BUY), rose-400 (SELL), amber-400 (HOLD); use CSS variable for smooth color transition via motion.div animate={{ color }} with transition={{ duration: 0.3 }}
• ModelConfidenceBars.tsx: horizontal progress bars per model using motion.div; set the animate width to confidence * 100 percent as a string value, transition={{ duration: 0.25 }}

​

23.4 Model voting panel

• ModelVotingPanel.tsx: 5-card grid (grid-cols-5 gap-3); each card is a motion.div with layout prop (enables FLIP animation on reorder); background color set via animate={{ backgroundColor }} — Framer Motion handles color interpolation
• Scale pop on vote change: track prevVote in useRef; if vote !== prevVote, trigger animate={{ scale: [1, 1.15, 1] }, { duration: 0.2 }}
• Consensus glow: when all 5 models agree — animate={{ boxShadow: "0 0 24px #10b981" }} (emerald for BUY) with transition={{ repeat: Infinity, repeatType: "reverse", duration: 1.2 }}
• Split signal badge: if Object.values(votes).filter(v => v === prediction).length <= 2, render amber ⚠ Split badge using AnimatePresence for enter/exit animation (slide down from top)
• Majority fraction badge: "4 / 5 BUY" string derived from vote counts; update without animation (content only)

​

23.5 Market regime indicator

const regimeBuffer = useRef<string[]>([]);
const confirmedRegime = useSignalStore(s => s.signal?.regime);

useEffect(() => {
  regimeBuffer.current.push(confirmedRegime);
  if (regimeBuffer.current.length > 3) regimeBuffer.current.shift();
  const dominant = mode(regimeBuffer.current);  // most frequent in last 3
  if (dominant !== displayedRegime) setDisplayedRegime(dominant);
}, [confirmedRegime]);

• RegimeIndicator.tsx: outer AnimatePresence mode="wait" — exit old card (opacity 0, y -20) then enter new card (opacity 1, y 0); transition={{ duration: 0.4 }}
• Glow border: animate={{ boxShadow: glowColor }} — map regime to glow: STRONG_TREND→"0 0 30px #10b981", RANGING→"0 0 30px #3b82f6", VOLATILE→"0 0 30px #ef4444", CHOPPY→"0 0 30px #eab308"
• Stats row: ADX value, ATR percentile (as %), recent WR from last 20 signals in signalStore.history; formatted as ADX 0.31 · ATR p72 · WR 68%
• Mini sparkline: <AreaChart width={120} height={40} data={adxHistory}> — no axes, no labels, just the shape; <Area dataKey="adx" stroke="#94a3b8" fill="transparent" strokeWidth={1.5} />

​

23.6 Live feature importance bar chart

• FeatureImportance.tsx: <BarChart layout="vertical" width={380} height={300}> — horizontal bars; <Bar dataKey="value" animationDuration={300} animationEasing="ease-out"> with <Cell fill={v > 0 ? "#14b8a6" : "#f43f5e"} /> per bar
• Sort top 10 by Math.abs(shap) descending; truncate feature name to 18 chars; <Tooltip formatter={(v) => v.toFixed(4)} />
• On update: Recharts re-renders with animationDuration=300 automatically animates bar width changes — no extra work needed
• Tabs component (shadcn/ui <Tabs>): tab 1 = SHAP, tab 2 = Attention (disabled/grayed until Phase 22.4 ships); when tab 2 is unlocked, render AttentionHeatmap inline

​

23.7 Trade flow animation

IDLE ──[new signal fires]──→ SCAN ──[confidence > threshold]──→ SIGNAL
     ──[risk check pass]──→ RISK CHECK ──[lot calculated]──→ SIZE
     ──[order sent]──→ EXECUTE ──[order confirmed]──→ MONITOR
     ──[position closes]──→ IDLE

• TradeFlowPipeline.tsx: horizontal node list with SVG connecting lines; each node is a 40px circle + label below
• Active node animation: motion.div animate={{ rotate: 360 }} transition={{ repeat: Infinity, duration: 2, ease: "linear" }} on the outer ring; inner circle static
• Node-to-node hop: use useEffect on signal.open_position to advance state machine; 200 ms delay between hops via sequential setTimeout calls (not sleep — use Promise chain)
• Connecting line fills as each node activates: motion.line animate={{ pathLength: isComplete ? 1 : 0 }} transition={{ duration: 0.2 }}
• Close event: subscribe to Supabase trades table INSERT; on INSERT, determine close type from close_type column → pulse MONITOR node emerald (TP / ML_EXIT) or rose (SL_HIT) with 3× scale keyframe, then reset state machine to IDLE after 2 s

​

23.8 TradingView Lightweight Charts integration

• CandlestickChart.tsx: initialize chart in useEffect with cleanup; use useRef<IChartApi> for the chart instance to survive re-renders
  const chart = createChart(containerRef.current, {
    layout: { background: { color: '#020617' }, textColor: '#94a3b8' },
    grid: { vertLines: { color: '#1e293b' }, horzLines: { color: '#1e293b' } },
    rightPriceScale: { borderColor: '#334155' },
    timeScale: { borderColor: '#334155', timeVisible: true },
    width: containerRef.current.clientWidth,
    height: 420,
  });
  
• On mount: fetch last 200 M15 bars from GET /api/bars?symbol=BTCUSD&tf=M15&limit=200 (add this Next.js API route that proxies to the MT5 HTTP API)
• On each SignalEvent: append signal.ohlcv to candleSeries.update() — Lightweight Charts handles the scrolling automatically
• BUY/SELL markers: accumulate markers array from signalStore.history; call candleSeries.setMarkers(markers) on each update — Lightweight Charts re-renders only changed markers
• SL/TP lines: use chart.addLineSeries({ lineStyle: LineStyle.Dashed }) for SL (rose) and TP (emerald); update price on position change; series.setData([]) when no position open
• Confidence histogram panel: add chart.addHistogramSeries({ priceScaleId: 'confidence', height: 80 }) — maps signal.confidence to bar height; color by direction
• ResizeObserver: watch container width changes → chart.applyOptions({ width: newWidth }) for responsiveness

​

23.9 Neural attention heatmap

const option = {
  tooltip: { formatter: ({ data }) => `Bar ${data[0]}: ${data[1]} = ${data[2].toFixed(3)}` },
  xAxis: { type: 'category', data: barLabels,   // e.g., ["-48", "-47", ..., "0"]
            axisLabel: { interval: 7 } },
  yAxis: { type: 'category', data: featureNames.slice(0, 10) },
  visualMap: { min: 0, max: 1, calculable: true,
                inRange: { color: ['#1e3a5f', '#2563eb', '#ef4444'] } },  // blue→red
  series: [{ type: 'heatmap', data: flatData,
              itemStyle: { borderRadius: 2, borderWidth: 0.5, borderColor: '#0f172a' } }],
};

// On new signal, reset and re-animate each column with increasing delay
flatData.forEach((cell, i) => {
  const col = cell[0];
  setTimeout(() => {
    updateCell(col, cell[1], cell[2]);
  }, col * 8);  // 8 ms per column × 48 columns = 384 ms total stagger
});

• AttentionHeatmap.tsx: use echarts-for-react wrapper; pass option as prop; style={{ height: 280 }}; update option via useState triggered by signal.attention_weights
• Transform attention_weights: number[48] into flatData: for each of the top 10 features (by SHAP), duplicate the bar-level attention weight — weight is the same per bar regardless of feature (TFT VSN gives per-feature weights separately; show VSN weights on Y axis if available)
• Show placeholder <div className="...">Attention available after Phase 22.4</div> if signal.attention_weights is null

​

23.10 Equity curve + live P&L panel

const TICK_MS = 100;
const lotSize = openPosition?.lot ?? 0;
const direction = openPosition?.direction === 'BUY' ? 1 : -1;

useEffect(() => {
  if (!openPosition) return;
  const interval = setInterval(() => {
    // Estimate tick P&L from last known close price in latest signal
    const currentPrice = latestSignal?.ohlcv?.close ?? openPosition.entry;
    const priceDiff = (currentPrice - openPosition.entry) * direction;
    const pnl = priceDiff * lotSize * 100;  // 100 USC per lot per point (BTCUSD CFD)
    setLivePnl(pnl);
  }, TICK_MS);
  return () => clearInterval(interval);
}, [openPosition, latestSignal]);