> ## Documentation Index
> Fetch the complete documentation index at: https://docs.novosky.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Optimization pipeline

> The weekly optimization pipeline is a 13-phase autonomous loop that retrains all four models, sweeps config parameters, validates on OOS data, and commits or rolls back — without human input.

## Overview

`scripts/weekly_optimize.py` runs every Sunday at 2 am UTC via cron. It takes approximately 2–3 hours on a 4-core VM.

```mermaid theme={null}
flowchart LR
    P0([Pre-flight\nbaseline]) --> P1([MT5 check\ndata refresh])
    P1 --> P2([SHAP\nanalysis])
    P2 --> P3([Optuna\ntuning])
    P3 --> P4([Retrain\nSignal · Position\nSLTP · Risk])
    P4 --> P5([Sweep\nscoped to\nrisk profile])
    P5 --> P6([Apply\nbest config])
    P6 --> P7([OOS eval\nfull window])
    P7 --> P8{Score ≥ baseline\n+ 2%?})
    P8 -->|Yes| P9([Push HF\ncommit\nnotify ✓])
    P8 -->|No| P10([Rollback\nnotify ✗])
```

***

## The 13 phases

<Steps>
  <Step title="Pre-flight and baseline">
    Loads current models, runs a quick backtest on recent OOS data, and records the baseline score. This score is the bar that the new run must beat.
  </Step>

  <Step title="MT5 connectivity check and data refresh">
    Verifies the MT5 API is reachable, then fetches the latest BTCUSD M15 candles to extend the training dataset.
  </Step>

  <Step title="SHAP analysis">
    `ml/shap_analysis.py` computes SHAP feature importances across all three signal models. The output informs which features are contributing and flags anomalies.

    SHAP results do **not** automatically drop features. Protected session/news features are never dropped regardless of SHAP values.
  </Step>

  <Step title="Optuna hyperparameter tuning">
    `ml/tune/hyperparams.py` runs an Optuna study to find the best hyperparameters for the signal ensemble. The objective function is a custom metric combining OOS precision, recall, and drawdown.

    Typical search: 50–100 trials. Results are saved to `results/optuna_study.pkl`.
  </Step>

  <Step title="Retrain — Signal model">
    Trains RF, XGB, and LGB signal classifiers with the tuned hyperparameters. Runs `calibrate_models()` after training to produce calibrated pkl files.
  </Step>

  <Step title="Retrain — Position model">
    Trains the position model using `ml/tune/position.py` hyperparameters. Requires the signal model to be already saved.
  </Step>

  <Step title="Retrain — SL/TP model">
    Trains the LightGBM SL/TP regressors. Requires signal and position models to be saved.
  </Step>

  <Step title="Retrain — Risk model">
    Trains the LightGBM risk multiplier model. Requires SL/TP models because training runs a full backtest to generate account-state labels.
  </Step>

  <Step title="Config sweep (scoped to risk profile)">
    `scripts/sweep.py` sweeps a grid of config parameters: `confidence_threshold`, `risk_percent`, `sl_multiplier`, `tp_multiplier`, and session filters. The sweep is scoped to your active risk profile — it only tests parameter combinations within the profile's bounds.

    The sweep runs on the **first 70% of the OOS window** only.
  </Step>

  <Step title="Apply best config">
    The parameter combination with the highest score is written to `config.json`.
  </Step>

  <Step title="Full OOS evaluation">
    A clean backtest runs on the **full OOS window** including the 30% holdout that the sweep never saw. This is the definitive score.
  </Step>

  <Step title="Commit or rollback">
    If `new_score >= baseline × 1.02`: push models to HF Hub, commit `config.json`, and send a Telegram success report.

    Otherwise: restore models and config from backup, send a Telegram failure report. The bot continues with the previous version.
  </Step>

  <Step title="Cleanup and cron reschedule">
    Cleans up temporary files and confirms the next Sunday cron run is scheduled.
  </Step>
</Steps>

***

## Running the pipeline

```bash theme={null}
# Full run with risk questionnaire (first time)
python scripts/weekly_optimize.py --balance 500

# Skip questionnaire — use stored profile
python scripts/weekly_optimize.py --profile balanced

# Skip retrain — sweep and OOS only (~45 min)
python scripts/weekly_optimize.py --skip-retrain

# Resume from a specific phase after a crash
python scripts/weekly_optimize.py --from-phase 5

# Dry run — shows what would run without executing
python scripts/weekly_optimize.py --dry-run
```

***

## Manual retrain path

To retrain outside of the weekly pipeline:

```bash theme={null}
# Minimal retrain — all 4 models in correct order
python scripts/retrain.py

# With SHAP analysis and data refresh
python scripts/retrain.py --shap --refresh

# With Optuna tuning first
python scripts/retrain.py --trials 50

# Validate and push
python backtest.py --balance 500 --no-swap --leverage 500 --spread 14.59 --oos-only --no-chart
python ml/hf_hub.py --push
```

<Warning>
  Never train models out of order. The Risk model depends on a backtest that requires SLTP models. The Position model depends on signal model outputs. Always use: Signal → Position → SLTP → Risk.
</Warning>

***

## Optuna tuning

Each model has its own Optuna study:

| Study                | File                     | Trials      | Objective                     |
| -------------------- | ------------------------ | ----------- | ----------------------------- |
| Signal hyperparams   | `ml/tune/hyperparams.py` | 50–100      | OOS F1 × (1 − max\_drawdown)  |
| Position hyperparams | `ml/tune/position.py`    | 30–50       | OOS accuracy × exit precision |
| Config sweep         | `scripts/sweep.py`       | Grid search | Score formula                 |

To run Optuna tuning independently:

```bash theme={null}
python ml/tune/hyperparams.py          # signal model tuning
python ml/tune/position.py             # position model tuning
python scripts/sweep.py --profile 3   # config sweep for profile 3
```

***

## Backtesting

`backtest.py` is a **config-faithful bar-by-bar backtester** in `backtest/run.py`. It replicates the live trading loop exactly — same feature engineering, same model inference, same risk sizing, same circuit breakers.

```bash theme={null}
# Standard OOS validation
python backtest.py \
  --balance 500 \
  --no-swap \
  --leverage 500 \
  --spread 14.59 \
  --oos-only \
  --no-chart

# Full window (IS + OOS)
python backtest.py --balance 500 --no-swap --leverage 500 --spread 14.59

# With chart output
python backtest.py --balance 500 --no-swap --leverage 500 --spread 14.59 --oos-only
```

`--oos-only` runs only on the held-out OOS segment. Always prefer this for realistic validation. Generic lookback backtests can be contaminated by in-sample data.