Optimization pipeline

Overview

scripts/weekly_optimize.py runs every Sunday at 2 am UTC via cron. It takes approximately 2–3 hours on a 4-core VM.

The 13 phases

Pre-flight and baseline

Loads current models, runs a quick backtest on recent OOS data, and records the baseline score. This score is the bar that the new run must beat.

MT5 connectivity check and data refresh

Verifies the MT5 API is reachable, then fetches the latest BTCUSD M15 candles to extend the training dataset.

SHAP analysis

ml/shap_analysis.py computes SHAP feature importances across all three signal models. The output informs which features are contributing and flags anomalies.SHAP results do not automatically drop features. Protected session/news features are never dropped regardless of SHAP values.

Optuna hyperparameter tuning

ml/tune/hyperparams.py runs an Optuna study to find the best hyperparameters for the signal ensemble. The objective function is a custom metric combining OOS precision, recall, and drawdown.Typical search: 50–100 trials. Results are saved to results/optuna_study.pkl.

Retrain — Signal model

Trains RF, XGB, and LGB signal classifiers with the tuned hyperparameters. Runs calibrate_models() after training to produce calibrated pkl files.

Retrain — Position model

Trains the position model using ml/tune/position.py hyperparameters. Requires the signal model to be already saved.

Retrain — SL/TP model

Trains the LightGBM SL/TP regressors. Requires signal and position models to be saved.

Retrain — Risk model

Trains the LightGBM risk multiplier model. Requires SL/TP models because training runs a full backtest to generate account-state labels.

Config sweep (scoped to risk profile)

scripts/sweep.py sweeps a grid of config parameters: confidence_threshold, risk_percent, sl_multiplier, tp_multiplier, and session filters. The sweep is scoped to your active risk profile — it only tests parameter combinations within the profile’s bounds.The sweep runs on the first 70% of the OOS window only.

Apply best config

The parameter combination with the highest score is written to config.json.

Full OOS evaluation

A clean backtest runs on the full OOS window including the 30% holdout that the sweep never saw. This is the definitive score.

Commit or rollback

If new_score >= baseline × 1.02: push models to HF Hub, commit config.json, and send a Telegram success report.Otherwise: restore models and config from backup, send a Telegram failure report. The bot continues with the previous version.

Cleanup and cron reschedule

Cleans up temporary files and confirms the next Sunday cron run is scheduled.

Running the pipeline

# Full run with risk questionnaire (first time)
python scripts/weekly_optimize.py --balance 500

# Skip questionnaire — use stored profile
python scripts/weekly_optimize.py --profile balanced

# Skip retrain — sweep and OOS only (~45 min)
python scripts/weekly_optimize.py --skip-retrain

# Resume from a specific phase after a crash
python scripts/weekly_optimize.py --from-phase 5

# Dry run — shows what would run without executing
python scripts/weekly_optimize.py --dry-run

Manual retrain path

To retrain outside of the weekly pipeline:

# Minimal retrain — all 4 models in correct order
python scripts/retrain.py

# With SHAP analysis and data refresh
python scripts/retrain.py --shap --refresh

# With Optuna tuning first
python scripts/retrain.py --trials 50

# Validate and push
python backtest.py --balance 500 --no-swap --leverage 500 --spread 14.59 --oos-only --no-chart
python ml/hf_hub.py --push

Never train models out of order. The Risk model depends on a backtest that requires SLTP models. The Position model depends on signal model outputs. Always use: Signal → Position → SLTP → Risk.

Optuna tuning

Each model has its own Optuna study:

Study	File	Trials	Objective
Signal hyperparams	`ml/tune/hyperparams.py`	50–100	OOS F1 × (1 − max_drawdown)
Position hyperparams	`ml/tune/position.py`	30–50	OOS accuracy × exit precision
Config sweep	`scripts/sweep.py`	Grid search	Score formula

To run Optuna tuning independently:

python ml/tune/hyperparams.py          # signal model tuning
python ml/tune/position.py             # position model tuning
python scripts/sweep.py --profile 3   # config sweep for profile 3

Backtesting

backtest.py is a config-faithful bar-by-bar backtester in backtest/run.py. It replicates the live trading loop exactly — same feature engineering, same model inference, same risk sizing, same circuit breakers.

# Standard OOS validation
python backtest.py \
  --balance 500 \
  --no-swap \
  --leverage 500 \
  --spread 14.59 \
  --oos-only \
  --no-chart

# Full window (IS + OOS)
python backtest.py --balance 500 --no-swap --leverage 500 --spread 14.59

# With chart output
python backtest.py --balance 500 --no-swap --leverage 500 --spread 14.59 --oos-only

--oos-only runs only on the held-out OOS segment. Always prefer this for realistic validation. Generic lookback backtests can be contaminated by in-sample data.

Models

Feature engineering

Pipeline

Overview

The 13 phases

Running the pipeline

Manual retrain path

Optuna tuning

Backtesting

​Overview

​The 13 phases

​Running the pipeline

​Manual retrain path

​Optuna tuning

​Backtesting

Overview

The 13 phases

Running the pipeline

Manual retrain path

Optuna tuning

Backtesting