Skip to main content
Developer/operator lane only. Regular users should run onboarding, select profile 1-5, pull approved model revisions from R2, and run trading.
This guide walks through the full optimization workflow from fastest to most expensive. Always start at the top — a config sweep often fixes a degraded Score without any retraining.

​
Before you start

Run the quick health check: 
python3 -c "
import json, os
mc = json.load(open('models/model_compat.json'))
ml = json.load(open('ml_config.json'))
feat_ok = mc['feature_count'] == len(ml['features'])
print(f'[{\"OK\" if feat_ok else \"FAIL\"}] Feature count: {len(ml[\"features\"])}')
model_files = ['ensemble_rf.pkl','ensemble_xgb.pkl','ensemble_lgb.pkl',
               'position_rf.pkl','position_xgb.pkl','position_lgb.pkl']
all_present = all(os.path.exists(f'models/{f}') for f in model_files)
print(f'[{\"OK\" if all_present else \"FAIL\"}] Model files present')
"
Then get your baseline Score: 
python backtest_config.py \
  --balance 500 --no-swap --leverage 500 \
  --spread 16.95 --oos-only --no-chart
Record: WR, PF, MaxDD, and Score = WR × PF / sqrt(MaxDD).

​
Step 1: Config sweep (no retrain, ~30 min)

The fastest way to improve Score. Tests ~68 parameter combinations without touching the models. 
# Sweep confidence threshold + prob_diff
python scripts/sweep.py --target signal --mode confidence

# Sweep TP/SL ATR multipliers
python scripts/sweep.py --target signal --mode sltp

# Sweep risk% + circuit breaker
python scripts/sweep.py --target signal --mode risk
Results are ranked by Score. Apply the best: 
# Example: set confidence to 0.60, prob_diff to 0.10
python3 -c "
import json
ml = json.load(open('ml_config.json'))
ml['prediction']['confidence_threshold'] = 0.60
ml['prediction']['min_probability_diff'] = 0.10
with open('ml_config.json', 'w') as f: json.dump(ml, f, indent=2)
print('Applied')
"

# Verify
python backtest_config.py \
  --balance 500 --no-swap --leverage 500 \
  --spread 16.95 --oos-only --no-chart
If Score improved: commit, push, go live. You’re done. If Score didn’t improve: proceed to Step 2.

​
Step 2: Position model sweep (no retrain, ~10 min)

Tune the ML active management thresholds — how aggressively the position model exits trades. 
python scripts/sweep.py --target pos --full
Apply best thresholds: 
python3 -c "
import json
ml = json.load(open('ml_config.json'))
pm = ml['position_model']['prediction']
pm['exit_threshold'] = 0.80
pm['min_prob_diff'] = 0.25
with open('ml_config.json', 'w') as f: json.dump(ml, f, indent=2)
print('Applied')
"

​
Step 3: Full optimization loop — 1 iteration (~2 hrs)

SHAP analysis + Optuna tuning + full retrain + OOS validation on local GPU/CPU.

​
Via Claude agent (recommended)

TASK="Fix tech debt first (TD-1, TD-4, TD-5). Then: python scripts/optimize_loop.py --iterations 1 --trials 50 --drop-threshold 0.0 --improvement-threshold 0.02. Do not remove/reorder features in ml_config.json. When done send a Telegram notification."
./scripts/run_agent.sh novosky-optimizer "$TASK"

​
Manually

# 1. SHAP analysis (diagnostic only)
python train_ml_model.py --shap-only

# 2. Tune signal model
python ml/tune/hyperparams.py

# 3. Tune position model
python ml/tune/position.py

# 4. Retrain all models
python train_ml_model.py --ensemble --position --no-warmstart

# 5. OOS backtest
python backtest_config.py \
  --balance 500 --no-swap --leverage 500 \
  --spread 16.95 --oos-only --no-chart
Compare the new Score against your baseline. If improved: push and go live. If not: try Step 4.

​
Step 4: Refresh training data and retrain

If the model keeps reverting, the training data may be stale. 
# Fetch fresh data and retrain
python scripts/retrain.py --refresh --local

# OOS backtest
python backtest_config.py \
  --balance 500 --no-swap --leverage 500 \
  --spread 16.95 --oos-only --no-chart
A --refresh retrain fetches the latest BTCUSD M15 candles from your MT5 API. This extends the training window and exposes the model to recent market regimes.

​
Step 5: Multi-iteration loop (heavy, ~6 hrs)

Three iterations on local GPU/CPU. Use this when a single iteration isn’t enough. 
TASK="Run the full NOVOSKY optimization pipeline. Phase 0: baseline OOS backtest. Phase 1: fix tech debt TD-1, TD-4, TD-5. Phase 2: python scripts/optimize_loop.py --iterations 3 --trials 50 --drop-threshold 0.0 --improvement-threshold 0.02. Keep the full ml_config.json feature list unchanged. Phase 3: final OOS backtest + python trading.py --dry. Phase 4: document results in strategy_params.json. Target WR>=60% PF>=2.5 MaxDD<=20% Score>=0.35. When done send a Telegram notification."
./scripts/run_agent.sh novosky-optimizer "$TASK"

​
After any improvement

# 1. Smoke test
python trading.py --dry

# 2. Push models to R2
python ml/r2_hub.py --push

# 3. Commit
git add config.json ml_config.json strategy_params.json
git commit -m "feat: Phase N -- WR=XX% PF=X.XX MaxDD=XX.X% Score=X.XX"

# 4. Go live
python trading.py

​
Decision framework

SituationAction
Score dropped less than 10%, last 2 weeksConfig sweep only
Score dropped more than 10%Config sweep + 1 optimization iteration
WR below 55% for 50+ tradesRefresh data + retrain
Model keeps reverting (3+ iters)Try --no-warmstart + fresh data
New feature ideaSee Three-File Rule
Score < 6.0Pause live trading, investigate

​
Score reference

Score = WR x PF / sqrt(MaxDD)
ScoreStatus
> 15Excellent
10–15Production-grade
6–10Below target — optimize
< 6Pause trading
Current production target: Score >= 10.0 (current: 21.34 on latest 37d OOS snapshot).