scripts/optimize_loop.py runs the core NOVOSKY optimization cycle. Each iteration: analyze features via SHAP, tune hyperparameters with Optuna, retrain all models, evaluate OOS, and keep or revert based on Score improvement. It is the engine that the novosky-optimizer Claude agent drives.
Developer/operator lane only. Regular users should run onboarding, select profile 1-5, pull approved model revisions from R2, and run trading.
Quick start
# 1 iteration (local GPU/CPU)
python scripts/optimize_loop.py --iterations 1 --trials 50 --drop-threshold 0.0
# 3 iterations
python scripts/optimize_loop.py --iterations 3 --trials 50 --drop-threshold 0.0
# SHAP + backtest only, no retrain
python scripts/optimize_loop.py --analyze-only
# Tune + retrain only, skip SHAP analysis
python scripts/optimize_loop.py --retrain-only
Always use --drop-threshold 0.0. The default (0.003) will drop features with low SHAP scores — but low training SHAP does not mean low live importance. Protected features like is_news_near and session flags are near-zero at training time by design.
CLI flags
| Flag | Default | Description |
|---|
--iterations N | 1 | Number of full optimize cycles to run |
--trials N | 50 | Optuna trials per model per iteration |
--drop-threshold F | 0.003 | SHAP threshold for feature dropping — use 0.0 to disable |
--improvement-threshold F | 0.02 | Minimum Score improvement (2%) to keep new models |
--local | false | Deprecated compatibility flag (loop is already local-only) |
--analyze-only | false | SHAP + backtest only, no tuning or retraining |
--retrain-only | false | Skip SHAP analysis, go straight to tune + retrain |
What one iteration does
Step 1 — SHAP analysis
python train_ml_model.py --shap-only
Reads models/shap_summary.json
Identifies features below --drop-threshold (diagnostic unless threshold > 0)
Step 2 — Baseline backtest
python backtest_config.py --balance 500 --no-swap --leverage 500
--spread 16.95 --oos-only --no-chart
Records current Score = WR x PF / sqrt(MaxDD)
Step 3 — Optuna tuning
Local: python ml/tune/hyperparams.py && python ml/tune/position.py
Step 4 — Retrain
Local: python train_ml_model.py --ensemble --position --risk --no-warmstart
Step 5 — OOS backtest
Same command as Step 2 — records new Score
Step 6 — Decision
new_score >= old_score * (1 + improvement_threshold) -> COMMIT
Otherwise -> REVERT (restore pkl snapshot + ml_config.json)
Training backend
The loop is local-only. It uses GPU automatically when available, and falls back to CPU.
Config authority and strict sync
optimize_loop.py treats root configs as canonical authority:
config.jsonml_config.json
At iteration start it validates canonical root configs.
Enable strict drift blocking:
NOVOSKY_STRICT_CONFIG_SYNC=1 python scripts/optimize_loop.py --iterations 1 --trials 50 --drop-threshold 0.0
python scripts/config_sync.py --check
Output files
| File | Contents |
|---|
models/optimize_log.json | Cumulative history of every iteration (decision, scores, params) |
models/optimize_best.json | Best Score ever achieved and the config that produced it |
models/_snapshot_<tag>/ | Pre-iteration model snapshot for rollback |
Reading the log
# Print last 3 iterations
python3 -c "
import json
log = json.load(open('models/optimize_log.json'))
for e in log[-3:]:
print(e.get('iteration'), e.get('decision'),
'score:', e.get('final_score','?'),
'->', e.get('improvement_pct','?'), '%')
"
Score formula
Score = WR x PF / sqrt(MaxDD)
| Score | Meaning |
|---|
| < 6.0 | Pause trading |
| 6–10 | Below target — trigger retrain |
| 10–15 | Production-grade |
| > 15 | Excellent |
When to use vs weekly_optimize.py
Use optimize_loop.py (via agent) for:
- On-demand runs triggered by performance degradation
- Multi-iteration deep dives (3–5 iterations)
- When the weekly cron isn’t enough
Use weekly_optimize.py for:
- The scheduled Sunday autonomous cron
- Zero-touch operation (includes sweep, push, commit, notify)
