Release Process Runbook¶
Overview¶
This runbook defines the process for releasing strategies and code changes to production. All releases require audit approval and staged rollout.
Release Types¶
| Type | Description | Approval Required | Rollout |
|---|---|---|---|
| Strategy Release | New or modified trading strategy | Full audit | Staged |
| Model Update | ML model retrain/update | Model audit | Staged |
| Bug Fix | Non-trading code fix | Code review | Direct |
| Hotfix | Critical live issue | Expedited review | Immediate |
| Config Change | Parameter adjustment | Risk review | Direct |
Versioning¶
Semantic Versioning¶
MAJOR.MINOR.PATCH
MAJOR: Breaking changes, strategy logic change
MINOR: New features, parameter additions
PATCH: Bug fixes, documentation updates
Version Tracking¶
# version.yaml (root of repo)
version: 1.2.3
release_date: 2024-01-15
changelog:
- "1.2.3: Fixed timeout handling in webhook"
- "1.2.2: Added kill switch API endpoint"
- "1.2.1: Updated slippage model for crypto"
- "1.2.0: Added session breakout strategy"
- "1.1.0: Implemented paper trading layer"
- "1.0.0: Initial release with ICT QM"
Strategy Versioning¶
# strategies/pine_v6/ict_qm.pine header
// @version=6
// Strategy: ICT_QM
// Version: 2.1.0
// Audit: AUD-2024-01-15-001 (PASS)
// Released: 2024-01-20
CI/CD Pipeline¶
Pre-Release Checks¶
# ci/release_checks.yaml
name: Release Checks
on:
push:
tags:
- 'v*'
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Lint Python
run: ruff check src/ tests/
- name: Type check
run: mypy src/
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Run unit tests
run: pytest tests/ -v --cov=src
- name: Run anti-bias tests
run: pytest tests/test_bias.py -v
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Scan for secrets
run: detect-secrets scan --all-files
- name: Dependency audit
run: pip-audit
audit_check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Verify audit approval
run: python scripts/verify_audit.py --version=${{ github.ref_name }}
Audit Verification Script¶
# scripts/verify_audit.py
import sys
import yaml
def verify_audit(version, strategy=None):
"""Verify audit approval for release."""
# Load audit log
with open('docs/audits/audit_log.md') as f:
audit_log = f.read()
# Check for PASS verdict
if strategy:
required_audit = f"{strategy}.*PASS"
if not re.search(required_audit, audit_log):
print(f"ERROR: No PASS audit found for {strategy}")
sys.exit(1)
# For any release, check version is documented
if version not in audit_log:
print(f"WARNING: Version {version} not found in audit log")
print("Audit verification passed")
sys.exit(0)
if __name__ == "__main__":
version = sys.argv[1] if len(sys.argv) > 1 else "unknown"
verify_audit(version)
Release Workflow¶
Phase 1: Preparation¶
1. Create release branch
git checkout -b release/v1.2.3
2. Update version.yaml
version: 1.2.3
release_date: YYYY-MM-DD
3. Update CHANGELOG.md
## [1.2.3] - YYYY-MM-DD
### Changed
- Description of changes
4. Run full test suite
pytest tests/ -v
5. Verify audit status
python scripts/verify_audit.py --strategy=ICT_QM
Phase 2: Review & Approval¶
1. Create pull request
- Title: "Release v1.2.3"
- Description: Changelog + audit reference
2. Required approvals:
- [ ] Code review (any team member)
- [ ] Audit approval (for strategy changes)
- [ ] Risk review (for config changes)
- [ ] Founder approval (for production)
3. CI checks must pass:
- [ ] Lint
- [ ] Tests
- [ ] Security scan
- [ ] Audit verification
Phase 3: Merge & Tag¶
# Merge to main
git checkout main
git merge --no-ff release/v1.2.3
# Tag release
git tag -a v1.2.3 -m "Release v1.2.3: Description"
# Push
git push origin main
git push origin v1.2.3
# Clean up
git branch -d release/v1.2.3
Phase 4: Staged Rollout¶
Day 0: Deploy to paper trading
├── Monitor for 24 hours
├── Check for errors
└── Verify signal generation
Day 1-3: Paper trading validation
├── Minimum 10 trades
├── No critical errors
└── Performance within expected range
Day 4: Deploy to live (minimal size)
├── 25% of target position size
├── Monitor closely
└── Kill switch ready
Day 7: Increase to 50% size
├── If Day 4-6 stable
├── Continue monitoring
Day 14: Full deployment
├── If Day 7-13 stable
├── 100% of target size
├── Standard monitoring
Audit Sign-Off Requirements¶
For Strategy Releases¶
| Requirement | Evidence | Location |
|---|---|---|
| Audit report | PASS or PASS-WITH-FIXES | docs/audits/{strategy}_audit.md |
| All fixes complete | Verified | Audit report Section 5 |
| Anti-bias tests | All pass | CI logs |
| Walk-forward | > 60% consistency | Audit report Section 6 |
| Paper results | Within 20% of backtest | Paper reconciliation report |
Audit Sign-Off Checklist¶
## Release Audit Sign-Off: v1.2.3
### Strategy: ICT_QM v2.1.0
- [ ] Audit report: docs/audits/ict_qm_audit.md
- [ ] Verdict: PASS
- [ ] Audit date: 2024-01-15
- [ ] Within 30 days: Yes
### Validation
- [ ] Anti-bias tests: PASS (CI run #123)
- [ ] Paper trading: 2 weeks, 25 trades
- [ ] Paper vs backtest: Sharpe 1.2 vs 1.4 (14% divergence)
### Approvals
- [ ] Auditor: _________________ Date: _______
- [ ] Risk: _________________ Date: _______
- [ ] Founder: _________________ Date: _______
### Release Decision
- [ ] APPROVED for production
- [ ] HOLD pending: _____________
- [ ] REJECTED reason: _____________
Rollback Procedure¶
When to Rollback¶
| Condition | Action |
|---|---|
| Live errors > 3 | Immediate rollback |
| Unexpected losses | Pause + review |
| Signal divergence > 30% | Rollback within 24h |
| System instability | Immediate rollback |
Rollback Steps¶
# 1. Activate kill switch (if trading)
python -m execution.mt5.kill_switch --activate --reason "Rollback"
# 2. Identify previous stable version
git log --oneline --tags | head -5
# 3. Checkout previous version
git checkout v1.2.2
# 4. Deploy previous version
# (For Pine Script: revert in TradingView)
# (For Python: restart services with old code)
# 5. Verify rollback
python scripts/verify_deployment.py --version=v1.2.2
# 6. Deactivate kill switch (if safe)
python -m execution.mt5.kill_switch --deactivate --authorized-by="founder"
# 7. Document incident
# See docs/runbooks/incident_response.md
Release Checklist Template¶
# Release Checklist: v{VERSION}
## Pre-Release
- [ ] Version.yaml updated
- [ ] CHANGELOG.md updated
- [ ] All tests passing
- [ ] Audit approval (if strategy)
- [ ] Paper trading complete (if strategy)
## Review
- [ ] PR created
- [ ] Code review approved
- [ ] Risk review approved (if needed)
- [ ] Founder approval
## Deployment
- [ ] Tag created
- [ ] Deployed to paper
- [ ] Paper validation (24h min)
- [ ] Deployed to live (minimal)
- [ ] Monitoring configured
- [ ] Kill switch tested
## Post-Release
- [ ] Announcement sent
- [ ] Documentation updated
- [ ] Metrics baseline recorded
## Sign-Off
Released by: _____________
Date: _____________
Emergency Release (Hotfix)¶
Expedited Process¶
1. Create hotfix branch from production tag
git checkout -b hotfix/critical-fix v1.2.3
2. Make minimal fix
3. Expedited review (Founder + 1 other)
4. Tag and deploy
git tag -a v1.2.4-hotfix -m "Hotfix: description"
5. Deploy directly to live (skip paper)
6. Post-mortem within 24 hours
7. Proper release within 48 hours
Hotfix Criteria¶
Only these qualify for hotfix: - Active money at risk - Critical security vulnerability - Complete system failure - Regulatory requirement
All others follow standard release process.