feat: Add core trading modules for risk management, backtesting, and execution algorithms, alongside a new ML transparency widget and related frontend dependencies.

docs/architecture/ml_improvements.md

@@ -0,0 +1,183 @@
+# Machine Learning Improvements
+
+This document describes the ML enhancements added to the intelligent autopilot system.
+
+## Overview
+
+The ML improvements focus on making the strategy selection model more robust, interpretable, and adaptive to changing market conditions.
+
+## Components
+
+### 1. Online Learning Pipeline
+
+**Location**: `src/autopilot/online_learning.py`
+
+**Features**:
+- Incremental model updates from live trading data
+- Concept drift detection using performance windows
+- Buffered training samples for efficient batch updates
+- Automatic full retraining on drift detection
+
+**Usage**:
+```python
+from src.autopilot.online_learning import get_online_learning_pipeline
+
+pipeline = get_online_learning_pipeline(model)
+
+# Add training sample after trade
+await pipeline.add_training_sample(
+    market_conditions=conditions,
+    strategy_name="selected_strategy",
+    performance=trade_return
+)
+
+# Check for drift and retrain if needed
+retrain_result = await pipeline.trigger_full_retrain_if_needed()
+```
+
+### 2. Confidence Calibration
+
+**Location**: `src/autopilot/confidence_calibration.py`
+
+**Features**:
+- Platt scaling (logistic regression calibration)
+- Isotonic regression calibration
+- Probability distribution calibration
+- Validation data integration
+
+**Methods**:
+- `Platt Scaling`: Fast, parametric calibration using logistic regression
+- `Isotonic Regression`: Non-parametric, more flexible but requires more data
+
+**Usage**:
+```python
+from src.autopilot.confidence_calibration import get_confidence_calibration_manager
+
+calibrator = get_confidence_calibration_manager()
+
+# Fit from validation data
+calibrator.fit_from_validation_data(
+    predicted_probs=[...],
+    true_labels=[...]
+)
+
+# Calibrate predictions
+strategy, calibrated_conf, calibrated_preds = calibrator.calibrate_prediction(
+    strategy_name="strategy",
+    confidence=0.85,
+    all_predictions={...}
+)
+```
+
+### 3. Model Explainability
+
+**Location**: `src/autopilot/explainability.py`
+
+**Features**:
+- SHAP (SHapley Additive exPlanations) value integration
+- Feature importance analysis (global and local)
+- Prediction explanations with top contributing features
+- Support for tree-based and kernel-based models
+
+**Usage**:
+```python
+from src.autopilot.explainability import get_model_explainer
+
+explainer = get_model_explainer(model)
+
+# Initialize with background data
+explainer.initialize_explainer(background_data_df)
+
+# Explain a prediction
+explanation = explainer.explain_prediction(features)
+# Returns: feature_importance, top_positive_features, top_negative_features, etc.
+
+# Get global feature importance
+global_importance = explainer.get_global_feature_importance()
+```
+
+### 4. Advanced Regime Detection
+
+**Location**: `src/autopilot/regime_detection.py`
+
+**Features**:
+- Hidden Markov Models (HMM) for regime detection
+- Gaussian Mixture Models (GMM) for regime detection
+- Hybrid detection combining multiple methods
+- Probabilistic regime predictions
+
+**Methods**:
+- `HMM`: Models regime transitions as Markov process
+- `GMM`: Clusters market states using Gaussian mixtures
+- `Hybrid`: Combines both methods for robust detection
+
+**Usage**:
+```python
+from src.autopilot.regime_detection import AdvancedRegimeDetector
+
+detector = AdvancedRegimeDetector(method="hmm")
+detector.fit_from_dataframe(ohlcv_df)
+
+regime = detector.detect_regime(returns=0.01, volatility=0.02)
+```
+
+### 5. Enhanced Feature Engineering
+
+**Location**: `src/autopilot/feature_engineering.py`
+
+**Enhancements**:
+- Multi-timeframe feature aggregation
+- Order book feature extraction
+- Feature interactions (products, ratios)
+- Regime-specific feature engineering
+- Lag features for temporal patterns
+
+## Integration
+
+These components integrate with the existing `IntelligentAutopilot` and `StrategySelector` classes:
+
+1. **Online Learning**: Integrated via `_record_trade_for_learning` method
+2. **Confidence Calibration**: Applied in `select_best_strategy` method
+3. **Explainability**: Available via API endpoints for UI visualization
+4. **Regime Detection**: Used in `MarketAnalyzer` for enhanced regime classification
+
+## Configuration
+
+Configuration options in `config/config.yaml`:
+
+```yaml
+autopilot:
+  intelligent:
+    online_learning:
+      drift_window: 100
+      drift_threshold: 0.1
+      buffer_size: 50
+      update_frequency: 100
+    confidence_calibration:
+      method: "isotonic"  # or "platt"
+    regime_detection:
+      method: "hmm"  # or "gmm" or "hybrid"
+      n_regimes: 4
+```
+
+## Dependencies
+
+Optional dependencies (with fallbacks):
+- `hmmlearn`: For HMM regime detection
+- `shap`: For model explainability
+- `scipy`: For calibration methods (isotonic regression)
+
+## Performance Considerations
+
+- **Online Learning**: Batches updates for efficiency (configurable buffer size)
+- **SHAP Values**: Can be slow for large models; consider caching or background computation
+- **HMM/GMM**: Training is fast, prediction is very fast
+- **Calibration**: Fitting is fast, prediction is O(1)
+
+## Testing
+
+Recommended testing approach:
+1. Use synthetic data for online learning pipeline
+2. Test calibration with known probability distributions
+3. Validate SHAP values against known feature importance
+4. Compare HMM/GMM regimes against rule-based classification

docs/architecture/overview.md

@@ -97,21 +97,30 @@ Crypto Trader follows a modular architecture with clear separation of concerns:
 
 ### 8. Risk Management
 
-- **Components**: Risk manager, stop-loss, position sizing, limits
-- **Integration**: Pre-trade checks, real-time monitoring
-- **Features**: Drawdown limits, daily loss limits, position limits
+- **Components**: Risk manager, stop-loss, position sizing, limits, VaR calculator, correlation analyzer
+- **Integration**: Pre-trade checks, real-time monitoring, portfolio-level analysis
+- **Features**: 
+  - Drawdown limits, daily loss limits, position limits
+  - Value at Risk (VaR) calculation (Historical, Parametric, Monte Carlo, CVaR)
+  - Portfolio correlation analysis and diversification scoring
+  - Correlation-based position limits
+  - Advanced position sizing (volatility-adjusted, fractional Kelly, regime-aware, confidence-based)
 
 ### 9. Backtesting Engine
 
-- **Features**: Historical data replay, realistic simulation
-- **Components**: Engine, metrics, slippage model, fee model
-- **Optimization**: Parameter optimization support
+- **Features**: Historical data replay, realistic simulation, walk-forward analysis, Monte Carlo simulation
+- **Components**: Engine, metrics, slippage model, fee model, walk-forward analyzer, Monte Carlo simulator
+- **Optimization**: 
+  - Parameter optimization (grid search, Bayesian, genetic algorithms)
+  - Walk-forward analysis with rolling window optimization
+  - Monte Carlo simulation for robustness testing
 
 ### 10. Portfolio Management
 
-- **Tracking**: Real-time position tracking
-- **Analytics**: Performance metrics, risk analysis
-- **Rebalancing**: Automatic portfolio rebalancing (planned)
+- **Tracking**: Real-time position tracking, performance analytics
+- **Analytics**: Performance metrics, risk analysis, correlation analysis, VaR calculation
+- **Rebalancing**: Automatic portfolio rebalancing with threshold and time-based triggers
+- **Components**: Portfolio tracker, correlation analyzer, rebalancing engine
 
 ## Data Flow
 

docs/architecture/risk_management.md

@@ -1,5 +1,120 @@
 # Risk Management Architecture
 
+## Overview
+
+The risk management system provides comprehensive risk control mechanisms including position sizing, stop-loss management, drawdown limits, Value at Risk (VaR) calculation, and portfolio correlation analysis.
+
+## Components
+
+### Position Sizing
+
+**Location**: `src/risk/position_sizing.py`
+
+**Methods**:
+- **Standard Position Sizing**: Percentage-based with fee accounting
+- **Kelly Criterion**: Optimal position sizing with fractional Kelly (configurable)
+- **Volatility-Adjusted**: ATR-based position sizing (lower vol = larger positions)
+- **Regime-Aware**: Adjusts position size based on market regime
+- **Confidence-Based**: ML model confidence-adjusted position sizing
+
+**Usage**:
+```python
+from src.risk.position_sizing import PositionSizingManager
+
+sizer = PositionSizingManager()
+
+# Standard sizing
+quantity = sizer.calculate_size(symbol, price, balance, risk_percent)
+
+# Kelly Criterion (fractional)
+kelly_pct = sizer.calculate_kelly_criterion(win_rate=0.6, avg_win=100, avg_loss=50, fractional=0.25)
+
+# Volatility-adjusted
+quantity = sizer.calculate_volatility_adjusted_size(symbol, price, balance, volatility_multiplier=0.8)
+
+# Regime-aware
+quantity = sizer.calculate_regime_aware_size(symbol, price, balance, market_regime="trending_up")
+```
+
+### Value at Risk (VaR)
+
+**Location**: `src/risk/var_calculator.py`
+
+**Methods**:
+1. **Historical VaR**: Uses historical portfolio returns distribution
+2. **Parametric VaR**: Assumes normal distribution (variance-covariance method)
+3. **Monte Carlo VaR**: Simulates future returns using estimated parameters
+4. **Conditional VaR (CVaR)**: Expected loss given that loss exceeds VaR
+
+**Usage**:
+```python
+from src.risk.var_calculator import get_var_calculator
+
+var_calc = get_var_calculator()
+
+# Calculate all methods
+results = await var_calc.calculate_all_var_methods(
+    portfolio_value=Decimal("10000.0"),
+    confidence_level=0.95,
+    holding_period_days=1
+)
+
+# Individual methods
+historical_var = await var_calc.calculate_historical_var(...)
+parametric_var = await var_calc.calculate_parametric_var(...)
+monte_carlo_var = await var_calc.calculate_monte_carlo_var(...)
+cvar = await var_calc.calculate_cvar(...)
+```
+
+### Portfolio Correlation Analysis
+
+**Location**: `src/portfolio/correlation_analyzer.py`
+
+**Features**:
+- Correlation matrix calculation for portfolio symbols
+- Diversification scoring (lower correlation = better)
+- Concentration risk analysis
+- Correlation-based position limits
+
+**Usage**:
+```python
+from src.portfolio.correlation_analyzer import get_correlation_analyzer
+
+analyzer = get_correlation_analyzer()
+
+# Analyze current portfolio
+analysis = await analyzer.analyze_portfolio_correlation(paper_trading=True)
+
+# Check correlation limits before adding position
+allowed, reason = await analyzer.check_correlation_limits(
+    symbol="ETH/USD",
+    new_position_value=Decimal("1000.0"),
+    max_correlation=0.8
+)
+```
+
+### Stop-Loss Management
+
+**Location**: `src/risk/stop_loss.py`
+
+Provides dynamic stop-loss adjustment and management.
+
+### Risk Limits
+
+**Location**: `src/risk/limits.py`
+
+Manages:
+- Daily loss limits
+- Maximum drawdown limits
+- Portfolio allocation limits
+
+### Risk Manager
+
+**Location**: `src/risk/manager.py`
+
+Orchestrates all risk management components and provides unified risk checking interface.
+
+
 This document describes the risk management system.
 
 ## Risk Management Components