crypto_trader/docs/architecture/autopilot.md

# Autopilot Architecture

This document describes the architecture of the Autopilot system, including both Pattern-Based and ML-Based modes.

## Overview

The Autopilot system provides autonomous trading signal generation using an **Intelligent Autopilot** powered by machine learning-based strategy selection.

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────┐
│                    Unified Autopilot API                    │
│                  (/api/autopilot/start-unified)            │
└───────────────────────┬─────────────────────────────────────┘
                        │
        ┌───────────────┴───────────────┐
        │                               │
        ▼                               ▼
┌───────────────────┐         ┌───────────────────┐
│ ML-Based          │         │  Signal Display   │
│ Autopilot         │         │  (Dashboard)      │
├───────────────────┤         └───────────────────┘
│ MarketAnalyzer    │
│ StrategySelector  │
│ PerformanceTracker│
└───────────────────┘
        │
        ▼
┌───────────────────────┐
│   Signal Generation   │
│   (StrategySignal)    │
└───────────────────────┘
        │
        ▼
┌───────────────────┐
│  Auto-Execution   │
│  (Optional)       │
└───────────────────┘
```

## ML-Based Autopilot Flow

> [!NOTE]
> **Global Model Architecture**: The system uses a single "Global Model" trained on data from *all* configured symbols. This allows for shared learning (patterns from BTC help predict ETH) and efficient resource usage.

> [!IMPORTANT]
> **Single-Asset Focus**: The current ML model is designed for **single-asset strategies** (e.g., Momentum, RSI, Grid). It analyzes each coin in isolation. It does **NOT** currently support multi-asset strategies like **Arbitrage**, which require analyzing correlations between multiple coins simultaneously.

```
Market Data (OHLCV)
        │
        ▼
┌───────────────────┐
│ MarketAnalyzer    │
│  - Analyze        │
│    conditions     │
│  - Determine      │
│    regime         │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ StrategySelector   │
│  - Global ML Model │
│  - Select best     │
│    strategy        │
│  - Calculate       │
│    confidence      │
└─────────┬─────────┘
          │
          ▼
┌───────────────────┐
│ Selected Strategy │
│  - Generate signal│
│  - Calculate size  │
└─────────┬─────────┘
          │
          ▼
    StrategySignal
```

## API Endpoints

### Unified Endpoints

- `POST /api/autopilot/start-unified` - Start autopilot
- `POST /api/autopilot/stop-unified` - Stop autopilot
- `GET /api/autopilot/status-unified/{symbol}` - Get autopilot status

### Legacy Endpoints (Removed)

- `POST /api/autopilot/start` - (Removed)
- `POST /api/autopilot/intelligent/start` - (Removed)

## Mode Selection Logic

The unified endpoints now exclusively support the "intelligent" mode. Code handling other modes has been removed.

```python
def start_unified_autopilot(config: UnifiedAutopilotConfig):
    # Always starts intelligent autopilot
    autopilot = get_intelligent_autopilot(
        symbol=config.symbol,
        exchange_id=config.exchange_id,
        timeframe=config.timeframe,
        interval=config.interval,
        paper_trading=config.paper_trading
    )
    
    if config.auto_execute:
        autopilot.enable_auto_execution = True
    
    autopilot.start()
```

## Auto-Execution Integration

Auto-execution is integrated directly into the ML-based workflow:

```python
# Auto-Execution
if auto_execute and signal and confidence > threshold:
    trading_engine.execute_order(
        symbol=signal.symbol,
        side=signal.signal_type,
        quantity=strategy.calculate_position_size(signal, balance),
        ...
    )
```

## Pre-Flight Order Validation

Before submitting orders, the autopilot validates that the order will succeed:

```python
async def _can_execute_order(side, quantity, price) -> Tuple[bool, str]:
    # 1. Check minimum order value ($1 USD)
    if order_value < 1.0:
        return False, "Order value below minimum"
    
    # 2. For BUY: check sufficient balance (with fee buffer)
    if side == BUY:
        if balance < (order_value + fee_estimate):
            return False, "Insufficient funds"
    
    # 3. For SELL: check position exists
    if side == SELL:
        if no_position:
            return False, "No position to sell"
    
    return True, "OK"
```

**Benefits:**
- Prevents creation of PENDING orders that would be REJECTED
- Cleaner order history (no garbage orders)
- Faster execution (skip validation at execution layer)

## Smart Order Type Selection

The autopilot intelligently chooses between LIMIT and MARKET orders:

| Scenario | Order Type | Rationale |
|----------|-----------|-----------|
| Strong signal (>80%) | MARKET | High confidence, execute now |
| Normal BUY signal | LIMIT (-0.1%) | Get better entry, pay maker fees |
| Take-profit SELL | LIMIT (+0.1%) | Get better exit price |
| Stop-loss SELL | MARKET | Urgent exit, protect capital |

```python
def _determine_order_type_and_price(side, signal_strength, price, is_stop_loss):
    # Strong signals or stop-losses use MARKET
    if signal_strength > 0.8 or is_stop_loss:
        return MARKET, None
    
    # Normal signals use LIMIT with 0.1% better price
    if side == BUY:
        limit_price = price * 0.999  # 0.1% below market
    else:
        limit_price = price * 1.001  # 0.1% above market
    
    return LIMIT, limit_price
```

**Stop-Loss vs Take-Profit Detection:**
- If `current_price < entry_price`: Stop-loss (use MARKET)
- If `current_price > entry_price`: Take-profit (use LIMIT)

## Data Flow

1. **Market Data Collection**: OHLCV data from database
2. **Market Analysis**: MarketAnalyzer determines conditions
3. **Strategy Selection**: ML model selects best strategy
4. **Signal Generation**: Selected strategy generates signal
5. **Opportunity Evaluation**: Check confidence and criteria
6. **Execution** (if enabled): Execute trade
7. **Learning**: Record trade for model improvement

## Component Responsibilities

- **MarketAnalyzer**: Analyzes market conditions and determines regime
- **StrategySelector**: ML model for selecting optimal strategy
- **PerformanceTracker**: Tracks strategy performance for learning
- **IntelligentAutopilot**: Orchestrates ML-based selection and execution

## Configuration

```python
{
    "symbol": "BTC/USD",
    "exchange_id": 1,
    "timeframe": "1h",
    "interval": 60.0,  # Analysis cycle in seconds
    "paper_trading": True,
    "min_confidence_threshold": 0.75,
    "max_trades_per_day": 10,
    "auto_execute": False
}
```

## State Management

- `_running`: Boolean flag
- `_last_analysis`: Last analysis result
- `_selected_strategy`: Currently selected strategy
- `_trades_today`: Trade count for daily limit
- `_ohlcv_data`: Current market data
- `_strategy_instances`: Cached strategy instances
- `_intelligent_autopilots`: Global registry of active instances

## Error Handling

Both modes implement error handling:

- **Market Data Errors**: Fallback to cached data or skip cycle
- **Pattern Detection Errors**: Return no pattern, continue monitoring
- **Sentiment Analysis Errors**: Use neutral sentiment, continue
- **ML Model Errors**: Fallback to rule-based selection
- **Execution Errors**: Log error, continue monitoring

## Performance Considerations

- **Moderate**: ML model adds overhead but is optimized for speed
- **Adaptive**: Performance improves with more training data
- **Scalable**: Model can handle multiple symbols with shared learning

## ML Model Training

The ML model training system supports background training via Celery with configurable parameters.

### Training Configuration

Training is configured via the Bootstrap Config API and persisted in the application config:

| Parameter | Description | Default |
|-----------|-------------|---------|
| `days` | Historical data (days) to fetch | 90 |
| `timeframe` | OHLCV candle timeframe | `1h` |
| `min_samples_per_strategy` | Minimum samples needed per strategy | 10 |
| `symbols` | Cryptocurrencies to train on | `["BTC/USD", "ETH/USD"]` |

### API Endpoints

```
GET  /api/autopilot/bootstrap-config     # Get current config
PUT  /api/autopilot/bootstrap-config     # Update config
POST /api/autopilot/intelligent/retrain  # Trigger retraining
GET  /api/autopilot/tasks/{task_id}      # Poll training status
GET  /api/autopilot/intelligent/model-info  # Get model info
POST /api/autopilot/intelligent/reset    # Reset model
```

### Background Training Architecture

```
┌─────────────────┐     ┌──────────────┐     ┌─────────────────┐
│ Settings Page   │────▶│ FastAPI      │────▶│ Celery Worker   │
│ (React)         │     │ Backend      │     │ (train_task)    │
└─────────────────┘     └──────────────┘     └─────────────────┘
        │                      │                      │
        │ Poll status          │ get_config()         │ Reset singletons
        │◀─────────────────────│                      │ Bootstrap data
        │                      │                      │ Train model
        │                      │                      │ Save to disk
        │                      │◀─────────────────────│
        │ SUCCESS + metrics    │                      │
        │◀─────────────────────│                      │
        │                      │                      │
        ▼                      ▼                      ▼
  UI shows results      Model auto-loads       Model file saved
```

### Training Task Flow

1. **Singleton Reset**: Reset `StrategySelector`, `PerformanceTracker`, and `Database` singletons to prevent async conflicts
2. **Check Existing Data**: Query performance tracker for existing training samples
3. **Bootstrap (if needed)**: Fetch OHLCV data for each symbol with progress updates
4. **Train Model**: Train ML model with cross-validation and walk-forward validation
5. **Save Model**: Persist trained model to `~/.local/share/crypto_trader/models/`
6. **Return Metrics**: Return training accuracy and model metadata

### Model Auto-Reload

The `StrategySelector.get_model_info()` method automatically checks for new model files:

```python
def get_model_info(self):
    # Always check if a newer model is available on disk
    self._try_load_saved_model()  # Compare timestamps, reload if newer
    return {...}
```

This ensures the API process picks up newly trained models without restart.

### Error Handling

- **Celery Task Errors**: Serialized to JSON with traceback for frontend display
- **Polling Errors**: Frontend stops polling after 3 consecutive failures
- **Connection Errors**: Singletons reset to prevent `asyncpg.InterfaceError`

## Strategy Grouping

To improve ML accuracy, individual strategies are grouped into 5 logical categories. The model predicts the best **group**, then rule-based logic selects the optimal strategy within that group.

### Strategy Groups

| Group | Strategies | Market Conditions |
|-------|-----------|-------------------|
| **Trend Following** | moving_average, macd, confirmed | Strong trends (ADX > 25) |
| **Mean Reversion** | rsi, bollinger_mean_reversion, grid, divergence | Ranging markets, oversold/overbought |
| **Momentum** | momentum, volatility_breakout | High volume spikes, breakouts |
| **Market Making** | market_making, dca | Low volatility, stable markets |
| **Sentiment Based** | sentiment, pairs_trading, consensus | External signals available |

### Benefits

- **Reduced Classes**: 5 groups vs 14 strategies (~20% random baseline vs ~7%)
- **Better Generalization**: Model learns group characteristics rather than memorizing individual strategies
- **Combined Confidence**: `group_confidence × strategy_confidence` for final score

## Improved Bootstrap Sampling

Training data is collected using intelligent sampling rather than fixed intervals:

### Sampling Strategies

1. **Regime-Change Detection**: Sample when market regime changes (e.g., trending → ranging)
2. **Minimum Time Spacing**: 24-hour gaps between samples for 1h timeframe
3. **Periodic Sampling**: Every 48 hours regardless of regime changes

### Timeframe Spacing

| Timeframe | Min Spacing (candles) | Actual Time |
|-----------|----------------------|-------------|
| 1m | 1440 | 24 hours |
| 15m | 96 | 24 hours |
| 1h | 24 | 24 hours |
| 4h | 6 | 24 hours |
| 1d | 1 | 24 hours |

## Security Considerations

- **Auto-Execution**: Requires explicit user confirmation
- **Paper Trading**: Default mode for safety
- **Risk Limits**: Enforced regardless of mode
- **API Keys**: Encrypted storage
- **Audit Logging**: All autopilot actions logged

## Future Enhancements

- Hybrid mode combining both approaches
- Real-time mode switching
- Multi-symbol autopilot management
- Advanced risk management integration
- Performance analytics dashboard
- Persistent training configuration (file/database storage)