379 lines
14 KiB
Markdown
379 lines
14 KiB
Markdown
# Autopilot Architecture
|
||
|
||
This document describes the architecture of the Autopilot system, including both Pattern-Based and ML-Based modes.
|
||
|
||
## Overview
|
||
|
||
The Autopilot system provides autonomous trading signal generation using an **Intelligent Autopilot** powered by machine learning-based strategy selection.
|
||
|
||
## Architecture Diagram
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Unified Autopilot API │
|
||
│ (/api/autopilot/start-unified) │
|
||
└───────────────────────┬─────────────────────────────────────┘
|
||
│
|
||
┌───────────────┴───────────────┐
|
||
│ │
|
||
▼ ▼
|
||
┌───────────────────┐ ┌───────────────────┐
|
||
│ ML-Based │ │ Signal Display │
|
||
│ Autopilot │ │ (Dashboard) │
|
||
├───────────────────┤ └───────────────────┘
|
||
│ MarketAnalyzer │
|
||
│ StrategySelector │
|
||
│ PerformanceTracker│
|
||
└───────────────────┘
|
||
│
|
||
▼
|
||
┌───────────────────────┐
|
||
│ Signal Generation │
|
||
│ (StrategySignal) │
|
||
└───────────────────────┘
|
||
│
|
||
▼
|
||
┌───────────────────┐
|
||
│ Auto-Execution │
|
||
│ (Optional) │
|
||
└───────────────────┘
|
||
```
|
||
|
||
## ML-Based Autopilot Flow
|
||
|
||
> [!NOTE]
|
||
> **Global Model Architecture**: The system uses a single "Global Model" trained on data from *all* configured symbols. This allows for shared learning (patterns from BTC help predict ETH) and efficient resource usage.
|
||
|
||
> [!IMPORTANT]
|
||
> **Single-Asset Focus**: The current ML model is designed for **single-asset strategies** (e.g., Momentum, RSI, Grid). It analyzes each coin in isolation. It does **NOT** currently support multi-asset strategies like **Arbitrage**, which require analyzing correlations between multiple coins simultaneously.
|
||
|
||
```
|
||
Market Data (OHLCV)
|
||
│
|
||
▼
|
||
┌───────────────────┐
|
||
│ MarketAnalyzer │
|
||
│ - Analyze │
|
||
│ conditions │
|
||
│ - Determine │
|
||
│ regime │
|
||
└─────────┬─────────┘
|
||
│
|
||
▼
|
||
┌───────────────────┐
|
||
│ StrategySelector │
|
||
│ - Global ML Model │
|
||
│ - Select best │
|
||
│ strategy │
|
||
│ - Calculate │
|
||
│ confidence │
|
||
└─────────┬─────────┘
|
||
│
|
||
▼
|
||
┌───────────────────┐
|
||
│ Selected Strategy │
|
||
│ - Generate signal│
|
||
│ - Calculate size │
|
||
└─────────┬─────────┘
|
||
│
|
||
▼
|
||
StrategySignal
|
||
```
|
||
|
||
## API Endpoints
|
||
|
||
### Unified Endpoints
|
||
|
||
- `POST /api/autopilot/start-unified` - Start autopilot
|
||
- `POST /api/autopilot/stop-unified` - Stop autopilot
|
||
- `GET /api/autopilot/status-unified/{symbol}` - Get autopilot status
|
||
|
||
### Legacy Endpoints (Removed)
|
||
|
||
- `POST /api/autopilot/start` - (Removed)
|
||
- `POST /api/autopilot/intelligent/start` - (Removed)
|
||
|
||
## Mode Selection Logic
|
||
|
||
The unified endpoints now exclusively support the "intelligent" mode. Code handling other modes has been removed.
|
||
|
||
```python
|
||
def start_unified_autopilot(config: UnifiedAutopilotConfig):
|
||
# Always starts intelligent autopilot
|
||
autopilot = get_intelligent_autopilot(
|
||
symbol=config.symbol,
|
||
exchange_id=config.exchange_id,
|
||
timeframe=config.timeframe,
|
||
interval=config.interval,
|
||
paper_trading=config.paper_trading
|
||
)
|
||
|
||
if config.auto_execute:
|
||
autopilot.enable_auto_execution = True
|
||
|
||
autopilot.start()
|
||
```
|
||
|
||
## Auto-Execution Integration
|
||
|
||
Auto-execution is integrated directly into the ML-based workflow:
|
||
|
||
```python
|
||
# Auto-Execution
|
||
if auto_execute and signal and confidence > threshold:
|
||
trading_engine.execute_order(
|
||
symbol=signal.symbol,
|
||
side=signal.signal_type,
|
||
quantity=strategy.calculate_position_size(signal, balance),
|
||
...
|
||
)
|
||
```
|
||
|
||
## Pre-Flight Order Validation
|
||
|
||
Before submitting orders, the autopilot validates that the order will succeed:
|
||
|
||
```python
|
||
async def _can_execute_order(side, quantity, price) -> Tuple[bool, str]:
|
||
# 1. Check minimum order value ($1 USD)
|
||
if order_value < 1.0:
|
||
return False, "Order value below minimum"
|
||
|
||
# 2. For BUY: check sufficient balance (with fee buffer)
|
||
if side == BUY:
|
||
if balance < (order_value + fee_estimate):
|
||
return False, "Insufficient funds"
|
||
|
||
# 3. For SELL: check position exists
|
||
if side == SELL:
|
||
if no_position:
|
||
return False, "No position to sell"
|
||
|
||
return True, "OK"
|
||
```
|
||
|
||
**Benefits:**
|
||
- Prevents creation of PENDING orders that would be REJECTED
|
||
- Cleaner order history (no garbage orders)
|
||
- Faster execution (skip validation at execution layer)
|
||
|
||
## Smart Order Type Selection
|
||
|
||
The autopilot intelligently chooses between LIMIT and MARKET orders:
|
||
|
||
| Scenario | Order Type | Rationale |
|
||
|----------|-----------|-----------|
|
||
| Strong signal (>80%) | MARKET | High confidence, execute now |
|
||
| Normal BUY signal | LIMIT (-0.1%) | Get better entry, pay maker fees |
|
||
| Take-profit SELL | LIMIT (+0.1%) | Get better exit price |
|
||
| Stop-loss SELL | MARKET | Urgent exit, protect capital |
|
||
|
||
```python
|
||
def _determine_order_type_and_price(side, signal_strength, price, is_stop_loss):
|
||
# Strong signals or stop-losses use MARKET
|
||
if signal_strength > 0.8 or is_stop_loss:
|
||
return MARKET, None
|
||
|
||
# Normal signals use LIMIT with 0.1% better price
|
||
if side == BUY:
|
||
limit_price = price * 0.999 # 0.1% below market
|
||
else:
|
||
limit_price = price * 1.001 # 0.1% above market
|
||
|
||
return LIMIT, limit_price
|
||
```
|
||
|
||
**Stop-Loss vs Take-Profit Detection:**
|
||
- If `current_price < entry_price`: Stop-loss (use MARKET)
|
||
- If `current_price > entry_price`: Take-profit (use LIMIT)
|
||
|
||
## Data Flow
|
||
|
||
1. **Market Data Collection**: OHLCV data from database
|
||
2. **Market Analysis**: MarketAnalyzer determines conditions
|
||
3. **Strategy Selection**: ML model selects best strategy
|
||
4. **Signal Generation**: Selected strategy generates signal
|
||
5. **Opportunity Evaluation**: Check confidence and criteria
|
||
6. **Execution** (if enabled): Execute trade
|
||
7. **Learning**: Record trade for model improvement
|
||
|
||
## Component Responsibilities
|
||
|
||
- **MarketAnalyzer**: Analyzes market conditions and determines regime
|
||
- **StrategySelector**: ML model for selecting optimal strategy
|
||
- **PerformanceTracker**: Tracks strategy performance for learning
|
||
- **IntelligentAutopilot**: Orchestrates ML-based selection and execution
|
||
|
||
## Configuration
|
||
|
||
```python
|
||
{
|
||
"symbol": "BTC/USD",
|
||
"exchange_id": 1,
|
||
"timeframe": "1h",
|
||
"interval": 60.0, # Analysis cycle in seconds
|
||
"paper_trading": True,
|
||
"min_confidence_threshold": 0.75,
|
||
"max_trades_per_day": 10,
|
||
"auto_execute": False
|
||
}
|
||
```
|
||
|
||
## State Management
|
||
|
||
- `_running`: Boolean flag
|
||
- `_last_analysis`: Last analysis result
|
||
- `_selected_strategy`: Currently selected strategy
|
||
- `_trades_today`: Trade count for daily limit
|
||
- `_ohlcv_data`: Current market data
|
||
- `_strategy_instances`: Cached strategy instances
|
||
- `_intelligent_autopilots`: Global registry of active instances
|
||
|
||
## Error Handling
|
||
|
||
Both modes implement error handling:
|
||
|
||
- **Market Data Errors**: Fallback to cached data or skip cycle
|
||
- **Pattern Detection Errors**: Return no pattern, continue monitoring
|
||
- **Sentiment Analysis Errors**: Use neutral sentiment, continue
|
||
- **ML Model Errors**: Fallback to rule-based selection
|
||
- **Execution Errors**: Log error, continue monitoring
|
||
|
||
## Performance Considerations
|
||
|
||
- **Moderate**: ML model adds overhead but is optimized for speed
|
||
- **Adaptive**: Performance improves with more training data
|
||
- **Scalable**: Model can handle multiple symbols with shared learning
|
||
|
||
## ML Model Training
|
||
|
||
The ML model training system supports background training via Celery with configurable parameters.
|
||
|
||
### Training Configuration
|
||
|
||
Training is configured via the Bootstrap Config API and persisted in the application config:
|
||
|
||
| Parameter | Description | Default |
|
||
|-----------|-------------|---------|
|
||
| `days` | Historical data (days) to fetch | 90 |
|
||
| `timeframe` | OHLCV candle timeframe | `1h` |
|
||
| `min_samples_per_strategy` | Minimum samples needed per strategy | 10 |
|
||
| `symbols` | Cryptocurrencies to train on | `["BTC/USD", "ETH/USD"]` |
|
||
|
||
### API Endpoints
|
||
|
||
```
|
||
GET /api/autopilot/bootstrap-config # Get current config
|
||
PUT /api/autopilot/bootstrap-config # Update config
|
||
POST /api/autopilot/intelligent/retrain # Trigger retraining
|
||
GET /api/autopilot/tasks/{task_id} # Poll training status
|
||
GET /api/autopilot/intelligent/model-info # Get model info
|
||
POST /api/autopilot/intelligent/reset # Reset model
|
||
```
|
||
|
||
### Background Training Architecture
|
||
|
||
```
|
||
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||
│ Settings Page │────▶│ FastAPI │────▶│ Celery Worker │
|
||
│ (React) │ │ Backend │ │ (train_task) │
|
||
└─────────────────┘ └──────────────┘ └─────────────────┘
|
||
│ │ │
|
||
│ Poll status │ get_config() │ Reset singletons
|
||
│◀─────────────────────│ │ Bootstrap data
|
||
│ │ │ Train model
|
||
│ │ │ Save to disk
|
||
│ │◀─────────────────────│
|
||
│ SUCCESS + metrics │ │
|
||
│◀─────────────────────│ │
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
UI shows results Model auto-loads Model file saved
|
||
```
|
||
|
||
### Training Task Flow
|
||
|
||
1. **Singleton Reset**: Reset `StrategySelector`, `PerformanceTracker`, and `Database` singletons to prevent async conflicts
|
||
2. **Check Existing Data**: Query performance tracker for existing training samples
|
||
3. **Bootstrap (if needed)**: Fetch OHLCV data for each symbol with progress updates
|
||
4. **Train Model**: Train ML model with cross-validation and walk-forward validation
|
||
5. **Save Model**: Persist trained model to `~/.local/share/crypto_trader/models/`
|
||
6. **Return Metrics**: Return training accuracy and model metadata
|
||
|
||
### Model Auto-Reload
|
||
|
||
The `StrategySelector.get_model_info()` method automatically checks for new model files:
|
||
|
||
```python
|
||
def get_model_info(self):
|
||
# Always check if a newer model is available on disk
|
||
self._try_load_saved_model() # Compare timestamps, reload if newer
|
||
return {...}
|
||
```
|
||
|
||
This ensures the API process picks up newly trained models without restart.
|
||
|
||
### Error Handling
|
||
|
||
- **Celery Task Errors**: Serialized to JSON with traceback for frontend display
|
||
- **Polling Errors**: Frontend stops polling after 3 consecutive failures
|
||
- **Connection Errors**: Singletons reset to prevent `asyncpg.InterfaceError`
|
||
|
||
## Strategy Grouping
|
||
|
||
To improve ML accuracy, individual strategies are grouped into 5 logical categories. The model predicts the best **group**, then rule-based logic selects the optimal strategy within that group.
|
||
|
||
### Strategy Groups
|
||
|
||
| Group | Strategies | Market Conditions |
|
||
|-------|-----------|-------------------|
|
||
| **Trend Following** | moving_average, macd, confirmed | Strong trends (ADX > 25) |
|
||
| **Mean Reversion** | rsi, bollinger_mean_reversion, grid, divergence | Ranging markets, oversold/overbought |
|
||
| **Momentum** | momentum, volatility_breakout | High volume spikes, breakouts |
|
||
| **Market Making** | market_making, dca | Low volatility, stable markets |
|
||
| **Sentiment Based** | sentiment, pairs_trading, consensus | External signals available |
|
||
|
||
### Benefits
|
||
|
||
- **Reduced Classes**: 5 groups vs 14 strategies (~20% random baseline vs ~7%)
|
||
- **Better Generalization**: Model learns group characteristics rather than memorizing individual strategies
|
||
- **Combined Confidence**: `group_confidence × strategy_confidence` for final score
|
||
|
||
## Improved Bootstrap Sampling
|
||
|
||
Training data is collected using intelligent sampling rather than fixed intervals:
|
||
|
||
### Sampling Strategies
|
||
|
||
1. **Regime-Change Detection**: Sample when market regime changes (e.g., trending → ranging)
|
||
2. **Minimum Time Spacing**: 24-hour gaps between samples for 1h timeframe
|
||
3. **Periodic Sampling**: Every 48 hours regardless of regime changes
|
||
|
||
### Timeframe Spacing
|
||
|
||
| Timeframe | Min Spacing (candles) | Actual Time |
|
||
|-----------|----------------------|-------------|
|
||
| 1m | 1440 | 24 hours |
|
||
| 15m | 96 | 24 hours |
|
||
| 1h | 24 | 24 hours |
|
||
| 4h | 6 | 24 hours |
|
||
| 1d | 1 | 24 hours |
|
||
|
||
## Security Considerations
|
||
|
||
- **Auto-Execution**: Requires explicit user confirmation
|
||
- **Paper Trading**: Default mode for safety
|
||
- **Risk Limits**: Enforced regardless of mode
|
||
- **API Keys**: Encrypted storage
|
||
- **Audit Logging**: All autopilot actions logged
|
||
|
||
## Future Enhancements
|
||
|
||
- Hybrid mode combining both approaches
|
||
- Real-time mode switching
|
||
- Multi-symbol autopilot management
|
||
- Advanced risk management integration
|
||
- Performance analytics dashboard
|
||
- Persistent training configuration (file/database storage)
|
||
|