Local changes: Updated model training, removed debug instrumentation, and configuration improvements
This commit is contained in:
kfox
359
docs/user_manual/configuration.md
Normal file
359
docs/user_manual/configuration.md
Normal file
@@ -0,0 +1,359 @@
|
||||
# Configuration Guide
|
||||
|
||||
This guide explains how to configure Crypto Trader to suit your needs.
|
||||
|
||||
## Configuration Files
|
||||
|
||||
Configuration files are stored in `~/.config/crypto_trader/` following the XDG Base Directory Specification.
|
||||
|
||||
### Main Configuration
|
||||
|
||||
The main configuration file is `config.yaml`. It contains:
|
||||
|
||||
- Database settings
|
||||
- Logging configuration
|
||||
- Paper trading settings
|
||||
- Update preferences
|
||||
- Notification settings
|
||||
- Backtesting defaults
|
||||
|
||||
### Logging Configuration
|
||||
|
||||
Logging settings are in `logging.yaml`. You can configure:
|
||||
|
||||
- Log levels (DEBUG, INFO, WARNING, ERROR)
|
||||
- Log file location
|
||||
- Log rotation settings
|
||||
- Retention policies
|
||||
|
||||
## Exchange Configuration
|
||||
|
||||
### Adding Exchange API Keys via UI
|
||||
|
||||
1. Open the application
|
||||
2. Navigate to the **Settings** tab
|
||||
3. In the **Exchanges** section, click **Add Exchange**
|
||||
4. Enter exchange details:
|
||||
- **Exchange Name**: Name of the exchange (e.g., Coinbase)
|
||||
- **API Key**: Your exchange API key
|
||||
- **API Secret**: Your exchange API secret
|
||||
- **Use Sandbox/Testnet**: Enable for testing
|
||||
- **Read-Only Mode**: Recommended for safety (prevents trading)
|
||||
- **Enabled**: Enable the exchange connection
|
||||
5. Click **Save**
|
||||
6. Test the connection using **Test Connection** button
|
||||
|
||||
### Editing Exchange Settings
|
||||
|
||||
1. Select an exchange from the exchanges table
|
||||
2. Click **Edit Exchange**
|
||||
3. Update API keys or settings as needed
|
||||
4. Click **Save**
|
||||
|
||||
### Managing Exchanges
|
||||
|
||||
The Settings page provides comprehensive exchange management:
|
||||
|
||||
- **Status Indicators**: Each exchange shows a color-coded status indicator:
|
||||
- Green: Enabled and connected
|
||||
- Gray: Disabled
|
||||
- Red: Error state
|
||||
- **Test Connection**: Click the checkmark icon to test the connection
|
||||
- You'll receive a notification with detailed connection status
|
||||
- Success: Green notification
|
||||
- Warning: Yellow notification with details
|
||||
- Error: Red notification with error message
|
||||
- **Edit Exchange**: Click the pencil icon to edit exchange settings
|
||||
- **Delete Exchange**: Click the trash icon to delete (removes API keys)
|
||||
- Confirmation dialog will appear
|
||||
- You'll receive a success notification when deleted
|
||||
|
||||
### API Key Security
|
||||
|
||||
- API keys are encrypted before storage
|
||||
- Use read-only keys when possible
|
||||
- Enable IP whitelisting on your exchange account
|
||||
- Never share your API keys
|
||||
|
||||
## Paper Trading Configuration
|
||||
|
||||
Paper trading settings can be configured in the Settings page:
|
||||
|
||||
1. Navigate to **Settings** page
|
||||
2. Click on **Paper Trading** tab
|
||||
3. Configure:
|
||||
- **Initial Capital ($)**: Starting capital in USD (default: $100)
|
||||
- **Fee Model (Exchange)**: Select which exchange's fee structure to simulate
|
||||
4. Click **Save Settings**
|
||||
5. You'll receive a success notification when settings are saved
|
||||
|
||||
### Fee Exchange Models
|
||||
|
||||
Choose from available exchange fee models:
|
||||
|
||||
| Exchange | Maker Fee | Taker Fee | Best For |
|
||||
|----------|-----------|-----------|----------|
|
||||
| **Default** | 0.10% | 0.10% | General testing |
|
||||
| **Coinbase** | 0.40% | 0.60% | Conservative estimates |
|
||||
| **Kraken** | 0.16% | 0.26% | Moderate fee simulation |
|
||||
| **Binance** | 0.10% | 0.10% | Low-fee simulation |
|
||||
|
||||
The fee rates display shows your current maker fee, taker fee, and estimated round-trip cost.
|
||||
|
||||
### Resetting Paper Account
|
||||
|
||||
To reset your paper trading account (closes all positions and resets balance):
|
||||
|
||||
1. Navigate to **Settings** > **Paper Trading** tab
|
||||
2. Click **Reset Paper Account** button
|
||||
3. Confirm the reset in the dialog
|
||||
4. All positions will be closed and balance reset to initial capital
|
||||
5. You'll receive a success notification when complete
|
||||
|
||||
**Warning**: This action cannot be undone. All paper trading history will be preserved, but positions and balance will be reset.
|
||||
|
||||
## Data Provider Configuration
|
||||
|
||||
Data providers are used to fetch real-time pricing data for paper trading, backtesting, and ML training. They work independently of exchange integrations and don't require API keys.
|
||||
|
||||
### Configuring Providers
|
||||
|
||||
1. Navigate to **Settings** page
|
||||
2. Click on **Data Providers** tab
|
||||
3. Configure provider settings:
|
||||
|
||||
#### Primary Providers (CCXT)
|
||||
|
||||
Primary providers use CCXT to connect to cryptocurrency exchanges:
|
||||
- **Kraken**: Default priority 1 (tried first)
|
||||
- **Coinbase**: Default priority 2
|
||||
- **Binance**: Default priority 3
|
||||
|
||||
Each provider can be:
|
||||
- **Enabled/Disabled**: Toggle using the switch
|
||||
- **Reordered**: Use up/down arrows to change priority order
|
||||
- **Monitored**: View health status, response times, and success rates
|
||||
|
||||
#### Fallback Provider
|
||||
|
||||
**CoinGecko** is used as a fallback when primary providers are unavailable:
|
||||
- Free tier API (no authentication required)
|
||||
- Lower rate limits than primary providers
|
||||
- Provides basic price data
|
||||
|
||||
#### Cache Settings
|
||||
|
||||
Configure caching behavior:
|
||||
- **Ticker TTL (seconds)**: How long to cache ticker data (default: 2 seconds)
|
||||
- **OHLCV TTL (seconds)**: How long to cache candlestick data (default: 60 seconds)
|
||||
|
||||
### Provider Status
|
||||
|
||||
The Data Providers tab shows:
|
||||
- **Active Provider**: Currently used provider
|
||||
- **Health Status**: Color-coded status for each provider:
|
||||
- Green: Healthy and working
|
||||
- Yellow: Degraded performance
|
||||
- Red: Unhealthy or unavailable
|
||||
- **Response Times**: Average response time for each provider
|
||||
- **Success/Failure Counts**: Historical performance metrics
|
||||
|
||||
### Automatic Failover
|
||||
|
||||
The system automatically:
|
||||
- Monitors provider health
|
||||
- Switches to the next available provider if current one fails
|
||||
- Opens circuit breakers for repeatedly failing providers
|
||||
- Retries failed providers after a timeout period
|
||||
|
||||
### Troubleshooting Providers
|
||||
|
||||
If all providers fail:
|
||||
1. Check your internet connection
|
||||
2. Verify firewall isn't blocking connections
|
||||
3. Check provider status in the Settings tab
|
||||
4. Try enabling/disabling specific providers
|
||||
5. Reset provider metrics if needed
|
||||
|
||||
## Risk Management Settings
|
||||
|
||||
Configure risk limits in the Settings page:
|
||||
|
||||
1. Navigate to **Settings** page
|
||||
2. Click on **Risk Management** tab
|
||||
3. Configure the following settings:
|
||||
- **Max Drawdown Limit (%)**: Maximum portfolio drawdown before trading stops
|
||||
- **Daily Loss Limit (%)**: Maximum daily loss percentage
|
||||
- **Default Position Size (%)**: Default percentage of capital to use per trade
|
||||
4. Click **Save Risk Settings**
|
||||
5. You'll receive a success notification when settings are saved
|
||||
|
||||
Settings are validated before saving and you'll see error messages if values are invalid.
|
||||
|
||||
## Data Storage
|
||||
|
||||
Data is stored in `~/.local/share/crypto_trader/`:
|
||||
|
||||
- `trading.db` - Legacy file (removed)
|
||||
- `historical/` - Historical market data
|
||||
- `logs/` - Application logs
|
||||
|
||||
### Database Options
|
||||
|
||||
### Database Options
|
||||
|
||||
**PostgreSQL (Required)**:
|
||||
- All production data is stored in PostgreSQL.
|
||||
- Requires a running PostgreSQL instance.
|
||||
- Configure connection in `config.yaml`.
|
||||
|
||||
**SQLite (Internal)**:
|
||||
- Used internally for unit testing only.
|
||||
- No configuration required for users.
|
||||
|
||||
## Redis Configuration
|
||||
|
||||
Redis is used for distributed state management, ensuring autopilot state persists across restarts and preventing duplicate instances.
|
||||
|
||||
### Default Settings
|
||||
|
||||
Redis settings are configured in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
redis:
|
||||
host: "127.0.0.1"
|
||||
port: 6379
|
||||
db: 0
|
||||
password: null # Set if Redis requires authentication
|
||||
socket_connect_timeout: 5
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
You can also configure Redis via environment variables:
|
||||
- `REDIS_HOST` - Redis server hostname
|
||||
- `REDIS_PORT` - Redis server port
|
||||
- `REDIS_PASSWORD` - Redis password (if required)
|
||||
|
||||
### Verifying Redis Connection
|
||||
|
||||
Run the verification script:
|
||||
```bash
|
||||
python scripts/verify_redis.py
|
||||
```
|
||||
|
||||
## Celery Configuration (Background Tasks)
|
||||
|
||||
Celery handles long-running tasks like ML model training in the background.
|
||||
|
||||
### Starting the Worker
|
||||
|
||||
```bash
|
||||
# From project root (with virtualenv activated)
|
||||
celery -A src.worker.app worker --loglevel=info
|
||||
```
|
||||
|
||||
Or use the helper script:
|
||||
```bash
|
||||
./scripts/start_all.sh
|
||||
```
|
||||
|
||||
### Task Queues
|
||||
|
||||
Tasks are routed to specific queues:
|
||||
- `ml_training` - Model training and data bootstrapping
|
||||
- `reporting` - Report generation
|
||||
- `celery` - Default queue
|
||||
|
||||
## ML Model Training Configuration
|
||||
|
||||
Configure how the autopilot ML model is trained via the Settings page.
|
||||
|
||||
### Training Data Configuration
|
||||
|
||||
Navigate to **Settings** > **Autopilot** tab to access these settings:
|
||||
|
||||
| Setting | Description | Recommended |
|
||||
|---------|-------------|-------------|
|
||||
| **Historical Data (days)** | Amount of historical data to fetch for training | 90-365 days |
|
||||
| **Timeframe** | OHLCV candle timeframe for training data | 1h (balanced) |
|
||||
| **Min Samples per Strategy** | Minimum samples required per strategy | 30+ |
|
||||
| **Training Symbols** | Cryptocurrencies to include in training | 5-10 major coins |
|
||||
|
||||
### Setting Training Symbols
|
||||
|
||||
1. Navigate to **Settings** > **Autopilot** tab
|
||||
2. In the **Training Data Configuration** section, find **Training Symbols**
|
||||
3. Click to add/remove symbols from the multi-select dropdown
|
||||
4. Common symbols: BTC/USD, ETH/USD, SOL/USD, XRP/USD, ADA/USD
|
||||
5. Click **Save Bootstrap Config** to persist your changes
|
||||
|
||||
### Triggering Model Training
|
||||
|
||||
1. Click **Retrain Model** button in the Model Management section
|
||||
2. A progress bar will appear showing:
|
||||
- Data fetching progress (20-60%)
|
||||
- Training progress (70-100%)
|
||||
3. Training typically takes 30-60 seconds depending on data volume
|
||||
4. Upon completion, the model card updates to show:
|
||||
- "Global Model Trained" badge
|
||||
- Number of strategies and features
|
||||
- Training accuracy
|
||||
- Which symbols the model was trained on
|
||||
|
||||
### Training Progress Persistence
|
||||
|
||||
Training progress persists across page navigation:
|
||||
- If you navigate away during training, progress resumes on return
|
||||
- The training task ID is stored in browser localStorage
|
||||
- On task completion or failure, the task ID is cleared
|
||||
|
||||
### Model Management
|
||||
|
||||
- **Retrain Model**: Triggers retraining with current configuration
|
||||
- **Reset Model**: Deletes all saved model files (confirmation required)
|
||||
|
||||
### Troubleshooting Training
|
||||
|
||||
If training fails:
|
||||
1. Check that Redis and Celery are running
|
||||
2. Review Celery worker logs: `tail -f celery.log`
|
||||
3. Ensure sufficient historical data is available
|
||||
4. Try reducing the number of training symbols
|
||||
5. Check backend logs for error details
|
||||
|
||||
### Monitoring Tasks
|
||||
|
||||
The API provides endpoints to monitor background tasks:
|
||||
- `POST /api/autopilot/intelligent/retrain` - Starts training, returns task ID
|
||||
- `GET /api/autopilot/tasks/{task_id}` - Check task status
|
||||
|
||||
## Environment Variables
|
||||
|
||||
You can override configuration using environment variables:
|
||||
|
||||
- `CRYPTO_TRADER_CONFIG_DIR` - Custom config directory
|
||||
- `CRYPTO_TRADER_DATA_DIR` - Custom data directory
|
||||
- `CRYPTO_TRADER_LOG_DIR` - Custom log directory
|
||||
- `DB_PASSWORD` - Database password (for PostgreSQL)
|
||||
- `REDIS_HOST` - Redis server hostname
|
||||
- `REDIS_PORT` - Redis server port
|
||||
- `REDIS_PASSWORD` - Redis password
|
||||
|
||||
## Backup and Restore
|
||||
|
||||
### Backup
|
||||
|
||||
1. Stop the application and Celery workers
|
||||
2. Backup PostgreSQL database
|
||||
3. Copy the entire `~/.local/share/crypto_trader/` directory
|
||||
4. Copy `~/.config/crypto_trader/` directory
|
||||
5. (Optional) Export Redis data with `redis-cli SAVE`
|
||||
|
||||
### Restore
|
||||
|
||||
1. Stop the application
|
||||
2. Restore PostgreSQL database
|
||||
3. Replace the directories with your backup
|
||||
4. Restart Redis, Celery worker, and application
|
||||
|
||||
Reference in New Issue
Block a user