crypto_trader/docs/user_manual/configuration.md

# Configuration Guide

This guide explains how to configure Crypto Trader to suit your needs.

## Configuration Files

Configuration files are stored in `~/.config/crypto_trader/` following the XDG Base Directory Specification.

### Main Configuration

The main configuration file is `config.yaml`. It contains:

- Database settings
- Logging configuration
- Paper trading settings
- Update preferences
- Notification settings
- Backtesting defaults

### Logging Configuration

Logging settings are in `logging.yaml`. You can configure:

- Log levels (DEBUG, INFO, WARNING, ERROR)
- Log file location
- Log rotation settings
- Retention policies

## Exchange Configuration

### Adding Exchange API Keys via UI

1. Open the application
2. Navigate to the **Settings** tab
3. In the **Exchanges** section, click **Add Exchange**
4. Enter exchange details:
   - **Exchange Name**: Name of the exchange (e.g., Coinbase)
   - **API Key**: Your exchange API key
   - **API Secret**: Your exchange API secret
   - **Use Sandbox/Testnet**: Enable for testing
   - **Read-Only Mode**: Recommended for safety (prevents trading)
   - **Enabled**: Enable the exchange connection
5. Click **Save**
6. Test the connection using **Test Connection** button

### Editing Exchange Settings

1. Select an exchange from the exchanges table
2. Click **Edit Exchange**
3. Update API keys or settings as needed
4. Click **Save**

### Managing Exchanges

The Settings page provides comprehensive exchange management:

- **Status Indicators**: Each exchange shows a color-coded status indicator:
  - Green: Enabled and connected
  - Gray: Disabled
  - Red: Error state
- **Test Connection**: Click the checkmark icon to test the connection
  - You'll receive a notification with detailed connection status
  - Success: Green notification
  - Warning: Yellow notification with details
  - Error: Red notification with error message
- **Edit Exchange**: Click the pencil icon to edit exchange settings
- **Delete Exchange**: Click the trash icon to delete (removes API keys)
  - Confirmation dialog will appear
  - You'll receive a success notification when deleted

### API Key Security

- API keys are encrypted before storage
- Use read-only keys when possible
- Enable IP whitelisting on your exchange account
- Never share your API keys

## Paper Trading Configuration

Paper trading settings can be configured in the Settings page:

1. Navigate to **Settings** page
2. Click on **Paper Trading** tab
3. Configure:
   - **Initial Capital ($)**: Starting capital in USD (default: $100)
   - **Fee Model (Exchange)**: Select which exchange's fee structure to simulate
4. Click **Save Settings**
5. You'll receive a success notification when settings are saved

### Fee Exchange Models

Choose from available exchange fee models:

| Exchange | Maker Fee | Taker Fee | Best For |
|----------|-----------|-----------|----------|
| **Default** | 0.10% | 0.10% | General testing |
| **Coinbase** | 0.40% | 0.60% | Conservative estimates |
| **Kraken** | 0.16% | 0.26% | Moderate fee simulation |
| **Binance** | 0.10% | 0.10% | Low-fee simulation |

The fee rates display shows your current maker fee, taker fee, and estimated round-trip cost.

### Resetting Paper Account

To reset your paper trading account (closes all positions and resets balance):

1. Navigate to **Settings** > **Paper Trading** tab
2. Click **Reset Paper Account** button
3. Confirm the reset in the dialog
4. All positions will be closed and balance reset to initial capital
5. You'll receive a success notification when complete

**Warning**: This action cannot be undone. All paper trading history will be preserved, but positions and balance will be reset.

## Data Provider Configuration

Data providers are used to fetch real-time pricing data for paper trading, backtesting, and ML training. They work independently of exchange integrations and don't require API keys.

### Configuring Providers

1. Navigate to **Settings** page
2. Click on **Data Providers** tab
3. Configure provider settings:

#### Primary Providers (CCXT)

Primary providers use CCXT to connect to cryptocurrency exchanges:
- **Kraken**: Default priority 1 (tried first)
- **Coinbase**: Default priority 2
- **Binance**: Default priority 3

Each provider can be:
- **Enabled/Disabled**: Toggle using the switch
- **Reordered**: Use up/down arrows to change priority order
- **Monitored**: View health status, response times, and success rates

#### Fallback Provider

**CoinGecko** is used as a fallback when primary providers are unavailable:
- Free tier API (no authentication required)
- Lower rate limits than primary providers
- Provides basic price data

#### Cache Settings

Configure caching behavior:
- **Ticker TTL (seconds)**: How long to cache ticker data (default: 2 seconds)
- **OHLCV TTL (seconds)**: How long to cache candlestick data (default: 60 seconds)

### Provider Status

The Data Providers tab shows:
- **Active Provider**: Currently used provider
- **Health Status**: Color-coded status for each provider:
  - Green: Healthy and working
  - Yellow: Degraded performance
  - Red: Unhealthy or unavailable
- **Response Times**: Average response time for each provider
- **Success/Failure Counts**: Historical performance metrics

### Automatic Failover

The system automatically:
- Monitors provider health
- Switches to the next available provider if current one fails
- Opens circuit breakers for repeatedly failing providers
- Retries failed providers after a timeout period

### Troubleshooting Providers

If all providers fail:
1. Check your internet connection
2. Verify firewall isn't blocking connections
3. Check provider status in the Settings tab
4. Try enabling/disabling specific providers
5. Reset provider metrics if needed

## Risk Management Settings

Configure risk limits in the Settings page:

1. Navigate to **Settings** page
2. Click on **Risk Management** tab
3. Configure the following settings:
   - **Max Drawdown Limit (%)**: Maximum portfolio drawdown before trading stops
   - **Daily Loss Limit (%)**: Maximum daily loss percentage
   - **Default Position Size (%)**: Default percentage of capital to use per trade
4. Click **Save Risk Settings**
5. You'll receive a success notification when settings are saved

Settings are validated before saving and you'll see error messages if values are invalid.

## Data Storage

Data is stored in `~/.local/share/crypto_trader/`:

- `trading.db` - Legacy file (removed)
- `historical/` - Historical market data
- `logs/` - Application logs

### Database Options

### Database Options
 
 **PostgreSQL (Required)**:
 - All production data is stored in PostgreSQL.
 - Requires a running PostgreSQL instance.
 - Configure connection in `config.yaml`.
 
 **SQLite (Internal)**:
 - Used internally for unit testing only.
 - No configuration required for users.

## Redis Configuration

Redis is used for distributed state management, ensuring autopilot state persists across restarts and preventing duplicate instances.

### Default Settings

Redis settings are configured in `config.yaml`:

```yaml
redis:
  host: "127.0.0.1"
  port: 6379
  db: 0
  password: null  # Set if Redis requires authentication
  socket_connect_timeout: 5
```

### Environment Variables

You can also configure Redis via environment variables:
- `REDIS_HOST` - Redis server hostname
- `REDIS_PORT` - Redis server port
- `REDIS_PASSWORD` - Redis password (if required)

### Verifying Redis Connection

Run the verification script:
```bash
python scripts/verify_redis.py
```

## Celery Configuration (Background Tasks)

Celery handles long-running tasks like ML model training in the background.

### Starting the Worker

```bash
# From project root (with virtualenv activated)
celery -A src.worker.app worker --loglevel=info
```

Or use the helper script:
```bash
./scripts/start_all.sh
```

### Task Queues

Tasks are routed to specific queues:
- `ml_training` - Model training and data bootstrapping
- `reporting` - Report generation
- `celery` - Default queue

## ML Model Training Configuration

Configure how the autopilot ML model is trained via the Settings page.

### Training Data Configuration

Navigate to **Settings** > **Autopilot** tab to access these settings:

| Setting | Description | Recommended |
|---------|-------------|-------------|
| **Historical Data (days)** | Amount of historical data to fetch for training | 90-365 days |
| **Timeframe** | OHLCV candle timeframe for training data | 1h (balanced) |
| **Min Samples per Strategy** | Minimum samples required per strategy | 30+ |
| **Training Symbols** | Cryptocurrencies to include in training | 5-10 major coins |

### Setting Training Symbols

1. Navigate to **Settings** > **Autopilot** tab
2. In the **Training Data Configuration** section, find **Training Symbols**
3. Click to add/remove symbols from the multi-select dropdown
4. Common symbols: BTC/USD, ETH/USD, SOL/USD, XRP/USD, ADA/USD
5. Click **Save Bootstrap Config** to persist your changes

### Triggering Model Training

1. Click **Retrain Model** button in the Model Management section
2. A progress bar will appear showing:
   - Data fetching progress (20-60%)
   - Training progress (70-100%)
3. Training typically takes 30-60 seconds depending on data volume
4. Upon completion, the model card updates to show:
   - "Global Model Trained" badge
   - Number of strategies and features
   - Training accuracy
   - Which symbols the model was trained on

### Training Progress Persistence

Training progress persists across page navigation:
- If you navigate away during training, progress resumes on return
- The training task ID is stored in browser localStorage
- On task completion or failure, the task ID is cleared

### Model Management

- **Retrain Model**: Triggers retraining with current configuration
- **Reset Model**: Deletes all saved model files (confirmation required)

### Troubleshooting Training

If training fails:
1. Check that Redis and Celery are running
2. Review Celery worker logs: `tail -f celery.log`
3. Ensure sufficient historical data is available
4. Try reducing the number of training symbols
5. Check backend logs for error details

### Monitoring Tasks

The API provides endpoints to monitor background tasks:
- `POST /api/autopilot/intelligent/retrain` - Starts training, returns task ID
- `GET /api/autopilot/tasks/{task_id}` - Check task status

## Environment Variables

You can override configuration using environment variables:

- `CRYPTO_TRADER_CONFIG_DIR` - Custom config directory
- `CRYPTO_TRADER_DATA_DIR` - Custom data directory
- `CRYPTO_TRADER_LOG_DIR` - Custom log directory
- `DB_PASSWORD` - Database password (for PostgreSQL)
- `REDIS_HOST` - Redis server hostname
- `REDIS_PORT` - Redis server port
- `REDIS_PASSWORD` - Redis password

## Backup and Restore

### Backup

1. Stop the application and Celery workers
2. Backup PostgreSQL database
3. Copy the entire `~/.local/share/crypto_trader/` directory
4. Copy `~/.config/crypto_trader/` directory
5. (Optional) Export Redis data with `redis-cli SAVE`

### Restore

1. Stop the application
2. Restore PostgreSQL database
3. Replace the directories with your backup
4. Restart Redis, Celery worker, and application