Local changes: Updated model training, removed debug instrumentation, and configuration improvements
This commit is contained in:
kfox
243
docs/developer/pricing_providers.md
Normal file
243
docs/developer/pricing_providers.md
Normal file
@@ -0,0 +1,243 @@
|
||||
# Pricing Provider Development Guide
|
||||
|
||||
This guide explains how to add new pricing data providers to the Crypto Trader system.
|
||||
|
||||
## Overview
|
||||
|
||||
Pricing providers are responsible for fetching market data (prices, OHLCV candlestick data) without requiring API keys. They differ from exchange adapters, which handle trading operations and require authentication.
|
||||
|
||||
The system uses a multi-tier provider strategy:
|
||||
- **Primary Providers**: CCXT-based providers (Kraken, Coinbase, Binance)
|
||||
- **Fallback Provider**: CoinGecko API
|
||||
|
||||
## Provider Interface
|
||||
|
||||
All pricing providers must implement the `BasePricingProvider` interface, located in `src/data/providers/base_provider.py`.
|
||||
|
||||
### Required Methods
|
||||
|
||||
1. **`name` (property)**: Return the provider's display name
|
||||
2. **`supports_websocket` (property)**: Return True if the provider supports WebSocket connections
|
||||
3. **`connect()`**: Establish connection to the provider, return True if successful
|
||||
4. **`disconnect()`**: Close connection and clean up resources
|
||||
5. **`get_ticker(symbol: str) -> Dict`**: Get current ticker data for a symbol
|
||||
6. **`get_ohlcv(symbol, timeframe, since, limit) -> List[List]`**: Get historical OHLCV data
|
||||
7. **`subscribe_ticker(symbol, callback) -> bool`**: Subscribe to real-time ticker updates
|
||||
|
||||
### Ticker Data Format
|
||||
|
||||
The `get_ticker()` method should return a dictionary with the following keys:
|
||||
|
||||
```python
|
||||
{
|
||||
'symbol': str, # Trading pair (e.g., 'BTC/USD')
|
||||
'bid': Decimal, # Best bid price
|
||||
'ask': Decimal, # Best ask price
|
||||
'last': Decimal, # Last traded price
|
||||
'high': Decimal, # 24h high
|
||||
'low': Decimal, # 24h low
|
||||
'volume': Decimal, # 24h volume
|
||||
'timestamp': int, # Unix timestamp in milliseconds
|
||||
}
|
||||
```
|
||||
|
||||
### OHLCV Data Format
|
||||
|
||||
The `get_ohlcv()` method should return a list of candles, where each candle is:
|
||||
|
||||
```python
|
||||
[timestamp_ms, open, high, low, close, volume]
|
||||
```
|
||||
|
||||
All values should be numeric (float or Decimal).
|
||||
|
||||
## Creating a New Provider
|
||||
|
||||
### Step 1: Create Provider Class
|
||||
|
||||
Create a new file in `src/data/providers/` (e.g., `my_provider.py`):
|
||||
|
||||
```python
|
||||
"""My custom pricing provider."""
|
||||
|
||||
from decimal import Decimal
|
||||
from typing import Dict, List, Optional, Any, Callable
|
||||
from datetime import datetime
|
||||
from .base_provider import BasePricingProvider
|
||||
from ...core.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class MyProvider(BasePricingProvider):
|
||||
"""My custom pricing provider implementation."""
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "MyProvider"
|
||||
|
||||
@property
|
||||
def supports_websocket(self) -> bool:
|
||||
return False # Set to True if WebSocket supported
|
||||
|
||||
def connect(self) -> bool:
|
||||
"""Connect to provider."""
|
||||
try:
|
||||
# Initialize connection
|
||||
self._connected = True
|
||||
logger.info(f"Connected to {self.name}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to connect: {e}")
|
||||
self._connected = False
|
||||
return False
|
||||
|
||||
def disconnect(self):
|
||||
"""Disconnect from provider."""
|
||||
self._connected = False
|
||||
logger.info(f"Disconnected from {self.name}")
|
||||
|
||||
def get_ticker(self, symbol: str) -> Dict[str, Any]:
|
||||
"""Get current ticker data."""
|
||||
# Implementation here
|
||||
pass
|
||||
|
||||
def get_ohlcv(
|
||||
self,
|
||||
symbol: str,
|
||||
timeframe: str = '1h',
|
||||
since: Optional[datetime] = None,
|
||||
limit: int = 100
|
||||
) -> List[List]:
|
||||
"""Get OHLCV data."""
|
||||
# Implementation here
|
||||
pass
|
||||
|
||||
def subscribe_ticker(self, symbol: str, callback: Callable) -> bool:
|
||||
"""Subscribe to ticker updates."""
|
||||
# Implementation here
|
||||
pass
|
||||
```
|
||||
|
||||
### Step 2: Register Provider
|
||||
|
||||
Update `src/data/providers/__init__.py` to include your provider:
|
||||
|
||||
```python
|
||||
from .my_provider import MyProvider
|
||||
|
||||
__all__ = [..., 'MyProvider']
|
||||
```
|
||||
|
||||
### Step 3: Add to Pricing Service
|
||||
|
||||
Update `src/data/pricing_service.py` to include your provider in the initialization:
|
||||
|
||||
```python
|
||||
# Add to _initialize_providers method
|
||||
try:
|
||||
my_provider = MyProvider()
|
||||
if my_provider.connect():
|
||||
self._providers[my_provider.name] = my_provider
|
||||
self._provider_priority.append(my_provider.name)
|
||||
except Exception as e:
|
||||
logger.error(f"Error initializing MyProvider: {e}")
|
||||
```
|
||||
|
||||
### Step 4: Add Configuration
|
||||
|
||||
Update `src/core/config.py` to include configuration options for your provider:
|
||||
|
||||
```python
|
||||
"data_providers": {
|
||||
"primary": [
|
||||
# ... existing providers ...
|
||||
{"name": "my_provider", "enabled": True, "priority": 4},
|
||||
],
|
||||
# ...
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Always catch exceptions in provider methods
|
||||
- Return empty data structures (`{}` or `[]`) on error rather than raising
|
||||
- Log errors with appropriate detail level
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
- Respect API rate limits
|
||||
- Implement appropriate delays between requests
|
||||
- Use exponential backoff for retries
|
||||
|
||||
### Symbol Normalization
|
||||
|
||||
- Override `normalize_symbol()` if your provider uses different symbol formats
|
||||
- Handle common variations (BTC/USD vs BTC-USD vs BTCUSD)
|
||||
|
||||
### Caching
|
||||
|
||||
- The pricing service handles caching automatically
|
||||
- Focus on providing fresh data from the API
|
||||
- Don't implement your own caching layer
|
||||
|
||||
### Testing
|
||||
|
||||
Create unit tests in `tests/unit/data/providers/test_my_provider.py`:
|
||||
|
||||
```python
|
||||
"""Unit tests for MyProvider."""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from src.data.providers.my_provider import MyProvider
|
||||
|
||||
class TestMyProvider:
|
||||
def test_connect(self):
|
||||
provider = MyProvider()
|
||||
result = provider.connect()
|
||||
assert result is True
|
||||
|
||||
def test_get_ticker(self):
|
||||
provider = MyProvider()
|
||||
provider.connect()
|
||||
ticker = provider.get_ticker("BTC/USD")
|
||||
assert 'last' in ticker
|
||||
assert ticker['last'] > 0
|
||||
```
|
||||
|
||||
## Example: CoinGecko Provider
|
||||
|
||||
See `src/data/providers/coingecko_provider.py` for a complete example of a REST API-based provider.
|
||||
|
||||
## Example: CCXT Provider
|
||||
|
||||
See `src/data/providers/ccxt_provider.py` for an example of a provider that wraps an existing library (CCXT).
|
||||
|
||||
## Health Monitoring
|
||||
|
||||
The pricing service automatically monitors provider health:
|
||||
- Tracks success/failure rates
|
||||
- Measures response times
|
||||
- Implements circuit breaker pattern
|
||||
- Automatically fails over to next provider
|
||||
|
||||
Your provider doesn't need to implement health monitoring - it's handled by the `HealthMonitor` class.
|
||||
|
||||
## Subscriptions
|
||||
|
||||
For real-time updates, implement `subscribe_ticker()`. The service expects:
|
||||
- Subscriptions to be persistent until `unsubscribe_ticker()` is called
|
||||
- Callbacks to be invoked with ticker data dictionaries
|
||||
- Graceful handling of connection failures
|
||||
|
||||
If WebSocket is not supported, use polling with appropriate intervals (typically 1-5 seconds for ticker data).
|
||||
|
||||
## Questions?
|
||||
|
||||
For more information, see:
|
||||
- `src/data/providers/base_provider.py` - Base interface
|
||||
- `src/data/pricing_service.py` - Service implementation
|
||||
- `src/data/health_monitor.py` - Health monitoring
|
||||
Reference in New Issue
Block a user