Local changes: Updated model training, removed debug instrumentation, and configuration improvements

2025-12-26 01:15:43 -05:00
commit cc60da49e7
388 changed files with 57127 additions and 0 deletions
--- a/docs/developer/setup.md
+++ b/docs/developer/setup.md
@@ -0,0 +1,315 @@
+# Development Environment Setup
+
+This guide will help you set up a development environment for Crypto Trader.
+
+## Prerequisites
+
+- Python 3.11 or higher
+- Node.js 18 or higher
+- Git
+- Virtual environment tool (venv or virtualenv)
+- Code editor (VS Code, PyCharm, etc.)
+- PostgreSQL 14 or higher
+- Redis 5.0 or higher
+
+## Initial Setup
+
+### 1. Clone the Repository
+
+```bash
+git clone <repository-url>
+cd crypto_trader
+```
+
+### 2. Create Virtual Environment
+
+```bash
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+### 3. Install Python Dependencies
+
+```bash
+# Install main dependencies
+pip install -r requirements.txt
+
+# Install development dependencies
+pip install -r tests/requirements.txt
+
+# Install documentation dependencies
+pip install -r docs/requirements.txt
+```
+
+### 4. Install Frontend Dependencies
+
+```bash
+cd frontend
+npm install
+cd ..
+```
+
+### 5. Install Pre-commit Hooks (Optional)
+
+```bash
+pip install pre-commit
+pre-commit install
+```
+
+## Development Tools
+
+### Recommended IDE Setup
+
+**VS Code**:
+- Python extension
+- Pylance for type checking
+- Python Test Explorer
+- ESLint and Prettier for frontend
+- YAML extension
+
+**PyCharm**:
+- Configure Python interpreter
+- Set up test runner
+- Configure code style
+- Enable JavaScript/TypeScript support
+
+### Code Quality Tools
+
+```bash
+# Install linting and formatting tools
+pip install black flake8 mypy pylint
+
+# Format code
+black src/ tests/
+
+# Lint code
+flake8 src/ tests/
+
+# Type checking
+mypy src/
+```
+
+## Database Setup
+
+### PostgreSQL (Required)
+
+You must have a PostgreSQL instance running for development.
+
+```bash
+# Install PostgreSQL
+# Create development database
+createdb crypto_trader_dev
+
+# Update config.yaml or set env var
+# DATABASE_URL=postgresql+asyncpg://user:password@localhost/crypto_trader_dev
+```
+
+### SQLite (Internal)
+
+Used internally for unit tests (in-memory) only. No setup required.
+
+## Redis Setup
+
+Redis is required for distributed state management and Celery background tasks (e.g., ML model retraining).
+
+```bash
+# Install Redis (Ubuntu/Debian)
+sudo apt-get install redis-server
+```
+
+**Starting Redis**:
+
+```bash
+# Option 1: Using system service (requires sudo)
+sudo service redis-server start
+
+# Option 2: Direct daemon mode (for containers/restricted environments)
+redis-server --daemonize yes
+
+# Verify
+redis-cli ping  # Should return PONG
+```
+
+> **Note**: In containerized environments (Toolbox, Distrobox, Docker, etc.) where `sudo` is not available, use the direct daemon mode option. If you see "Connection refused" errors when using features like "Retrain Model", Redis is likely not running.
+
+### Default Configuration
+
+Redis defaults to `localhost:6379`. Override in `config.yaml`:
+
+```yaml
+redis:
+  host: "127.0.0.1"
+  port: 6379
+  db: 0
+```
+
+## Running the Application
+
+### Start All Services (Recommended)
+
+Use the helper script to start all services:
+
+```bash
+./scripts/start_all.sh
+```
+
+### Start Services Manually
+
+```bash
+# 1. Start Redis
+# Use sudo if available, otherwise direct daemon mode
+sudo service redis-server start  # OR: redis-server --daemonize yes
+
+# 2. Start Celery Worker (background tasks)
+celery -A src.worker.app worker --loglevel=info &
+
+# 3. Start Backend API
+uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000 &
+
+# 4. Start Frontend
+cd frontend && npm run dev
+```
+
+### Access Points
+
+- **Frontend**: http://localhost:3000
+- **Backend API**: http://localhost:8000
+- **API Docs**: http://localhost:8000/docs
+
+### Verify Redis/Celery Integration
+
+```bash
+python scripts/verify_redis.py
+```
+
+## Running Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=src --cov-report=html
+
+# Run specific test file
+pytest tests/unit/core/test_redis.py
+
+# Run Redis/Celery tests
+pytest tests/unit/core/test_redis.py tests/unit/worker/test_tasks.py -v
+
+# Run with verbose output
+pytest -v
+```
+
+## Project Structure
+
+```
+crypto_trader/
+├── src/                    # Backend source code
+│   ├── autopilot/          # Intelligent autopilot
+│   ├── core/               # Core utilities (config, redis, logging)
+│   ├── strategies/         # Trading strategies
+│   ├── trading/            # Trading engine
+│   └── worker/             # Celery tasks
+├── backend/                # FastAPI application
+│   ├── api/                # API endpoints
+│   └── main.py             # Application entry point
+├── frontend/               # React frontend
+│   ├── src/                # React source
+│   └── package.json        # Frontend dependencies
+├── tests/                  # Test suite
+├── scripts/                # Utility scripts
+│   ├── start_all.sh        # Start all services
+│   └── verify_redis.py     # Verify Redis/Celery
+├── docs/                   # Documentation
+└── requirements.txt        # Python dependencies
+```
+
+## Common Development Tasks
+
+### Adding a New Celery Task
+
+1. Add task function in `src/worker/tasks.py`
+2. Configure task routing in `src/worker/app.py` (optional)
+3. Create API endpoint in `backend/api/`
+4. Write unit tests in `tests/unit/worker/`
+
+### Adding a New API Endpoint
+
+1. Create/update router in `backend/api/`
+2. Register router in `backend/main.py`
+3. Add frontend API client in `frontend/src/api/`
+4. Write tests
+
+### Running Documentation
+
+```bash
+cd docs/api
+make html
+# Open build/html/index.html
+```
+
+### Building AppImage
+
+```bash
+./packaging/build_appimage.sh
+```
+
+## Debugging
+
+### Enable Debug Logging
+
+Set in `config.yaml`:
+```yaml
+logging:
+  level: DEBUG
+```
+
+### Using Debugger
+
+```bash
+# VS Code: Set breakpoints and use debugger
+# PyCharm: Configure debug configuration
+# Command line: Use pdb
+python -m pdb -c continue backend/main.py
+```
+
+### Checking Celery Tasks
+
+```bash
+# Monitor tasks
+celery -A src.worker.app events
+
+# Inspect active tasks
+celery -A src.worker.app inspect active
+```
+
+## Troubleshooting
+
+**Import errors?**
+- Verify virtual environment is activated
+- Check PYTHONPATH
+- Reinstall dependencies
+
+**Redis connection failed / "Connection refused" error?**
+- Ensure Redis is running: `redis-cli ping` (should return `PONG`)
+- Start Redis if not running:
+  - With sudo: `sudo service redis-server start`
+  - Without sudo: `redis-server --daemonize yes`
+- Check host/port configuration in `config.yaml`
+- Verify firewall rules
+
+**Celery tasks not executing?**
+- Ensure worker is running: `ps aux | grep celery`
+- Check worker logs: `tail -f celery.log`
+- Verify Redis is accessible
+
+**Tests failing?**
+- Check test database setup
+- Verify test fixtures
+- Review test logs
+
+**Frontend not connecting?**
+- Check API is running on port 8000
+- Verify CORS settings in backend
+- Check browser console for errors