StarPunk/docs/decisions/ADR-006-python-virtual-environment-uv.md

# ADR-006: Python Virtual Environment Management with uv

## Status
Accepted

## Context
StarPunk is a Python-based web application that requires dependency management and virtual environment isolation. Developer agents (AI assistants like Claude Code) need clear, unambiguous standards for:

- Creating and managing Python virtual environments
- Installing and tracking dependencies
- Ensuring reproducible development environments
- Avoiding common pitfalls (polluting global Python, dependency conflicts)
- Maintaining consistency across development and deployment

Traditional tools (pip, venv, virtualenv, poetry, pipenv) have various limitations:
- **pip + venv**: Slow dependency resolution, manual requirements.txt management
- **poetry**: Complex configuration, slow, dependency lock issues
- **pipenv**: Abandoned maintenance, slow performance
- **conda**: Heavyweight, non-standard for web development

We need a tool that is fast, simple, and provides excellent developer experience while maintaining compatibility with standard Python packaging.

## Decision
Use **uv** for all Python virtual environment and dependency management in StarPunk.

uv will be the standard tool for:
- Creating virtual environments
- Installing dependencies
- Managing requirements
- Running Python commands in the virtual environment
- Synchronizing dependencies

## Rationale

### Simplicity Score: 10/10
- Single tool for all environment management
- Simple command syntax (uv venv, uv pip install, uv run)
- Drop-in replacement for pip and virtualenv
- No complex configuration files
- Works with standard requirements.txt
- Written in Rust, installed as single binary

### Performance Score: 10/10
- 10-100x faster than pip for dependency resolution
- Parallel downloads and installations
- Efficient caching mechanism
- Near-instant virtual environment creation
- Minimal overhead for running commands

### Fitness Score: 9/10
- Perfect for small to medium Python projects
- Excellent for single-developer projects
- Works with standard Python packaging (PEP 517/518)
- Compatible with requirements.txt workflow
- Supports editable installs for development
- Works seamlessly with Flask and all our dependencies

### Maintenance Score: 9/10
- Actively developed by Astral (creators of ruff)
- Strong community adoption
- Excellent documentation
- Regular updates and improvements
- Modern codebase (Rust)
- Backed by funding and commercial support

### Standards Compliance: Pass
- Full compatibility with pip
- Works with PyPI and all standard package indices
- Supports PEP 440 version specifiers
- Compatible with requirements.txt format
- Works with standard Python virtual environments
- No proprietary lock files (uses standard formats)

## Implementation Details

### 1. Installation Standards

#### System-Level uv Installation
Developer agents MUST ensure uv is installed before creating environments:

```bash
# Check if uv is installed
which uv

# If not installed, install via pip (fallback)
pip install uv

# Or install via official installer (preferred on Linux/macOS)
curl -LsSf https://astral.sh/uv/install.sh | sh
```

#### Verification
```bash
# Verify uv installation
uv --version
# Expected output: uv 0.x.x (or newer)
```

### 2. Virtual Environment Creation Standards

#### Location and Naming
- **Standard location**: `/home/phil/Projects/starpunk/.venv`
- **Name**: Always use `.venv` (hidden directory)
- **DO NOT** use: `venv`, `env`, `virtualenv`, or custom names

#### Creation Command
```bash
# Create virtual environment with uv
cd /home/phil/Projects/starpunk
uv venv .venv

# Specify Python version (recommended)
uv venv .venv --python 3.11
```

#### Post-Creation Verification
```bash
# Verify .venv directory exists
ls -la /home/phil/Projects/starpunk/.venv

# Verify Python executable
/home/phil/Projects/starpunk/.venv/bin/python --version
```

### 3. Dependency Installation Standards

#### Using requirements.txt (Primary Method)
```bash
# Install all dependencies from requirements.txt
uv pip install -r /home/phil/Projects/starpunk/requirements.txt

# Verify installation
uv pip list
```

#### Installing Individual Packages
```bash
# Install a single package
uv pip install flask==3.0.*

# Install multiple packages
uv pip install flask markdown feedgen
```

#### Development Dependencies
```bash
# Install dev dependencies (if requirements-dev.txt exists)
uv pip install -r /home/phil/Projects/starpunk/requirements-dev.txt
```

### 4. Running Commands in Virtual Environment

#### Using uv run (Recommended)
```bash
# Run Python script
uv run /home/phil/Projects/starpunk/.venv/bin/python script.py

# Run Flask development server
uv run /home/phil/Projects/starpunk/.venv/bin/flask run

# Run pytest
uv run /home/phil/Projects/starpunk/.venv/bin/pytest

# Run Python REPL
uv run /home/phil/Projects/starpunk/.venv/bin/python
```

#### Direct Execution (Alternative)
```bash
# Execute using absolute path to venv Python
/home/phil/Projects/starpunk/.venv/bin/python script.py
/home/phil/Projects/starpunk/.venv/bin/flask run
/home/phil/Projects/starpunk/.venv/bin/pytest
```

### 5. Dependency Tracking Standards

#### Generating requirements.txt
```bash
# Freeze current environment to requirements.txt
uv pip freeze > /home/phil/Projects/starpunk/requirements.txt

# Freeze with sorted output for consistency
uv pip freeze | sort > /home/phil/Projects/starpunk/requirements.txt
```

#### Adding New Dependencies
When adding a new dependency:
1. Install the package: `uv pip install package-name`
2. Update requirements.txt: `uv pip freeze | sort > requirements.txt`
3. Verify installation: `uv pip list | grep package-name`

### 6. Environment Updates and Maintenance

#### Updating Dependencies
```bash
# Update a specific package
uv pip install --upgrade flask

# Update all packages (use with caution)
uv pip install --upgrade -r requirements.txt

# Regenerate requirements.txt after updates
uv pip freeze | sort > requirements.txt
```

#### Cleaning and Rebuilding
```bash
# Remove virtual environment
rm -rf /home/phil/Projects/starpunk/.venv

# Recreate from scratch
uv venv .venv --python 3.11
uv pip install -r requirements.txt
```

## Developer Agent Standards

### Critical Rules for AI Assistants

#### Rule 1: ALWAYS Check for Existing Virtual Environment
Before creating a new virtual environment, ALWAYS check:

```bash
# Check if .venv exists
if [ -d "/home/phil/Projects/starpunk/.venv" ]; then
  echo "Virtual environment exists"
  /home/phil/Projects/starpunk/.venv/bin/python --version
else
  echo "Virtual environment does not exist"
fi
```

**NEVER** create a new virtual environment if one already exists without explicit user permission.

#### Rule 2: ALWAYS Use Absolute Paths
Agent threads reset cwd between bash calls. ALWAYS use absolute paths:

**CORRECT:**
```bash
uv venv /home/phil/Projects/starpunk/.venv
/home/phil/Projects/starpunk/.venv/bin/python script.py
uv pip install -r /home/phil/Projects/starpunk/requirements.txt
```

**INCORRECT:**
```bash
uv venv .venv  # Relative path - WRONG
./venv/bin/python script.py  # Relative path - WRONG
uv pip install -r requirements.txt  # Relative path - WRONG
```

#### Rule 3: Verify Before Executing
Before running Python commands, verify the virtual environment:

```bash
# Verification checklist
[ -d "/home/phil/Projects/starpunk/.venv" ] && echo "✓ venv exists" || echo "✗ venv missing"
[ -f "/home/phil/Projects/starpunk/.venv/bin/python" ] && echo "✓ Python exists" || echo "✗ Python missing"
/home/phil/Projects/starpunk/.venv/bin/python --version
```

#### Rule 4: Handle Errors Gracefully
If virtual environment operations fail:

1. **Check uv installation**: `which uv`
2. **Check Python version**: `python3 --version`
3. **Check disk space**: `df -h /home/phil/Projects/starpunk`
4. **Report specific error** to user with context
5. **DO NOT** silently continue with global Python

#### Rule 5: Never Modify Global Python
**NEVER** run these commands:
```bash
# FORBIDDEN - modifies global Python
pip install package
python3 -m pip install package
sudo pip install package
```

**ALWAYS** use virtual environment:
```bash
# CORRECT - uses virtual environment
uv pip install package
/home/phil/Projects/starpunk/.venv/bin/pip install package
```

#### Rule 6: Track Dependency Changes
After installing or removing packages:

1. Update requirements.txt: `uv pip freeze | sort > requirements.txt`
2. Verify changes: `git diff requirements.txt` (if applicable)
3. Inform user of changes made

### Standard Agent Workflow

#### Scenario 1: First-Time Setup
```bash
# 1. Check if venv exists
if [ ! -d "/home/phil/Projects/starpunk/.venv" ]; then
  echo "Creating virtual environment..."
  uv venv /home/phil/Projects/starpunk/.venv --python 3.11
fi

# 2. Verify creation
/home/phil/Projects/starpunk/.venv/bin/python --version

# 3. Install dependencies (if requirements.txt exists)
if [ -f "/home/phil/Projects/starpunk/requirements.txt" ]; then
  uv pip install -r /home/phil/Projects/starpunk/requirements.txt
fi

# 4. Verify installation
uv pip list
```

#### Scenario 2: Running Development Server
```bash
# 1. Verify venv exists
[ -d "/home/phil/Projects/starpunk/.venv" ] || echo "ERROR: Virtual environment missing"

# 2. Verify Flask is installed
/home/phil/Projects/starpunk/.venv/bin/python -c "import flask; print(flask.__version__)"

# 3. Run Flask development server
/home/phil/Projects/starpunk/.venv/bin/flask --app /home/phil/Projects/starpunk/app.py run
```

#### Scenario 3: Adding New Dependency
```bash
# 1. Install package
uv pip install httpx

# 2. Verify installation
uv pip show httpx

# 3. Update requirements.txt
uv pip freeze | sort > /home/phil/Projects/starpunk/requirements.txt

# 4. Confirm to user
echo "Added httpx to project dependencies"
```

#### Scenario 4: Running Tests
```bash
# 1. Verify pytest is installed
/home/phil/Projects/starpunk/.venv/bin/python -c "import pytest; print(pytest.__version__)"

# 2. Run tests
/home/phil/Projects/starpunk/.venv/bin/pytest /home/phil/Projects/starpunk/tests/

# 3. Run tests with coverage (if pytest-cov installed)
/home/phil/Projects/starpunk/.venv/bin/pytest --cov=/home/phil/Projects/starpunk/src /home/phil/Projects/starpunk/tests/
```

## Project-Specific Standards

### Python Version Requirements
- **Minimum**: Python 3.11
- **Recommended**: Python 3.11 or 3.12
- **Rationale**: Modern Python features, improved performance, security updates

### Directory Structure
```
/home/phil/Projects/starpunk/
├── .venv/                 # Virtual environment (NEVER commit)
├── requirements.txt       # Production dependencies
├── requirements-dev.txt   # Development dependencies (optional)
├── src/                   # Application source code
├── tests/                 # Test files
└── docs/                  # Documentation
```

### .gitignore Requirements
The following MUST be in .gitignore:
```
# Virtual Environment
.venv/
venv/
env/
ENV/

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
```

### Environment Variables
Use python-dotenv for configuration:
```bash
# .env file (NEVER commit to git)
FLASK_APP=app.py
FLASK_ENV=development
SECRET_KEY=your-secret-key
DATABASE_PATH=/home/phil/Projects/starpunk/data/starpunk.db
```

Load in application:
```python
from dotenv import load_dotenv
load_dotenv()
```

### Requirements.txt Format
Follow these conventions:
```
# Requirements.txt - StarPunk Dependencies
# Generated: 2025-11-18

# Web Framework
flask==3.0.*

# Content Processing
markdown==3.5.*

# Feed Generation
feedgen==1.0.*

# HTTP Client
httpx==0.27.*

# Configuration
python-dotenv==1.0.*
```

## Consequences

### Positive
- **10-100x faster** dependency resolution and installation
- **Consistent environments** across development and deployment
- **Simple workflow** - one tool for all Python environment tasks
- **No activation required** - uv run handles environment automatically
- **Excellent caching** - faster subsequent installations
- **Standard compatibility** - works with all existing Python tools
- **Clear agent guidelines** - reduces errors in automated workflows
- **Isolated dependencies** - no conflicts with system Python

### Negative
- **Additional tool dependency** - requires uv installation
- **Less familiar** - newer tool, smaller community than pip
- **Rust dependency** - uv is written in Rust (but distributed as binary)

### Mitigation
- uv is easy to install (single binary, no compilation needed)
- uv is pip-compatible (drop-in replacement)
- Fallback to pip + venv is always possible
- Documentation and agent standards make adoption easy
- Active development and growing adoption reduce risk

### Trade-offs Accepted
- **uv vs poetry**: We chose simplicity over advanced features
- **uv vs pipenv**: We chose active maintenance and speed
- **uv vs pip**: We chose performance over ubiquity
- **Single tool complexity**: Better than managing multiple tools

## Verification Checklist

Before considering the environment correctly set up, verify:

- [ ] uv is installed and accessible: `which uv`
- [ ] Virtual environment exists: `ls -la /home/phil/Projects/starpunk/.venv`
- [ ] Python version is 3.11+: `/home/phil/Projects/starpunk/.venv/bin/python --version`
- [ ] Dependencies installed: `uv pip list` shows Flask, markdown, feedgen, httpx
- [ ] requirements.txt exists and is up to date
- [ ] .venv is in .gitignore
- [ ] Flask runs: `/home/phil/Projects/starpunk/.venv/bin/flask --version`

## Integration with Development Workflow

### Running Flask Application
```bash
# Development server
/home/phil/Projects/starpunk/.venv/bin/flask --app app.py run --debug

# Production server (using gunicorn)
/home/phil/Projects/starpunk/.venv/bin/gunicorn app:app
```

### Running Tests
```bash
# All tests
/home/phil/Projects/starpunk/.venv/bin/pytest

# Specific test file
/home/phil/Projects/starpunk/.venv/bin/pytest tests/test_api.py

# With coverage
/home/phil/Projects/starpunk/.venv/bin/pytest --cov=src tests/
```

### Code Quality Tools
```bash
# Format code with black
/home/phil/Projects/starpunk/.venv/bin/black src/

# Lint with flake8
/home/phil/Projects/starpunk/.venv/bin/flake8 src/

# Type checking with mypy (if added)
/home/phil/Projects/starpunk/.venv/bin/mypy src/
```

## Alternatives Considered

### pip + venv (Rejected)
- **Simplicity**: 8/10 - Standard Python tools, well-known
- **Performance**: 4/10 - Very slow dependency resolution
- **Fitness**: 7/10 - Works but painful for larger dependency trees
- **Maintenance**: 10/10 - Built into Python, always maintained
- **Verdict**: Too slow, poor developer experience, but acceptable fallback

### poetry (Rejected)
- **Simplicity**: 5/10 - Complex pyproject.toml, lock file management
- **Performance**: 5/10 - Slow dependency resolution
- **Fitness**: 6/10 - Overkill for simple project, lock files add complexity
- **Maintenance**: 7/10 - Maintained but has had reliability issues
- **Verdict**: Too complex for "minimal code" philosophy

### pipenv (Rejected)
- **Simplicity**: 6/10 - Simpler than poetry, but still adds abstraction
- **Performance**: 4/10 - Known performance issues
- **Fitness**: 5/10 - Previously recommended, now effectively abandoned
- **Maintenance**: 2/10 - Minimal maintenance, community has moved on
- **Verdict**: Dead project, poor performance

### conda (Rejected)
- **Simplicity**: 3/10 - Heavy, complex environment management
- **Performance**: 5/10 - Slower than uv, larger downloads
- **Fitness**: 2/10 - Designed for data science, not web development
- **Maintenance**: 9/10 - Well maintained, large ecosystem
- **Verdict**: Wrong tool for web application development

### PDM (Considered)
- **Simplicity**: 7/10 - Modern, PEP 582 support
- **Performance**: 8/10 - Fast, but not as fast as uv
- **Fitness**: 7/10 - Good for modern Python projects
- **Maintenance**: 8/10 - Actively maintained, growing community
- **Verdict**: Good alternative, but uv is faster and simpler

## References
- uv Documentation: https://docs.astral.sh/uv/
- uv GitHub: https://github.com/astral-sh/uv
- Python Virtual Environments: https://docs.python.org/3/library/venv.html
- PEP 405 (Python Virtual Environments): https://peps.python.org/pep-0405/
- requirements.txt format: https://pip.pypa.io/en/stable/reference/requirements-file-format/
- Astral (uv creators): https://astral.sh/

## Change Log
- 2025-11-18: Initial version - Established uv as standard tool for StarPunk Python environment management