StarPunk/docs/operations/troubleshooting.md

# StarPunk Troubleshooting Guide

**Version**: 1.1.1
**Last Updated**: 2025-11-25

This guide helps diagnose and resolve common issues with StarPunk.

## Quick Diagnostics

### Check System Health

```bash
# Basic health check
curl http://localhost:5000/health

# Detailed health check (requires authentication)
curl -H "Authorization: Bearer YOUR_TOKEN" \
  http://localhost:5000/health?detailed=true

# Full diagnostics
curl -H "Authorization: Bearer YOUR_TOKEN" \
  http://localhost:5000/admin/health
```

### Check Logs

```bash
# View recent logs
tail -f data/logs/starpunk.log

# Search for errors
grep ERROR data/logs/starpunk.log | tail -20

# Search for warnings
grep WARNING data/logs/starpunk.log | tail -20
```

### Check Database

```bash
# Verify database exists and is accessible
ls -lh data/starpunk.db

# Check database integrity
sqlite3 data/starpunk.db "PRAGMA integrity_check;"

# Check migrations
sqlite3 data/starpunk.db "SELECT * FROM schema_migrations;"
```

## Common Issues

### Application Won't Start

#### Symptom
StarPunk fails to start or crashes immediately.

#### Possible Causes

1. **Missing configuration**
   ```bash
   # Check required environment variables
   echo $SITE_URL
   echo $SITE_NAME
   echo $ADMIN_ME
   ```

   **Solution**: Set all required variables in `.env`:
   ```bash
   SITE_URL=https://your-domain.com/
   SITE_NAME=Your Site Name
   ADMIN_ME=https://your-domain.com/
   ```

2. **Database locked**
   ```bash
   # Check for other processes
   lsof data/starpunk.db
   ```

   **Solution**: Stop other StarPunk instances or wait for lock release

3. **Permission issues**
   ```bash
   # Check permissions
   ls -ld data/
   ls -l data/starpunk.db
   ```

   **Solution**: Fix permissions:
   ```bash
   chmod 755 data/
   chmod 644 data/starpunk.db
   ```

4. **Missing dependencies**
   ```bash
   # Re-sync dependencies
   uv sync
   ```

### Database Connection Errors

#### Symptom
Errors like "database is locked" or "unable to open database file"

#### Solutions

1. **Check database path**
   ```bash
   # Verify DATABASE_PATH in config
   echo $DATABASE_PATH
   ls -l $DATABASE_PATH
   ```

2. **Check file permissions**
   ```bash
   # Database file needs write permission
   chmod 644 data/starpunk.db
   chmod 755 data/
   ```

3. **Check disk space**
   ```bash
   df -h
   ```

4. **Check connection pool**
   ```bash
   # View pool statistics
   curl http://localhost:5000/admin/metrics | jq '.database.pool'
   ```

   If pool is exhausted, increase `DB_POOL_SIZE`:
   ```bash
   export DB_POOL_SIZE=10
   ```

### IndieAuth Login Fails

#### Symptom
Cannot log in to admin interface, redirects fail, or authentication errors.

#### Solutions

1. **Check ADMIN_ME configuration**
   ```bash
   echo $ADMIN_ME
   ```

   Must be a valid URL that matches your identity.

2. **Check IndieAuth endpoints**
   ```bash
   # Verify endpoints are discoverable
   curl -I $ADMIN_ME | grep Link
   ```

   Should show authorization_endpoint and token_endpoint.

3. **Check callback URL**
   - Verify `/auth/callback` is accessible
   - Check for HTTPS in production
   - Verify no trailing slash issues

4. **Check session secret**
   ```bash
   echo $SESSION_SECRET
   ```

   Must be set and persistent across restarts.

### RSS Feed Issues

#### Symptom
Feed not displaying, validation errors, or empty feed.

#### Solutions

1. **Check feed endpoint**
   ```bash
   curl http://localhost:5000/feed.xml | head -50
   ```

2. **Verify published notes**
   ```bash
   sqlite3 data/starpunk.db \
     "SELECT COUNT(*) FROM notes WHERE published=1;"
   ```

3. **Check feed cache**
   ```bash
   # Clear cache by restarting
   # Cache duration controlled by FEED_CACHE_SECONDS
   ```

4. **Validate feed**
   ```bash
   curl http://localhost:5000/feed.xml | \
     xmllint --format - | head -100
   ```

### Search Not Working

#### Symptom
Search returns no results or errors.

#### Solutions

1. **Check FTS5 availability**
   ```bash
   sqlite3 data/starpunk.db \
     "SELECT COUNT(*) FROM notes_fts;"
   ```

2. **Rebuild search index**
   ```bash
   uv run python -c "from starpunk.search import rebuild_fts_index; \
     rebuild_fts_index('data/starpunk.db', 'data')"
   ```

3. **Check for FTS5 support**
   ```bash
   sqlite3 data/starpunk.db \
     "PRAGMA compile_options;" | grep FTS5
   ```

   If not available, StarPunk will fall back to LIKE queries automatically.

### Performance Issues

#### Symptom
Slow response times, high memory usage, or timeouts.

#### Diagnostics

1. **Check performance metrics**
   ```bash
   curl http://localhost:5000/admin/metrics | jq '.performance'
   ```

2. **Check database pool**
   ```bash
   curl http://localhost:5000/admin/metrics | jq '.database.pool'
   ```

3. **Check system resources**
   ```bash
   # Memory usage
   ps aux | grep starpunk

   # Disk usage
   df -h

   # Open files
   lsof -p $(pgrep -f starpunk)
   ```

#### Solutions

1. **Increase connection pool**
   ```bash
   export DB_POOL_SIZE=10
   ```

2. **Adjust metrics sampling**
   ```bash
   # Reduce sampling for high-traffic sites
   export METRICS_SAMPLING_HTTP=0.01  # 1% sampling
   export METRICS_SAMPLING_RENDER=0.01
   ```

3. **Increase cache duration**
   ```bash
   export FEED_CACHE_SECONDS=600  # 10 minutes
   ```

4. **Check slow queries**
   ```bash
   grep "SLOW" data/logs/starpunk.log
   ```

### Log Rotation Not Working

#### Symptom
Log files growing unbounded, disk space issues.

#### Solutions

1. **Check log directory**
   ```bash
   ls -lh data/logs/
   ```

2. **Verify log rotation configuration**
   - RotatingFileHandler configured for 10MB files
   - Keeps 10 backup files
   - Automatic rotation on size limit

3. **Manual log rotation**
   ```bash
   # Backup and truncate
   mv data/logs/starpunk.log data/logs/starpunk.log.old
   touch data/logs/starpunk.log
   chmod 644 data/logs/starpunk.log
   ```

4. **Check permissions**
   ```bash
   ls -l data/logs/
   chmod 755 data/logs/
   chmod 644 data/logs/*.log
   ```

### Metrics Dashboard Not Loading

#### Symptom
Blank dashboard, 404 errors, or JavaScript errors.

#### Solutions

1. **Check authentication**
   - Must be logged in as admin
   - Navigate to `/admin/dashboard`

2. **Check JavaScript console**
   - Open browser developer tools
   - Look for CDN loading errors
   - Verify htmx and Chart.js load

3. **Check network connectivity**
   ```bash
   # Test CDN access
   curl -I https://unpkg.com/htmx.org@1.9.10
   curl -I https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js
   ```

4. **Test metrics endpoint**
   ```bash
   curl http://localhost:5000/admin/metrics
   ```

## Log File Locations

- **Application logs**: `data/logs/starpunk.log`
- **Rotated logs**: `data/logs/starpunk.log.1` through `starpunk.log.10`
- **Container logs**: `podman logs starpunk` or `docker logs starpunk`
- **System logs**: `/var/log/syslog` or `journalctl -u starpunk`

## Health Check Interpretation

### Basic Health (`/health`)

```json
{
  "status": "healthy"
}
```

- **healthy**: All systems operational
- **unhealthy**: Critical issues detected

### Detailed Health (`/health?detailed=true`)

```json
{
  "status": "healthy",
  "version": "1.1.1",
  "checks": {
    "database": {"status": "healthy"},
    "filesystem": {"status": "healthy"},
    "fts_index": {"status": "healthy"}
  }
}
```

Check each component status individually.

### Full Diagnostics (`/admin/health`)

Includes all above plus:
- Performance metrics
- Database pool statistics
- System resource usage
- Error budget status

## Performance Monitoring Tips

### Normal Metrics

- **Database queries**: avg < 50ms
- **HTTP requests**: avg < 200ms
- **Template rendering**: avg < 50ms
- **Pool usage**: < 80% connections active

### Warning Signs

- **Database**: avg > 100ms consistently
- **HTTP**: avg > 500ms
- **Pool**: 100% connections active
- **Memory**: continuous growth

### Metrics Sampling

Adjust sampling rates based on traffic:

```bash
# Low traffic (< 100 req/day)
METRICS_SAMPLING_DATABASE=1.0
METRICS_SAMPLING_HTTP=1.0
METRICS_SAMPLING_RENDER=1.0

# Medium traffic (100-1000 req/day)
METRICS_SAMPLING_DATABASE=1.0
METRICS_SAMPLING_HTTP=0.1
METRICS_SAMPLING_RENDER=0.1

# High traffic (> 1000 req/day)
METRICS_SAMPLING_DATABASE=0.1
METRICS_SAMPLING_HTTP=0.01
METRICS_SAMPLING_RENDER=0.01
```

## Database Pool Issues

### Pool Exhaustion

**Symptom**: "No available connections" errors

**Solution**:
```bash
# Increase pool size
export DB_POOL_SIZE=10

# Or reduce request concurrency
```

### Pool Leaks

**Symptom**: Connections not returned to pool

**Check**:
```bash
curl http://localhost:5000/admin/metrics | \
  jq '.database.pool'
```

Look for high `active_connections` that don't decrease.

**Solution**: Restart application to reset pool

## Getting Help

### Before Filing an Issue

1. Check this troubleshooting guide
2. Review logs for specific errors
3. Run health checks
4. Try with minimal configuration
5. Search existing issues

### Information to Include

When filing an issue, include:

1. **Version**: `uv run python -c "import starpunk; print(starpunk.__version__)"`
2. **Environment**: Development or production
3. **Configuration**: Sanitized `.env` (remove secrets)
4. **Logs**: Recent errors from `data/logs/starpunk.log`
5. **Health check**: Output from `/admin/health`
6. **Steps to reproduce**: Exact commands that trigger the issue

### Debug Mode

Enable verbose logging:

```bash
export LOG_LEVEL=DEBUG
# Restart StarPunk
```

**WARNING**: Debug logs may contain sensitive information. Don't share publicly.

## Emergency Recovery

### Complete Reset (DESTRUCTIVE)

**WARNING**: This deletes all data.

```bash
# Stop StarPunk
sudo systemctl stop starpunk

# Backup everything
cp -r data data.backup.$(date +%Y%m%d)

# Remove database
rm data/starpunk.db

# Remove logs
rm -rf data/logs/

# Restart (will reinitialize)
sudo systemctl start starpunk
```

### Restore from Backup

```bash
# Stop StarPunk
sudo systemctl stop starpunk

# Restore database
cp data.backup/starpunk.db data/

# Restore notes
cp -r data.backup/notes/* data/notes/

# Restart
sudo systemctl start starpunk
```

## Related Documentation

- `/docs/operations/upgrade-to-v1.1.1.md` - Upgrade procedures
- `/docs/operations/performance-tuning.md` - Optimization guide
- `/docs/architecture/overview.md` - System architecture
- `CHANGELOG.md` - Version history and changes