{{ uptime }}
-{{ summary.http.count }}
-{{ summary.http.avg_ms|round(2) }}ms
-{{ current_memory }}MB
-| Time | -Duration | -Query | -
|---|---|---|
| {{ query.timestamp|timeago }} | -{{ query.duration_ms|round(2) }}ms | -{{ query.metadata.query|truncate(100) }} |
-
| Endpoint | -Calls | -Avg (ms) | -P95 (ms) | -P99 (ms) | -
|---|---|---|---|---|
| {{ endpoint }} | -{{ stats.count }} | -{{ stats.avg_ms|round(2) }} | -{{ stats.p95_ms|round(2) }} | -{{ stats.p99_ms|round(2) }} | -
This is XHTML content
-HTML content
", - "date_published": "2024-11-25T12:00:00Z", - "tags": ["tag1", "tag2"] - } - ] -} -``` - -### 2.3 Content Negotiation (1.5 hours) - -**Location**: `starpunk/feed/negotiator.py` - -**Implementation Steps**: - -1. **Create Content Negotiator** - ```python - class FeedNegotiator: - def negotiate(self, accept_header): - # Parse Accept header - # Score each format - # Return best match - ``` - -2. **Parse Accept Header** - - Split on comma - - Extract MIME type - - Parse quality factors (q=) - - Handle wildcards (*/*) - -3. **Score Formats** - - Exact match: 1.0 - - Wildcard match: 0.5 - - Type/* match: 0.7 - - Default RSS: 0.1 - -4. **Format Mapping** - ```python - FORMAT_MIME_TYPES = { - 'rss': ['application/rss+xml', 'application/xml', 'text/xml'], - 'atom': ['application/atom+xml'], - 'json': ['application/json', 'application/feed+json'] - } - ``` - -### 2.4 Feed Validation (1.5 hours) - -**Location**: `starpunk/feed/validators.py` - -**Implementation Steps**: - -1. **Create Validation Framework** - ```python - class FeedValidator(Protocol): - def validate(self, content: str) -> List[ValidationError]: - pass - ``` - -2. **RSS Validator** - - Check required elements - - Verify date formats - - Validate URLs - - Check CDATA escaping - -3. **ATOM Validator** - - Verify namespace - - Check required elements - - Validate RFC 3339 dates - - Verify ID uniqueness - -4. **JSON Feed Validator** - - Validate against schema - - Check required fields - - Verify URL formats - - Validate date strings - -**Validation Levels**: -- ERROR: Feed is invalid -- WARNING: Non-critical issue -- INFO: Suggestion for improvement - -## Phase 3: Feed Enhancements (4 hours) - -### Objective -Add caching, statistics, and operational improvements to the feed system. - -### 3.1 Feed Caching Layer (1.5 hours) - -**Location**: `starpunk/feed/cache.py` - -**Implementation Steps**: - -1. **Create Cache Manager** - ```python - class FeedCache: - def __init__(self, max_size=100, ttl=300): - self.cache = LRU(max_size) - self.ttl = ttl - ``` - -2. **Cache Key Generation** - - Format type - - Item limit - - Content checksum - - Last modified - -3. **Cache Operations** - - Get with TTL check - - Set with expiration - - Invalidate on changes - - Clear entire cache - -4. **Memory Management** - - Monitor cache size - - Implement eviction - - Track hit rates - - Report statistics - -**Cache Strategy**: -```python -def get_or_generate(format, limit): - key = generate_cache_key(format, limit) - cached = cache.get(key) - - if cached and not expired(cached): - metrics.record_cache_hit() - return cached - - content = generate_feed(format, limit) - cache.set(key, content, ttl=300) - metrics.record_cache_miss() - return content -``` - -### 3.2 Statistics Dashboard (1.5 hours) - -**Location**: `starpunk/admin/syndication.py` - -**Template**: `templates/admin/syndication.html` - -**Implementation Steps**: - -1. **Create Dashboard Route** - ```python - @app.route('/admin/syndication') - @require_admin - def syndication_dashboard(): - stats = gather_syndication_stats() - return render_template('admin/syndication.html', stats=stats) - ``` - -2. **Gather Statistics** - - Requests by format (pie chart) - - Cache hit rates (line graph) - - Generation times (histogram) - - Popular user agents (table) - - Recent errors (log) - -3. **Create Dashboard UI** - - Overview cards - - Time series graphs - - Format breakdown - - Performance metrics - - Configuration status - -**Dashboard Sections**: -- Feed Format Usage -- Cache Performance -- Generation Times -- Client Analysis -- Error Log -- Configuration - -### 3.3 OPML Export (1 hour) - -**Location**: `starpunk/feed/opml.py` - -**Implementation Steps**: - -1. **Create OPML Generator** - ```python - def generate_opml(site_config): - # Generate OPML header - # Add feed outlines - # Include metadata - return opml_content - ``` - -2. **OPML Structure** - ```xml - -HTML content
", - "content_text": "Plain text content", - "summary": "Brief summary", - "image": "https://example.com/image.jpg", - "banner_image": "https://example.com/banner.jpg", - "date_published": "2024-11-25T12:00:00Z", - "date_modified": "2024-11-25T13:00:00Z", - "authors": [], - "tags": ["tag1", "tag2"], - "language": "en", - "attachments": [], - "_custom": {} -} -``` - -### Required Item Fields - -| Field | Type | Description | -|-------|------|-------------| -| `id` | String | Unique, stable ID | - -### Optional Item Fields - -| Field | Type | Description | -|-------|------|-------------| -| `url` | String | Item permalink | -| `external_url` | String | Link to external content | -| `title` | String | Item title | -| `content_html` | String | HTML content | -| `content_text` | String | Plain text content | -| `summary` | String | Brief summary | -| `image` | String | Main image URL | -| `banner_image` | String | Wide banner image | -| `date_published` | String | RFC 3339 date | -| `date_modified` | String | RFC 3339 date | -| `authors` | Array | Item authors | -| `tags` | Array | String tags | -| `language` | String | Language code | -| `attachments` | Array | File attachments | - -### Author Object - -```json -{ - "name": "Author Name", - "url": "https://example.com/about", - "avatar": "https://example.com/avatar.jpg" -} -``` - -### Attachment Object - -```json -{ - "url": "https://example.com/file.pdf", - "mime_type": "application/pdf", - "title": "Attachment Title", - "size_in_bytes": 1024000, - "duration_in_seconds": 300 -} -``` - -## Implementation Design - -### JSON Feed Generator Class - -```python -import json -from typing import List, Dict, Any, Iterator -from datetime import datetime, timezone - -class JsonFeedGenerator: - """JSON Feed 1.1 generator with streaming support""" - - def __init__(self, site_url: str, site_name: str, site_description: str, - author_name: str = None, author_url: str = None, author_avatar: str = None): - self.site_url = site_url.rstrip('/') - self.site_name = site_name - self.site_description = site_description - self.author = { - 'name': author_name, - 'url': author_url, - 'avatar': author_avatar - } if author_name else None - - def generate(self, notes: List[Note], limit: int = 50) -> str: - """Generate complete JSON feed - - IMPORTANT: Notes are expected to be in DESC order (newest first) - from the database. This order MUST be preserved in the feed. - """ - feed = self._build_feed_object(notes[:limit]) - return json.dumps(feed, ensure_ascii=False, indent=2) - - def generate_streaming(self, notes: List[Note], limit: int = 50) -> Iterator[str]: - """Generate JSON feed as stream of chunks - - IMPORTANT: Notes are expected to be in DESC order (newest first) - from the database. This order MUST be preserved in the feed. - """ - # Start feed object - yield '{\n' - yield ' "version": "https://jsonfeed.org/version/1.1",\n' - yield f' "title": {json.dumps(self.site_name)},\n' - - # Add optional feed metadata - yield from self._stream_feed_metadata() - - # Start items array - yield ' "items": [\n' - - # Stream items - maintain DESC order (newest first) - # DO NOT reverse! Database order is correct - items = notes[:limit] - for i, note in enumerate(items): - item_json = json.dumps(self._build_item_object(note), indent=4) - # Indent items properly - indented = '\n'.join(' ' + line for line in item_json.split('\n')) - yield indented - - if i < len(items) - 1: - yield ',\n' - else: - yield '\n' - - # Close items array and feed - yield ' ]\n' - yield '}\n' - - def _build_feed_object(self, notes: List[Note]) -> Dict[str, Any]: - """Build complete feed object""" - feed = { - 'version': 'https://jsonfeed.org/version/1.1', - 'title': self.site_name, - 'home_page_url': self.site_url, - 'feed_url': f'{self.site_url}/feed.json', - 'description': self.site_description, - 'items': [self._build_item_object(note) for note in notes] - } - - # Add optional fields - if self.author: - feed['authors'] = [self._clean_author(self.author)] - - feed['language'] = 'en' # Make configurable - - # Add icon/favicon if configured - icon_url = self._get_icon_url() - if icon_url: - feed['icon'] = icon_url - - favicon_url = self._get_favicon_url() - if favicon_url: - feed['favicon'] = favicon_url - - return feed - - def _build_item_object(self, note: Note) -> Dict[str, Any]: - """Build item object from note""" - permalink = f'{self.site_url}{note.permalink}' - - item = { - 'id': permalink, - 'url': permalink, - 'title': note.title or self._format_date_title(note.created_at), - 'date_published': self._format_json_date(note.created_at) - } - - # Add content (prefer HTML) - if note.html: - item['content_html'] = note.html - elif note.content: - item['content_text'] = note.content - - # Add modified date if different - if hasattr(note, 'updated_at') and note.updated_at != note.created_at: - item['date_modified'] = self._format_json_date(note.updated_at) - - # Add summary if available - if hasattr(note, 'summary') and note.summary: - item['summary'] = note.summary - - # Add tags if available - if hasattr(note, 'tags') and note.tags: - item['tags'] = note.tags - - # Add author if different from feed author - if hasattr(note, 'author') and note.author != self.author: - item['authors'] = [self._clean_author(note.author)] - - # Add image if available - image_url = self._extract_image_url(note) - if image_url: - item['image'] = image_url - - # Add custom extensions - item['_starpunk'] = { - 'permalink_path': note.permalink, - 'word_count': len(note.content.split()) if note.content else 0 - } - - return item - - def _clean_author(self, author: Any) -> Dict[str, str]: - """Clean author object for JSON""" - clean = {} - - if isinstance(author, dict): - if author.get('name'): - clean['name'] = author['name'] - if author.get('url'): - clean['url'] = author['url'] - if author.get('avatar'): - clean['avatar'] = author['avatar'] - elif hasattr(author, 'name'): - clean['name'] = author.name - if hasattr(author, 'url'): - clean['url'] = author.url - if hasattr(author, 'avatar'): - clean['avatar'] = author.avatar - else: - clean['name'] = str(author) - - return clean - - def _format_json_date(self, dt: datetime) -> str: - """Format datetime to RFC 3339 for JSON Feed - - Format: 2024-11-25T12:00:00Z or 2024-11-25T12:00:00-05:00 - """ - if dt.tzinfo is None: - dt = dt.replace(tzinfo=timezone.utc) - - # Use Z for UTC - if dt.tzinfo == timezone.utc: - return dt.strftime('%Y-%m-%dT%H:%M:%SZ') - else: - return dt.isoformat() - - def _extract_image_url(self, note: Note) -> Optional[str]: - """Extract first image URL from note content""" - if not note.html: - return None - - # Simple regex to find first img tag - import re - match = re.search(r'This is my first note with bold text.
", - "summary": "Introduction to my notes", - "image": "https://example.com/images/first.jpg", - "date_published": "2024-11-25T10:00:00Z", - "date_modified": "2024-11-25T10:30:00Z", - "tags": ["personal", "introduction"], - "_starpunk": { - "permalink_path": "/notes/2024/11/25/first-note", - "word_count": 8 - } - }, - { - "id": "https://example.com/notes/2024/11/24/another-note", - "url": "https://example.com/notes/2024/11/24/another-note", - "title": "Another Note", - "content_text": "Plain text content for this note.", - "date_published": "2024-11-24T15:45:00Z", - "tags": ["thoughts"], - "_starpunk": { - "permalink_path": "/notes/2024/11/24/another-note", - "word_count": 6 - } - } - ] -} -``` - -## Validation - -### JSON Feed Validator - -Validate against the official validator: -- https://validator.jsonfeed.org/ - -### Common Validation Issues - -1. **Invalid JSON Syntax** - - Proper escaping of quotes - - Valid UTF-8 encoding - - No trailing commas - -2. **Missing Required Fields** - - version, title, items required - - Each item needs id - -3. **Invalid Date Format** - - Must be RFC 3339 - - Include timezone - -4. **Invalid URLs** - - Must be absolute URLs - - Properly encoded - -## Testing Strategy - -### Unit Tests - -```python -class TestJsonFeedGenerator: - def test_required_fields(self): - """Test all required fields are present""" - generator = JsonFeedGenerator(site_url, site_name, site_description) - feed_json = generator.generate(notes) - feed = json.loads(feed_json) - - assert feed['version'] == 'https://jsonfeed.org/version/1.1' - assert 'title' in feed - assert 'items' in feed - - def test_feed_order_newest_first(self): - """Test JSON feed shows newest entries first (spec convention)""" - # Create notes with different timestamps - old_note = Note( - title="Old Note", - created_at=datetime(2024, 11, 20, 10, 0, 0, tzinfo=timezone.utc) - ) - new_note = Note( - title="New Note", - created_at=datetime(2024, 11, 25, 10, 0, 0, tzinfo=timezone.utc) - ) - - # Generate feed with notes in DESC order (as from database) - generator = JsonFeedGenerator(site_url, site_name, site_description) - feed_json = generator.generate([new_note, old_note]) - feed = json.loads(feed_json) - - # First item should be newest - assert feed['items'][0]['title'] == "New Note" - assert '2024-11-25' in feed['items'][0]['date_published'] - - # Second item should be oldest - assert feed['items'][1]['title'] == "Old Note" - assert '2024-11-20' in feed['items'][1]['date_published'] - - def test_json_validity(self): - """Test output is valid JSON""" - generator = JsonFeedGenerator(site_url, site_name, site_description) - feed_json = generator.generate(notes) - - # Should parse without error - feed = json.loads(feed_json) - assert isinstance(feed, dict) - - def test_date_formatting(self): - """Test RFC 3339 date formatting""" - dt = datetime(2024, 11, 25, 12, 0, 0, tzinfo=timezone.utc) - formatted = generator._format_json_date(dt) - - assert formatted == '2024-11-25T12:00:00Z' - - def test_streaming_generation(self): - """Test streaming produces valid JSON""" - generator = JsonFeedGenerator(site_url, site_name, site_description) - chunks = list(generator.generate_streaming(notes)) - feed_json = ''.join(chunks) - - # Should be valid JSON - feed = json.loads(feed_json) - assert feed['version'] == 'https://jsonfeed.org/version/1.1' - - def test_custom_extensions(self): - """Test custom _starpunk extension""" - generator = JsonFeedGenerator(site_url, site_name, site_description) - feed_json = generator.generate([sample_note]) - feed = json.loads(feed_json) - - item = feed['items'][0] - assert '_starpunk' in item - assert 'permalink_path' in item['_starpunk'] - assert 'word_count' in item['_starpunk'] -``` - -### Integration Tests - -```python -def test_json_feed_endpoint(): - """Test JSON feed endpoint""" - response = client.get('/feed.json') - - assert response.status_code == 200 - assert response.content_type == 'application/feed+json' - - feed = json.loads(response.data) - assert feed['version'] == 'https://jsonfeed.org/version/1.1' - -def test_content_negotiation_json(): - """Test content negotiation prefers JSON""" - response = client.get('/feed', headers={'Accept': 'application/json'}) - - assert response.status_code == 200 - assert 'json' in response.content_type.lower() - -def test_feed_reader_compatibility(): - """Test with JSON Feed readers""" - readers = [ - 'Feedbin', - 'Inoreader', - 'NewsBlur', - 'NetNewsWire' - ] - - for reader in readers: - assert validate_with_reader(feed_url, reader, format='json') -``` - -### Validation Tests - -```python -def test_jsonfeed_validation(): - """Validate against official validator""" - generator = JsonFeedGenerator(site_url, site_name, site_description) - feed_json = generator.generate(sample_notes) - - # Submit to validator - result = validate_json_feed(feed_json) - assert result['valid'] == True - assert len(result['errors']) == 0 -``` - -## Performance Benchmarks - -### Generation Speed - -```python -def benchmark_json_generation(): - """Benchmark JSON feed generation""" - notes = generate_sample_notes(100) - generator = JsonFeedGenerator(site_url, site_name, site_description) - - start = time.perf_counter() - feed_json = generator.generate(notes, limit=50) - duration = time.perf_counter() - start - - assert duration < 0.05 # Less than 50ms - assert len(feed_json) > 0 -``` - -### Size Comparison - -```python -def test_json_vs_xml_size(): - """Compare JSON feed size to RSS/ATOM""" - notes = generate_sample_notes(50) - - # Generate all formats - json_feed = json_generator.generate(notes) - rss_feed = rss_generator.generate(notes) - atom_feed = atom_generator.generate(notes) - - # JSON should be more compact - print(f"JSON: {len(json_feed)} bytes") - print(f"RSS: {len(rss_feed)} bytes") - print(f"ATOM: {len(atom_feed)} bytes") - - # Typically JSON is 20-30% smaller -``` - -## Configuration - -### JSON Feed Settings - -```ini -# JSON Feed configuration -STARPUNK_FEED_JSON_ENABLED=true -STARPUNK_FEED_JSON_AUTHOR_NAME=John Doe -STARPUNK_FEED_JSON_AUTHOR_URL=https://example.com/about -STARPUNK_FEED_JSON_AUTHOR_AVATAR=https://example.com/avatar.jpg -STARPUNK_FEED_JSON_ICON=https://example.com/icon.png -STARPUNK_FEED_JSON_FAVICON=https://example.com/favicon.ico -STARPUNK_FEED_JSON_LANGUAGE=en -STARPUNK_FEED_JSON_HUB_URL= # WebSub hub URL (optional) -``` - -## Security Considerations - -1. **JSON Injection Prevention** - - Proper JSON escaping - - No raw user input - - Validate all URLs - -2. **Content Security** - - HTML content sanitized - - No script injection - - Safe JSON encoding - -3. **Size Limits** - - Maximum feed size - - Item count limits - - Timeout protection - -## Migration Notes - -### Adding JSON Feed - -- Runs parallel to RSS/ATOM -- No changes to existing feeds -- Shared caching infrastructure -- Same data source - -## Advanced Features - -### WebSub Support (Future) - -```json -{ - "hubs": [ - { - "type": "WebSub", - "url": "https://example.com/hub" - } - ] -} -``` - -### Pagination - -```json -{ - "next_url": "https://example.com/feed.json?page=2" -} -``` - -### Attachments - -```json -{ - "attachments": [ - { - "url": "https://example.com/podcast.mp3", - "mime_type": "audio/mpeg", - "title": "Podcast Episode", - "size_in_bytes": 25000000, - "duration_in_seconds": 1800 - } - ] -} -``` - -## Acceptance Criteria - -1. β Valid JSON Feed 1.1 generation -2. β All required fields present -3. β RFC 3339 dates correct -4. β Valid JSON syntax -5. β Streaming generation working -6. β Official validator passing -7. β Works with 5+ JSON Feed readers -8. β Performance target met (<50ms) -9. β Custom extensions working -10. β Security review passed \ No newline at end of file diff --git a/docs/design/v1.1.2/metrics-instrumentation-spec.md b/docs/design/v1.1.2/metrics-instrumentation-spec.md deleted file mode 100644 index 6212940..0000000 --- a/docs/design/v1.1.2/metrics-instrumentation-spec.md +++ /dev/null @@ -1,534 +0,0 @@ -# Metrics Instrumentation Specification - v1.1.2 - -## Overview - -This specification completes the metrics instrumentation foundation started in v1.1.1, adding comprehensive coverage for database operations, HTTP requests, memory monitoring, and business-specific syndication metrics. - -## Requirements - -### Functional Requirements - -1. **Database Performance Metrics** - - Time all database operations - - Track query patterns and frequency - - Detect slow queries (>1 second) - - Monitor connection pool utilization - - Count rows affected/returned - -2. **HTTP Request/Response Metrics** - - Full request lifecycle timing - - Request and response size tracking - - Status code distribution - - Per-endpoint performance metrics - - Client identification (user agent) - -3. **Memory Monitoring** - - Continuous RSS memory tracking - - Memory growth detection - - High water mark tracking - - Garbage collection statistics - - Leak detection algorithms - -4. **Business Metrics** - - Feed request counts by format - - Cache hit/miss rates - - Content publication rates - - Syndication success tracking - - Format popularity analysis - -### Non-Functional Requirements - -1. **Performance Impact** - - Total overhead <1% when enabled - - Zero impact when disabled - - Efficient metric storage (<2MB) - - Non-blocking collection - -2. **Data Retention** - - In-memory circular buffer - - Last 1000 metrics retained - - 15-minute detail window - - Automatic cleanup - -## Design - -### Database Instrumentation - -#### Connection Wrapper - -```python -class MonitoredConnection: - """SQLite connection wrapper with performance monitoring""" - - def __init__(self, db_path: str, metrics_collector: MetricsCollector): - self.conn = sqlite3.connect(db_path) - self.metrics = metrics_collector - - def execute(self, query: str, params: Optional[tuple] = None) -> sqlite3.Cursor: - """Execute query with timing""" - query_type = self._get_query_type(query) - table_name = self._extract_table_name(query) - - start_time = time.perf_counter() - try: - cursor = self.conn.execute(query, params or ()) - duration = time.perf_counter() - start_time - - # Record successful execution - self.metrics.record_database_operation( - operation_type=query_type, - table_name=table_name, - duration_ms=duration * 1000, - rows_affected=cursor.rowcount if query_type != 'SELECT' else len(cursor.fetchall()) - ) - - # Check for slow query - if duration > 1.0: - self.metrics.record_slow_query(query, duration, params) - - return cursor - - except Exception as e: - duration = time.perf_counter() - start_time - self.metrics.record_database_error(query_type, table_name, str(e), duration * 1000) - raise - - def _get_query_type(self, query: str) -> str: - """Extract query type from SQL""" - query_upper = query.strip().upper() - for query_type in ['SELECT', 'INSERT', 'UPDATE', 'DELETE', 'CREATE', 'DROP']: - if query_upper.startswith(query_type): - return query_type - return 'OTHER' - - def _extract_table_name(self, query: str) -> Optional[str]: - """Extract primary table name from query""" - # Simple regex patterns for common cases - patterns = [ - r'FROM\s+(\w+)', - r'INTO\s+(\w+)', - r'UPDATE\s+(\w+)', - r'DELETE\s+FROM\s+(\w+)' - ] - # Implementation details... -``` - -#### Metrics Collected - -| Metric | Type | Description | -|--------|------|-------------| -| `db.query.duration` | Histogram | Query execution time in ms | -| `db.query.count` | Counter | Total queries by type | -| `db.query.errors` | Counter | Failed queries by type | -| `db.rows.affected` | Histogram | Rows modified per query | -| `db.rows.returned` | Histogram | Rows returned per SELECT | -| `db.slow_queries` | List | Queries exceeding threshold | -| `db.connection.active` | Gauge | Active connections | -| `db.transaction.duration` | Histogram | Transaction time in ms | - -### HTTP Instrumentation - -#### Request Middleware - -```python -class HTTPMetricsMiddleware: - """Flask middleware for HTTP metrics collection""" - - def __init__(self, app: Flask, metrics_collector: MetricsCollector): - self.app = app - self.metrics = metrics_collector - self.setup_hooks() - - def setup_hooks(self): - """Register Flask hooks for metrics""" - - @self.app.before_request - def start_request_timer(): - """Initialize request metrics""" - g.request_metrics = { - 'start_time': time.perf_counter(), - 'start_memory': self._get_memory_usage(), - 'request_id': str(uuid.uuid4()), - 'method': request.method, - 'endpoint': request.endpoint, - 'path': request.path, - 'content_length': request.content_length or 0 - } - - @self.app.after_request - def record_response_metrics(response): - """Record response metrics""" - if not hasattr(g, 'request_metrics'): - return response - - # Calculate metrics - duration = time.perf_counter() - g.request_metrics['start_time'] - memory_delta = self._get_memory_usage() - g.request_metrics['start_memory'] - - # Record to collector - self.metrics.record_http_request( - method=g.request_metrics['method'], - endpoint=g.request_metrics['endpoint'], - status_code=response.status_code, - duration_ms=duration * 1000, - request_size=g.request_metrics['content_length'], - response_size=len(response.get_data()), - memory_delta_mb=memory_delta - ) - - # Add timing header for debugging - if self.app.config.get('DEBUG'): - response.headers['X-Response-Time'] = f"{duration * 1000:.2f}ms" - - return response -``` - -#### Metrics Collected - -| Metric | Type | Description | -|--------|------|-------------| -| `http.request.duration` | Histogram | Total request processing time | -| `http.request.count` | Counter | Requests by method and endpoint | -| `http.request.size` | Histogram | Request body size distribution | -| `http.response.size` | Histogram | Response body size distribution | -| `http.status.{code}` | Counter | Response status code counts | -| `http.endpoint.{name}.duration` | Histogram | Per-endpoint timing | -| `http.memory.delta` | Gauge | Memory change per request | - -### Memory Monitoring - -#### Background Monitor Thread - -```python -class MemoryMonitor(Thread): - """Background thread for continuous memory monitoring""" - - def __init__(self, metrics_collector: MetricsCollector, interval: int = 10): - super().__init__(daemon=True) - self.metrics = metrics_collector - self.interval = interval - self.running = True - self.baseline_memory = None - self.high_water_mark = 0 - - def run(self): - """Main monitoring loop""" - # Establish baseline after startup - time.sleep(5) - self.baseline_memory = self._get_memory_info() - - while self.running: - try: - memory_info = self._get_memory_info() - - # Update high water mark - self.high_water_mark = max(self.high_water_mark, memory_info['rss']) - - # Calculate growth rate - if self.baseline_memory: - growth_rate = (memory_info['rss'] - self.baseline_memory['rss']) / - (time.time() - self.baseline_memory['timestamp']) * 3600 - - # Detect potential leak (>10MB/hour growth) - if growth_rate > 10: - self.metrics.record_memory_leak_warning(growth_rate) - - # Record metrics - self.metrics.record_memory_usage( - rss_mb=memory_info['rss'], - vms_mb=memory_info['vms'], - high_water_mb=self.high_water_mark, - gc_stats=self._get_gc_stats() - ) - - except Exception as e: - logger.error(f"Memory monitoring error: {e}") - - time.sleep(self.interval) - - def _get_memory_info(self) -> dict: - """Get current memory usage""" - import resource - usage = resource.getrusage(resource.RUSAGE_SELF) - return { - 'timestamp': time.time(), - 'rss': usage.ru_maxrss / 1024, # Convert to MB - 'vms': usage.ru_idrss - } - - def _get_gc_stats(self) -> dict: - """Get garbage collection statistics""" - import gc - return { - 'collections': gc.get_count(), - 'collected': gc.collect(0), - 'uncollectable': len(gc.garbage) - } -``` - -#### Metrics Collected - -| Metric | Type | Description | -|--------|------|-------------| -| `memory.rss` | Gauge | Resident set size in MB | -| `memory.vms` | Gauge | Virtual memory size in MB | -| `memory.high_water` | Gauge | Maximum RSS observed | -| `memory.growth_rate` | Gauge | MB/hour growth rate | -| `gc.collections` | Counter | GC collection counts by generation | -| `gc.collected` | Counter | Objects collected | -| `gc.uncollectable` | Gauge | Uncollectable object count | - -### Business Metrics - -#### Syndication Metrics - -```python -class SyndicationMetrics: - """Business metrics specific to content syndication""" - - def __init__(self, metrics_collector: MetricsCollector): - self.metrics = metrics_collector - - def record_feed_request(self, format: str, cached: bool, generation_time: float): - """Record feed request metrics""" - self.metrics.increment(f'feed.requests.{format}') - - if cached: - self.metrics.increment('feed.cache.hits') - else: - self.metrics.increment('feed.cache.misses') - self.metrics.record_histogram('feed.generation.time', generation_time * 1000) - - def record_content_negotiation(self, accept_header: str, selected_format: str): - """Track content negotiation results""" - self.metrics.increment(f'feed.negotiation.{selected_format}') - - # Track client preferences - if 'json' in accept_header.lower(): - self.metrics.increment('feed.client.prefers_json') - elif 'atom' in accept_header.lower(): - self.metrics.increment('feed.client.prefers_atom') - - def record_publication(self, note_length: int, has_media: bool): - """Track content publication metrics""" - self.metrics.increment('content.notes.published') - self.metrics.record_histogram('content.note.length', note_length) - - if has_media: - self.metrics.increment('content.notes.with_media') -``` - -#### Metrics Collected - -| Metric | Type | Description | -|--------|------|-------------| -| `feed.requests.{format}` | Counter | Requests by feed format | -| `feed.cache.hits` | Counter | Cache hit count | -| `feed.cache.misses` | Counter | Cache miss count | -| `feed.cache.hit_rate` | Gauge | Cache hit percentage | -| `feed.generation.time` | Histogram | Feed generation duration | -| `feed.negotiation.{format}` | Counter | Format selection results | -| `content.notes.published` | Counter | Total notes published | -| `content.note.length` | Histogram | Note size distribution | -| `content.syndication.success` | Counter | Successful syndications | - -## Implementation Details - -### Metrics Collector - -```python -class MetricsCollector: - """Central metrics collection and storage""" - - def __init__(self, buffer_size: int = 1000): - self.buffer = deque(maxlen=buffer_size) - self.counters = defaultdict(int) - self.gauges = {} - self.histograms = defaultdict(list) - self.slow_queries = deque(maxlen=100) - - def record_metric(self, category: str, name: str, value: float, metadata: dict = None): - """Record a generic metric""" - metric = { - 'timestamp': time.time(), - 'category': category, - 'name': name, - 'value': value, - 'metadata': metadata or {} - } - self.buffer.append(metric) - - def increment(self, name: str, amount: int = 1): - """Increment a counter""" - self.counters[name] += amount - - def set_gauge(self, name: str, value: float): - """Set a gauge value""" - self.gauges[name] = value - - def record_histogram(self, name: str, value: float): - """Add value to histogram""" - self.histograms[name].append(value) - # Keep only last 1000 values - if len(self.histograms[name]) > 1000: - self.histograms[name] = self.histograms[name][-1000:] - - def get_summary(self, window_seconds: int = 900) -> dict: - """Get metrics summary for dashboard""" - cutoff = time.time() - window_seconds - recent = [m for m in self.buffer if m['timestamp'] > cutoff] - - summary = { - 'counters': dict(self.counters), - 'gauges': dict(self.gauges), - 'histograms': self._calculate_histogram_stats(), - 'recent_metrics': recent[-100:], # Last 100 metrics - 'slow_queries': list(self.slow_queries) - } - - return summary - - def _calculate_histogram_stats(self) -> dict: - """Calculate statistics for histograms""" - stats = {} - for name, values in self.histograms.items(): - if values: - sorted_values = sorted(values) - stats[name] = { - 'count': len(values), - 'min': min(values), - 'max': max(values), - 'mean': sum(values) / len(values), - 'p50': sorted_values[len(values) // 2], - 'p95': sorted_values[int(len(values) * 0.95)], - 'p99': sorted_values[int(len(values) * 0.99)] - } - return stats -``` - -## Configuration - -### Environment Variables - -```ini -# Metrics collection toggles -STARPUNK_METRICS_ENABLED=true -STARPUNK_METRICS_DB_TIMING=true -STARPUNK_METRICS_HTTP_TIMING=true -STARPUNK_METRICS_MEMORY_MONITOR=true -STARPUNK_METRICS_BUSINESS=true - -# Thresholds -STARPUNK_METRICS_SLOW_QUERY_THRESHOLD=1.0 # seconds -STARPUNK_METRICS_MEMORY_LEAK_THRESHOLD=10 # MB/hour - -# Storage -STARPUNK_METRICS_BUFFER_SIZE=1000 -STARPUNK_METRICS_RETENTION_SECONDS=900 # 15 minutes - -# Monitoring intervals -STARPUNK_METRICS_MEMORY_INTERVAL=10 # seconds -``` - -## Testing Strategy - -### Unit Tests - -1. **Collector Tests** - ```python - def test_metrics_buffer_circular(): - collector = MetricsCollector(buffer_size=10) - for i in range(20): - collector.record_metric('test', 'metric', i) - assert len(collector.buffer) == 10 - assert collector.buffer[0]['value'] == 10 # Oldest is 10, not 0 - ``` - -2. **Instrumentation Tests** - ```python - def test_database_timing(): - conn = MonitoredConnection(':memory:', collector) - conn.execute('CREATE TABLE test (id INTEGER)') - - metrics = collector.get_summary() - assert 'db.query.duration' in metrics['histograms'] - assert metrics['counters']['db.query.count'] == 1 - ``` - -### Integration Tests - -1. **End-to-End Request Tracking** - ```python - def test_request_metrics(): - response = client.get('/feed.xml') - - metrics = app.metrics_collector.get_summary() - assert 'http.request.duration' in metrics['histograms'] - assert metrics['counters']['http.status.200'] > 0 - ``` - -2. **Memory Leak Detection** - ```python - def test_memory_monitoring(): - monitor = MemoryMonitor(collector) - monitor.start() - - # Simulate memory growth - large_list = [0] * 1000000 - time.sleep(15) - - metrics = collector.get_summary() - assert metrics['gauges']['memory.rss'] > 0 - ``` - -## Performance Benchmarks - -### Overhead Measurement - -```python -def benchmark_instrumentation_overhead(): - # Baseline without instrumentation - config.METRICS_ENABLED = False - start = time.perf_counter() - for _ in range(1000): - execute_operation() - baseline = time.perf_counter() - start - - # With instrumentation - config.METRICS_ENABLED = True - start = time.perf_counter() - for _ in range(1000): - execute_operation() - instrumented = time.perf_counter() - start - - overhead_percent = ((instrumented - baseline) / baseline) * 100 - assert overhead_percent < 1.0 # Less than 1% overhead -``` - -## Security Considerations - -1. **No Sensitive Data**: Never log query parameters that might contain passwords -2. **Rate Limiting**: Metrics endpoints should be rate-limited -3. **Access Control**: Metrics dashboard requires admin authentication -4. **Data Sanitization**: Escape all user-provided data in metrics - -## Migration Notes - -### From v1.1.1 - -- Existing performance monitoring configuration remains compatible -- New metrics are additive, no breaking changes -- Dashboard enhanced but backward compatible - -## Acceptance Criteria - -1. β All database operations are timed -2. β HTTP requests fully instrumented -3. β Memory monitoring thread operational -4. β Business metrics for syndication tracked -5. β Performance overhead <1% -6. β Metrics dashboard shows all new data -7. β Slow query detection working -8. β Memory leak detection functional -9. β All metrics properly documented -10. β Security review passed \ No newline at end of file diff --git a/docs/design/v1.1.2/phase2-completion-update.md b/docs/design/v1.1.2/phase2-completion-update.md deleted file mode 100644 index f5cbe1f..0000000 --- a/docs/design/v1.1.2/phase2-completion-update.md +++ /dev/null @@ -1,159 +0,0 @@ -# StarPunk v1.1.2 Phase 2 - Completion Update - -**Date**: 2025-11-26 -**Phase**: 2 - Feed Formats -**Status**: COMPLETE β - -## Summary - -Phase 2 of the v1.1.2 "Syndicate" release has been fully completed by the developer. All sub-phases (2.0 through 2.4) have been implemented, tested, and reviewed. - -## Implementation Status - -### Phase 2.0: RSS Feed Ordering Fix β COMPLETE -- **Status**: COMPLETE (2025-11-26) -- **Time**: 0.5 hours (as estimated) -- **Result**: Critical bug fixed, RSS now shows newest-first - -### Phase 2.1: Feed Module Restructuring β COMPLETE -- **Status**: COMPLETE (2025-11-26) -- **Time**: 1.5 hours -- **Result**: Clean module organization in `starpunk/feeds/` - -### Phase 2.2: ATOM Feed Generation β COMPLETE -- **Status**: COMPLETE (2025-11-26) -- **Time**: 2.5 hours -- **Result**: Full RFC 4287 compliance with 11 passing tests - -### Phase 2.3: JSON Feed Generation β COMPLETE -- **Status**: COMPLETE (2025-11-26) -- **Time**: 2.5 hours -- **Result**: JSON Feed 1.1 compliance with 13 passing tests - -### Phase 2.4: Content Negotiation β COMPLETE -- **Status**: COMPLETE (2025-11-26) -- **Time**: 1 hour -- **Result**: HTTP Accept header negotiation with 63 passing tests - -## Total Phase 2 Metrics - -- **Total Time**: 8 hours (vs 6-8 hours estimated) -- **Total Tests**: 132 (all passing) -- **Lines of Code**: ~2,540 (production + tests) -- **Standards**: Full compliance with RSS 2.0, ATOM 1.0, JSON Feed 1.1 - -## Deliverables - -### Production Code -- `starpunk/feeds/rss.py` - RSS 2.0 generator (moved from feed.py) -- `starpunk/feeds/atom.py` - ATOM 1.0 generator (new) -- `starpunk/feeds/json_feed.py` - JSON Feed 1.1 generator (new) -- `starpunk/feeds/negotiation.py` - Content negotiation (new) -- `starpunk/feeds/__init__.py` - Module exports -- `starpunk/feed.py` - Backward compatibility shim -- `starpunk/routes/public.py` - Feed endpoints - -### Test Code -- `tests/helpers/feed_ordering.py` - Shared ordering test helper -- `tests/test_feeds_atom.py` - ATOM tests (11 tests) -- `tests/test_feeds_json.py` - JSON Feed tests (13 tests) -- `tests/test_feeds_negotiation.py` - Negotiation tests (41 tests) -- `tests/test_routes_feeds.py` - Integration tests (22 tests) - -### Documentation -- `docs/reports/2025-11-26-v1.1.2-phase2-complete.md` - Developer's implementation report -- `docs/reviews/2025-11-26-phase2-architect-review.md` - Architect's review (APPROVED) - -## Available Endpoints - -``` -GET /feed # Content negotiation (RSS/ATOM/JSON) -GET /feed.rss # Explicit RSS 2.0 -GET /feed.atom # Explicit ATOM 1.0 -GET /feed.json # Explicit JSON Feed 1.1 -GET /feed.xml # Backward compat (β /feed.rss) -``` - -## Quality Metrics - -### Test Results -```bash -$ uv run pytest tests/test_feed*.py tests/test_routes_feed*.py -q -132 passed in 11.42s -``` - -### Standards Compliance -- β RSS 2.0: Full specification compliance -- β ATOM 1.0: RFC 4287 compliance -- β JSON Feed 1.1: Full specification compliance -- β HTTP: Practical content negotiation - -### Performance -- RSS generation: ~2-5ms for 50 items -- ATOM generation: ~2-5ms for 50 items -- JSON generation: ~1-3ms for 50 items -- Content negotiation: <1ms overhead - -## Architect's Review - -**Verdict**: APPROVED WITH COMMENDATION - -Key points from review: -- Exceptional adherence to architectural principles -- Perfect implementation of StarPunk philosophy -- Zero defects identified -- Ready for immediate production deployment - -## Next Steps - -### Immediate -1. β Merge to main branch (approved by architect) -2. β Deploy to production (includes critical RSS fix) -3. β³ Begin Phase 3: Feed Caching - -### Phase 3 Preview -- Checksum-based feed caching -- ETag support -- Conditional GET (304 responses) -- Cache invalidation strategy -- Estimated time: 4-6 hours - -## Updates Required - -### Project Plan -The main implementation guide (`docs/design/v1.1.2/implementation-guide.md`) should be updated to reflect: -- Phase 2 marked as COMPLETE -- Actual time taken (8 hours) -- Link to completion documentation -- Phase 3 ready to begin - -### CHANGELOG -Add entry for Phase 2 completion: -```markdown -### [Unreleased] - Phase 2 Complete - -#### Added -- ATOM 1.0 feed support with RFC 4287 compliance -- JSON Feed 1.1 support with full specification compliance -- HTTP content negotiation for automatic format selection -- Explicit feed endpoints (/feed.rss, /feed.atom, /feed.json) -- Comprehensive feed test suite (132 tests) - -#### Fixed -- Critical: RSS feed ordering now shows newest entries first -- Removed misleading comments about feedgen behavior - -#### Changed -- Restructured feed code into `starpunk/feeds/` module -- Improved feed generation performance with streaming -``` - -## Conclusion - -Phase 2 is complete and exceeds all requirements. The implementation is production-ready and approved for immediate deployment. The developer has demonstrated exceptional skill in delivering a comprehensive, standards-compliant solution with minimal code. - ---- - -**Updated by**: StarPunk Architect (AI) -**Date**: 2025-11-26 -**Phase Status**: β COMPLETE - Ready for Phase 3 \ No newline at end of file diff --git a/docs/design/v1.2.0/2025-12-09-v1.2.0-release.md b/docs/design/v1.2.0/2025-12-09-v1.2.0-release.md new file mode 100644 index 0000000..d383107 --- /dev/null +++ b/docs/design/v1.2.0/2025-12-09-v1.2.0-release.md @@ -0,0 +1,258 @@ +# v1.2.0 Release Report + +**Date**: 2025-12-09 +**Version**: 1.2.0 +**Release Type**: Stable Minor Release +**Previous Version**: 1.1.2 + +## Overview + +Successfully promoted v1.2.0-rc.2 to stable v1.2.0 release. This is a major feature release adding comprehensive media support, author discovery, custom slugs, and enhanced syndication feeds. + +## Release Process + +### 1. Version Updates + +**File**: `starpunk/__init__.py` +- Updated `__version__` from `"1.2.0-rc.2"` to `"1.2.0"` +- Updated `__version_info__` from `(1, 2, 0, "dev")` to `(1, 2, 0)` + +### 2. CHANGELOG Updates + +**File**: `CHANGELOG.md` +- Merged rc.1 and rc.2 entries into single `[1.2.0]` section +- Added release date: 2025-12-09 +- Consolidated all features and fixes from both release candidates +- Maintained chronological order of changes + +### 3. Git Operations + +**Commit**: `927db4a` +``` +release: Bump version to 1.2.0 + +Promote v1.2.0-rc.2 to stable v1.2.0 release + +- Merged rc.1 and rc.2 changelog entries +- Updated version in starpunk/__init__.py +- All features tested in production +``` + +**Tag**: `v1.2.0` (annotated) +- Comprehensive release notes included +- Documents all major features +- Notes standards compliance +- Includes upgrade instructions + +### 4. Container Images + +Built and pushed container images: +- `git.thesatelliteoflove.com/phil/starpunk:v1.2.0` +- `git.thesatelliteoflove.com/phil/starpunk:latest` + +**Image Size**: 190 MB +**Base**: Python 3.11-slim +**Build**: Multi-stage with uv package manager + +### 5. Registry Push + +Successfully pushed to remote: +- Git commit pushed to `origin/main` +- Git tag `v1.2.0` pushed to remote +- Container images pushed to `git.thesatelliteoflove.com` registry + +## Release Contents + +### Major Features + +#### Media Upload & Display +- Upload up to 4 images per note (JPEG, PNG, GIF, WebP) +- Automatic image optimization with Pillow library +- File size limit: 10MB per image +- Dimension limit: 4096x4096 pixels +- Auto-resize images over 2048px +- EXIF orientation correction +- Social media style layout (media first, then text) +- Optional captions for accessibility +- Responsive image sizing with proper CSS + +#### Feed Media Enhancement +- Media RSS namespace (xmlns:media) for structured metadata +- RSS enclosure element for first image (per RSS 2.0 spec) +- Media RSS media:content elements for all images +- Media RSS media:thumbnail element for preview +- JSON Feed image field (per JSON Feed 1.1 spec) +- Enhanced display in modern feed readers (Feedly, Inoreader, NetNewsWire) + +#### Author Profile Discovery +- Automatic h-card discovery from IndieAuth identity +- Caches author information (name, photo, bio, rel-me links) +- 24-hour cache TTL +- Graceful fallback to domain name +- Never blocks login functionality +- Eliminates need for manual author configuration + +#### Complete Microformats2 Support +- Full h-entry markup with required properties +- Author h-card nested within each h-entry +- Proper p-name handling (only when explicit title) +- u-uid and u-url match for permalink stability +- Homepage as h-feed with proper structure +- rel-me links from discovered profile +- dt-updated property when note modified +- Passes Microformats2 validation + +#### Custom Slugs +- Web UI custom slug input field +- Optional field with auto-generation fallback +- Read-only after creation (preserves permalinks) +- Automatic validation and sanitization +- Helpful placeholder text and guidance +- Matches Micropub mp-slug behavior + +### Fixes from RC Releases + +#### RC.2 Fixes +- Media display on homepage (not just individual note pages) +- Responsive image sizing with container constraints +- Caption display (alt text only, not visible text) +- Logging correlation ID crash in non-request contexts + +#### RC.1 Fixes +- All features tested and validated in production + +## Standards Compliance + +- W3C Micropub Specification +- Microformats2 h-entry, h-card, h-feed +- RSS 2.0 with Media RSS extension +- JSON Feed 1.1 specification +- IndieWeb best practices + +## Testing + +- 600+ tests passing +- All features tested in production (rc.1 and rc.2) +- Enhanced feed reader compatibility verified +- Media upload and display validated +- Author discovery tested with multiple profiles + +## Upgrade Instructions + +### From v1.1.2 + +No breaking changes. Simple upgrade process: + +1. Pull latest code: `git pull origin main` +2. Checkout tag: `git checkout v1.2.0` +3. Restart application + +### Configuration + +No configuration changes required. All new features work automatically. + +Optional configuration for media: +- `MEDIA_MAX_SIZE` - Max file size in bytes (default: 10MB) +- `MEDIA_MAX_DIMENSION` - Max dimension in pixels (default: 4096) +- `MEDIA_RESIZE_THRESHOLD` - Auto-resize threshold (default: 2048) + +## Verification + +### Version Check +```bash +$ uv run python -c "from starpunk import __version__; print(__version__)" +1.2.0 +``` + +### Git Tag +```bash +$ git tag -l v1.2.0 +v1.2.0 + +$ git log -1 --oneline +927db4a release: Bump version to 1.2.0 +``` + +### Container Images +```bash +$ podman images | grep starpunk | grep v1.2.0 +git.thesatelliteoflove.com/phil/starpunk v1.2.0 20853617ebf1 190 MB +git.thesatelliteoflove.com/phil/starpunk latest 20853617ebf1 190 MB +``` + +## Documentation + +### Updated Files +- `/home/phil/Projects/starpunk/starpunk/__init__.py` +- `/home/phil/Projects/starpunk/CHANGELOG.md` + +### Release Documentation +- Git tag annotation with full release notes +- This implementation report +- CHANGELOG.md with complete details + +### Existing Documentation (Unchanged) +- `/home/phil/Projects/starpunk/docs/design/v1.2.0-media-css-design.md` +- `/home/phil/Projects/starpunk/docs/design/v1.1.2-caption-alttext-update.md` +- `/home/phil/Projects/starpunk/docs/design/media-display-fixes.md` +- `/home/phil/Projects/starpunk/docs/reports/2025-11-28-media-display-fixes.md` + +## Release Timeline + +- **2025-11-28**: v1.2.0-rc.1 released (initial feature complete) +- **2025-12-09**: v1.2.0-rc.2 released (media display fixes) +- **2025-12-09**: v1.2.0 stable released (production validated) + +## Backwards Compatibility + +Fully backward compatible with v1.1.2. No breaking changes. + +- Existing notes display correctly +- Existing feeds continue working +- Existing configuration valid +- Existing clients unaffected + +## Known Issues + +None identified. All features tested and stable in production. + +## Next Steps + +### Post-Release +1. Monitor production deployment +2. Update any documentation references to version numbers +3. Announce release to users + +### Future Development (v1.3.0 or v2.0.0) +- Additional IndieWeb features (Webmentions, etc.) +- Enhanced search capabilities +- Performance optimizations +- User-requested features + +## Related Documentation + +- `/home/phil/Projects/starpunk/docs/standards/versioning-strategy.md` +- `/home/phil/Projects/starpunk/docs/standards/git-branching-strategy.md` +- `/home/phil/Projects/starpunk/CHANGELOG.md` + +## Compliance + +This release follows: +- Semantic Versioning 2.0.0 +- Keep a Changelog format +- Git workflow from versioning-strategy.md +- Developer protocol from CLAUDE.md + +## Summary + +Successfully promoted v1.2.0-rc.2 to stable v1.2.0 release. All steps completed: + +- Version updated in `starpunk/__init__.py` +- CHANGELOG.md updated with merged entries +- Git commit created and pushed +- Annotated tag `v1.2.0` created and pushed +- Container images built (v1.2.0 and latest) +- Container images pushed to registry +- All verification checks passed + +The release is now available for production deployment. diff --git a/docs/design/v1.2.0/2025-12-10-documentation-audit-v1.2.0.md b/docs/design/v1.2.0/2025-12-10-documentation-audit-v1.2.0.md new file mode 100644 index 0000000..cf1ab67 --- /dev/null +++ b/docs/design/v1.2.0/2025-12-10-documentation-audit-v1.2.0.md @@ -0,0 +1,300 @@ +# Documentation Audit Report - Post v1.2.0 Release + +**Date**: 2025-12-10 +**Agent**: Documentation Manager +**Scope**: Comprehensive documentation audit and cleanup after v1.2.0 release + +## Executive Summary + +Performed a comprehensive documentation audit of the StarPunk project following the v1.2.0 release. The audit focused on repository structure compliance, design document organization, ADR integrity, and README currency. All issues identified have been resolved, resulting in a well-organized and maintainable documentation system. + +**Overall Documentation Health**: Excellent + +## Audit Findings and Actions + +### 1. Repository Root Compliance + +**Status**: PASS + +**Finding**: Repository root contains only the three approved documentation files: +- README.md +- CLAUDE.md +- CHANGELOG.md + +**Action**: No action required. Root structure is compliant with documentation standards. + +--- + +### 2. Misplaced Design Documents + +**Status**: RESOLVED + +**Finding**: Three design documents were located in `/docs/design/` root instead of version-specific subdirectories: +- `media-display-fixes.md` (v1.2.0 content) +- `v1.1.2-caption-alttext-update.md` (v1.1.2 content, marked as superseded) +- `v1.2.0-media-css-design.md` (v1.2.0 content, marked as superseded) + +**Actions Taken**: +1. Moved `media-display-fixes.md` β `v1.2.0/media-display-fixes.md` +2. Moved `v1.1.2-caption-alttext-update.md` β `v1.1.2/caption-alttext-update.md` +3. Moved `v1.2.0-media-css-design.md` β `v1.2.0/media-css-design.md` + +**Rationale**: Version-based organization improves discoverability and maintains clear historical record of design evolution. + +--- + +### 3. Legacy Design Documents Organization + +**Status**: RESOLVED + +**Finding**: 31 design documents from v1.0.0 and v1.1.x development remained in `/docs/design/` root, including: +- 19 phase-based documents from initial implementation (v1.0.0) +- 9 hotfix and diagnostic documents (v1.1.1) +- 3 feed enhancement documents (v1.1.2) + +**Actions Taken**: + +**Created Version Folders**: +- `/docs/design/v1.0.0/` - For initial implementation (phase-based) +- `/docs/design/v1.1.1/` - For authentication hotfix documents + +**Moved v1.0.0 Documents** (19 files): +- All `phase-*.md` files (phase 1.1 through 5) +- `initial-files.md`, `initial-schema-*.md` +- `project-structure.md` +- `micropub-endpoint-design.md` + +**Moved v1.1.1 Documents** (9 files): +- `auth-redirect-loop-*.md` (diagnosis and fix) +- `hotfix-v1.1.1-*.md` +- `indieauth-pkce-authentication.md` +- `token-security-migration.md` + +**Moved v1.1.2 Documents** (3 files): +- `feed-media-handling-options.md` +- `feed-media-option2-design.md` +- `caption-alttext-update.md` + +**Result**: Only `INDEX.md` remains in `/docs/design/` root, which is correct and expected. + +--- + +### 4. Design Documentation INDEX Update + +**Status**: COMPLETED + +**Finding**: The `/docs/design/INDEX.md` file referenced the old flat structure with phase documents in the root. + +**Actions Taken**: +1. Rewrote INDEX.md to reflect version-based organization +2. Added clear organization section listing all version folders +3. Documented key design documents for each version +4. Updated "How to Use" section for version-based navigation +5. Updated "Document Types" to reflect current patterns +6. Updated last-modified date to 2025-12-10 + +**New Structure**: +- Organization section with version folder listing +- Version-specific sections (v1.0.0, v1.1.1, v1.1.2, v1.2.0) +- Key documents highlighted for each version +- Updated usage guidance for developers + +--- + +### 5. ADR Numbering Sequence + +**Status**: VERIFIED - No Issues + +**Finding**: ADR sequence shows a gap: jumps from ADR-059 to ADR-061, missing ADR-060. + +**Investigation**: +- ADR-059 references "Option 2 (ADR-060)" for Media RSS implementation +- Media RSS was implemented in v1.2.0 (confirmed in CHANGELOG) +- No separate ADR-060 document was created + +**Conclusion**: ADR-060 was planned but implementation happened without creating a separate ADR. The decision is adequately documented in ADR-059 itself, which describes both Option 2 (basic Media RSS) and Option 3 (full standardization). This is acceptable - not every decision requires a separate ADR when well-documented in a related ADR. + +**Recommendation**: If future work requires more detailed Media RSS decisions, create ADR-060 at that time. For now, ADR-059 provides sufficient documentation. + +--- + +### 6. README.md Currency + +**Status**: UPDATED + +**Finding**: README.md showed version 1.1.0 but current version is 1.2.0. Major v1.2.0 features were not documented. + +**Actions Taken**: + +**Version Updates**: +- Changed "Current Version: 1.1.0" β "1.2.0" +- Updated versioning section to reflect current 1.2.0 stable release + +**Features Section Enhancements**: +Added new v1.2.0 features: +- Media attachments (image upload and display) +- Microformats2 (full h-entry, h-card, h-feed markup) +- Author discovery (automatic profile discovery) +- RSS, ATOM, JSON Feed (multiple syndication formats) +- Custom slugs (permalink control) + +Updated from "6 core dependencies" to "Core dependencies" (more accurate, less maintenance burden). + +**IndieWeb Compliance Section**: +Enhanced to include: +- "RSS 2.0 with Media RSS extensions" (was just "RSS 2.0") +- Added "ATOM 1.0 - Syndication format" +- Added "JSON Feed 1.1 - Modern feed format" +- Enhanced Microformats2 description to mention h-entry, h-card, h-feed + +**Result**: README now accurately reflects v1.2.0 capabilities and feature set. + +--- + +### 7. Reports Folder Organization + +**Status**: EXCELLENT - No Action Required + +**Finding**: The `/docs/reports/` folder is well-organized with 76 implementation reports. + +**Observations**: +- All reports follow naming convention: `YYYY-MM-DD-description.md` +- Clear chronological ordering (oldest: 2025-11-18, newest: 2025-12-09) +- Version-tagged reports (e.g., `v1.2.0-phase1-custom-slugs.md`) +- One untracked file: `2025-12-09-v1.2.0-release.md` (appropriate for v1.2.0) + +**Action**: Renamed one report for consistency: +- `2025-11-28-media-display-fixes.md` β `2025-11-28-v1.2.0-media-display-fixes.md` + +**Assessment**: Reports folder follows best practices and needs no further cleanup. + +--- + +### 8. Superseded Documents + +**Status**: VERIFIED + +**Finding**: Several documents marked as "Superseded" were found: +- `v1.1.2/caption-alttext-update.md` - Superseded by `media-display-fixes.md` +- `v1.2.0/media-css-design.md` - Superseded by `media-display-fixes.md` +- Various ADRs with superseded status headers + +**Assessment**: +- Superseded documents are properly marked with status headers +- They are retained for historical context (correct approach) +- They are now organized in version folders (improves discoverability) +- Cross-references to superseding documents are present + +**Action**: No action required. Superseded documents are properly handled. + +--- + +## Documentation Organization Summary + +### Repository Root +``` +/ +βββ README.md β Updated to v1.2.0 +βββ CLAUDE.md β Current +βββ CHANGELOG.md β Current +``` + +### Design Documentation Structure +``` +docs/design/ +βββ INDEX.md β Updated for version-based structure +βββ v1.0.0/ β 19 documents (initial implementation) +βββ v1.1.1/ β 9 documents (hotfix) +βββ v1.1.2/ β 10 documents (feed enhancements) +βββ v1.2.0/ β 6 documents (media and IndieWeb) +``` + +### ADR Status +- Total ADRs: 56 (ADR-001 through ADR-061, excluding ADR-060) +- Gap at ADR-060: Acceptable (documented in ADR-059) +- All ADRs properly numbered and sequenced +- Superseded ADRs have status headers + +### Reports Status +- Total reports: 76 implementation reports +- All follow naming convention: `YYYY-MM-DD-description.md` +- Date range: 2025-11-18 to 2025-12-10 +- Well-organized, chronologically ordered + +--- + +## Git Changes Summary + +The following files were moved/renamed using `git mv` to preserve history: + +**Design Document Relocations** (34 files): +- 19 files β `docs/design/v1.0.0/` +- 9 files β `docs/design/v1.1.1/` +- 3 files β `docs/design/v1.1.2/` +- 3 files β `docs/design/v1.2.0/` + +**Report Rename** (1 file): +- `2025-11-28-media-display-fixes.md` β `2025-11-28-v1.2.0-media-display-fixes.md` + +**Documentation Updates** (2 files): +- `README.md` - Version and features updated +- `docs/design/INDEX.md` - Complete restructure for version-based organization + +**Total Changes**: 37 file operations + 2 content updates + +--- + +## Recommendations + +### Immediate Actions +None required. All issues have been resolved. + +### Future Maintenance + +1. **Design Document Discipline** + - Always create new design docs in appropriate version folder + - Use version prefixes in filenames for cross-version documents + - Update INDEX.md when adding new version folders + +2. **ADR Management** + - Continue sequential numbering (next: ADR-062) + - Consider creating ADR-060 if Media RSS needs detailed decision doc + - Always mark superseded ADRs with status headers + +3. **README Maintenance** + - Update version number on each release + - Add new features to features section + - Keep IndieWeb compliance section current + +4. **Reports Best Practices** + - Continue using `YYYY-MM-DD-description.md` format + - Include version prefix for version-specific work + - Create reports for all significant implementations + +### Documentation Health Indicators + +Monitor these metrics to maintain documentation quality: + +- **Root Cleanliness**: Only README.md, CLAUDE.md, CHANGELOG.md in root +- **Design Organization**: All design docs in version folders (except INDEX.md) +- **ADR Sequence**: Sequential numbering with documented gaps +- **Report Consistency**: All reports follow naming convention +- **README Currency**: Version and features match current release + +--- + +## Conclusion + +The StarPunk documentation is now in excellent health following the v1.2.0 release. All structural issues have been resolved, historical documents are properly organized by version, and the README accurately reflects current capabilities. + +The version-based organization of design documents provides a clear historical record and improves discoverability. The reports folder demonstrates excellent discipline with consistent naming and comprehensive coverage of implementation work. + +**Documentation Health Score**: A+ (Excellent) + +**Ready for v1.3.0 Development**: Yes + +--- + +**Audit Completed**: 2025-12-10 +**Maintained By**: Documentation Manager Agent +**Next Audit Recommended**: After v1.3.0 release diff --git a/docs/design/v1.2.0/developer-qa.md b/docs/design/v1.2.0/developer-qa.md deleted file mode 100644 index 4ca5816..0000000 --- a/docs/design/v1.2.0/developer-qa.md +++ /dev/null @@ -1,303 +0,0 @@ -# v1.2.0 Developer Q&A - -**Date**: 2025-11-28 -**Architect**: StarPunk Architect Subagent -**Purpose**: Answer critical implementation questions for v1.2.0 - -## Custom Slugs Answers - -**Q1: Validation pattern conflict - should we apply new lowercase validation to existing slugs?** -- **Answer:** Validate only new custom slugs, don't migrate existing slugs -- **Rationale:** Existing slugs work, no need to change them retroactively -- **Implementation:** In `validate_and_sanitize_custom_slug()`, apply lowercase enforcement only to new/edited slugs - -**Q2: Form field readonly behavior - how should the slug field behave on edit forms?** -- **Answer:** Display as readonly input field with current value visible -- **Rationale:** Users need to see the current slug but understand it cannot be changed -- **Implementation:** Use `readonly` attribute, not `disabled` (disabled fields don't submit with form) - -**Q3: Slug uniqueness validation - where should this happen?** -- **Answer:** Both client-side (for UX) and server-side (for security) -- **Rationale:** Client-side prevents unnecessary submissions, server-side is authoritative -- **Implementation:** Database unique constraint + Python validation in `validate_and_sanitize_custom_slug()` - -## Media Upload Answers - -**Q4: Media upload flow - how should upload and note association work?** -- **Answer:** Upload during note creation, associate via note_id after creation -- **Rationale:** Simpler than pre-upload with temporary IDs -- **Implementation:** Upload files in `create_note_submit()` after note is created, store associations in media table - -**Q5: Storage directory structure - exact path format?** -- **Answer:** `data/media/YYYY/MM/filename-uuid.ext` -- **Rationale:** Date organization helps with backups and management -- **Implementation:** Use `os.makedirs(path, exist_ok=True)` to create directories as needed - -**Q6: File naming convention - how to ensure uniqueness?** -- **Answer:** `{original_name_slug}-{uuid4()[:8]}.{extension}` -- **Rationale:** Preserves original name for SEO while ensuring uniqueness -- **Implementation:** Slugify original filename, append 8-char UUID, preserve extension - -**Q7: MIME type validation - which types exactly?** -- **Answer:** Allow: image/jpeg, image/png, image/gif, image/webp. Reject all others -- **Rationale:** Common web formats only, no SVG (XSS risk) -- **Implementation:** Use python-magic for reliable MIME detection, not just file extension - -**Q8: Upload size limits - what's reasonable?** -- **Answer:** 10MB per file, 40MB total per note (4 files Γ 10MB) -- **Rationale:** Sufficient for high-quality images without overwhelming storage -- **Implementation:** Check in both client-side JavaScript and server-side validation - -**Q9: Database schema for media table - exact columns?** -- **Answer:** id, note_id, filename, mime_type, size_bytes, width, height, uploaded_at -- **Rationale:** Minimal but sufficient metadata for display and management -- **Implementation:** Use Pillow to extract image dimensions on upload - -**Q10: Orphaned file cleanup - how to handle?** -- **Answer:** Keep orphaned files, add admin cleanup tool in future version -- **Rationale:** Data preservation is priority, cleanup can be manual for v1.2.0 -- **Implementation:** Log orphaned files but don't auto-delete - -**Q11: Upload progress indication - required for v1.2.0?** -- **Answer:** No, simple form submission is sufficient for v1.2.0 -- **Rationale:** Keep it simple, can enhance in future version -- **Implementation:** Standard HTML form with enctype="multipart/form-data" - -**Q12: Image display order - how to maintain?** -- **Answer:** Use upload sequence, store display_order in media table -- **Rationale:** Predictable and simple -- **Implementation:** Auto-increment display_order starting at 0 - -**Q13: Thumbnail generation - needed for v1.2.0?** -- **Answer:** No, use CSS for responsive sizing -- **Rationale:** Simplicity over optimization for v1 -- **Implementation:** Use `max-width: 100%` and lazy loading - -**Q14: Edit form media handling - can users remove media?** -- **Answer:** Yes, checkbox to mark for deletion -- **Rationale:** Essential editing capability -- **Implementation:** "Remove" checkboxes next to each image in edit form - -**Q15: Media URL structure - exact format?** -- **Answer:** `/media/YYYY/MM/filename.ext` (matches storage path) -- **Rationale:** Clean URLs, date organization visible -- **Implementation:** Route in `starpunk/routes/public.py` using send_from_directory - -## Author Discovery Answers - -**Q16: Discovery failure handling - what if profile URL is unreachable?** -- **Answer:** Use defaults: name from IndieAuth me URL domain, no photo -- **Rationale:** Always provide something, never break -- **Implementation:** Try discovery, catch all exceptions, use defaults - -**Q17: h-card parsing library - which one?** -- **Answer:** Use mf2py (already in requirements for Micropub) -- **Rationale:** Already a dependency, well-maintained -- **Implementation:** `import mf2py; result = mf2py.parse(url=profile_url)` - -**Q18: Multiple h-cards on profile - which to use?** -- **Answer:** First h-card with url property matching the profile URL -- **Rationale:** Most specific match per IndieWeb convention -- **Implementation:** Loop through h-cards, check url property - -**Q19: Discovery caching duration - how long?** -- **Answer:** 24 hours, with manual refresh button in admin -- **Rationale:** Balance between freshness and performance -- **Implementation:** Store discovered_at timestamp, check age - -**Q20: Profile update mechanism - when to refresh?** -- **Answer:** On login + manual refresh button + 24hr expiry -- **Rationale:** Login is natural refresh point -- **Implementation:** Call discovery in auth callback - -**Q21: Missing properties handling - what if no name/photo?** -- **Answer:** name = domain from URL, photo = None (no image) -- **Rationale:** Graceful degradation -- **Implementation:** Use get() with defaults on parsed properties - -**Q22: Database schema for author_profile - exact columns?** -- **Answer:** me_url (PK), name, photo, url, discovered_at, raw_data (JSON) -- **Rationale:** Cache parsed data + raw for debugging -- **Implementation:** Single row table, upsert on discovery - -## Microformats2 Answers - -**Q23: h-card placement - where exactly in templates?** -- **Answer:** Only within h-entry author property (p-author h-card) -- **Rationale:** Correct semantic placement per spec -- **Implementation:** In note partial template, not standalone - -**Q24: h-feed container - which pages need it?** -- **Answer:** Homepage (/) and any paginated list pages -- **Rationale:** Feed pages only, not single note pages -- **Implementation:** Wrap note list in div.h-feed with h1.p-name - -**Q25: Optional properties - which to include?** -- **Answer:** Only what we have: author, name, url, published, content -- **Rationale:** Don't add empty properties -- **Implementation:** Use conditional template blocks - -**Q26: Micropub compatibility - any changes needed?** -- **Answer:** No, Micropub already handles microformats correctly -- **Rationale:** Micropub creates data, templates display it -- **Implementation:** Ensure templates match Micropub's data model - -## Feed Integration Answers - -**Q27: RSS/Atom changes for media - how to include images?** -- **Answer:** Add as enclosures (RSS) and link rel="enclosure" (Atom) -- **Rationale:** Standard podcast/media pattern -- **Implementation:** Loop through note.media, add enclosure elements - -**Q28: JSON Feed media handling - which property?** -- **Answer:** Use "attachments" array per JSON Feed 1.1 spec -- **Rationale:** Designed for exactly this use case -- **Implementation:** Create attachment objects with url, mime_type - -**Q29: Feed caching - any changes needed?** -- **Answer:** No, existing cache logic is sufficient -- **Rationale:** Media URLs are stable once uploaded -- **Implementation:** No changes required - -**Q30: Author in feeds - use discovered data?** -- **Answer:** Yes, use discovered name and photo in feed metadata -- **Rationale:** Consistency across all outputs -- **Implementation:** Pass author_profile to feed templates - -## Database Migration Answers - -**Q31: Migration naming convention - what number?** -- **Answer:** Use next sequential: 005_add_media_support.sql -- **Rationale:** Continue existing pattern -- **Implementation:** Check latest migration, increment - -**Q32: Migration rollback - needed?** -- **Answer:** No, forward-only migrations per project convention -- **Rationale:** Simplicity, follows existing pattern -- **Implementation:** CREATE IF NOT EXISTS, never DROP - -**Q33: Migration testing - how to verify?** -- **Answer:** Test on copy of production database -- **Rationale:** Real-world data is best test -- **Implementation:** Copy data/starpunk.db, run migration, verify - -## Testing Strategy Answers - -**Q34: Test data for media - what to use?** -- **Answer:** Generate 1x1 pixel PNG in tests, don't use real files -- **Rationale:** Minimal, fast, no binary files in repo -- **Implementation:** Use Pillow to generate test images in memory - -**Q35: Author discovery mocking - how to test?** -- **Answer:** Mock HTTP responses with test h-card HTML -- **Rationale:** Deterministic, no external dependencies -- **Implementation:** Use responses library or unittest.mock - -**Q36: Integration test priority - which are critical?** -- **Answer:** Upload β Display β Edit β Delete flow -- **Rationale:** Core user journey must work -- **Implementation:** Single test that exercises full lifecycle - -## Error Handling Answers - -**Q37: Upload failure recovery - how to handle?** -- **Answer:** Show error, preserve form data, allow retry -- **Rationale:** Don't lose user's work -- **Implementation:** Flash error, return to form with content preserved - -**Q38: Discovery network timeout - how long to wait?** -- **Answer:** 5 second timeout for profile fetch -- **Rationale:** Balance between patience and responsiveness -- **Implementation:** Use requests timeout parameter - -## Deployment Answers - -**Q39: Media directory permissions - what's needed?** -- **Answer:** data/media/ needs write permission for app user -- **Rationale:** Same as existing data/ directory -- **Implementation:** Document in deployment guide, create in setup - -**Q40: Upgrade path from v1.1.2 - any special steps?** -- **Answer:** Run migration, create media directory, restart app -- **Rationale:** Minimal disruption -- **Implementation:** Add to CHANGELOG upgrade notes - -**Q41: Configuration changes - any new env vars?** -- **Answer:** No, all settings have sensible defaults -- **Rationale:** Maintain zero-config philosophy -- **Implementation:** Hardcode limits in code with constants - -## Critical Path Decisions Summary - -These are the key decisions to unblock implementation: - -1. **Media upload flow**: Upload after note creation, associate via note_id -2. **Author discovery**: Use mf2py, cache for 24hrs, graceful fallbacks -3. **h-card parsing**: First h-card with matching URL property -4. **h-card placement**: Only within h-entry as p-author -5. **Migration strategy**: Sequential numbering (005), forward-only - -## Implementation Order - -Based on dependencies and complexity: - -### Phase 1: Custom Slugs (2 hours) -- Simplest feature -- No database changes -- Template and validation only - -### Phase 2: Author Discovery (4 hours) -- Build discovery module -- Add author_profile table -- Integrate with auth flow -- Update templates - -### Phase 3: Media Upload (6 hours) -- Most complex feature -- Media table and migration -- Upload handling -- Template updates -- Storage management - -## File Structure - -Key files to create/modify: - -### New Files -- `starpunk/discovery.py` - Author discovery module -- `starpunk/media.py` - Media handling module -- `migrations/005_add_media_support.sql` - Database changes -- `static/js/media-upload.js` - Optional enhancement - -### Modified Files -- `templates/admin/new.html` - Add slug and media fields -- `templates/admin/edit.html` - Add slug (readonly) and media -- `templates/partials/note.html` - Add microformats markup -- `templates/public/index.html` - Add h-feed container -- `starpunk/routes/admin.py` - Handle slugs and uploads -- `starpunk/routes/auth.py` - Trigger discovery on login -- `starpunk/models/note.py` - Add media relationship - -## Success Metrics - -Implementation is complete when: - -1. β Custom slug can be specified on creation -2. β Images can be uploaded and displayed -3. β Author info is discovered from IndieAuth profile -4. β IndieWebify.me validates h-feed and h-entry -5. β All tests pass -6. β No regressions in existing functionality -7. β Media files are tracked in database -8. β Errors are handled gracefully - -## Final Notes - -- Keep it simple - this is v1.2.0, not v2.0.0 -- Data preservation over premature optimization -- When uncertain, choose the more explicit option -- Document any deviations from this guidance - ---- - -This Q&A document serves as the authoritative implementation guide for v1.2.0. Any questions not covered here should follow the principle of maximum simplicity. \ No newline at end of file diff --git a/docs/design/v1.2.0/feature-specification.md b/docs/design/v1.2.0/feature-specification.md deleted file mode 100644 index 200e964..0000000 --- a/docs/design/v1.2.0/feature-specification.md +++ /dev/null @@ -1,872 +0,0 @@ -# v1.2.0 Feature Specification - -## Overview - -Version 1.2.0 focuses on three essential improvements to the StarPunk web interface: -1. Custom slug support in the web UI -2. Media upload capability (web UI only, not Micropub) -3. Complete Microformats2 implementation - -## Feature 1: Custom Slugs in Web UI - -### Current State -- Slugs are auto-generated from the first line of content -- Custom slugs only possible via Micropub API (mp-slug property) -- Web UI has no option to specify custom slugs - -### Requirements -- Add optional "Slug" field to note creation form -- Validate slug format (URL-safe, unique) -- If empty, fall back to auto-generation -- Support custom slugs in edit form as well - -### Design Specification - -#### Form Updates -Location: `templates/admin/new.html` and `templates/admin/edit.html` - -Add new form field: -```html -
-
- Note text content here...
-{{ author.bio }}
- {% endif %} -` element + +This is already the behavior via Jinja2 conditionals. + +--- + +## 3. h-feed Minimum Properties + +### 3.1 Required Properties (per spec) + +The h-feed specification requires: +- `p-name` - Feed title +- `p-author` - Feed author (h-card) +- `u-url` - Feed URL +- `u-photo` - Feed photo (author photo) + +### 3.2 Current State Analysis + +Current `templates/index.html`: +```html +
No notes with this tag.
+ {% endif %} +