Phil Skentelbery
f10d0679da
Implements tag/category system backend following microformats2 p-category specification. Database changes: - Migration 008: Add tags and note_tags tables - Normalized tag storage (case-insensitive lookup, display name preserved) - Indexes for performance New module: - starpunk/tags.py: Tag management functions - normalize_tag: Normalize tag strings - get_or_create_tag: Get or create tag records - add_tags_to_note: Associate tags with notes (replaces existing) - get_note_tags: Retrieve note tags (alphabetically ordered) - get_tag_by_name: Lookup tag by normalized name - get_notes_by_tag: Get all notes with specific tag - parse_tag_input: Parse comma-separated tag input Model updates: - Note.tags property (lazy-loaded, prefer pre-loading in routes) - Note.to_dict() add include_tags parameter CRUD updates: - create_note() accepts tags parameter - update_note() accepts tags parameter (None = no change, [] = remove all) Micropub integration: - Pass tags to create_note() (tags already extracted by extract_tags()) - Return tags in q=source response Per design doc: docs/design/v1.3.0/microformats-tags-design.md Generated with Claude Code Co-Authored-By: Claude <noreply@anthropic.com>
375 lines
13 KiB
Markdown
375 lines
13 KiB
Markdown
# StarPunk v1.1.0 Feature Architecture
|
|
|
|
## Overview
|
|
This document defines the architectural design for the three major features in v1.1.0: Migration System Redesign, Full-Text Search, and Custom Slugs. Each component has been designed following our core principle of minimal, elegant solutions.
|
|
|
|
## System Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ StarPunk CMS v1.1.0 │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
|
│ │ Micropub │ │ Web UI │ │ Search API │ │
|
|
│ │ Endpoint │ │ │ │ /api/search │ │
|
|
│ └──────┬──────┘ └──────┬───────┘ └────────┬─────────┘ │
|
|
│ │ │ │ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌──────────────────────────────────────────────────────┐ │
|
|
│ │ Application Layer │ │
|
|
│ │ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │
|
|
│ │ │ Custom │ │ Note │ │ Search │ │ │
|
|
│ │ │ Slugs │ │ CRUD │ │ Engine │ │ │
|
|
│ │ └────────────┘ └────────────┘ └────────────────┘ │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────────────────────────────────────────┐ │
|
|
│ │ Data Layer (SQLite) │ │
|
|
│ │ ┌────────────┐ ┌────────────┐ ┌────────────────┐ │ │
|
|
│ │ │ notes │ │ notes_fts │ │ migrations │ │ │
|
|
│ │ │ table │◄─┤ (FTS5) │ │ table │ │ │
|
|
│ │ └────────────┘ └────────────┘ └────────────────┘ │ │
|
|
│ │ │ ▲ │ │ │
|
|
│ │ └──────────────┴───────────────────┘ │ │
|
|
│ │ Triggers keep FTS in sync │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌──────────────────────────────────────────────────────┐ │
|
|
│ │ File System Layer │ │
|
|
│ │ data/notes/YYYY/MM/[slug].md │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Component Architecture
|
|
|
|
### 1. Migration System Redesign
|
|
|
|
#### Current Problem
|
|
```
|
|
[Fresh Install] [Upgrade Path]
|
|
│ │
|
|
▼ ▼
|
|
SCHEMA_SQL Migration Files
|
|
(full schema) (partial schema)
|
|
│ │
|
|
└────────┬───────────────┘
|
|
▼
|
|
DUPLICATION!
|
|
```
|
|
|
|
#### New Architecture
|
|
```
|
|
[Fresh Install] [Upgrade Path]
|
|
│ │
|
|
▼ ▼
|
|
INITIAL_SCHEMA_SQL ──────► Migrations
|
|
(v1.0.0 only) (changes only)
|
|
│ │
|
|
└────────┬───────────────┘
|
|
▼
|
|
Single Source
|
|
```
|
|
|
|
#### Key Components
|
|
- **INITIAL_SCHEMA_SQL**: Frozen v1.0.0 schema
|
|
- **Migration Files**: Only incremental changes
|
|
- **Migration Runner**: Handles both paths intelligently
|
|
|
|
### 2. Full-Text Search Architecture
|
|
|
|
#### Data Flow
|
|
```
|
|
1. User Query
|
|
│
|
|
▼
|
|
2. Query Parser
|
|
│
|
|
▼
|
|
3. FTS5 Engine ───► SQLite Query Planner
|
|
│ │
|
|
▼ ▼
|
|
4. BM25 Ranking Index Lookup
|
|
│ │
|
|
└──────────┬───────────┘
|
|
▼
|
|
5. Results + Snippets
|
|
```
|
|
|
|
#### Database Schema
|
|
```sql
|
|
notes (main table) notes_fts (virtual table)
|
|
┌──────────────┐ ┌──────────────────┐
|
|
│ id (PK) │◄───────────┤ rowid (FK) │
|
|
│ slug │ │ slug (UNINDEXED) │
|
|
│ content │───trigger──► title │
|
|
│ published │ │ content │
|
|
└──────────────┘ └──────────────────┘
|
|
```
|
|
|
|
#### Synchronization Strategy
|
|
- **INSERT Trigger**: Automatically indexes new notes
|
|
- **UPDATE Trigger**: Re-indexes modified notes
|
|
- **DELETE Trigger**: Removes deleted notes from index
|
|
- **Initial Build**: One-time indexing of existing notes
|
|
|
|
### 3. Custom Slugs Architecture
|
|
|
|
#### Request Flow
|
|
```
|
|
Micropub Request
|
|
│
|
|
▼
|
|
Extract mp-slug ──► No mp-slug ──► Auto-generate
|
|
│ │
|
|
▼ │
|
|
Validate Format │
|
|
│ │
|
|
▼ │
|
|
Check Uniqueness │
|
|
│ │
|
|
├─► Unique ────────────────────┤
|
|
│ │
|
|
└─► Duplicate │
|
|
│ │
|
|
▼ ▼
|
|
Add suffix Create Note
|
|
(my-slug-2)
|
|
```
|
|
|
|
#### Validation Pipeline
|
|
```
|
|
Input: "My/Cool/../Post!"
|
|
│
|
|
▼
|
|
1. Lowercase: "my/cool/../post!"
|
|
│
|
|
▼
|
|
2. Remove Invalid: "my/cool/post"
|
|
│
|
|
▼
|
|
3. Security Check: Reject "../"
|
|
│
|
|
▼
|
|
4. Pattern Match: ^[a-z0-9-/]+$
|
|
│
|
|
▼
|
|
5. Reserved Check: Not in blocklist
|
|
│
|
|
▼
|
|
Output: "my-cool-post"
|
|
```
|
|
|
|
## Data Models
|
|
|
|
### Migration Record
|
|
```python
|
|
class Migration:
|
|
version: str # "001", "002", etc.
|
|
description: str # Human-readable
|
|
applied_at: datetime
|
|
checksum: str # Verify integrity
|
|
```
|
|
|
|
### Search Result
|
|
```python
|
|
class SearchResult:
|
|
slug: str
|
|
title: str
|
|
snippet: str # With <mark> highlights
|
|
rank: float # BM25 score
|
|
published: bool
|
|
created_at: datetime
|
|
```
|
|
|
|
### Slug Validation
|
|
```python
|
|
class SlugValidator:
|
|
pattern: regex = r'^[a-z0-9-/]+$'
|
|
max_length: int = 200
|
|
reserved: set = {'api', 'admin', 'auth', 'feed'}
|
|
|
|
def validate(slug: str) -> bool
|
|
def sanitize(slug: str) -> str
|
|
def ensure_unique(slug: str) -> str
|
|
```
|
|
|
|
## Interface Specifications
|
|
|
|
### Search API Contract
|
|
```yaml
|
|
endpoint: GET /api/search
|
|
parameters:
|
|
q: string (required) - Search query
|
|
limit: int (optional, default: 20, max: 100)
|
|
offset: int (optional, default: 0)
|
|
published_only: bool (optional, default: true)
|
|
|
|
response:
|
|
200 OK:
|
|
content-type: application/json
|
|
schema:
|
|
query: string
|
|
total: integer
|
|
results: array[SearchResult]
|
|
|
|
400 Bad Request:
|
|
error: "invalid_query"
|
|
description: string
|
|
```
|
|
|
|
### Micropub Slug Extension
|
|
```yaml
|
|
property: mp-slug
|
|
type: string
|
|
required: false
|
|
validation:
|
|
- URL-safe characters only
|
|
- Maximum 200 characters
|
|
- Not in reserved list
|
|
- Unique (or auto-incremented)
|
|
|
|
example:
|
|
properties:
|
|
content: ["My post"]
|
|
mp-slug: ["my-custom-url"]
|
|
```
|
|
|
|
## Performance Characteristics
|
|
|
|
### Migration System
|
|
- Fresh install: ~100ms (schema + migrations)
|
|
- Upgrade: ~50ms per migration
|
|
- Rollback: Not supported (forward-only)
|
|
|
|
### Full-Text Search
|
|
- Index build: 1ms per note
|
|
- Query latency: <10ms for 10K notes
|
|
- Index size: ~30% of text
|
|
- Memory usage: Negligible (SQLite managed)
|
|
|
|
### Custom Slugs
|
|
- Validation: <1ms
|
|
- Uniqueness check: <5ms
|
|
- Conflict resolution: <10ms
|
|
- No performance impact on existing flows
|
|
|
|
## Security Architecture
|
|
|
|
### Search Security
|
|
1. **Input Sanitization**: FTS5 handles SQL injection
|
|
2. **Output Escaping**: HTML escaped in snippets
|
|
3. **Rate Limiting**: 100 requests/minute per IP
|
|
4. **Access Control**: Unpublished notes require auth
|
|
|
|
### Slug Security
|
|
1. **Path Traversal Prevention**: Reject `..` patterns
|
|
2. **Reserved Routes**: Block system endpoints
|
|
3. **Length Limits**: Prevent DoS via long slugs
|
|
4. **Character Whitelist**: Only allow safe chars
|
|
|
|
### Migration Security
|
|
1. **Checksum Verification**: Detect tampering
|
|
2. **Transaction Safety**: All-or-nothing execution
|
|
3. **No User Input**: Migrations are code-only
|
|
4. **Audit Trail**: Track all applied migrations
|
|
|
|
## Deployment Considerations
|
|
|
|
### Database Upgrade Path
|
|
```bash
|
|
# v1.0.x → v1.1.0
|
|
1. Backup database
|
|
2. Apply migration 002 (FTS5 tables)
|
|
3. Build initial search index
|
|
4. Verify functionality
|
|
5. Remove backup after confirmation
|
|
```
|
|
|
|
### Rollback Strategy
|
|
```bash
|
|
# Emergency rollback (data preserved)
|
|
1. Stop application
|
|
2. Restore v1.0.x code
|
|
3. Database remains compatible
|
|
4. FTS tables ignored by old code
|
|
5. Custom slugs work as regular slugs
|
|
```
|
|
|
|
### Container Deployment
|
|
```dockerfile
|
|
# No changes to container required
|
|
# SQLite FTS5 included by default
|
|
# No new dependencies added
|
|
```
|
|
|
|
## Testing Strategy
|
|
|
|
### Unit Test Coverage
|
|
- Migration path logic: 100%
|
|
- Slug validation: 100%
|
|
- Search query parsing: 100%
|
|
- Trigger behavior: 100%
|
|
|
|
### Integration Test Scenarios
|
|
1. Fresh installation flow
|
|
2. Upgrade from each version
|
|
3. Search with special characters
|
|
4. Micropub with various slugs
|
|
5. Concurrent note operations
|
|
|
|
### Performance Benchmarks
|
|
- 1,000 notes: <5ms search
|
|
- 10,000 notes: <10ms search
|
|
- 100,000 notes: <50ms search
|
|
- Index size: Confirm ~30% ratio
|
|
|
|
## Monitoring & Observability
|
|
|
|
### Key Metrics
|
|
1. Search query latency (p50, p95, p99)
|
|
2. Index size growth rate
|
|
3. Slug conflict frequency
|
|
4. Migration execution time
|
|
|
|
### Log Events
|
|
```python
|
|
# Search
|
|
INFO: "Search query: {query}, results: {count}, latency: {ms}"
|
|
|
|
# Slugs
|
|
WARN: "Slug conflict resolved: {original} → {final}"
|
|
|
|
# Migrations
|
|
INFO: "Migration {version} applied in {ms}ms"
|
|
ERROR: "Migration {version} failed: {error}"
|
|
```
|
|
|
|
## Future Considerations
|
|
|
|
### Potential Enhancements
|
|
1. **Search Filters**: by date, author, tags
|
|
2. **Hierarchical Slugs**: `/2024/11/25/post`
|
|
3. **Migration Rollback**: Bi-directional migrations
|
|
4. **Search Suggestions**: Auto-complete support
|
|
|
|
### Scaling Considerations
|
|
1. **Search Index Sharding**: If >1M notes
|
|
2. **External Search**: Meilisearch for multi-user
|
|
3. **Slug Namespaces**: Per-user slug spaces
|
|
4. **Migration Parallelization**: For large datasets
|
|
|
|
## Conclusion
|
|
|
|
The v1.1.0 architecture maintains StarPunk's commitment to minimalism while adding essential features. Each component:
|
|
- Solves a specific user need
|
|
- Uses standard, proven technologies
|
|
- Avoids external dependencies
|
|
- Maintains backward compatibility
|
|
- Follows the principle: "Every line of code must justify its existence"
|
|
|
|
The architecture is designed to be understood, maintained, and extended by a single developer, staying true to the IndieWeb philosophy of personal publishing platforms. |