phil/StarPunk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(tags): Add database schema and tags module (v1.3.0 Phase 1)

Browse Source 
Implements tag/category system backend following microformats2 p-category specification.

Database changes:
- Migration 008: Add tags and note_tags tables
- Normalized tag storage (case-insensitive lookup, display name preserved)
- Indexes for performance

New module:
- starpunk/tags.py: Tag management functions
  - normalize_tag: Normalize tag strings
  - get_or_create_tag: Get or create tag records
  - add_tags_to_note: Associate tags with notes (replaces existing)
  - get_note_tags: Retrieve note tags (alphabetically ordered)
  - get_tag_by_name: Lookup tag by normalized name
  - get_notes_by_tag: Get all notes with specific tag
  - parse_tag_input: Parse comma-separated tag input

Model updates:
- Note.tags property (lazy-loaded, prefer pre-loading in routes)
- Note.to_dict() add include_tags parameter

CRUD updates:
- create_note() accepts tags parameter
- update_note() accepts tags parameter (None = no change, [] = remove all)

Micropub integration:
- Pass tags to create_note() (tags already extracted by extract_tags())
- Return tags in q=source response

Per design doc: docs/design/v1.3.0/microformats-tags-design.md

Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Phil Skentelbery
2025-12-10 11:24:23 -07:00
parent 927db4aea0
commit f10d0679da
188 changed files with 601 additions and 945 deletions
Download Patch File Download Diff File Expand all files Collapse all files

572
docs/design/v1.1.1/v1.1.0-validation-report.md Normal file
View File

@@ -0,0 +1,572 @@
# StarPunk v1.1.0 Implementation Validation & Search UI Design
**Date**: 2025-11-25
**Architect**: Claude (StarPunk Architect Agent)
**Status**: Review Complete
## Executive Summary
The v1.1.0 implementation by the developer is **APPROVED** with minor suggestions. All four completed components meet architectural requirements and maintain backward compatibility. The deferred Search UI components have been fully specified below for implementation.
## Part 1: Implementation Validation
### 1. RSS Feed Fix
**Status**: ✅ **Approved**
**Review Findings**:
- Line 97 in `starpunk/feed.py` correctly applies `reversed()` to compensate for feedgen's internal ordering
- Regression test `test_generate_feed_newest_first()` adequately verifies correct ordering
- Test creates 3 notes with distinct timestamps and verifies both database and feed ordering
- Clear comment explains the feedgen behavior requiring the fix
**Code Quality**:
- Minimal change (single line with `reversed()`)
- Well-documented with explanatory comment
- Comprehensive regression test prevents future issues
**Approval**: Ready as-is. The fix is elegant and properly tested.
### 2. Migration System Redesign
**Status**: ✅ **Approved**
**Review Findings**:
- `SCHEMA_SQL` renamed to `INITIAL_SCHEMA_SQL` in `database.py` (line 13)
- Clear documentation: "DO NOT MODIFY - This represents the v1.0.0 schema state"
- Comment properly directs future changes to migration files
- No functional changes, purely documentation improvement
**Architecture Alignment**:
- Follows ADR-033's philosophy of frozen baseline schema
- Makes intent clear for future developers
- Prevents accidental modifications to baseline
**Approval**: Ready as-is. The rename clarifies intent without breaking changes.
### 3. Full-Text Search (Core)
**Status**: ✅ **Approved with minor suggestions**
**Review Findings**:
**Migration (005_add_fts5_search.sql)**:
- FTS5 virtual table schema is correct
- Porter stemming and Unicode61 tokenizer appropriate for international support
- DELETE trigger correctly handles cleanup
- Good documentation explaining why INSERT/UPDATE triggers aren't used
**Search Module (search.py)**:
- Well-structured with clear separation of concerns
- `check_fts5_support()`: Properly tests FTS5 availability
- `update_fts_index()`: Correctly extracts title and updates index
- `search_notes()`: Implements ranking and snippet generation
- `rebuild_fts_index()`: Provides recovery mechanism
- Graceful degradation implemented throughout
**Integration (notes.py)**:
- Lines 299-307: FTS update after create with proper error handling
- Lines 699-708: FTS update after content change with proper error handling
- Graceful degradation ensures note operations succeed even if FTS fails
**Minor Suggestions**:
1. Consider adding a config flag `ENABLE_FTS` to allow disabling FTS entirely
2. The 100-character title truncation (line 94 in search.py) could be configurable
3. Consider logging FTS rebuild progress for large datasets
**Approval**: Approved. Core functionality is solid with excellent error handling.
### 4. Custom Slugs
**Status**: ✅ **Approved**
**Review Findings**:
**Slug Utils Module (slug_utils.py)**:
- Comprehensive `RESERVED_SLUGS` list protects application routes
- `sanitize_slug()`: Properly converts to valid format
- `validate_slug()`: Strong validation with regex pattern
- `make_slug_unique_with_suffix()`: Sequential numbering is predictable and clean
- `validate_and_sanitize_custom_slug()`: Full validation pipeline
**Security**:
- Path traversal prevented by rejecting `/` in slugs
- Reserved slugs protect application routes
- Max length enforced (200 chars)
- Proper sanitization prevents injection attacks
**Integration**:
- Notes.py (lines 217-223): Proper custom slug handling
- Micropub.py (lines 300-304): Correct mp-slug extraction
- Error messages are clear and actionable
**Architecture Alignment**:
- Sequential suffixes (-2, -3) are predictable for users
- Hierarchical slugs properly deferred to v1.2.0
- Maintains backward compatibility with auto-generation
**Approval**: Ready as-is. Implementation is secure and well-designed.
### 5. Testing & Overall Quality
**Test Coverage**: 556 tests passing (1 flaky timing test unrelated to v1.1.0)
**Version Management**:
- Version correctly bumped to 1.1.0 in `__init__.py`
- CHANGELOG.md properly documents all changes
- Semantic versioning followed correctly
**Backward Compatibility**: 100% maintained
- Existing notes work unchanged
- Micropub clients need no modifications
- Database migrations handle all upgrade paths
## Part 2: Search UI Design Specification
### A. Search API Endpoint
**File**: Create new `starpunk/routes/search.py`
```python
# Route Definition
@app.route('/api/search', methods=['GET'])
def api_search():
"""
Search API endpoint
Query Parameters:
q (required): Search query string
limit (optional): Results limit, default 20, max 100
offset (optional): Pagination offset, default 0
Returns:
JSON response with search results
Status Codes:
200: Success (even with 0 results)
400: Bad request (empty query)
503: Service unavailable (FTS5 not available)
"""
```
**Request Validation**:
```python
# Extract and validate parameters
query = request.args.get('q', '').strip()
if not query:
return jsonify({
'error': 'Missing required parameter: q',
'message': 'Search query cannot be empty'
}), 400
# Parse limit with bounds checking
try:
limit = min(int(request.args.get('limit', 20)), 100)
if limit < 1:
limit = 20
except ValueError:
limit = 20
# Parse offset
try:
offset = max(int(request.args.get('offset', 0)), 0)
except ValueError:
offset = 0
```
**Authentication Consideration**:
```python
# Check if user is authenticated (for unpublished notes)
from starpunk.auth import get_current_user
user = get_current_user()
published_only = (user is None) # Anonymous users see only published
```
**Search Execution**:
```python
from starpunk.search import search_notes, has_fts_table
from pathlib import Path
db_path = Path(app.config['DATABASE_PATH'])
# Check FTS availability
if not has_fts_table(db_path):
return jsonify({
'error': 'Search unavailable',
'message': 'Full-text search is not configured on this server'
}), 503
try:
results = search_notes(
query=query,
db_path=db_path,
published_only=published_only,
limit=limit,
offset=offset
)
except Exception as e:
app.logger.error(f"Search failed: {e}")
return jsonify({
'error': 'Search failed',
'message': 'An error occurred during search'
}), 500
```
**Response Format**:
```python
# Format response
response = {
'query': query,
'count': len(results),
'limit': limit,
'offset': offset,
'results': [
{
'slug': r['slug'],
'title': r['title'] or f"Note from {r['created_at'][:10]}",
'excerpt': r['snippet'], # Already has <mark> tags
'published_at': r['created_at'],
'url': f"/notes/{r['slug']}"
}
for r in results
]
}
return jsonify(response), 200
```
### B. Search Box UI Component
**File to Modify**: `templates/base.html`
**Location**: In the navigation bar, after the existing nav links
**HTML Structure**:
```html
<!-- Add to navbar after existing nav items, before auth section -->
<form class="d-flex ms-auto me-3" action="/search" method="get" role="search">
<input
class="form-control form-control-sm me-2"
type="search"
name="q"
placeholder="Search notes..."
aria-label="Search"
value="{{ request.args.get('q', '') }}"
minlength="2"
maxlength="100"
required
>
<button class="btn btn-outline-secondary btn-sm" type="submit">
<i class="bi bi-search"></i>
</button>
</form>
```
**Behavior**:
- Form submission (full page load, no AJAX for v1.1.0)
- Minimum query length: 2 characters (HTML5 validation)
- Maximum query length: 100 characters
- Preserves query in search box when on search results page
### C. Search Results Page
**File**: Create new `templates/search.html`
```html
{% extends "base.html" %}
{% block title %}Search{% if query %}: {{ query }}{% endif %} - {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<div class="container py-4">
<div class="row">
<div class="col-lg-8 mx-auto">
<!-- Search Header -->
<div class="mb-4">
<h1 class="h3">Search Results</h1>
{% if query %}
<p class="text-muted">
Found {{ results|length }} result{{ 's' if results|length != 1 else '' }}
for "<strong>{{ query }}</strong>"
</p>
{% endif %}
</div>
<!-- Search Form (for new searches) -->
<div class="card mb-4">
<div class="card-body">
<form action="/search" method="get" role="search">
<div class="input-group">
<input
type="search"
class="form-control"
name="q"
placeholder="Enter search terms..."
value="{{ query }}"
minlength="2"
maxlength="100"
required
autofocus
>
<button class="btn btn-primary" type="submit">
Search
</button>
</div>
</form>
</div>
</div>
<!-- Results -->
{% if query %}
{% if results %}
<div class="search-results">
{% for result in results %}
<article class="card mb-3">
<div class="card-body">
<h2 class="h5 card-title">
<a href="{{ result.url }}" class="text-decoration-none">
{{ result.title }}
</a>
</h2>
<div class="card-text">
<!-- Excerpt with highlighted terms (safe because we control the <mark> tags) -->
<p class="mb-2">{{ result.excerpt|safe }}</p>
<small class="text-muted">
<time datetime="{{ result.published_at }}">
{{ result.published_at|format_date }}
</time>
</small>
</div>
</div>
</article>
{% endfor %}
</div>
<!-- Pagination (if more than limit results possible) -->
{% if results|length == limit %}
<nav aria-label="Search pagination">
<ul class="pagination justify-content-center">
{% if offset > 0 %}
<li class="page-item">
<a class="page-link" href="/search?q={{ query|urlencode }}&offset={{ max(0, offset - limit) }}">
Previous
</a>
</li>
{% endif %}
<li class="page-item">
<a class="page-link" href="/search?q={{ query|urlencode }}&offset={{ offset + limit }}">
Next
</a>
</li>
</ul>
</nav>
{% endif %}
{% else %}
<!-- No results -->
<div class="alert alert-info" role="alert">
<h4 class="alert-heading">No results found</h4>
<p>Your search for "<strong>{{ query }}</strong>" didn't match any notes.</p>
<hr>
<p class="mb-0">Try different keywords or check your spelling.</p>
</div>
{% endif %}
{% else %}
<!-- No query yet -->
<div class="text-center text-muted py-5">
<i class="bi bi-search" style="font-size: 3rem;"></i>
<p class="mt-3">Enter search terms above to find notes</p>
</div>
{% endif %}
<!-- Error state (if search unavailable) -->
{% if error %}
<div class="alert alert-warning" role="alert">
<h4 class="alert-heading">Search Unavailable</h4>
<p>{{ error }}</p>
<hr>
<p class="mb-0">Full-text search is temporarily unavailable. Please try again later.</p>
</div>
{% endif %}
</div>
</div>
</div>
{% endblock %}
```
**Route Handler**: Add to `starpunk/routes/search.py`
```python
@app.route('/search')
def search_page():
"""
Search results HTML page
"""
query = request.args.get('q', '').strip()
limit = 20 # Fixed for HTML view
offset = 0
try:
offset = max(int(request.args.get('offset', 0)), 0)
except ValueError:
offset = 0
# Check authentication for unpublished notes
from starpunk.auth import get_current_user
user = get_current_user()
published_only = (user is None)
results = []
error = None
if query:
from starpunk.search import search_notes, has_fts_table
from pathlib import Path
db_path = Path(app.config['DATABASE_PATH'])
if not has_fts_table(db_path):
error = "Full-text search is not configured on this server"
else:
try:
results = search_notes(
query=query,
db_path=db_path,
published_only=published_only,
limit=limit,
offset=offset
)
except Exception as e:
app.logger.error(f"Search failed: {e}")
error = "An error occurred during search"
return render_template(
'search.html',
query=query,
results=results,
error=error,
limit=limit,
offset=offset
)
```
### D. Integration Points
1. **Route Registration**: In `starpunk/routes/__init__.py`, add:
```python
from starpunk.routes.search import register_search_routes
register_search_routes(app)
```
2. **Template Filter**: Add to `starpunk/app.py` or template filters:
```python
@app.template_filter('format_date')
def format_date(date_string):
"""Format ISO date for display"""
from datetime import datetime
try:
dt = datetime.fromisoformat(date_string.replace('Z', '+00:00'))
return dt.strftime('%B %d, %Y')
except:
return date_string
```
3. **App Startup FTS Index**: Add to `create_app()` after database init:
```python
# Initialize FTS index if needed
from starpunk.search import has_fts_table, rebuild_fts_index
from pathlib import Path
db_path = Path(app.config['DATABASE_PATH'])
data_path = Path(app.config['DATA_PATH'])
if has_fts_table(db_path):
# Check if index is empty (fresh migration)
import sqlite3
conn = sqlite3.connect(db_path)
count = conn.execute("SELECT COUNT(*) FROM notes_fts").fetchone()[0]
conn.close()
if count == 0:
app.logger.info("Populating FTS index on first run...")
try:
rebuild_fts_index(db_path, data_path)
except Exception as e:
app.logger.error(f"Failed to populate FTS index: {e}")
```
### E. Testing Requirements
**Unit Tests** (`tests/test_search_api.py`):
```python
def test_search_api_requires_query()
def test_search_api_validates_limit()
def test_search_api_returns_results()
def test_search_api_handles_no_results()
def test_search_api_respects_authentication()
def test_search_api_handles_fts_unavailable()
```
**Integration Tests** (`tests/test_search_integration.py`):
```python
def test_search_page_renders()
def test_search_page_displays_results()
def test_search_page_handles_no_results()
def test_search_page_pagination()
def test_search_box_in_navigation()
```
**Security Tests**:
```python
def test_search_prevents_xss_in_query()
def test_search_prevents_sql_injection()
def test_search_escapes_html_in_results()
def test_search_respects_published_status()
```
## Implementation Recommendations
### Priority Order
1. Implement `/api/search` endpoint first (enables programmatic access)
2. Add search box to base.html navigation
3. Create search results page template
4. Add FTS index population on startup
5. Write comprehensive tests
### Estimated Effort
- API Endpoint: 1 hour
- Search UI (box + results page): 1.5 hours
- FTS startup population: 0.5 hours
- Testing: 1 hour
- **Total: 4 hours**
### Performance Considerations
1. FTS5 queries are fast but consider caching frequent searches
2. Limit default results to 20 for HTML view
3. Add index on `notes_fts(rank)` if performance issues arise
4. Consider async FTS index updates for large notes
### Security Notes
1. Always escape user input in templates
2. Use `|safe` filter only for our controlled `<mark>` tags
3. Validate query length to prevent DoS
4. Rate limiting recommended for production (not required for v1.1.0)
## Conclusion
The v1.1.0 implementation is **APPROVED** for release pending Search UI completion. The developer has delivered high-quality, well-tested code that maintains architectural principles and backward compatibility.
The Search UI specifications provided above are complete and ready for implementation. Following these specifications will result in a fully functional search feature that integrates seamlessly with the existing FTS5 implementation.
### Next Steps
1. Developer implements Search UI per specifications (4 hours)
2. Run full test suite including new search tests
3. Update version and CHANGELOG if needed
4. Create v1.1.0-rc.1 release candidate
5. Deploy and test in staging environment
6. Release v1.1.0
---
**Architect Sign-off**: ✅ Approved
**Date**: 2025-11-25
**StarPunk Architect Agent**