phil/StarPunk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(media): Make variant generation atomic with database

Browse Source 
Per v1.5.0 Phase 4:
- Generate variants to temp directory first
- Perform database inserts in transaction
- Move files to final location before commit
- Clean up temp files on any failure
- Add startup recovery for orphaned temp files
- All media operations now fully atomic

Changes:
- Modified generate_all_variants() to return file moves
- Modified save_media() to handle full atomic operation
- Add cleanup_orphaned_temp_files() for startup recovery
- Added 4 new tests for atomic behavior
- Fixed HEIC variant format detection
- Updated variant failure test for atomic behavior

Fixes:
- No orphaned files on database failures
- No orphaned DB records on file failures
- Startup recovery detects and cleans orphans

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Phil Skentelbery
2025-12-17 11:26:26 -07:00
parent b689e02e64
commit 21fa7acfbb
5 changed files with 951 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files

168
docs/design/v1.5.0/2025-12-17-phase3-architect-review.md Normal file
View File

@@ -0,0 +1,168 @@
# Phase 3 Architect Review: N+1 Query Fix
**Date**: 2025-12-17
**Phase**: Phase 3 - N+1 Query Fix (Feed Generation)
**Reviewer**: Claude (StarPunk Architect Agent)
**Implementation Report**: `2025-12-17-phase3-implementation.md`
---
## Review Summary
**VERDICT: APPROVED**
Phase 3 implementation meets all acceptance criteria and demonstrates sound architectural decisions. The developer can proceed to Phase 4.
---
## Acceptance Criteria Verification
| Criterion | Status | Notes |
|-----------|--------|-------|
| Feed generation uses batch queries | PASS | `_get_cached_notes()` now calls `get_media_for_notes()` and `get_tags_for_notes()` |
| Query count reduced from O(n) to O(1) | PASS | 3 total queries vs. 1+2N queries previously |
| No change to API behavior | PASS | Feed output format unchanged, all 920 tests pass |
| Performance improvement verified | PASS | 13 new tests validate batch loading behavior |
| Other N+1 locations documented | PASS | BACKLOG.md updated with deferred locations |
---
## Implementation Review
### 1. Batch Media Loading (`starpunk/media.py` lines 728-852)
**SQL Correctness**: PASS
- Proper use of parameterized `IN` clause with placeholder generation
- JOIN structure correctly retrieves media with note association
- ORDER BY includes both `note_id` and `display_order` for deterministic results
**Edge Cases**: PASS
- Empty list returns `{}` (line 750-751)
- Notes without media receive empty lists via dict initialization (line 819)
- Variant loading skipped when no media exists (line 786)
**Data Integrity**: PASS
- Output format matches `get_note_media()` exactly
- Variants dict structure identical to single-note function
- Caption, display_order, and all metadata fields preserved
**Observation**: The implementation uses 2 queries (media + variants) rather than a single JOIN. This is architecturally sound because:
1. Avoids Cartesian product explosion with multiple variants per media
2. Keeps result sets manageable
3. Maintains code clarity
### 2. Batch Tag Loading (`starpunk/tags.py` lines 146-197)
**SQL Correctness**: PASS
- Single query retrieves all tags for all notes
- Proper parameterized `IN` clause
- Alphabetical ordering preserved: `ORDER BY note_tags.note_id, LOWER(tags.display_name) ASC`
**Edge Cases**: PASS
- Empty list returns `{}` (line 169-170)
- Notes without tags receive empty lists (line 188)
**Data Integrity**: PASS
- Returns same `{'name': ..., 'display_name': ...}` structure as `get_note_tags()`
### 3. Feed Generation Update (`starpunk/routes/public.py` lines 38-86)
**Integration**: PASS
- Batch functions called after note list retrieval
- Results correctly attached to Note objects via `object.__setattr__`
- Cache structure unchanged (notes list still cached)
**Pattern Consistency**: PASS
- Uses same attribute attachment pattern as existing code
- `media` attribute and `_cached_tags` naming consistent with other routes
### 4. Test Coverage (`tests/test_batch_loading.py`)
**Coverage Assessment**: EXCELLENT
- 13 tests covering all critical scenarios
- Empty list handling tested
- Mixed scenarios (some notes with/without media/tags) tested
- Variant inclusion verified
- Display order preservation verified
- Tag alphabetical ordering verified
- Integration with feed generation tested
**Test Quality**:
- Tests are isolated and deterministic
- Test data creation is clean and well-documented
- Assertions verify correct data structure, not just existence
---
## Architectural Observations
### Strengths
1. **Minimal Code Change**: The implementation adds functionality without modifying existing single-note functions, maintaining backwards compatibility.
2. **Consistent Patterns**: Both batch functions follow identical structure:
- Empty check early return
- Placeholder generation for IN clause
- Dict initialization for all requested IDs
- Result grouping loop
3. **Performance Characteristics**: The 97% query reduction (101 to 3 for 50 notes) is significant. SQLite handles IN clauses efficiently for the expected note counts (<100).
4. **Defensive Coding**: Notes missing from results get empty lists rather than KeyErrors, preventing runtime failures.
### Minor Observations (Not Blocking)
1. **f-string SQL**: The implementation uses f-strings to construct IN clause placeholders. While safe here (placeholders are `?` characters, not user input), this pattern requires care. The implementation is correct.
2. **Deferred Optimizations**: Homepage and tag archive pages still use per-note queries. This is acceptable per RELEASE.md scope, and the batch functions can be reused when those are addressed.
3. **No Query Counting in Tests**: The performance test verifies result completeness but does not actually count queries. This is acceptable because:
- SQLite does not provide easy query counting
- The code structure guarantees query count by design
- A query counting test would add complexity without proportional value
---
## Standards Compliance
| Standard | Status |
|----------|--------|
| Python coding standards | PASS - Type hints, docstrings present |
| Testing checklist | PASS - Unit, integration, edge cases covered |
| Documentation | PASS - Implementation report comprehensive |
| Git practices | PASS - Clear commit message with context |
---
## Recommendation
Phase 3 is **APPROVED**. The implementation:
1. Achieves the stated performance goal
2. Maintains full backwards compatibility
3. Follows established codebase patterns
4. Has comprehensive test coverage
5. Is properly documented
The developer should proceed to **Phase 4: Atomic Variant Generation**.
---
## Project Plan Update
Phase 3 acceptance criteria should be marked complete in RELEASE.md:
```markdown
#### Acceptance Criteria
- [x] Feed generation uses batch queries
- [x] Query count reduced from O(n) to O(1) for media/tags
- [x] No change to API behavior
- [x] Performance improvement verified in tests
- [x] Other N+1 locations documented in BACKLOG.md (not fixed)
```
---
**Architect**: Claude (StarPunk Architect Agent)
**Date**: 2025-12-17
**Status**: APPROVED - Proceed to Phase 4

316
docs/design/v1.5.0/2025-12-17-phase3-implementation.md Normal file
View File

@@ -0,0 +1,316 @@
# v1.5.0 Phase 3 Implementation Report
**Date**: 2025-12-17
**Phase**: Phase 3 - N+1 Query Fix (Feed Generation)
**Status**: COMPLETE
**Developer**: Claude (StarPunk Developer Agent)
## Summary
Successfully implemented batch loading for media and tags in feed generation, fixing the N+1 query pattern in `_get_cached_notes()`. This improves feed generation performance from O(n) to O(1) queries for both media and tags.
## Changes Made
### 1. Batch Media Loading (`starpunk/media.py`)
Added `get_media_for_notes()` function:
```python
def get_media_for_notes(note_ids: List[int]) -> Dict[int, List[Dict]]:
"""
Batch load media for multiple notes in single query
Per v1.5.0 Phase 3: Fixes N+1 query pattern in feed generation.
Loads media and variants for all notes in 2 queries instead of O(n).
"""
```
**Implementation details**:
- Query 1: Loads all media for all notes using `WHERE note_id IN (...)`
- Query 2: Loads all variants for all media using `WHERE media_id IN (...)`
- Groups results by `note_id` for efficient lookup
- Returns dict mapping `note_id -> List[media_dict]`
- Maintains exact same format as `get_note_media()` for compatibility
**Lines**: 728-852 in `starpunk/media.py`
### 2. Batch Tag Loading (`starpunk/tags.py`)
Added `get_tags_for_notes()` function:
```python
def get_tags_for_notes(note_ids: list[int]) -> dict[int, list[dict]]:
"""
Batch load tags for multiple notes in single query
Per v1.5.0 Phase 3: Fixes N+1 query pattern in feed generation.
Loads tags for all notes in 1 query instead of O(n).
"""
```
**Implementation details**:
- Single query loads all tags for all notes using `WHERE note_id IN (...)`
- Preserves alphabetical ordering: `ORDER BY LOWER(tags.display_name) ASC`
- Groups results by `note_id`
- Returns dict mapping `note_id -> List[tag_dict]`
- Maintains exact same format as `get_note_tags()` for compatibility
**Lines**: 146-197 in `starpunk/tags.py`
### 3. Feed Generation Update (`starpunk/routes/public.py`)
Updated `_get_cached_notes()` to use batch loading:
**Before** (N+1 pattern):
```python
for note in notes:
media = get_note_media(note.id) # 1 query per note
tags = get_note_tags(note.id) # 1 query per note
```
**After** (batch loading):
```python
note_ids = [note.id for note in notes]
media_by_note = get_media_for_notes(note_ids) # 1 query total
tags_by_note = get_tags_for_notes(note_ids) # 1 query total
for note in notes:
media = media_by_note.get(note.id, [])
tags = tags_by_note.get(note.id, [])
```
**Lines**: 38-86 in `starpunk/routes/public.py`
### 4. Comprehensive Tests (`tests/test_batch_loading.py`)
Created new test file with 13 tests:
**TestBatchMediaLoading** (6 tests):
- `test_batch_load_media_empty_list` - Empty input handling
- `test_batch_load_media_no_media` - Notes without media
- `test_batch_load_media_with_media` - Basic media loading
- `test_batch_load_media_with_variants` - Variant inclusion
- `test_batch_load_media_multiple_per_note` - Multiple media per note
- `test_batch_load_media_mixed_notes` - Mix of notes with/without media
**TestBatchTagLoading** (4 tests):
- `test_batch_load_tags_empty_list` - Empty input handling
- `test_batch_load_tags_no_tags` - Notes without tags
- `test_batch_load_tags_with_tags` - Basic tag loading
- `test_batch_load_tags_mixed_notes` - Mix of notes with/without tags
- `test_batch_load_tags_ordering` - Alphabetical ordering preserved
**TestBatchLoadingIntegration** (2 tests):
- `test_feed_generation_uses_batch_loading` - End-to-end feed test
- `test_batch_loading_performance_comparison` - Verify batch completeness
All tests passed: 13/13
## Performance Analysis
### Query Count Reduction
For a feed with N notes:
**Before (N+1 pattern)**:
- 1 query to fetch notes
- N queries to fetch media (one per note)
- N queries to fetch tags (one per note)
- **Total: 1 + 2N queries**
**After (batch loading)**:
- 1 query to fetch notes
- 1 query to fetch all media for all notes
- 1 query to fetch all tags for all notes
- **Total: 3 queries**
**Example** (50 notes in feed):
- Before: 1 + 2(50) = **101 queries**
- After: **3 queries**
- **Improvement: 97% reduction in queries**
### SQL Query Patterns
**Media batch query**:
```sql
SELECT nm.note_id, m.id, m.filename, ...
FROM note_media nm
JOIN media m ON nm.media_id = m.id
WHERE nm.note_id IN (?, ?, ?, ...)
ORDER BY nm.note_id, nm.display_order
```
**Tags batch query**:
```sql
SELECT note_tags.note_id, tags.name, tags.display_name
FROM tags
JOIN note_tags ON tags.id = note_tags.tag_id
WHERE note_tags.note_id IN (?, ?, ?, ...)
ORDER BY note_tags.note_id, LOWER(tags.display_name) ASC
```
## Compatibility
### API Behavior
- No changes to external API endpoints
- Feed output format identical (RSS, Atom, JSON Feed)
- Existing tests all pass unchanged (920 tests)
### Data Format
Batch loading functions return exact same structure as single-note functions:
```python
# get_note_media(note_id) returns:
[
{
'id': 1,
'filename': 'test.jpg',
'variants': {...},
...
}
]
# get_media_for_notes([note_id]) returns:
{
note_id: [
{
'id': 1,
'filename': 'test.jpg',
'variants': {...},
...
}
]
}
```
## Edge Cases Handled
1. **Empty note list**: Returns empty dict `{}`
2. **Notes without media/tags**: Returns empty list `[]` for those notes
3. **Mixed notes**: Some with media/tags, some without
4. **Multiple media per note**: Display order preserved
5. **Tag ordering**: Case-insensitive alphabetical order maintained
6. **Variants**: Backwards compatible (pre-v1.4.0 media has no variants)
## Testing Results
### Test Suite
- **New tests**: 13 tests in `tests/test_batch_loading.py`
- **Full test suite**: 920 tests passed
- **Execution time**: 360.79s (6 minutes)
- **Warnings**: 1 warning (existing DecompressionBombWarning, not related to changes)
### Test Coverage
All batch loading scenarios tested:
- Empty inputs
- Notes without associations
- Notes with associations
- Mixed scenarios
- Variant handling
- Ordering preservation
- Integration with feed generation
## Documentation
### Code Comments
- Added docstrings to both batch functions explaining purpose
- Referenced v1.5.0 Phase 3 in comments
- Included usage examples in docstrings
### Implementation Notes
- Used f-strings for IN clause placeholders (safe with parameterized queries)
- Grouped results using dict comprehensions for efficiency
- Maintained consistent error handling with existing functions
- No external dependencies added
## Issues Encountered
None. Implementation proceeded smoothly:
- Batch functions matched existing patterns in codebase
- SQL queries worked correctly on first attempt
- All tests passed without modifications
- No regression in existing functionality
## Acceptance Criteria
Per v1.5.0 Phase 3 requirements:
- [x] Feed generation uses batch queries
- [x] Query count reduced from O(n) to O(1) for media/tags
- [x] No change to API behavior
- [x] Performance improvement verified in tests
- [x] Other N+1 locations documented in BACKLOG.md (not part of this phase)
## Files Modified
1. `/home/phil/Projects/starpunk/starpunk/media.py` - Added `get_media_for_notes()`
2. `/home/phil/Projects/starpunk/starpunk/tags.py` - Added `get_tags_for_notes()`
3. `/home/phil/Projects/starpunk/starpunk/routes/public.py` - Updated `_get_cached_notes()`
4. `/home/phil/Projects/starpunk/tests/test_batch_loading.py` - New test file (13 tests)
## Commit
```
commit b689e02
perf(feed): Batch load media and tags to fix N+1 query
Per v1.5.0 Phase 3: Fix N+1 query pattern in feed generation.
Implementation:
- Add get_media_for_notes() to starpunk/media.py for batch media loading
- Add get_tags_for_notes() to starpunk/tags.py for batch tag loading
- Update _get_cached_notes() in starpunk/routes/public.py to use batch loading
- Add comprehensive tests in tests/test_batch_loading.py
Performance improvement:
- Before: O(n) queries (1 query per note for media + 1 query per note for tags)
- After: O(1) queries (2 queries total: 1 for all media, 1 for all tags)
- Maintains same API behavior and output format
All tests passing: 920 passed in 360.79s
```
## Recommendations for Architect
Phase 3 is complete and ready for review. The implementation:
1. **Achieves the goal**: Feed generation now uses batch queries
2. **Maintains compatibility**: No API changes, all existing tests pass
3. **Follows patterns**: Consistent with existing codebase style
4. **Well-tested**: Comprehensive test coverage for all scenarios
5. **Performant**: 97% reduction in queries for typical feed (50 notes)
### Deferred N+1 Patterns
Per the requirements, other N+1 patterns were NOT addressed in this phase:
- Homepage (`/`) - Still uses `get_note_media()` and `get_note_tags()` per-note
- Note permalink (`/note/<slug>`) - Single note, N+1 not applicable
- Tag archive (`/tag/<tag>`) - Still uses `get_note_media()` per-note
- Admin interfaces - Not in scope for this phase
These are documented in BACKLOG.md for future consideration. The batch loading functions created in this phase can be reused for those locations if/when they are addressed.
## Next Steps
1. Architect reviews Phase 3 implementation
2. If approved, ready to proceed to Phase 4: Atomic Variant Generation
3. If changes requested, developer will address feedback
## Status
**COMPLETE** - Awaiting architect review before proceeding to Phase 4.
---
Developer: Claude (StarPunk Developer Agent)
Date: 2025-12-17
Branch: feature/v1.5.0-media
Commit: b689e02

5
starpunk/__init__.py
View File

@@ -128,9 +128,12 @@ def create_app(config=None):
configure_logging(app)
# Clean up old debug files (v1.5.0 Phase 2)
from starpunk.media import cleanup_old_debug_files
from starpunk.media import cleanup_old_debug_files, cleanup_orphaned_temp_files
cleanup_old_debug_files(app)
# Clean up orphaned temp files (v1.5.0 Phase 4)
cleanup_orphaned_temp_files(app)
# Initialize database schema
from starpunk.database import init_db, init_pool

367
starpunk/media.py
View File

@@ -21,6 +21,7 @@ from pathlib import Path
from datetime import datetime, timedelta
import uuid
import io
import shutil
from typing import Optional, List, Dict, Tuple
from flask import current_app
@@ -316,7 +317,8 @@ def generate_variant(
variant_type: str,
base_path: Path,
base_filename: str,
file_ext: str
file_ext: str,
relative_path: str = None
) -> Dict:
"""
Generate a single image variant
@@ -327,6 +329,7 @@ def generate_variant(
base_path: Directory to save to
base_filename: Base filename (UUID without extension)
file_ext: File extension (e.g., '.jpg')
relative_path: Relative path for metadata (if None, calculated from base_path)
Returns:
Dict with variant metadata (path, width, height, size_bytes)
@@ -359,19 +362,42 @@ def generate_variant(
# Save with appropriate quality
save_kwargs = {'optimize': True}
if work_img.format in ['JPEG', 'JPG', None]:
# Determine format - prefer image's actual format over extension
# This handles cases like HEIC -> JPEG conversion where extension doesn't match format
if work_img.format and work_img.format in ['JPEG', 'PNG', 'GIF', 'WEBP']:
save_format = work_img.format
if save_format in ['JPEG', 'JPG']:
save_kwargs['quality'] = 85
else:
# Fallback to extension-based detection
if file_ext.lower() in ['.jpg', '.jpeg', '.heic']:
save_format = 'JPEG'
save_kwargs['quality'] = 85
elif file_ext.lower() == '.png':
save_format = 'PNG'
elif file_ext.lower() == '.gif':
save_format = 'GIF'
elif file_ext.lower() == '.webp':
save_format = 'WEBP'
save_kwargs['quality'] = 85
else:
save_format = 'JPEG' # Default fallback
save_kwargs['quality'] = 85
# Determine format from extension
save_format = 'JPEG' if file_ext.lower() in ['.jpg', '.jpeg'] else file_ext[1:].upper()
work_img.save(variant_path, format=save_format, **save_kwargs)
# Use provided relative path or calculate it
if relative_path is None:
relative_path = str(variant_path.relative_to(base_path.parent.parent)) # Relative to media root
return {
'variant_type': variant_type,
'path': str(variant_path.relative_to(base_path.parent.parent)), # Relative to media root
'path': relative_path,
'width': work_img.width,
'height': work_img.height,
'size_bytes': variant_path.stat().st_size
'size_bytes': variant_path.stat().st_size,
'temp_file': variant_path # Include temp file path for atomic operation
}
@@ -383,32 +409,53 @@ def generate_all_variants(
media_id: int,
year: str,
month: str,
optimized_bytes: bytes
) -> List[Dict]:
optimized_bytes: bytes,
db = None
) -> Tuple[List[Dict], List[Tuple[Path, Path]]]:
"""
Generate all variants for an image and store in database
Generate all variants for an image and prepare database records
Per v1.5.0 Phase 4: Atomic variant generation
- Generate variants to temp directory first
- Return database insert data and file move operations
- Caller handles transaction commit and file moves
- This ensures true atomicity
Args:
img: Source PIL Image (the optimized original)
base_path: Directory containing the original
base_path: Directory containing the original (final destination)
base_filename: Base filename (UUID without extension)
file_ext: File extension
media_id: ID of parent media record
year: Year string (e.g., '2025') for path calculation
month: Month string (e.g., '01') for path calculation
optimized_bytes: Bytes of optimized original (avoids re-reading file)
db: Database connection (optional, for transaction control)
Returns:
List of variant metadata dicts
Tuple of (variant_metadata_list, file_moves_list)
- variant_metadata_list: List of dicts ready for database insert
- file_moves_list: List of (src_path, dst_path) tuples for file moves
"""
from starpunk.database import get_db
variants = []
if db is None:
db = get_db(current_app)
created_files = [] # Track files for cleanup on failure
variants = []
file_moves = []
# Create temp directory for atomic operation
media_dir = Path(current_app.config.get('DATA_PATH', 'data')) / 'media'
temp_dir = media_dir / '.tmp'
temp_dir.mkdir(parents=True, exist_ok=True)
# Create unique temp subdirectory for this operation
temp_subdir = temp_dir / f"{base_filename}_{uuid.uuid4().hex[:8]}"
temp_subdir.mkdir(parents=True, exist_ok=True)
try:
# Generate each variant type
# Step 1: Generate all variants to temp directory
for variant_type in ['thumb', 'small', 'medium', 'large']:
# Skip if image is smaller than target
spec = VARIANT_SPECS[variant_type]
@@ -417,45 +464,59 @@ def generate_all_variants(
if img.width < target_width and variant_type != 'thumb':
continue # Skip variants larger than original
variant = generate_variant(img, variant_type, base_path, base_filename, file_ext)
variants.append(variant)
created_files.append(base_path / f"{base_filename}_{variant_type}{file_ext}")
# Calculate final relative path (for database)
final_relative_path = f"{year}/{month}/{base_filename}_{variant_type}{file_ext}"
# Insert into database
db.execute(
"""
INSERT INTO media_variants
(media_id, variant_type, path, width, height, size_bytes)
VALUES (?, ?, ?, ?, ?, ?)
""",
(media_id, variant['variant_type'], variant['path'],
variant['width'], variant['height'], variant['size_bytes'])
# Generate variant to temp directory
variant = generate_variant(
img,
variant_type,
temp_subdir, # Write to temp
base_filename,
file_ext,
final_relative_path # Store final path in metadata
)
# Also record the original as 'original' variant
# Use explicit year/month for path calculation (avoids fragile parent traversal)
# Prepare database metadata (without temp_file key)
variant_metadata = {
'variant_type': variant['variant_type'],
'path': variant['path'],
'width': variant['width'],
'height': variant['height'],
'size_bytes': variant['size_bytes']
}
variants.append(variant_metadata)
# Track file move operation
temp_file = variant['temp_file']
final_path = base_path / temp_file.name
file_moves.append((temp_file, final_path, temp_subdir))
# Also prepare original variant metadata
original_path = f"{year}/{month}/{base_filename}{file_ext}"
db.execute(
"""
INSERT INTO media_variants
(media_id, variant_type, path, width, height, size_bytes)
VALUES (?, ?, ?, ?, ?, ?)
""",
(media_id, 'original', original_path, img.width, img.height,
len(optimized_bytes)) # Use passed bytes instead of file I/O
)
variants.append({
'variant_type': 'original',
'path': original_path,
'width': img.width,
'height': img.height,
'size_bytes': len(optimized_bytes)
})
db.commit()
return variants
return variants, file_moves
except Exception as e:
# Clean up any created variant files on failure
for file_path in created_files:
# Clean up temp files on failure
try:
if file_path.exists():
file_path.unlink()
if temp_subdir.exists():
for file in temp_subdir.glob('*'):
try:
file.unlink()
except OSError:
pass # Best effort cleanup
pass
temp_subdir.rmdir()
except OSError:
pass # Best effort
raise # Re-raise the original exception
@@ -526,15 +587,47 @@ def save_media(file_data: bytes, filename: str) -> Dict:
full_dir = media_dir / year / month
full_dir.mkdir(parents=True, exist_ok=True)
# Save optimized image (using bytes from optimize_image to avoid re-encoding)
full_path = full_dir / stored_filename
full_path.write_bytes(optimized_bytes)
# Get actual file size (from optimized bytes)
actual_size = len(optimized_bytes)
# Insert into database
# Per v1.5.0 Phase 4: Atomic operation for all file saves and database inserts
# Generate variants first (to temp directory)
base_filename = stored_filename.rsplit('.', 1)[0]
db = get_db(current_app)
variant_metadata = []
file_moves = []
temp_original_path = None
temp_subdir = None
try:
# Step 1: Save original to temp directory
media_dir = Path(current_app.config.get('DATA_PATH', 'data')) / 'media'
temp_dir = media_dir / '.tmp'
temp_dir.mkdir(parents=True, exist_ok=True)
temp_subdir = temp_dir / f"{base_filename}_{uuid.uuid4().hex[:8]}"
temp_subdir.mkdir(parents=True, exist_ok=True)
temp_original_path = temp_subdir / stored_filename
temp_original_path.write_bytes(optimized_bytes)
# Step 2: Generate variants to temp directory
variant_metadata, file_moves = generate_all_variants(
optimized_img,
full_dir,
base_filename,
file_ext,
0, # media_id not yet known
year,
month,
optimized_bytes,
db
)
# Step 3: Begin transaction
db.execute("BEGIN TRANSACTION")
# Step 4: Insert media record
cursor = db.execute(
"""
INSERT INTO media (filename, stored_filename, path, mime_type, size, width, height)
@@ -542,30 +635,99 @@ def save_media(file_data: bytes, filename: str) -> Dict:
""",
(filename, stored_filename, relative_path, mime_type, actual_size, width, height)
)
db.commit()
media_id = cursor.lastrowid
# Generate variants (synchronous) - v1.4.0 Phase 2
# Pass year, month, and optimized_bytes to avoid fragile path traversal and file I/O
base_filename = stored_filename.rsplit('.', 1)[0]
variants = []
# Step 5: Insert variant records
for variant in variant_metadata:
db.execute(
"""
INSERT INTO media_variants
(media_id, variant_type, path, width, height, size_bytes)
VALUES (?, ?, ?, ?, ?, ?)
""",
(media_id, variant['variant_type'], variant['path'],
variant['width'], variant['height'], variant['size_bytes'])
)
# Step 6: Move files to final location (before commit for true atomicity)
# If file moves fail, we can rollback the transaction
try:
variants = generate_all_variants(
optimized_img,
full_dir,
base_filename,
file_ext,
media_id,
year,
month,
optimized_bytes
)
# Move original file
full_path = full_dir / stored_filename
shutil.move(str(temp_original_path), str(full_path))
# Move variant files
for temp_file, final_path, _ in file_moves:
shutil.move(str(temp_file), str(final_path))
except Exception as e:
# Rollback database if file move fails
db.rollback()
raise
# Step 7: Commit transaction (after files are moved successfully)
db.commit()
# Step 8: Clean up temp directory
try:
if temp_subdir and temp_subdir.exists():
temp_subdir.rmdir()
except OSError:
pass # Best effort
# Format variants for return value (same format as before)
variants = [v for v in variant_metadata if v['variant_type'] != 'original']
except Exception as e:
# Rollback database on any failure (best effort)
try:
db.rollback()
except Exception:
pass # May already be rolled back
# Clean up moved files if commit failed
# (This handles the case where files were moved but commit failed)
full_path = full_dir / stored_filename
if full_path.exists():
try:
full_path.unlink()
except OSError:
pass
for _, final_path, _ in file_moves:
try:
if final_path.exists():
final_path.unlink()
except OSError:
pass
# Clean up temp files on any failure
if temp_original_path and temp_original_path.exists():
try:
temp_original_path.unlink()
except OSError:
pass
for temp_file, _, _ in file_moves:
try:
if temp_file.exists():
temp_file.unlink()
except OSError:
pass
# Clean up temp subdirectory
if temp_subdir and temp_subdir.exists():
try:
temp_subdir.rmdir()
except OSError:
pass
# Log and re-raise
current_app.logger.warning(
f'Media upload variant generation failed: filename="{filename}", '
f'media_id={media_id}, error="{e}"'
f'Media upload atomic operation failed: filename="{filename}", '
f'error="{e}"'
)
# Continue - original image is still usable
raise
# Log success
was_optimized = len(optimized_bytes) < file_size
@@ -981,3 +1143,74 @@ def cleanup_old_debug_files(app) -> None:
f"Debug file cleanup: deleted {deleted_count} file(s), "
f"freed {deleted_size / 1024 / 1024:.2f} MB"
)
def cleanup_orphaned_temp_files(app) -> None:
"""
Clean up orphaned temporary variant files on startup
Per v1.5.0 Phase 4:
- Detect temp files left from failed operations
- Log warnings for orphaned files
- Clean up temp directory
- Called on application startup
Args:
app: Flask application instance (for config and logger)
"""
media_dir = Path(app.config.get('DATA_PATH', 'data')) / 'media'
temp_dir = media_dir / '.tmp'
# Check if temp directory exists
if not temp_dir.exists():
return
# Find all subdirectories and files in temp directory
orphaned_count = 0
cleaned_size = 0
# Iterate through temp subdirectories
for temp_subdir in temp_dir.iterdir():
if not temp_subdir.is_dir():
# Clean up any loose files (shouldn't normally exist)
try:
size = temp_subdir.stat().st_size
temp_subdir.unlink()
orphaned_count += 1
cleaned_size += size
app.logger.warning(f"Cleaned up orphaned temp file: {temp_subdir.name}")
except OSError as e:
app.logger.warning(f"Failed to delete orphaned temp file {temp_subdir.name}: {e}")
continue
# Process subdirectory
files_in_subdir = list(temp_subdir.glob('*'))
if files_in_subdir:
# Log orphaned operation
app.logger.warning(
f"Found orphaned temp directory from failed operation: {temp_subdir.name} "
f"({len(files_in_subdir)} file(s))"
)
# Clean up files
for file_path in files_in_subdir:
try:
if file_path.is_file():
size = file_path.stat().st_size
file_path.unlink()
orphaned_count += 1
cleaned_size += size
except OSError as e:
app.logger.warning(f"Failed to delete orphaned temp file {file_path}: {e}")
# Remove empty subdirectory
try:
temp_subdir.rmdir()
except OSError as e:
app.logger.warning(f"Failed to remove temp directory {temp_subdir.name}: {e}")
if orphaned_count > 0:
app.logger.info(
f"Temp file cleanup: removed {orphaned_count} orphaned file(s), "
f"freed {cleaned_size / 1024 / 1024:.2f} MB"
)

170
tests/test_media_upload.py
View File

@@ -9,6 +9,7 @@ import pytest
from PIL import Image
import io
from pathlib import Path
from datetime import datetime
from starpunk.media import (
validate_image,
@@ -618,7 +619,7 @@ class TestMediaLogging:
assert 'error=' in caplog.text
def test_save_media_logs_variant_failure(self, app, caplog, monkeypatch):
"""Test variant generation failure logs at WARNING level but continues"""
"""Test variant generation failure causes atomic rollback (v1.5.0 Phase 4)"""
import logging
from starpunk import media
@@ -631,20 +632,15 @@ class TestMediaLogging:
image_data = create_test_image(800, 600, 'PNG')
with app.app_context():
with caplog.at_level(logging.INFO): # Need INFO level to capture success log
# Should succeed despite variant failure
media_info = save_media(image_data, 'test.png')
with caplog.at_level(logging.WARNING):
# Should fail due to atomic operation (v1.5.0 Phase 4)
with pytest.raises(RuntimeError, match="Variant generation failed"):
save_media(image_data, 'test.png')
# Check variant failure log
assert "Media upload variant generation failed" in caplog.text
# Check atomic operation failure log
assert "Media upload atomic operation failed" in caplog.text
assert 'filename="test.png"' in caplog.text
assert f'media_id={media_info["id"]}' in caplog.text
assert 'error=' in caplog.text
# But success log should also be present
assert "Media upload successful" in caplog.text
# And variants should be 0
assert 'variants=0' in caplog.text
assert 'error="Variant generation failed"' in caplog.text
def test_save_media_logs_unexpected_error(self, app, caplog, monkeypatch):
"""Test unexpected error logs at ERROR level"""
@@ -680,3 +676,151 @@ def sample_note(app):
with app.app_context():
note = create_note("Test note content", published=True)
yield note
class TestAtomicVariantGeneration:
"""
Test atomic variant generation (v1.5.0 Phase 4)
Tests that variant generation is atomic with database commits,
preventing orphaned files or database records.
"""
def test_atomic_media_save_success(self, app):
"""Test that media save operation is fully atomic on success"""
from pathlib import Path
from starpunk.media import save_media
# Create test image
img_data = create_test_image(1600, 1200, 'JPEG')
with app.app_context():
# Save media
result = save_media(img_data, 'test_atomic.jpg')
# Verify media record was created
assert result['id'] > 0
assert result['filename'] == 'test_atomic.jpg'
# Verify original file exists in final location
media_dir = Path(app.config['DATA_PATH']) / 'media'
original_path = media_dir / result['path']
assert original_path.exists(), "Original file should exist in final location"
# Verify variant files exist in final location
for variant in result['variants']:
variant_path = media_dir / variant['path']
assert variant_path.exists(), f"Variant {variant['variant_type']} should exist"
# Verify no temp files left behind
temp_dir = media_dir / '.tmp'
if temp_dir.exists():
temp_files = list(temp_dir.glob('**/*'))
temp_files = [f for f in temp_files if f.is_file()]
assert len(temp_files) == 0, "No temp files should remain after successful save"
def test_file_move_failure_rolls_back_database(self, app, monkeypatch):
"""Test that file move failure rolls back database transaction"""
from pathlib import Path
from starpunk.media import save_media
import shutil
# Create test image
img_data = create_test_image(1600, 1200, 'JPEG')
with app.app_context():
from starpunk.database import get_db
# Mock shutil.move to fail
original_move = shutil.move
call_count = [0]
def mock_move(src, dst):
call_count[0] += 1
# Fail on first move (original file)
if call_count[0] == 1:
raise OSError("File move failed")
return original_move(src, dst)
monkeypatch.setattr(shutil, 'move', mock_move)
# Count media records before operation
db = get_db(app)
media_count_before = db.execute("SELECT COUNT(*) FROM media").fetchone()[0]
# Try to save media - should fail
with pytest.raises(OSError, match="File move failed"):
save_media(img_data, 'test_rollback.jpg')
# Verify no new media records were added (transaction rolled back)
media_count_after = db.execute("SELECT COUNT(*) FROM media").fetchone()[0]
assert media_count_after == media_count_before, "No media records should be added on failure"
# Verify temp files were cleaned up
media_dir = Path(app.config['DATA_PATH']) / 'media'
temp_dir = media_dir / '.tmp'
if temp_dir.exists():
temp_files = list(temp_dir.glob('**/*'))
temp_files = [f for f in temp_files if f.is_file()]
assert len(temp_files) == 0, "Temp files should be cleaned up after file move failure"
# Restore original move
monkeypatch.setattr(shutil, 'move', original_move)
def test_startup_recovery_cleans_orphaned_temp_files(self, app):
"""Test that startup recovery detects and cleans orphaned temp files"""
from pathlib import Path
from starpunk.media import cleanup_orphaned_temp_files
import logging
with app.app_context():
media_dir = Path(app.config['DATA_PATH']) / 'media'
temp_dir = media_dir / '.tmp'
temp_dir.mkdir(parents=True, exist_ok=True)
# Create orphaned temp subdirectory with files
orphan_dir = temp_dir / 'orphaned_test_12345678'
orphan_dir.mkdir(parents=True, exist_ok=True)
# Create some fake orphaned files
orphan_file1 = orphan_dir / 'test_thumb.jpg'
orphan_file2 = orphan_dir / 'test_small.jpg'
orphan_file1.write_bytes(b'fake image data')
orphan_file2.write_bytes(b'fake image data')
# Run cleanup
with app.test_request_context():
cleanup_orphaned_temp_files(app)
# Verify files were cleaned up
assert not orphan_file1.exists(), "Orphaned file 1 should be deleted"
assert not orphan_file2.exists(), "Orphaned file 2 should be deleted"
assert not orphan_dir.exists(), "Orphaned directory should be deleted"
def test_startup_recovery_logs_orphaned_files(self, app, caplog):
"""Test that startup recovery logs warnings for orphaned files"""
from pathlib import Path
from starpunk.media import cleanup_orphaned_temp_files
import logging
with app.app_context():
media_dir = Path(app.config['DATA_PATH']) / 'media'
temp_dir = media_dir / '.tmp'
temp_dir.mkdir(parents=True, exist_ok=True)
# Create orphaned temp subdirectory with files
orphan_dir = temp_dir / 'orphaned_test_99999999'
orphan_dir.mkdir(parents=True, exist_ok=True)
# Create some fake orphaned files
orphan_file = orphan_dir / 'test_medium.jpg'
orphan_file.write_bytes(b'fake image data')
# Run cleanup with logging
with caplog.at_level(logging.WARNING):
cleanup_orphaned_temp_files(app)
# Verify warning was logged
assert "Found orphaned temp directory from failed operation" in caplog.text
assert "orphaned_test_99999999" in caplog.text