phil/StarPunk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Implement Phase 3 Feed Caching (Partial)

Browse Source 
Implements feed caching layer with LRU eviction, TTL expiration, and ETag support.

Phase 3.1: Feed Caching (Complete)
- LRU cache with configurable max_size (default: 50 feeds)
- TTL-based expiration (default: 300 seconds = 5 minutes)
- SHA-256 checksums for cache keys and ETags
- Weak ETag generation (W/"checksum")
- If-None-Match header support for 304 Not Modified responses
- Cache invalidation (全体 or per-format)
- Hit/miss/eviction statistics tracking
- Content-based cache keys (changes when notes are modified)

Implementation:
- Created starpunk/feeds/cache.py with FeedCache class
- Integrated caching into feed routes (RSS, ATOM, JSON Feed)
- Added ETag headers to all feed responses
- 304 Not Modified responses for conditional requests
- Configuration: FEED_CACHE_ENABLED, FEED_CACHE_MAX_SIZE
- Global cache instance with singleton pattern

Architecture:
- Two-level caching:
  1. Note list cache (simple dict, existing)
  2. Feed content cache (LRU with TTL, new)
- Cache keys include format + notes checksum
- Checksums based on note IDs + updated timestamps
- Non-streaming generators used for cacheable content

Testing:
- 25 comprehensive cache tests (100% passing)
- Tests for LRU eviction, TTL expiration, statistics
- Tests for checksum generation and consistency
- Tests for ETag generation and uniqueness
- All 114 feed tests passing (no regressions)

Quality Metrics:
- 114/114 tests passing (100%)
- Zero breaking changes
- Full backward compatibility
- Cache disabled mode supported (FEED_CACHE_ENABLED=false)

Performance Benefits:
- Database queries reduced (note list cached)
- Feed generation reduced (content cached)
- Bandwidth saved (304 responses)
- Memory efficient (LRU eviction)

Note: Phase 3 is partially complete. Still pending:
- Feed statistics dashboard
- OPML 2.0 export endpoint

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Phil Skentelbery
2025-11-27 21:14:03 -07:00
parent f59cbb30a5
commit c1dd706b8f
6 changed files with 827 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
starpunk/routes/public.py
View File

@@ -13,11 +13,15 @@ from flask import Blueprint, abort, render_template, Response, current_app, requ
from starpunk.notes import list_notes, get_note
from starpunk.feed import generate_feed_streaming # Legacy RSS
from starpunk.feeds import (
generate_rss,
generate_rss_streaming,
generate_atom,
generate_atom_streaming,
generate_json_feed,
generate_json_feed_streaming,
negotiate_feed_format,
get_mime_type,
get_cache,
)
# Create blueprint
@@ -25,7 +29,7 @@ bp = Blueprint("public", __name__)
# Simple in-memory cache for feed note list
# Caches the database query results to avoid repeated DB hits
# Feed content (XML/JSON) is streamed, not cached (memory optimization)
# Feed content is now cached via FeedCache (Phase 3)
# Structure: {'notes': list[Note], 'timestamp': datetime}
_feed_cache = {"notes": None, "timestamp": None}
@@ -61,6 +65,98 @@ def _get_cached_notes():
return notes
def _generate_feed_with_cache(format_name: str, non_streaming_generator):
"""
Generate feed with caching and ETag support.
Implements Phase 3 feed caching:
- Checks If-None-Match header for conditional requests
- Uses FeedCache for content caching
- Returns 304 Not Modified when appropriate
- Adds ETag header to all responses
Args:
format_name: Feed format (rss, atom, json)
non_streaming_generator: Function that returns full feed content (not streaming)
Returns:
Flask Response with appropriate headers and status
"""
# Get cached notes
notes = _get_cached_notes()
# Check if caching is enabled
cache_enabled = current_app.config.get("FEED_CACHE_ENABLED", True)
if not cache_enabled:
# Caching disabled, generate fresh feed
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
# Generate feed content (non-streaming)
content = non_streaming_generator(
site_url=current_app.config["SITE_URL"],
site_name=current_app.config["SITE_NAME"],
site_description=current_app.config.get("SITE_DESCRIPTION", ""),
notes=notes,
limit=max_items,
)
response = Response(content, mimetype=get_mime_type(format_name))
response.headers["Cache-Control"] = f"public, max-age={cache_seconds}"
return response
# Caching enabled - use FeedCache
feed_cache = get_cache()
notes_checksum = feed_cache.generate_notes_checksum(notes)
# Check If-None-Match header for conditional requests
if_none_match = request.headers.get('If-None-Match')
# Try to get cached feed
cached_result = feed_cache.get(format_name, notes_checksum)
if cached_result:
content, etag = cached_result
# Check if client has current version
if if_none_match and if_none_match == etag:
# Client has current version, return 304 Not Modified
response = Response(status=304)
response.headers["ETag"] = etag
return response
# Return cached content with ETag
response = Response(content, mimetype=get_mime_type(format_name))
response.headers["ETag"] = etag
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
response.headers["Cache-Control"] = f"public, max-age={cache_seconds}"
return response
# Cache miss - generate fresh feed
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
# Generate feed content (non-streaming)
content = non_streaming_generator(
site_url=current_app.config["SITE_URL"],
site_name=current_app.config["SITE_NAME"],
site_description=current_app.config.get("SITE_DESCRIPTION", ""),
notes=notes,
limit=max_items,
)
# Store in cache and get ETag
etag = feed_cache.set(format_name, content, notes_checksum)
# Return fresh content with ETag
response = Response(content, mimetype=get_mime_type(format_name))
response.headers["ETag"] = etag
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
response.headers["Cache-Control"] = f"public, max-age={cache_seconds}"
return response
@bp.route("/")
def index():
"""
@@ -171,32 +267,27 @@ def feed():
@bp.route("/feed.rss")
def feed_rss():
"""
Explicit RSS 2.0 feed endpoint
Explicit RSS 2.0 feed endpoint (with caching)
Generates standards-compliant RSS 2.0 feed using memory-efficient streaming.
Instead of building the entire feed in memory, yields XML chunks directly
to the client for optimal memory usage with large feeds.
Cache duration is configurable via FEED_CACHE_SECONDS (default: 300 seconds
= 5 minutes). Cache stores note list to avoid repeated database queries,
but streaming prevents holding full XML in memory.
Generates standards-compliant RSS 2.0 feed with Phase 3 caching:
- LRU cache with TTL (default 5 minutes)
- ETag support for conditional requests
- 304 Not Modified responses
- SHA-256 checksums
Returns:
Streaming RSS 2.0 feed response
Cached or fresh RSS 2.0 feed response
Headers:
Content-Type: application/rss+xml; charset=utf-8
Cache-Control: public, max-age={FEED_CACHE_SECONDS}
ETag: W/"sha256_hash"
Streaming Strategy:
- Database query cached (avoid repeated DB hits)
- XML generation streamed (avoid full XML in memory)
- Client-side: Cache-Control header with max-age
Performance:
- Memory usage: O(1) instead of O(n) for feed size
- Latency: Lower time-to-first-byte (TTFB)
- Recommended for feeds with 100+ items
Caching Strategy:
- Database query cached (note list)
- Feed content cached (full XML)
- Conditional requests (If-None-Match)
- Cache invalidation on content changes
Examples:
>>> response = client.get('/feed.rss')
@@ -204,44 +295,32 @@ def feed_rss():
200
>>> response.headers['Content-Type']
'application/rss+xml; charset=utf-8'
>>> response.headers['ETag']
'W/"abc123..."'
>>> # Conditional request
>>> response = client.get('/feed.rss', headers={'If-None-Match': 'W/"abc123..."'})
>>> response.status_code
304
"""
# Get cached notes
notes = _get_cached_notes()
# Get cache duration for response header
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
# Generate streaming RSS feed
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
generator = generate_rss_streaming(
site_url=current_app.config["SITE_URL"],
site_name=current_app.config["SITE_NAME"],
site_description=current_app.config.get("SITE_DESCRIPTION", ""),
notes=notes,
limit=max_items,
)
# Return streaming response with appropriate headers
response = Response(generator, mimetype="application/rss+xml; charset=utf-8")
response.headers["Cache-Control"] = f"public, max-age={cache_seconds}"
return response
return _generate_feed_with_cache('rss', generate_rss)
@bp.route("/feed.atom")
def feed_atom():
"""
Explicit ATOM 1.0 feed endpoint
Explicit ATOM 1.0 feed endpoint (with caching)
Generates standards-compliant ATOM 1.0 feed using memory-efficient streaming.
Generates standards-compliant ATOM 1.0 feed with Phase 3 caching.
Follows RFC 4287 specification for ATOM syndication format.
Returns:
Streaming ATOM 1.0 feed response
Cached or fresh ATOM 1.0 feed response
Headers:
Content-Type: application/atom+xml; charset=utf-8
Cache-Control: public, max-age={FEED_CACHE_SECONDS}
ETag: W/"sha256_hash"
Examples:
>>> response = client.get('/feed.atom')
@@ -249,44 +328,27 @@ def feed_atom():
200
>>> response.headers['Content-Type']
'application/atom+xml; charset=utf-8'
>>> response.headers['ETag']
'W/"abc123..."'
"""
# Get cached notes
notes = _get_cached_notes()
# Get cache duration for response header
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
# Generate streaming ATOM feed
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
generator = generate_atom_streaming(
site_url=current_app.config["SITE_URL"],
site_name=current_app.config["SITE_NAME"],
site_description=current_app.config.get("SITE_DESCRIPTION", ""),
notes=notes,
limit=max_items,
)
# Return streaming response with appropriate headers
response = Response(generator, mimetype="application/atom+xml; charset=utf-8")
response.headers["Cache-Control"] = f"public, max-age={cache_seconds}"
return response
return _generate_feed_with_cache('atom', generate_atom)
@bp.route("/feed.json")
def feed_json():
"""
Explicit JSON Feed 1.1 endpoint
Explicit JSON Feed 1.1 endpoint (with caching)
Generates standards-compliant JSON Feed 1.1 feed using memory-efficient streaming.
Generates standards-compliant JSON Feed 1.1 feed with Phase 3 caching.
Follows JSON Feed specification (https://jsonfeed.org/version/1.1).
Returns:
Streaming JSON Feed 1.1 response
Cached or fresh JSON Feed 1.1 response
Headers:
Content-Type: application/feed+json; charset=utf-8
Cache-Control: public, max-age={FEED_CACHE_SECONDS}
ETag: W/"sha256_hash"
Examples:
>>> response = client.get('/feed.json')
@@ -294,28 +356,10 @@ def feed_json():
200
>>> response.headers['Content-Type']
'application/feed+json; charset=utf-8'
>>> response.headers['ETag']
'W/"abc123..."'
"""
# Get cached notes
notes = _get_cached_notes()
# Get cache duration for response header
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
# Generate streaming JSON Feed
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
generator = generate_json_feed_streaming(
site_url=current_app.config["SITE_URL"],
site_name=current_app.config["SITE_NAME"],
site_description=current_app.config.get("SITE_DESCRIPTION", ""),
notes=notes,
limit=max_items,
)
# Return streaming response with appropriate headers
response = Response(generator, mimetype="application/feed+json; charset=utf-8")
response.headers["Cache-Control"] = f"public, max-age={cache_seconds}"
return response
return _generate_feed_with_cache('json', generate_json_feed)
@bp.route("/feed.xml")