Compare commits

..
6 Commits
Author SHA1 Message Date
Phil 0cfc9965df Merge branch 'fix/v2.6.9-webmention-processing' for v2.6.9 release
Build Container / build (push) Successful in 12s
2026-11-03 07:53:19 -07:00
PhilandClaude Opus 5.5 106495d114 fix: Webmention processing works end to end (v2.6.9)
- Normalise mf2 author values: u-photo with alt text is a dict in mf2py and
  broke the database update for most real senders
- Pass str(response.url) to urljoin during endpoint discovery; the TypeError
  marked every target as having no endpoint. Retry on discovery errors
- Hold a cross-process flock so only one gunicorn process runs the worker
- Keep author URL/photo only if http(s) (XSS via javascript: URLs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:53:19 -07:00
Phil 3c1e109358 Merge branch 'fix/v2.6.8-webmention-worker' for v2.6.8 release
Build Container / build (push) Successful in 20s
2026-11-03 07:41:11 -07:00
PhilandClaude Opus 5.5 d43361d505 fix: Webmention worker runs inside the app context (v2.6.8)
The worker thread called the SSRF checks, which read current_app.config,
without an app context, so every incoming and outgoing webmention failed with
'Working outside of application context'. A teardown_appcontext hook also
stopped the worker whenever a request ended.

Pass the app to WebmentionWorker and run each poll in app.app_context(), like
SyndicationWorker, and drop the teardown hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:41:06 -07:00
Phil f924413934 Merge branch 'fix/v2.6.7-admin-layout' for v2.6.7 release
Build Container / build (push) Successful in 7s
2026-11-02 18:24:15 -07:00
PhilandClaude Opus 5.5 4b2ae4c7fc fix: Admin pages line up with the header (v2.6.7)
Every page sat in the 40rem reading column, so the admin wrapper's 60rem
never applied and the notes table, which can't shrink that far, ran past
the header's right edge. Admin pages now set a 60rem frame for header,
content and footer; post previews wrap, and tables scroll sideways on
narrow screens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:24:14 -07:00
13 changed files with 326 additions and 30 deletions

No files matched your search

+25
View File
@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.6.9] - 2026-11-03
### Fixed
- Received webmentions failed when the sender's profile photo had alt text (Bridgy, Mastodon and most IndieWeb sites do this). The photo is now stored as its URL
- Outgoing webmentions were never sent: endpoint discovery crashed on every target that had an endpoint, and the target was then marked as having none. Discovery now works, and network errors during discovery are retried instead of giving up
- Each server process ran its own webmention worker, so a webmention could be processed or sent more than once. Only one process handles the queues at a time now; if it stops, another takes over
### Security
- A webmention sender could set its author link to a `javascript:` URL, which was shown as a clickable link on the note page. Author links and photos are now only kept if they are http or https URLs
## [2.6.8] - 2026-11-03
### Fixed
- Webmentions were never processed. The background worker checked URLs against this site's settings without the app loaded, so every incoming webmention was marked "error" ("Working outside of application context") and every outgoing one kept failing and retrying. The worker now runs with the app loaded. A received webmention showed up in the admin list but never on the note
- The webmention worker also stopped as soon as any request finished, so after startup it often wasn't running at all. It now runs for the life of the server process
## [2.6.7] - 2026-11-02
### Fixed
- Admin pages were misaligned: the notes table ran past the right edge of the header and admin menu, because every page sat in the 40rem reading column and the table couldn't shrink to fit it. Admin pages now use a 60rem frame for the header, menu, content and footer alike, long post previews wrap, and on narrow screens tables scroll sideways instead of spilling out
## [2.6.6] - 2026-11-02
### Fixed
@@ -0,0 +1,37 @@
# v2.6.8: Webmention worker app context
## Problem
Production webmention #1 (Bridgy, reply from Mastodon to note 20261101162512)
was stored with `validation_status = 'error'`:
"Working outside of application context". Both outgoing queue items had the
same error and were on their last retry.
## Causes
1. `webmention_ssrf.safe_resolve()` reads `current_app.config['DEV_MODE']`.
`WebmentionWorker` is a plain thread with no Flask app context, so every
`validate_url_for_fetch()` call raised `RuntimeError`, for both
validation (incoming) and endpoint discovery and sending (outgoing).
2. `create_app()` registered a `teardown_appcontext` handler that called
`webmention_worker.stop()`. Teardown runs whenever any app context ends
(every request), so the worker stopped soon after startup.
## Fix
- `WebmentionWorker` takes the Flask app and runs each poll inside
`app.app_context()`, as `SyndicationWorker` already does.
- The teardown handler is removed. The worker is a daemon thread and ends with
the process.
## Tests
`tests/test_webmention_worker.py`: an outgoing send completes through the
worker thread, and the worker is still alive after an app context and a
request end. Both fail without the fix.
## Production data
After deploying, reset webmention 1 to `validation_status = 'pending'` and the
two queue items to `retry_count = 0, next_retry_at = NULL` so the worker
processes them again.
@@ -0,0 +1,36 @@
# v2.6.9: Webmention processing fixes
Found after v2.6.8 let the worker run in production, and by a review of the
implementation against the W3C Webmention Recommendation.
## Fixes
1. **Author photo with alt text** (`webmention_processor.extract_author`).
mf2py returns `u-photo` with an `alt` attribute as
`{'value': url, 'alt': ...}`; the dict was bound to SQLite and the update
failed. `_first_str()` reduces mf2 values to their `value`.
2. **Endpoint discovery** (`WebmentionWorker._discover_endpoint`).
`response.url` is an `httpx.URL`; `urljoin(str, URL)` raised `TypeError`,
which was caught and returned as "no endpoint", so every target was marked
`no_endpoint`. Pass `str(response.url)`. Fetch errors now propagate so the
caller schedules a retry, instead of being treated as "no endpoint".
3. **Duplicate processing.** gunicorn runs 4 processes, each with a worker,
and queue items were claimed by SELECT then UPDATE. The worker now takes a
non-blocking `flock` on `<data>/webmention-worker.lock` before processing;
others wait and retry each poll, so one takes over if the holder dies.
Orphan cleanup runs when the lock is acquired.
4. **XSS (spec 4.1).** `author_url` and `author_photo` are kept only if they
are absolute http(s) URLs; otherwise the source URL / no photo is used.
## Tests
`tests/test_webmention_worker.py`: the send test uses a real `httpx.Response`
(the old `MagicMock` with a string URL hid fix 2), plus tests for the lock,
an alt-text photo and a `javascript:` author URL. All fail on v2.6.8.
## Remaining spec gaps
Tracked separately: rel-token and document-order endpoint discovery, exact
target match during verification, Accept header, re-sending on delete, 410
for deleted notes, SSRF re-validation on redirects, and the others listed in
the compliance review.
+3 -8
View File
@@ -330,6 +330,7 @@ def create_app(config=None):
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
from starpunk.webmention_worker import WebmentionWorker
webmention_worker = WebmentionWorker(
app,
db_path=app.config['DATABASE_PATH'],
site_url=app.config['SITE_URL'],
poll_interval=app.config.get('WEBMENTION_POLL_INTERVAL', 10),
@@ -345,12 +346,6 @@ def create_app(config=None):
f"batch_size={webmention_worker.batch_size})"
)
# Register cleanup handler
@app.teardown_appcontext
def cleanup_webmention_worker(error=None):
if hasattr(app, 'webmention_worker') and app.webmention_worker.is_alive():
app.webmention_worker.stop()
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
@@ -464,5 +459,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.6.6"
__version_info__ = (2, 6, 4)
__version__ = "2.6.9"
__version_info__ = (2, 6, 9)
+32 -7
View File
@@ -182,6 +182,28 @@ def detect_mention_type(parsed_mf2: Dict, target_url: str) -> str:
return 'mention'
def _first_str(props: Dict, name: str) -> Optional[str]:
"""
First value of an mf2 property as a string
mf2py gives u-photo with alt text as {'value': url, 'alt': ...} and nested
microformats as dicts with a 'value'; reduce those to the plain value.
"""
values = props.get(name) or [None]
value = values[0]
if isinstance(value, dict):
value = value.get('value')
return value if isinstance(value, str) and value else None
def _is_http_url(value: Optional[str]) -> bool:
"""True if value is an absolute http or https URL"""
if not value:
return False
parsed = urlparse(value.strip())
return parsed.scheme.lower() in ('http', 'https') and bool(parsed.netloc)
def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]]:
"""
Extract author information from h-card.
@@ -212,26 +234,29 @@ def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]
if isinstance(author_data, dict):
# h-card object
props = author_data.get('properties', {})
author_name = props.get('name', [None])[0]
author_url = props.get('url', [None])[0]
author_photo = props.get('photo', [None])[0]
author_name = _first_str(props, 'name')
author_url = _first_str(props, 'url')
author_photo = _first_str(props, 'photo')
# If no author in h-entry, find representative h-card
if not author_name:
for item in parsed_mf2.get('items', []):
if 'h-card' in item.get('type', []):
props = item.get('properties', {})
author_name = props.get('name', [None])[0]
author_url = props.get('url', [None])[0]
author_photo = props.get('photo', [None])[0]
author_name = _first_str(props, 'name')
author_url = _first_str(props, 'url')
author_photo = _first_str(props, 'photo')
break
# Fallback to hostname if no author found
if not author_name:
author_name = urlparse(source_url).netloc
if not author_url:
# Only http(s) URLs are rendered as links (blocks javascript: and data:)
if not _is_http_url(author_url):
author_url = source_url
if not _is_http_url(author_photo):
author_photo = None
return {
'author_name': author_name,
+52 -12
View File
@@ -12,6 +12,7 @@ References:
- ADR-063: Webmention Implementation
"""
import fcntl
import logging
import threading
import time
@@ -25,7 +26,6 @@ from lxml import html as lxml_html
from starpunk.webmention_processor import process_webmention, ValidationError
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
from flask import has_app_context
logger = logging.getLogger(__name__)
@@ -47,6 +47,7 @@ class WebmentionWorker(threading.Thread):
def __init__(
self,
app,
db_path: Path,
site_url: str,
poll_interval: int = 10,
@@ -58,6 +59,7 @@ class WebmentionWorker(threading.Thread):
Initialize webmention worker thread
Args:
app: Flask app (the SSRF checks read its config)
db_path: Path to SQLite database
site_url: Site URL for User-Agent header
poll_interval: Seconds between queue checks (default: 10)
@@ -66,6 +68,7 @@ class WebmentionWorker(threading.Thread):
auto_approve: Auto-approve valid webmentions (default: True)
"""
super().__init__(daemon=True, name="webmention-worker")
self.app = app
self.db_path = db_path
self.site_url = site_url
self.poll_interval = poll_interval
@@ -74,6 +77,7 @@ class WebmentionWorker(threading.Thread):
self.auto_approve = auto_approve
self._stop_event = threading.Event()
self._last_processed = None
self._lock_file = None
# Phase 3: Discovery cache (worker instance variable)
# Per architect's answer to Q4
@@ -92,19 +96,23 @@ class WebmentionWorker(threading.Thread):
try:
logger.info("Webmention worker started")
# Cleanup orphaned items from previous crashes
self._cleanup_orphaned_items()
# Main processing loop
while not self._stop_event.is_set():
# Every gunicorn process starts a worker; only the one holding
# the lock processes queues, so nothing is sent twice
if not self._acquire_lock():
self._stop_event.wait(self.poll_interval)
continue
try:
processed = 0
# Process incoming webmentions
processed += self._process_pending_validations()
with self.app.app_context():
# Process incoming webmentions
processed += self._process_pending_validations()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Update last processed time
if processed > 0:
@@ -121,8 +129,40 @@ class WebmentionWorker(threading.Thread):
except Exception as e:
logger.error(f"Webmention worker thread failed: {e}", exc_info=True)
finally:
self._release_lock()
logger.info("Webmention worker stopped")
def _acquire_lock(self) -> bool:
"""
Take the cross-process worker lock (non-blocking)
On first acquisition, also reset items left mid-send by a previous
holder. The OS releases the lock if this process dies.
Returns:
True if this worker holds the lock
"""
if self._lock_file:
return True
lock_file = open(Path(self.db_path).parent / "webmention-worker.lock", "w")
try:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
lock_file.close()
return False
self._lock_file = lock_file
logger.info("Webmention worker holds the queue lock")
self._cleanup_orphaned_items()
return True
def _release_lock(self):
"""Release the cross-process worker lock if held"""
if self._lock_file:
self._lock_file.close()
self._lock_file = None
def _cleanup_orphaned_items(self):
"""
Clean up orphaned 'sending' status items on startup
@@ -507,7 +547,7 @@ class WebmentionWorker(threading.Thread):
# 1. Check Link header
link_header = response.headers.get('Link', '')
if link_header:
endpoint = self._parse_link_header(link_header, response.url)
endpoint = self._parse_link_header(link_header, str(response.url))
if endpoint:
self._cache_endpoint(target_url, endpoint, hours=1)
return endpoint
@@ -515,7 +555,7 @@ class WebmentionWorker(threading.Thread):
# 2. Parse HTML for <link> and <a> elements
content_type = response.headers.get('Content-Type', '')
if 'html' in content_type.lower():
endpoint = self._parse_html_for_endpoint(response.text, response.url)
endpoint = self._parse_html_for_endpoint(response.text, str(response.url))
if endpoint:
self._cache_endpoint(target_url, endpoint, hours=1)
return endpoint
@@ -526,9 +566,9 @@ class WebmentionWorker(threading.Thread):
return None
except Exception as e:
# Don't cache; the caller schedules a retry
logger.warning(f"Failed to discover endpoint for {target_url}: {e}")
# Don't cache on error - allow retry
return None
raise
def _parse_link_header(self, link_header: str, base_url: str) -> Optional[str]:
"""
+3 -2
View File
@@ -136,7 +136,8 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.button-warning { background: var(--color-warning); color: #333; border-color: var(--color-warning); }
.button-small { padding: 0.25rem 0.5rem; font-size: 0.875rem; }
.admin-container { max-width: 60rem; }
.admin-page { --max-width: 60rem; }
.table-scroll { overflow-x: auto; }
.admin-nav { display: flex; gap: var(--spacing-md); padding: var(--spacing-md); background: var(--color-bg-alt); border-radius: var(--border-radius); margin-bottom: var(--spacing-lg); flex-wrap: wrap; align-items: center; }
.admin-nav .logout-form { margin-left: auto; }
.admin-nav .logout-form button { margin: 0; }
@@ -147,7 +148,7 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.note-table { width: 100%; border-collapse: collapse; margin-top: var(--spacing-md); }
.note-table th, .note-table td { padding: var(--spacing-md); text-align: left; border-bottom: 1px solid var(--color-border); }
.note-table th { background: var(--color-bg-alt); font-weight: 600; }
.note-content-preview { max-width: 30rem; }
.note-content-preview { min-width: 14rem; overflow-wrap: anywhere; }
.note-excerpt { margin-bottom: var(--spacing-xs); }
.note-slug { color: var(--color-text-light); font-family: var(--font-mono); font-size: 0.75rem; }
.note-date { white-space: nowrap; color: var(--color-text-light); }
+3
View File
@@ -1,5 +1,8 @@
{% extends "base.html" %}
{# Admin pages use a wider frame so tables fit under the header #}
{% block body_class %} class="admin-page"{% endblock %}
{% block content %}
<div class="admin-container">
<nav class="admin-nav">
+2
View File
@@ -21,6 +21,7 @@
</div>
{% if notes %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -67,6 +68,7 @@
{% endfor %}
</tbody>
</table>
</div>
{# Pagination (v1.6.0) #}
{% if total_pages > 1 %}
+2
View File
@@ -8,6 +8,7 @@
<p>Micropub apps you have approved. Revoke any you no longer use.</p>
{% if tokens %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -37,6 +38,7 @@
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p>No apps connected.</p>
{% endif %}
+2
View File
@@ -43,6 +43,7 @@
</button>
</div>
<div class="table-scroll">
<table class="webmention-table">
<thead>
<tr>
@@ -85,6 +86,7 @@
{% endfor %}
</tbody>
</table>
</div>
</form>
{# Pagination #}
+1 -1
View File
@@ -27,7 +27,7 @@
{% block head %}{% endblock %}
</head>
<body>
<body{% block body_class %}{% endblock %}>
{% if config.DEV_MODE %}
<div class="dev-mode-warning">
WARNING: DEVELOPMENT MODE - Authentication bypassed
+128
View File
@@ -0,0 +1,128 @@
"""
Webmention worker regression tests (v2.6.8)
The worker thread used Flask's current_app (via the SSRF checks) without an
app context, so every send and validation failed. It was also stopped
whenever any app context ended.
"""
import sqlite3
import httpx
import time
from unittest.mock import MagicMock, patch
from starpunk.webmention_worker import WebmentionWorker
def _wait_for(predicate, timeout=5):
deadline = time.time() + timeout
while time.time() < deadline:
if predicate():
return True
time.sleep(0.05)
return False
def test_worker_sends_queued_webmention(app):
db_path = app.config["DATABASE_PATH"]
conn = sqlite3.connect(db_path)
conn.execute(
"INSERT INTO webmention_queue (source_url, target_url, status) "
"VALUES (?, ?, 'pending')",
("http://localhost:5000/note/1", "https://example.org/post"),
)
conn.commit()
conn.close()
page = httpx.Response(200, headers={"Link": '</wm>; rel="webmention"'},
request=httpx.Request("GET", "https://example.org/post"))
accepted = MagicMock(status_code=202, text="")
worker = WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"],
poll_interval=0.05)
with patch("starpunk.webmention_ssrf.socket.gethostbyname", return_value="93.184.216.34"), \
patch("starpunk.webmention_worker.httpx.get", return_value=page), \
patch("starpunk.webmention_worker.httpx.post", return_value=accepted) as post:
worker.start()
try:
def status():
c = sqlite3.connect(db_path)
row = c.execute("SELECT status, last_error FROM webmention_queue").fetchone()
c.close()
return row
assert _wait_for(lambda: status()[0] != "pending"), status()
assert status() == ("sent", None)
post.assert_called_once()
assert post.call_args.args[0] == "https://example.org/wm"
finally:
worker.stop()
worker.join(timeout=2)
def test_worker_survives_app_context_teardown(app):
worker = WebmentionWorker(app, db_path=app.config["DATABASE_PATH"],
site_url=app.config["SITE_URL"], poll_interval=0.05)
worker.start()
try:
with app.app_context():
pass
app.test_client().get("/")
time.sleep(0.2)
assert worker.is_alive()
finally:
worker.stop()
worker.join(timeout=2)
def test_only_one_worker_processes_queues(app):
"""Each gunicorn process starts a worker; only one may hold the queue lock"""
kwargs = dict(db_path=app.config["DATABASE_PATH"], site_url=app.config["SITE_URL"])
first = WebmentionWorker(app, **kwargs)
second = WebmentionWorker(app, **kwargs)
try:
assert first._acquire_lock()
assert not second._acquire_lock()
first._release_lock()
assert second._acquire_lock()
finally:
first._release_lock()
second._release_lock()
def test_author_photo_with_alt_text_is_a_string():
"""mf2py returns u-photo with alt as a dict; the database needs a URL string"""
import mf2py
from starpunk.webmention_processor import extract_author
html = """
<div class="h-entry">
<span class="p-author h-card">
<a class="u-url p-name" href="https://a.example/">A Person</a>
<img class="u-photo" src="https://a.example/me.jpg" alt=""></span>
<a href="https://site.example/note/1">target</a>
</div>"""
author = extract_author(mf2py.parse(doc=html, url="https://src.example/"),
"https://src.example/")
assert author == {
"author_name": "A Person",
"author_url": "https://a.example/",
"author_photo": "https://a.example/me.jpg",
}
def test_author_urls_must_be_http():
"""A javascript: author URL would be rendered as a clickable link (XSS)"""
import mf2py
from starpunk.webmention_processor import extract_author
html = """
<div class="h-entry">
<span class="p-author h-card">
<a class="u-url p-name" href="javascript:alert(1)">Evil</a>
<img class="u-photo" src="data:image/svg+xml,x" alt=""></span>
</div>"""
author = extract_author(mf2py.parse(doc=html, url="https://src.example/"),
"https://src.example/")
assert author["author_url"] == "https://src.example/"
assert author["author_photo"] is None