Compare commits

...
6 Commits
Author SHA1 Message Date
Phil 3c1e109358 Merge branch 'fix/v2.6.8-webmention-worker' for v2.6.8 release
Build Container / build (push) Successful in 20s
2026-11-03 07:41:11 -07:00
PhilandClaude Opus 5.5 d43361d505 fix: Webmention worker runs inside the app context (v2.6.8)
The worker thread called the SSRF checks, which read current_app.config,
without an app context, so every incoming and outgoing webmention failed with
'Working outside of application context'. A teardown_appcontext hook also
stopped the worker whenever a request ended.

Pass the app to WebmentionWorker and run each poll in app.app_context(), like
SyndicationWorker, and drop the teardown hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:41:06 -07:00
Phil f924413934 Merge branch 'fix/v2.6.7-admin-layout' for v2.6.7 release
Build Container / build (push) Successful in 7s
2026-11-02 18:24:15 -07:00
PhilandClaude Opus 5.5 4b2ae4c7fc fix: Admin pages line up with the header (v2.6.7)
Every page sat in the 40rem reading column, so the admin wrapper's 60rem
never applied and the notes table, which can't shrink that far, ran past
the header's right edge. Admin pages now set a 60rem frame for header,
content and footer; post previews wrap, and tables scroll sideways on
narrow screens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:24:14 -07:00
Phil d3148063e3 Merge branch 'fix/v2.6.6-inline-image-crosspost' for v2.6.6 release
Build Container / build (push) Successful in 15s
2026-11-02 18:12:51 -07:00
PhilandClaude Opus 5.5 39c8259f3f fix: Crossposts include photos embedded in the post text (v2.6.6)
Some Micropub clients (OutPost) upload a photo to the media endpoint and
then embed it in content instead of sending it as `photo`. Those posts
have no attached media, so crossposts went out without the image.

Images in the text that are in this site's media library are now sent
like attached photos, with their alt text, and GoToSocial's text no
longer carries the image Markdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:11:27 -07:00
13 changed files with 267 additions and 22 deletions

No files matched your search

+19
View File
@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.6.8] - 2026-11-03
### Fixed
- Webmentions were never processed. The background worker checked URLs against this site's settings without the app loaded, so every incoming webmention was marked "error" ("Working outside of application context") and every outgoing one kept failing and retrying. The worker now runs with the app loaded. A received webmention showed up in the admin list but never on the note
- The webmention worker also stopped as soon as any request finished, so after startup it often wasn't running at all. It now runs for the life of the server process
## [2.6.7] - 2026-11-02
### Fixed
- Admin pages were misaligned: the notes table ran past the right edge of the header and admin menu, because every page sat in the 40rem reading column and the table couldn't shrink to fit it. Admin pages now use a 60rem frame for the header, menu, content and footer alike, long post previews wrap, and on narrow screens tables scroll sideways instead of spilling out
## [2.6.6] - 2026-11-02
### Fixed
- Crossposts left out photos that were embedded in the post text instead of attached. Some Micropub clients (OutPost, for one) upload a photo to the media endpoint and then put it in `content` as an image rather than sending it as `photo`; both are valid Micropub. Images in the text that are in this site's media library are now sent to GoToSocial, Bluesky and Threads like attached photos, with their alt text, after any attached ones and within each target's photo limit. GoToSocial's text no longer includes the image Markdown. Images hosted on other sites are still left out
## [2.6.5] - 2026-11-01
### Fixed
@@ -0,0 +1,37 @@
# v2.6.8: Webmention worker app context
## Problem
Production webmention #1 (Bridgy, reply from Mastodon to note 20261101162512)
was stored with `validation_status = 'error'`:
"Working outside of application context". Both outgoing queue items had the
same error and were on their last retry.
## Causes
1. `webmention_ssrf.safe_resolve()` reads `current_app.config['DEV_MODE']`.
`WebmentionWorker` is a plain thread with no Flask app context, so every
`validate_url_for_fetch()` call raised `RuntimeError`, for both
validation (incoming) and endpoint discovery and sending (outgoing).
2. `create_app()` registered a `teardown_appcontext` handler that called
`webmention_worker.stop()`. Teardown runs whenever any app context ends
(every request), so the worker stopped soon after startup.
## Fix
- `WebmentionWorker` takes the Flask app and runs each poll inside
`app.app_context()`, as `SyndicationWorker` already does.
- The teardown handler is removed. The worker is a daemon thread and ends with
the process.
## Tests
`tests/test_webmention_worker.py`: an outgoing send completes through the
worker thread, and the worker is still alive after an app context and a
request end. Both fail without the fix.
## Production data
After deploying, reset webmention 1 to `validation_status = 'pending'` and the
two queue items to `retry_count = 0, next_retry_at = NULL` so the worker
processes them again.
+3 -8
View File
@@ -330,6 +330,7 @@ def create_app(config=None):
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
from starpunk.webmention_worker import WebmentionWorker
webmention_worker = WebmentionWorker(
app,
db_path=app.config['DATABASE_PATH'],
site_url=app.config['SITE_URL'],
poll_interval=app.config.get('WEBMENTION_POLL_INTERVAL', 10),
@@ -345,12 +346,6 @@ def create_app(config=None):
f"batch_size={webmention_worker.batch_size})"
)
# Register cleanup handler
@app.teardown_appcontext
def cleanup_webmention_worker(error=None):
if hasattr(app, 'webmention_worker') and app.webmention_worker.is_alive():
app.webmention_worker.stop()
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
@@ -464,5 +459,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.6.5"
__version_info__ = (2, 6, 4)
__version__ = "2.6.8"
__version_info__ = (2, 6, 8)
+68 -6
View File
@@ -138,7 +138,9 @@ class MastodonClient:
def _prepare(self, post: "Post") -> tuple:
if post.kind == "repost":
return compose_repost_text(post, post.note.content, post.max_chars), []
text = compose_text(post.note, post.permalink, post.tags, post.max_chars)
# Images in the text are sent as attachments instead (v2.6.6)
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=strip_images(post.note.content))
media_ids = [self.upload_media(f["path"], f["mime_type"], f["description"]) for f in post.media]
return text, media_ids
@@ -1135,25 +1137,84 @@ def bluesky_facets(text: str, anchors: List[tuple] = ()) -> List[dict]:
return facets
def _media_files(note_id: int, limit: int) -> List[dict]:
"""Image files for a note: the 'large' variant when there is one"""
_MD_IMAGE_RE = re.compile(r"!\[[^\]]*\]\([^)]*\)")
_HTML_IMAGE_RE = re.compile(r"<img\b[^>]*>", re.IGNORECASE)
def inline_images(markdown_text: str) -> List[tuple]:
"""(url, alt) for each image embedded in a note's text, in reading order"""
images = []
for img in _note_soup(markdown_text).find_all("img"):
if img.get("src"):
images.append((img["src"], img.get("alt") or ""))
return images
def strip_images(markdown_text: str) -> str:
"""A note's Markdown without its embedded images"""
text = _HTML_IMAGE_RE.sub("", _MD_IMAGE_RE.sub("", markdown_text))
return re.sub(r"\n{3,}", "\n\n", text).strip()
def _own_media_path(url: str) -> Optional[str]:
"""Path in the media library for one of this site's /media/ URLs"""
site = current_app.config["SITE_URL"].rstrip("/")
if url.startswith(site):
url = url[len(site):]
return url[len("/media/"):] if url.startswith("/media/") else None
def _media_files(note_id: int, limit: int, content: str = "") -> List[dict]:
"""
Image files for a note: the 'large' variant when there is one.
Attached photos come first, then images embedded in the text that are in
this site's media library. Some Micropub clients (OutPost, for one)
upload a photo and put it in the content rather than in `photo` (v2.6.6).
"""
from starpunk.media import get_note_media
media_dir = Path(current_app.config["DATA_PATH"]) / "media"
site = current_app.config["SITE_URL"].rstrip("/")
items = [(m, m.get("caption") or "") for m in get_note_media(note_id)]
seen = {m["path"] for m, _ in items}
for url, alt in inline_images(content) if content else []:
path = _own_media_path(url)
if not path or path in seen:
continue
media = _media_by_path(path)
if media:
seen.add(path)
items.append((media, alt or media.get("caption") or ""))
files = []
for media in get_note_media(note_id)[:limit]:
for media, description in items[:limit]:
variant = (media.get("variants") or {}).get("large")
rel = variant["path"] if variant else media["path"]
files.append({
"path": media_dir / rel,
"url": f"{site}/media/{rel}", # for targets that fetch images themselves
"mime_type": media.get("mime_type") or "image/jpeg",
"description": media.get("caption") or "",
"description": description,
})
return files
def _media_by_path(path: str) -> Optional[dict]:
"""A media row with its variants, found by its path in the library"""
db = get_db(current_app)
row = db.execute("SELECT id, path, mime_type FROM media WHERE path = ?", (path,)).fetchone()
if not row:
return None
variants = {
v["variant_type"]: {"path": v["path"]}
for v in db.execute(
"SELECT variant_type, path FROM media_variants WHERE media_id = ?", (row["id"],)
)
}
return {"path": row["path"], "mime_type": row["mime_type"], "variants": variants, "caption": ""}
# --- Worker -----------------------------------------------------------------
def _claim(db, row_id: int) -> bool:
@@ -1242,7 +1303,8 @@ def _reconcile(row, target: Target) -> tuple:
permalink=f"{site}/note/{note.slug}",
tags=[t["display_name"] for t in get_note_tags(note.id, entered_order=True)],
max_chars=target.max_chars,
media=_media_files(note.id, target.max_media) if note.kind == "note" else [],
media=_media_files(note.id, target.max_media, note.content)
if note.kind == "note" else [],
kind=note.kind,
target_url=note.reacts_to,
target_title=_target_title(note.reacts_to),
+8 -5
View File
@@ -25,7 +25,6 @@ from lxml import html as lxml_html
from starpunk.webmention_processor import process_webmention, ValidationError
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
from flask import has_app_context
logger = logging.getLogger(__name__)
@@ -47,6 +46,7 @@ class WebmentionWorker(threading.Thread):
def __init__(
self,
app,
db_path: Path,
site_url: str,
poll_interval: int = 10,
@@ -58,6 +58,7 @@ class WebmentionWorker(threading.Thread):
Initialize webmention worker thread
Args:
app: Flask app (the SSRF checks read its config)
db_path: Path to SQLite database
site_url: Site URL for User-Agent header
poll_interval: Seconds between queue checks (default: 10)
@@ -66,6 +67,7 @@ class WebmentionWorker(threading.Thread):
auto_approve: Auto-approve valid webmentions (default: True)
"""
super().__init__(daemon=True, name="webmention-worker")
self.app = app
self.db_path = db_path
self.site_url = site_url
self.poll_interval = poll_interval
@@ -100,11 +102,12 @@ class WebmentionWorker(threading.Thread):
try:
processed = 0
# Process incoming webmentions
processed += self._process_pending_validations()
with self.app.app_context():
# Process incoming webmentions
processed += self._process_pending_validations()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Update last processed time
if processed > 0:
+3 -2
View File
@@ -136,7 +136,8 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.button-warning { background: var(--color-warning); color: #333; border-color: var(--color-warning); }
.button-small { padding: 0.25rem 0.5rem; font-size: 0.875rem; }
.admin-container { max-width: 60rem; }
.admin-page { --max-width: 60rem; }
.table-scroll { overflow-x: auto; }
.admin-nav { display: flex; gap: var(--spacing-md); padding: var(--spacing-md); background: var(--color-bg-alt); border-radius: var(--border-radius); margin-bottom: var(--spacing-lg); flex-wrap: wrap; align-items: center; }
.admin-nav .logout-form { margin-left: auto; }
.admin-nav .logout-form button { margin: 0; }
@@ -147,7 +148,7 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.note-table { width: 100%; border-collapse: collapse; margin-top: var(--spacing-md); }
.note-table th, .note-table td { padding: var(--spacing-md); text-align: left; border-bottom: 1px solid var(--color-border); }
.note-table th { background: var(--color-bg-alt); font-weight: 600; }
.note-content-preview { max-width: 30rem; }
.note-content-preview { min-width: 14rem; overflow-wrap: anywhere; }
.note-excerpt { margin-bottom: var(--spacing-xs); }
.note-slug { color: var(--color-text-light); font-family: var(--font-mono); font-size: 0.75rem; }
.note-date { white-space: nowrap; color: var(--color-text-light); }
+3
View File
@@ -1,5 +1,8 @@
{% extends "base.html" %}
{# Admin pages use a wider frame so tables fit under the header #}
{% block body_class %} class="admin-page"{% endblock %}
{% block content %}
<div class="admin-container">
<nav class="admin-nav">
+2
View File
@@ -21,6 +21,7 @@
</div>
{% if notes %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -67,6 +68,7 @@
{% endfor %}
</tbody>
</table>
</div>
{# Pagination (v1.6.0) #}
{% if total_pages > 1 %}
+2
View File
@@ -8,6 +8,7 @@
<p>Micropub apps you have approved. Revoke any you no longer use.</p>
{% if tokens %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -37,6 +38,7 @@
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p>No apps connected.</p>
{% endif %}
+2
View File
@@ -43,6 +43,7 @@
</button>
</div>
<div class="table-scroll">
<table class="webmention-table">
<thead>
<tr>
@@ -85,6 +86,7 @@
{% endfor %}
</tbody>
</table>
</div>
</form>
{# Pagination #}
+1 -1
View File
@@ -27,7 +27,7 @@
{% block head %}{% endblock %}
</head>
<body>
<body{% block body_class %}{% endblock %}>
{% if config.DEV_MODE %}
<div class="dev-mode-warning">
WARNING: DEVELOPMENT MODE - Authentication bypassed
+47
View File
@@ -32,6 +32,7 @@ class FakeServer:
self.statuses = {}
self.calls = []
self.media = 0
self.uploads = [] # raw multipart bodies
self.fail_next = None # status code to return once
def __call__(self, request: httpx.Request) -> httpx.Response:
@@ -43,6 +44,7 @@ class FakeServer:
path = request.url.path
if path == "/api/v2/media":
self.media += 1
self.uploads.append(request.read())
return httpx.Response(200, json={"id": f"m{self.media}"})
if path == "/api/v1/statuses" and request.method == "POST":
body = json.loads(request.content)
@@ -271,6 +273,51 @@ class TestMedia:
assert server.statuses["1"]["media_ids"] == ["m1"]
class TestInlineImages:
"""Clients like OutPost upload a photo and embed it in the content (v2.6.6)"""
def _upload(self, app):
from starpunk.media import save_media
buf = io.BytesIO()
Image.new("RGB", (800, 600), "orange").save(buf, format="JPEG")
with app.test_request_context():
return save_media(buf.getvalue(), "cat.jpg")
def test_embedded_own_image_is_attached(self, app, server):
media = self._upload(app)
url = f"https://blog.example.com/media/{media['path']}"
publish(app, content=f"![Tortie cat on a chair]({url})")
status = server.statuses["1"]
assert status["media_ids"] == ["m1"]
assert b"Tortie cat on a chair" in server.uploads[0]
assert "![" not in status["status"] and url not in status["status"]
assert status["status"].startswith("https://blog.example.com/note/")
def test_text_around_image_is_kept(self, app, server):
media = self._upload(app)
publish(app, content=f"Look\n\n<img src=\"/media/{media['path']}\" alt=\"cat\">\n\nNice")
status = server.statuses["1"]
assert status["media_ids"] == ["m1"]
assert status["status"].startswith("Look\n\nNice")
def test_image_both_attached_and_embedded_is_sent_once(self, app, server):
from starpunk.media import attach_media_to_note
media = self._upload(app)
with app.test_request_context():
note = create_note(content=f"![x](/media/{media['path']})", published=True)
attach_media_to_note(note.id, [media["id"]], ["caption"])
select_targets(note.id, ["gotosocial"])
process_due()
assert server.statuses["1"]["media_ids"] == ["m1"]
def test_other_sites_images_are_ignored(self, app, server):
publish(app, content="![x](https://elsewhere.example/cat.jpg) hi")
assert server.statuses["1"]["media_ids"] == []
assert server.uploads == []
class TestEntryPoints:
@pytest.fixture
def admin(self, app):
+72
View File
@@ -0,0 +1,72 @@
"""
Webmention worker regression tests (v2.6.8)
The worker thread used Flask's current_app (via the SSRF checks) without an
app context, so every send and validation failed. It was also stopped
whenever any app context ended.
"""
import sqlite3
import time
from unittest.mock import MagicMock, patch
from starpunk.webmention_worker import WebmentionWorker
def _wait_for(predicate, timeout=5):
deadline = time.time() + timeout
while time.time() < deadline:
if predicate():
return True
time.sleep(0.05)
return False
def test_worker_sends_queued_webmention(app):
db_path = app.config["DATABASE_PATH"]
conn = sqlite3.connect(db_path)
conn.execute(
"INSERT INTO webmention_queue (source_url, target_url, status) "
"VALUES (?, ?, 'pending')",
("http://localhost:5000/note/1", "https://example.org/post"),
)
conn.commit()
conn.close()
page = MagicMock(status_code=200, url="https://example.org/post",
headers={"Link": '<https://example.org/wm>; rel="webmention"'})
accepted = MagicMock(status_code=202, text="")
worker = WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"],
poll_interval=0.05)
with patch("starpunk.webmention_ssrf.socket.gethostbyname", return_value="93.184.216.34"), \
patch("starpunk.webmention_worker.httpx.get", return_value=page), \
patch("starpunk.webmention_worker.httpx.post", return_value=accepted) as post:
worker.start()
try:
def status():
c = sqlite3.connect(db_path)
row = c.execute("SELECT status, last_error FROM webmention_queue").fetchone()
c.close()
return row
assert _wait_for(lambda: status()[0] != "pending"), status()
assert status() == ("sent", None)
post.assert_called_once()
finally:
worker.stop()
worker.join(timeout=2)
def test_worker_survives_app_context_teardown(app):
worker = WebmentionWorker(app, db_path=app.config["DATABASE_PATH"],
site_url=app.config["SITE_URL"], poll_interval=0.05)
worker.start()
try:
with app.app_context():
pass
app.test_client().get("/")
time.sleep(0.2)
assert worker.is_alive()
finally:
worker.stop()
worker.join(timeout=2)