Compare commits

...
8 Commits
Author SHA1 Message Date
Phil 57682165a4 Merge branch 'fix/v2.8.1-migration-race' for v2.8.1 release
Build Container / build (push) Successful in 7m34s
2026-11-03 09:58:20 -07:00
PhilandClaude Opus 5.5 8c51b52ed6 fix: Hold the migration lock for the whole run (v2.8.1)
create_migrations_table() committed right after BEGIN IMMEDIATE, and
executescript() commits before running, so gunicorn workers applied
migrations concurrently; the v2.8.0 deploy failed to boot once with
'duplicate column name'. Run migration statements with conn.execute inside
the transaction and don't commit when creating the tracking table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:58:20 -07:00
Phil 42c77e34b7 Merge branch 'feat/v2.8.0-microformats' for v2.8.0 release
Build Container / build (push) Successful in 9m35s
2026-11-03 09:36:53 -07:00
PhilandClaude Opus 5.5 438465592c feat: Microformats2 review fixes (v2.8.0)
- Responses as u-comment/u-like/u-repost/u-bookmark/u-mention h-cites
  with author h-cards; source's own url and published (migration 014)
- dt-* and Micropub published in RFC 3339 UTC
- Title only for a real Markdown heading (hashtag notes stay notes)
- Ignore webmentions from a note's own syndicated copies
- Micropub/IndieAuth links independent of Webmention; rel=me/rel=author
- Representative h-card and authorship algorithm (starpunk/mf2_utils.py)
- Tag/likes h-feeds get url and author; previews get u-uid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:36:49 -07:00
Phil c8a7ef309f Merge branch 'feat/v2.7.2-webmention-shoulds' for v2.7.2 release
Build Container / build (push) Successful in 9m28s
2026-11-03 09:01:42 -07:00
PhilandClaude Opus 5.5 0640730416 feat: Pin webmention connections; re-send when responses change (v2.7.2)
- Connect to the address that passed the SSRF check (Host header and SNI
  carry the hostname, so TLS still verifies it), closing the DNS rebinding
  gap (spec 4.5). The webmention POST uses the same path.
- Treat a change in the responses a note shows as an update and re-send its
  webmentions (spec 3.1.4), from validation and from admin moderation.
  Unchanged re-validations send nothing, so there is no ping-pong.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:01:38 -07:00
Phil 51f711ea4a Merge branch 'ci/v2.7.1-pytest' for v2.7.1 release
Build Container / build (push) Successful in 9m13s
2026-11-03 08:38:40 -07:00
PhilandClaude Opus 5.5 6a76ee26af ci: Run the test suite before building release images (v2.7.1)
Add a 'test' stage to the Containerfile (production base image plus pytest)
and build it in the release workflow before the push step, so a failing test
stops the release.

Fix the eight tests that were failing: media validation returns a PIL Image
since v1.5.0, and the migration test needs TESTING to skip production config
checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:28:08 -07:00
34 changed files with 1318 additions and 322 deletions

No files matched your search

+5 -1
View File
@@ -1,5 +1,5 @@
# Gitea Actions workflow for StarPunk
# Builds and pushes container images on version tags.
# Runs the tests, then builds and pushes container images on version tags.
#
# The runner has no Docker daemon (that would give workflows root on the
# server). Images are built by the rootless BuildKit container next to it,
@@ -30,6 +30,10 @@ jobs:
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "$REGISTRY" -u "${{ github.repository_owner }}" --password-stdin
- name: Use the BuildKit socket
run: docker buildx create --name ci --driver remote "$BUILDKIT" --use
# Builds the Containerfile's test stage, which runs pytest; a failure
# stops the workflow before anything is pushed
- name: Run tests
run: docker buildx build --target test --progress plain -f Containerfile .
- name: Build and push
run: |
image="$REGISTRY/$(echo '${{ github.repository }}' | tr '[:upper:]' '[:lower:]')"
+48
View File
@@ -7,6 +7,54 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.8.1] - 2026-11-03
### Fixed
- Starting the server after an upgrade with a database migration could fail: several server processes applied the same migration at once and one stopped with "duplicate column name", taking the server down until it restarted (this happened deploying v2.8.0, for about two seconds). The migration lock was released right after it was taken, and again before each migration was recorded. Migrations now run entirely inside the lock
## [2.8.0] - 2026-11-03
Microformats2 review: the markup now matches the [microformats2](https://microformats.org/wiki/microformats2) vocabularies, so IndieWeb readers and tools see what the page shows.
### Fixed
- Replies to a note weren't marked up as its comments, so readers and threading tools saw none. They are now `u-comment h-cite`s with the replier's h-card (name, link, photo) and the reply's HTML
- Likes, reposts, bookmarks and mentions on a note had no microformats at all. Each is now a `u-like`, `u-repost`, `u-bookmark` or `u-mention` h-cite with its author, including likes past the eight faces shown
- Responses link to and are dated by the original post (its own URL and published time) instead of the address that sent the webmention and when it arrived. New database columns hold these (migration 014)
- Post times had no time zone (and production showed microseconds), so readers could shift them by hours. Times on pages and in Micropub `q=source` are now UTC with a `Z`, matching the Atom feed
- A note starting with a hashtag (`#indieweb is great…`) was treated as a titled article: its text became a heading and the body was empty, and a note starting with a bare `#` line got its slug as a title. Only a Markdown heading (`#` then a space) makes a title now
- A webmention from one of a note's own crossposts (for example your GoToSocial copy, sent back through Bridgy) showed up as a response on the note. It is now ignored
- Micropub and IndieAuth discovery links disappeared from pages when webmentions were turned off
- A new site had no `rel="me"` link to its author, so readers couldn't tell whose site it was; pages now always link `rel="me"` and `rel="author"` to the author's home page
- Finding the author's profile could pick someone else's h-card from their home page. It now follows the [representative h-card](https://microformats.org/wiki/representative-h-card-parsing) rules
- A webmention's author is found by the [authorship algorithm](https://indieweb.org/authorship-spec), from the entry on the source page that actually responds to the note: an author given as a name or link, and the author of a feed the entry is in, now count
### Added
- Tag pages and the likes page have a feed URL and author, like the home page
- Note previews have `u-uid`
## [2.7.2] - 2026-11-03
### Added
- When the replies, likes and other responses shown on a note change (a webmention arrives, is updated or removed, or you approve, hide or delete one), the note's webmentions are sent again, as the Webmention spec recommends, so the sites it links to can pick up the conversation. A response that is checked again without changing sends nothing
### Security
- Webmention fetches and sends connect to the exact address that passed the private-address check. Before, the hostname was looked up a second time when connecting, so a DNS server that changed its answer could still point StarPunk at a private address. TLS certificates are still checked against the hostname. IPv4 addresses are preferred when a host has both
## [2.7.1] - 2026-11-03
### Changed
- Releases run the full test suite first: CI builds the Containerfile's new `test` stage (same Python image as production) and stops before pushing an image if any test fails
### Fixed
- Eight tests that had been failing: the image validation tests still expected bytes (validation has returned an image object since v1.5.0), and a migration test started the app in production mode
## [2.7.0] - 2026-11-03
Webmention now follows the [W3C Recommendation](https://www.w3.org/TR/webmention/) throughout.
+21 -1
View File
@@ -23,7 +23,27 @@ RUN uv venv /opt/venv && \
uv pip install --no-cache -r requirements.txt
# ============================================================================
# Runtime Stage - Minimal production image
# Test Stage - Runs the test suite (CI builds it with --target test)
# ============================================================================
FROM python:3.11-slim AS test
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
COPY --from=builder /opt/venv /opt/venv
RUN uv pip install --no-cache --python /opt/venv/bin/python pytest pytest-mock
ENV PATH="/opt/venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
SESSION_SECRET=ci-test-secret \
SITE_URL=http://localhost:5000 \
ADMIN_ME=https://test.example.com
WORKDIR /app
COPY . .
RUN python -m pytest -q -p no:cacheprovider tests
# ============================================================================
# Runtime Stage - Minimal production image (default target: keep it last)
# ============================================================================
FROM python:3.11-slim
@@ -40,9 +40,10 @@ v2.6.8 and v2.6.9 fixed the bugs that stopped webmentions working at all.
## Known limits
- DNS is checked before connecting, not pinned for the connection, so a
rebinding DNS server could still answer differently on connect.
- Homepage mentions are refused rather than stored.
- Discovery does not apply HTTP cache headers (5.3); targets are only
fetched when a note is created, edited or deleted.
- The 410 page is a generic tombstone without microformats.
- `error` rows (unexpected failures) are not retried automatically.
## Tests
@@ -0,0 +1,36 @@
# v2.7.2: Remaining Webmention SHOULDs
Follow-up to the v2.7.0 compliance review. Every SHOULD in the
Recommendation was checked; two were not met.
## 4.5 DNS pinning
`validate_url_for_fetch()` now returns the checked address.
`_pinned_request()` builds each request against that IP, with the original
host in the `Host` header and, for https, in the `sni_hostname` extension.
httpcore uses `sni_hostname` as `server_hostname` with `check_hostname`
on, so the certificate is verified against the real hostname. Used by
`safe_get()` (every redirect hop) and the new `safe_post()` (the webmention
POST). IPv4 is preferred when a host has both families.
## 3.1.4 Displayed responses are an update
"If the source post displays responses... then sender SHOULD treat that as
an update of the source URL and re-send any previously sent Webmentions."
`note_responses_changed()` re-queues a published note's previous webmentions.
It is called when:
- the worker validates a webmention and what the note shows for it changed
(`_display_state()` before and after: shown or not, type, author, snippet).
A re-send from the source resets the row to `pending`; a previously valid
row still counts as shown, so an unchanged re-validation sends nothing.
This keeps two sites that both implement this from ping-ponging.
- an admin approves, hides or deletes webmentions and a valid one changes
between shown and not shown.
## Tests
`tests/test_webmention_spec.py`: pinning (URL, Host, SNI, one lookup per
request), IPv4 preference, re-send on a new response, none on an unchanged
one, re-send on moderation changes.
@@ -0,0 +1,38 @@
# v2.8.0: Microformats2 review
Review of StarPunk's published and consumed microformats against
microformats2, h-entry, h-card, h-feed, h-cite, representative h-card
parsing, the authorship spec and Post Type Discovery. Every page type was
rendered, parsed with mf2py and checked with mf2util.
Posts (h-entry) were largely correct: no implied names, PTD right for
note/article/reply/like/repost/photo, author h-card, u-url/u-uid,
p-category, u-syndication.
## Fixes
| # | Problem | Fix |
|---|---|---|
| 1 | Replies were unattached `h-cite` children with a plain-text author | `u-comment h-cite`, `p-author h-card` (u-url on an inner link, u-photo, p-name), `e-content` (`templates/partials/responses.html`) |
| 2 | Likes/reposts/bookmarks/mentions had no mf2 | `u-like` / `u-repost` / `u-bookmark` / `u-mention` h-cites; faces past 8 are `hidden`, not omitted |
| 3 | `dt-*` naive, with microseconds | `iso_utc` filter/helper: RFC 3339 UTC `Z`, no fraction; Micropub `q=source` too |
| 4 | `#hashtag` first line = title; bare `#` = slug title | `split_title` requires an ATX heading (`#{1,6}` + space); templates use `Note.has_title` |
| 5 | Note's own syndicated copy shown as a mention | Worker rejects a source whose url/uid (or URL) is one of the note's `syndications.remote_url` as invalid |
| 6 | Micropub/IndieAuth links gated on WEBMENTION_ENABLED | Only `rel=webmention` is gated |
| 7 | No rel=me on a fresh install | `rel=me` and `rel=author` to `author.url or author.me or ADMIN_ME`, always |
| 8 | Author discovery guessed h-cards | `mf2_utils.representative_hcard` (uid+url, rel=me, single h-card); parsed with the URL after redirects |
| 9 | Webmention entry/author/time | `mf2_utils.responding_entry` (entry whose in-reply-to/like-of/... or content refers to the target), `entry_author` (author property as h-card/URL/name, parent h-feed author, rel=author, representative h-card), `source_post_url` and `source_published` stored (migration 014) |
Also: tag and likes h-feeds have `u-url` and `p-author`; previews have `u-uid`.
## Not done
- Reply-context h-cite does not carry the cached author h-card, name or published time.
- Photo alt falls back to "Image" when a photo-only post has no caption.
- Person tags (`p-category h-card`) are not supported.
- Existing webmentions keep NULL source details until they are re-validated.
## Tests
`tests/test_microformats_v280.py`. `tests/test_microformats.py`
`test_no_relme_without_author` became `test_relme_to_admin_me_without_author`.
@@ -0,0 +1,33 @@
# v2.8.1: Migration lock held for the whole run
## Incident
Deploying v2.8.0 (migration 014 adds two columns): one gunicorn worker
applied 014, another failed with `duplicate column name: source_post_url`,
gunicorn shut down ("Worker failed to boot") and Docker restarted the
container. About two seconds of downtime; schema correct afterwards.
## Causes
`run_migrations` takes `BEGIN IMMEDIATE` so one worker migrates at a time,
but the lock was released early twice:
1. `create_migrations_table()`, called right after `BEGIN IMMEDIATE`,
ended with `conn.commit()`. Everything after it ran without the lock.
2. `conn.executescript()` COMMITs any open transaction before running the
script, so the lock was dropped before the migration was recorded in
`schema_migrations`.
## Fix
- `create_migrations_table(conn, commit=True)`; `run_migrations` passes
`commit=False`.
- `execute_in_transaction()` runs each statement from
`split_sql_statements()` (built on `sqlite3.complete_statement`, so
trigger bodies stay whole) with `conn.execute`, inside the transaction.
## Verification
Four processes migrating a database at 013 simultaneously: 6/30 trials
failed before, 0/30 after. `tests/test_migration_concurrency.py` runs this
five times, plus a rollback test showing a migration commits nothing early.
@@ -0,0 +1,6 @@
-- Migration 014: Source details for received webmentions (v2.8.0)
-- The source's own u-url and dt-published, so a response links to and dates
-- the original post rather than the URL that sent it and when it arrived.
ALTER TABLE webmentions ADD COLUMN source_post_url TEXT;
ALTER TABLE webmentions ADD COLUMN source_published TIMESTAMP;
+6 -2
View File
@@ -219,6 +219,10 @@ def create_app(config=None):
return {'author': author}
# Template filter - RFC 3339 UTC for dt-* properties (v2.8.0)
from starpunk.utils import iso_utc
app.add_template_filter(iso_utc, 'iso_utc')
# Template filter - URL to color for facepile (v1.6.0)
@app.template_filter('url_to_color')
def url_to_color_filter(url):
@@ -459,5 +463,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.7.0"
__version_info__ = (2, 7, 0)
__version__ = "2.8.1"
__version_info__ = (2, 8, 1)
+13 -37
View File
@@ -82,10 +82,12 @@ def discover_author_profile(me_url: str) -> Optional[Dict]:
response.raise_for_status()
# Parse microformats from HTML
parsed = mf2py.parse(doc=response.text, url=me_url)
# Relative URLs and the representative h-card use the URL after redirects
page_url = str(response.url) if isinstance(response.url, (str, httpx.URL)) else me_url
parsed = mf2py.parse(doc=response.text, url=page_url)
# Extract h-card (per Q16: first representative h-card)
hcard = _find_representative_hcard(parsed, me_url)
# Representative h-card (v2.8.0: per the parsing spec)
hcard = _find_representative_hcard(parsed, page_url)
if not hcard:
current_app.logger.warning(f"No h-card found at {me_url}")
@@ -132,47 +134,21 @@ def discover_author_profile(me_url: str) -> Optional[Dict]:
def _find_representative_hcard(parsed: dict, me_url: str) -> Optional[dict]:
"""
Find representative h-card from parsed microformats
Find the representative h-card of the author's page (v2.8.0)
Per Q16: First representative h-card = first h-card with p-name
Per Q18: First h-card with url property matching profile URL
Follows https://microformats.org/wiki/representative-h-card-parsing
(uid+url match, rel=me match, or the only h-card with a matching url).
Other h-cards on the page may be other people, so none is guessed.
Args:
parsed: Parsed microformats data from mf2py
me_url: Profile URL for matching
me_url: Profile URL (the page that was parsed)
Returns:
h-card dict or None if not found
h-card dict or None if the page has no representative h-card
"""
items = parsed.get('items', [])
# First try: h-card with matching URL (most specific)
for item in items:
if 'h-card' in item.get('type', []):
properties = item.get('properties', {})
urls = properties.get('url', [])
# Check if any URL matches the profile URL
for url in urls:
if isinstance(url, dict):
url = url.get('value', '')
if _normalize_url(url) == _normalize_url(me_url):
# Found matching h-card
return item
# Second try: First h-card with p-name (representative h-card)
for item in items:
if 'h-card' in item.get('type', []):
properties = item.get('properties', {})
if properties.get('name'):
return item
# Third try: Just use first h-card if any
for item in items:
if 'h-card' in item.get('type', []):
return item
return None
from starpunk.mf2_utils import representative_hcard
return representative_hcard(parsed, me_url)
def _get_property(hcard: dict, prop_name: str) -> Optional[str]:
+168
View File
@@ -0,0 +1,168 @@
"""
Microformats2 consuming helpers (v2.8.0)
Shared by Webmention processing and author discovery:
- representative h-card of a page
- the h-entry on a page that responds to a given URL
- the author of an h-entry (authorship algorithm)
References:
- https://microformats.org/wiki/representative-h-card-parsing
- https://indieweb.org/authorship-spec
"""
from typing import Iterator, Optional
from urllib.parse import urldefrag, urlparse
RESPONSE_PROPERTIES = ('in-reply-to', 'like-of', 'repost-of', 'bookmark-of')
def normalize_url(url: str) -> str:
"""URL for comparison: no fragment, no trailing slash, lowercase scheme/host"""
if not url:
return ''
url = urldefrag(url.strip())[0]
parsed = urlparse(url)
url = parsed._replace(scheme=parsed.scheme.lower(), netloc=parsed.netloc.lower()).geturl()
return url.rstrip('/')
def value_of(value) -> Optional[str]:
"""Plain value of an mf2 property value (strings, {'value':...}, {'html':...})"""
if isinstance(value, dict):
value = value.get('value')
return value if isinstance(value, str) and value else None
def first(item: dict, prop: str) -> Optional[str]:
"""First plain value of an item's property"""
for value in item.get('properties', {}).get(prop, []):
result = value_of(value)
if result:
return result
return None
def values(item: dict, prop: str) -> list[str]:
"""All plain values of an item's property"""
return [v for v in (value_of(x) for x in item.get('properties', {}).get(prop, [])) if v]
def walk(items: list) -> Iterator[dict]:
"""Every microformat in items, depth first: top level, children, nested properties"""
for item in items:
if not isinstance(item, dict) or 'type' not in item:
continue
yield item
yield from walk(item.get('children', []))
for prop_values in item.get('properties', {}).values():
yield from walk([v for v in prop_values if isinstance(v, dict)])
def is_type(item: dict, mf_type: str) -> bool:
return mf_type in item.get('type', [])
def representative_hcard(parsed: dict, page_url: str) -> Optional[dict]:
"""
The page's representative h-card, per the parsing spec, in order:
1. An h-card whose uid and url both match the page URL
2. An h-card whose url matches one of the page's rel=me URLs
3. The only h-card on the page, if its url matches the page URL
Otherwise the page has none.
"""
page = normalize_url(page_url)
cards = [item for item in walk(parsed.get('items', [])) if is_type(item, 'h-card')]
for card in cards:
if page in map(normalize_url, values(card, 'uid')) and \
page in map(normalize_url, values(card, 'url')):
return card
rel_me = {normalize_url(u) for u in parsed.get('rels', {}).get('me', [])}
for card in cards:
if rel_me & set(map(normalize_url, values(card, 'url'))):
return card
if len(cards) == 1 and page in map(normalize_url, values(cards[0], 'url')):
return cards[0]
return None
def responding_entry(parsed: dict, target_url: str) -> Optional[dict]:
"""
The h-entry on a page that refers to target_url
Prefers an entry whose in-reply-to / like-of / repost-of / bookmark-of
is the target, then one whose content links to it, then the first
top-level h-entry.
"""
target = normalize_url(target_url)
entries = [item for item in walk(parsed.get('items', [])) if is_type(item, 'h-entry')]
top = [item for item in parsed.get('items', []) if is_type(item, 'h-entry')]
if not target:
return top[0] if top else (entries[0] if entries else None)
for entry in entries:
for prop in RESPONSE_PROPERTIES:
if target in map(normalize_url, values(entry, prop)):
return entry
for entry in entries:
for content in entry.get('properties', {}).get('content', []):
html = content.get('html', '') if isinstance(content, dict) else str(content)
if target_url in html or target in html:
return entry
return top[0] if top else (entries[0] if entries else None)
def response_type(entry: dict, target_url: str) -> str:
"""reply / like / repost / bookmark if the entry is one to target_url, else mention"""
target = normalize_url(target_url)
for prop, kind in zip(RESPONSE_PROPERTIES, ('reply', 'like', 'repost', 'bookmark')):
if target in map(normalize_url, values(entry, prop)):
return kind
return 'mention'
def entry_author(parsed: dict, entry: Optional[dict], page_url: str) -> dict:
"""
Author of an h-entry, following the authorship algorithm without fetching
1. The entry's author property (an h-card, a URL, or a name)
2. The author of an h-feed that contains the entry
3. The page's rel=author link
4. The page's representative h-card
Returns:
{'name', 'url', 'photo'}; values may be None
"""
author = None
if entry:
author = (entry.get('properties', {}).get('author') or [None])[0]
if author is None:
for feed in walk(parsed.get('items', [])):
if is_type(feed, 'h-feed') and entry in feed.get('children', []):
author = (feed.get('properties', {}).get('author') or [None])[0]
break
if author is None:
rel_author = parsed.get('rels', {}).get('author', [])
author = rel_author[0] if rel_author else representative_hcard(parsed, page_url)
if isinstance(author, dict) and 'type' in author:
return {
'name': first(author, 'name'),
'url': first(author, 'url'),
'photo': first(author, 'photo'),
}
text = value_of(author)
if text and urlparse(text).scheme in ('http', 'https'):
return {'name': None, 'url': text, 'photo': None}
return {'name': text, 'url': None, 'photo': None}
+2 -2
View File
@@ -36,7 +36,7 @@ from starpunk.notes import (
InvalidNoteDataError, NoteNotFoundError,
)
from starpunk.indieauth import check_scope
from starpunk.utils import absolute_url_for
from starpunk.utils import absolute_url_for, iso_utc
# Custom Exceptions
@@ -743,7 +743,7 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
title, content = split_title(note.content)
props = {
"content": [content],
"published": [note.created_at.isoformat()],
"published": [iso_utc(note.created_at)],
"url": [absolute_url_for("public.note", slug=note.slug)],
"post-status": ["published" if note.published else "draft"],
}
+42 -6
View File
@@ -31,12 +31,47 @@ class MigrationError(Exception):
pass
def create_migrations_table(conn):
def split_sql_statements(script: str) -> list[str]:
"""
Split a SQL script into complete statements (v2.8.1)
Uses sqlite3.complete_statement, so semicolons inside strings and
trigger bodies (BEGIN ... END) don't split a statement.
"""
statements = []
current = ""
for line in script.splitlines(keepends=True):
current += line
if sqlite3.complete_statement(current):
if current.strip().strip(";").strip():
statements.append(current.strip())
current = ""
if current.strip():
statements.append(current.strip())
return statements
def execute_in_transaction(conn, script: str) -> None:
"""
Run a SQL script inside the connection's open transaction
Not conn.executescript(): that COMMITs first, which released the
migration lock mid-migration and let another gunicorn worker apply the
same migration again ("duplicate column name", v2.8.0 deploy).
"""
for statement in split_sql_statements(script):
conn.execute(statement)
def create_migrations_table(conn, commit=True):
"""
Create schema_migrations tracking table if it doesn't exist
Args:
conn: SQLite connection
commit: Commit afterwards. run_migrations passes False: committing
there released its BEGIN IMMEDIATE lock straight away, so
gunicorn workers applied migrations concurrently (v2.8.1)
"""
conn.execute("""
CREATE TABLE IF NOT EXISTS schema_migrations (
@@ -51,7 +86,8 @@ def create_migrations_table(conn):
ON schema_migrations(migration_name)
""")
conn.commit()
if commit:
conn.commit()
def is_schema_current(conn):
@@ -287,7 +323,7 @@ def apply_migration(conn, migration_name, migration_path, logger=None):
# Execute migration in transaction
conn.execute("BEGIN")
conn.executescript(migration_sql)
execute_in_transaction(conn, migration_sql)
# Record migration as applied
conn.execute(
@@ -374,7 +410,7 @@ def run_migrations(db_path, logger=None):
try:
# Ensure migrations tracking table exists
create_migrations_table(conn)
create_migrations_table(conn, commit=False)
# Quick check: have migrations already been applied by another worker?
cursor = conn.execute("SELECT COUNT(*) FROM schema_migrations")
@@ -472,8 +508,8 @@ def run_migrations(db_path, logger=None):
logger.debug(f"Applying migration: {migration_name}")
# Execute migration (already in transaction)
conn.executescript(migration_sql)
# Execute migration inside the locked transaction
execute_in_transaction(conn, migration_sql)
# Record migration as applied
conn.execute(
+19 -5
View File
@@ -11,6 +11,7 @@ from dataclasses import dataclass, field, replace
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Optional
import re
import sqlite3
# Third-party imports
@@ -40,6 +41,9 @@ MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br", BareUrlExtension()]
# Constants - Content limits
MAX_TITLE_LENGTH = 200
# A Markdown ATX heading: up to 6 '#', whitespace, then text
TITLE_RE = re.compile(r"#{1,6}[ \t]+(\S.*)$")
EXCERPT_LENGTH = 200
@@ -47,17 +51,19 @@ def split_title(content: str) -> tuple[Optional[str], str]:
"""
Split a note's '# Heading' first line (its title, per Q22) off its content
The first line must be a Markdown heading: 1-6 '#' then a space and text.
'#indieweb ...' (a hashtag) or a bare '#' is not a title. This is the one
place that decides whether a note has a title (templates use has_title).
Returns:
(title, remaining content); title is None if the note has no heading
"""
stripped = content.lstrip()
if not stripped.startswith("#"):
return None, content
first_line, _, rest = stripped.partition("\n")
title = first_line.lstrip("#").strip()
if not title:
match = TITLE_RE.match(first_line)
if not match:
return None, content
return title, rest.strip()
return match.group(1).strip().rstrip("#").strip(), rest.strip()
@dataclass(frozen=True)
@@ -278,6 +284,14 @@ class Note:
return self.html
return markdown.markdown(body, extensions=MARKDOWN_EXTENSIONS)
@property
def has_title(self) -> bool:
"""True if the note starts with a '# Heading' title (an article)"""
try:
return split_title(self.content)[0] is not None
except (FileNotFoundError, OSError):
return False
@property
def title(self) -> str:
"""
+24 -1
View File
@@ -11,7 +11,7 @@ import hashlib
import re
import secrets
import shutil
from datetime import datetime
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
@@ -693,3 +693,26 @@ def absolute_url_for(endpoint: str, **values) -> str:
from flask import current_app, url_for
return current_app.config["SITE_URL"].rstrip("/") + url_for(endpoint, **values)
def iso_utc(value) -> str:
"""
Format a timestamp as RFC 3339 UTC for dt-* and Micropub (v2.8.0)
Stored times are naive UTC. Without an offset, microformats parsers read
them as local time, so this adds 'Z' and drops microseconds (HTML
<time datetime> allows at most 3 fractional digits).
Args:
value: datetime (naive = UTC) or ISO 8601 string
Returns:
e.g. '2026-11-01T16:25:12Z', or '' for empty values
"""
if not value:
return ''
if isinstance(value, str):
value = datetime.fromisoformat(value.strip().replace('Z', '+00:00'))
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
+65 -133
View File
@@ -19,10 +19,11 @@ import logging
import mf2py
import nh3
import httpx
from datetime import datetime
from typing import Dict, Any, Optional
from urllib.parse import urldefrag, urlparse
from starpunk import mf2_utils
from starpunk.utils import iso_utc
from starpunk.webmention_ssrf import (
FetchTooLargeError, SSRFBlockedError, safe_get, user_agent,
)
@@ -147,66 +148,13 @@ def verify_link_presence(content: str, target_url: str, content_type: str,
def detect_mention_type(parsed_mf2: Dict, target_url: str) -> str:
"""
Detect mention type from microformats.
Priority: reply > like > repost > bookmark > mention
Args:
parsed_mf2: Parsed microformats data from mf2py
target_url: Target URL being mentioned
Detect mention type from the h-entry that responds to the target
Returns:
Mention type: 'reply', 'like', 'repost', 'bookmark', or 'mention'
'reply', 'like', 'repost', 'bookmark', or 'mention'
"""
target_normalized = urlparse(target_url)._replace(fragment='').geturl().rstrip('/')
# Find h-entry that references target
for item in parsed_mf2.get('items', []):
if 'h-entry' not in item.get('type', []):
continue
props = item.get('properties', {})
# Check in-reply-to (highest priority)
for reply_to in props.get('in-reply-to', []):
url = reply_to if isinstance(reply_to, str) else reply_to.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'reply'
# Check like-of
for like in props.get('like-of', []):
url = like if isinstance(like, str) else like.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'like'
# Check repost-of
for repost in props.get('repost-of', []):
url = repost if isinstance(repost, str) else repost.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'repost'
# Check bookmark-of
for bookmark in props.get('bookmark-of', []):
url = bookmark if isinstance(bookmark, str) else bookmark.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'bookmark'
# Default fallback
return 'mention'
def _first_str(props: Dict, name: str) -> Optional[str]:
"""
First value of an mf2 property as a string
mf2py gives u-photo with alt text as {'value': url, 'alt': ...} and nested
microformats as dicts with a 'value'; reduce those to the plain value.
"""
values = props.get(name) or [None]
value = values[0]
if isinstance(value, dict):
value = value.get('value')
return value if isinstance(value, str) and value else None
entry = mf2_utils.responding_entry(parsed_mf2, target_url)
return mf2_utils.response_type(entry, target_url) if entry else 'mention'
def _is_http_url(value: Optional[str]) -> bool:
@@ -217,59 +165,27 @@ def _is_http_url(value: Optional[str]) -> bool:
return parsed.scheme.lower() in ('http', 'https') and bool(parsed.netloc)
def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]]:
def extract_author(parsed_mf2: Dict, source_url: str,
entry: Optional[Dict] = None) -> Dict[str, Optional[str]]:
"""
Extract author information from h-card.
Author of the source's h-entry (authorship algorithm, without fetching)
Args:
parsed_mf2: Parsed microformats data from mf2py
source_url: Source URL (fallback for author URL)
source_url: Source URL (fallback for author URL and name)
entry: The h-entry to attribute (default: first top-level h-entry)
Returns:
Dict with author_name, author_url, author_photo
Dict with author_name, author_url, author_photo; only http(s) URLs
are kept, so they are safe to render as links (XSS)
"""
author_name = None
author_url = None
author_photo = None
if entry is None:
entry = mf2_utils.responding_entry(parsed_mf2, '')
author = mf2_utils.entry_author(parsed_mf2, entry, source_url)
# Find h-entry first
entry = None
for item in parsed_mf2.get('items', []):
if 'h-entry' in item.get('type', []):
entry = item
break
# Check for author property in h-entry
if entry:
author_prop = entry.get('properties', {}).get('author', [])
if author_prop:
author_data = author_prop[0]
if isinstance(author_data, dict):
# h-card object
props = author_data.get('properties', {})
author_name = _first_str(props, 'name')
author_url = _first_str(props, 'url')
author_photo = _first_str(props, 'photo')
# If no author in h-entry, find representative h-card
if not author_name:
for item in parsed_mf2.get('items', []):
if 'h-card' in item.get('type', []):
props = item.get('properties', {})
author_name = _first_str(props, 'name')
author_url = _first_str(props, 'url')
author_photo = _first_str(props, 'photo')
break
# Fallback to hostname if no author found
if not author_name:
author_name = urlparse(source_url).netloc
# Only http(s) URLs are rendered as links (blocks javascript: and data:)
if not _is_http_url(author_url):
author_url = source_url
if not _is_http_url(author_photo):
author_photo = None
author_url = author['url'] if _is_http_url(author['url']) else source_url
author_photo = author['photo'] if _is_http_url(author['photo']) else None
author_name = author['name'] or urlparse(author_url).netloc
return {
'author_name': author_name,
@@ -278,45 +194,56 @@ def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]
}
def extract_content(parsed_mf2: Dict) -> Optional[str]:
def extract_content(parsed_mf2: Dict, entry: Optional[Dict] = None) -> Optional[str]:
"""
Extract content snippet from h-entry.
Content snippet of an h-entry (e-content, else p-summary)
Args:
parsed_mf2: Parsed microformats data from mf2py
entry: The h-entry (default: first top-level h-entry)
Returns:
Sanitized content snippet (max 500 chars) or None
"""
# Find h-entry
for item in parsed_mf2.get('items', []):
if 'h-entry' not in item.get('type', []):
continue
if entry is None:
entry = mf2_utils.responding_entry(parsed_mf2, '')
if not entry:
return None
props = item.get('properties', {})
props = entry.get('properties', {})
for content in props.get('content', []):
html = (content.get('html') or content.get('value')) if isinstance(content, dict) else content
if html:
return _snippet(html)
# Try e-content first (HTML)
content_prop = props.get('content', [])
if content_prop:
content_data = content_prop[0]
if isinstance(content_data, dict):
content = content_data.get('html', '') or content_data.get('value', '')
else:
content = content_data
summary = mf2_utils.first(entry, 'summary')
return _snippet(summary) if summary else None
if content:
return _snippet(content)
# Try p-summary as fallback
summary = props.get('summary', [])
if summary:
content = summary[0]
if isinstance(content, dict):
content = content.get('value', '')
if content:
return _snippet(content)
def extract_source_details(entry: Optional[Dict]) -> Dict[str, Optional[str]]:
"""
The source post's own URL, uid and published time
return None
Returns:
source_post_url, source_uid (http(s) only) and source_published
(RFC 3339 UTC, or None if missing or unparseable)
"""
if not entry:
return {'source_post_url': None, 'source_uid': None, 'source_published': None}
url = mf2_utils.first(entry, 'url')
uid = mf2_utils.first(entry, 'uid')
published = mf2_utils.first(entry, 'published')
try:
published = iso_utc(published) or None
except (ValueError, TypeError):
published = None
return {
'source_post_url': url if _is_http_url(url) else None,
'source_uid': uid,
'source_published': published,
}
def _snippet(html: str, limit: int = 500) -> str:
@@ -399,14 +326,19 @@ def process_webmention(source_url: str, target_url: str, site_url: str) -> Dict[
}
content_snippet = None
source_details = extract_source_details(None)
if parsed_mf2:
mention_type = detect_mention_type(parsed_mf2, target_url)
author_info = extract_author(parsed_mf2, source_url)
content_snippet = extract_content(parsed_mf2)
entry = mf2_utils.responding_entry(parsed_mf2, target_url)
mention_type = mf2_utils.response_type(entry, target_url) if entry else 'mention'
author_info = extract_author(parsed_mf2, final_url, entry)
content_snippet = extract_content(parsed_mf2, entry)
source_details = extract_source_details(entry)
return {
'valid': True,
'mention_type': mention_type,
**author_info,
'content_snippet': content_snippet
'content_snippet': content_snippet,
**source_details,
}
+46 -3
View File
@@ -17,7 +17,10 @@ Functions:
import logging
import math
from datetime import datetime
from pathlib import Path
from typing import List, Dict, Any, Optional
from flask import current_app
from starpunk.database import get_db
logger = logging.getLogger(__name__)
@@ -71,9 +74,13 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
for row in rows:
wm = dict(row)
# Parse received_at timestamp
if isinstance(wm.get('received_at'), str):
wm['received_at'] = datetime.fromisoformat(wm['received_at'].replace('Z', '+00:00'))
# Parse timestamps (source_published may be missing or malformed)
for key in ('received_at', 'source_published'):
if isinstance(wm.get(key), str):
try:
wm[key] = datetime.fromisoformat(wm[key].replace('Z', '+00:00'))
except ValueError:
wm[key] = None
mention_type = wm.get('mention_type', 'mention')
if mention_type == 'reply':
@@ -233,10 +240,19 @@ def update_webmention_status(webmention_ids: List[int], status: str) -> int:
WHERE id IN ({placeholders})
"""
# Notes whose displayed mentions change (valid ones switching between
# approved and not)
changed_notes = _visibility_changes(
db, webmention_ids,
"validation_status = 'valid' AND (moderation_status = 'approved') != (? = 'approved')",
[status],
)
params = [status] + webmention_ids
cursor = db.execute(query, params)
db.commit()
_notify_responses_changed(changed_notes)
return cursor.rowcount
@@ -255,15 +271,42 @@ def delete_webmentions(webmention_ids: List[int]) -> int:
db = get_db()
changed_notes = _visibility_changes(
db, webmention_ids, "validation_status = 'valid' AND moderation_status = 'approved'", [],
)
placeholders = ','.join('?' * len(webmention_ids))
query = f"DELETE FROM webmentions WHERE id IN ({placeholders})"
cursor = db.execute(query, webmention_ids)
db.commit()
_notify_responses_changed(changed_notes)
return cursor.rowcount
def _visibility_changes(db, webmention_ids: List[int], condition: str, params: list) -> List[int]:
"""Target note IDs of the given webmentions that match condition"""
placeholders = ','.join('?' * len(webmention_ids))
rows = db.execute(
f"SELECT DISTINCT target_note_id FROM webmentions "
f"WHERE id IN ({placeholders}) AND {condition}",
list(webmention_ids) + params,
).fetchall()
return [row[0] for row in rows]
def _notify_responses_changed(note_ids: List[int]) -> None:
"""Re-send webmentions for notes whose shown responses changed (best effort)"""
if not note_ids:
return
try:
from starpunk.webmention_sender import note_responses_changed
note_responses_changed(Path(current_app.config['DATABASE_PATH']), note_ids)
except Exception as e:
current_app.logger.warning(f"Failed to re-send webmentions after moderation: {e}")
def get_reply_context(url: str) -> Optional[Dict[str, Any]]:
"""
Get cached reply context for a URL.
+31
View File
@@ -129,6 +129,37 @@ def requeue_previous_webmentions(db_path: Path, note_id: int, detach: bool = Fal
return queued
def note_responses_changed(db_path: Path, note_ids) -> int:
"""
Re-send a note's webmentions because the responses it shows changed
Spec 3.1.4: if the source post displays responses (e.g. comments), a
change to them is an update of the post. Only published, undeleted notes.
Args:
db_path: Path to SQLite database
note_ids: IDs of notes whose displayed responses changed
Returns:
Number of webmentions queued
"""
ids = sorted({i for i in note_ids if i is not None})
if not ids:
return 0
conn = sqlite3.connect(db_path)
try:
placeholders = ','.join('?' * len(ids))
live = [row[0] for row in conn.execute(
f"SELECT id FROM notes WHERE id IN ({placeholders}) "
"AND published = 1 AND deleted_at IS NULL", ids,
)]
finally:
conn.close()
return sum(requeue_previous_webmentions(db_path, note_id) for note_id in live)
def queue_webmentions_for_note(
db_path: Path,
note_id: int,
+52 -6
View File
@@ -4,6 +4,8 @@ SSRF protection for webmention fetches
Per ADR-063 Decision 9 and Webmention sections 4.2, 4.3 and 4.5:
- Every address a hostname resolves to must be public
- Redirects are followed by hand so each hop is checked
- Each request connects to the address that was checked (no second DNS
lookup that a rebinding DNS server could answer differently)
- Redirect count, response size and total time are limited
- DEV_MODE exception for localhost testing
@@ -89,7 +91,7 @@ def safe_resolve(hostname: str) -> str:
Per background-worker.md: DEV_MODE exception for localhost testing.
Returns:
First resolved IP address
Address to connect to (IPv4 preferred)
Raises:
SSRFBlockedError: If any address is blocked
@@ -98,7 +100,8 @@ def safe_resolve(hostname: str) -> str:
if current_app.config.get('DEV_MODE') and hostname in ('localhost', '127.0.0.1', '::1'):
return '127.0.0.1'
addresses = sorted({info[4][0] for info in socket.getaddrinfo(hostname, None)})
addresses = sorted({info[4][0] for info in socket.getaddrinfo(hostname, None)},
key=lambda ip: (':' in ip, ip))
for ip in addresses:
if is_private_ip(ip):
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
@@ -106,10 +109,13 @@ def safe_resolve(hostname: str) -> str:
return addresses[0]
def validate_url_for_fetch(url: str) -> None:
def validate_url_for_fetch(url: str) -> str:
"""
Validate URL is safe to fetch (http(s), public addresses only).
Returns:
The checked IP address to connect to
Raises:
SSRFBlockedError: If URL points to a blocked address
ValueError: If URL is malformed or its host does not resolve
@@ -123,11 +129,48 @@ def validate_url_for_fetch(url: str) -> None:
raise ValueError("URL must include a hostname")
try:
safe_resolve(parsed.hostname)
return safe_resolve(parsed.hostname)
except socket.gaierror as e:
raise ValueError(f"Could not resolve {parsed.hostname}: {e}")
def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
**kwargs) -> httpx.Request:
"""
Build a request that connects to the checked address for url's host.
The URL's host is replaced by the IP that passed the SSRF check; the
real host goes in the Host header and, for https, in SNI, so virtual
hosting and certificate verification still use the hostname.
"""
ip = validate_url_for_fetch(url)
parsed = urlparse(url)
ip_host = f"[{ip}]" if ':' in ip else ip
port = f":{parsed.port}" if parsed.port else ''
pinned_url = parsed._replace(netloc=f"{ip_host}{port}").geturl()
host_header = f"{parsed.hostname}{port}"
extensions = {'sni_hostname': parsed.hostname} if parsed.scheme == 'https' else {}
return client.build_request(
method, pinned_url, headers={**headers, 'Host': host_header},
extensions=extensions, **kwargs,
)
def safe_post(url: str, data: dict, headers: dict, timeout: float = TIMEOUT) -> httpx.Response:
"""
POST form data to a checked, pinned address. Redirects are not followed.
Raises:
SSRFBlockedError, ValueError: URL is not allowed
httpx.HTTPError: Network errors
"""
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
response = client.send(_pinned_request(client, 'POST', url, headers, data=data))
response.read()
return response
@dataclass
class FetchResult:
"""A fetched response, after redirects"""
@@ -169,12 +212,13 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client:
for _ in range(max_redirects + 1):
validate_url_for_fetch(url)
remaining = deadline - time.monotonic()
if remaining <= 0:
raise httpx.TimeoutException("Fetch exceeded time limit")
with client.stream('GET', url, headers=headers, timeout=remaining) as response:
request = _pinned_request(client, 'GET', url, headers, timeout=remaining)
response = client.send(request, stream=True)
try:
location = response.headers.get('Location')
if response.is_redirect and location:
url = urljoin(url, location)
@@ -199,5 +243,7 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
content=content,
encoding=response.charset_encoding or 'utf-8',
)
finally:
response.close()
raise httpx.TooManyRedirects(f"More than {max_redirects} redirects")
+79 -9
View File
@@ -26,8 +26,10 @@ from urllib.parse import urljoin
from lxml import html as lxml_html
from starpunk.webmention_processor import process_webmention, TransientError, ValidationError
from starpunk.webmention_sender import note_responses_changed
from starpunk.mf2_utils import normalize_url
from starpunk.webmention_ssrf import (
FetchTooLargeError, SSRFBlockedError, safe_get, user_agent, validate_url_for_fetch,
FetchTooLargeError, SSRFBlockedError, safe_get, safe_post, user_agent,
)
@@ -244,6 +246,69 @@ class WebmentionWorker(threading.Thread):
source_url: str,
target_url: str,
target_note_id: Optional[int]
):
"""
Validate a webmention, then re-send the target note's webmentions if
what the note shows changed (spec 3.1.4: displayed responses are an
update of the post). Unchanged re-validations send nothing, so two
sites doing this don't ping-pong.
"""
before = self._display_state(webmention_id)
try:
self._validate_and_store(webmention_id, source_url, target_url, target_note_id)
finally:
after = self._display_state(webmention_id)
if before != after and target_note_id is not None:
try:
note_responses_changed(self.db_path, [target_note_id])
except Exception as e:
logger.warning(f"Failed to re-send webmentions for note {target_note_id}: {e}")
def _is_own_syndicated_copy(self, note_id: Optional[int], source_url: str,
result: Dict[str, Any]) -> bool:
"""True if the source is one of the note's own u-syndication copies"""
if note_id is None:
return False
conn = sqlite3.connect(self.db_path)
try:
copies = {normalize_url(row[0]) for row in conn.execute(
"SELECT remote_url FROM syndications WHERE note_id = ? AND remote_url IS NOT NULL",
(note_id,),
)}
finally:
conn.close()
source = {normalize_url(u) for u in (
source_url, result.get('source_post_url'), result.get('source_uid')) if u}
return bool(copies & source)
def _display_state(self, webmention_id: int) -> Optional[tuple]:
"""
What the note page shows for this webmention, or None if hidden
A row reset to 'pending' by a re-send still counts as shown if it was
valid before (validated with no error), as it was until now.
"""
conn = sqlite3.connect(self.db_path)
try:
row = conn.execute("""
SELECT mention_type, author_name, author_url, author_photo, content_snippet
FROM webmentions
WHERE id = ?
AND moderation_status = 'approved'
AND (validation_status = 'valid'
OR (validation_status = 'pending' AND validated_at IS NOT NULL
AND validation_error IS NULL))
""", (webmention_id,)).fetchone()
finally:
conn.close()
return tuple(row) if row else None
def _validate_and_store(
self,
webmention_id: int,
source_url: str,
target_url: str,
target_note_id: Optional[int]
):
"""
Validate a single webmention
@@ -258,6 +323,11 @@ class WebmentionWorker(threading.Thread):
# Process webmention (fetch, validate, extract)
result = process_webmention(source_url, target_url, self.site_url)
# A copy of this very note on a silo (e.g. backfed by Bridgy) is
# not a response to it
if self._is_own_syndicated_copy(target_note_id, source_url, result):
raise ValidationError("Source is this note's own syndicated copy")
# Update database with successful validation
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
@@ -283,6 +353,8 @@ class WebmentionWorker(threading.Thread):
author_url = ?,
author_photo = ?,
content_snippet = ?,
source_post_url = ?,
source_published = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (
@@ -292,6 +364,8 @@ class WebmentionWorker(threading.Thread):
result['author_url'],
result['author_photo'],
result['content_snippet'],
result.get('source_post_url'),
result.get('source_published'),
webmention_id
))
@@ -348,7 +422,7 @@ class WebmentionWorker(threading.Thread):
# All other errors are 'error' status
if transient:
status = 'error'
elif any(x in error.lower() for x in ['does not link', 'returned 4']):
elif any(x in error.lower() for x in ['does not link', 'returned 4', 'own syndicated copy']):
status = 'invalid'
else:
status = 'error'
@@ -495,20 +569,16 @@ class WebmentionWorker(threading.Thread):
# Send webmention POST (spec 3.1.3)
try:
# SSRF protection for endpoint URL (spec 4.3)
validate_url_for_fetch(endpoint)
# Redirects are not followed: httpx would turn a 301/302 POST
# SSRF-checked and pinned to the checked address (spec 4.3).
# Redirects are not followed: a 301/302 would turn the POST
# into a GET without the body
response = httpx.post(
response = safe_post(
endpoint,
data={
'source': source_url,
'target': target_url
},
timeout=10,
headers={'User-Agent': user_agent(self.site_url)},
follow_redirects=False
)
# Any 2xx is success (spec 3.1.3)
+16
View File
@@ -332,6 +332,9 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
}
.webmention-author a {
display: inline-flex;
align-items: center;
gap: var(--spacing-sm);
font-weight: bold;
}
@@ -361,6 +364,13 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
margin-left: 0;
}
.facepile .face > .h-card,
.facepile .face > .h-card > a {
display: block;
width: 100%;
height: 100%;
}
.facepile .face img {
width: 100%;
height: 100%;
@@ -392,6 +402,12 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
border-radius: 4px;
}
.webmention-mention .h-card img {
border-radius: 50%;
vertical-align: middle;
margin-right: var(--spacing-xs);
}
.webmention-mention p {
margin: var(--spacing-xs) 0 0 0;
font-size: 0.875rem;
+11 -1
View File
@@ -12,15 +12,25 @@
{# Webmention endpoint (v1.6.0) #}
{% if config.WEBMENTION_ENABLED %}
<link rel="webmention" href="/webmention">
{% endif %}
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
<link rel="authorization_endpoint" href="{{ absolute_url_for('indieauth.authorize') }}">
<link rel="token_endpoint" href="{{ absolute_url_for('indieauth.token') }}">
{# Author's home page: rel=me makes the h-card linking to it this site's
representative h-card; rel=author is the authorship fallback (v2.8.0) #}
{% set author_home = (author.url or author.me) if author else config.ADMIN_ME %}
{% if author_home %}
<link rel="me" href="{{ author_home }}">
<link rel="author" href="{{ author_home }}">
{% endif %}
{# rel-me links from discovered author profile (v1.2.0 Phase 2) #}
{% if author and author.rel_me_links %}
{% for profile_url in author.rel_me_links %}
{% for profile_url in author.rel_me_links if profile_url != author_home %}
<link rel="me" href="{{ profile_url }}">
{% endfor %}
{% endif %}
+6
View File
@@ -7,6 +7,12 @@
<div class="h-feed">
<h1 class="p-name page-title">Likes</h1>
{# Feed URL and author, as on the home page (v2.8.0) #}
<a class="u-url" href="{{ absolute_url_for(request.endpoint, **request.view_args) }}" hidden>Feed URL</a>
{% if author %}
<span class="p-author h-card" hidden><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
{% endif %}
{% if notes %}
{% for note in notes %}
{{ note_preview(note, author) }}
+11 -81
View File
@@ -1,13 +1,14 @@
{% extends "base.html" %}
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% import "partials/responses.html" as responses with context %}
{% block title %}{% if note.content.strip().startswith('#') %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block title %}{% if note.has_title %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<article class="h-entry">
{# Detect if note has explicit title (starts with # heading) - per Q22 #}
{% set has_explicit_title = note.content.strip().startswith('#') %}
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
{% set has_explicit_title = note.has_title %}
{# p-name only if note has explicit title (per Q22) #}
{% if has_explicit_title %}
@@ -53,9 +54,9 @@
<footer class="note-meta">
{# u-url and u-uid same for notes (per Q23) #}
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at.isoformat() }}">{{ note.created_at.strftime('%B %-d, %Y · %-I:%M %p') }}</time></a>
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at|iso_utc }}">{{ note.created_at.strftime('%B %-d, %Y · %-I:%M %p') }}</time></a>
{% if note.updated_at and note.updated_at != note.created_at %}
<span class="updated">Updated <time class="dt-updated" datetime="{{ note.updated_at.isoformat() }}">{{ note.updated_at.strftime('%b %-d, %Y') }}</time></span>
<span class="updated">Updated <time class="dt-updated" datetime="{{ note.updated_at|iso_utc }}">{{ note.updated_at.strftime('%b %-d, %Y') }}</time></span>
{% endif %}
{# rel="tag" per microformats2 p-category #}
{% if note.tags %}
@@ -89,106 +90,35 @@
{% if replies %}
<div class="webmention-group webmention-replies">
<h3>{{ replies|length }} {{ 'Reply' if replies|length == 1 else 'Replies' }}</h3>
{% for reply in replies %}
<article class="webmention webmention-reply h-cite">
<div class="webmention-author">
{% if reply.author_photo %}
<img src="{{ reply.author_photo }}" alt="" width="48" height="48" loading="lazy">
{% endif %}
<a class="p-author" href="{{ reply.author_url or reply.source_url }}">
{{ reply.author_name or 'Anonymous' }}
</a>
</div>
{% if reply.content_snippet %}
<div class="p-content">{{ reply.content_snippet|safe }}</div>
{% endif %}
<a class="u-url webmention-date" href="{{ reply.source_url }}">
<time class="dt-published" datetime="{{ reply.received_at.isoformat() }}">
{{ reply.received_at.strftime('%B %d, %Y') }}
</time>
</a>
</article>
{% endfor %}
{% for wm in replies %}{{ responses.reply(wm) }}{% endfor %}
</div>
{% endif %}
{% if likes %}
<div class="webmention-group webmention-likes">
<h3>{{ likes|length }} {{ 'Like' if likes|length == 1 else 'Likes' }}</h3>
<div class="facepile">
{% for wm in likes[:8] %}
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
{% if wm.author_photo %}
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
{% else %}
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
{{ (wm.author_name or '?')[0]|upper }}
</span>
{% endif %}
</a>
{% endfor %}
{% if likes|length > 8 %}
<span class="face-overflow">+{{ likes|length - 8 }} others</span>
{% endif %}
</div>
{{ responses.facepile(likes, 'u-like') }}
</div>
{% endif %}
{% if reposts %}
<div class="webmention-group webmention-reposts">
<h3>{{ reposts|length }} {{ 'Repost' if reposts|length == 1 else 'Reposts' }}</h3>
<div class="facepile">
{% for wm in reposts[:8] %}
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
{% if wm.author_photo %}
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
{% else %}
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
{{ (wm.author_name or '?')[0]|upper }}
</span>
{% endif %}
</a>
{% endfor %}
{% if reposts|length > 8 %}
<span class="face-overflow">+{{ reposts|length - 8 }} others</span>
{% endif %}
</div>
{{ responses.facepile(reposts, 'u-repost') }}
</div>
{% endif %}
{% if bookmarks %}
<div class="webmention-group webmention-bookmarks">
<h3>{{ bookmarks|length }} {{ 'Bookmark' if bookmarks|length == 1 else 'Bookmarks' }}</h3>
<div class="facepile">
{% for wm in bookmarks[:8] %}
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
{% if wm.author_photo %}
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
{% else %}
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
{{ (wm.author_name or '?')[0]|upper }}
</span>
{% endif %}
</a>
{% endfor %}
{% if bookmarks|length > 8 %}
<span class="face-overflow">+{{ bookmarks|length - 8 }} others</span>
{% endif %}
</div>
{{ responses.facepile(bookmarks, 'u-bookmark') }}
</div>
{% endif %}
{% if mentions %}
<div class="webmention-group webmention-mentions">
<h3>{{ mentions|length }} {{ 'Mention' if mentions|length == 1 else 'Mentions' }}</h3>
{% for mention in mentions %}
<div class="webmention webmention-mention">
<a href="{{ mention.source_url }}">{{ mention.author_name or mention.source_url }}</a>
{% if mention.content_snippet %}
<p>{{ mention.content_snippet|striptags|truncate(200) }}</p>
{% endif %}
</div>
{% endfor %}
{% for wm in mentions %}{{ responses.mention(wm) }}{% endfor %}
</div>
{% endif %}
</section>
+3 -3
View File
@@ -3,8 +3,8 @@
{% from "partials/reaction.html" import reaction %}
{% macro note_preview(note, author) %}
<article class="h-entry note-preview">
{# Notes starting with a # heading have a title (Q22) #}
{% set has_explicit_title = note.content.strip().startswith('#') %}
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
{% set has_explicit_title = note.has_title %}
{% if has_explicit_title %}
<h2 class="p-name"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">{{ note.title }}</a></h2>
{% endif %}
@@ -31,7 +31,7 @@
{% endif %}
<footer class="note-meta">
<a class="u-url" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at.isoformat() }}">{{ note.created_at.strftime('%b %-d, %Y') }}</time></a>
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at|iso_utc }}">{{ note.created_at.strftime('%b %-d, %Y') }}</time></a>
{% if note.tags %}
<span class="note-tags">
{% for tag in note.tags %}<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>{% endfor %}
+62
View File
@@ -0,0 +1,62 @@
{# Received webmentions as microformats2 responses (v2.8.0)
Each response is an h-cite in a property of the note's h-entry:
u-comment (replies), u-like, u-repost, u-bookmark, u-mention.
See https://indieweb.org/comments-presentation and microformats.org/wiki/h-entry #}
{# When the source said when it was posted, use that; else when we received it #}
{% macro response_time(wm, fmt='%B %-d, %Y') %}
{% set when = wm.source_published or wm.received_at %}
<time class="dt-published" datetime="{{ when|iso_utc }}">{{ when.strftime(fmt) if when.strftime is defined else when }}</time>
{%- endmacro %}
{# The response's own URL: the source's u-url if it gave one #}
{% macro response_url(wm) %}{{ wm.source_post_url or wm.source_url }}{% endmacro %}
{# Author h-card: name, URL and photo. u-url is on an inner link: on the
h-card element itself it would belong to the enclosing h-cite #}
{% macro author_card(wm, class='', photo_size=48, show_name=True) %}
<span class="p-author h-card {{ class }}"><a class="u-url" href="{{ wm.author_url or wm.source_url }}" title="{{ wm.author_name or 'Anonymous' }}">
{%- if wm.author_photo %}<img class="u-photo" src="{{ wm.author_photo }}" alt="" width="{{ photo_size }}" height="{{ photo_size }}" loading="lazy">
{%- elif not show_name %}<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}" aria-hidden="true">{{ (wm.author_name or '?')[0]|upper }}</span>
{%- endif %}
<span class="p-name{{ '' if show_name else ' visually-hidden' }}">{{ wm.author_name or 'Anonymous' }}</span>
</a></span>
{%- endmacro %}
{# A reply, shown as a comment #}
{% macro reply(wm) %}
<article class="webmention webmention-reply u-comment h-cite">
<div class="webmention-author">{{ author_card(wm) }}</div>
{% if wm.content_snippet %}
<div class="e-content">{{ wm.content_snippet|safe }}</div>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
</article>
{%- endmacro %}
{# Likes, reposts or bookmarks as a facepile; prop is u-like / u-repost / u-bookmark.
Faces past `shown` are hidden on screen but kept for parsers. #}
{% macro facepile(items, prop, shown=8) %}
<div class="facepile">
{% for wm in items %}
<span class="face {{ prop }} h-cite"{% if loop.index > shown %} hidden{% endif %}>
{{ author_card(wm, photo_size=32, show_name=False) }}
<a class="u-url" href="{{ response_url(wm) }}" hidden>{{ response_time(wm) }}</a>
</span>
{% endfor %}
{% if items|length > shown %}
<span class="face-overflow">+{{ items|length - shown }} others</span>
{% endif %}
</div>
{%- endmacro %}
{# A post that mentions this note without replying #}
{% macro mention(wm) %}
<div class="webmention webmention-mention u-mention h-cite">
{{ author_card(wm, photo_size=24) }}
{% if wm.content_snippet %}
<p class="p-content">{{ wm.content_snippet|striptags|truncate(200) }}</p>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
</div>
{%- endmacro %}
+6
View File
@@ -7,6 +7,12 @@
<div class="h-feed">
<h1 class="p-name page-title">Notes tagged <span class="tag-name">{{ tag.display_name }}</span></h1>
{# Feed URL and author, as on the home page (v2.8.0) #}
<a class="u-url" href="{{ absolute_url_for(request.endpoint, **request.view_args) }}" hidden>Feed URL</a>
{% if author %}
<span class="p-author h-card" hidden><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
{% endif %}
{% if notes %}
{% for note in notes %}
{{ note_preview(note, author) }}
+9 -14
View File
@@ -110,7 +110,7 @@ class TestImageValidation:
assert mime_type == 'image/jpeg'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_valid_png(self):
"""Test validation of valid PNG image"""
@@ -120,7 +120,7 @@ class TestImageValidation:
assert mime_type == 'image/png'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_valid_gif(self):
"""Test validation of valid GIF image"""
@@ -130,7 +130,7 @@ class TestImageValidation:
assert mime_type == 'image/gif'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_valid_webp(self):
"""Test validation of valid WebP image"""
@@ -140,7 +140,7 @@ class TestImageValidation:
assert mime_type == 'image/webp'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_file_too_large(self):
"""Test rejection of >10MB file (per Q6)"""
@@ -183,11 +183,8 @@ class TestHEICSupport:
assert mime_type == 'image/jpeg'
assert width == 800
assert height == 600
# file_data should be different (converted to JPEG)
assert file_data != heic_data
# Verify it's actually JPEG by opening it
img = Image.open(io.BytesIO(file_data))
assert img.format == 'JPEG'
# Converted to JPEG (validate_image returns a PIL Image)
assert file_data.format == 'JPEG'
def test_heic_with_rgba_mode(self):
"""Test HEIC with alpha channel is converted to RGB JPEG"""
@@ -209,8 +206,7 @@ class TestHEICSupport:
# Should be converted to JPEG (no alpha)
assert mime_type == 'image/jpeg'
# Verify it's RGB (no alpha)
img = Image.open(io.BytesIO(file_data))
assert img.mode == 'RGB'
assert file_data.mode == 'RGB'
def test_heic_dimensions_preserved(self):
"""Test HEIC conversion preserves dimensions"""
@@ -273,9 +269,8 @@ class TestMPOSupport:
assert mime_type == 'image/jpeg'
assert width == 800
assert height == 600
# Verify it's actually JPEG by opening it
img = Image.open(io.BytesIO(file_data))
assert img.format == 'JPEG'
# Converted to JPEG (validate_image returns a PIL Image)
assert file_data.format == 'JPEG'
# Note: MPO with single frame may have identical bytes to JPEG
# The important part is the format detection and MIME type correction
+8 -4
View File
@@ -289,16 +289,20 @@ class TestRelMe:
assert 'https://github.com/testuser' in rels['me']
assert 'https://mastodon.social/@testuser' in rels['me']
def test_no_relme_without_author(self, client, app):
"""No rel=me links if no author profile"""
def test_relme_to_admin_me_without_author(self, client, app):
"""Without a discovered profile, rel=me and rel=author still point at ADMIN_ME (v2.8.0)
The home page's p-author h-card links to the author's site; rel=me to
the same URL makes it the representative h-card.
"""
with patch('starpunk.author_discovery.get_author_profile', return_value=None):
response = client.get('/')
parsed = mf2py.parse(doc=response.data.decode(), url='http://localhost/')
rels = parsed.get('rels', {})
# Should not have rel=me, or it should be empty
assert len(rels.get('me', [])) == 0, "Should not have rel=me without author"
assert rels.get('me') == [app.config['ADMIN_ME']]
assert rels.get('author') == [app.config['ADMIN_ME']]
class TestV130Microformats:
+268
View File
@@ -0,0 +1,268 @@
"""
Microformats2 fixes (v2.8.0)
Pages are rendered and parsed with mf2py; assertions are on the parsed data,
which is what IndieWeb readers and Webmention receivers see.
"""
import re
import sqlite3
from unittest.mock import patch
import mf2py
import pytest
from starpunk import mf2_utils
from starpunk.models import split_title
from starpunk.webmention_processor import extract_author, extract_source_details
from starpunk.webmention_worker import WebmentionWorker
BASE = 'http://localhost:5000'
def parse(client, path):
response = client.get(path)
assert response.status_code == 200
return mf2py.parse(doc=response.data.decode(), url=BASE + path)
def entry_of(parsed):
return next(i for i in parsed['items'] if 'h-entry' in i['type'])
def make_note(app, content, **kw):
from starpunk.notes import create_note
with app.test_request_context():
return create_note(content, published=True, **kw)
def add_mention(app, note, kind, n, photo=True, **cols):
db = sqlite3.connect(app.config['DATABASE_PATH'])
fields = {
'source_url': f'https://src.example/{kind}/{n}',
'target_url': f'{BASE}/note/{note.slug}',
'target_note_id': note.id,
'mention_type': kind,
'author_name': f'{kind.title()} Person {n}',
'author_url': f'https://{kind}{n}.example/',
'author_photo': f'https://{kind}{n}.example/me.jpg' if photo else None,
'content_snippet': f'<p>{kind} text <a href="https://x.example/">link</a></p>',
'validation_status': 'valid',
'moderation_status': 'approved',
**cols,
}
db.execute(f"INSERT INTO webmentions ({', '.join(fields)}) VALUES ({', '.join('?' * len(fields))})",
tuple(fields.values()))
db.commit()
db.close()
# A1/A2: received responses are properties of the entry
def test_responses_are_entry_properties(app, client):
note = make_note(app, "A note")
add_mention(app, note, 'reply', 1, source_post_url='https://src.example/reply/1#canonical',
source_published='2026-11-01T10:00:00Z')
for n in range(10):
add_mention(app, note, 'like', n, photo=n % 2 == 0)
add_mention(app, note, 'repost', 1)
add_mention(app, note, 'bookmark', 1)
add_mention(app, note, 'mention', 1)
props = entry_of(parse(client, f'/note/{note.slug}'))['properties']
assert 'children' not in entry_of(parse(client, f'/note/{note.slug}'))
(comment,) = props['comment']
assert comment['type'] == ['h-cite']
author = comment['properties']['author'][0]
assert author['type'] == ['h-card']
assert author['properties']['name'] == ['Reply Person 1']
assert author['properties']['url'] == ['https://reply1.example/']
assert author['properties']['photo'][0] in ('https://reply1.example/me.jpg',
{'value': 'https://reply1.example/me.jpg', 'alt': ''})
assert '<a href="https://x.example/"' in comment['properties']['content'][0]['html']
# the source's own URL and published time, not ours
assert comment['properties']['url'] == ['https://src.example/reply/1#canonical']
assert comment['properties']['published'] == ['2026-11-01T10:00:00Z']
# all ten likes, including the ones past the visible eight
assert len(props['like']) == 10
names = {like['properties']['author'][0]['properties']['name'][0] for like in props['like']}
assert names == {f'Like Person {n}' for n in range(10)}
assert props['like'][0]['properties']['url'] == ['https://src.example/like/0']
assert props['repost'][0]['properties']['author'][0]['properties']['name'] == ['Repost Person 1']
assert props['bookmark'][0]['type'] == ['h-cite']
assert props['mention'][0]['properties']['author'][0]['properties']['url'] == ['https://mention1.example/']
# A3: dt-* are RFC 3339 UTC
UTC = re.compile(r'^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$')
def test_dates_are_utc(app, client):
from starpunk.notes import update_note
note = make_note(app, "Dated")
with app.test_request_context():
update_note(id=note.id, content="Dated, edited")
add_mention(app, note, 'reply', 1)
props = entry_of(parse(client, f'/note/{note.slug}'))['properties']
assert UTC.match(props['published'][0]), props['published']
assert UTC.match(props['updated'][0]), props['updated']
assert UTC.match(props['comment'][0]['properties']['published'][0])
home = parse(client, '/')['items'][0]['children'][0]['properties']
assert UTC.match(home['published'][0])
def test_micropub_source_published_is_utc():
from datetime import datetime, timezone, timedelta
from starpunk.utils import iso_utc
assert iso_utc(datetime(2026, 11, 1, 16, 25, 12, 921768)) == '2026-11-01T16:25:12Z'
assert iso_utc(datetime(2026, 11, 1, 9, 0, tzinfo=timezone(timedelta(hours=-7)))) == '2026-11-01T16:00:00Z'
assert iso_utc('2026-11-01T16:25:15+0000') == '2026-11-01T16:25:15Z'
# A4: only a real Markdown heading is a title
@pytest.mark.parametrize("content,title", [
("# Title\n\nBody", "Title"),
("## Sub title ##\n\nBody", "Sub title"),
("#indieweb is great", None),
("#\n\nbody text", None),
("####### seven", None),
("Plain note", None),
])
def test_split_title(content, title):
assert split_title(content)[0] == title
@pytest.mark.parametrize("content,kind", [
("#indieweb is great, posting from my phone", "note"),
("#\n\nbody text", "note"),
("# A Title\n\nThe body", "article"),
])
def test_post_type_of_hash_notes(app, client, content, kind):
note = make_note(app, content)
for path in (f'/note/{note.slug}', '/'):
parsed = parse(client, path)
entry = entry_of(parsed) if path != '/' else parsed['items'][0]['children'][0]
props = entry['properties']
if kind == 'note':
assert 'name' not in props
assert props['content'][0]['value'].strip()
else:
assert props['name'] == ['A Title']
# A5: the note's own syndicated copy is not a response
def test_own_syndicated_copy_rejected(app):
note = make_note(app, "Crossposted")
db = sqlite3.connect(app.config['DATABASE_PATH'])
db.execute("INSERT INTO syndications (note_id, target, remote_url) VALUES (?, 'gotosocial', ?)",
(note.id, 'https://social.example/@me/statuses/1'))
db.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) VALUES (?, ?, ?)",
('https://brid.gy/post/mastodon/@someone/1', f'{BASE}/note/{note.slug}', note.id))
db.commit()
result = {'valid': True, 'mention_type': 'mention', 'author_name': 'Me', 'author_url': 'https://me.example/',
'author_photo': None, 'content_snippet': 'x',
'source_post_url': 'https://social.example/@me/statuses/1', 'source_uid': None,
'source_published': None}
worker = WebmentionWorker(app, db_path=app.config['DATABASE_PATH'], site_url=BASE)
with patch("starpunk.webmention_worker.process_webmention", return_value=result):
worker._validate_webmention(1, 'https://brid.gy/post/mastodon/@someone/1', 'x', note.id)
status, error = db.execute("SELECT validation_status, validation_error FROM webmentions").fetchone()
assert status == 'invalid' and 'own syndicated copy' in error
# B1/B3: discovery links
def test_head_links_without_webmention(app, client):
app.config['WEBMENTION_ENABLED'] = False
rels = parse(client, '/')['rels']
assert 'webmention' not in rels
for rel in ('micropub', 'indieauth-metadata', 'authorization_endpoint', 'token_endpoint'):
assert rels.get(rel), rel
def test_home_has_representative_hcard(app, client):
# Fresh install: discovery fails, the fallback profile is used
parsed = parse(client, '/')
card = mf2_utils.representative_hcard(parsed, BASE + '/')
assert card and card['properties']['url'] == [app.config['ADMIN_ME']]
@pytest.mark.parametrize("path", ['/likes', '/tag/python'])
def test_list_feeds_have_url_and_author(app, client, path):
make_note(app, "Tagged", tags=["python"])
feed = parse(client, path)['items'][0]
assert feed['type'] == ['h-feed']
assert feed['properties']['url'] == [BASE + path]
assert feed['properties']['author'][0]['type'] == ['h-card']
# B2: representative h-card
def _cards(html, url='https://me.example/'):
return mf2py.parse(doc=html, url=url)
def test_representative_hcard_rules():
other = '<div class="h-card"><a class="p-name u-url" href="https://bob.example/">Bob</a></div>'
# someone else's h-card only: none
assert mf2_utils.representative_hcard(_cards(other), 'https://me.example/') is None
# uid + url
me = '<div class="h-card"><a class="p-name u-url u-uid" href="/">Me</a></div>'
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(other + me), 'https://me.example/'), 'name') == 'Me'
# rel=me
relme = ('<a rel="me" href="https://social.example/@me">x</a>'
'<div class="h-card"><a class="p-name u-url" href="https://social.example/@me">Me</a></div>')
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(other + relme), 'https://me.example/'), 'name') == 'Me'
# only one h-card, url = page
single = '<div class="h-card"><a class="p-name u-url" href="/">Me</a></div>'
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(single), 'https://me.example/'), 'name') == 'Me'
# B4: the responding entry and its author
FEED_PAGE = '''
<div class="h-feed">
<div class="p-author h-card"><a class="p-name u-url" href="https://feed.example/">Feed Owner</a></div>
<div class="h-entry"><a class="u-url" href="https://feed.example/1">one</a>
<div class="e-content">unrelated</div></div>
<div class="h-entry"><a class="u-url" href="https://feed.example/2">two</a>
<time class="dt-published" datetime="2026-11-01T09:00:00-07:00">x</time>
<a class="u-in-reply-to" href="https://me.example/note/a">re</a>
<div class="e-content">my reply</div></div>
</div>'''
def test_responding_entry_and_feed_author():
parsed = mf2py.parse(doc=FEED_PAGE, url='https://feed.example/')
entry = mf2_utils.responding_entry(parsed, 'https://me.example/note/a#frag')
assert mf2_utils.first(entry, 'url') == 'https://feed.example/2'
assert mf2_utils.response_type(entry, 'https://me.example/note/a') == 'reply'
author = extract_author(parsed, 'https://feed.example/', entry)
assert author['author_name'] == 'Feed Owner'
assert author['author_url'] == 'https://feed.example/'
details = extract_source_details(entry)
assert details['source_post_url'] == 'https://feed.example/2'
assert details['source_published'] == '2026-11-01T16:00:00Z'
def test_author_as_url_or_name():
url_author = '<div class="h-entry"><a class="u-author" href="https://al.example/">al</a></div>'
parsed = mf2py.parse(doc=url_author, url='https://s.example/')
assert extract_author(parsed, 'https://s.example/')['author_url'] == 'https://al.example/'
name_author = '<div class="h-entry"><span class="p-author">Al Smith</span></div>'
parsed = mf2py.parse(doc=name_author, url='https://s.example/')
author = extract_author(parsed, 'https://s.example/')
assert author['author_name'] == 'Al Smith' and author['author_url'] == 'https://s.example/'
+86
View File
@@ -0,0 +1,86 @@
"""
Migrations under concurrent gunicorn workers (v2.8.1)
The v2.8.0 deploy failed to boot: one worker applied migration 014 while
another applied it again ("duplicate column name"), because
conn.executescript() committed and released the migration lock early.
"""
import multiprocessing
import sqlite3
from pathlib import Path
from starpunk.database.schema import INITIAL_SCHEMA_SQL
from starpunk.migrations import execute_in_transaction, run_migrations, split_sql_statements
MIGRATIONS = Path(__file__).parent.parent / "migrations"
def test_script_runs_inside_the_transaction(tmp_path):
conn = sqlite3.connect(tmp_path / "t.db")
conn.execute("CREATE TABLE t (a INTEGER)")
conn.commit()
conn.execute("BEGIN IMMEDIATE")
execute_in_transaction(conn, "ALTER TABLE t ADD COLUMN b TEXT;\nALTER TABLE t ADD COLUMN c TEXT;")
conn.rollback()
assert [row[1] for row in conn.execute("PRAGMA table_info(t)")] == ["a"]
def test_split_keeps_trigger_bodies_together():
script = """
-- comment; with a semicolon
CREATE TABLE t (a TEXT DEFAULT 'x;y');
CREATE TRIGGER tr AFTER DELETE ON t BEGIN
DELETE FROM t WHERE a = old.a;
SELECT 1;
END;
"""
statements = split_sql_statements(script)
assert len(statements) == 2
assert statements[1].startswith("CREATE TRIGGER") and statements[1].endswith("END;")
def _migrate(db_path, barrier, errors):
try:
barrier.wait()
run_migrations(db_path)
except Exception as e: # report to the parent
errors.put(repr(e))
import pytest
@pytest.mark.parametrize("trial", range(5)) # the race hit ~1 in 5 runs
def test_concurrent_workers_apply_each_migration_once(tmp_path, trial):
# A database at migration 013, like production before v2.8.0
db_path = tmp_path / "starpunk.db"
conn = sqlite3.connect(db_path)
conn.executescript(INITIAL_SCHEMA_SQL) # as init_db does
conn.close()
run_migrations(db_path)
conn = sqlite3.connect(db_path)
conn.execute("DELETE FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'")
conn.commit()
columns = [row[1] for row in conn.execute("PRAGMA table_info(webmentions)")]
if "source_post_url" in columns:
conn.execute("ALTER TABLE webmentions DROP COLUMN source_post_url")
conn.execute("ALTER TABLE webmentions DROP COLUMN source_published")
conn.commit()
conn.close()
ctx = multiprocessing.get_context("fork")
barrier, errors = ctx.Barrier(4), ctx.Queue()
workers = [ctx.Process(target=_migrate, args=(db_path, barrier, errors)) for _ in range(4)]
for w in workers:
w.start()
for w in workers:
w.join(timeout=60)
assert errors.empty(), errors.get()
conn = sqlite3.connect(db_path)
assert conn.execute(
"SELECT COUNT(*) FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'"
).fetchone()[0] == 1
+4 -3
View File
@@ -43,7 +43,7 @@ class TestRetryLogic:
from starpunk.database import init_db
from starpunk import create_app
app = create_app({'DATABASE_PATH': str(temp_db)})
app = create_app({'DATABASE_PATH': str(temp_db), 'TESTING': True})
init_db(app)
# Verify migrations table exists and has records
@@ -302,12 +302,13 @@ class TestErrorHandling:
def test_rollback_failure_causes_system_exit(self, temp_db):
"""Test that rollback failure raises SystemExit"""
with patch('sqlite3.connect') as mock_connect:
with patch('sqlite3.connect') as mock_connect, \
patch('starpunk.migrations.execute_in_transaction',
side_effect=Exception("Migration failed")):
mock_conn = MagicMock()
mock_connect.return_value = mock_conn
# Make both migration and rollback fail
mock_conn.executescript.side_effect = Exception("Migration failed")
mock_conn.rollback.side_effect = Exception("Rollback failed")
mock_conn.execute.return_value.fetchone.return_value = (0,)
+82 -1
View File
@@ -17,6 +17,11 @@ from starpunk.webmention_processor import (
from starpunk.webmention_worker import WebmentionWorker
PUBLIC = [(2, 1, 6, '', ('93.184.216.34', 0))]
def original_url(request):
"""The URL a pinned request was for (connections go to the checked IP)"""
return f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
BASE = 'https://t.example/test/1?x=1'
@@ -32,7 +37,7 @@ def mock_web():
routes = {}
def handler(request):
return routes[str(request.url)](request)
return routes[original_url(request)](request)
with patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC), \
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
@@ -106,6 +111,7 @@ def test_4_5_redirect_to_private_address_blocked(app):
return [(2, 1, 6, '', (ip, 0))]
def handler(request):
assert request.url.host == '93.184.216.34' # never connects to the private one
return httpx.Response(302, headers={'Location': 'http://metadata.internal/'})
with app.app_context(), \
@@ -281,3 +287,78 @@ def test_3_1_5_delete_resends_and_returns_410(app, client, soft):
assert rows[0][2] is None # detached, so it survives the delete
assert client.get(f"/note/{note.slug}").status_code == 410
assert client.get("/note/never-existed").status_code == 404
# 4.5 Connections go to the address that was checked (no DNS rebinding)
def test_4_5_connection_pinned_to_checked_address(app):
seen = []
def handler(request):
seen.append((str(request.url), request.headers['host'],
request.extensions.get('sni_hostname')))
return httpx.Response(200, text='ok')
with app.app_context(), \
patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC) as resolve, \
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
webmention_ssrf.safe_get('https://site.example:8443/p?q=1', {})
webmention_ssrf.safe_post('http://site.example/wm', {'source': 's'}, {})
assert seen == [
('https://93.184.216.34:8443/p?q=1', 'site.example:8443', 'site.example'),
('http://93.184.216.34/wm', 'site.example', None),
]
assert resolve.call_count == 2 # one lookup per request, then pinned
def test_4_5_ipv4_preferred(app):
with app.app_context(), patch("starpunk.webmention_ssrf.socket.getaddrinfo",
return_value=[(10, 1, 6, '', ('2606:2800::1', 0, 0, 0))] + PUBLIC):
assert webmention_ssrf.safe_resolve('x.example') == '93.184.216.34'
# 3.1.4 A change in the responses a note shows is an update of the note
def _note_with_sent_mention(app):
from starpunk.notes import create_note
with app.test_request_context():
note = create_note("See [a](https://a.example/x)", published=True)
_sent(app)
db = sqlite3.connect(app.config['DATABASE_PATH'])
db.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) VALUES (?, ?, ?)",
('https://s.example/reply', f'http://localhost:5000/note/{note.slug}', note.id))
db.commit()
return note, db
def test_3_1_4_new_response_resends(app, worker):
note, db = _note_with_sent_mention(app)
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
assert [(t, s) for t, s, _ in _queue(app)] == [('https://a.example/x', 'pending')]
def test_3_1_4_unchanged_response_does_not_resend(app, worker):
note, db = _note_with_sent_mention(app)
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
_sent(app)
db.execute("UPDATE webmentions SET validation_status = 'pending'") # re-sent by source
db.commit()
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
assert [(t, s) for t, s, _ in _queue(app)] == [('https://a.example/x', 'sent')]
def test_3_1_4_moderation_change_resends(app, worker):
from starpunk.webmention_queries import update_webmention_status
note, db = _note_with_sent_mention(app)
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
_sent(app)
with app.app_context():
update_webmention_status([1], 'approved') # already approved: no change
assert _queue(app)[0][1] == 'sent'
with app.app_context():
update_webmention_status([1], 'hidden')
assert _queue(app)[0][1] == 'pending'
+9 -7
View File
@@ -37,16 +37,18 @@ def test_worker_sends_queued_webmention(app):
conn.commit()
conn.close()
def target_site(request):
return httpx.Response(200, headers={"Link": '</wm>; rel="webmention"'})
posts = []
accepted = MagicMock(status_code=202, text="")
def target_site(request):
if request.method == "POST":
posts.append((request.headers["host"], request.url.path, request.content))
return httpx.Response(202)
return httpx.Response(200, headers={"Link": '</wm>; rel="webmention"'})
worker = WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"],
poll_interval=0.05)
with patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC), \
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(target_site)), \
patch("starpunk.webmention_worker.httpx.post", return_value=accepted) as post:
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(target_site)):
worker.start()
try:
def status():
@@ -56,8 +58,8 @@ def test_worker_sends_queued_webmention(app):
return row
assert _wait_for(lambda: status()[0] != "pending"), status()
assert status() == ("sent", None)
post.assert_called_once()
assert post.call_args.args[0] == "https://example.org/wm"
assert [(h, p) for h, p, _ in posts] == [("example.org", "/wm")]
assert b"source=" in posts[0][2] and b"target=" in posts[0][2]
finally:
worker.stop()
worker.join(timeout=2)