Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
57682165a4 | ||
|
|
8c51b52ed6 | ||
|
|
42c77e34b7 | ||
|
|
438465592c | ||
|
|
c8a7ef309f | ||
|
|
0640730416 | ||
|
|
51f711ea4a | ||
|
|
6a76ee26af | ||
|
|
43ec7cb8be | ||
|
|
4aaa7a8122 | ||
|
|
0cfc9965df | ||
|
|
106495d114 |
No files matched your search
@@ -1,5 +1,5 @@
|
||||
# Gitea Actions workflow for StarPunk
|
||||
# Builds and pushes container images on version tags.
|
||||
# Runs the tests, then builds and pushes container images on version tags.
|
||||
#
|
||||
# The runner has no Docker daemon (that would give workflows root on the
|
||||
# server). Images are built by the rootless BuildKit container next to it,
|
||||
@@ -30,6 +30,10 @@ jobs:
|
||||
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "$REGISTRY" -u "${{ github.repository_owner }}" --password-stdin
|
||||
- name: Use the BuildKit socket
|
||||
run: docker buildx create --name ci --driver remote "$BUILDKIT" --use
|
||||
# Builds the Containerfile's test stage, which runs pytest; a failure
|
||||
# stops the workflow before anything is pushed
|
||||
- name: Run tests
|
||||
run: docker buildx build --target test --progress plain -f Containerfile .
|
||||
- name: Build and push
|
||||
run: |
|
||||
image="$REGISTRY/$(echo '${{ github.repository }}' | tr '[:upper:]' '[:lower:]')"
|
||||
|
||||
@@ -7,6 +7,95 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [2.8.1] - 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- Starting the server after an upgrade with a database migration could fail: several server processes applied the same migration at once and one stopped with "duplicate column name", taking the server down until it restarted (this happened deploying v2.8.0, for about two seconds). The migration lock was released right after it was taken, and again before each migration was recorded. Migrations now run entirely inside the lock
|
||||
|
||||
## [2.8.0] - 2026-11-03
|
||||
|
||||
Microformats2 review: the markup now matches the [microformats2](https://microformats.org/wiki/microformats2) vocabularies, so IndieWeb readers and tools see what the page shows.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Replies to a note weren't marked up as its comments, so readers and threading tools saw none. They are now `u-comment h-cite`s with the replier's h-card (name, link, photo) and the reply's HTML
|
||||
- Likes, reposts, bookmarks and mentions on a note had no microformats at all. Each is now a `u-like`, `u-repost`, `u-bookmark` or `u-mention` h-cite with its author, including likes past the eight faces shown
|
||||
- Responses link to and are dated by the original post (its own URL and published time) instead of the address that sent the webmention and when it arrived. New database columns hold these (migration 014)
|
||||
- Post times had no time zone (and production showed microseconds), so readers could shift them by hours. Times on pages and in Micropub `q=source` are now UTC with a `Z`, matching the Atom feed
|
||||
- A note starting with a hashtag (`#indieweb is great…`) was treated as a titled article: its text became a heading and the body was empty, and a note starting with a bare `#` line got its slug as a title. Only a Markdown heading (`#` then a space) makes a title now
|
||||
- A webmention from one of a note's own crossposts (for example your GoToSocial copy, sent back through Bridgy) showed up as a response on the note. It is now ignored
|
||||
- Micropub and IndieAuth discovery links disappeared from pages when webmentions were turned off
|
||||
- A new site had no `rel="me"` link to its author, so readers couldn't tell whose site it was; pages now always link `rel="me"` and `rel="author"` to the author's home page
|
||||
- Finding the author's profile could pick someone else's h-card from their home page. It now follows the [representative h-card](https://microformats.org/wiki/representative-h-card-parsing) rules
|
||||
- A webmention's author is found by the [authorship algorithm](https://indieweb.org/authorship-spec), from the entry on the source page that actually responds to the note: an author given as a name or link, and the author of a feed the entry is in, now count
|
||||
|
||||
### Added
|
||||
|
||||
- Tag pages and the likes page have a feed URL and author, like the home page
|
||||
- Note previews have `u-uid`
|
||||
|
||||
## [2.7.2] - 2026-11-03
|
||||
|
||||
### Added
|
||||
|
||||
- When the replies, likes and other responses shown on a note change (a webmention arrives, is updated or removed, or you approve, hide or delete one), the note's webmentions are sent again, as the Webmention spec recommends, so the sites it links to can pick up the conversation. A response that is checked again without changing sends nothing
|
||||
|
||||
### Security
|
||||
|
||||
- Webmention fetches and sends connect to the exact address that passed the private-address check. Before, the hostname was looked up a second time when connecting, so a DNS server that changed its answer could still point StarPunk at a private address. TLS certificates are still checked against the hostname. IPv4 addresses are preferred when a host has both
|
||||
|
||||
## [2.7.1] - 2026-11-03
|
||||
|
||||
### Changed
|
||||
|
||||
- Releases run the full test suite first: CI builds the Containerfile's new `test` stage (same Python image as production) and stops before pushing an image if any test fails
|
||||
|
||||
### Fixed
|
||||
|
||||
- Eight tests that had been failing: the image validation tests still expected bytes (validation has returned an image object since v1.5.0), and a migration test started the app in production mode
|
||||
|
||||
## [2.7.0] - 2026-11-03
|
||||
|
||||
Webmention now follows the [W3C Recommendation](https://www.w3.org/TR/webmention/) throughout.
|
||||
|
||||
### Changed
|
||||
|
||||
- Webmentions are only accepted for published notes. Mentions of the home page or other pages were stored but never shown; they're now refused with 400
|
||||
- Deleted notes answer 410 Gone with a "This post has been deleted" page instead of 404
|
||||
- A received mention that is sent again keeps its moderation: one you hid stays hidden, one you approved stays approved
|
||||
|
||||
### Fixed
|
||||
|
||||
- Sending: endpoint discovery now handles `rel` values with several words (`rel="webmention other"`), any capitalisation, `rel` after other Link parameters, and several Link headers, and takes the first `<link>` or `<a>` in page order
|
||||
- Sending: the endpoint is looked up again every time (the 1–24 hour cache broke the spec's rule for updated posts)
|
||||
- Sending: editing a note notifies every page it mentioned before, including an old reply target and links the edit removed. Deleting or restoring a note notifies them too, so they can remove or restore the mention
|
||||
- Sending: a redirect from the endpoint is no longer treated as success (the POST was turned into a GET); "429 Too Many Requests" is retried
|
||||
- Receiving: the source must link to the exact target URL (in a link, image, or other HTML attribute, any JSON value, or plain text). A longer URL that starts with the target, or the target mentioned in plain text or an HTML comment, no longer counts
|
||||
- Receiving: relative links in the source are resolved, using the URL after redirects
|
||||
- Receiving: the source is fetched with an `Accept` header and decoded in its declared character set
|
||||
- Receiving: if the source can't be reached when a mention is sent again, the existing mention stays up instead of disappearing
|
||||
- Receiving: the target's host is compared without regard to case or a default port, and `#fragment`s are dropped, so one mention isn't stored twice
|
||||
- Mention snippets cut at 500 characters no longer leave HTML tags open, and the "Mentions" list no longer shows raw HTML tags
|
||||
- The `Link: rel="webmention"` header is added alongside other Link headers instead of replacing them
|
||||
- Links at the top level of a code block were still sent webmentions
|
||||
|
||||
### Security
|
||||
|
||||
- Fetches for webmentions check every redirect, not just the first URL, so a public page can't redirect StarPunk to a private address. Every address a hostname resolves to (IPv4 and IPv6) must be public; carrier-grade NAT, `0.0.0.0/8`, multicast, reserved and IPv4-mapped addresses are blocked too. Each fetch has a total time limit and a 1 MB size limit, including endpoint discovery
|
||||
|
||||
## [2.6.9] - 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- Received webmentions failed when the sender's profile photo had alt text (Bridgy, Mastodon and most IndieWeb sites do this). The photo is now stored as its URL
|
||||
- Outgoing webmentions were never sent: endpoint discovery crashed on every target that had an endpoint, and the target was then marked as having none. Discovery now works, and network errors during discovery are retried instead of giving up
|
||||
- Each server process ran its own webmention worker, so a webmention could be processed or sent more than once. Only one process handles the queues at a time now; if it stops, another takes over
|
||||
|
||||
### Security
|
||||
|
||||
- A webmention sender could set its author link to a `javascript:` URL, which was shown as a clickable link on the note page. Author links and photos are now only kept if they are http or https URLs
|
||||
|
||||
## [2.6.8] - 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
+21
-1
@@ -23,7 +23,27 @@ RUN uv venv /opt/venv && \
|
||||
uv pip install --no-cache -r requirements.txt
|
||||
|
||||
# ============================================================================
|
||||
# Runtime Stage - Minimal production image
|
||||
# Test Stage - Runs the test suite (CI builds it with --target test)
|
||||
# ============================================================================
|
||||
FROM python:3.11-slim AS test
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
COPY --from=builder /opt/venv /opt/venv
|
||||
RUN uv pip install --no-cache --python /opt/venv/bin/python pytest pytest-mock
|
||||
|
||||
ENV PATH="/opt/venv/bin:$PATH" \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
SESSION_SECRET=ci-test-secret \
|
||||
SITE_URL=http://localhost:5000 \
|
||||
ADMIN_ME=https://test.example.com
|
||||
|
||||
WORKDIR /app
|
||||
COPY . .
|
||||
RUN python -m pytest -q -p no:cacheprovider tests
|
||||
|
||||
# ============================================================================
|
||||
# Runtime Stage - Minimal production image (default target: keep it last)
|
||||
# ============================================================================
|
||||
FROM python:3.11-slim
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# v2.6.9: Webmention processing fixes
|
||||
|
||||
Found after v2.6.8 let the worker run in production, and by a review of the
|
||||
implementation against the W3C Webmention Recommendation.
|
||||
|
||||
## Fixes
|
||||
|
||||
1. **Author photo with alt text** (`webmention_processor.extract_author`).
|
||||
mf2py returns `u-photo` with an `alt` attribute as
|
||||
`{'value': url, 'alt': ...}`; the dict was bound to SQLite and the update
|
||||
failed. `_first_str()` reduces mf2 values to their `value`.
|
||||
2. **Endpoint discovery** (`WebmentionWorker._discover_endpoint`).
|
||||
`response.url` is an `httpx.URL`; `urljoin(str, URL)` raised `TypeError`,
|
||||
which was caught and returned as "no endpoint", so every target was marked
|
||||
`no_endpoint`. Pass `str(response.url)`. Fetch errors now propagate so the
|
||||
caller schedules a retry, instead of being treated as "no endpoint".
|
||||
3. **Duplicate processing.** gunicorn runs 4 processes, each with a worker,
|
||||
and queue items were claimed by SELECT then UPDATE. The worker now takes a
|
||||
non-blocking `flock` on `<data>/webmention-worker.lock` before processing;
|
||||
others wait and retry each poll, so one takes over if the holder dies.
|
||||
Orphan cleanup runs when the lock is acquired.
|
||||
4. **XSS (spec 4.1).** `author_url` and `author_photo` are kept only if they
|
||||
are absolute http(s) URLs; otherwise the source URL / no photo is used.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_webmention_worker.py`: the send test uses a real `httpx.Response`
|
||||
(the old `MagicMock` with a string URL hid fix 2), plus tests for the lock,
|
||||
an alt-text photo and a `javascript:` author URL. All fail on v2.6.8.
|
||||
|
||||
## Remaining spec gaps
|
||||
|
||||
Tracked separately: rel-token and document-order endpoint discovery, exact
|
||||
target match during verification, Accept header, re-sending on delete, 410
|
||||
for deleted notes, SSRF re-validation on redirects, and the others listed in
|
||||
the compliance review.
|
||||
@@ -0,0 +1,51 @@
|
||||
# v2.7.0: Webmention W3C compliance
|
||||
|
||||
Review of StarPunk's Webmention support against the W3C Recommendation
|
||||
(https://www.w3.org/TR/webmention/, 12 January 2017), and the changes made.
|
||||
v2.6.8 and v2.6.9 fixed the bugs that stopped webmentions working at all.
|
||||
|
||||
## Sending
|
||||
|
||||
| Section | Requirement | Change |
|
||||
|---|---|---|
|
||||
| 3.1.2 | Link header with rel=webmention first (MUST) | RFC 8288 parsing: several links per header, several headers, `rel` in any position, token list, case-insensitive |
|
||||
| 3.1.2 | Then first `<link>`/`<a>` rel=webmention in document order (MUST) | Single pass over `link` and `a` in document order; rel token match; elements without `href` skipped |
|
||||
| 3.1.2 | Relative URL resolved against target; empty href is the target; query kept (MUST) | `urljoin` against the final URL after redirects |
|
||||
| 3.1.2 | Follow redirects | `safe_get` follows up to 10, checking each hop |
|
||||
| 3.1.2 | User-Agent with "Webmention" (MAY) | `StarPunk/<version> Webmention (+site)` |
|
||||
| 3.1.3 | Form-encoded POST; any 2xx success (MUST) | Unchanged. Redirects not followed (httpx turned 301/302 POST into GET); 429 retried |
|
||||
| 3.1.4 | Re-discover endpoint on update (MUST) | Discovery cache removed |
|
||||
| 3.1.4 | Re-send to previously sent targets, including removed links (SHOULD) | `previous_targets()` from the queue, plus removed links; not capped |
|
||||
| 3.1.5 | Deleted post returns 410 (SHOULD), tombstone (SHOULD) | `public.note` aborts 410 for soft-deleted notes and hard-deleted notes that sent webmentions; `410.html` |
|
||||
| 3.1.5 | Re-send on delete (SHOULD) | `requeue_previous_webmentions()`; for a hard delete the rows are detached (`source_note_id = NULL`) first, since the foreign key cascades. Also on undelete |
|
||||
|
||||
## Receiving
|
||||
|
||||
| Section | Requirement | Change |
|
||||
|---|---|---|
|
||||
| 3.2 | 202 for async processing (MUST) | Unchanged |
|
||||
| 3.2.1 | Valid http(s) URLs; source ≠ target; 400 on error (MUST) | Unchanged |
|
||||
| 3.2.1 | Target is a valid resource (SHOULD) | Only published notes; others 400 |
|
||||
| 3.2.1 | Fragment ignored | Stripped before storing (no duplicate rows) |
|
||||
| 3.2.2 | GET source, follow redirects, limit redirects | `safe_get` |
|
||||
| 3.2.2 | Accept header (SHOULD) | Added |
|
||||
| 3.2.2 | Exact match of target, per media type (MUST) | HTML: every link attribute (`lxml iterlinks`) resolved against the final URL; JSON: any string value; `text/*`: substring; other types: no match. Fragments ignored |
|
||||
| 3.2.4 | Update existing data; 410 / no link removes it (SHOULD) | Unchanged in effect. Moderation decisions kept on update; transient errors keep a valid mention |
|
||||
|
||||
## Security (section 4)
|
||||
|
||||
- 4.1 XSS: author URL/photo must be http(s) (v2.6.9); snippets re-sanitized after truncation; Mentions list uses `striptags`.
|
||||
- 4.2 / 4.5: `safe_get` checks every redirect hop, checks all `getaddrinfo` addresses, blocks private, loopback, link-local, multicast, reserved, unspecified, CGNAT, `0.0.0.0/8` and IPv4-mapped forms; 1 MB and 10 s total limits.
|
||||
- 4.3: endpoint URL checked before POST, no redirects followed.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Homepage mentions are refused rather than stored.
|
||||
- Discovery does not apply HTTP cache headers (5.3); targets are only
|
||||
fetched when a note is created, edited or deleted.
|
||||
- The 410 page is a generic tombstone without microformats.
|
||||
- `error` rows (unexpected failures) are not retried automatically.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_webmention_spec.py` (section-numbered), `tests/test_webmention_worker.py`.
|
||||
@@ -0,0 +1,36 @@
|
||||
# v2.7.2: Remaining Webmention SHOULDs
|
||||
|
||||
Follow-up to the v2.7.0 compliance review. Every SHOULD in the
|
||||
Recommendation was checked; two were not met.
|
||||
|
||||
## 4.5 DNS pinning
|
||||
|
||||
`validate_url_for_fetch()` now returns the checked address.
|
||||
`_pinned_request()` builds each request against that IP, with the original
|
||||
host in the `Host` header and, for https, in the `sni_hostname` extension.
|
||||
httpcore uses `sni_hostname` as `server_hostname` with `check_hostname`
|
||||
on, so the certificate is verified against the real hostname. Used by
|
||||
`safe_get()` (every redirect hop) and the new `safe_post()` (the webmention
|
||||
POST). IPv4 is preferred when a host has both families.
|
||||
|
||||
## 3.1.4 Displayed responses are an update
|
||||
|
||||
"If the source post displays responses... then sender SHOULD treat that as
|
||||
an update of the source URL and re-send any previously sent Webmentions."
|
||||
|
||||
`note_responses_changed()` re-queues a published note's previous webmentions.
|
||||
It is called when:
|
||||
|
||||
- the worker validates a webmention and what the note shows for it changed
|
||||
(`_display_state()` before and after: shown or not, type, author, snippet).
|
||||
A re-send from the source resets the row to `pending`; a previously valid
|
||||
row still counts as shown, so an unchanged re-validation sends nothing.
|
||||
This keeps two sites that both implement this from ping-ponging.
|
||||
- an admin approves, hides or deletes webmentions and a valid one changes
|
||||
between shown and not shown.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_webmention_spec.py`: pinning (URL, Host, SNI, one lookup per
|
||||
request), IPv4 preference, re-send on a new response, none on an unchanged
|
||||
one, re-send on moderation changes.
|
||||
@@ -0,0 +1,38 @@
|
||||
# v2.8.0: Microformats2 review
|
||||
|
||||
Review of StarPunk's published and consumed microformats against
|
||||
microformats2, h-entry, h-card, h-feed, h-cite, representative h-card
|
||||
parsing, the authorship spec and Post Type Discovery. Every page type was
|
||||
rendered, parsed with mf2py and checked with mf2util.
|
||||
|
||||
Posts (h-entry) were largely correct: no implied names, PTD right for
|
||||
note/article/reply/like/repost/photo, author h-card, u-url/u-uid,
|
||||
p-category, u-syndication.
|
||||
|
||||
## Fixes
|
||||
|
||||
| # | Problem | Fix |
|
||||
|---|---|---|
|
||||
| 1 | Replies were unattached `h-cite` children with a plain-text author | `u-comment h-cite`, `p-author h-card` (u-url on an inner link, u-photo, p-name), `e-content` (`templates/partials/responses.html`) |
|
||||
| 2 | Likes/reposts/bookmarks/mentions had no mf2 | `u-like` / `u-repost` / `u-bookmark` / `u-mention` h-cites; faces past 8 are `hidden`, not omitted |
|
||||
| 3 | `dt-*` naive, with microseconds | `iso_utc` filter/helper: RFC 3339 UTC `Z`, no fraction; Micropub `q=source` too |
|
||||
| 4 | `#hashtag` first line = title; bare `#` = slug title | `split_title` requires an ATX heading (`#{1,6}` + space); templates use `Note.has_title` |
|
||||
| 5 | Note's own syndicated copy shown as a mention | Worker rejects a source whose url/uid (or URL) is one of the note's `syndications.remote_url` as invalid |
|
||||
| 6 | Micropub/IndieAuth links gated on WEBMENTION_ENABLED | Only `rel=webmention` is gated |
|
||||
| 7 | No rel=me on a fresh install | `rel=me` and `rel=author` to `author.url or author.me or ADMIN_ME`, always |
|
||||
| 8 | Author discovery guessed h-cards | `mf2_utils.representative_hcard` (uid+url, rel=me, single h-card); parsed with the URL after redirects |
|
||||
| 9 | Webmention entry/author/time | `mf2_utils.responding_entry` (entry whose in-reply-to/like-of/... or content refers to the target), `entry_author` (author property as h-card/URL/name, parent h-feed author, rel=author, representative h-card), `source_post_url` and `source_published` stored (migration 014) |
|
||||
|
||||
Also: tag and likes h-feeds have `u-url` and `p-author`; previews have `u-uid`.
|
||||
|
||||
## Not done
|
||||
|
||||
- Reply-context h-cite does not carry the cached author h-card, name or published time.
|
||||
- Photo alt falls back to "Image" when a photo-only post has no caption.
|
||||
- Person tags (`p-category h-card`) are not supported.
|
||||
- Existing webmentions keep NULL source details until they are re-validated.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_microformats_v280.py`. `tests/test_microformats.py`
|
||||
`test_no_relme_without_author` became `test_relme_to_admin_me_without_author`.
|
||||
@@ -0,0 +1,33 @@
|
||||
# v2.8.1: Migration lock held for the whole run
|
||||
|
||||
## Incident
|
||||
|
||||
Deploying v2.8.0 (migration 014 adds two columns): one gunicorn worker
|
||||
applied 014, another failed with `duplicate column name: source_post_url`,
|
||||
gunicorn shut down ("Worker failed to boot") and Docker restarted the
|
||||
container. About two seconds of downtime; schema correct afterwards.
|
||||
|
||||
## Causes
|
||||
|
||||
`run_migrations` takes `BEGIN IMMEDIATE` so one worker migrates at a time,
|
||||
but the lock was released early twice:
|
||||
|
||||
1. `create_migrations_table()`, called right after `BEGIN IMMEDIATE`,
|
||||
ended with `conn.commit()`. Everything after it ran without the lock.
|
||||
2. `conn.executescript()` COMMITs any open transaction before running the
|
||||
script, so the lock was dropped before the migration was recorded in
|
||||
`schema_migrations`.
|
||||
|
||||
## Fix
|
||||
|
||||
- `create_migrations_table(conn, commit=True)`; `run_migrations` passes
|
||||
`commit=False`.
|
||||
- `execute_in_transaction()` runs each statement from
|
||||
`split_sql_statements()` (built on `sqlite3.complete_statement`, so
|
||||
trigger bodies stay whole) with `conn.execute`, inside the transaction.
|
||||
|
||||
## Verification
|
||||
|
||||
Four processes migrating a database at 013 simultaneously: 6/30 trials
|
||||
failed before, 0/30 after. `tests/test_migration_concurrency.py` runs this
|
||||
five times, plus a rollback test showing a migration commits nothing early.
|
||||
@@ -0,0 +1,6 @@
|
||||
-- Migration 014: Source details for received webmentions (v2.8.0)
|
||||
-- The source's own u-url and dt-published, so a response links to and dates
|
||||
-- the original post rather than the URL that sent it and when it arrived.
|
||||
|
||||
ALTER TABLE webmentions ADD COLUMN source_post_url TEXT;
|
||||
ALTER TABLE webmentions ADD COLUMN source_published TIMESTAMP;
|
||||
@@ -219,6 +219,10 @@ def create_app(config=None):
|
||||
|
||||
return {'author': author}
|
||||
|
||||
# Template filter - RFC 3339 UTC for dt-* properties (v2.8.0)
|
||||
from starpunk.utils import iso_utc
|
||||
app.add_template_filter(iso_utc, 'iso_utc')
|
||||
|
||||
# Template filter - URL to color for facepile (v1.6.0)
|
||||
@app.template_filter('url_to_color')
|
||||
def url_to_color_filter(url):
|
||||
@@ -302,7 +306,7 @@ def create_app(config=None):
|
||||
if (app.config.get('WEBMENTION_ENABLED', True) and
|
||||
response.content_type and
|
||||
'text/html' in response.content_type):
|
||||
response.headers['Link'] = '</webmention>; rel="webmention"'
|
||||
response.headers.add('Link', '</webmention>; rel="webmention"')
|
||||
return response
|
||||
|
||||
# Register centralized error handlers
|
||||
@@ -459,5 +463,5 @@ def create_app(config=None):
|
||||
|
||||
# Package version (Semantic Versioning 2.0.0)
|
||||
# See docs/standards/versioning-strategy.md for details
|
||||
__version__ = "2.6.8"
|
||||
__version_info__ = (2, 6, 8)
|
||||
__version__ = "2.8.1"
|
||||
__version_info__ = (2, 8, 1)
|
||||
@@ -82,10 +82,12 @@ def discover_author_profile(me_url: str) -> Optional[Dict]:
|
||||
response.raise_for_status()
|
||||
|
||||
# Parse microformats from HTML
|
||||
parsed = mf2py.parse(doc=response.text, url=me_url)
|
||||
# Relative URLs and the representative h-card use the URL after redirects
|
||||
page_url = str(response.url) if isinstance(response.url, (str, httpx.URL)) else me_url
|
||||
parsed = mf2py.parse(doc=response.text, url=page_url)
|
||||
|
||||
# Extract h-card (per Q16: first representative h-card)
|
||||
hcard = _find_representative_hcard(parsed, me_url)
|
||||
# Representative h-card (v2.8.0: per the parsing spec)
|
||||
hcard = _find_representative_hcard(parsed, page_url)
|
||||
|
||||
if not hcard:
|
||||
current_app.logger.warning(f"No h-card found at {me_url}")
|
||||
@@ -132,47 +134,21 @@ def discover_author_profile(me_url: str) -> Optional[Dict]:
|
||||
|
||||
def _find_representative_hcard(parsed: dict, me_url: str) -> Optional[dict]:
|
||||
"""
|
||||
Find representative h-card from parsed microformats
|
||||
Find the representative h-card of the author's page (v2.8.0)
|
||||
|
||||
Per Q16: First representative h-card = first h-card with p-name
|
||||
Per Q18: First h-card with url property matching profile URL
|
||||
Follows https://microformats.org/wiki/representative-h-card-parsing
|
||||
(uid+url match, rel=me match, or the only h-card with a matching url).
|
||||
Other h-cards on the page may be other people, so none is guessed.
|
||||
|
||||
Args:
|
||||
parsed: Parsed microformats data from mf2py
|
||||
me_url: Profile URL for matching
|
||||
me_url: Profile URL (the page that was parsed)
|
||||
|
||||
Returns:
|
||||
h-card dict or None if not found
|
||||
h-card dict or None if the page has no representative h-card
|
||||
"""
|
||||
items = parsed.get('items', [])
|
||||
|
||||
# First try: h-card with matching URL (most specific)
|
||||
for item in items:
|
||||
if 'h-card' in item.get('type', []):
|
||||
properties = item.get('properties', {})
|
||||
urls = properties.get('url', [])
|
||||
|
||||
# Check if any URL matches the profile URL
|
||||
for url in urls:
|
||||
if isinstance(url, dict):
|
||||
url = url.get('value', '')
|
||||
if _normalize_url(url) == _normalize_url(me_url):
|
||||
# Found matching h-card
|
||||
return item
|
||||
|
||||
# Second try: First h-card with p-name (representative h-card)
|
||||
for item in items:
|
||||
if 'h-card' in item.get('type', []):
|
||||
properties = item.get('properties', {})
|
||||
if properties.get('name'):
|
||||
return item
|
||||
|
||||
# Third try: Just use first h-card if any
|
||||
for item in items:
|
||||
if 'h-card' in item.get('type', []):
|
||||
return item
|
||||
|
||||
return None
|
||||
from starpunk.mf2_utils import representative_hcard
|
||||
return representative_hcard(parsed, me_url)
|
||||
|
||||
|
||||
def _get_property(hcard: dict, prop_name: str) -> Optional[str]:
|
||||
|
||||
@@ -82,6 +82,11 @@ def register_error_handlers(app):
|
||||
|
||||
return render_template('404.html'), 404
|
||||
|
||||
@app.errorhandler(410)
|
||||
def gone(error):
|
||||
"""Handle 410 Gone (deleted notes, Webmention 3.1.5)"""
|
||||
return render_template('410.html'), 410
|
||||
|
||||
@app.errorhandler(405)
|
||||
def method_not_allowed(error):
|
||||
"""Handle 405 Method Not Allowed errors"""
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
"""
|
||||
Microformats2 consuming helpers (v2.8.0)
|
||||
|
||||
Shared by Webmention processing and author discovery:
|
||||
- representative h-card of a page
|
||||
- the h-entry on a page that responds to a given URL
|
||||
- the author of an h-entry (authorship algorithm)
|
||||
|
||||
References:
|
||||
- https://microformats.org/wiki/representative-h-card-parsing
|
||||
- https://indieweb.org/authorship-spec
|
||||
"""
|
||||
|
||||
from typing import Iterator, Optional
|
||||
from urllib.parse import urldefrag, urlparse
|
||||
|
||||
RESPONSE_PROPERTIES = ('in-reply-to', 'like-of', 'repost-of', 'bookmark-of')
|
||||
|
||||
|
||||
def normalize_url(url: str) -> str:
|
||||
"""URL for comparison: no fragment, no trailing slash, lowercase scheme/host"""
|
||||
if not url:
|
||||
return ''
|
||||
url = urldefrag(url.strip())[0]
|
||||
parsed = urlparse(url)
|
||||
url = parsed._replace(scheme=parsed.scheme.lower(), netloc=parsed.netloc.lower()).geturl()
|
||||
return url.rstrip('/')
|
||||
|
||||
|
||||
def value_of(value) -> Optional[str]:
|
||||
"""Plain value of an mf2 property value (strings, {'value':...}, {'html':...})"""
|
||||
if isinstance(value, dict):
|
||||
value = value.get('value')
|
||||
return value if isinstance(value, str) and value else None
|
||||
|
||||
|
||||
def first(item: dict, prop: str) -> Optional[str]:
|
||||
"""First plain value of an item's property"""
|
||||
for value in item.get('properties', {}).get(prop, []):
|
||||
result = value_of(value)
|
||||
if result:
|
||||
return result
|
||||
return None
|
||||
|
||||
|
||||
def values(item: dict, prop: str) -> list[str]:
|
||||
"""All plain values of an item's property"""
|
||||
return [v for v in (value_of(x) for x in item.get('properties', {}).get(prop, [])) if v]
|
||||
|
||||
|
||||
def walk(items: list) -> Iterator[dict]:
|
||||
"""Every microformat in items, depth first: top level, children, nested properties"""
|
||||
for item in items:
|
||||
if not isinstance(item, dict) or 'type' not in item:
|
||||
continue
|
||||
yield item
|
||||
yield from walk(item.get('children', []))
|
||||
for prop_values in item.get('properties', {}).values():
|
||||
yield from walk([v for v in prop_values if isinstance(v, dict)])
|
||||
|
||||
|
||||
def is_type(item: dict, mf_type: str) -> bool:
|
||||
return mf_type in item.get('type', [])
|
||||
|
||||
|
||||
def representative_hcard(parsed: dict, page_url: str) -> Optional[dict]:
|
||||
"""
|
||||
The page's representative h-card, per the parsing spec, in order:
|
||||
|
||||
1. An h-card whose uid and url both match the page URL
|
||||
2. An h-card whose url matches one of the page's rel=me URLs
|
||||
3. The only h-card on the page, if its url matches the page URL
|
||||
|
||||
Otherwise the page has none.
|
||||
"""
|
||||
page = normalize_url(page_url)
|
||||
cards = [item for item in walk(parsed.get('items', [])) if is_type(item, 'h-card')]
|
||||
|
||||
for card in cards:
|
||||
if page in map(normalize_url, values(card, 'uid')) and \
|
||||
page in map(normalize_url, values(card, 'url')):
|
||||
return card
|
||||
|
||||
rel_me = {normalize_url(u) for u in parsed.get('rels', {}).get('me', [])}
|
||||
for card in cards:
|
||||
if rel_me & set(map(normalize_url, values(card, 'url'))):
|
||||
return card
|
||||
|
||||
if len(cards) == 1 and page in map(normalize_url, values(cards[0], 'url')):
|
||||
return cards[0]
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def responding_entry(parsed: dict, target_url: str) -> Optional[dict]:
|
||||
"""
|
||||
The h-entry on a page that refers to target_url
|
||||
|
||||
Prefers an entry whose in-reply-to / like-of / repost-of / bookmark-of
|
||||
is the target, then one whose content links to it, then the first
|
||||
top-level h-entry.
|
||||
"""
|
||||
target = normalize_url(target_url)
|
||||
entries = [item for item in walk(parsed.get('items', [])) if is_type(item, 'h-entry')]
|
||||
top = [item for item in parsed.get('items', []) if is_type(item, 'h-entry')]
|
||||
|
||||
if not target:
|
||||
return top[0] if top else (entries[0] if entries else None)
|
||||
|
||||
for entry in entries:
|
||||
for prop in RESPONSE_PROPERTIES:
|
||||
if target in map(normalize_url, values(entry, prop)):
|
||||
return entry
|
||||
|
||||
for entry in entries:
|
||||
for content in entry.get('properties', {}).get('content', []):
|
||||
html = content.get('html', '') if isinstance(content, dict) else str(content)
|
||||
if target_url in html or target in html:
|
||||
return entry
|
||||
|
||||
return top[0] if top else (entries[0] if entries else None)
|
||||
|
||||
|
||||
def response_type(entry: dict, target_url: str) -> str:
|
||||
"""reply / like / repost / bookmark if the entry is one to target_url, else mention"""
|
||||
target = normalize_url(target_url)
|
||||
for prop, kind in zip(RESPONSE_PROPERTIES, ('reply', 'like', 'repost', 'bookmark')):
|
||||
if target in map(normalize_url, values(entry, prop)):
|
||||
return kind
|
||||
return 'mention'
|
||||
|
||||
|
||||
def entry_author(parsed: dict, entry: Optional[dict], page_url: str) -> dict:
|
||||
"""
|
||||
Author of an h-entry, following the authorship algorithm without fetching
|
||||
|
||||
1. The entry's author property (an h-card, a URL, or a name)
|
||||
2. The author of an h-feed that contains the entry
|
||||
3. The page's rel=author link
|
||||
4. The page's representative h-card
|
||||
|
||||
Returns:
|
||||
{'name', 'url', 'photo'}; values may be None
|
||||
"""
|
||||
author = None
|
||||
if entry:
|
||||
author = (entry.get('properties', {}).get('author') or [None])[0]
|
||||
if author is None:
|
||||
for feed in walk(parsed.get('items', [])):
|
||||
if is_type(feed, 'h-feed') and entry in feed.get('children', []):
|
||||
author = (feed.get('properties', {}).get('author') or [None])[0]
|
||||
break
|
||||
|
||||
if author is None:
|
||||
rel_author = parsed.get('rels', {}).get('author', [])
|
||||
author = rel_author[0] if rel_author else representative_hcard(parsed, page_url)
|
||||
|
||||
if isinstance(author, dict) and 'type' in author:
|
||||
return {
|
||||
'name': first(author, 'name'),
|
||||
'url': first(author, 'url'),
|
||||
'photo': first(author, 'photo'),
|
||||
}
|
||||
|
||||
text = value_of(author)
|
||||
if text and urlparse(text).scheme in ('http', 'https'):
|
||||
return {'name': None, 'url': text, 'photo': None}
|
||||
return {'name': text, 'url': None, 'photo': None}
|
||||
@@ -36,7 +36,7 @@ from starpunk.notes import (
|
||||
InvalidNoteDataError, NoteNotFoundError,
|
||||
)
|
||||
from starpunk.indieauth import check_scope
|
||||
from starpunk.utils import absolute_url_for
|
||||
from starpunk.utils import absolute_url_for, iso_utc
|
||||
|
||||
|
||||
# Custom Exceptions
|
||||
@@ -743,7 +743,7 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
|
||||
title, content = split_title(note.content)
|
||||
props = {
|
||||
"content": [content],
|
||||
"published": [note.created_at.isoformat()],
|
||||
"published": [iso_utc(note.created_at)],
|
||||
"url": [absolute_url_for("public.note", slug=note.slug)],
|
||||
"post-status": ["published" if note.published else "draft"],
|
||||
}
|
||||
|
||||
+42
-6
@@ -31,12 +31,47 @@ class MigrationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def create_migrations_table(conn):
|
||||
def split_sql_statements(script: str) -> list[str]:
|
||||
"""
|
||||
Split a SQL script into complete statements (v2.8.1)
|
||||
|
||||
Uses sqlite3.complete_statement, so semicolons inside strings and
|
||||
trigger bodies (BEGIN ... END) don't split a statement.
|
||||
"""
|
||||
statements = []
|
||||
current = ""
|
||||
for line in script.splitlines(keepends=True):
|
||||
current += line
|
||||
if sqlite3.complete_statement(current):
|
||||
if current.strip().strip(";").strip():
|
||||
statements.append(current.strip())
|
||||
current = ""
|
||||
if current.strip():
|
||||
statements.append(current.strip())
|
||||
return statements
|
||||
|
||||
|
||||
def execute_in_transaction(conn, script: str) -> None:
|
||||
"""
|
||||
Run a SQL script inside the connection's open transaction
|
||||
|
||||
Not conn.executescript(): that COMMITs first, which released the
|
||||
migration lock mid-migration and let another gunicorn worker apply the
|
||||
same migration again ("duplicate column name", v2.8.0 deploy).
|
||||
"""
|
||||
for statement in split_sql_statements(script):
|
||||
conn.execute(statement)
|
||||
|
||||
|
||||
def create_migrations_table(conn, commit=True):
|
||||
"""
|
||||
Create schema_migrations tracking table if it doesn't exist
|
||||
|
||||
Args:
|
||||
conn: SQLite connection
|
||||
commit: Commit afterwards. run_migrations passes False: committing
|
||||
there released its BEGIN IMMEDIATE lock straight away, so
|
||||
gunicorn workers applied migrations concurrently (v2.8.1)
|
||||
"""
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
@@ -51,7 +86,8 @@ def create_migrations_table(conn):
|
||||
ON schema_migrations(migration_name)
|
||||
""")
|
||||
|
||||
conn.commit()
|
||||
if commit:
|
||||
conn.commit()
|
||||
|
||||
|
||||
def is_schema_current(conn):
|
||||
@@ -287,7 +323,7 @@ def apply_migration(conn, migration_name, migration_path, logger=None):
|
||||
|
||||
# Execute migration in transaction
|
||||
conn.execute("BEGIN")
|
||||
conn.executescript(migration_sql)
|
||||
execute_in_transaction(conn, migration_sql)
|
||||
|
||||
# Record migration as applied
|
||||
conn.execute(
|
||||
@@ -374,7 +410,7 @@ def run_migrations(db_path, logger=None):
|
||||
|
||||
try:
|
||||
# Ensure migrations tracking table exists
|
||||
create_migrations_table(conn)
|
||||
create_migrations_table(conn, commit=False)
|
||||
|
||||
# Quick check: have migrations already been applied by another worker?
|
||||
cursor = conn.execute("SELECT COUNT(*) FROM schema_migrations")
|
||||
@@ -472,8 +508,8 @@ def run_migrations(db_path, logger=None):
|
||||
|
||||
logger.debug(f"Applying migration: {migration_name}")
|
||||
|
||||
# Execute migration (already in transaction)
|
||||
conn.executescript(migration_sql)
|
||||
# Execute migration inside the locked transaction
|
||||
execute_in_transaction(conn, migration_sql)
|
||||
|
||||
# Record migration as applied
|
||||
conn.execute(
|
||||
|
||||
+19
-5
@@ -11,6 +11,7 @@ from dataclasses import dataclass, field, replace
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
import re
|
||||
import sqlite3
|
||||
|
||||
# Third-party imports
|
||||
@@ -40,6 +41,9 @@ MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br", BareUrlExtension()]
|
||||
|
||||
# Constants - Content limits
|
||||
MAX_TITLE_LENGTH = 200
|
||||
|
||||
# A Markdown ATX heading: up to 6 '#', whitespace, then text
|
||||
TITLE_RE = re.compile(r"#{1,6}[ \t]+(\S.*)$")
|
||||
EXCERPT_LENGTH = 200
|
||||
|
||||
|
||||
@@ -47,17 +51,19 @@ def split_title(content: str) -> tuple[Optional[str], str]:
|
||||
"""
|
||||
Split a note's '# Heading' first line (its title, per Q22) off its content
|
||||
|
||||
The first line must be a Markdown heading: 1-6 '#' then a space and text.
|
||||
'#indieweb ...' (a hashtag) or a bare '#' is not a title. This is the one
|
||||
place that decides whether a note has a title (templates use has_title).
|
||||
|
||||
Returns:
|
||||
(title, remaining content); title is None if the note has no heading
|
||||
"""
|
||||
stripped = content.lstrip()
|
||||
if not stripped.startswith("#"):
|
||||
return None, content
|
||||
first_line, _, rest = stripped.partition("\n")
|
||||
title = first_line.lstrip("#").strip()
|
||||
if not title:
|
||||
match = TITLE_RE.match(first_line)
|
||||
if not match:
|
||||
return None, content
|
||||
return title, rest.strip()
|
||||
return match.group(1).strip().rstrip("#").strip(), rest.strip()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -278,6 +284,14 @@ class Note:
|
||||
return self.html
|
||||
return markdown.markdown(body, extensions=MARKDOWN_EXTENSIONS)
|
||||
|
||||
@property
|
||||
def has_title(self) -> bool:
|
||||
"""True if the note starts with a '# Heading' title (an article)"""
|
||||
try:
|
||||
return split_title(self.content)[0] is not None
|
||||
except (FileNotFoundError, OSError):
|
||||
return False
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
"""
|
||||
|
||||
@@ -989,6 +989,17 @@ def delete_note(
|
||||
"Path validation failed",
|
||||
)
|
||||
|
||||
# Re-send webmentions so receivers learn the post is gone (spec 3.1.5).
|
||||
# A hard delete would cascade the queue rows away, so detach them first.
|
||||
if existing_note.published:
|
||||
try:
|
||||
from starpunk.webmention_sender import requeue_previous_webmentions
|
||||
requeue_previous_webmentions(
|
||||
Path(current_app.config["DATABASE_PATH"]), existing_note.id, detach=not soft
|
||||
)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue webmentions for deleted note: {e}")
|
||||
|
||||
# 5. PERFORM DELETION
|
||||
db = get_db(current_app)
|
||||
|
||||
@@ -1102,4 +1113,10 @@ def undelete_note(slug: str) -> Note:
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue syndication after undelete: {e}")
|
||||
|
||||
try:
|
||||
from starpunk.webmention_sender import requeue_previous_webmentions
|
||||
requeue_previous_webmentions(Path(current_app.config["DATABASE_PATH"]), row["id"])
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
|
||||
|
||||
return get_note(slug=slug)
|
||||
@@ -294,6 +294,27 @@ def likes():
|
||||
return render_template("likes.html", notes=notes)
|
||||
|
||||
|
||||
def _note_was_deleted(slug: str) -> bool:
|
||||
"""
|
||||
True if this slug was a note that has been deleted
|
||||
|
||||
Soft-deleted notes keep their row; hard-deleted notes that sent
|
||||
webmentions keep their queue rows (detached from the note).
|
||||
"""
|
||||
from starpunk.database import get_db
|
||||
from starpunk.utils import absolute_url_for
|
||||
|
||||
db = get_db(current_app)
|
||||
if db.execute(
|
||||
"SELECT 1 FROM notes WHERE slug = ? AND deleted_at IS NOT NULL", (slug,)
|
||||
).fetchone():
|
||||
return True
|
||||
return db.execute(
|
||||
"SELECT 1 FROM webmention_queue WHERE source_url = ? AND source_note_id IS NULL",
|
||||
(absolute_url_for("public.note", slug=slug),),
|
||||
).fetchone() is not None
|
||||
|
||||
|
||||
@bp.route("/note/<slug>")
|
||||
def note(slug: str):
|
||||
"""
|
||||
@@ -319,8 +340,10 @@ def note(slug: str):
|
||||
# Get note by slug
|
||||
note_obj = get_note(slug=slug)
|
||||
|
||||
# Return 404 if note doesn't exist or isn't published
|
||||
# Deleted notes are 410 Gone (Webmention 3.1.5); unknown or draft 404
|
||||
if not note_obj or not note_obj.published:
|
||||
if not note_obj and _note_was_deleted(slug):
|
||||
abort(410)
|
||||
abort(404)
|
||||
|
||||
# Get media for note (v1.2.0 Phase 3)
|
||||
|
||||
@@ -11,7 +11,7 @@ References:
|
||||
"""
|
||||
|
||||
from flask import Blueprint, current_app, request, jsonify
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import urldefrag, urlparse
|
||||
from starpunk.database import get_db
|
||||
from starpunk.notes import get_note, NoteNotFoundError
|
||||
from starpunk.rate_limiter import check_rate_limit
|
||||
@@ -107,17 +107,24 @@ def validate_url(url: str) -> tuple[bool, str | None]:
|
||||
return False, "URL is malformed"
|
||||
|
||||
|
||||
def _host_port(url: str) -> tuple[str, int | None]:
|
||||
parsed = urlparse(url)
|
||||
default = {'http': 80, 'https': 443}.get(parsed.scheme.lower())
|
||||
try:
|
||||
port = parsed.port or default
|
||||
except ValueError:
|
||||
port = None
|
||||
return (parsed.hostname or '').lower(), port
|
||||
|
||||
|
||||
def is_target_on_site(target_url: str) -> bool:
|
||||
"""
|
||||
Check if target URL is on this site.
|
||||
|
||||
Compares hostname of target_url with SITE_URL configuration.
|
||||
Compares host (case-insensitively) and effective port with SITE_URL.
|
||||
"""
|
||||
site_url = current_app.config.get('SITE_URL', '')
|
||||
site_parsed = urlparse(site_url)
|
||||
target_parsed = urlparse(target_url)
|
||||
|
||||
return site_parsed.netloc == target_parsed.netloc
|
||||
return _host_port(site_url) == _host_port(target_url)
|
||||
|
||||
|
||||
def resolve_target_note(target_url: str) -> int | None:
|
||||
@@ -228,16 +235,17 @@ def receive_webmention():
|
||||
"error_description": "Target URL is not on this site"
|
||||
}), 400
|
||||
|
||||
# Resolve target to note (may be None for homepage/other pages)
|
||||
target_note_id = resolve_target_note(target)
|
||||
# The fragment is ignored (spec 3.2.1) and not stored, so target#a and
|
||||
# target#b are the same mention
|
||||
target = urldefrag(target)[0]
|
||||
|
||||
# If target looks like a note URL but doesn't resolve, it's not found
|
||||
target_parsed = urlparse(target)
|
||||
target_path = target_parsed.path.rstrip('/')
|
||||
if target_path.startswith('/note/') and target_note_id is None:
|
||||
# Only published notes accept webmentions (spec 3.2.1: target must be a
|
||||
# valid resource that accepts them)
|
||||
target_note_id = resolve_target_note(target)
|
||||
if target_note_id is None:
|
||||
return jsonify({
|
||||
"error": "invalid_target",
|
||||
"error_description": "Target URL not found"
|
||||
"error_description": "Target URL does not accept webmentions"
|
||||
}), 400
|
||||
|
||||
# Insert webmention into database (queued for validation)
|
||||
|
||||
+24
-1
@@ -11,7 +11,7 @@ import hashlib
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
@@ -693,3 +693,26 @@ def absolute_url_for(endpoint: str, **values) -> str:
|
||||
from flask import current_app, url_for
|
||||
|
||||
return current_app.config["SITE_URL"].rstrip("/") + url_for(endpoint, **values)
|
||||
|
||||
|
||||
def iso_utc(value) -> str:
|
||||
"""
|
||||
Format a timestamp as RFC 3339 UTC for dt-* and Micropub (v2.8.0)
|
||||
|
||||
Stored times are naive UTC. Without an offset, microformats parsers read
|
||||
them as local time, so this adds 'Z' and drops microseconds (HTML
|
||||
<time datetime> allows at most 3 fractional digits).
|
||||
|
||||
Args:
|
||||
value: datetime (naive = UTC) or ISO 8601 string
|
||||
|
||||
Returns:
|
||||
e.g. '2026-11-01T16:25:12Z', or '' for empty values
|
||||
"""
|
||||
if not value:
|
||||
return ''
|
||||
if isinstance(value, str):
|
||||
value = datetime.fromisoformat(value.strip().replace('Z', '+00:00'))
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
|
||||
+171
-200
@@ -14,15 +14,19 @@ References:
|
||||
- W3C Webmention: https://www.w3.org/TR/webmention/
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import mf2py
|
||||
import nh3
|
||||
import httpx
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any, Optional
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import urldefrag, urlparse
|
||||
|
||||
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
|
||||
from starpunk import mf2_utils
|
||||
from starpunk.utils import iso_utc
|
||||
from starpunk.webmention_ssrf import (
|
||||
FetchTooLargeError, SSRFBlockedError, safe_get, user_agent,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -33,205 +37,155 @@ class ValidationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def fetch_source(url: str, site_url: str, timeout: int = 10, max_size: int = 1048576) -> tuple[str, str]:
|
||||
ACCEPT = ('text/html, application/xhtml+xml;q=0.9, application/json;q=0.8, '
|
||||
'text/plain;q=0.7, */*;q=0.1')
|
||||
|
||||
|
||||
class TransientError(ValidationError):
|
||||
"""Source could not be checked right now (network error, 5xx)"""
|
||||
pass
|
||||
|
||||
|
||||
def fetch_source(url: str, site_url: str) -> tuple[str, str, str]:
|
||||
"""
|
||||
Fetch source URL content with SSRF protection.
|
||||
|
||||
Per spec 3.2.2: GET following redirects (limited), with an Accept header.
|
||||
|
||||
Args:
|
||||
url: Source URL to fetch
|
||||
site_url: Site URL for User-Agent header
|
||||
timeout: Request timeout in seconds (default: 10)
|
||||
max_size: Maximum content size in bytes (default: 1MB)
|
||||
|
||||
Returns:
|
||||
Tuple of (content, content_type)
|
||||
Tuple of (content, content_type, final_url)
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError: If URL points to private IP
|
||||
ValidationError: If fetch fails or content too large
|
||||
ValidationError: Source is blocked, too large, or returned 4xx
|
||||
TransientError: Network error, timeout or 5xx
|
||||
"""
|
||||
# SSRF protection
|
||||
try:
|
||||
validate_url_for_fetch(url)
|
||||
result = safe_get(url, headers={'User-Agent': user_agent(site_url), 'Accept': ACCEPT})
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
logger.warning(f"SSRF protection blocked URL {url}: {e}")
|
||||
raise ValidationError(f"Blocked request: {e}")
|
||||
|
||||
# Fetch URL
|
||||
try:
|
||||
# Use streaming to check size before downloading entire content
|
||||
with httpx.stream(
|
||||
'GET',
|
||||
url,
|
||||
timeout=timeout,
|
||||
headers={'User-Agent': f'StarPunk/1.6.0 (+{site_url})'},
|
||||
follow_redirects=True
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
|
||||
# Check content length if provided
|
||||
content_length = response.headers.get('Content-Length')
|
||||
if content_length and int(content_length) > max_size:
|
||||
raise ValidationError(f"Source exceeds size limit ({max_size} bytes)")
|
||||
|
||||
# Read content with size limit
|
||||
content = b''
|
||||
for chunk in response.iter_bytes(chunk_size=8192):
|
||||
content += chunk
|
||||
if len(content) > max_size:
|
||||
raise ValidationError(f"Source exceeds size limit ({max_size} bytes)")
|
||||
|
||||
content_type = response.headers.get('Content-Type', '')
|
||||
|
||||
return content.decode('utf-8', errors='replace'), content_type
|
||||
|
||||
except FetchTooLargeError as e:
|
||||
raise ValidationError(f"Source {e}")
|
||||
except httpx.TimeoutException:
|
||||
raise ValidationError("Source fetch timed out")
|
||||
except httpx.HTTPStatusError as e:
|
||||
if 400 <= e.response.status_code < 500:
|
||||
raise ValidationError(f"Source returned {e.response.status_code}")
|
||||
else:
|
||||
raise ValidationError(f"Source server error: {e.response.status_code}")
|
||||
raise TransientError("Source fetch timed out")
|
||||
except httpx.HTTPError as e:
|
||||
raise ValidationError(f"Failed to fetch source: {e}")
|
||||
except UnicodeDecodeError:
|
||||
raise ValidationError("Source content encoding is not supported")
|
||||
raise TransientError(f"Failed to fetch source: {e}")
|
||||
|
||||
if 400 <= result.status_code < 500:
|
||||
raise ValidationError(f"Source returned {result.status_code}")
|
||||
if result.status_code >= 300:
|
||||
raise TransientError(f"Source server error: {result.status_code}")
|
||||
|
||||
return result.text, result.headers.get('Content-Type', ''), result.url
|
||||
|
||||
|
||||
def verify_link_presence(content: str, target_url: str, content_type: str) -> bool:
|
||||
def _strip_fragment(url: str) -> str:
|
||||
return urldefrag(url.strip())[0]
|
||||
|
||||
|
||||
def _json_strings(value):
|
||||
"""Yield every string in a parsed JSON document"""
|
||||
if isinstance(value, str):
|
||||
yield value
|
||||
elif isinstance(value, dict):
|
||||
for item in value.values():
|
||||
yield from _json_strings(item)
|
||||
elif isinstance(value, list):
|
||||
for item in value:
|
||||
yield from _json_strings(item)
|
||||
|
||||
|
||||
def verify_link_presence(content: str, target_url: str, content_type: str,
|
||||
source_url: str = '') -> bool:
|
||||
"""
|
||||
Verify that source content contains a link to target URL.
|
||||
Verify the source links to the target (spec 3.2.2).
|
||||
|
||||
The source MUST have an exact match of the target URL, checked per media
|
||||
type: in HTML, any link-bearing attribute (a/link href, img/video/audio
|
||||
src, ...) resolved against the source URL; in JSON, any string value; in
|
||||
other text, the URL anywhere. Fragments are ignored on both sides.
|
||||
|
||||
Args:
|
||||
content: Source content (HTML or text)
|
||||
content: Source content
|
||||
target_url: Target URL to find
|
||||
content_type: Content type of source
|
||||
source_url: Final source URL, for resolving relative links
|
||||
|
||||
Returns:
|
||||
True if target link found, False otherwise
|
||||
True if the target is linked, False otherwise
|
||||
"""
|
||||
# Normalize URLs for comparison (remove fragments, trailing slashes)
|
||||
target_normalized = urlparse(target_url)._replace(fragment='').geturl().rstrip('/')
|
||||
target = _strip_fragment(target_url)
|
||||
media_type = content_type.split(';')[0].strip().lower()
|
||||
|
||||
# For HTML, parse and check all links
|
||||
if 'html' in content_type.lower():
|
||||
if 'html' in media_type:
|
||||
from lxml import html as lxml_html
|
||||
try:
|
||||
from lxml import html as lxml_html
|
||||
doc = lxml_html.fromstring(content)
|
||||
links = doc.xpath('//a/@href')
|
||||
|
||||
for link in links:
|
||||
# Make absolute if relative
|
||||
link_normalized = urlparse(link)._replace(fragment='').geturl().rstrip('/')
|
||||
if link_normalized == target_normalized:
|
||||
return True
|
||||
doc = lxml_html.document_fromstring(content)
|
||||
except Exception as e:
|
||||
logger.debug(f"HTML parsing failed, falling back to string search: {e}")
|
||||
logger.debug(f"HTML parsing failed: {e}")
|
||||
return False
|
||||
if source_url:
|
||||
doc.make_links_absolute(source_url, resolve_base_href=True,
|
||||
handle_failures='ignore')
|
||||
return any(_strip_fragment(link) == target for _, _, link, _ in doc.iterlinks())
|
||||
|
||||
# Fallback: simple string search
|
||||
return target_normalized in content
|
||||
if media_type == 'application/json' or media_type.endswith('+json'):
|
||||
try:
|
||||
data = json.loads(content)
|
||||
except ValueError:
|
||||
return False
|
||||
return any(_strip_fragment(s) == target for s in _json_strings(data))
|
||||
|
||||
if media_type.startswith('text/'):
|
||||
return target in content
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def detect_mention_type(parsed_mf2: Dict, target_url: str) -> str:
|
||||
"""
|
||||
Detect mention type from microformats.
|
||||
Detect mention type from the h-entry that responds to the target
|
||||
|
||||
Priority: reply > like > repost > bookmark > mention
|
||||
Returns:
|
||||
'reply', 'like', 'repost', 'bookmark', or 'mention'
|
||||
"""
|
||||
entry = mf2_utils.responding_entry(parsed_mf2, target_url)
|
||||
return mf2_utils.response_type(entry, target_url) if entry else 'mention'
|
||||
|
||||
|
||||
def _is_http_url(value: Optional[str]) -> bool:
|
||||
"""True if value is an absolute http or https URL"""
|
||||
if not value:
|
||||
return False
|
||||
parsed = urlparse(value.strip())
|
||||
return parsed.scheme.lower() in ('http', 'https') and bool(parsed.netloc)
|
||||
|
||||
|
||||
def extract_author(parsed_mf2: Dict, source_url: str,
|
||||
entry: Optional[Dict] = None) -> Dict[str, Optional[str]]:
|
||||
"""
|
||||
Author of the source's h-entry (authorship algorithm, without fetching)
|
||||
|
||||
Args:
|
||||
parsed_mf2: Parsed microformats data from mf2py
|
||||
target_url: Target URL being mentioned
|
||||
source_url: Source URL (fallback for author URL and name)
|
||||
entry: The h-entry to attribute (default: first top-level h-entry)
|
||||
|
||||
Returns:
|
||||
Mention type: 'reply', 'like', 'repost', 'bookmark', or 'mention'
|
||||
Dict with author_name, author_url, author_photo; only http(s) URLs
|
||||
are kept, so they are safe to render as links (XSS)
|
||||
"""
|
||||
target_normalized = urlparse(target_url)._replace(fragment='').geturl().rstrip('/')
|
||||
if entry is None:
|
||||
entry = mf2_utils.responding_entry(parsed_mf2, '')
|
||||
author = mf2_utils.entry_author(parsed_mf2, entry, source_url)
|
||||
|
||||
# Find h-entry that references target
|
||||
for item in parsed_mf2.get('items', []):
|
||||
if 'h-entry' not in item.get('type', []):
|
||||
continue
|
||||
|
||||
props = item.get('properties', {})
|
||||
|
||||
# Check in-reply-to (highest priority)
|
||||
for reply_to in props.get('in-reply-to', []):
|
||||
url = reply_to if isinstance(reply_to, str) else reply_to.get('value', '')
|
||||
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
|
||||
return 'reply'
|
||||
|
||||
# Check like-of
|
||||
for like in props.get('like-of', []):
|
||||
url = like if isinstance(like, str) else like.get('value', '')
|
||||
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
|
||||
return 'like'
|
||||
|
||||
# Check repost-of
|
||||
for repost in props.get('repost-of', []):
|
||||
url = repost if isinstance(repost, str) else repost.get('value', '')
|
||||
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
|
||||
return 'repost'
|
||||
|
||||
# Check bookmark-of
|
||||
for bookmark in props.get('bookmark-of', []):
|
||||
url = bookmark if isinstance(bookmark, str) else bookmark.get('value', '')
|
||||
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
|
||||
return 'bookmark'
|
||||
|
||||
# Default fallback
|
||||
return 'mention'
|
||||
|
||||
|
||||
def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]]:
|
||||
"""
|
||||
Extract author information from h-card.
|
||||
|
||||
Args:
|
||||
parsed_mf2: Parsed microformats data from mf2py
|
||||
source_url: Source URL (fallback for author URL)
|
||||
|
||||
Returns:
|
||||
Dict with author_name, author_url, author_photo
|
||||
"""
|
||||
author_name = None
|
||||
author_url = None
|
||||
author_photo = None
|
||||
|
||||
# Find h-entry first
|
||||
entry = None
|
||||
for item in parsed_mf2.get('items', []):
|
||||
if 'h-entry' in item.get('type', []):
|
||||
entry = item
|
||||
break
|
||||
|
||||
# Check for author property in h-entry
|
||||
if entry:
|
||||
author_prop = entry.get('properties', {}).get('author', [])
|
||||
if author_prop:
|
||||
author_data = author_prop[0]
|
||||
if isinstance(author_data, dict):
|
||||
# h-card object
|
||||
props = author_data.get('properties', {})
|
||||
author_name = props.get('name', [None])[0]
|
||||
author_url = props.get('url', [None])[0]
|
||||
author_photo = props.get('photo', [None])[0]
|
||||
|
||||
# If no author in h-entry, find representative h-card
|
||||
if not author_name:
|
||||
for item in parsed_mf2.get('items', []):
|
||||
if 'h-card' in item.get('type', []):
|
||||
props = item.get('properties', {})
|
||||
author_name = props.get('name', [None])[0]
|
||||
author_url = props.get('url', [None])[0]
|
||||
author_photo = props.get('photo', [None])[0]
|
||||
break
|
||||
|
||||
# Fallback to hostname if no author found
|
||||
if not author_name:
|
||||
author_name = urlparse(source_url).netloc
|
||||
|
||||
if not author_url:
|
||||
author_url = source_url
|
||||
author_url = author['url'] if _is_http_url(author['url']) else source_url
|
||||
author_photo = author['photo'] if _is_http_url(author['photo']) else None
|
||||
author_name = author['name'] or urlparse(author_url).netloc
|
||||
|
||||
return {
|
||||
'author_name': author_name,
|
||||
@@ -240,53 +194,65 @@ def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]
|
||||
}
|
||||
|
||||
|
||||
def extract_content(parsed_mf2: Dict) -> Optional[str]:
|
||||
def extract_content(parsed_mf2: Dict, entry: Optional[Dict] = None) -> Optional[str]:
|
||||
"""
|
||||
Extract content snippet from h-entry.
|
||||
Content snippet of an h-entry (e-content, else p-summary)
|
||||
|
||||
Args:
|
||||
parsed_mf2: Parsed microformats data from mf2py
|
||||
entry: The h-entry (default: first top-level h-entry)
|
||||
|
||||
Returns:
|
||||
Sanitized content snippet (max 500 chars) or None
|
||||
"""
|
||||
# Find h-entry
|
||||
for item in parsed_mf2.get('items', []):
|
||||
if 'h-entry' not in item.get('type', []):
|
||||
continue
|
||||
if entry is None:
|
||||
entry = mf2_utils.responding_entry(parsed_mf2, '')
|
||||
if not entry:
|
||||
return None
|
||||
|
||||
props = item.get('properties', {})
|
||||
props = entry.get('properties', {})
|
||||
for content in props.get('content', []):
|
||||
html = (content.get('html') or content.get('value')) if isinstance(content, dict) else content
|
||||
if html:
|
||||
return _snippet(html)
|
||||
|
||||
# Try e-content first (HTML)
|
||||
content_prop = props.get('content', [])
|
||||
if content_prop:
|
||||
content_data = content_prop[0]
|
||||
if isinstance(content_data, dict):
|
||||
content = content_data.get('html', '') or content_data.get('value', '')
|
||||
else:
|
||||
content = content_data
|
||||
summary = mf2_utils.first(entry, 'summary')
|
||||
return _snippet(summary) if summary else None
|
||||
|
||||
if content:
|
||||
# Sanitize HTML
|
||||
sanitized = sanitize_content(content)
|
||||
# Truncate to 500 chars
|
||||
if len(sanitized) > 500:
|
||||
sanitized = sanitized[:497] + '...'
|
||||
return sanitized
|
||||
|
||||
# Try p-summary as fallback
|
||||
summary = props.get('summary', [])
|
||||
if summary:
|
||||
content = summary[0]
|
||||
if isinstance(content, dict):
|
||||
content = content.get('value', '')
|
||||
if content:
|
||||
sanitized = sanitize_content(content)
|
||||
if len(sanitized) > 500:
|
||||
sanitized = sanitized[:497] + '...'
|
||||
return sanitized
|
||||
def extract_source_details(entry: Optional[Dict]) -> Dict[str, Optional[str]]:
|
||||
"""
|
||||
The source post's own URL, uid and published time
|
||||
|
||||
return None
|
||||
Returns:
|
||||
source_post_url, source_uid (http(s) only) and source_published
|
||||
(RFC 3339 UTC, or None if missing or unparseable)
|
||||
"""
|
||||
if not entry:
|
||||
return {'source_post_url': None, 'source_uid': None, 'source_published': None}
|
||||
|
||||
url = mf2_utils.first(entry, 'url')
|
||||
uid = mf2_utils.first(entry, 'uid')
|
||||
published = mf2_utils.first(entry, 'published')
|
||||
try:
|
||||
published = iso_utc(published) or None
|
||||
except (ValueError, TypeError):
|
||||
published = None
|
||||
|
||||
return {
|
||||
'source_post_url': url if _is_http_url(url) else None,
|
||||
'source_uid': uid,
|
||||
'source_published': published,
|
||||
}
|
||||
|
||||
|
||||
def _snippet(html: str, limit: int = 500) -> str:
|
||||
"""Sanitized HTML of at most ~limit characters, with tags closed"""
|
||||
sanitized = sanitize_content(html)
|
||||
if len(sanitized) > limit:
|
||||
# Cutting can leave a tag or attribute open; sanitizing again closes it
|
||||
sanitized = sanitize_content(sanitized[:limit - 3] + '...')
|
||||
return sanitized
|
||||
|
||||
|
||||
def sanitize_content(html: str) -> str:
|
||||
@@ -337,17 +303,17 @@ def process_webmention(source_url: str, target_url: str, site_url: str) -> Dict[
|
||||
ValidationError: If validation fails with specific error
|
||||
"""
|
||||
# Fetch source content
|
||||
content, content_type = fetch_source(source_url, site_url)
|
||||
content, content_type, final_url = fetch_source(source_url, site_url)
|
||||
|
||||
# Verify target link presence
|
||||
if not verify_link_presence(content, target_url, content_type):
|
||||
if not verify_link_presence(content, target_url, content_type, final_url):
|
||||
raise ValidationError("Source does not link to target")
|
||||
|
||||
# Parse microformats (only for HTML)
|
||||
parsed_mf2 = None
|
||||
if 'html' in content_type.lower():
|
||||
try:
|
||||
parsed_mf2 = mf2py.parse(doc=content, url=source_url)
|
||||
parsed_mf2 = mf2py.parse(doc=content, url=final_url)
|
||||
except Exception as e:
|
||||
logger.debug(f"Microformat parsing failed for {source_url}: {e}")
|
||||
|
||||
@@ -360,14 +326,19 @@ def process_webmention(source_url: str, target_url: str, site_url: str) -> Dict[
|
||||
}
|
||||
content_snippet = None
|
||||
|
||||
source_details = extract_source_details(None)
|
||||
|
||||
if parsed_mf2:
|
||||
mention_type = detect_mention_type(parsed_mf2, target_url)
|
||||
author_info = extract_author(parsed_mf2, source_url)
|
||||
content_snippet = extract_content(parsed_mf2)
|
||||
entry = mf2_utils.responding_entry(parsed_mf2, target_url)
|
||||
mention_type = mf2_utils.response_type(entry, target_url) if entry else 'mention'
|
||||
author_info = extract_author(parsed_mf2, final_url, entry)
|
||||
content_snippet = extract_content(parsed_mf2, entry)
|
||||
source_details = extract_source_details(entry)
|
||||
|
||||
return {
|
||||
'valid': True,
|
||||
'mention_type': mention_type,
|
||||
**author_info,
|
||||
'content_snippet': content_snippet
|
||||
'content_snippet': content_snippet,
|
||||
**source_details,
|
||||
}
|
||||
@@ -17,7 +17,10 @@ Functions:
|
||||
import logging
|
||||
import math
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
from flask import current_app
|
||||
from starpunk.database import get_db
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -71,9 +74,13 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
|
||||
|
||||
for row in rows:
|
||||
wm = dict(row)
|
||||
# Parse received_at timestamp
|
||||
if isinstance(wm.get('received_at'), str):
|
||||
wm['received_at'] = datetime.fromisoformat(wm['received_at'].replace('Z', '+00:00'))
|
||||
# Parse timestamps (source_published may be missing or malformed)
|
||||
for key in ('received_at', 'source_published'):
|
||||
if isinstance(wm.get(key), str):
|
||||
try:
|
||||
wm[key] = datetime.fromisoformat(wm[key].replace('Z', '+00:00'))
|
||||
except ValueError:
|
||||
wm[key] = None
|
||||
|
||||
mention_type = wm.get('mention_type', 'mention')
|
||||
if mention_type == 'reply':
|
||||
@@ -233,10 +240,19 @@ def update_webmention_status(webmention_ids: List[int], status: str) -> int:
|
||||
WHERE id IN ({placeholders})
|
||||
"""
|
||||
|
||||
# Notes whose displayed mentions change (valid ones switching between
|
||||
# approved and not)
|
||||
changed_notes = _visibility_changes(
|
||||
db, webmention_ids,
|
||||
"validation_status = 'valid' AND (moderation_status = 'approved') != (? = 'approved')",
|
||||
[status],
|
||||
)
|
||||
|
||||
params = [status] + webmention_ids
|
||||
cursor = db.execute(query, params)
|
||||
db.commit()
|
||||
|
||||
_notify_responses_changed(changed_notes)
|
||||
return cursor.rowcount
|
||||
|
||||
|
||||
@@ -255,15 +271,42 @@ def delete_webmentions(webmention_ids: List[int]) -> int:
|
||||
|
||||
db = get_db()
|
||||
|
||||
changed_notes = _visibility_changes(
|
||||
db, webmention_ids, "validation_status = 'valid' AND moderation_status = 'approved'", [],
|
||||
)
|
||||
|
||||
placeholders = ','.join('?' * len(webmention_ids))
|
||||
query = f"DELETE FROM webmentions WHERE id IN ({placeholders})"
|
||||
|
||||
cursor = db.execute(query, webmention_ids)
|
||||
db.commit()
|
||||
|
||||
_notify_responses_changed(changed_notes)
|
||||
return cursor.rowcount
|
||||
|
||||
|
||||
def _visibility_changes(db, webmention_ids: List[int], condition: str, params: list) -> List[int]:
|
||||
"""Target note IDs of the given webmentions that match condition"""
|
||||
placeholders = ','.join('?' * len(webmention_ids))
|
||||
rows = db.execute(
|
||||
f"SELECT DISTINCT target_note_id FROM webmentions "
|
||||
f"WHERE id IN ({placeholders}) AND {condition}",
|
||||
list(webmention_ids) + params,
|
||||
).fetchall()
|
||||
return [row[0] for row in rows]
|
||||
|
||||
|
||||
def _notify_responses_changed(note_ids: List[int]) -> None:
|
||||
"""Re-send webmentions for notes whose shown responses changed (best effort)"""
|
||||
if not note_ids:
|
||||
return
|
||||
try:
|
||||
from starpunk.webmention_sender import note_responses_changed
|
||||
note_responses_changed(Path(current_app.config['DATABASE_PATH']), note_ids)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to re-send webmentions after moderation: {e}")
|
||||
|
||||
|
||||
def get_reply_context(url: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get cached reply context for a URL.
|
||||
|
||||
+111
-61
@@ -14,9 +14,8 @@ References:
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Set
|
||||
from typing import Optional
|
||||
from lxml import html as lxml_html
|
||||
from urllib.parse import urlparse
|
||||
|
||||
@@ -45,16 +44,16 @@ def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True)
|
||||
return []
|
||||
|
||||
try:
|
||||
# Parse HTML
|
||||
doc = lxml_html.fromstring(html_content)
|
||||
# Parse HTML inside a wrapper, so a top-level <pre> can be removed too
|
||||
doc = lxml_html.fragment_fromstring(html_content, create_parent='div')
|
||||
|
||||
if exclude_code_blocks:
|
||||
# Remove <pre> and <code> elements before extracting links
|
||||
for element in doc.xpath('//pre | //code'):
|
||||
element.getparent().remove(element)
|
||||
for element in doc.xpath('.//pre | .//code'):
|
||||
element.drop_tree()
|
||||
|
||||
# Extract all <a href> attributes in document order
|
||||
links = doc.xpath('//a/@href')
|
||||
links = doc.xpath('.//a/@href')
|
||||
|
||||
# Filter to valid absolute URLs (http/https)
|
||||
absolute_links = []
|
||||
@@ -70,6 +69,97 @@ def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True)
|
||||
return []
|
||||
|
||||
|
||||
def _dedupe(urls: list) -> list:
|
||||
"""Drop empty and repeated URLs, keeping the first occurrence"""
|
||||
seen = set()
|
||||
result = []
|
||||
for url in urls:
|
||||
if url and url not in seen:
|
||||
seen.add(url)
|
||||
result.append(url)
|
||||
return result
|
||||
|
||||
|
||||
def previous_targets(db_path: Path, note_id: int) -> list[str]:
|
||||
"""Targets this note has been queued to send webmentions to before"""
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT target_url FROM webmention_queue WHERE source_note_id = ? ORDER BY id",
|
||||
(note_id,),
|
||||
).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
return [row[0] for row in rows]
|
||||
|
||||
|
||||
def requeue_previous_webmentions(db_path: Path, note_id: int, detach: bool = False) -> int:
|
||||
"""
|
||||
Re-send every webmention this note has sent (spec 3.1.5 deleted posts)
|
||||
|
||||
Args:
|
||||
db_path: Path to SQLite database
|
||||
note_id: Note being deleted or restored
|
||||
detach: Clear source_note_id so the rows survive a hard delete
|
||||
(the foreign key cascades)
|
||||
|
||||
Returns:
|
||||
Number of webmentions queued
|
||||
"""
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
cursor = conn.execute(f"""
|
||||
UPDATE webmention_queue
|
||||
SET status = 'pending',
|
||||
retry_count = 0,
|
||||
next_retry_at = NULL,
|
||||
last_error = NULL,
|
||||
endpoint_url = NULL,
|
||||
created_at = CURRENT_TIMESTAMP
|
||||
{', source_note_id = NULL' if detach else ''}
|
||||
WHERE source_note_id = ?
|
||||
""", (note_id,))
|
||||
conn.commit()
|
||||
queued = cursor.rowcount
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
if queued:
|
||||
logger.info(f"Re-queued {queued} webmentions for note {note_id}")
|
||||
return queued
|
||||
|
||||
|
||||
def note_responses_changed(db_path: Path, note_ids) -> int:
|
||||
"""
|
||||
Re-send a note's webmentions because the responses it shows changed
|
||||
|
||||
Spec 3.1.4: if the source post displays responses (e.g. comments), a
|
||||
change to them is an update of the post. Only published, undeleted notes.
|
||||
|
||||
Args:
|
||||
db_path: Path to SQLite database
|
||||
note_ids: IDs of notes whose displayed responses changed
|
||||
|
||||
Returns:
|
||||
Number of webmentions queued
|
||||
"""
|
||||
ids = sorted({i for i in note_ids if i is not None})
|
||||
if not ids:
|
||||
return 0
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
placeholders = ','.join('?' * len(ids))
|
||||
live = [row[0] for row in conn.execute(
|
||||
f"SELECT id FROM notes WHERE id IN ({placeholders}) "
|
||||
"AND published = 1 AND deleted_at IS NULL", ids,
|
||||
)]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
return sum(requeue_previous_webmentions(db_path, note_id) for note_id in live)
|
||||
|
||||
|
||||
def queue_webmentions_for_note(
|
||||
db_path: Path,
|
||||
note_id: int,
|
||||
@@ -83,10 +173,9 @@ def queue_webmentions_for_note(
|
||||
|
||||
Per architect's answers (Phase 6):
|
||||
- Called from notes.py AFTER successful database commit
|
||||
- For updates: diff old vs new URLs, queue for new/existing/removed
|
||||
- For creates: queue for all URLs in content
|
||||
- Max 10 webmentions per operation
|
||||
- Priority: in_reply_to URL first, then document order
|
||||
- For creates: reply target first, then links in document order (max 10)
|
||||
- For updates: the same, plus every previously sent target and links
|
||||
removed by the edit (spec 3.1.4)
|
||||
- Webmention sending is best-effort; failures do not affect note creation
|
||||
|
||||
Args:
|
||||
@@ -100,58 +189,19 @@ def queue_webmentions_for_note(
|
||||
Returns:
|
||||
Number of webmentions queued
|
||||
"""
|
||||
# Phase 6: Implement URL diffing for updates
|
||||
# Targets for the current content: reply target first, then links in
|
||||
# document order, at most 10
|
||||
links = extract_links_from_html(content_html) if content_html else []
|
||||
current = _dedupe(([in_reply_to] if in_reply_to else []) + links)[:10]
|
||||
|
||||
if old_content_html is not None:
|
||||
# This is an update - compute URL diff
|
||||
old_links = set(extract_links_from_html(old_content_html))
|
||||
|
||||
# Handle unpublishing case (content_html is None)
|
||||
if content_html is None:
|
||||
# Unpublishing - queue for all old URLs (they're all removed)
|
||||
new_links = set()
|
||||
else:
|
||||
new_links = set(extract_links_from_html(content_html))
|
||||
|
||||
# Compute sets for URL diffing
|
||||
new_urls = new_links - old_links # In new, not in old
|
||||
existing_urls = new_links & old_links # In both (re-validate)
|
||||
removed_urls = old_links - new_links # In old, not in new
|
||||
|
||||
# Build target list: new + existing + removed
|
||||
# (All need webmentions per W3C spec)
|
||||
all_update_urls = list(new_urls) + list(existing_urls) + list(removed_urls)
|
||||
|
||||
# Add in_reply_to if changed
|
||||
target_urls = []
|
||||
if in_reply_to and in_reply_to not in all_update_urls:
|
||||
target_urls.append(in_reply_to)
|
||||
|
||||
# Add the diffed URLs
|
||||
target_urls.extend(all_update_urls)
|
||||
|
||||
# Limit to 10
|
||||
target_urls = target_urls[:10]
|
||||
|
||||
# Update (spec 3.1.4): also re-send to every target notified before,
|
||||
# and to links the edit removed, so they can update or remove the
|
||||
# mention. These are not capped.
|
||||
removed = [u for u in extract_links_from_html(old_content_html) if u not in links]
|
||||
target_urls = _dedupe(current + previous_targets(db_path, note_id) + removed)
|
||||
else:
|
||||
# This is a create (no old content) - use original logic
|
||||
links = extract_links_from_html(content_html) if content_html else []
|
||||
|
||||
# Build priority-ordered list of target URLs
|
||||
target_urls = []
|
||||
|
||||
# 1. in_reply_to gets highest priority
|
||||
if in_reply_to:
|
||||
target_urls.append(in_reply_to)
|
||||
|
||||
# 2. Add extracted links in document order (deduplicated)
|
||||
seen = {in_reply_to} if in_reply_to else set()
|
||||
for link in links:
|
||||
if link not in seen:
|
||||
target_urls.append(link)
|
||||
seen.add(link)
|
||||
|
||||
# 3. Limit to 10 webmentions per operation
|
||||
target_urls = target_urls[:10]
|
||||
target_urls = current
|
||||
|
||||
if not target_urls:
|
||||
logger.debug(f"No links to send webmentions for note {note_id}")
|
||||
|
||||
+188
-52
@@ -1,113 +1,249 @@
|
||||
"""
|
||||
SSRF protection utilities for webmention processing
|
||||
SSRF protection for webmention fetches
|
||||
|
||||
Per ADR-063 Decision 9 and background-worker.md:
|
||||
- Block private IP address ranges
|
||||
- Resolve hostname before making requests
|
||||
Per ADR-063 Decision 9 and Webmention sections 4.2, 4.3 and 4.5:
|
||||
- Every address a hostname resolves to must be public
|
||||
- Redirects are followed by hand so each hop is checked
|
||||
- Each request connects to the address that was checked (no second DNS
|
||||
lookup that a rebinding DNS server could answer differently)
|
||||
- Redirect count, response size and total time are limited
|
||||
- DEV_MODE exception for localhost testing
|
||||
- Applied to all outgoing HTTP requests
|
||||
|
||||
References:
|
||||
- ADR-063: Webmention Implementation
|
||||
- Background Worker Design: /docs/design/v1.6.0/background-worker.md
|
||||
- W3C Webmention: https://www.w3.org/TR/webmention/#security-considerations
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import httpx
|
||||
from flask import current_app
|
||||
|
||||
MAX_REDIRECTS = 10
|
||||
MAX_BYTES = 1024 * 1024
|
||||
TIMEOUT = 10
|
||||
|
||||
# Tests can set an httpx transport here (e.g. httpx.MockTransport)
|
||||
TRANSPORT = None
|
||||
|
||||
CGNAT = ipaddress.ip_network('100.64.0.0/10')
|
||||
THIS_NETWORK = ipaddress.ip_network('0.0.0.0/8')
|
||||
|
||||
|
||||
class SSRFBlockedError(Exception):
|
||||
"""Raised when SSRF protection blocks a request"""
|
||||
pass
|
||||
|
||||
|
||||
# Private IP ranges per RFC 1918 and link-local ranges
|
||||
BLOCKED_NETWORKS = [
|
||||
ipaddress.ip_network('10.0.0.0/8'), # RFC 1918 private
|
||||
ipaddress.ip_network('172.16.0.0/12'), # RFC 1918 private
|
||||
ipaddress.ip_network('192.168.0.0/16'), # RFC 1918 private
|
||||
ipaddress.ip_network('127.0.0.0/8'), # Localhost
|
||||
ipaddress.ip_network('169.254.0.0/16'), # Link-local
|
||||
ipaddress.ip_network('::1/128'), # IPv6 localhost
|
||||
ipaddress.ip_network('fc00::/7'), # IPv6 unique local
|
||||
ipaddress.ip_network('fe80::/10'), # IPv6 link-local
|
||||
]
|
||||
class FetchTooLargeError(Exception):
|
||||
"""Raised when a response is larger than the limit"""
|
||||
pass
|
||||
|
||||
|
||||
def user_agent(site_url: str) -> str:
|
||||
"""User-Agent for webmention requests (spec 3.1.2 suggests including 'Webmention')"""
|
||||
from starpunk import __version__
|
||||
return f"StarPunk/{__version__} Webmention (+{site_url})"
|
||||
|
||||
|
||||
def is_private_ip(ip: str) -> bool:
|
||||
"""
|
||||
Check if IP address is in a private/blocked range.
|
||||
Check if an IP address must not be fetched.
|
||||
|
||||
Blocks private, loopback, link-local, multicast, reserved, unspecified,
|
||||
carrier-grade NAT and IPv4-mapped forms of these.
|
||||
|
||||
Args:
|
||||
ip: IP address string (IPv4 or IPv6)
|
||||
|
||||
Returns:
|
||||
True if IP is private/blocked, False otherwise
|
||||
True if IP is blocked, False otherwise
|
||||
"""
|
||||
try:
|
||||
addr = ipaddress.ip_address(ip)
|
||||
return any(addr in network for network in BLOCKED_NETWORKS)
|
||||
addr = ipaddress.ip_address(ip.split('%')[0])
|
||||
except ValueError:
|
||||
# Invalid IP format, treat as blocked
|
||||
return True
|
||||
|
||||
if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped:
|
||||
addr = addr.ipv4_mapped
|
||||
|
||||
if isinstance(addr, ipaddress.IPv4Address) and (addr in CGNAT or addr in THIS_NETWORK):
|
||||
return True
|
||||
|
||||
return (
|
||||
addr.is_private
|
||||
or addr.is_loopback
|
||||
or addr.is_link_local
|
||||
or addr.is_multicast
|
||||
or addr.is_reserved
|
||||
or addr.is_unspecified
|
||||
)
|
||||
|
||||
|
||||
def safe_resolve(hostname: str) -> str:
|
||||
"""
|
||||
Resolve hostname and validate IP is not private.
|
||||
Resolve hostname and check every address is public.
|
||||
|
||||
Per background-worker.md: DEV_MODE exception for localhost testing.
|
||||
|
||||
Args:
|
||||
hostname: Hostname to resolve
|
||||
|
||||
Returns:
|
||||
Resolved IP address if safe
|
||||
Address to connect to (IPv4 preferred)
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError: If hostname resolves to private IP
|
||||
SSRFBlockedError: If any address is blocked
|
||||
socket.gaierror: If hostname cannot be resolved
|
||||
"""
|
||||
# DEV_MODE exception for localhost
|
||||
if current_app.config.get('DEV_MODE') and hostname in ('localhost', '127.0.0.1', '::1'):
|
||||
return '127.0.0.1'
|
||||
|
||||
# Resolve hostname
|
||||
ip = socket.gethostbyname(hostname)
|
||||
addresses = sorted({info[4][0] for info in socket.getaddrinfo(hostname, None)},
|
||||
key=lambda ip: (':' in ip, ip))
|
||||
for ip in addresses:
|
||||
if is_private_ip(ip):
|
||||
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
|
||||
|
||||
# Check if resolved IP is private
|
||||
if is_private_ip(ip):
|
||||
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
|
||||
|
||||
return ip
|
||||
return addresses[0]
|
||||
|
||||
|
||||
def validate_url_for_fetch(url: str) -> None:
|
||||
def validate_url_for_fetch(url: str) -> str:
|
||||
"""
|
||||
Validate URL is safe to fetch (not private IP).
|
||||
Validate URL is safe to fetch (http(s), public addresses only).
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
Returns:
|
||||
The checked IP address to connect to
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError: If URL points to private IP
|
||||
ValueError: If URL is malformed
|
||||
SSRFBlockedError: If URL points to a blocked address
|
||||
ValueError: If URL is malformed or its host does not resolve
|
||||
"""
|
||||
from urllib.parse import urlparse
|
||||
|
||||
parsed = urlparse(url)
|
||||
|
||||
if not parsed.scheme in ('http', 'https'):
|
||||
if parsed.scheme not in ('http', 'https'):
|
||||
raise ValueError(f"Unsupported URL scheme: {parsed.scheme}")
|
||||
|
||||
if not parsed.netloc:
|
||||
if not parsed.hostname:
|
||||
raise ValueError("URL must include a hostname")
|
||||
|
||||
# Extract hostname (remove port if present)
|
||||
hostname = parsed.hostname
|
||||
if not hostname:
|
||||
raise ValueError("Could not extract hostname from URL")
|
||||
try:
|
||||
return safe_resolve(parsed.hostname)
|
||||
except socket.gaierror as e:
|
||||
raise ValueError(f"Could not resolve {parsed.hostname}: {e}")
|
||||
|
||||
# Validate hostname resolves to public IP
|
||||
safe_resolve(hostname)
|
||||
|
||||
def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
||||
**kwargs) -> httpx.Request:
|
||||
"""
|
||||
Build a request that connects to the checked address for url's host.
|
||||
|
||||
The URL's host is replaced by the IP that passed the SSRF check; the
|
||||
real host goes in the Host header and, for https, in SNI, so virtual
|
||||
hosting and certificate verification still use the hostname.
|
||||
"""
|
||||
ip = validate_url_for_fetch(url)
|
||||
parsed = urlparse(url)
|
||||
ip_host = f"[{ip}]" if ':' in ip else ip
|
||||
port = f":{parsed.port}" if parsed.port else ''
|
||||
pinned_url = parsed._replace(netloc=f"{ip_host}{port}").geturl()
|
||||
host_header = f"{parsed.hostname}{port}"
|
||||
|
||||
extensions = {'sni_hostname': parsed.hostname} if parsed.scheme == 'https' else {}
|
||||
return client.build_request(
|
||||
method, pinned_url, headers={**headers, 'Host': host_header},
|
||||
extensions=extensions, **kwargs,
|
||||
)
|
||||
|
||||
|
||||
def safe_post(url: str, data: dict, headers: dict, timeout: float = TIMEOUT) -> httpx.Response:
|
||||
"""
|
||||
POST form data to a checked, pinned address. Redirects are not followed.
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL is not allowed
|
||||
httpx.HTTPError: Network errors
|
||||
"""
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
||||
response = client.send(_pinned_request(client, 'POST', url, headers, data=data))
|
||||
response.read()
|
||||
return response
|
||||
|
||||
|
||||
@dataclass
|
||||
class FetchResult:
|
||||
"""A fetched response, after redirects"""
|
||||
url: str
|
||||
status_code: int
|
||||
headers: httpx.Headers
|
||||
content: bytes
|
||||
encoding: str
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
try:
|
||||
return self.content.decode(self.encoding, errors='replace')
|
||||
except LookupError:
|
||||
return self.content.decode('utf-8', errors='replace')
|
||||
|
||||
|
||||
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
|
||||
"""
|
||||
GET a URL, following redirects by hand and checking each hop.
|
||||
|
||||
Args:
|
||||
url: URL to fetch
|
||||
headers: Request headers
|
||||
max_bytes: Response body limit
|
||||
max_redirects: Redirect limit
|
||||
timeout: Total time limit in seconds
|
||||
|
||||
Returns:
|
||||
FetchResult for the final response (any status code)
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL or a redirect target is not allowed
|
||||
FetchTooLargeError: Body exceeds max_bytes
|
||||
httpx.HTTPError: Network errors, too many redirects, time limit
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise httpx.TimeoutException("Fetch exceeded time limit")
|
||||
|
||||
request = _pinned_request(client, 'GET', url, headers, timeout=remaining)
|
||||
response = client.send(request, stream=True)
|
||||
try:
|
||||
location = response.headers.get('Location')
|
||||
if response.is_redirect and location:
|
||||
url = urljoin(url, location)
|
||||
continue
|
||||
|
||||
content_length = response.headers.get('Content-Length', '')
|
||||
if content_length.isdigit() and int(content_length) > max_bytes:
|
||||
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
|
||||
|
||||
content = b''
|
||||
for chunk in response.iter_bytes(chunk_size=8192):
|
||||
content += chunk
|
||||
if len(content) > max_bytes:
|
||||
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
|
||||
if time.monotonic() > deadline:
|
||||
raise httpx.TimeoutException("Fetch exceeded time limit")
|
||||
|
||||
return FetchResult(
|
||||
url=url,
|
||||
status_code=response.status_code,
|
||||
headers=response.headers,
|
||||
content=content,
|
||||
encoding=response.charset_encoding or 'utf-8',
|
||||
)
|
||||
finally:
|
||||
response.close()
|
||||
|
||||
raise httpx.TooManyRedirects(f"More than {max_redirects} redirects")
|
||||
+240
-146
@@ -12,29 +12,34 @@ References:
|
||||
- ADR-063: Webmention Implementation
|
||||
"""
|
||||
|
||||
import fcntl
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
import sqlite3
|
||||
import httpx
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Optional, TypedDict
|
||||
from urllib.parse import urlparse
|
||||
from typing import Dict, Any, Optional
|
||||
from urllib.parse import urljoin
|
||||
from lxml import html as lxml_html
|
||||
|
||||
from starpunk.webmention_processor import process_webmention, ValidationError
|
||||
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
|
||||
from starpunk.webmention_processor import process_webmention, TransientError, ValidationError
|
||||
from starpunk.webmention_sender import note_responses_changed
|
||||
from starpunk.mf2_utils import normalize_url
|
||||
from starpunk.webmention_ssrf import (
|
||||
FetchTooLargeError, SSRFBlockedError, safe_get, safe_post, user_agent,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Link header (RFC 8288): <uri> then ;-separated params; quoted values may
|
||||
# contain , and ;
|
||||
LINK_RE = re.compile(r'<([^>]*)>((?:\s*;\s*[^;,"=\s]+\s*(?:=\s*(?:"[^"]*"|[^;,]*))?)*)')
|
||||
LINK_PARAM_RE = re.compile(r';\s*([^;,"=\s]+)\s*(?:=\s*(?:"([^"]*)"|([^;,]*)))?')
|
||||
|
||||
# Discovery cache entry type
|
||||
class DiscoveryCacheEntry(TypedDict):
|
||||
"""Cache entry for endpoint discovery results"""
|
||||
endpoint: Optional[str] # None = negative cache (no endpoint found)
|
||||
expires_at: datetime
|
||||
|
||||
|
||||
class WebmentionWorker(threading.Thread):
|
||||
@@ -76,10 +81,7 @@ class WebmentionWorker(threading.Thread):
|
||||
self.auto_approve = auto_approve
|
||||
self._stop_event = threading.Event()
|
||||
self._last_processed = None
|
||||
|
||||
# Phase 3: Discovery cache (worker instance variable)
|
||||
# Per architect's answer to Q4
|
||||
self._discovery_cache: Dict[str, DiscoveryCacheEntry] = {}
|
||||
self._lock_file = None
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
@@ -94,11 +96,14 @@ class WebmentionWorker(threading.Thread):
|
||||
try:
|
||||
logger.info("Webmention worker started")
|
||||
|
||||
# Cleanup orphaned items from previous crashes
|
||||
self._cleanup_orphaned_items()
|
||||
|
||||
# Main processing loop
|
||||
while not self._stop_event.is_set():
|
||||
# Every gunicorn process starts a worker; only the one holding
|
||||
# the lock processes queues, so nothing is sent twice
|
||||
if not self._acquire_lock():
|
||||
self._stop_event.wait(self.poll_interval)
|
||||
continue
|
||||
|
||||
try:
|
||||
processed = 0
|
||||
|
||||
@@ -124,8 +129,40 @@ class WebmentionWorker(threading.Thread):
|
||||
except Exception as e:
|
||||
logger.error(f"Webmention worker thread failed: {e}", exc_info=True)
|
||||
finally:
|
||||
self._release_lock()
|
||||
logger.info("Webmention worker stopped")
|
||||
|
||||
def _acquire_lock(self) -> bool:
|
||||
"""
|
||||
Take the cross-process worker lock (non-blocking)
|
||||
|
||||
On first acquisition, also reset items left mid-send by a previous
|
||||
holder. The OS releases the lock if this process dies.
|
||||
|
||||
Returns:
|
||||
True if this worker holds the lock
|
||||
"""
|
||||
if self._lock_file:
|
||||
return True
|
||||
|
||||
lock_file = open(Path(self.db_path).parent / "webmention-worker.lock", "w")
|
||||
try:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
lock_file.close()
|
||||
return False
|
||||
|
||||
self._lock_file = lock_file
|
||||
logger.info("Webmention worker holds the queue lock")
|
||||
self._cleanup_orphaned_items()
|
||||
return True
|
||||
|
||||
def _release_lock(self):
|
||||
"""Release the cross-process worker lock if held"""
|
||||
if self._lock_file:
|
||||
self._lock_file.close()
|
||||
self._lock_file = None
|
||||
|
||||
def _cleanup_orphaned_items(self):
|
||||
"""
|
||||
Clean up orphaned 'sending' status items on startup
|
||||
@@ -209,6 +246,69 @@ class WebmentionWorker(threading.Thread):
|
||||
source_url: str,
|
||||
target_url: str,
|
||||
target_note_id: Optional[int]
|
||||
):
|
||||
"""
|
||||
Validate a webmention, then re-send the target note's webmentions if
|
||||
what the note shows changed (spec 3.1.4: displayed responses are an
|
||||
update of the post). Unchanged re-validations send nothing, so two
|
||||
sites doing this don't ping-pong.
|
||||
"""
|
||||
before = self._display_state(webmention_id)
|
||||
try:
|
||||
self._validate_and_store(webmention_id, source_url, target_url, target_note_id)
|
||||
finally:
|
||||
after = self._display_state(webmention_id)
|
||||
if before != after and target_note_id is not None:
|
||||
try:
|
||||
note_responses_changed(self.db_path, [target_note_id])
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to re-send webmentions for note {target_note_id}: {e}")
|
||||
|
||||
def _is_own_syndicated_copy(self, note_id: Optional[int], source_url: str,
|
||||
result: Dict[str, Any]) -> bool:
|
||||
"""True if the source is one of the note's own u-syndication copies"""
|
||||
if note_id is None:
|
||||
return False
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
copies = {normalize_url(row[0]) for row in conn.execute(
|
||||
"SELECT remote_url FROM syndications WHERE note_id = ? AND remote_url IS NOT NULL",
|
||||
(note_id,),
|
||||
)}
|
||||
finally:
|
||||
conn.close()
|
||||
source = {normalize_url(u) for u in (
|
||||
source_url, result.get('source_post_url'), result.get('source_uid')) if u}
|
||||
return bool(copies & source)
|
||||
|
||||
def _display_state(self, webmention_id: int) -> Optional[tuple]:
|
||||
"""
|
||||
What the note page shows for this webmention, or None if hidden
|
||||
|
||||
A row reset to 'pending' by a re-send still counts as shown if it was
|
||||
valid before (validated with no error), as it was until now.
|
||||
"""
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
try:
|
||||
row = conn.execute("""
|
||||
SELECT mention_type, author_name, author_url, author_photo, content_snippet
|
||||
FROM webmentions
|
||||
WHERE id = ?
|
||||
AND moderation_status = 'approved'
|
||||
AND (validation_status = 'valid'
|
||||
OR (validation_status = 'pending' AND validated_at IS NOT NULL
|
||||
AND validation_error IS NULL))
|
||||
""", (webmention_id,)).fetchone()
|
||||
finally:
|
||||
conn.close()
|
||||
return tuple(row) if row else None
|
||||
|
||||
def _validate_and_store(
|
||||
self,
|
||||
webmention_id: int,
|
||||
source_url: str,
|
||||
target_url: str,
|
||||
target_note_id: Optional[int]
|
||||
):
|
||||
"""
|
||||
Validate a single webmention
|
||||
@@ -223,31 +323,49 @@ class WebmentionWorker(threading.Thread):
|
||||
# Process webmention (fetch, validate, extract)
|
||||
result = process_webmention(source_url, target_url, self.site_url)
|
||||
|
||||
# A copy of this very note on a silo (e.g. backfed by Bridgy) is
|
||||
# not a response to it
|
||||
if self._is_own_syndicated_copy(target_note_id, source_url, result):
|
||||
raise ValidationError("Source is this note's own syndicated copy")
|
||||
|
||||
# Update database with successful validation
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
moderation_status = 'approved' if self.auto_approve else 'pending'
|
||||
|
||||
# An update to an existing mention (spec 3.2.4) keeps the admin's
|
||||
# moderation decision; only a first-time valid mention gets the
|
||||
# default. Previously valid = validated with no error.
|
||||
cursor.execute("""
|
||||
UPDATE webmentions
|
||||
SET validation_status = 'valid',
|
||||
SET moderation_status = CASE
|
||||
WHEN moderation_status IN ('hidden', 'deleted') THEN moderation_status
|
||||
WHEN validated_at IS NOT NULL AND validation_error IS NULL
|
||||
THEN moderation_status
|
||||
ELSE ?
|
||||
END,
|
||||
validation_status = 'valid',
|
||||
validation_error = NULL,
|
||||
validated_at = CURRENT_TIMESTAMP,
|
||||
mention_type = ?,
|
||||
author_name = ?,
|
||||
author_url = ?,
|
||||
author_photo = ?,
|
||||
content_snippet = ?,
|
||||
moderation_status = ?,
|
||||
source_post_url = ?,
|
||||
source_published = ?,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""", (
|
||||
moderation_status,
|
||||
result['mention_type'],
|
||||
result['author_name'],
|
||||
result['author_url'],
|
||||
result['author_photo'],
|
||||
result['content_snippet'],
|
||||
moderation_status,
|
||||
result.get('source_post_url'),
|
||||
result.get('source_published'),
|
||||
webmention_id
|
||||
))
|
||||
|
||||
@@ -256,11 +374,17 @@ class WebmentionWorker(threading.Thread):
|
||||
|
||||
logger.info(
|
||||
f"Validated webmention: {source_url} -> {target_url} "
|
||||
f"(type={result['mention_type']}, status={moderation_status})"
|
||||
f"(type={result['mention_type']})"
|
||||
)
|
||||
|
||||
except TransientError as e:
|
||||
# Source unreachable right now: keep a previously valid mention
|
||||
self._mark_validation_failed(webmention_id, source_url, target_url, str(e),
|
||||
transient=True)
|
||||
|
||||
except ValidationError as e:
|
||||
# Validation failed - update with error
|
||||
# Validation failed (410/4xx, no link): the mention is removed
|
||||
# from display (spec 3.2.4)
|
||||
self._mark_validation_failed(webmention_id, source_url, target_url, str(e))
|
||||
|
||||
except Exception as e:
|
||||
@@ -278,11 +402,15 @@ class WebmentionWorker(threading.Thread):
|
||||
webmention_id: int,
|
||||
source_url: str,
|
||||
target_url: str,
|
||||
error: str
|
||||
error: str,
|
||||
transient: bool = False
|
||||
):
|
||||
"""
|
||||
Mark webmention validation as failed/invalid
|
||||
|
||||
A transient failure of a mention that was valid before restores it to
|
||||
valid with its previous data, rather than hiding it.
|
||||
|
||||
Args:
|
||||
webmention_id: Database ID
|
||||
source_url: Source URL
|
||||
@@ -292,7 +420,9 @@ class WebmentionWorker(threading.Thread):
|
||||
# Determine if error or invalid
|
||||
# Per background-worker.md: HTTP 4xx and "no link" are invalid
|
||||
# All other errors are 'error' status
|
||||
if any(x in error.lower() for x in ['does not link', 'returned 4']):
|
||||
if transient:
|
||||
status = 'error'
|
||||
elif any(x in error.lower() for x in ['does not link', 'returned 4', 'own syndicated copy']):
|
||||
status = 'invalid'
|
||||
else:
|
||||
status = 'error'
|
||||
@@ -301,6 +431,22 @@ class WebmentionWorker(threading.Thread):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
if transient:
|
||||
cursor.execute("""
|
||||
UPDATE webmentions
|
||||
SET validation_status = 'valid',
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ? AND validated_at IS NOT NULL AND validation_error IS NULL
|
||||
""", (webmention_id,))
|
||||
if cursor.rowcount:
|
||||
conn.commit()
|
||||
conn.close()
|
||||
logger.warning(
|
||||
f"Could not re-check webmention, keeping it: "
|
||||
f"{source_url} -> {target_url}: {error}"
|
||||
)
|
||||
return
|
||||
|
||||
cursor.execute("""
|
||||
UPDATE webmentions
|
||||
SET validation_status = ?,
|
||||
@@ -421,37 +567,34 @@ class WebmentionWorker(threading.Thread):
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
# Send webmention POST
|
||||
# Send webmention POST (spec 3.1.3)
|
||||
try:
|
||||
# SSRF protection for endpoint URL
|
||||
validate_url_for_fetch(endpoint)
|
||||
|
||||
response = httpx.post(
|
||||
# SSRF-checked and pinned to the checked address (spec 4.3).
|
||||
# Redirects are not followed: a 301/302 would turn the POST
|
||||
# into a GET without the body
|
||||
response = safe_post(
|
||||
endpoint,
|
||||
data={
|
||||
'source': source_url,
|
||||
'target': target_url
|
||||
},
|
||||
timeout=10,
|
||||
headers={'User-Agent': f'StarPunk/1.6.0 (+{self.site_url})'},
|
||||
follow_redirects=True
|
||||
headers={'User-Agent': user_agent(self.site_url)},
|
||||
)
|
||||
|
||||
# Check response
|
||||
# Any 2xx is success (spec 3.1.3)
|
||||
if 200 <= response.status_code < 300:
|
||||
# Success
|
||||
self._mark_send_success(queue_id, source_url, target_url)
|
||||
|
||||
elif 400 <= response.status_code < 500:
|
||||
# Permanent failure (4xx)
|
||||
error = f"HTTP {response.status_code}: {response.text[:200]}"
|
||||
self._mark_send_failed(queue_id, source_url, target_url, error)
|
||||
|
||||
else:
|
||||
# Temporary failure (5xx) - retry
|
||||
elif response.status_code == 429 or response.status_code >= 500:
|
||||
# Rate limited or server error - retry
|
||||
error = f"HTTP {response.status_code}: {response.text[:200]}"
|
||||
self._mark_send_retry(queue_id, source_url, target_url, retry_count, error)
|
||||
|
||||
else:
|
||||
# Permanent failure (3xx, 4xx)
|
||||
error = f"HTTP {response.status_code}: {response.text[:200]}"
|
||||
self._mark_send_failed(queue_id, source_url, target_url, error)
|
||||
|
||||
except (httpx.TimeoutException, httpx.HTTPError) as e:
|
||||
# Connection error - retry
|
||||
self._mark_send_retry(queue_id, source_url, target_url, retry_count, str(e))
|
||||
@@ -467,100 +610,81 @@ class WebmentionWorker(threading.Thread):
|
||||
|
||||
def _discover_endpoint(self, target_url: str) -> Optional[str]:
|
||||
"""
|
||||
Discover webmention endpoint for target URL with caching
|
||||
Discover the webmention endpoint for a target URL (spec 3.1.2)
|
||||
|
||||
Per architect's answers to Q3, Q4, Q10:
|
||||
- GET only (no HEAD)
|
||||
- Check Link header → <link rel="webmention"> → <a rel="webmention">
|
||||
- First match wins
|
||||
- Cache results: 1h for found, 24h for not found
|
||||
- SSRF protection on target URL
|
||||
|
||||
Args:
|
||||
target_url: Target URL to discover endpoint for
|
||||
- GET the target, following redirects (each hop SSRF-checked)
|
||||
- The first Link header with rel=webmention wins, then the first
|
||||
<link> or <a> with rel=webmention and an href, in document order
|
||||
- rel is a space-separated token list, matched case-insensitively
|
||||
- Relative endpoints resolve against the final URL; an empty href
|
||||
is the target itself; query strings are kept
|
||||
- Not cached: an updated post must re-discover (spec 3.1.4)
|
||||
|
||||
Returns:
|
||||
Endpoint URL if found, None otherwise
|
||||
"""
|
||||
# Check cache first
|
||||
cached = self._discovery_cache.get(target_url)
|
||||
if cached and cached['expires_at'] > datetime.utcnow():
|
||||
logger.debug(f"Using cached endpoint for {target_url}: {cached['endpoint']}")
|
||||
return cached['endpoint']
|
||||
Endpoint URL, or None if the target has none (or is blocked/4xx)
|
||||
|
||||
# SSRF protection
|
||||
Raises:
|
||||
httpx.HTTPError: Network errors and 5xx, so the caller retries
|
||||
"""
|
||||
try:
|
||||
validate_url_for_fetch(target_url)
|
||||
result = safe_get(target_url, headers={'User-Agent': user_agent(self.site_url)})
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
logger.warning(f"SSRF protection blocked target {target_url}: {e}")
|
||||
# Cache negative result
|
||||
self._cache_endpoint(target_url, None, hours=24)
|
||||
return None
|
||||
except FetchTooLargeError:
|
||||
logger.info(f"Target too large for discovery: {target_url}")
|
||||
return None
|
||||
|
||||
# Fetch target URL
|
||||
try:
|
||||
response = httpx.get(
|
||||
target_url,
|
||||
timeout=10,
|
||||
headers={'User-Agent': f'StarPunk/1.6.0 (+{self.site_url})'},
|
||||
follow_redirects=True
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
# 1. Check Link header
|
||||
link_header = response.headers.get('Link', '')
|
||||
if link_header:
|
||||
endpoint = self._parse_link_header(link_header, response.url)
|
||||
if endpoint:
|
||||
self._cache_endpoint(target_url, endpoint, hours=1)
|
||||
return endpoint
|
||||
|
||||
# 2. Parse HTML for <link> and <a> elements
|
||||
content_type = response.headers.get('Content-Type', '')
|
||||
if 'html' in content_type.lower():
|
||||
endpoint = self._parse_html_for_endpoint(response.text, response.url)
|
||||
if endpoint:
|
||||
self._cache_endpoint(target_url, endpoint, hours=1)
|
||||
return endpoint
|
||||
|
||||
# No endpoint found - cache negative result
|
||||
logger.info(f"No webmention endpoint found for {target_url}")
|
||||
self._cache_endpoint(target_url, None, hours=24)
|
||||
if result.status_code >= 500:
|
||||
raise httpx.HTTPError(f"Target returned {result.status_code}")
|
||||
if result.status_code >= 400:
|
||||
logger.info(f"Target returned {result.status_code}: {target_url}")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to discover endpoint for {target_url}: {e}")
|
||||
# Don't cache on error - allow retry
|
||||
return None
|
||||
for link_header in result.headers.get_list('Link'):
|
||||
endpoint = self._parse_link_header(link_header, result.url)
|
||||
if endpoint is not None:
|
||||
return endpoint
|
||||
|
||||
content_type = result.headers.get('Content-Type', '')
|
||||
if 'html' in content_type.lower():
|
||||
endpoint = self._parse_html_for_endpoint(result.text, result.url)
|
||||
if endpoint is not None:
|
||||
return endpoint
|
||||
|
||||
logger.info(f"No webmention endpoint found for {target_url}")
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _has_webmention_rel(rel: str) -> bool:
|
||||
return 'webmention' in rel.lower().split()
|
||||
|
||||
def _parse_link_header(self, link_header: str, base_url: str) -> Optional[str]:
|
||||
"""
|
||||
Parse Link header for webmention endpoint
|
||||
Parse a Link header (RFC 8288) for a rel=webmention link
|
||||
|
||||
Args:
|
||||
link_header: Raw Link header value
|
||||
link_header: Link header value (may hold several links)
|
||||
base_url: Base URL for resolving relative URLs
|
||||
|
||||
Returns:
|
||||
Absolute endpoint URL if found, None otherwise
|
||||
Absolute endpoint URL of the first match, None otherwise
|
||||
"""
|
||||
# Simple parsing - look for rel="webmention"
|
||||
# Format: <URL>; rel="webmention"
|
||||
import re
|
||||
pattern = r'<([^>]+)>\s*;\s*rel\s*=\s*["\']?webmention["\']?'
|
||||
match = re.search(pattern, link_header, re.IGNORECASE)
|
||||
|
||||
if match:
|
||||
endpoint = match.group(1)
|
||||
# Make absolute if relative
|
||||
from urllib.parse import urljoin
|
||||
return urljoin(base_url, endpoint)
|
||||
for match in LINK_RE.finditer(link_header):
|
||||
uri, params = match.group(1), match.group(2)
|
||||
for param in LINK_PARAM_RE.finditer(params):
|
||||
if param.group(1).lower() != 'rel':
|
||||
continue
|
||||
rel = param.group(2) if param.group(2) is not None else (param.group(3) or '')
|
||||
if self._has_webmention_rel(rel):
|
||||
return urljoin(base_url, uri.strip())
|
||||
break # only the first rel parameter counts (RFC 8288)
|
||||
|
||||
return None
|
||||
|
||||
def _parse_html_for_endpoint(self, html_content: str, base_url: str) -> Optional[str]:
|
||||
"""
|
||||
Parse HTML for webmention endpoint in <link> or <a> elements
|
||||
Find the first <link> or <a> with rel=webmention, in document order
|
||||
|
||||
Args:
|
||||
html_content: HTML content
|
||||
@@ -570,48 +694,18 @@ class WebmentionWorker(threading.Thread):
|
||||
Absolute endpoint URL if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
doc = lxml_html.fromstring(html_content)
|
||||
|
||||
# Check <link rel="webmention"> first
|
||||
links = doc.xpath('//link[@rel="webmention"]/@href')
|
||||
if links:
|
||||
from urllib.parse import urljoin
|
||||
return urljoin(base_url, links[0])
|
||||
|
||||
# Check <a rel="webmention">
|
||||
anchors = doc.xpath('//a[@rel="webmention"]/@href')
|
||||
if anchors:
|
||||
from urllib.parse import urljoin
|
||||
return urljoin(base_url, anchors[0])
|
||||
|
||||
doc = lxml_html.document_fromstring(html_content)
|
||||
except Exception as e:
|
||||
logger.debug(f"HTML parsing failed: {e}")
|
||||
return None
|
||||
|
||||
for element in doc.iter('link', 'a'):
|
||||
href = element.get('href')
|
||||
if href is not None and self._has_webmention_rel(element.get('rel', '')):
|
||||
return urljoin(base_url, href.strip())
|
||||
|
||||
return None
|
||||
|
||||
def _cache_endpoint(self, target_url: str, endpoint: Optional[str], hours: int):
|
||||
"""
|
||||
Cache endpoint discovery result
|
||||
|
||||
Args:
|
||||
target_url: Target URL
|
||||
endpoint: Discovered endpoint (None = negative cache)
|
||||
hours: TTL in hours
|
||||
"""
|
||||
self._discovery_cache[target_url] = {
|
||||
'endpoint': endpoint,
|
||||
'expires_at': datetime.utcnow() + timedelta(hours=hours)
|
||||
}
|
||||
|
||||
# Optional: Evict old entries (keep cache bounded)
|
||||
if len(self._discovery_cache) > 1000:
|
||||
# Remove expired entries
|
||||
now = datetime.utcnow()
|
||||
self._discovery_cache = {
|
||||
k: v for k, v in self._discovery_cache.items()
|
||||
if v['expires_at'] > now
|
||||
}
|
||||
|
||||
def _mark_send_success(self, queue_id: int, source_url: str, target_url: str):
|
||||
"""Mark webmention send as successful"""
|
||||
try:
|
||||
|
||||
@@ -332,6 +332,9 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
|
||||
}
|
||||
|
||||
.webmention-author a {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: var(--spacing-sm);
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
@@ -361,6 +364,13 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
|
||||
margin-left: 0;
|
||||
}
|
||||
|
||||
.facepile .face > .h-card,
|
||||
.facepile .face > .h-card > a {
|
||||
display: block;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.facepile .face img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
@@ -392,6 +402,12 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.webmention-mention .h-card img {
|
||||
border-radius: 50%;
|
||||
vertical-align: middle;
|
||||
margin-right: var(--spacing-xs);
|
||||
}
|
||||
|
||||
.webmention-mention p {
|
||||
margin: var(--spacing-xs) 0 0 0;
|
||||
font-size: 0.875rem;
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
{% extends "base.html" %}
|
||||
|
||||
{% block title %}Post Deleted - {{ config.SITE_NAME }}{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<article class="error-page">
|
||||
<h1>410 - Post Deleted</h1>
|
||||
<p>This post has been deleted.</p>
|
||||
<p><a href="/">Return to homepage</a></p>
|
||||
</article>
|
||||
{% endblock %}
|
||||
+11
-1
@@ -12,15 +12,25 @@
|
||||
{# Webmention endpoint (v1.6.0) #}
|
||||
{% if config.WEBMENTION_ENABLED %}
|
||||
<link rel="webmention" href="/webmention">
|
||||
{% endif %}
|
||||
|
||||
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
|
||||
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
|
||||
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
|
||||
<link rel="authorization_endpoint" href="{{ absolute_url_for('indieauth.authorize') }}">
|
||||
<link rel="token_endpoint" href="{{ absolute_url_for('indieauth.token') }}">
|
||||
|
||||
{# Author's home page: rel=me makes the h-card linking to it this site's
|
||||
representative h-card; rel=author is the authorship fallback (v2.8.0) #}
|
||||
{% set author_home = (author.url or author.me) if author else config.ADMIN_ME %}
|
||||
{% if author_home %}
|
||||
<link rel="me" href="{{ author_home }}">
|
||||
<link rel="author" href="{{ author_home }}">
|
||||
{% endif %}
|
||||
|
||||
{# rel-me links from discovered author profile (v1.2.0 Phase 2) #}
|
||||
{% if author and author.rel_me_links %}
|
||||
{% for profile_url in author.rel_me_links %}
|
||||
{% for profile_url in author.rel_me_links if profile_url != author_home %}
|
||||
<link rel="me" href="{{ profile_url }}">
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
@@ -7,6 +7,12 @@
|
||||
<div class="h-feed">
|
||||
<h1 class="p-name page-title">Likes</h1>
|
||||
|
||||
{# Feed URL and author, as on the home page (v2.8.0) #}
|
||||
<a class="u-url" href="{{ absolute_url_for(request.endpoint, **request.view_args) }}" hidden>Feed URL</a>
|
||||
{% if author %}
|
||||
<span class="p-author h-card" hidden><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
|
||||
{% endif %}
|
||||
|
||||
{% if notes %}
|
||||
{% for note in notes %}
|
||||
{{ note_preview(note, author) }}
|
||||
|
||||
+11
-81
@@ -1,13 +1,14 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "partials/media.html" import display_media %}
|
||||
{% from "partials/reaction.html" import reaction %}
|
||||
{% import "partials/responses.html" as responses with context %}
|
||||
|
||||
{% block title %}{% if note.content.strip().startswith('#') %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
|
||||
{% block title %}{% if note.has_title %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<article class="h-entry">
|
||||
{# Detect if note has explicit title (starts with # heading) - per Q22 #}
|
||||
{% set has_explicit_title = note.content.strip().startswith('#') %}
|
||||
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
|
||||
{% set has_explicit_title = note.has_title %}
|
||||
|
||||
{# p-name only if note has explicit title (per Q22) #}
|
||||
{% if has_explicit_title %}
|
||||
@@ -53,9 +54,9 @@
|
||||
|
||||
<footer class="note-meta">
|
||||
{# u-url and u-uid same for notes (per Q23) #}
|
||||
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at.isoformat() }}">{{ note.created_at.strftime('%B %-d, %Y · %-I:%M %p') }}</time></a>
|
||||
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at|iso_utc }}">{{ note.created_at.strftime('%B %-d, %Y · %-I:%M %p') }}</time></a>
|
||||
{% if note.updated_at and note.updated_at != note.created_at %}
|
||||
<span class="updated">Updated <time class="dt-updated" datetime="{{ note.updated_at.isoformat() }}">{{ note.updated_at.strftime('%b %-d, %Y') }}</time></span>
|
||||
<span class="updated">Updated <time class="dt-updated" datetime="{{ note.updated_at|iso_utc }}">{{ note.updated_at.strftime('%b %-d, %Y') }}</time></span>
|
||||
{% endif %}
|
||||
{# rel="tag" per microformats2 p-category #}
|
||||
{% if note.tags %}
|
||||
@@ -89,106 +90,35 @@
|
||||
{% if replies %}
|
||||
<div class="webmention-group webmention-replies">
|
||||
<h3>{{ replies|length }} {{ 'Reply' if replies|length == 1 else 'Replies' }}</h3>
|
||||
{% for reply in replies %}
|
||||
<article class="webmention webmention-reply h-cite">
|
||||
<div class="webmention-author">
|
||||
{% if reply.author_photo %}
|
||||
<img src="{{ reply.author_photo }}" alt="" width="48" height="48" loading="lazy">
|
||||
{% endif %}
|
||||
<a class="p-author" href="{{ reply.author_url or reply.source_url }}">
|
||||
{{ reply.author_name or 'Anonymous' }}
|
||||
</a>
|
||||
</div>
|
||||
{% if reply.content_snippet %}
|
||||
<div class="p-content">{{ reply.content_snippet|safe }}</div>
|
||||
{% endif %}
|
||||
<a class="u-url webmention-date" href="{{ reply.source_url }}">
|
||||
<time class="dt-published" datetime="{{ reply.received_at.isoformat() }}">
|
||||
{{ reply.received_at.strftime('%B %d, %Y') }}
|
||||
</time>
|
||||
</a>
|
||||
</article>
|
||||
{% endfor %}
|
||||
{% for wm in replies %}{{ responses.reply(wm) }}{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if likes %}
|
||||
<div class="webmention-group webmention-likes">
|
||||
<h3>{{ likes|length }} {{ 'Like' if likes|length == 1 else 'Likes' }}</h3>
|
||||
<div class="facepile">
|
||||
{% for wm in likes[:8] %}
|
||||
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
|
||||
{% if wm.author_photo %}
|
||||
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
|
||||
{% else %}
|
||||
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
|
||||
{{ (wm.author_name or '?')[0]|upper }}
|
||||
</span>
|
||||
{% endif %}
|
||||
</a>
|
||||
{% endfor %}
|
||||
{% if likes|length > 8 %}
|
||||
<span class="face-overflow">+{{ likes|length - 8 }} others</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{{ responses.facepile(likes, 'u-like') }}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if reposts %}
|
||||
<div class="webmention-group webmention-reposts">
|
||||
<h3>{{ reposts|length }} {{ 'Repost' if reposts|length == 1 else 'Reposts' }}</h3>
|
||||
<div class="facepile">
|
||||
{% for wm in reposts[:8] %}
|
||||
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
|
||||
{% if wm.author_photo %}
|
||||
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
|
||||
{% else %}
|
||||
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
|
||||
{{ (wm.author_name or '?')[0]|upper }}
|
||||
</span>
|
||||
{% endif %}
|
||||
</a>
|
||||
{% endfor %}
|
||||
{% if reposts|length > 8 %}
|
||||
<span class="face-overflow">+{{ reposts|length - 8 }} others</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{{ responses.facepile(reposts, 'u-repost') }}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if bookmarks %}
|
||||
<div class="webmention-group webmention-bookmarks">
|
||||
<h3>{{ bookmarks|length }} {{ 'Bookmark' if bookmarks|length == 1 else 'Bookmarks' }}</h3>
|
||||
<div class="facepile">
|
||||
{% for wm in bookmarks[:8] %}
|
||||
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
|
||||
{% if wm.author_photo %}
|
||||
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
|
||||
{% else %}
|
||||
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
|
||||
{{ (wm.author_name or '?')[0]|upper }}
|
||||
</span>
|
||||
{% endif %}
|
||||
</a>
|
||||
{% endfor %}
|
||||
{% if bookmarks|length > 8 %}
|
||||
<span class="face-overflow">+{{ bookmarks|length - 8 }} others</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{{ responses.facepile(bookmarks, 'u-bookmark') }}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if mentions %}
|
||||
<div class="webmention-group webmention-mentions">
|
||||
<h3>{{ mentions|length }} {{ 'Mention' if mentions|length == 1 else 'Mentions' }}</h3>
|
||||
{% for mention in mentions %}
|
||||
<div class="webmention webmention-mention">
|
||||
<a href="{{ mention.source_url }}">{{ mention.author_name or mention.source_url }}</a>
|
||||
{% if mention.content_snippet %}
|
||||
<p>{{ mention.content_snippet|truncate(200) }}</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
{% for wm in mentions %}{{ responses.mention(wm) }}{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
</section>
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
{% from "partials/reaction.html" import reaction %}
|
||||
{% macro note_preview(note, author) %}
|
||||
<article class="h-entry note-preview">
|
||||
{# Notes starting with a # heading have a title (Q22) #}
|
||||
{% set has_explicit_title = note.content.strip().startswith('#') %}
|
||||
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
|
||||
{% set has_explicit_title = note.has_title %}
|
||||
{% if has_explicit_title %}
|
||||
<h2 class="p-name"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">{{ note.title }}</a></h2>
|
||||
{% endif %}
|
||||
@@ -31,7 +31,7 @@
|
||||
{% endif %}
|
||||
|
||||
<footer class="note-meta">
|
||||
<a class="u-url" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at.isoformat() }}">{{ note.created_at.strftime('%b %-d, %Y') }}</time></a>
|
||||
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at|iso_utc }}">{{ note.created_at.strftime('%b %-d, %Y') }}</time></a>
|
||||
{% if note.tags %}
|
||||
<span class="note-tags">
|
||||
{% for tag in note.tags %}<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>{% endfor %}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
{# Received webmentions as microformats2 responses (v2.8.0)
|
||||
Each response is an h-cite in a property of the note's h-entry:
|
||||
u-comment (replies), u-like, u-repost, u-bookmark, u-mention.
|
||||
See https://indieweb.org/comments-presentation and microformats.org/wiki/h-entry #}
|
||||
|
||||
{# When the source said when it was posted, use that; else when we received it #}
|
||||
{% macro response_time(wm, fmt='%B %-d, %Y') %}
|
||||
{% set when = wm.source_published or wm.received_at %}
|
||||
<time class="dt-published" datetime="{{ when|iso_utc }}">{{ when.strftime(fmt) if when.strftime is defined else when }}</time>
|
||||
{%- endmacro %}
|
||||
|
||||
{# The response's own URL: the source's u-url if it gave one #}
|
||||
{% macro response_url(wm) %}{{ wm.source_post_url or wm.source_url }}{% endmacro %}
|
||||
|
||||
{# Author h-card: name, URL and photo. u-url is on an inner link: on the
|
||||
h-card element itself it would belong to the enclosing h-cite #}
|
||||
{% macro author_card(wm, class='', photo_size=48, show_name=True) %}
|
||||
<span class="p-author h-card {{ class }}"><a class="u-url" href="{{ wm.author_url or wm.source_url }}" title="{{ wm.author_name or 'Anonymous' }}">
|
||||
{%- if wm.author_photo %}<img class="u-photo" src="{{ wm.author_photo }}" alt="" width="{{ photo_size }}" height="{{ photo_size }}" loading="lazy">
|
||||
{%- elif not show_name %}<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}" aria-hidden="true">{{ (wm.author_name or '?')[0]|upper }}</span>
|
||||
{%- endif %}
|
||||
<span class="p-name{{ '' if show_name else ' visually-hidden' }}">{{ wm.author_name or 'Anonymous' }}</span>
|
||||
</a></span>
|
||||
{%- endmacro %}
|
||||
|
||||
{# A reply, shown as a comment #}
|
||||
{% macro reply(wm) %}
|
||||
<article class="webmention webmention-reply u-comment h-cite">
|
||||
<div class="webmention-author">{{ author_card(wm) }}</div>
|
||||
{% if wm.content_snippet %}
|
||||
<div class="e-content">{{ wm.content_snippet|safe }}</div>
|
||||
{% endif %}
|
||||
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
|
||||
</article>
|
||||
{%- endmacro %}
|
||||
|
||||
{# Likes, reposts or bookmarks as a facepile; prop is u-like / u-repost / u-bookmark.
|
||||
Faces past `shown` are hidden on screen but kept for parsers. #}
|
||||
{% macro facepile(items, prop, shown=8) %}
|
||||
<div class="facepile">
|
||||
{% for wm in items %}
|
||||
<span class="face {{ prop }} h-cite"{% if loop.index > shown %} hidden{% endif %}>
|
||||
{{ author_card(wm, photo_size=32, show_name=False) }}
|
||||
<a class="u-url" href="{{ response_url(wm) }}" hidden>{{ response_time(wm) }}</a>
|
||||
</span>
|
||||
{% endfor %}
|
||||
{% if items|length > shown %}
|
||||
<span class="face-overflow">+{{ items|length - shown }} others</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{%- endmacro %}
|
||||
|
||||
{# A post that mentions this note without replying #}
|
||||
{% macro mention(wm) %}
|
||||
<div class="webmention webmention-mention u-mention h-cite">
|
||||
{{ author_card(wm, photo_size=24) }}
|
||||
{% if wm.content_snippet %}
|
||||
<p class="p-content">{{ wm.content_snippet|striptags|truncate(200) }}</p>
|
||||
{% endif %}
|
||||
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
|
||||
</div>
|
||||
{%- endmacro %}
|
||||
@@ -7,6 +7,12 @@
|
||||
<div class="h-feed">
|
||||
<h1 class="p-name page-title">Notes tagged <span class="tag-name">{{ tag.display_name }}</span></h1>
|
||||
|
||||
{# Feed URL and author, as on the home page (v2.8.0) #}
|
||||
<a class="u-url" href="{{ absolute_url_for(request.endpoint, **request.view_args) }}" hidden>Feed URL</a>
|
||||
{% if author %}
|
||||
<span class="p-author h-card" hidden><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
|
||||
{% endif %}
|
||||
|
||||
{% if notes %}
|
||||
{% for note in notes %}
|
||||
{{ note_preview(note, author) }}
|
||||
|
||||
@@ -110,7 +110,7 @@ class TestImageValidation:
|
||||
assert mime_type == 'image/jpeg'
|
||||
assert width == 800
|
||||
assert height == 600
|
||||
assert file_data == image_data # No conversion
|
||||
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
|
||||
|
||||
def test_valid_png(self):
|
||||
"""Test validation of valid PNG image"""
|
||||
@@ -120,7 +120,7 @@ class TestImageValidation:
|
||||
assert mime_type == 'image/png'
|
||||
assert width == 800
|
||||
assert height == 600
|
||||
assert file_data == image_data # No conversion
|
||||
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
|
||||
|
||||
def test_valid_gif(self):
|
||||
"""Test validation of valid GIF image"""
|
||||
@@ -130,7 +130,7 @@ class TestImageValidation:
|
||||
assert mime_type == 'image/gif'
|
||||
assert width == 800
|
||||
assert height == 600
|
||||
assert file_data == image_data # No conversion
|
||||
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
|
||||
|
||||
def test_valid_webp(self):
|
||||
"""Test validation of valid WebP image"""
|
||||
@@ -140,7 +140,7 @@ class TestImageValidation:
|
||||
assert mime_type == 'image/webp'
|
||||
assert width == 800
|
||||
assert height == 600
|
||||
assert file_data == image_data # No conversion
|
||||
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
|
||||
|
||||
def test_file_too_large(self):
|
||||
"""Test rejection of >10MB file (per Q6)"""
|
||||
@@ -183,11 +183,8 @@ class TestHEICSupport:
|
||||
assert mime_type == 'image/jpeg'
|
||||
assert width == 800
|
||||
assert height == 600
|
||||
# file_data should be different (converted to JPEG)
|
||||
assert file_data != heic_data
|
||||
# Verify it's actually JPEG by opening it
|
||||
img = Image.open(io.BytesIO(file_data))
|
||||
assert img.format == 'JPEG'
|
||||
# Converted to JPEG (validate_image returns a PIL Image)
|
||||
assert file_data.format == 'JPEG'
|
||||
|
||||
def test_heic_with_rgba_mode(self):
|
||||
"""Test HEIC with alpha channel is converted to RGB JPEG"""
|
||||
@@ -209,8 +206,7 @@ class TestHEICSupport:
|
||||
# Should be converted to JPEG (no alpha)
|
||||
assert mime_type == 'image/jpeg'
|
||||
# Verify it's RGB (no alpha)
|
||||
img = Image.open(io.BytesIO(file_data))
|
||||
assert img.mode == 'RGB'
|
||||
assert file_data.mode == 'RGB'
|
||||
|
||||
def test_heic_dimensions_preserved(self):
|
||||
"""Test HEIC conversion preserves dimensions"""
|
||||
@@ -273,9 +269,8 @@ class TestMPOSupport:
|
||||
assert mime_type == 'image/jpeg'
|
||||
assert width == 800
|
||||
assert height == 600
|
||||
# Verify it's actually JPEG by opening it
|
||||
img = Image.open(io.BytesIO(file_data))
|
||||
assert img.format == 'JPEG'
|
||||
# Converted to JPEG (validate_image returns a PIL Image)
|
||||
assert file_data.format == 'JPEG'
|
||||
# Note: MPO with single frame may have identical bytes to JPEG
|
||||
# The important part is the format detection and MIME type correction
|
||||
|
||||
|
||||
@@ -289,16 +289,20 @@ class TestRelMe:
|
||||
assert 'https://github.com/testuser' in rels['me']
|
||||
assert 'https://mastodon.social/@testuser' in rels['me']
|
||||
|
||||
def test_no_relme_without_author(self, client, app):
|
||||
"""No rel=me links if no author profile"""
|
||||
def test_relme_to_admin_me_without_author(self, client, app):
|
||||
"""Without a discovered profile, rel=me and rel=author still point at ADMIN_ME (v2.8.0)
|
||||
|
||||
The home page's p-author h-card links to the author's site; rel=me to
|
||||
the same URL makes it the representative h-card.
|
||||
"""
|
||||
with patch('starpunk.author_discovery.get_author_profile', return_value=None):
|
||||
response = client.get('/')
|
||||
|
||||
parsed = mf2py.parse(doc=response.data.decode(), url='http://localhost/')
|
||||
|
||||
rels = parsed.get('rels', {})
|
||||
# Should not have rel=me, or it should be empty
|
||||
assert len(rels.get('me', [])) == 0, "Should not have rel=me without author"
|
||||
assert rels.get('me') == [app.config['ADMIN_ME']]
|
||||
assert rels.get('author') == [app.config['ADMIN_ME']]
|
||||
|
||||
|
||||
class TestV130Microformats:
|
||||
|
||||
@@ -0,0 +1,268 @@
|
||||
"""
|
||||
Microformats2 fixes (v2.8.0)
|
||||
|
||||
Pages are rendered and parsed with mf2py; assertions are on the parsed data,
|
||||
which is what IndieWeb readers and Webmention receivers see.
|
||||
"""
|
||||
|
||||
import re
|
||||
import sqlite3
|
||||
from unittest.mock import patch
|
||||
|
||||
import mf2py
|
||||
import pytest
|
||||
|
||||
from starpunk import mf2_utils
|
||||
from starpunk.models import split_title
|
||||
from starpunk.webmention_processor import extract_author, extract_source_details
|
||||
from starpunk.webmention_worker import WebmentionWorker
|
||||
|
||||
BASE = 'http://localhost:5000'
|
||||
|
||||
|
||||
def parse(client, path):
|
||||
response = client.get(path)
|
||||
assert response.status_code == 200
|
||||
return mf2py.parse(doc=response.data.decode(), url=BASE + path)
|
||||
|
||||
|
||||
def entry_of(parsed):
|
||||
return next(i for i in parsed['items'] if 'h-entry' in i['type'])
|
||||
|
||||
|
||||
def make_note(app, content, **kw):
|
||||
from starpunk.notes import create_note
|
||||
with app.test_request_context():
|
||||
return create_note(content, published=True, **kw)
|
||||
|
||||
|
||||
def add_mention(app, note, kind, n, photo=True, **cols):
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
fields = {
|
||||
'source_url': f'https://src.example/{kind}/{n}',
|
||||
'target_url': f'{BASE}/note/{note.slug}',
|
||||
'target_note_id': note.id,
|
||||
'mention_type': kind,
|
||||
'author_name': f'{kind.title()} Person {n}',
|
||||
'author_url': f'https://{kind}{n}.example/',
|
||||
'author_photo': f'https://{kind}{n}.example/me.jpg' if photo else None,
|
||||
'content_snippet': f'<p>{kind} text <a href="https://x.example/">link</a></p>',
|
||||
'validation_status': 'valid',
|
||||
'moderation_status': 'approved',
|
||||
**cols,
|
||||
}
|
||||
db.execute(f"INSERT INTO webmentions ({', '.join(fields)}) VALUES ({', '.join('?' * len(fields))})",
|
||||
tuple(fields.values()))
|
||||
db.commit()
|
||||
db.close()
|
||||
|
||||
|
||||
# A1/A2: received responses are properties of the entry
|
||||
|
||||
def test_responses_are_entry_properties(app, client):
|
||||
note = make_note(app, "A note")
|
||||
add_mention(app, note, 'reply', 1, source_post_url='https://src.example/reply/1#canonical',
|
||||
source_published='2026-11-01T10:00:00Z')
|
||||
for n in range(10):
|
||||
add_mention(app, note, 'like', n, photo=n % 2 == 0)
|
||||
add_mention(app, note, 'repost', 1)
|
||||
add_mention(app, note, 'bookmark', 1)
|
||||
add_mention(app, note, 'mention', 1)
|
||||
|
||||
props = entry_of(parse(client, f'/note/{note.slug}'))['properties']
|
||||
assert 'children' not in entry_of(parse(client, f'/note/{note.slug}'))
|
||||
|
||||
(comment,) = props['comment']
|
||||
assert comment['type'] == ['h-cite']
|
||||
author = comment['properties']['author'][0]
|
||||
assert author['type'] == ['h-card']
|
||||
assert author['properties']['name'] == ['Reply Person 1']
|
||||
assert author['properties']['url'] == ['https://reply1.example/']
|
||||
assert author['properties']['photo'][0] in ('https://reply1.example/me.jpg',
|
||||
{'value': 'https://reply1.example/me.jpg', 'alt': ''})
|
||||
assert '<a href="https://x.example/"' in comment['properties']['content'][0]['html']
|
||||
# the source's own URL and published time, not ours
|
||||
assert comment['properties']['url'] == ['https://src.example/reply/1#canonical']
|
||||
assert comment['properties']['published'] == ['2026-11-01T10:00:00Z']
|
||||
|
||||
# all ten likes, including the ones past the visible eight
|
||||
assert len(props['like']) == 10
|
||||
names = {like['properties']['author'][0]['properties']['name'][0] for like in props['like']}
|
||||
assert names == {f'Like Person {n}' for n in range(10)}
|
||||
assert props['like'][0]['properties']['url'] == ['https://src.example/like/0']
|
||||
|
||||
assert props['repost'][0]['properties']['author'][0]['properties']['name'] == ['Repost Person 1']
|
||||
assert props['bookmark'][0]['type'] == ['h-cite']
|
||||
assert props['mention'][0]['properties']['author'][0]['properties']['url'] == ['https://mention1.example/']
|
||||
|
||||
|
||||
# A3: dt-* are RFC 3339 UTC
|
||||
|
||||
UTC = re.compile(r'^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$')
|
||||
|
||||
|
||||
def test_dates_are_utc(app, client):
|
||||
from starpunk.notes import update_note
|
||||
note = make_note(app, "Dated")
|
||||
with app.test_request_context():
|
||||
update_note(id=note.id, content="Dated, edited")
|
||||
add_mention(app, note, 'reply', 1)
|
||||
|
||||
props = entry_of(parse(client, f'/note/{note.slug}'))['properties']
|
||||
assert UTC.match(props['published'][0]), props['published']
|
||||
assert UTC.match(props['updated'][0]), props['updated']
|
||||
assert UTC.match(props['comment'][0]['properties']['published'][0])
|
||||
|
||||
home = parse(client, '/')['items'][0]['children'][0]['properties']
|
||||
assert UTC.match(home['published'][0])
|
||||
|
||||
|
||||
def test_micropub_source_published_is_utc():
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from starpunk.utils import iso_utc
|
||||
assert iso_utc(datetime(2026, 11, 1, 16, 25, 12, 921768)) == '2026-11-01T16:25:12Z'
|
||||
assert iso_utc(datetime(2026, 11, 1, 9, 0, tzinfo=timezone(timedelta(hours=-7)))) == '2026-11-01T16:00:00Z'
|
||||
assert iso_utc('2026-11-01T16:25:15+0000') == '2026-11-01T16:25:15Z'
|
||||
|
||||
|
||||
# A4: only a real Markdown heading is a title
|
||||
|
||||
@pytest.mark.parametrize("content,title", [
|
||||
("# Title\n\nBody", "Title"),
|
||||
("## Sub title ##\n\nBody", "Sub title"),
|
||||
("#indieweb is great", None),
|
||||
("#\n\nbody text", None),
|
||||
("####### seven", None),
|
||||
("Plain note", None),
|
||||
])
|
||||
def test_split_title(content, title):
|
||||
assert split_title(content)[0] == title
|
||||
|
||||
|
||||
@pytest.mark.parametrize("content,kind", [
|
||||
("#indieweb is great, posting from my phone", "note"),
|
||||
("#\n\nbody text", "note"),
|
||||
("# A Title\n\nThe body", "article"),
|
||||
])
|
||||
def test_post_type_of_hash_notes(app, client, content, kind):
|
||||
note = make_note(app, content)
|
||||
for path in (f'/note/{note.slug}', '/'):
|
||||
parsed = parse(client, path)
|
||||
entry = entry_of(parsed) if path != '/' else parsed['items'][0]['children'][0]
|
||||
props = entry['properties']
|
||||
if kind == 'note':
|
||||
assert 'name' not in props
|
||||
assert props['content'][0]['value'].strip()
|
||||
else:
|
||||
assert props['name'] == ['A Title']
|
||||
|
||||
|
||||
# A5: the note's own syndicated copy is not a response
|
||||
|
||||
def test_own_syndicated_copy_rejected(app):
|
||||
note = make_note(app, "Crossposted")
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
db.execute("INSERT INTO syndications (note_id, target, remote_url) VALUES (?, 'gotosocial', ?)",
|
||||
(note.id, 'https://social.example/@me/statuses/1'))
|
||||
db.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) VALUES (?, ?, ?)",
|
||||
('https://brid.gy/post/mastodon/@someone/1', f'{BASE}/note/{note.slug}', note.id))
|
||||
db.commit()
|
||||
|
||||
result = {'valid': True, 'mention_type': 'mention', 'author_name': 'Me', 'author_url': 'https://me.example/',
|
||||
'author_photo': None, 'content_snippet': 'x',
|
||||
'source_post_url': 'https://social.example/@me/statuses/1', 'source_uid': None,
|
||||
'source_published': None}
|
||||
worker = WebmentionWorker(app, db_path=app.config['DATABASE_PATH'], site_url=BASE)
|
||||
with patch("starpunk.webmention_worker.process_webmention", return_value=result):
|
||||
worker._validate_webmention(1, 'https://brid.gy/post/mastodon/@someone/1', 'x', note.id)
|
||||
|
||||
status, error = db.execute("SELECT validation_status, validation_error FROM webmentions").fetchone()
|
||||
assert status == 'invalid' and 'own syndicated copy' in error
|
||||
|
||||
|
||||
# B1/B3: discovery links
|
||||
|
||||
def test_head_links_without_webmention(app, client):
|
||||
app.config['WEBMENTION_ENABLED'] = False
|
||||
rels = parse(client, '/')['rels']
|
||||
assert 'webmention' not in rels
|
||||
for rel in ('micropub', 'indieauth-metadata', 'authorization_endpoint', 'token_endpoint'):
|
||||
assert rels.get(rel), rel
|
||||
|
||||
|
||||
def test_home_has_representative_hcard(app, client):
|
||||
# Fresh install: discovery fails, the fallback profile is used
|
||||
parsed = parse(client, '/')
|
||||
card = mf2_utils.representative_hcard(parsed, BASE + '/')
|
||||
assert card and card['properties']['url'] == [app.config['ADMIN_ME']]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", ['/likes', '/tag/python'])
|
||||
def test_list_feeds_have_url_and_author(app, client, path):
|
||||
make_note(app, "Tagged", tags=["python"])
|
||||
feed = parse(client, path)['items'][0]
|
||||
assert feed['type'] == ['h-feed']
|
||||
assert feed['properties']['url'] == [BASE + path]
|
||||
assert feed['properties']['author'][0]['type'] == ['h-card']
|
||||
|
||||
|
||||
# B2: representative h-card
|
||||
|
||||
def _cards(html, url='https://me.example/'):
|
||||
return mf2py.parse(doc=html, url=url)
|
||||
|
||||
|
||||
def test_representative_hcard_rules():
|
||||
other = '<div class="h-card"><a class="p-name u-url" href="https://bob.example/">Bob</a></div>'
|
||||
# someone else's h-card only: none
|
||||
assert mf2_utils.representative_hcard(_cards(other), 'https://me.example/') is None
|
||||
# uid + url
|
||||
me = '<div class="h-card"><a class="p-name u-url u-uid" href="/">Me</a></div>'
|
||||
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(other + me), 'https://me.example/'), 'name') == 'Me'
|
||||
# rel=me
|
||||
relme = ('<a rel="me" href="https://social.example/@me">x</a>'
|
||||
'<div class="h-card"><a class="p-name u-url" href="https://social.example/@me">Me</a></div>')
|
||||
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(other + relme), 'https://me.example/'), 'name') == 'Me'
|
||||
# only one h-card, url = page
|
||||
single = '<div class="h-card"><a class="p-name u-url" href="/">Me</a></div>'
|
||||
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(single), 'https://me.example/'), 'name') == 'Me'
|
||||
|
||||
|
||||
# B4: the responding entry and its author
|
||||
|
||||
FEED_PAGE = '''
|
||||
<div class="h-feed">
|
||||
<div class="p-author h-card"><a class="p-name u-url" href="https://feed.example/">Feed Owner</a></div>
|
||||
<div class="h-entry"><a class="u-url" href="https://feed.example/1">one</a>
|
||||
<div class="e-content">unrelated</div></div>
|
||||
<div class="h-entry"><a class="u-url" href="https://feed.example/2">two</a>
|
||||
<time class="dt-published" datetime="2026-11-01T09:00:00-07:00">x</time>
|
||||
<a class="u-in-reply-to" href="https://me.example/note/a">re</a>
|
||||
<div class="e-content">my reply</div></div>
|
||||
</div>'''
|
||||
|
||||
|
||||
def test_responding_entry_and_feed_author():
|
||||
parsed = mf2py.parse(doc=FEED_PAGE, url='https://feed.example/')
|
||||
entry = mf2_utils.responding_entry(parsed, 'https://me.example/note/a#frag')
|
||||
assert mf2_utils.first(entry, 'url') == 'https://feed.example/2'
|
||||
assert mf2_utils.response_type(entry, 'https://me.example/note/a') == 'reply'
|
||||
|
||||
author = extract_author(parsed, 'https://feed.example/', entry)
|
||||
assert author['author_name'] == 'Feed Owner'
|
||||
assert author['author_url'] == 'https://feed.example/'
|
||||
|
||||
details = extract_source_details(entry)
|
||||
assert details['source_post_url'] == 'https://feed.example/2'
|
||||
assert details['source_published'] == '2026-11-01T16:00:00Z'
|
||||
|
||||
|
||||
def test_author_as_url_or_name():
|
||||
url_author = '<div class="h-entry"><a class="u-author" href="https://al.example/">al</a></div>'
|
||||
parsed = mf2py.parse(doc=url_author, url='https://s.example/')
|
||||
assert extract_author(parsed, 'https://s.example/')['author_url'] == 'https://al.example/'
|
||||
|
||||
name_author = '<div class="h-entry"><span class="p-author">Al Smith</span></div>'
|
||||
parsed = mf2py.parse(doc=name_author, url='https://s.example/')
|
||||
author = extract_author(parsed, 'https://s.example/')
|
||||
assert author['author_name'] == 'Al Smith' and author['author_url'] == 'https://s.example/'
|
||||
@@ -161,7 +161,7 @@ class TestDelete:
|
||||
url = create(client, h, category=["kept"])
|
||||
path = url.replace(SITE, "")
|
||||
assert client.post("/micropub", data={"action": "delete", "url": url}, headers=h).status_code == 204
|
||||
assert client.get(path).status_code == 404
|
||||
assert client.get(path).status_code == 410 # Gone (Webmention 3.1.5)
|
||||
response = client.post("/micropub", json={"action": "undelete", "url": url}, headers=h)
|
||||
assert response.status_code == 204
|
||||
assert client.get(path).status_code == 200
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
"""
|
||||
Migrations under concurrent gunicorn workers (v2.8.1)
|
||||
|
||||
The v2.8.0 deploy failed to boot: one worker applied migration 014 while
|
||||
another applied it again ("duplicate column name"), because
|
||||
conn.executescript() committed and released the migration lock early.
|
||||
"""
|
||||
|
||||
import multiprocessing
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from starpunk.database.schema import INITIAL_SCHEMA_SQL
|
||||
from starpunk.migrations import execute_in_transaction, run_migrations, split_sql_statements
|
||||
|
||||
MIGRATIONS = Path(__file__).parent.parent / "migrations"
|
||||
|
||||
|
||||
def test_script_runs_inside_the_transaction(tmp_path):
|
||||
conn = sqlite3.connect(tmp_path / "t.db")
|
||||
conn.execute("CREATE TABLE t (a INTEGER)")
|
||||
conn.commit()
|
||||
|
||||
conn.execute("BEGIN IMMEDIATE")
|
||||
execute_in_transaction(conn, "ALTER TABLE t ADD COLUMN b TEXT;\nALTER TABLE t ADD COLUMN c TEXT;")
|
||||
conn.rollback()
|
||||
|
||||
assert [row[1] for row in conn.execute("PRAGMA table_info(t)")] == ["a"]
|
||||
|
||||
|
||||
def test_split_keeps_trigger_bodies_together():
|
||||
script = """
|
||||
-- comment; with a semicolon
|
||||
CREATE TABLE t (a TEXT DEFAULT 'x;y');
|
||||
CREATE TRIGGER tr AFTER DELETE ON t BEGIN
|
||||
DELETE FROM t WHERE a = old.a;
|
||||
SELECT 1;
|
||||
END;
|
||||
"""
|
||||
statements = split_sql_statements(script)
|
||||
assert len(statements) == 2
|
||||
assert statements[1].startswith("CREATE TRIGGER") and statements[1].endswith("END;")
|
||||
|
||||
|
||||
def _migrate(db_path, barrier, errors):
|
||||
try:
|
||||
barrier.wait()
|
||||
run_migrations(db_path)
|
||||
except Exception as e: # report to the parent
|
||||
errors.put(repr(e))
|
||||
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.parametrize("trial", range(5)) # the race hit ~1 in 5 runs
|
||||
def test_concurrent_workers_apply_each_migration_once(tmp_path, trial):
|
||||
# A database at migration 013, like production before v2.8.0
|
||||
db_path = tmp_path / "starpunk.db"
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.executescript(INITIAL_SCHEMA_SQL) # as init_db does
|
||||
conn.close()
|
||||
run_migrations(db_path)
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.execute("DELETE FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'")
|
||||
conn.commit()
|
||||
columns = [row[1] for row in conn.execute("PRAGMA table_info(webmentions)")]
|
||||
if "source_post_url" in columns:
|
||||
conn.execute("ALTER TABLE webmentions DROP COLUMN source_post_url")
|
||||
conn.execute("ALTER TABLE webmentions DROP COLUMN source_published")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
ctx = multiprocessing.get_context("fork")
|
||||
barrier, errors = ctx.Barrier(4), ctx.Queue()
|
||||
workers = [ctx.Process(target=_migrate, args=(db_path, barrier, errors)) for _ in range(4)]
|
||||
for w in workers:
|
||||
w.start()
|
||||
for w in workers:
|
||||
w.join(timeout=60)
|
||||
|
||||
assert errors.empty(), errors.get()
|
||||
conn = sqlite3.connect(db_path)
|
||||
assert conn.execute(
|
||||
"SELECT COUNT(*) FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'"
|
||||
).fetchone()[0] == 1
|
||||
@@ -43,7 +43,7 @@ class TestRetryLogic:
|
||||
from starpunk.database import init_db
|
||||
from starpunk import create_app
|
||||
|
||||
app = create_app({'DATABASE_PATH': str(temp_db)})
|
||||
app = create_app({'DATABASE_PATH': str(temp_db), 'TESTING': True})
|
||||
init_db(app)
|
||||
|
||||
# Verify migrations table exists and has records
|
||||
@@ -302,12 +302,13 @@ class TestErrorHandling:
|
||||
|
||||
def test_rollback_failure_causes_system_exit(self, temp_db):
|
||||
"""Test that rollback failure raises SystemExit"""
|
||||
with patch('sqlite3.connect') as mock_connect:
|
||||
with patch('sqlite3.connect') as mock_connect, \
|
||||
patch('starpunk.migrations.execute_in_transaction',
|
||||
side_effect=Exception("Migration failed")):
|
||||
mock_conn = MagicMock()
|
||||
mock_connect.return_value = mock_conn
|
||||
|
||||
# Make both migration and rollback fail
|
||||
mock_conn.executescript.side_effect = Exception("Migration failed")
|
||||
mock_conn.rollback.side_effect = Exception("Rollback failed")
|
||||
mock_conn.execute.return_value.fetchone.return_value = (0,)
|
||||
|
||||
|
||||
@@ -0,0 +1,364 @@
|
||||
"""
|
||||
W3C Webmention conformance tests (v2.7.0)
|
||||
|
||||
https://www.w3.org/TR/webmention/ — section numbers in test names.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
from unittest.mock import patch
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from starpunk import webmention_ssrf
|
||||
from starpunk.webmention_processor import (
|
||||
TransientError, ValidationError, fetch_source, verify_link_presence,
|
||||
)
|
||||
from starpunk.webmention_worker import WebmentionWorker
|
||||
|
||||
PUBLIC = [(2, 1, 6, '', ('93.184.216.34', 0))]
|
||||
|
||||
|
||||
def original_url(request):
|
||||
"""The URL a pinned request was for (connections go to the checked IP)"""
|
||||
return f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
|
||||
BASE = 'https://t.example/test/1?x=1'
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def worker(app):
|
||||
return WebmentionWorker(app, db_path=app.config["DATABASE_PATH"],
|
||||
site_url=app.config["SITE_URL"])
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_web():
|
||||
"""Route safe_get through a MockTransport; all hosts resolve public"""
|
||||
routes = {}
|
||||
|
||||
def handler(request):
|
||||
return routes[original_url(request)](request)
|
||||
|
||||
with patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC), \
|
||||
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
|
||||
yield routes
|
||||
|
||||
|
||||
# 3.1.2 Sender discovers receiver Webmention endpoint
|
||||
|
||||
@pytest.mark.parametrize("header,expected", [
|
||||
('<https://e.example/wm>; rel="webmention"', 'https://e.example/wm'),
|
||||
('</wm>; rel=webmention', 'https://t.example/wm'),
|
||||
('<wm>; rel="other webmention"', 'https://t.example/test/wm'),
|
||||
('<wm>; title="a;b,c"; rel="webmention"', 'https://t.example/test/wm'),
|
||||
('<wm>; REL="WebMention"', 'https://t.example/test/wm'),
|
||||
('<a>; rel="other", </wm?q=1>; rel="webmention"', 'https://t.example/wm?q=1'),
|
||||
('<x>; rel="webmention-not"', None),
|
||||
('<x>; rel="not-webmention"', None),
|
||||
])
|
||||
def test_3_1_2_link_header(worker, header, expected):
|
||||
assert worker._parse_link_header(header, BASE) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("html,expected", [
|
||||
# <a> before <link>: first in document order wins
|
||||
('<a rel="webmention" href="/a">x</a><link rel="webmention" href="/l">', 'https://t.example/a'),
|
||||
('<link rel="webmention somethingelse" href="/l">', 'https://t.example/l'),
|
||||
('<link rel="Webmention" href="/l">', 'https://t.example/l'),
|
||||
# empty href is the target itself
|
||||
('<link rel="webmention" href="">', BASE),
|
||||
# no href: skipped
|
||||
('<link rel="webmention"><a rel="webmention" href="/a">', 'https://t.example/a'),
|
||||
# comments, escaped markup and other rels are ignored
|
||||
('<!-- <link rel="webmention" href="/bad"> -->'
|
||||
'<a rel="not-webmention" href="/bad"></a>'
|
||||
'<code><a rel="webmention" href="/bad"></code>'
|
||||
'<link rel="webmention" href="/ok">', 'https://t.example/ok'),
|
||||
('<p>nothing</p>', None),
|
||||
])
|
||||
def test_3_1_2_html(worker, html, expected):
|
||||
assert worker._parse_html_for_endpoint(html, BASE) == expected
|
||||
|
||||
|
||||
def test_3_1_2_header_before_html_and_relative_to_final_url(app, worker, mock_web):
|
||||
mock_web['https://t.example/old'] = lambda r: httpx.Response(
|
||||
301, headers={'Location': 'https://t.example/new/page'})
|
||||
mock_web['https://t.example/new/page'] = lambda r: httpx.Response(
|
||||
200, headers={'Link': '<wm>; rel="webmention"', 'Content-Type': 'text/html'},
|
||||
text='<link rel="webmention" href="/html">')
|
||||
with app.app_context():
|
||||
assert worker._discover_endpoint('https://t.example/old') == 'https://t.example/new/wm'
|
||||
|
||||
|
||||
def test_3_1_2_multiple_link_headers(app, worker, mock_web):
|
||||
mock_web['https://t.example/p'] = lambda r: httpx.Response(
|
||||
200, headers=[('Link', '</other>; rel="other"'), ('Link', '</wm>; rel="webmention"')])
|
||||
with app.app_context():
|
||||
assert worker._discover_endpoint('https://t.example/p') == 'https://t.example/wm'
|
||||
|
||||
|
||||
def test_3_1_2_server_error_is_retried_not_final(app, worker, mock_web):
|
||||
mock_web['https://t.example/p'] = lambda r: httpx.Response(503)
|
||||
with app.app_context(), pytest.raises(httpx.HTTPError):
|
||||
worker._discover_endpoint('https://t.example/p')
|
||||
|
||||
|
||||
# 4.5 SSRF: every redirect hop is checked
|
||||
|
||||
def test_4_5_redirect_to_private_address_blocked(app):
|
||||
def resolve(host, *a, **k):
|
||||
ip = '169.254.169.254' if host == 'metadata.internal' else '93.184.216.34'
|
||||
return [(2, 1, 6, '', (ip, 0))]
|
||||
|
||||
def handler(request):
|
||||
assert request.url.host == '93.184.216.34' # never connects to the private one
|
||||
return httpx.Response(302, headers={'Location': 'http://metadata.internal/'})
|
||||
|
||||
with app.app_context(), \
|
||||
patch("starpunk.webmention_ssrf.socket.getaddrinfo", side_effect=resolve), \
|
||||
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
|
||||
with pytest.raises(ValidationError, match="Blocked"):
|
||||
fetch_source('https://evil.example/', 'http://localhost:5000')
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ip", [
|
||||
'127.0.0.1', '10.1.2.3', '169.254.169.254', '0.0.0.0', '100.64.0.1',
|
||||
'::1', '::ffff:127.0.0.1', 'fe80::1', '224.0.0.1',
|
||||
])
|
||||
def test_4_5_blocked_addresses(ip):
|
||||
assert webmention_ssrf.is_private_ip(ip)
|
||||
|
||||
|
||||
def test_4_5_any_blocked_address_blocks(app):
|
||||
mixed = PUBLIC + [(10, 1, 6, '', ('::1', 0, 0, 0))]
|
||||
with app.app_context(), patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=mixed):
|
||||
with pytest.raises(webmention_ssrf.SSRFBlockedError):
|
||||
webmention_ssrf.validate_url_for_fetch('https://x.example/')
|
||||
|
||||
|
||||
# 3.2.2 Webmention verification
|
||||
|
||||
TARGET = 'https://me.example/note/abc'
|
||||
|
||||
|
||||
@pytest.mark.parametrize("content,content_type,expected", [
|
||||
('<a href="/note/abc#c">x</a>', 'text/html', True),
|
||||
('<img src="https://me.example/note/abc">', 'text/html; charset=utf-8', True),
|
||||
('<a href="https://me.example/note/abcdef">x</a>', 'text/html', False),
|
||||
('<p>https://me.example/note/abc</p>', 'text/html', False),
|
||||
('<!-- <a href="https://me.example/note/abc"> -->', 'text/html', False),
|
||||
('{"a": ["https://me.example/note/abc"]}', 'application/json', True),
|
||||
('{"a": "https://me.example/note/abcdef"}', 'application/json', False),
|
||||
('see https://me.example/note/abc', 'text/plain', True),
|
||||
])
|
||||
def test_3_2_2_exact_match(content, content_type, expected):
|
||||
assert verify_link_presence(content, TARGET, content_type, 'https://me.example/x') is expected
|
||||
|
||||
|
||||
def test_3_2_2_accept_header_and_status(app, mock_web):
|
||||
seen = {}
|
||||
|
||||
def page(request):
|
||||
seen['accept'] = request.headers.get('Accept')
|
||||
return httpx.Response(410)
|
||||
|
||||
mock_web['https://s.example/p'] = page
|
||||
with app.app_context():
|
||||
with pytest.raises(ValidationError, match="410"):
|
||||
fetch_source('https://s.example/p', 'http://localhost:5000')
|
||||
assert 'text/html' in seen['accept']
|
||||
|
||||
mock_web['https://s.example/p'] = lambda r: httpx.Response(500)
|
||||
with app.app_context(), pytest.raises(TransientError):
|
||||
fetch_source('https://s.example/p', 'http://localhost:5000')
|
||||
|
||||
|
||||
# 3.2.1 Request verification
|
||||
|
||||
def _post(client, source, target):
|
||||
return client.post('/webmention', data={'source': source, 'target': target})
|
||||
|
||||
|
||||
def test_3_2_1_target_must_be_a_note(app, client, sample_note):
|
||||
note_url = f"http://localhost:5000/note/{sample_note.slug}"
|
||||
assert _post(client, 'https://s.example/a', note_url).status_code == 202
|
||||
assert _post(client, 'https://s.example/a', 'http://localhost:5000/nope').status_code == 400
|
||||
assert _post(client, 'https://s.example/a', 'http://localhost:5000/').status_code == 400
|
||||
assert _post(client, 'https://s.example/a', 'https://other.example/note/x').status_code == 400
|
||||
assert _post(client, note_url, note_url).status_code == 400
|
||||
assert _post(client, 'ftp://s.example/a', note_url).status_code == 400
|
||||
|
||||
|
||||
def test_3_2_1_host_case_and_fragment(app, client, sample_note):
|
||||
note_url = f"http://LOCALHOST:5000/note/{sample_note.slug}"
|
||||
assert _post(client, 'https://s.example/a', note_url + '#one').status_code == 202
|
||||
assert _post(client, 'https://s.example/a', note_url + '#two').status_code == 202
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
rows = db.execute("SELECT target_url FROM webmentions").fetchall()
|
||||
assert rows == [(note_url,)]
|
||||
|
||||
|
||||
# 3.2.4 Updating existing Webmentions
|
||||
|
||||
def _insert_mention(app, note_id, **cols):
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
fields = {'source_url': 'https://s.example/a', 'target_url': 'http://localhost:5000/n',
|
||||
'target_note_id': note_id, **cols}
|
||||
db.execute(f"INSERT INTO webmentions ({', '.join(fields)}) VALUES ({', '.join('?' * len(fields))})",
|
||||
tuple(fields.values()))
|
||||
db.commit()
|
||||
return db
|
||||
|
||||
|
||||
VALID = {'valid': True, 'mention_type': 'reply', 'author_name': 'A', 'author_url': 'https://a.example/',
|
||||
'author_photo': None, 'content_snippet': 'hi'}
|
||||
|
||||
|
||||
def test_3_2_4_update_keeps_moderation(app, worker, sample_note):
|
||||
db = _insert_mention(app, sample_note.id, validation_status='valid', moderation_status='hidden',
|
||||
validated_at='2026-01-01')
|
||||
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
|
||||
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
|
||||
assert db.execute("SELECT validation_status, moderation_status FROM webmentions").fetchone() == ('valid', 'hidden')
|
||||
|
||||
|
||||
def test_3_2_4_new_mention_gets_default_moderation(app, worker, sample_note):
|
||||
db = _insert_mention(app, sample_note.id)
|
||||
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
|
||||
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
|
||||
assert db.execute("SELECT validation_status, moderation_status FROM webmentions").fetchone() == ('valid', 'approved')
|
||||
|
||||
|
||||
def test_3_2_4_transient_error_keeps_valid_mention(app, worker, sample_note):
|
||||
db = _insert_mention(app, sample_note.id, validation_status='pending', moderation_status='approved',
|
||||
validated_at='2026-01-01', content_snippet='kept')
|
||||
with patch("starpunk.webmention_worker.process_webmention", side_effect=TransientError("timeout")):
|
||||
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
|
||||
assert db.execute("SELECT validation_status, content_snippet FROM webmentions").fetchone() == ('valid', 'kept')
|
||||
|
||||
|
||||
def test_3_2_4_gone_source_removes_mention(app, worker, sample_note):
|
||||
db = _insert_mention(app, sample_note.id, validation_status='pending', moderation_status='approved',
|
||||
validated_at='2026-01-01')
|
||||
with patch("starpunk.webmention_worker.process_webmention",
|
||||
side_effect=ValidationError("Source returned 410")):
|
||||
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
|
||||
assert db.execute("SELECT validation_status FROM webmentions").fetchone() == ('invalid',)
|
||||
|
||||
|
||||
# 3.1.4 Updated posts / 3.1.5 Deleted posts
|
||||
|
||||
def _queue(app):
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
return db.execute(
|
||||
"SELECT target_url, status, source_note_id FROM webmention_queue ORDER BY target_url"
|
||||
).fetchall()
|
||||
|
||||
|
||||
def _sent(app):
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
db.execute("UPDATE webmention_queue SET status = 'sent'")
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_3_1_4_update_resends_previous_targets(app):
|
||||
from starpunk.notes import create_note, update_note
|
||||
|
||||
with app.test_request_context():
|
||||
note = create_note("Reply", published=True, in_reply_to="https://a.example/post")
|
||||
_sent(app)
|
||||
update_note(id=note.id, content="Now about [b](https://b.example/x)", in_reply_to="")
|
||||
targets = {t: s for t, s, _ in _queue(app)}
|
||||
assert targets == {'https://a.example/post': 'pending', 'https://b.example/x': 'pending'}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("soft", [True, False])
|
||||
def test_3_1_5_delete_resends_and_returns_410(app, client, soft):
|
||||
from starpunk.notes import create_note, delete_note
|
||||
|
||||
with app.test_request_context():
|
||||
note = create_note("See [a](https://a.example/x)", published=True)
|
||||
_sent(app)
|
||||
delete_note(id=note.id, soft=soft)
|
||||
|
||||
rows = _queue(app)
|
||||
assert [(t, s) for t, s, _ in rows] == [('https://a.example/x', 'pending')]
|
||||
if not soft:
|
||||
assert rows[0][2] is None # detached, so it survives the delete
|
||||
assert client.get(f"/note/{note.slug}").status_code == 410
|
||||
assert client.get("/note/never-existed").status_code == 404
|
||||
|
||||
|
||||
# 4.5 Connections go to the address that was checked (no DNS rebinding)
|
||||
|
||||
def test_4_5_connection_pinned_to_checked_address(app):
|
||||
seen = []
|
||||
|
||||
def handler(request):
|
||||
seen.append((str(request.url), request.headers['host'],
|
||||
request.extensions.get('sni_hostname')))
|
||||
return httpx.Response(200, text='ok')
|
||||
|
||||
with app.app_context(), \
|
||||
patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC) as resolve, \
|
||||
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
|
||||
webmention_ssrf.safe_get('https://site.example:8443/p?q=1', {})
|
||||
webmention_ssrf.safe_post('http://site.example/wm', {'source': 's'}, {})
|
||||
|
||||
assert seen == [
|
||||
('https://93.184.216.34:8443/p?q=1', 'site.example:8443', 'site.example'),
|
||||
('http://93.184.216.34/wm', 'site.example', None),
|
||||
]
|
||||
assert resolve.call_count == 2 # one lookup per request, then pinned
|
||||
|
||||
|
||||
def test_4_5_ipv4_preferred(app):
|
||||
with app.app_context(), patch("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||
return_value=[(10, 1, 6, '', ('2606:2800::1', 0, 0, 0))] + PUBLIC):
|
||||
assert webmention_ssrf.safe_resolve('x.example') == '93.184.216.34'
|
||||
|
||||
|
||||
# 3.1.4 A change in the responses a note shows is an update of the note
|
||||
|
||||
def _note_with_sent_mention(app):
|
||||
from starpunk.notes import create_note
|
||||
with app.test_request_context():
|
||||
note = create_note("See [a](https://a.example/x)", published=True)
|
||||
_sent(app)
|
||||
db = sqlite3.connect(app.config['DATABASE_PATH'])
|
||||
db.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) VALUES (?, ?, ?)",
|
||||
('https://s.example/reply', f'http://localhost:5000/note/{note.slug}', note.id))
|
||||
db.commit()
|
||||
return note, db
|
||||
|
||||
|
||||
def test_3_1_4_new_response_resends(app, worker):
|
||||
note, db = _note_with_sent_mention(app)
|
||||
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
|
||||
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
|
||||
assert [(t, s) for t, s, _ in _queue(app)] == [('https://a.example/x', 'pending')]
|
||||
|
||||
|
||||
def test_3_1_4_unchanged_response_does_not_resend(app, worker):
|
||||
note, db = _note_with_sent_mention(app)
|
||||
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
|
||||
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
|
||||
_sent(app)
|
||||
db.execute("UPDATE webmentions SET validation_status = 'pending'") # re-sent by source
|
||||
db.commit()
|
||||
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
|
||||
assert [(t, s) for t, s, _ in _queue(app)] == [('https://a.example/x', 'sent')]
|
||||
|
||||
|
||||
def test_3_1_4_moderation_change_resends(app, worker):
|
||||
from starpunk.webmention_queries import update_webmention_status
|
||||
note, db = _note_with_sent_mention(app)
|
||||
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
|
||||
worker._validate_webmention(1, 'https://s.example/reply', 'x', note.id)
|
||||
_sent(app)
|
||||
with app.app_context():
|
||||
update_webmention_status([1], 'approved') # already approved: no change
|
||||
assert _queue(app)[0][1] == 'sent'
|
||||
with app.app_context():
|
||||
update_webmention_status([1], 'hidden')
|
||||
assert _queue(app)[0][1] == 'pending'
|
||||
@@ -7,11 +7,15 @@ whenever any app context ended.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
import httpx
|
||||
import time
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from starpunk.webmention_worker import WebmentionWorker
|
||||
|
||||
PUBLIC = [(2, 1, 6, '', ('93.184.216.34', 0))]
|
||||
|
||||
|
||||
def _wait_for(predicate, timeout=5):
|
||||
deadline = time.time() + timeout
|
||||
@@ -33,15 +37,18 @@ def test_worker_sends_queued_webmention(app):
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
page = MagicMock(status_code=200, url="https://example.org/post",
|
||||
headers={"Link": '<https://example.org/wm>; rel="webmention"'})
|
||||
accepted = MagicMock(status_code=202, text="")
|
||||
posts = []
|
||||
|
||||
def target_site(request):
|
||||
if request.method == "POST":
|
||||
posts.append((request.headers["host"], request.url.path, request.content))
|
||||
return httpx.Response(202)
|
||||
return httpx.Response(200, headers={"Link": '</wm>; rel="webmention"'})
|
||||
|
||||
worker = WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"],
|
||||
poll_interval=0.05)
|
||||
with patch("starpunk.webmention_ssrf.socket.gethostbyname", return_value="93.184.216.34"), \
|
||||
patch("starpunk.webmention_worker.httpx.get", return_value=page), \
|
||||
patch("starpunk.webmention_worker.httpx.post", return_value=accepted) as post:
|
||||
with patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC), \
|
||||
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(target_site)):
|
||||
worker.start()
|
||||
try:
|
||||
def status():
|
||||
@@ -51,7 +58,8 @@ def test_worker_sends_queued_webmention(app):
|
||||
return row
|
||||
assert _wait_for(lambda: status()[0] != "pending"), status()
|
||||
assert status() == ("sent", None)
|
||||
post.assert_called_once()
|
||||
assert [(h, p) for h, p, _ in posts] == [("example.org", "/wm")]
|
||||
assert b"source=" in posts[0][2] and b"target=" in posts[0][2]
|
||||
finally:
|
||||
worker.stop()
|
||||
worker.join(timeout=2)
|
||||
@@ -70,3 +78,56 @@ def test_worker_survives_app_context_teardown(app):
|
||||
finally:
|
||||
worker.stop()
|
||||
worker.join(timeout=2)
|
||||
|
||||
|
||||
def test_only_one_worker_processes_queues(app):
|
||||
"""Each gunicorn process starts a worker; only one may hold the queue lock"""
|
||||
kwargs = dict(db_path=app.config["DATABASE_PATH"], site_url=app.config["SITE_URL"])
|
||||
first = WebmentionWorker(app, **kwargs)
|
||||
second = WebmentionWorker(app, **kwargs)
|
||||
try:
|
||||
assert first._acquire_lock()
|
||||
assert not second._acquire_lock()
|
||||
first._release_lock()
|
||||
assert second._acquire_lock()
|
||||
finally:
|
||||
first._release_lock()
|
||||
second._release_lock()
|
||||
|
||||
|
||||
def test_author_photo_with_alt_text_is_a_string():
|
||||
"""mf2py returns u-photo with alt as a dict; the database needs a URL string"""
|
||||
import mf2py
|
||||
from starpunk.webmention_processor import extract_author
|
||||
|
||||
html = """
|
||||
<div class="h-entry">
|
||||
<span class="p-author h-card">
|
||||
<a class="u-url p-name" href="https://a.example/">A Person</a>
|
||||
<img class="u-photo" src="https://a.example/me.jpg" alt=""></span>
|
||||
<a href="https://site.example/note/1">target</a>
|
||||
</div>"""
|
||||
author = extract_author(mf2py.parse(doc=html, url="https://src.example/"),
|
||||
"https://src.example/")
|
||||
assert author == {
|
||||
"author_name": "A Person",
|
||||
"author_url": "https://a.example/",
|
||||
"author_photo": "https://a.example/me.jpg",
|
||||
}
|
||||
|
||||
|
||||
def test_author_urls_must_be_http():
|
||||
"""A javascript: author URL would be rendered as a clickable link (XSS)"""
|
||||
import mf2py
|
||||
from starpunk.webmention_processor import extract_author
|
||||
|
||||
html = """
|
||||
<div class="h-entry">
|
||||
<span class="p-author h-card">
|
||||
<a class="u-url p-name" href="javascript:alert(1)">Evil</a>
|
||||
<img class="u-photo" src="data:image/svg+xml,x" alt=""></span>
|
||||
</div>"""
|
||||
author = extract_author(mf2py.parse(doc=html, url="https://src.example/"),
|
||||
"https://src.example/")
|
||||
assert author["author_url"] == "https://src.example/"
|
||||
assert author["author_photo"] is None
|
||||
Reference in new issue
Block a user