Compare commits

...
57 Commits
Author SHA1 Message Date
Phil fa551de197 Merge branch 'fix/v2.10.1-websub-hardening' for v2.10.1 release
Build Container / build (push) Successful in 7m53s
2026-11-03 11:23:36 -07:00
PhilandClaude Opus 5.5 b2043286fa fix: WebSub hardening from the adversarial review (v2.10.1)
- Per-item error handling in the worker; hard time limits on requests
- Notes with photos announced after attach; feed cache/ETag follow
  content_version
- Keep failing subscriptions until lease end; retry DNS and render errors;
  hub.mode=denied for gone topics
- Spec 5.1: decode unreserved chars, echo requested topic, ignore lease and
  secret on unsubscribe, case-insensitive Content-Type, refuse empty secret
- Limits count pending requests per callback address; callback validation;
  request_id guards the verification race (migration 017)
- Render once per topic per pass; likes only push /likes; Vary: Accept

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:23:36 -07:00
Phil 409b8b31b4 Merge branch 'feat/v2.10.0-websub' for v2.10.0 release
Build Container / build (push) Successful in 7m29s
2026-11-03 10:33:41 -07:00
PhilandClaude Opus 5.5 b520b8b4d3 feat: WebSub with a built-in hub (v2.10.0)
ADR-071. Topics (/, /feed.atom, /feed.rss, /feed.json, /likes, /tag/<name>)
advertise the hub and themselves in Link headers and embedded links. Note
changes bump a content version (fresh feeds in every process) and queue
deliveries. /websub accepts subscriptions for these topics only; the worker
verifies intent, delivers signed content with retries, and prunes expired
leases, all through the SSRF-safe pinned fetcher. WEBSUB_HUB_URL switches
to an external hub (publish pings). Also: the memory monitor no longer
stops after the first request; RSS self link is /feed.rss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:33:41 -07:00
PhilandClaude Opus 5.5 720a1e6d6f docs: ADR-071 WebSub publishing with a built-in hub (proposed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:23:00 -07:00
Phil c89c2cd5ca Merge branch 'feat/v2.9.0-micropub' for v2.9.0 release
Build Container / build (push) Successful in 7m28s
2026-11-03 10:11:23 -07:00
PhilandClaude Opus 5.5 ab6ff60f5c feat: Micropub review fixes (v2.9.0)
- mp-syndicate-to/mp-slug read from JSON properties; absent means no
  crossposting
- Sanitize rendered HTML (nh3 allowlist)
- q=source/q=category need a posting scope; draft scope creates drafts;
  private/unlisted visibility saved as drafts
- Photos that can't be fetched or found fail the request; remote photos
  via the SSRF-checked, pinned fetcher
- JSON scalars wrapped, update delete values must be arrays
- bookmark-of stored; HTML content returned as {html} (migration 015)
- published stored as UTC, bad dates 400; malformed JSON 400
- Auth: case-insensitive Bearer, header+body 400, WWW-Authenticate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:11:23 -07:00
Phil 57682165a4 Merge branch 'fix/v2.8.1-migration-race' for v2.8.1 release
Build Container / build (push) Successful in 7m34s
2026-11-03 09:58:20 -07:00
PhilandClaude Opus 5.5 8c51b52ed6 fix: Hold the migration lock for the whole run (v2.8.1)
create_migrations_table() committed right after BEGIN IMMEDIATE, and
executescript() commits before running, so gunicorn workers applied
migrations concurrently; the v2.8.0 deploy failed to boot once with
'duplicate column name'. Run migration statements with conn.execute inside
the transaction and don't commit when creating the tracking table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:58:20 -07:00
Phil 42c77e34b7 Merge branch 'feat/v2.8.0-microformats' for v2.8.0 release
Build Container / build (push) Successful in 9m35s
2026-11-03 09:36:53 -07:00
PhilandClaude Opus 5.5 438465592c feat: Microformats2 review fixes (v2.8.0)
- Responses as u-comment/u-like/u-repost/u-bookmark/u-mention h-cites
  with author h-cards; source's own url and published (migration 014)
- dt-* and Micropub published in RFC 3339 UTC
- Title only for a real Markdown heading (hashtag notes stay notes)
- Ignore webmentions from a note's own syndicated copies
- Micropub/IndieAuth links independent of Webmention; rel=me/rel=author
- Representative h-card and authorship algorithm (starpunk/mf2_utils.py)
- Tag/likes h-feeds get url and author; previews get u-uid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:36:49 -07:00
Phil c8a7ef309f Merge branch 'feat/v2.7.2-webmention-shoulds' for v2.7.2 release
Build Container / build (push) Successful in 9m28s
2026-11-03 09:01:42 -07:00
PhilandClaude Opus 5.5 0640730416 feat: Pin webmention connections; re-send when responses change (v2.7.2)
- Connect to the address that passed the SSRF check (Host header and SNI
  carry the hostname, so TLS still verifies it), closing the DNS rebinding
  gap (spec 4.5). The webmention POST uses the same path.
- Treat a change in the responses a note shows as an update and re-send its
  webmentions (spec 3.1.4), from validation and from admin moderation.
  Unchanged re-validations send nothing, so there is no ping-pong.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:01:38 -07:00
Phil 51f711ea4a Merge branch 'ci/v2.7.1-pytest' for v2.7.1 release
Build Container / build (push) Successful in 9m13s
2026-11-03 08:38:40 -07:00
PhilandClaude Opus 5.5 6a76ee26af ci: Run the test suite before building release images (v2.7.1)
Add a 'test' stage to the Containerfile (production base image plus pytest)
and build it in the release workflow before the push step, so a failing test
stops the release.

Fix the eight tests that were failing: media validation returns a PIL Image
since v1.5.0, and the migration test needs TESTING to skip production config
checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:28:08 -07:00
Phil 43ec7cb8be Merge branch 'feat/v2.7.0-webmention-spec' for v2.7.0 release
Build Container / build (push) Successful in 11s
2026-11-03 08:07:12 -07:00
PhilandClaude Opus 5.5 4aaa7a8122 feat: W3C Webmention compliance (v2.7.0)
Sending: RFC 8288 Link header parsing, rel token matching and document
order for <link>/<a>, endpoint re-discovery on every send, re-send to
previous targets on update/delete/undelete, 410 Gone for deleted notes,
no redirect following on the POST, 429 retried.

Receiving: exact target match per media type, Accept header, final URL
for relative links, only published notes accepted, fragments ignored,
moderation kept on updates, transient errors keep valid mentions.

Security: every redirect hop SSRF-checked, all resolved addresses
checked, broader blocklist, total time and size limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:07:04 -07:00
Phil 0cfc9965df Merge branch 'fix/v2.6.9-webmention-processing' for v2.6.9 release
Build Container / build (push) Successful in 12s
2026-11-03 07:53:19 -07:00
PhilandClaude Opus 5.5 106495d114 fix: Webmention processing works end to end (v2.6.9)
- Normalise mf2 author values: u-photo with alt text is a dict in mf2py and
  broke the database update for most real senders
- Pass str(response.url) to urljoin during endpoint discovery; the TypeError
  marked every target as having no endpoint. Retry on discovery errors
- Hold a cross-process flock so only one gunicorn process runs the worker
- Keep author URL/photo only if http(s) (XSS via javascript: URLs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:53:19 -07:00
Phil 3c1e109358 Merge branch 'fix/v2.6.8-webmention-worker' for v2.6.8 release
Build Container / build (push) Successful in 20s
2026-11-03 07:41:11 -07:00
PhilandClaude Opus 5.5 d43361d505 fix: Webmention worker runs inside the app context (v2.6.8)
The worker thread called the SSRF checks, which read current_app.config,
without an app context, so every incoming and outgoing webmention failed with
'Working outside of application context'. A teardown_appcontext hook also
stopped the worker whenever a request ended.

Pass the app to WebmentionWorker and run each poll in app.app_context(), like
SyndicationWorker, and drop the teardown hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:41:06 -07:00
Phil f924413934 Merge branch 'fix/v2.6.7-admin-layout' for v2.6.7 release
Build Container / build (push) Successful in 7s
2026-11-02 18:24:15 -07:00
PhilandClaude Opus 5.5 4b2ae4c7fc fix: Admin pages line up with the header (v2.6.7)
Every page sat in the 40rem reading column, so the admin wrapper's 60rem
never applied and the notes table, which can't shrink that far, ran past
the header's right edge. Admin pages now set a 60rem frame for header,
content and footer; post previews wrap, and tables scroll sideways on
narrow screens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:24:14 -07:00
Phil d3148063e3 Merge branch 'fix/v2.6.6-inline-image-crosspost' for v2.6.6 release
Build Container / build (push) Successful in 15s
2026-11-02 18:12:51 -07:00
PhilandClaude Opus 5.5 39c8259f3f fix: Crossposts include photos embedded in the post text (v2.6.6)
Some Micropub clients (OutPost) upload a photo to the media endpoint and
then embed it in content instead of sending it as `photo`. Those posts
have no attached media, so crossposts went out without the image.

Images in the text that are in this site's media library are now sent
like attached photos, with their alt text, and GoToSocial's text no
longer carries the image Markdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:11:27 -07:00
Phil 1ce88b38a5 Merge branch 'fix/v2.6.5-micropub-media' for v2.6.5 release
Build Container / build (push) Successful in 6s
2026-11-01 10:53:37 -07:00
PhilandClaude Opus 5.5 4ffe36f9d3 fix: Photos without alt text use the post's text (v2.6.5)
Uncaptioned photos rendered alt="Image", which mf2 parsers report as
the photo's alt text. They now take the post's text (excerpt, up to 150
characters), falling back to "Image" only for text-free posts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 10:44:54 -07:00
PhilandClaude Opus 5.5 a551770c76 fix: Replies and photo posts read as the right post type (v2.6.5)
Checked rendered pages with mf2py against W3C Post Type Discovery:

- Note pages put u-in-reply-to inside a bare h-cite, making it the
  h-cite's property, so replies parsed as notes (or photo posts). The
  reply context is now the entry's u-in-reply-to h-cite.
- Home page previews had no reply markup; they now show "In reply to"
  with u-in-reply-to.
- Previews always emit e-content, so a text-free photo post can't get
  an implied name.
- Text-free page titles follow PTD order (Repost, Like, Reply, Photo).

tests/test_post_types.py runs PTD on parsed note pages and the home page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 10:06:44 -07:00
PhilandClaude Opus 5.5 d0bfc80c2e fix: Micropub photo posts and failed creates (v2.6.5)
- A create that failed after the note was written (e.g. the same photo
  twice) answered 500 but left the post published without photos.
  Photos are now resolved first, and a failed note is removed.
- Pooled connections roll back unfinished writes, so one failure no
  longer locks the database for the next ~10 s.
- The same photo twice in one post is shown once.
- Photo posts no longer need text (Micropub doesn't require content,
  and q=config already advertised a photo-only post type). Updates may
  clear the text while photos remain. Crossposts start with the link;
  the page is titled "Photo".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 09:50:30 -07:00
PhilandClaude Opus 5.5 d16463ef8a Merge branch 'fix/v2.6.4-tag-order' for v2.6.4 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:59:19 -07:00
PhilandClaude Opus 5.5 4b56f7cd4b fix: Tags keep the order they were entered in (v2.6.4)
add_tags_to_note() rewrites note_tags in entry order on every save, so
note_tags.id already records it; get_note_tags(entered_order=True) reads
it back. Crossposts (Threads topic, which hashtags fit on Bluesky), the
editor and Micropub q=source/update use it, so saving no longer
reorders tags alphabetically. Public pages stay alphabetical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:59:18 -07:00
PhilandClaude Opus 5.5 2474628646 Merge branch 'fix/v2.6.3-link-cards' for v2.6.3 release
Build Container / build (push) Successful in 8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:25 -07:00
PhilandClaude Opus 5.5 0fe341161e feat: Crosspost cards show the note's own link (v2.6.3)
When a note links somewhere (an article, say), the first link gets the
preview card instead of the note (ADR-070). Bluesky cards are built from
the page's OG/Twitter/HTML title, description and image (thumb shrunk
under 1 MB); Threads previews the first link in the text. Markdown links
no longer lose their URL on Bluesky (link text becomes a facet) or Threads
("text (url)"); Threads keeps to Meta's 5-link limit. Page fetches are
public-address only, checked at every redirect, size-capped, and never
fail the post.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:24 -07:00
PhilandClaude Opus 5.5 a62077ab30 Merge branch 'fix/v2.6.2-threads-link-preview' for v2.6.2 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 fade75fd2d fix: Let Threads build the link preview itself (v2.6.2)
An explicit link_attachment for a URL Meta hasn't fetched yet fails at
threads_publish with "Invalid Link Attachment" (subcode 4279047), which
is every new note; it only went out on the retry. Tested live with fresh
URLs: with link_attachment it fails, without it it publishes first time
and Threads still sets link_attachment_url from the first link in the
text. Reposts likewise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 0dd59fbc18 Merge branch 'fix/v2.6.1-threads-publish-retry' for v2.6.1 release
Build Container / build (push) Successful in 6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:18:11 -06:00
PhilandClaude Opus 5.5 11e09a0d02 fix: Retry Threads publish errors, show full error detail (v2.6.1)
Meta's threads_publish returned a one-off HTTP 400 "Fatal" for a plain
text post; the same post published fine on retry. Publish-step errors
are now retryable unless they are auth/permission errors (codes 10, 190,
200); the retry's existing-post check prevents doubles. Errors carry the
Graph code, subcode, user message and fbtrace id. A short pause before
the first container status poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:18:11 -06:00
PhilandClaude Opus 5.5 565adaeecf Merge branch 'feature/v2.6.0-threads' for v2.6.0 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:54:15 -06:00
PhilandClaude Opus 5.5 53d10ed271 feat: Crosspost notes to Threads (v2.6.0)
Third crossposting target on the ADR-065 loop (ADR-069). Text posts get a
link preview, 1 image posts as IMAGE and 2-4 as a CAROUSEL (Meta fetches
them from /media/), with alt text and the first tag as topic_tag. Posts are
shortened to 500 characters, counting non-ASCII as UTF-8 bytes. Edits don't
follow (the API has no edit); unpublish and delete remove the post. Retried
creates check for an existing post first.

The 60-day token is refreshed daily from the worker loop and kept in
DATA_PATH/threads_token.json under a file lock; a new configured token
replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:46:56 -06:00
PhilandClaude Opus 5.5 e8c1b03980 feat: Likes and reposts (v2.5.0)
Build Container / build (push) Successful in 6s
Micropub like-of/repost-of with optional comment; likes on /likes, off the
home page and feeds; reposts in the stream. Native favourites/boosts on
GoToSocial and likes/reposts/quotes on Bluesky when the target is on that
network, link posts for other reposts. Fix double slash in outgoing
webmention source URLs. ADR-068.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:16:21 -06:00
PhilandClaude Opus 5.5 0d3a5cd519 feat: Micropub update, delete/undelete, drafts, replies (v2.4.0)
Build Container / build (push) Successful in 6s
Update (replace/add/delete) for content, name, category, photo,
in-reply-to, post-status; soft delete with undelete; post-status drafts;
in-reply-to; multipart and remote photos; q=source lists and properties[];
q=category; media scope. Fix media endpoint 404 at the advertised
/micropub/media path. ADR-067.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 19:53:18 -06:00
Phil 88088e10e1 Merge branch 'fix/v2.3.3-micropub-title-source-url' for v2.3.3 release
Build Container / build (push) Successful in 7s
2026-10-31 19:34:24 -06:00
PhilandClaude Opus 5.5 391bdb9db3 fix: Micropub titles, q=source URL, single title display (v2.3.3)
Micropub `name` was dropped; store it as the note's '# Heading' first
line, which is how StarPunk titles work. q=source returned /notes/<slug>
(a 404) and a name for every note; it now returns the real permalink and
name only for titled notes, without the heading in content. Titled notes
no longer show their title twice (Note.body_html, excerpt).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 19:34:23 -06:00
PhilandClaude Opus 5.5 c51cf726b0 fix: Header shows Admin when logged in (v2.3.2)
Build Container / build (push) Successful in 9s
Public pages never checked the session, so they always showed Log in.
Set g.me in before_request when a session cookie is present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 18:57:12 -06:00
PhilandClaude Opus 5.5 2616ef5060 fix: Put Log in back in the header nav (v2.3.1)
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 18:46:33 -06:00
Phil 0ae16bd1fc Merge branch 'feature/v2.3.0-style-pass' for v2.3.0 release
Build Container / build (push) Successful in 8s
2026-10-31 18:43:20 -06:00
PhilandClaude Opus 5.5 cc09098741 style: Public page pass (v2.3.0)
Fix note metadata inheriting page-footer styles, show full notes instead of
HTML cut at 300 chars, auto-link bare URLs, one-line meta, SITE_NAME in the
header and titles, serif reading font, dark mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 18:24:29 -06:00
Phil 16925663a3 Merge branch 'feature/v2.2.0-bluesky' for v2.2.0 release
Build Container / build (push) Successful in 7s
2026-10-31 14:19:17 -06:00
PhilandClaude Opus 5.5 e92c27a84c feat: Crosspost notes to Bluesky (v2.2.0)
BlueskyClient behind the ADR-065 reconcile loop: plain text with link and
hashtag facets, images under 1 MB with alt text, link cards, app-password
sessions with refresh. Deletes follow; edits don't (ADR-066).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 14:19:17 -06:00
PhilandClaude Opus 5.5 ed63e70c46 Merge branch 'feature/v2.1.0-gotosocial-syndication' for v2.1.0 release
Build Container / build (push) Successful in 8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:46:28 -06:00
PhilandClaude Opus 5.5 57b02ad6c7 feat: Crosspost notes to GoToSocial (v2.1.0)
Published notes are copied to a GoToSocial (Mastodon-API) account and
edits, unpublishing and deletes follow them (ADR-065). On by default for
new notes, untickable in the editor and selectable from Micropub via
mp-syndicate-to. A background worker reconciles each (note, target)
row with atomic claims, an idempotency key and a version so edits during
a send are not lost. Synced copies are linked with u-syndication.

Also fixes the Micropub Location header and search result links, which
pointed at /notes/<slug> (404), and the nested delete form on the edit
page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:46:26 -06:00
PhilandClaude Opus 5.5 94e8c672df release: v2.0.2 (CI registry login)
Build Container / build (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:20:01 -06:00
PhilandClaude Opus 5.5 011f7cc87d ci: Log in to the registry with a write:package token
The automatic Actions token is refused by the container registry
(run 107: unauthorized). Use the REGISTRY_TOKEN repository secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:13:43 -06:00
PhilandClaude Opus 5.5 f46130569f Merge branch 'fix/v2.0.1-ci-and-feed-urls' for v2.0.1 release
Build Container / build (push) Failing after 12s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:12:29 -06:00
PhilandClaude Opus 5.5 b4e6cf3cc3 fix: Build absolute URLs from SITE_URL, not the request (v2.0.1)
Behind Caddy the app is reached over plain HTTP, so url_for(_external=True)
produced http:// links on the https site: the RSS/OPML <link>s and the
microformats u-url/u-uid permalinks that IndieWeb parsers treat as
canonical. Templates now use absolute_url_for(), shared with the
IndieAuth metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:12:28 -06:00
PhilandClaude Opus 5.5 2d0568d9c2 Build the container image with the server's rootless BuildKit
The runner no longer has a Docker daemon (it gave every workflow root on
the server), and this workflow never worked anyway: it logged in to
gitea:3000, which the daemon couldn't resolve. Log in to
git.thesatelliteoflove.com and build through the BuildKit socket instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 11:53:59 -06:00
PhilandClaude Opus 5.5 f117913797 Merge branch 'feature/v2.0.0-oidc' for v2.0.0 release
Build Container / build (push) Failing after 2m38s
Includes the v1.6.0-rc.2 source recovered from production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 11:45:20 -06:00
106 changed files with 10527 additions and 1254 deletions

No files matched your search

+39
View File
@@ -50,6 +50,27 @@ SESSION_LIFETIME=30
# Lifetime of Micropub access tokens StarPunk issues, in days (default: 90)
INDIEAUTH_TOKEN_LIFETIME_DAYS=90
# =============================================================================
# CROSSPOSTING (optional, v2.1.0)
# =============================================================================
# GoToSocial or Mastodon account to copy new notes to. Create an app token
# with scopes: write:statuses write:media
GOTOSOCIAL_URL=
GOTOSOCIAL_ACCESS_TOKEN=
# Name shown in the editor and Micropub apps (default: GoToSocial)
GOTOSOCIAL_NAME=GoToSocial
# public, unlisted, private (default: public)
GOTOSOCIAL_VISIBILITY=public
# Threads account (v2.6.0). A long-lived user token with threads_basic,
# threads_content_publish and threads_delete; StarPunk refreshes it daily and
# keeps the current one in DATA_PATH/threads_token.json.
# How to get one: docs/design/v2.6.0/2026-10-31-threads-crossposting.md
THREADS_ACCESS_TOKEN=
# Your Threads username (for the Micropub syndicate-to uid)
THREADS_USERNAME=
# =============================================================================
# DATA STORAGE
# =============================================================================
@@ -104,6 +125,24 @@ WORKER_TIMEOUT=30
# Max requests per worker before restart (prevents memory leaks)
MAX_REQUESTS=1000
# Behind a reverse proxy (Caddy, nginx): take the client address from
# X-Forwarded-For. Without it every visitor shares the proxy's address, so one
# busy client can rate-limit everyone on /webmention and /websub.
# Only enable when the proxy sets X-Forwarded-For itself (Caddy does).
TRUSTED_PROXY=true
# =============================================================================
# WEBSUB (v2.10.0)
# =============================================================================
# Built-in hub at /websub for this site's feeds. Set WEBSUB_HUB_URL to use
# another hub instead (StarPunk then pings it on changes).
WEBSUB_ENABLED=true
# WEBSUB_HUB_URL=https://hub.example.com/
# WEBSUB_MAX_SUBSCRIPTIONS=1000
# Per callback server (resolved address)
# WEBSUB_MAX_PER_HOST=25
# =============================================================================
# DEVELOPMENT OPTIONS
# =============================================================================
+30 -42
View File
@@ -1,58 +1,46 @@
# Gitea Actions workflow for StarPunk
# Builds and pushes container images on version tags
# Runs the tests, then builds and pushes container images on version tags.
#
# The runner has no Docker daemon (that would give workflows root on the
# server). Images are built by the rootless BuildKit container next to it,
# reached through its socket.
name: Build Container
on:
# Trigger on version tags
push:
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'
# Allow manual trigger from Gitea UI
workflow_dispatch:
env:
REGISTRY: git.thesatelliteoflove.com
BUILDKIT: unix:///run/buildkit/buildkit/buildkitd.sock
jobs:
build:
runs-on: docker
steps:
- name: Install dependencies
run: |
if command -v apk > /dev/null; then
apk add --no-cache nodejs npm docker git
elif command -v apt-get > /dev/null; then
apt-get update && apt-get install -y nodejs npm docker.io git
elif command -v yum > /dev/null; then
yum install -y nodejs npm docker git
fi
run: apk add --no-cache nodejs git docker-cli docker-cli-buildx
- name: Checkout
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Extract registry URL
id: registry
run: |
# Extract hostname from server URL (remove protocol)
REGISTRY_URL=$(echo "${{ github.server_url }}" | sed 's|https://||' | sed 's|http://||')
echo "url=${REGISTRY_URL}" >> $GITHUB_OUTPUT
- name: Login to Gitea Container Registry
uses: docker/login-action@v3
with:
registry: ${{ steps.registry.outputs.url }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# The automatic Actions token can't log in to the container registry, so
# this uses a personal access token with only the write:package scope,
# stored as the repository secret REGISTRY_TOKEN.
- name: Log in to the Gitea registry
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "$REGISTRY" -u "${{ github.repository_owner }}" --password-stdin
- name: Use the BuildKit socket
run: docker buildx create --name ci --driver remote "$BUILDKIT" --use
# Builds the Containerfile's test stage, which runs pytest; a failure
# stops the workflow before anything is pushed
- name: Run tests
run: docker buildx build --target test --progress plain -f Containerfile .
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
file: ./Containerfile
push: true
tags: |
${{ steps.registry.outputs.url }}/${{ github.repository }}:${{ github.ref_name }}
${{ steps.registry.outputs.url }}/${{ github.repository }}:latest
cache-from: type=gha
cache-to: type=gha,mode=max
run: |
image="$REGISTRY/$(echo '${{ github.repository }}' | tr '[:upper:]' '[:lower:]')"
docker buildx build --push -f Containerfile \
-t "$image:${{ github.ref_name }}" -t "$image:latest" .
- name: Clean up
if: always()
run: |
docker buildx rm ci || true
docker logout "$REGISTRY" || true
+362
View File
@@ -7,6 +7,368 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.10.1] - 2026-11-03
WebSub fixes from an adversarial review of v2.10.0 against the [W3C Recommendation](https://www.w3.org/TR/websub/) (details in ADR-071, Amendments).
### Security
- One subscription request could stop the hub for good: a callback that answered verification with a large page, or a callback URL with a control character, crashed the worker on every pass. Each queued item is now handled on its own
- A subscriber sending its response one byte at a time could hold the hub for hours. Every request the hub (and webmention fetching) makes now has a hard time limit
- The subscription limits only counted verified subscriptions and could be dodged with subdomains. They now count pending requests too, per callback server address, and at most 200 requests can wait for verification
- Callbacks must be printable URLs of reasonable length, not on this site, and on ports 80, 443 or 1024 and up
### Fixed
- New posts with photos were sent to subscribers (and cached in the feeds) without their photos, and photo-only edits sent the old feed. Notes with photos are announced once the photos are attached, and the feed cache follows photo and caption changes
- A subscriber whose deliveries failed for three days was removed; WebSub says to keep it until its lease ends. Deliveries are retried and given up, the subscription stays
- A temporary DNS failure or an error rendering the feed lost the delivery instead of retrying it. A feed that no longer exists now ends its subscriptions with a `hub.mode=denied` notice
- Verification sent our normalised topic (`/tag/python`) instead of the one requested (`/tag/Python`), so strict subscribers refused it. Percent-encoded topics (`/%66eed.atom`) are now understood, the Content-Type check ignores case, unsubscribe requests ignore `hub.lease_seconds`, and an empty `hub.secret` is refused instead of turning signatures off
- A newer subscription request arriving while an older one was being verified was lost (migration 017)
- Unsubscribing works while `WEBSUB_HUB_URL` points to another hub
- Feed ETags differed between server processes, so conditional requests rarely matched; `/feed` now sends `Vary: Accept`
- A like no longer causes pushes of the home page and feeds, which don't show likes; each feed is rendered once per delivery round, not once per subscriber
- Internationalised callback domains can be verified
### Changed
- `.env.example` documents `TRUSTED_PROXY` (needed behind a reverse proxy for per-client rate limits) and the WebSub settings
## [2.10.0] - 2026-11-03
### Added
- **WebSub** ([ADR-071](docs/decisions/ADR-071-websub.md)): readers that support it get new posts, edits and deletions within seconds instead of polling. StarPunk runs its own hub at `/websub` for its own feeds: the home page, `/feed.atom`, `/feed.rss`, `/feed.json`, `/likes` and each tag page. Each advertises the hub and itself (`Link` headers, and links inside the page or feed). Subscriptions are verified with a challenge, leases expire, deliveries carry the full feed, signed with `X-Hub-Signature` when the subscriber gave a secret, and failures are retried for about nine hours. Requests to subscribers go through the same private-address protections as webmentions, and there are limits per subscriber host and in total. Set `WEBSUB_HUB_URL` to use another hub instead; `WEBSUB_ENABLED=false` turns it off
- The admin dashboard shows WebSub subscribers per feed
### Fixed
- Feeds could be up to five minutes out of date after posting, editing or deleting, and each server process cached its own copy. Any change to a public note now refreshes the feeds everywhere at once
- The memory monitor stopped after the first request (the same bug the webmention worker had before v2.6.8)
### Changed
- The RSS feed calls itself `/feed.rss` instead of the older `/feed.xml`. Both addresses still work
## [2.9.0] - 2026-11-03
Micropub review against the [W3C Recommendation](https://www.w3.org/TR/micropub/).
### Changed
- **Micropub posts are only crossposted to the targets the client asks for.** If a client doesn't send `mp-syndicate-to`, nothing is crossposted (before, everything was). Posts made in the admin editor are unchanged
- Posts sent with `visibility` private or unlisted are saved as drafts instead of being published (StarPunk has no private posts)
- A token with only the `draft` scope can create drafts, and nothing else
- `q=source` and `q=category` need a token that can post (`create`, `update`, `delete` or `draft`); before, any token could list every post, drafts included
- Sending the access token in both the header and the body is refused (400)
### Added
- Bookmarks: Micropub `bookmark-of` is stored and shown ("🔖 Bookmarked …", `u-bookmark-of`), returned by `q=source`, listed in `q=config`, and crossposts lead with the bookmarked link (migration 015)
- A post with only a `summary` uses it as the text; a post with only a `name` is a titled post with no body
### Fixed
- Quill and other JSON clients put `mp-syndicate-to` and `mp-slug` inside `properties`; they were ignored, so posts crossposted everywhere and slugs were not used
- HTML in posts is sanitized: scripts, event handlers, iframes and `javascript:` links are removed when notes are shown. Formatting, images, code highlighting, footnotes and microformats are kept
- A photo that can't be downloaded or found made the post without it while reporting success. Create and update now fail with 400 and change nothing
- Remote photos are fetched the same way as webmentions: every redirect is checked for private addresses and the connection goes to the checked address
- JSON values that weren't lists were read a character at a time (`"category": "solo"` became tags s, o, l). Single values are now treated as a one-item list; `delete` values in updates must be lists
- Posts written as HTML are returned by `q=source` as `{"html": …}`, as the spec requires; empty `content` is left out
- A client-supplied `published` time is stored as UTC like every other post; a date that can't be read is refused instead of silently replaced with now
- Malformed JSON gets 400 instead of 500, and error messages no longer include Python internals
- `video` and `audio` (not supported) are refused with 400 instead of being dropped; person tags in `category` are ignored instead of failing the post
- An update with no `replace`, `add` or `delete` is refused
- `Authorization: bearer …` is accepted (the scheme is case-insensitive), and 401 responses include `WWW-Authenticate`
## [2.8.1] - 2026-11-03
### Fixed
- Starting the server after an upgrade with a database migration could fail: several server processes applied the same migration at once and one stopped with "duplicate column name", taking the server down until it restarted (this happened deploying v2.8.0, for about two seconds). The migration lock was released right after it was taken, and again before each migration was recorded. Migrations now run entirely inside the lock
## [2.8.0] - 2026-11-03
Microformats2 review: the markup now matches the [microformats2](https://microformats.org/wiki/microformats2) vocabularies, so IndieWeb readers and tools see what the page shows.
### Fixed
- Replies to a note weren't marked up as its comments, so readers and threading tools saw none. They are now `u-comment h-cite`s with the replier's h-card (name, link, photo) and the reply's HTML
- Likes, reposts, bookmarks and mentions on a note had no microformats at all. Each is now a `u-like`, `u-repost`, `u-bookmark` or `u-mention` h-cite with its author, including likes past the eight faces shown
- Responses link to and are dated by the original post (its own URL and published time) instead of the address that sent the webmention and when it arrived. New database columns hold these (migration 014)
- Post times had no time zone (and production showed microseconds), so readers could shift them by hours. Times on pages and in Micropub `q=source` are now UTC with a `Z`, matching the Atom feed
- A note starting with a hashtag (`#indieweb is great…`) was treated as a titled article: its text became a heading and the body was empty, and a note starting with a bare `#` line got its slug as a title. Only a Markdown heading (`#` then a space) makes a title now
- A webmention from one of a note's own crossposts (for example your GoToSocial copy, sent back through Bridgy) showed up as a response on the note. It is now ignored
- Micropub and IndieAuth discovery links disappeared from pages when webmentions were turned off
- A new site had no `rel="me"` link to its author, so readers couldn't tell whose site it was; pages now always link `rel="me"` and `rel="author"` to the author's home page
- Finding the author's profile could pick someone else's h-card from their home page. It now follows the [representative h-card](https://microformats.org/wiki/representative-h-card-parsing) rules
- A webmention's author is found by the [authorship algorithm](https://indieweb.org/authorship-spec), from the entry on the source page that actually responds to the note: an author given as a name or link, and the author of a feed the entry is in, now count
### Added
- Tag pages and the likes page have a feed URL and author, like the home page
- Note previews have `u-uid`
## [2.7.2] - 2026-11-03
### Added
- When the replies, likes and other responses shown on a note change (a webmention arrives, is updated or removed, or you approve, hide or delete one), the note's webmentions are sent again, as the Webmention spec recommends, so the sites it links to can pick up the conversation. A response that is checked again without changing sends nothing
### Security
- Webmention fetches and sends connect to the exact address that passed the private-address check. Before, the hostname was looked up a second time when connecting, so a DNS server that changed its answer could still point StarPunk at a private address. TLS certificates are still checked against the hostname. IPv4 addresses are preferred when a host has both
## [2.7.1] - 2026-11-03
### Changed
- Releases run the full test suite first: CI builds the Containerfile's new `test` stage (same Python image as production) and stops before pushing an image if any test fails
### Fixed
- Eight tests that had been failing: the image validation tests still expected bytes (validation has returned an image object since v1.5.0), and a migration test started the app in production mode
## [2.7.0] - 2026-11-03
Webmention now follows the [W3C Recommendation](https://www.w3.org/TR/webmention/) throughout.
### Changed
- Webmentions are only accepted for published notes. Mentions of the home page or other pages were stored but never shown; they're now refused with 400
- Deleted notes answer 410 Gone with a "This post has been deleted" page instead of 404
- A received mention that is sent again keeps its moderation: one you hid stays hidden, one you approved stays approved
### Fixed
- Sending: endpoint discovery now handles `rel` values with several words (`rel="webmention other"`), any capitalisation, `rel` after other Link parameters, and several Link headers, and takes the first `<link>` or `<a>` in page order
- Sending: the endpoint is looked up again every time (the 1–24 hour cache broke the spec's rule for updated posts)
- Sending: editing a note notifies every page it mentioned before, including an old reply target and links the edit removed. Deleting or restoring a note notifies them too, so they can remove or restore the mention
- Sending: a redirect from the endpoint is no longer treated as success (the POST was turned into a GET); "429 Too Many Requests" is retried
- Receiving: the source must link to the exact target URL (in a link, image, or other HTML attribute, any JSON value, or plain text). A longer URL that starts with the target, or the target mentioned in plain text or an HTML comment, no longer counts
- Receiving: relative links in the source are resolved, using the URL after redirects
- Receiving: the source is fetched with an `Accept` header and decoded in its declared character set
- Receiving: if the source can't be reached when a mention is sent again, the existing mention stays up instead of disappearing
- Receiving: the target's host is compared without regard to case or a default port, and `#fragment`s are dropped, so one mention isn't stored twice
- Mention snippets cut at 500 characters no longer leave HTML tags open, and the "Mentions" list no longer shows raw HTML tags
- The `Link: rel="webmention"` header is added alongside other Link headers instead of replacing them
- Links at the top level of a code block were still sent webmentions
### Security
- Fetches for webmentions check every redirect, not just the first URL, so a public page can't redirect StarPunk to a private address. Every address a hostname resolves to (IPv4 and IPv6) must be public; carrier-grade NAT, `0.0.0.0/8`, multicast, reserved and IPv4-mapped addresses are blocked too. Each fetch has a total time limit and a 1 MB size limit, including endpoint discovery
## [2.6.9] - 2026-11-03
### Fixed
- Received webmentions failed when the sender's profile photo had alt text (Bridgy, Mastodon and most IndieWeb sites do this). The photo is now stored as its URL
- Outgoing webmentions were never sent: endpoint discovery crashed on every target that had an endpoint, and the target was then marked as having none. Discovery now works, and network errors during discovery are retried instead of giving up
- Each server process ran its own webmention worker, so a webmention could be processed or sent more than once. Only one process handles the queues at a time now; if it stops, another takes over
### Security
- A webmention sender could set its author link to a `javascript:` URL, which was shown as a clickable link on the note page. Author links and photos are now only kept if they are http or https URLs
## [2.6.8] - 2026-11-03
### Fixed
- Webmentions were never processed. The background worker checked URLs against this site's settings without the app loaded, so every incoming webmention was marked "error" ("Working outside of application context") and every outgoing one kept failing and retrying. The worker now runs with the app loaded. A received webmention showed up in the admin list but never on the note
- The webmention worker also stopped as soon as any request finished, so after startup it often wasn't running at all. It now runs for the life of the server process
## [2.6.7] - 2026-11-02
### Fixed
- Admin pages were misaligned: the notes table ran past the right edge of the header and admin menu, because every page sat in the 40rem reading column and the table couldn't shrink to fit it. Admin pages now use a 60rem frame for the header, menu, content and footer alike, long post previews wrap, and on narrow screens tables scroll sideways instead of spilling out
## [2.6.6] - 2026-11-02
### Fixed
- Crossposts left out photos that were embedded in the post text instead of attached. Some Micropub clients (OutPost, for one) upload a photo to the media endpoint and then put it in `content` as an image rather than sending it as `photo`; both are valid Micropub. Images in the text that are in this site's media library are now sent to GoToSocial, Bluesky and Threads like attached photos, with their alt text, after any attached ones and within each target's photo limit. GoToSocial's text no longer includes the image Markdown. Images hosted on other sites are still left out
## [2.6.5] - 2026-11-01
### Fixed
- A Micropub post that failed partway (for example, the same photo listed twice) answered "Failed to create post" but stayed published without its photos, so a client retrying would post it twice. Photos are now looked up before the post is made, and a post that fails after that is removed again
- After such a failure, every request failed with "database is locked" for about 10 seconds: database connections now roll back unfinished writes when they're returned to the pool
- The same photo listed twice in one post is shown once instead of failing the post
- Replies didn't read as replies to IndieWeb readers and webmention receivers: on note pages `u-in-reply-to` sat inside an unrelated `h-cite`, and the home page didn't mark replies at all, so [Post Type Discovery](https://www.w3.org/TR/post-type-discovery/) saw notes (or photo posts). The reply context is now the entry's `u-in-reply-to h-cite`, and home page entries show "↩ In reply to"
- Photos without alt text used "Image" as their description, which IndieWeb readers passed on as if you'd written it. They now use the post's text (up to 150 characters), and fall back to "Image" only when the post has no text
- Home page entries always have an `e-content`, so a photo post without text never gets an implied name (which readers would show as its caption)
### Changed
- Photo posts no longer need text. Micropub doesn't require `content`, and `q=config` already offered a Photo post type with only `photo`. A Micropub update may also clear the text of a post that keeps its photos. Crossposts of a photo post without text start with its link, and its page is titled "Photo"
## [2.6.4] - 2026-11-01
### Fixed
- Tags keep the order you entered them in. The first tag is the note's main tag: it's the Threads topic, and Bluesky keeps the first tags when space runs out. Before, both used the alphabetically first tag. Public pages still list tags alphabetically
- Opening a note in the editor or updating it via Micropub reordered its tags alphabetically on save; the editor and Micropub `q=source` now show, and keep, the entered order
## [2.6.3] - 2026-11-01
### Changed
- **A note's own link gets the preview card** (ADR-070). When a note links somewhere (say, an article), the crosspost's card shows that page instead of the note. Bluesky cards are built from the page's title, description and image; Threads previews it from the text. Notes without links keep their own card
### Fixed
- Markdown links (`[text](url)`) lost their URL on Bluesky and Threads. On Bluesky the link text is now clickable; on Threads it's written "text (url)"
- Notes with many links failed on Threads (Meta allows 5 per post); extra links now keep just their text
## [2.6.2] - 2026-11-01
### Fixed
- New notes failed their first Threads crosspost with "Invalid Link Attachment" (Meta rejects an explicit link preview for a page it hasn't fetched yet) and only went out on the retry a minute later. StarPunk no longer sends the explicit link preview; Threads still shows one, built from the note's link in the text
## [2.6.1] - 2026-11-01
### Fixed
- Threads crossposts gave up after Meta's publish step returned a one-off `HTTP 400 Fatal`; publish errors are now retried (a retry checks for an existing post first, so nothing is posted twice). Only a bad token or missing permission stops retries
- Threads errors now include Meta's error code, subcode, user message and trace id, so the editor shows why a post failed
- Wait a moment before the first readiness check on a Threads post, since Meta can report it ready too early
## [2.6.0] - 2026-11-01
### Added
- **Crossposting to Threads** (ADR-069): set `THREADS_ACCESS_TOKEN` (and optionally `THREADS_USERNAME`) to add a Threads target. Text posts get a link preview; 1 image posts as an image, 2–4 as a carousel, with alt text; the note's first tag becomes the post's topic. Shortened to 500 characters. Edits don't follow (the API can't edit); unpublish and delete remove the post. Likes aren't crossposted to Threads; reposts post "🔁 title url" (with the comment, if any)
- The Threads token (60-day lifetime) is refreshed automatically once a day and kept in `data/threads_token.json`, so it survives restarts; setting a new token in the config replaces it. Setup steps: `docs/design/v2.6.0/`
## [2.5.0] - 2026-11-01
### Added
- **Likes and reposts** via Micropub `like-of` / `repost-of` (ADR-068); a comment is optional
- Likes are kept off the home page and feeds and listed on a new `/likes` page (linked in the header)
- Reposts appear in the stream and feeds with a "🔁 Reposted" line
- The liked/reposted post's title and author are fetched and shown; it is sent a webmention
- GoToSocial: likes/reposts of fediverse posts become native favourites/boosts; reposts of other pages (or with a comment) post "🔁 title url"
- Bluesky: likes/reposts of Bluesky posts become native likes/reposts; a repost with a comment is a quote post; reposts of other pages post with a link card
- Likes of ordinary web pages are not crossposted; deleting a like/repost undoes it everywhere
- Migration 013: `notes.like_of`, `notes.repost_of`
### Fixed
- Outgoing webmentions named posts `https://site//note/<slug>` (double slash)
### Notes
- The GoToSocial token needs the extra scopes `read:search` and `write:favourites`
## [2.4.0] - 2026-10-31
### Added
- **Micropub update** (`action=update`, JSON): replace/add/delete for `content`, `name`, `category`, `photo`, `in-reply-to`, `post-status`; edits follow to crossposts (ADR-067)
- **Micropub delete and undelete**: soft delete, restorable; crossposts are removed and re-posted
- **Drafts** via `post-status: draft`; **replies** via `in-reply-to`
- Photos uploaded in a multipart create request; photos hosted elsewhere are downloaded (public addresses only, 20 MB cap)
- `q=source` without `url` lists recent posts (drafts included, `limit`/`offset`); `properties[]` filter; returns photos, syndication links, reply and post status
- `q=category` (tag suggestions, optional `filter`); `q=config` lists supported queries and post types
- Media endpoint accepts the `media` scope
### Fixed
- Media endpoint was advertised at `/micropub/media` but served at `/media` (404 for clients); now both work
- Unsupported Micropub types (e.g. `h=event`) were silently stored as notes; now rejected
## [2.3.3] - 2026-10-31
### Fixed
- Micropub `name` was ignored, so titles sent by Micropub clients never appeared. It is now stored as the note's `# Heading` first line, which is how StarPunk titles work
- Micropub `q=source` returned `/notes/<slug>` links (a 404); it now returns the real permalink `/note/<slug>`
- Micropub `q=source` reported every note's first line as its `name`; it now returns `name` only for titled notes, with the heading left out of `content`, matching what was posted
- Titled notes showed their title twice: as the page heading and again at the top of the body (and of the home-page summary)
## [2.3.2] - 2026-10-31
### Changed
- Header link reads "Admin" (was "Dashboard") when logged in
### Fixed
- Public pages never knew you were logged in (only `/admin` routes checked the session), so the header always said "Log in"
## [2.3.1] - 2026-10-31
### Fixed
- "Log in" is back in the header nav (2.3.0 had moved it to the footer, where it was easy to miss)
## [2.3.0] - 2026-10-31
### Changed
- **Public pages restyled**: compact header with the site name from `SITE_NAME`, serif reading font for notes, one-line metadata (date · #tags · Also on …), dark mode via `prefers-color-scheme`, footer pinned to the bottom, "Log in" moved to the footer
- Home and tag pages show short notes in full; titled notes show a summary and "Read more"
- Author h-card kept for parsers but visually hidden on this single-author site
- Page titles use the note text and `SITE_NAME` instead of the slug and "StarPunk"
### Fixed
- Every note's metadata was styled as the page footer (centered, ruled, padded)
- Previews cut the note's HTML at 300 characters, which could break markup
- Bare URLs in notes are now links
## [2.2.0] - 2026-10-31
### Added
- **Crossposting to Bluesky**, ADR-066
- Plain-text note (Markdown stripped) with clickable link and hashtag facets, shortened to 300 characters with the permalink kept
- Images (up to 4, shrunk under Bluesky's 1 MB limit) with alt text; notes without images get a link card
- Unpublishing, unticking and deleting remove the Bluesky post; edits don't follow (Bluesky has no edits)
- Logs in with an app password: `BLUESKY_HANDLE`, `BLUESKY_APP_PASSWORD` (optional `BLUESKY_SERVICE`, default `https://bsky.social`)
- Shown in the editor and Micropub `syndicate-to` next to GoToSocial
## [2.1.0] - 2026-10-31
### Added
- **Crossposting to GoToSocial** (or any Mastodon-API server), ADR-065
- Full note as Markdown with images (alt text from captions), permalink and hashtags
- Edits, unpublishing and deletes follow the note
- On by default for new notes; untick in the editor, or send `mp-syndicate-to` from Micropub apps
- Micropub `q=syndicate-to` and `q=config` list the target
- Editor shows crosspost status, errors and a retry button; note pages link the copy with `u-syndication`
- Configure with `GOTOSOCIAL_URL` and `GOTOSOCIAL_ACCESS_TOKEN` (scopes `write:statuses write:media`)
- Migration 012: `syndications`
### Fixed
- Micropub `Location` header pointed at `/notes/<slug>` (404); now the real permalink
- "Delete Note" on the edit page submitted the edit form (nested forms)
- Search results linked to `/notes/<slug>` (404); now `/note/<slug>`
## [2.0.2] - 2026-10-31
### Fixed
- Container builds in Gitea Actions log in to the registry with the
`REGISTRY_TOKEN` secret (a `write:package` token); the automatic Actions
token is refused by the registry. No application changes.
## [2.0.1] - 2026-10-31
### Fixed
- Page links built from the request said `http://` behind an HTTPS reverse proxy
(RSS/OPML `<link>`s and microformats `u-url`/`u-uid` permalinks). All absolute
URLs in templates now come from `SITE_URL` via `absolute_url_for()`.
- Container builds in Gitea Actions: build through the server's rootless BuildKit
and log in to the public registry name (the old workflow never succeeded).
## [2.0.0] - 2026-10-31
### Changed (breaking)
+21 -1
View File
@@ -23,7 +23,27 @@ RUN uv venv /opt/venv && \
uv pip install --no-cache -r requirements.txt
# ============================================================================
# Runtime Stage - Minimal production image
# Test Stage - Runs the test suite (CI builds it with --target test)
# ============================================================================
FROM python:3.11-slim AS test
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
COPY --from=builder /opt/venv /opt/venv
RUN uv pip install --no-cache --python /opt/venv/bin/python pytest pytest-mock
ENV PATH="/opt/venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
SESSION_SECRET=ci-test-secret \
SITE_URL=http://localhost:5000 \
ADMIN_ME=https://test.example.com
WORKDIR /app
COPY . .
RUN python -m pytest -q -p no:cacheprovider tests
# ============================================================================
# Runtime Stage - Minimal production image (default target: keep it last)
# ============================================================================
FROM python:3.11-slim
@@ -0,0 +1,45 @@
# ADR-065: Direct Syndication (POSSE) to GoToSocial
## Status
Accepted (v2.1.0, 2026-10-31)
## Context
StarPunk posts were copied to other networks by Echofeed reading the feed.
That gives no control per post, no record of where a post went, and edits
and deletes never follow. The operator runs GoToSocial locally and wants
StarPunk to post there directly, with Bluesky likely next.
## Decision
1. **Publish here, copy out (POSSE).** Each (note, target) has a row in
`syndications` (migration 012). Creating, editing, unpublishing or
deleting a note marks its rows pending; a background worker reconciles
the remote post: create, update (`PUT /api/v1/statuses/:id`) or delete.
2. **Defaults.** New notes go to every configured target unless unticked in
the editor or a Micropub client sends `mp-syndicate-to` (an explicit
list, possibly empty). Existing notes are never backfilled.
3. **Content.** The full note as Markdown (`content_type: text/markdown`),
then the permalink and the note's tags as hashtags. Longer than the
target's limit: shortened at a word boundary with the link kept.
Images are uploaded first (the `large` variant) with their captions as
alt text.
4. **Delivery.** Rows are claimed atomically so the four gunicorn workers'
threads never send twice; an `Idempotency-Key` guards retries; a row
`version` keeps edits made during a send. Network errors and 5xx/429
retry with exponential backoff (8 attempts); other 4xx fail at once.
A remote post deleted by hand is posted again on the next edit.
5. **Backlinks.** Synced URLs are shown on the note page as `u-syndication`
links, and in the editor with status, last error and a retry button.
6. **Targets** are config-driven. One kind exists, a Mastodon-API client
(`GOTOSOCIAL_URL`, `GOTOSOCIAL_ACCESS_TOKEN`); it also works for
Mastodon. A Bluesky target would add a second client behind the same
reconcile loop.
## Consequences
- Echofeed must stop posting to GoToSocial or posts appear twice.
- The token needs only `write:statuses write:media`.
- `syndications.note_id` has no foreign key so a hard-deleted note's
remote post can still be deleted.
@@ -0,0 +1,37 @@
# ADR-066: Crossposting to Bluesky
## Status
Accepted (v2.2.0, 2026-10-31)
## Context
ADR-065 added direct crossposting to GoToSocial. Bluesky was still fed by
Echofeed, with the same problems: no per-post control, no record of the
copy, deletes never follow.
## Decision
1. **Second client, same loop.** `BlueskyClient` sits behind the reconcile
loop and `syndications` table from ADR-065, as target id `bluesky`.
Clients share a small interface: `create`, `update`, `delete`, `can_edit`.
2. **Auth.** An app password (`BLUESKY_HANDLE`, `BLUESKY_APP_PASSWORD`).
`createSession` against `BLUESKY_SERVICE` (default `https://bsky.social`);
writes go to the PDS named in the returned DID document. Expired access
tokens are refreshed, falling back to a new session.
3. **Content.** Plain text (Markdown rendered and stripped), then the
permalink and hashtags (at most a third of the space), shortened to 300
characters at a word boundary. Link and hashtag facets use UTF-8 byte
offsets. Images (up to 4) are re-encoded as JPEG if over 1 MB, with alt
text and aspect ratio; a note without images gets a link card.
4. **Edits don't follow.** Bluesky has no edit; rewriting the record would
break likes and replies, and delete-and-repost would lose them. The post
stays as first sent. Unpublish, untick and delete remove it.
5. **No double posts.** When a create is retried, recent posts are checked
for one already linking to the note (a send whose reply was lost).
## Consequences
- Echofeed is no longer needed; it must stop posting to Bluesky or posts
appear twice.
- A post deleted by hand on Bluesky is not re-created.
@@ -0,0 +1,34 @@
# ADR-067: Micropub update, delete, drafts and replies
## Status
Accepted (v2.4.0, 2026-10-31)
## Context
The Micropub endpoint could only create published notes. Apps could not edit
or delete posts, drafts and replies were dropped, and photos sent with the
post or hosted elsewhere were lost.
## Decision
1. **Update** (JSON only, `update` scope). Supported properties are the ones a
note can store: `content`, `name` (the `# Heading` first line),
`category`, `photo`, `in-reply-to`, `post-status`. Anything else is a 400,
not silently ignored. Order: delete, then replace, then add (`add` only for
`category` and `photo`). Success is 204.
2. **Delete is soft** (`delete` scope) so `undelete` can restore the note and
its file from `.trash/`. Crossposts are removed on delete and posted again
on undelete.
3. **Drafts**: `post-status: draft` creates an unpublished note; crossposts
wait until it is published.
4. **Photos**: multipart uploads are saved like media-endpoint uploads; remote
URLs are downloaded only from public IP addresses, up to 20 MB, max 4.
5. **Queries**: `q=source` returns everything create accepts (round-trip);
without `url` it lists recent posts, drafts included, for editing apps.
`q=category` lists tags by use.
## Consequences
- `published` (backdating) can be set on create but not changed by update.
- Likes, reposts and bookmarks remain unsupported (round 2).
@@ -0,0 +1,32 @@
# ADR-068: Likes and reposts
## Status
Accepted (v2.5.0, 2026-11-01). Bookmarks were considered and declined.
## Decision
1. **Storage.** `notes.like_of` / `notes.repost_of` (migration 013). The note's
content is an optional comment and may be empty.
2. **Display.** Likes stay off the home page and feeds and are listed on
`/likes`. Reposts appear in the stream and feeds. Both show the target's
fetched title and author (the reply-context cache) and carry
`u-like-of` / `u-repost-of`. The target is sent a webmention.
3. **Crossposting, per target.**
| | Target is a post on that network | Anything else |
|---|---|---|
| Like | native favourite / like | nothing |
| Repost, no comment | native boost / repost | "🔁 title url" post |
| Repost, with comment | Bluesky quote post; GoToSocial status | comment + "🔁 title url" post |
GoToSocial finds fediverse posts with `/api/v2/search?resolve=true`
(needs `read:search`; favourites need `write:favourites`). Bluesky
recognises `bsky.app/profile/<actor>/post/<rkey>` URLs. Native actions
are stored as `favourite:<id>` / `reblog:<id>` or the like/repost record
URI, and deleting the note undoes them.
## Consequences
- Editing a like/repost doesn't change a native favourite/boost.
- A repost with a comment can't be a GoToSocial boost (no quote posts there).
@@ -0,0 +1,57 @@
# ADR-069: Crossposting to Threads
## Status
Accepted (v2.6.0, 2026-10-31)
## Context
ADR-065 and ADR-066 crosspost notes to GoToSocial and Bluesky. Threads
(Meta) has a publishing API, so it can be a third target on the same loop.
## Decision
1. **Third client, same loop.** `ThreadsClient` is target id `threads`
behind the `syndications` table and reconcile loop from ADR-065.
2. **Auth.** A long-lived Threads user token (`THREADS_ACCESS_TOKEN`) with
`threads_basic`, `threads_content_publish` and `threads_delete`. These
tokens expire after 60 days, so StarPunk refreshes the token about once a
day (`th_refresh_token`) from the worker loop, even when nothing is being
posted. The current token is kept in `DATA_PATH/threads_token.json`
(mode 0600) with a hash of the configured token. A different configured
token replaces the saved one. A file lock stops gunicorn workers from
refreshing at the same time. The file avoids a migration for one value.
3. **Posting.** Create a container, poll its status until `FINISHED`, then
`threads_publish`. Text-only notes are `TEXT` posts. Threads previews
the first link in the text (the permalink) by itself. An explicit
`link_attachment` is not sent, because Meta rejects one for a page it
hasn't fetched yet ("Invalid Link Attachment", subcode 4279047), which
is always the case for a new note (v2.6.2). One image is an `IMAGE` post and 2–4 images are a
`CAROUSEL`. Meta fetches images from their public `/media/` URL. Only
JPEG and PNG up to 8 MB are allowed, so GIF and WebP images are left off.
Alt text goes in `alt_text`.
4. **Content.** Plain text (as for Bluesky), then the permalink, shortened
to 500 characters. Non-ASCII characters are counted as UTF-8 bytes,
because Threads counts emoji that way. Threads allows one tag per post,
so the note's first tag is sent as `topic_tag` and no hashtags are added
to the text.
5. **Edits don't follow.** The API has no way to edit a post. As with
Bluesky, the post stays as first sent, and unpublishing, unticking the
target or deleting the note removes it.
6. **Likes and reposts (ADR-068).** The API can't like or repost outside
content, so likes are not crossposted. Reposts are text posts with
"🔁 title" and the URL, plus the comment if there is one, and the
reposted page as the link preview.
7. **No double posts.** A retried create first looks for a recent post that
already contains the permalink.
## Consequences
- Meta limits accounts to 250 posts and 100 deletes per day. Rate-limit
errors (Graph codes 4, 17, 32, 613) are retried.
- If StarPunk is down for more than 60 days, the token expires and a new
one is needed.
- Getting the first token requires a Meta developer app with the Threads
use case. See the setup notes in `docs/design/v2.6.0/`.
- Notes with more than five links fail on Threads (a Meta limit). The
error is shown in the editor.
@@ -0,0 +1,44 @@
# ADR-070: Links and link cards in crossposts
## Status
Accepted (v2.6.3, 2026-11-01)
## Context
Every crosspost carries the note's permalink. Notes often link elsewhere
too, usually to the article being discussed. Before this change:
- Bluesky and Threads get plain text, and a Markdown link `[text](url)`
lost its URL, so readers there couldn't follow it.
- The Bluesky card always showed the note, and the Threads preview showed
whichever link came first.
- Threads rejects posts with more than 5 different links.
## Decision
1. **The card shows what the note links to.** If the note has its own link
(a Markdown link, `<url>` or a bare URL), the first one gets the
preview card. A note without links gets a card for itself, as before.
Images still take the place of a card on Bluesky.
2. **Bluesky.** The card is built by StarPunk: it fetches the page's Open
Graph, Twitter or plain HTML title, description and image, and uploads
the image (shrunk to under 1 MB) as the thumbnail. If the page can't
be fetched, the card has just the URL. Markdown link text becomes a
clickable link facet, so no characters are spent on the URL.
3. **Threads.** It has no clickable link text, so Markdown links are
written "text (url)". Threads previews the first link in the text, which
is now the linked page. To stay within Meta's 5-link limit, after 4
different links in the body a Markdown link keeps only its text.
4. **GoToSocial** already gets Markdown, so its links are unchanged.
Receiving servers build their own cards.
5. **Fetching is careful.** Only http(s) URLs on public addresses are
fetched, and this is checked again at every redirect. At most 512 KB of
a page and 5 MB of an image are read. A failed fetch never fails the
post.
## Consequences
- One or two extra requests to the linked site when a note is crossposted
to Bluesky.
- Reposts (ADR-068) are unchanged: their card is the reposted page.
+206
View File
@@ -0,0 +1,206 @@
# ADR-071: WebSub publishing with a built-in hub
## Status
Accepted (v2.10.0, 2026-11-03)
## Context
Readers poll StarPunk's feeds. WebSub (W3C Recommendation,
https://www.w3.org/TR/websub/) lets readers that support it (Monocle,
Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and
have new content pushed to them within seconds.
WebSub has three roles. StarPunk is the **publisher**. Someone has to run
the **hub** that keeps subscriptions and delivers content. Public hubs
exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a
third-party dependency and some have shut down. The rest of this site is
self-hosted.
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
Two existing problems matter here:
- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears
the cache when a note changes. A hub fetching right after a change would
get stale content. Regular readers are also up to 5 minutes behind.
- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and
calls itself `/feed.xml`.
## Decision
StarPunk runs its own hub, **for its own topics only**, at `/websub`.
`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk
advertises that hub, pings it on changes, and its own `/websub` is off.
### 1. Topics
Absolute URLs under `SITE_URL`:
| Topic | Content type |
|---|---|
| `/` (h-feed) | text/html |
| `/feed.atom` | application/atom+xml |
| `/feed.rss` | application/rss+xml |
| `/feed.json` | application/feed+json |
| `/likes` | text/html |
| `/tag/<name>` | text/html |
`/feed.xml` and the negotiated `/feed` advertise the topic of the format
they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The
RSS feed's own self link changes from `/feed.xml` to `/feed.rss`.
### 2. Discovery (publisher, spec 4)
Every topic response carries `Link: <hub>; rel="hub"` and exactly one
`Link: <topic>; rel="self"`. Embedded links as well: `<link rel="hub">` and
`<link rel="self">` in the HTML head of topic pages, `<link rel="hub">`
next to the existing self link in Atom, `<atom:link rel="hub">` in RSS, and
`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed.
### 3. What changed, and when
`websub.note_changed(note, old_tags=())` is called wherever
`syndication.note_changed` is, plus on create. It:
1. clears the feed cache (all formats), and
2. if the note is or was public, queues delivery for `/`, the three
feeds, `/likes` if it is a like, and each `/tag/<name>` for its tags
before and after the change.
Drafts that stay drafts change nothing.
### 4. Hub endpoint (spec 5)
`POST /websub`, `application/x-www-form-urlencoded`:
- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the
topics above (after normalising), else 400 with a plain-text reason;
`hub.callback` must be an absolute http(s) URL that passes the SSRF
check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to
1 day to 30 days (default 10 days). Unknown parameters are ignored.
- Response: 202 Accepted, independent of what verification later finds.
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions
per callback host and 1,000 in total; beyond that, 429.
### 5. Verification of intent (spec 5.3)
A background worker sends `GET <callback>` with the callback's query
string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32
bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body
equal to the challenge; anything else, including redirects, fails and
leaves the subscription state unchanged. On success a subscribe becomes
active with `expires_at = now + lease`, and an unsubscribe deletes it.
A re-subscribe replaces the secret and lease only once verified.
### 6. Content distribution (spec 6, 7)
For each change, one delivery per active, unexpired subscription to the
topic. The worker renders the topic inside the app (no HTTP round trip
through Caddy), so subscribers get exactly what the public URL serves,
then `POST`s it to the callback with:
- `Content-Type` of the topic,
- `Link` headers with `rel=hub` and `rel=self`,
- `X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>` if a secret was
given.
A 2xx is success. 410 Gone removes the subscription. Anything else is
retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a
subscription whose deliveries fail for 3 days is removed. Only the latest
content is sent: a queued delivery for a topic is replaced by a newer one.
### 7. Outbound safety
Callbacks are arbitrary URLs, so verification and delivery go through
`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the
connection pinned to the checked address, no redirects, 10 s time limit,
1 MB response cap.
### 8. Data (migration 016)
- `websub_subscriptions(id, topic, callback, secret, lease_seconds,
state ['pending','active'], expires_at, created_at, verified_at,
last_success_at, last_error)`, unique on `(topic, callback)`.
- `websub_intents(id, topic, callback, mode, secret, lease_seconds,
attempts, next_attempt_at, created_at)`: verifications to do.
- `websub_deliveries(id, subscription_id, attempts, next_attempt_at,
last_error, created_at)`, unique on `subscription_id`.
### 9. Worker
The same pattern as the Webmention worker: one thread per gunicorn
process, a file lock so only one process works the queues, app context
around each poll. Expired subscriptions are pruned once a day.
### 10. Admin
The admin dashboard shows active subscribers per topic and failing
deliveries.
### 11. Configuration
`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in),
`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25).
## Consequences
- Readers that support WebSub get posts and edits within seconds.
- Feeds are fresh right after a change for every reader, because the cache
is cleared.
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF
protections, per-host and total limits, and rate limiting bound that.
- websub.rocks' hub tests use websub.rocks' own topics, which this hub
refuses by design. Testing uses unit tests of each spec section and a
real subscription from a reader after deploy.
- The RSS self link changes to `/feed.rss`. Readers keep whichever URL
they subscribed to; both keep working.
## Alternatives considered
- **Public hub only.** Least code, but a third-party dependency for a
self-hosted site. Still available through `WEBSUB_HUB_URL`.
- **A separate hub service** (another container). More to deploy and
keep updated, and few maintained hubs exist.
- **Accepting any topic** (a general-purpose hub). Not needed, and it
would let the hub be used to fetch and push arbitrary URLs.
## Amendments (v2.10.1, 2026-11-03)
An adversarial review of v2.10.0 against the Recommendation led to these
changes; where they differ, they replace the text above.
- **Worker robustness.** Each queued item is handled on its own: an
unexpected error drops (verification) or retries (delivery) that item
only. Before, a callback answering with a large page or containing a
control character stopped the worker for good.
- **Time limits.** `safe_get` and `safe_post` close the connection when
the whole exchange exceeds its time limit (10 s), not just a single read.
Verification replies over 4 KB fail; delivery responses are read up to
4 KB.
- **Leases are kept (spec 7).** A failing subscriber is not removed; its
deliveries are retried and then given up, and the next change is
delivered again. Only lease expiry, 410 Gone or unsubscribing end a
subscription. DNS failures are retried. A topic that renders 404/410
ends its subscriptions with a `hub.mode=denied` notice; other render
errors are retried.
- **Photos.** A note with photos is announced once they are attached
(`attach_media_to_note` notifies), and the feed cache key and ETag
include `content_version`, so photo and caption changes reach feeds and
subscribers. ETags no longer depend on the feed's build time.
- **Subscription requests (spec 5.1).** Unreserved percent-encodings are
decoded; the Content-Type check is case-insensitive; `hub.secret` and
`hub.lease_seconds` are ignored for unsubscriptions; an empty secret is
refused; verification echoes `hub.topic` exactly as requested
(`requested_topic`); unsubscribing works while an external hub is
configured. Callbacks must be printable, at most 2,048 characters, not on
this site, on port 80, 443 or 1024+, and are stored with an IDNA host.
- **Limits** count pending requests as well as subscriptions, per callback
*address* (so subdomains on one server share a limit), with at most 200
pending requests. Rate limiting needs `TRUSTED_PROXY=true` behind Caddy.
- **Races.** Each request has a `request_id`; verifying an older request
doesn't consume a newer one (migration 017).
- **Less work.** A topic is rendered once per worker pass; likes only
update `/likes` and tag pages.
- Corrections: there is no `state` column (a subscription row exists only
once verified); expired leases are pruned hourly.
@@ -0,0 +1,23 @@
# v2.3.3: Micropub titles, q=source URL, single title display
Found while building tuipub, a terminal Micropub client, against StarPunk.
## Problems
1. **Micropub `name` dropped.** `handle_create` called `extract_title()` and threw the result away. StarPunk has no title column: `Note.title` is derived from the content's first line, and a `# Heading` first line marks a titled note (Q22). So a `name` sent by a client never showed up.
2. **`q=source` URL wrong.** It built `{SITE_URL}notes/{slug}`, but the route is `/note/<slug>`, so the URL was a 404.
3. **`q=source` name wrong.** It returned `note.title` for every note. For untitled notes that is just the first line of the content, so clients saw a `name` they never sent.
4. **Title shown twice.** `note.html` renders the title as `<h1 class="p-name">` and then the whole content, which includes the `# Heading` again. Home-page previews did the same with the excerpt.
## Decisions
- Store `name` as a `# {name}` first line (`micropub.add_title_heading`) rather than adding a title column. This matches how titled notes already work everywhere (pages, feeds, search, admin editor), and needs no migration. If the content already starts with that exact heading, it isn't added twice.
- `models.split_title()` is the single definition of "title heading": the first non-blank line starting with `#` (the same rule as the templates' `has_explicit_title`). `q=source` uses it to return `name` + `content` the way they were posted.
- `Note.body_html` renders the content without the title heading; `note.html` uses it. `Note.excerpt` also leaves the title out. `Note.html` is unchanged, so feeds and syndication behave as before (feed items carry a separate title anyway).
- `extract_title()` no longer derives a fallback title from content: that value was never used.
## Tests
- `tests/test_micropub.py`: name stored as heading, no duplicate heading, `q=source` url equals the Location permalink, `name` only for titled notes, round trip.
- `tests/test_models.py`: `split_title`, `body_html`, excerpt without title.
- Checked end to end with tuipub against a local dev server: titled post → page shows the title once, `q=source` returns `name`, `content` and a working url.
@@ -0,0 +1,62 @@
# v2.6.0: Crossposting to Threads
Implements ADR-069.
## What changed
- `starpunk/syndication.py`: `ThreadsClient` (target `threads`); media
entries now carry their public `url`; `process_due()` calls each client's
optional `maintain()` hook, which Threads uses to refresh its token.
- `starpunk/config.py`: `THREADS_ACCESS_TOKEN`, `THREADS_USERNAME`,
`THREADS_NAME`, `THREADS_MAX_CHARS` (500), `THREADS_MAX_MEDIA` (4).
- `tests/test_syndication_threads.py`: a fake Graph API covering text,
image and carousel posts; container polling; edits not following; deletes
(including posts already deleted by hand); a lost publish reply;
permanent and rate-limit errors; and token refresh, persistence and
replacement.
## Configuration
| Variable | Meaning |
|---|---|
| `THREADS_ACCESS_TOKEN` | Long-lived Threads user token. Enables the target. |
| `THREADS_USERNAME` | Optional. Used for the Micropub `syndicate-to` uid (`https://www.threads.com/@name`) and for fallback links. |
| `THREADS_NAME` | Label in the editor (default `Threads`) |
After the first daily refresh, the working token lives in
`DATA_PATH/threads_token.json`. The configured value only seeds it.
Changing `THREADS_ACCESS_TOKEN` switches to the new token.
## Getting a token (one time)
1. At <https://developers.facebook.com/apps>, create an app with the
**Access the Threads API** use case.
2. Under the use case's permissions, add `threads_basic`,
`threads_content_publish` and `threads_delete`.
3. Under **App roles → Roles**, add your Threads account as a Threads
tester. Accept the invite in Threads (Settings → Account → Website
permissions → Invites). The app can stay in development mode.
4. In the use case settings, add `https://<site>/` as a redirect URI. Then
open:
https://threads.net/oauth/authorize?client_id=APP_ID&redirect_uri=https://<site>/&scope=threads_basic,threads_content_publish,threads_delete&response_type=code
After you approve, the browser lands on `https://<site>/?code=...`. Copy
the code (drop a trailing `#_`).
5. Exchange the code for a short-lived token, then exchange that for a
long-lived one:
curl -s -X POST https://graph.threads.net/oauth/access_token \
-F client_id=APP_ID -F client_secret=APP_SECRET \
-F grant_type=authorization_code -F redirect_uri=https://<site>/ -F code=CODE
curl -s "https://graph.threads.net/access_token?grant_type=th_exchange_token&client_secret=APP_SECRET&access_token=SHORT_TOKEN"
6. Set `THREADS_ACCESS_TOKEN` to the long-lived `access_token`.
The app secret is only needed for these steps. StarPunk doesn't use it.
## Not done
- Video, GIF and WebP images (Threads takes video by URL. StarPunk only
stores still images, and Threads rejects GIF and WebP).
- Edits (no API for them).
@@ -0,0 +1,89 @@
# v2.6.5: Micropub media fixes
Found while adding media uploads to tuipub, a Micropub client written to the
spec rather than to StarPunk.
## Problems
1. **Failed creates left a post behind.** `handle_create` made the note, then
attached photos. Listing one photo twice broke the `note_media` unique key,
so the client got a 500 while the note stayed published without photos.
A retry would post it twice.
2. **Then the database stayed locked.** `attach_media_to_note` didn't roll
back, and `ConnectionPool.return_connection` pooled the connection with
its write transaction still open. Other requests got "database is locked"
until the 10 s timeout.
3. **Photo posts needed text.** The Micropub spec doesn't require `content`,
and `q=config` advertised a `photo` post type with only `photo`, yet
creates without content got 400 "Content is required".
## Changes
- `database/pool.py`: `return_connection` rolls back an open transaction
(or closes the connection if that fails).
- `media.py`: `attach_media_to_note` rolls back on error.
- `micropub.py`:
- `handle_create` resolves photos (by media URL, downloading remote ones)
before `create_note`. If attaching or crosspost selection fails, it
hard-deletes the note before answering 500.
- `_resolve_photos` skips repeated media.
- Content is optional when photos resolve. Content-less posts whose photos
all fail still get a 400 from `create_note`.
- `handle_update` allows `replace: {content: [""]}` and `delete: ["content"]`
while the note keeps photos (or is a like or repost). Otherwise it answers
400 "A post needs text or a photo".
- `notes.py`: `create_note` and `update_note` take `has_media`, so
empty content is allowed for photo posts.
- `syndication.py`: `compose_text` drops the leading blank line when there's
no body.
## Post types (IndieWeb standards check)
Post Type Discovery (W3C Note) has no separate "photo" type to store. A post
with a valid `photo` is a photo post, unless it's first a repost, like or
reply (step order: repost, like, reply, video, photo). Otherwise it's a note,
or an article when its `name` isn't a prefix of its content. h-entry treats
`u-photo` as the primary content, with `name` as a caption. mf2 implies a
`name` only when an entry has no other `p-*`/`e-*` properties and no nested
microformats.
StarPunk already stores photo posts as notes with media and renders
`u-photo` outside `e-content`, which is what the standards ask for. Checking
the rendered pages with mf2py found:
- **Replies weren't replies.** In `note.html`, `u-in-reply-to` was inside a
bare `h-cite`, so it was the h-cite's property, not the entry's.
`note_preview.html` had no reply markup at all. Fixed: the blockquote is
`u-in-reply-to h-cite` (with `u-url` inside), and previews show
"↩ In reply to" with `u-in-reply-to`.
- **Implied names were possible.** Previews left out `e-content` for posts
without text. They worked only because the author h-card blocked an
implied name. Previews now always emit `e-content`.
- Page title for text-free posts follows PTD order: Repost, Like, Reply,
Photo.
Photos without a caption take the post's text as alt (`partials/media.html`
`fallback_alt`) instead of the made-up "Image". The Micropub `q=source`
still reports the stored caption (`""`).
`tests/test_post_types.py` parses note pages and the home page with mf2py and
runs PTD for notes, articles, photo posts (no text, caption, title), replies
(with and without a photo) and reposts.
## Not changed (noted for later)
- Text-free photos without alt text still render `alt="Image"`. Captioned
photos use their caption, and others now use the post's text (`display_media`
`fallback_alt`, the excerpt cut to 150 characters).
- More than 4 photos are still cut to 4 without an error (ADR-057).
- `video` and `audio` are ignored. This is allowed by the spec, and they
aren't in `q=config` post-types.
- Media endpoint errors for non-images expose Python text
("cannot identify image file <_io.BytesIO …>").
## Tests
`tests/test_micropub_v265.py`. The full suite runs with
`SESSION_SECRET=test`. 8 tests in `test_media_upload.py` and
`test_migration_race_condition.py` already fail on main, and still fail here.
@@ -0,0 +1,37 @@
# v2.6.8: Webmention worker app context
## Problem
Production webmention #1 (Bridgy, reply from Mastodon to note 20261101162512)
was stored with `validation_status = 'error'`:
"Working outside of application context". Both outgoing queue items had the
same error and were on their last retry.
## Causes
1. `webmention_ssrf.safe_resolve()` reads `current_app.config['DEV_MODE']`.
`WebmentionWorker` is a plain thread with no Flask app context, so every
`validate_url_for_fetch()` call raised `RuntimeError`, for both
validation (incoming) and endpoint discovery and sending (outgoing).
2. `create_app()` registered a `teardown_appcontext` handler that called
`webmention_worker.stop()`. Teardown runs whenever any app context ends
(every request), so the worker stopped soon after startup.
## Fix
- `WebmentionWorker` takes the Flask app and runs each poll inside
`app.app_context()`, as `SyndicationWorker` already does.
- The teardown handler is removed. The worker is a daemon thread and ends with
the process.
## Tests
`tests/test_webmention_worker.py`: an outgoing send completes through the
worker thread, and the worker is still alive after an app context and a
request end. Both fail without the fix.
## Production data
After deploying, reset webmention 1 to `validation_status = 'pending'` and the
two queue items to `retry_count = 0, next_retry_at = NULL` so the worker
processes them again.
@@ -0,0 +1,36 @@
# v2.6.9: Webmention processing fixes
Found after v2.6.8 let the worker run in production, and by a review of the
implementation against the W3C Webmention Recommendation.
## Fixes
1. **Author photo with alt text** (`webmention_processor.extract_author`).
mf2py returns `u-photo` with an `alt` attribute as
`{'value': url, 'alt': ...}`; the dict was bound to SQLite and the update
failed. `_first_str()` reduces mf2 values to their `value`.
2. **Endpoint discovery** (`WebmentionWorker._discover_endpoint`).
`response.url` is an `httpx.URL`; `urljoin(str, URL)` raised `TypeError`,
which was caught and returned as "no endpoint", so every target was marked
`no_endpoint`. Pass `str(response.url)`. Fetch errors now propagate so the
caller schedules a retry, instead of being treated as "no endpoint".
3. **Duplicate processing.** gunicorn runs 4 processes, each with a worker,
and queue items were claimed by SELECT then UPDATE. The worker now takes a
non-blocking `flock` on `<data>/webmention-worker.lock` before processing;
others wait and retry each poll, so one takes over if the holder dies.
Orphan cleanup runs when the lock is acquired.
4. **XSS (spec 4.1).** `author_url` and `author_photo` are kept only if they
are absolute http(s) URLs; otherwise the source URL / no photo is used.
## Tests
`tests/test_webmention_worker.py`: the send test uses a real `httpx.Response`
(the old `MagicMock` with a string URL hid fix 2), plus tests for the lock,
an alt-text photo and a `javascript:` author URL. All fail on v2.6.8.
## Remaining spec gaps
Tracked separately: rel-token and document-order endpoint discovery, exact
target match during verification, Accept header, re-sending on delete, 410
for deleted notes, SSRF re-validation on redirects, and the others listed in
the compliance review.
@@ -0,0 +1,51 @@
# v2.7.0: Webmention W3C compliance
Review of StarPunk's Webmention support against the W3C Recommendation
(https://www.w3.org/TR/webmention/, 12 January 2017), and the changes made.
v2.6.8 and v2.6.9 fixed the bugs that stopped webmentions working at all.
## Sending
| Section | Requirement | Change |
|---|---|---|
| 3.1.2 | Link header with rel=webmention first (MUST) | RFC 8288 parsing: several links per header, several headers, `rel` in any position, token list, case-insensitive |
| 3.1.2 | Then first `<link>`/`<a>` rel=webmention in document order (MUST) | Single pass over `link` and `a` in document order; rel token match; elements without `href` skipped |
| 3.1.2 | Relative URL resolved against target; empty href is the target; query kept (MUST) | `urljoin` against the final URL after redirects |
| 3.1.2 | Follow redirects | `safe_get` follows up to 10, checking each hop |
| 3.1.2 | User-Agent with "Webmention" (MAY) | `StarPunk/<version> Webmention (+site)` |
| 3.1.3 | Form-encoded POST; any 2xx success (MUST) | Unchanged. Redirects not followed (httpx turned 301/302 POST into GET); 429 retried |
| 3.1.4 | Re-discover endpoint on update (MUST) | Discovery cache removed |
| 3.1.4 | Re-send to previously sent targets, including removed links (SHOULD) | `previous_targets()` from the queue, plus removed links; not capped |
| 3.1.5 | Deleted post returns 410 (SHOULD), tombstone (SHOULD) | `public.note` aborts 410 for soft-deleted notes and hard-deleted notes that sent webmentions; `410.html` |
| 3.1.5 | Re-send on delete (SHOULD) | `requeue_previous_webmentions()`; for a hard delete the rows are detached (`source_note_id = NULL`) first, since the foreign key cascades. Also on undelete |
## Receiving
| Section | Requirement | Change |
|---|---|---|
| 3.2 | 202 for async processing (MUST) | Unchanged |
| 3.2.1 | Valid http(s) URLs; source ≠ target; 400 on error (MUST) | Unchanged |
| 3.2.1 | Target is a valid resource (SHOULD) | Only published notes; others 400 |
| 3.2.1 | Fragment ignored | Stripped before storing (no duplicate rows) |
| 3.2.2 | GET source, follow redirects, limit redirects | `safe_get` |
| 3.2.2 | Accept header (SHOULD) | Added |
| 3.2.2 | Exact match of target, per media type (MUST) | HTML: every link attribute (`lxml iterlinks`) resolved against the final URL; JSON: any string value; `text/*`: substring; other types: no match. Fragments ignored |
| 3.2.4 | Update existing data; 410 / no link removes it (SHOULD) | Unchanged in effect. Moderation decisions kept on update; transient errors keep a valid mention |
## Security (section 4)
- 4.1 XSS: author URL/photo must be http(s) (v2.6.9); snippets re-sanitized after truncation; Mentions list uses `striptags`.
- 4.2 / 4.5: `safe_get` checks every redirect hop, checks all `getaddrinfo` addresses, blocks private, loopback, link-local, multicast, reserved, unspecified, CGNAT, `0.0.0.0/8` and IPv4-mapped forms; 1 MB and 10 s total limits.
- 4.3: endpoint URL checked before POST, no redirects followed.
## Known limits
- Homepage mentions are refused rather than stored.
- Discovery does not apply HTTP cache headers (5.3); targets are only
fetched when a note is created, edited or deleted.
- The 410 page is a generic tombstone without microformats.
- `error` rows (unexpected failures) are not retried automatically.
## Tests
`tests/test_webmention_spec.py` (section-numbered), `tests/test_webmention_worker.py`.
@@ -0,0 +1,36 @@
# v2.7.2: Remaining Webmention SHOULDs
Follow-up to the v2.7.0 compliance review. Every SHOULD in the
Recommendation was checked; two were not met.
## 4.5 DNS pinning
`validate_url_for_fetch()` now returns the checked address.
`_pinned_request()` builds each request against that IP, with the original
host in the `Host` header and, for https, in the `sni_hostname` extension.
httpcore uses `sni_hostname` as `server_hostname` with `check_hostname`
on, so the certificate is verified against the real hostname. Used by
`safe_get()` (every redirect hop) and the new `safe_post()` (the webmention
POST). IPv4 is preferred when a host has both families.
## 3.1.4 Displayed responses are an update
"If the source post displays responses... then sender SHOULD treat that as
an update of the source URL and re-send any previously sent Webmentions."
`note_responses_changed()` re-queues a published note's previous webmentions.
It is called when:
- the worker validates a webmention and what the note shows for it changed
(`_display_state()` before and after: shown or not, type, author, snippet).
A re-send from the source resets the row to `pending`; a previously valid
row still counts as shown, so an unchanged re-validation sends nothing.
This keeps two sites that both implement this from ping-ponging.
- an admin approves, hides or deletes webmentions and a valid one changes
between shown and not shown.
## Tests
`tests/test_webmention_spec.py`: pinning (URL, Host, SNI, one lookup per
request), IPv4 preference, re-send on a new response, none on an unchanged
one, re-send on moderation changes.
@@ -0,0 +1,38 @@
# v2.8.0: Microformats2 review
Review of StarPunk's published and consumed microformats against
microformats2, h-entry, h-card, h-feed, h-cite, representative h-card
parsing, the authorship spec and Post Type Discovery. Every page type was
rendered, parsed with mf2py and checked with mf2util.
Posts (h-entry) were largely correct: no implied names, PTD right for
note/article/reply/like/repost/photo, author h-card, u-url/u-uid,
p-category, u-syndication.
## Fixes
| # | Problem | Fix |
|---|---|---|
| 1 | Replies were unattached `h-cite` children with a plain-text author | `u-comment h-cite`, `p-author h-card` (u-url on an inner link, u-photo, p-name), `e-content` (`templates/partials/responses.html`) |
| 2 | Likes/reposts/bookmarks/mentions had no mf2 | `u-like` / `u-repost` / `u-bookmark` / `u-mention` h-cites; faces past 8 are `hidden`, not omitted |
| 3 | `dt-*` naive, with microseconds | `iso_utc` filter/helper: RFC 3339 UTC `Z`, no fraction; Micropub `q=source` too |
| 4 | `#hashtag` first line = title; bare `#` = slug title | `split_title` requires an ATX heading (`#{1,6}` + space); templates use `Note.has_title` |
| 5 | Note's own syndicated copy shown as a mention | Worker rejects a source whose url/uid (or URL) is one of the note's `syndications.remote_url` as invalid |
| 6 | Micropub/IndieAuth links gated on WEBMENTION_ENABLED | Only `rel=webmention` is gated |
| 7 | No rel=me on a fresh install | `rel=me` and `rel=author` to `author.url or author.me or ADMIN_ME`, always |
| 8 | Author discovery guessed h-cards | `mf2_utils.representative_hcard` (uid+url, rel=me, single h-card); parsed with the URL after redirects |
| 9 | Webmention entry/author/time | `mf2_utils.responding_entry` (entry whose in-reply-to/like-of/... or content refers to the target), `entry_author` (author property as h-card/URL/name, parent h-feed author, rel=author, representative h-card), `source_post_url` and `source_published` stored (migration 014) |
Also: tag and likes h-feeds have `u-url` and `p-author`; previews have `u-uid`.
## Not done
- Reply-context h-cite does not carry the cached author h-card, name or published time.
- Photo alt falls back to "Image" when a photo-only post has no caption.
- Person tags (`p-category h-card`) are not supported.
- Existing webmentions keep NULL source details until they are re-validated.
## Tests
`tests/test_microformats_v280.py`. `tests/test_microformats.py`
`test_no_relme_without_author` became `test_relme_to_admin_me_without_author`.
@@ -0,0 +1,33 @@
# v2.8.1: Migration lock held for the whole run
## Incident
Deploying v2.8.0 (migration 014 adds two columns): one gunicorn worker
applied 014, another failed with `duplicate column name: source_post_url`,
gunicorn shut down ("Worker failed to boot") and Docker restarted the
container. About two seconds of downtime; schema correct afterwards.
## Causes
`run_migrations` takes `BEGIN IMMEDIATE` so one worker migrates at a time,
but the lock was released early twice:
1. `create_migrations_table()`, called right after `BEGIN IMMEDIATE`,
ended with `conn.commit()`. Everything after it ran without the lock.
2. `conn.executescript()` COMMITs any open transaction before running the
script, so the lock was dropped before the migration was recorded in
`schema_migrations`.
## Fix
- `create_migrations_table(conn, commit=True)`; `run_migrations` passes
`commit=False`.
- `execute_in_transaction()` runs each statement from
`split_sql_statements()` (built on `sqlite3.complete_statement`, so
trigger bodies stay whole) with `conn.execute`, inside the transaction.
## Verification
Four processes migrating a database at 013 simultaneously: 6/30 trials
failed before, 0/30 after. `tests/test_migration_concurrency.py` runs this
five times, plus a rollback test showing a migration commits nothing early.
@@ -0,0 +1,51 @@
# v2.9.0: Micropub review
Review against the W3C Micropub Recommendation (https://www.w3.org/TR/micropub/),
run through the Flask test client and mapped to the micropub.rocks server
tests. Everyday create/update/delete/query/media/auth already passed; the
gaps were privacy, security and silent data loss.
## Decisions (user)
- No `mp-syndicate-to` from a Micropub client means no crossposting
(was: all targets, v2.1.0). The admin editor's default is unchanged.
- `visibility` private/unlisted are saved as drafts.
- Rendered HTML is sanitized with an allowlist (nh3 defaults plus class,
title, id, lang, dir on all tags; video/audio/source/track/picture).
## Changes
| Area | Before | After |
|---|---|---|
| `mp-*` in JSON `properties` | ignored (all targets, timestamp slug) | read as commands, never stored |
| absent `mp-syndicate-to` | all targets | none |
| `q=source` / `q=category` | any token | create, update, delete or draft scope |
| HTML content | rendered raw (stored XSS) | `render_markdown()` = Markdown + nh3 |
| `visibility` | ignored (published) | private/unlisted -> draft; unknown -> 400 |
| `draft` scope | 403 | creates drafts only |
| photo fetch/lookup failure | dropped, 201 | 400, nothing created or changed |
| remote photo fetch | first host checked, redirects followed | `webmention_ssrf.safe_get` (per hop, pinned) |
| JSON scalars | indexed per character | wrapped on create; update `delete` values must be arrays |
| `bookmark-of` | dropped | `notes.bookmark_of`, kind `bookmark` (migration 015) |
| HTML-authored content in `q=source` | string | `{"html": ...}` via `notes.content_format` |
| `summary` / `name` only | 400 | used as text / titled post |
| `video` / `audio` | dropped | 400 |
| `published` | stored with offset; garbage -> now | naive UTC; garbage -> 400 |
| malformed JSON | 500 | 400 |
| update without operations | 204 | 400 |
| auth | `Bearer` only; header+body accepted; no challenge | case-insensitive; both -> 400; `WWW-Authenticate` |
## Kept as is
- 403 (RFC 6750) for insufficient_scope; the Micropub table says 401.
- `add` only for category and photo; `published`, `syndication`,
`location` not updatable.
- `location`/`checkin` ignored (3.3.5 allows).
- GET `access_token` query parameter still accepted.
## Tests
`tests/test_micropub_v290.py` (31 of 34 fail on v2.8.1). Updated:
likes/reposts helper sends `mp-syndicate-to` in properties;
`test_micropub_default_is_no_targets`; remote photo test uses the pinned
fetcher; corrupted-image message.
+29
View File
@@ -0,0 +1,29 @@
-- Migration 012: Crossposting (POSSE) to syndication targets
-- Version: 2.1.0
-- Per ADR-065: Direct syndication to GoToSocial / Mastodon-API servers
--
-- One row per (note, target). The worker reconciles the remote post with the
-- note: create, update, or delete as needed. note_id has no foreign key on
-- purpose: a row must outlive a hard-deleted note so the remote post can be
-- deleted too.
CREATE TABLE IF NOT EXISTS syndications (
id INTEGER PRIMARY KEY AUTOINCREMENT,
note_id INTEGER NOT NULL,
target TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
remote_id TEXT,
remote_url TEXT,
state TEXT NOT NULL DEFAULT 'pending'
CHECK (state IN ('pending', 'synced', 'failed')),
version INTEGER NOT NULL DEFAULT 1,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
claimed_until TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE (note_id, target)
);
CREATE INDEX IF NOT EXISTS idx_syndications_due ON syndications(state, next_attempt_at);
+8
View File
@@ -0,0 +1,8 @@
-- Migration 013: Likes and reposts (v2.5.0, ADR-068)
-- A note with like_of is a like; with repost_of, a repost. Content is the
-- optional comment and may be empty.
ALTER TABLE notes ADD COLUMN like_of TEXT;
ALTER TABLE notes ADD COLUMN repost_of TEXT;
CREATE INDEX IF NOT EXISTS idx_notes_like_of ON notes(like_of);
@@ -0,0 +1,6 @@
-- Migration 014: Source details for received webmentions (v2.8.0)
-- The source's own u-url and dt-published, so a response links to and dates
-- the original post rather than the URL that sent it and when it arrived.
ALTER TABLE webmentions ADD COLUMN source_post_url TEXT;
ALTER TABLE webmentions ADD COLUMN source_published TIMESTAMP;
@@ -0,0 +1,7 @@
-- Migration 015: Bookmarks and HTML-authored content (v2.9.0)
-- bookmark_of: URL bookmarked (Micropub bookmark-of); the note is a bookmark.
-- content_format: 'html' when a Micropub client sent the content as HTML,
-- so q=source returns it as {"html": ...} (Micropub 3.7.2). NULL = Markdown.
ALTER TABLE notes ADD COLUMN bookmark_of TEXT;
ALTER TABLE notes ADD COLUMN content_format TEXT;
+55
View File
@@ -0,0 +1,55 @@
-- Migration 016: WebSub hub (v2.10.0, ADR-071)
-- Active and pending subscriptions to this site's topics
CREATE TABLE IF NOT EXISTS websub_subscriptions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
topic TEXT NOT NULL,
callback TEXT NOT NULL,
secret TEXT,
lease_seconds INTEGER NOT NULL,
expires_at TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
verified_at TIMESTAMP,
last_success_at TIMESTAMP,
last_error TEXT,
UNIQUE(topic, callback)
);
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_topic ON websub_subscriptions(topic);
-- Subscribe/unsubscribe requests waiting for verification of intent
CREATE TABLE IF NOT EXISTS websub_intents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
topic TEXT NOT NULL,
callback TEXT NOT NULL,
mode TEXT NOT NULL CHECK (mode IN ('subscribe', 'unsubscribe')),
secret TEXT,
lease_seconds INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE(topic, callback)
);
-- Content to push: one row per subscription (built-in hub) or per topic
-- (ping to an external hub, subscription_id NULL)
CREATE TABLE IF NOT EXISTS websub_deliveries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
subscription_id INTEGER REFERENCES websub_subscriptions(id) ON DELETE CASCADE,
topic TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_error TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_subscription
ON websub_deliveries(subscription_id) WHERE subscription_id IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_ping
ON websub_deliveries(topic) WHERE subscription_id IS NULL;
-- Site-wide values. content_version changes whenever a public note does, so
-- every gunicorn process knows its cached feed note list is stale.
CREATE TABLE IF NOT EXISTS site_state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
INSERT OR IGNORE INTO site_state (key, value) VALUES ('content_version', '0');
+12
View File
@@ -0,0 +1,12 @@
-- Migration 017: WebSub hardening (v2.10.1)
-- request_id: each subscription request gets its own id, so verifying an
-- older request can't consume a newer one for the same topic and callback.
-- requested_topic: the hub.topic exactly as sent; verification echoes it.
-- callback_ip: address the callback resolved to, for per-address limits.
ALTER TABLE websub_intents ADD COLUMN request_id TEXT;
ALTER TABLE websub_intents ADD COLUMN requested_topic TEXT;
ALTER TABLE websub_intents ADD COLUMN callback_ip TEXT;
ALTER TABLE websub_subscriptions ADD COLUMN callback_ip TEXT;
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_ip ON websub_subscriptions(callback_ip);
CREATE INDEX IF NOT EXISTS idx_websub_intents_ip ON websub_intents(callback_ip);
+48 -14
View File
@@ -191,6 +191,10 @@ def create_app(config=None):
register_routes(app)
# Template context processor - Inject author profile (v1.2.0 Phase 2)
from starpunk.utils import absolute_url_for
app.jinja_env.globals["absolute_url_for"] = absolute_url_for
@app.context_processor
def inject_author():
"""
@@ -215,6 +219,10 @@ def create_app(config=None):
return {'author': author}
# Template filter - RFC 3339 UTC for dt-* properties (v2.8.0)
from starpunk.utils import iso_utc
app.add_template_filter(iso_utc, 'iso_utc')
# Template filter - URL to color for facepile (v1.6.0)
@app.template_filter('url_to_color')
def url_to_color_filter(url):
@@ -276,6 +284,21 @@ def create_app(config=None):
"""Add correlation ID to request context for tracing"""
add_correlation_id()
# Know who is logged in on every page, not just /admin (v2.3.2), so
# the header can say "Admin" instead of "Log in". Only requests that
# carry a session cookie touch the database.
from flask import g, request
token = request.cookies.get("starpunk_session")
if token:
from starpunk.auth import verify_session
try:
info = verify_session(token)
except Exception:
info = None
if info:
g.user = info
g.me = info["me"]
# Response middleware - Add webmention Link header (v1.6.0)
@app.after_request
def add_webmention_headers(response):
@@ -283,9 +306,14 @@ def create_app(config=None):
if (app.config.get('WEBMENTION_ENABLED', True) and
response.content_type and
'text/html' in response.content_type):
response.headers['Link'] = '</webmention>; rel="webmention"'
response.headers.add('Link', '</webmention>; rel="webmention"')
return response
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
from starpunk.websub import add_discovery_headers, template_links
app.after_request(add_discovery_headers)
app.add_template_global(template_links, 'websub_links')
# Register centralized error handlers
from starpunk.errors import register_error_handlers
@@ -299,18 +327,15 @@ def create_app(config=None):
memory_monitor.start()
app.memory_monitor = memory_monitor
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
# Register cleanup handler
@app.teardown_appcontext
def cleanup_memory_monitor(error=None):
if hasattr(app, 'memory_monitor') and app.memory_monitor.is_alive():
app.memory_monitor.stop()
# No teardown_appcontext stop: that ran after every request and stopped
# the monitor straight away (v2.10.0). It is a daemon thread.
# Start webmention worker thread (v1.6.0 Phase 2)
# Per background-worker.md: Skip in test mode
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
from starpunk.webmention_worker import WebmentionWorker
webmention_worker = WebmentionWorker(
app,
db_path=app.config['DATABASE_PATH'],
site_url=app.config['SITE_URL'],
poll_interval=app.config.get('WEBMENTION_POLL_INTERVAL', 10),
@@ -326,11 +351,20 @@ def create_app(config=None):
f"batch_size={webmention_worker.batch_size})"
)
# Register cleanup handler
@app.teardown_appcontext
def cleanup_webmention_worker(error=None):
if hasattr(app, 'webmention_worker') and app.webmention_worker.is_alive():
app.webmention_worker.stop()
# Start WebSub worker (v2.10.0)
from starpunk.websub_worker import start_worker
app.websub_worker = start_worker(app)
if app.websub_worker:
app.logger.info("WebSub worker started")
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
app.syndication_worker = SyndicationWorker(
app, poll_interval=app.config["SYNDICATION_POLL_INTERVAL"]
)
app.syndication_worker.start()
app.logger.info("Syndication worker started")
# Health check endpoint for containers and monitoring
@app.route("/health")
@@ -436,5 +470,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.0.0"
__version_info__ = (2, 0, 0)
__version__ = "2.10.1"
__version_info__ = (2, 10, 1)
+13 -37
View File
@@ -82,10 +82,12 @@ def discover_author_profile(me_url: str) -> Optional[Dict]:
response.raise_for_status()
# Parse microformats from HTML
parsed = mf2py.parse(doc=response.text, url=me_url)
# Relative URLs and the representative h-card use the URL after redirects
page_url = str(response.url) if isinstance(response.url, (str, httpx.URL)) else me_url
parsed = mf2py.parse(doc=response.text, url=page_url)
# Extract h-card (per Q16: first representative h-card)
hcard = _find_representative_hcard(parsed, me_url)
# Representative h-card (v2.8.0: per the parsing spec)
hcard = _find_representative_hcard(parsed, page_url)
if not hcard:
current_app.logger.warning(f"No h-card found at {me_url}")
@@ -132,47 +134,21 @@ def discover_author_profile(me_url: str) -> Optional[Dict]:
def _find_representative_hcard(parsed: dict, me_url: str) -> Optional[dict]:
"""
Find representative h-card from parsed microformats
Find the representative h-card of the author's page (v2.8.0)
Per Q16: First representative h-card = first h-card with p-name
Per Q18: First h-card with url property matching profile URL
Follows https://microformats.org/wiki/representative-h-card-parsing
(uid+url match, rel=me match, or the only h-card with a matching url).
Other h-cards on the page may be other people, so none is guessed.
Args:
parsed: Parsed microformats data from mf2py
me_url: Profile URL for matching
me_url: Profile URL (the page that was parsed)
Returns:
h-card dict or None if not found
h-card dict or None if the page has no representative h-card
"""
items = parsed.get('items', [])
# First try: h-card with matching URL (most specific)
for item in items:
if 'h-card' in item.get('type', []):
properties = item.get('properties', {})
urls = properties.get('url', [])
# Check if any URL matches the profile URL
for url in urls:
if isinstance(url, dict):
url = url.get('value', '')
if _normalize_url(url) == _normalize_url(me_url):
# Found matching h-card
return item
# Second try: First h-card with p-name (representative h-card)
for item in items:
if 'h-card' in item.get('type', []):
properties = item.get('properties', {})
if properties.get('name'):
return item
# Third try: Just use first h-card if any
for item in items:
if 'h-card' in item.get('type', []):
return item
return None
from starpunk.mf2_utils import representative_hcard
return representative_hcard(parsed, me_url)
def _get_property(hcard: dict, prop_name: str) -> Optional[str]:
+28
View File
@@ -111,6 +111,13 @@ def load_config(app, config_override=None):
# Webmention configuration (v1.6.0)
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
# WebSub (v2.10.0, ADR-071): built-in hub unless WEBSUB_HUB_URL names another
app.config["WEBSUB_ENABLED"] = os.getenv("WEBSUB_ENABLED", "true").lower() == "true"
app.config["WEBSUB_HUB_URL"] = os.getenv("WEBSUB_HUB_URL", "").strip()
app.config["WEBSUB_MAX_SUBSCRIPTIONS"] = int(os.getenv("WEBSUB_MAX_SUBSCRIPTIONS", "1000"))
app.config["WEBSUB_MAX_PER_HOST"] = int(os.getenv("WEBSUB_MAX_PER_HOST", "25"))
app.config["WEBSUB_POLL_INTERVAL"] = int(os.getenv("WEBSUB_POLL_INTERVAL", "5"))
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
@@ -124,6 +131,27 @@ def load_config(app, config_override=None):
app.config["WEBMENTION_RATE_LIMIT_WINDOW"] = int(os.getenv("WEBMENTION_RATE_LIMIT_WINDOW", "60"))
app.config["TRUSTED_PROXY"] = os.getenv("TRUSTED_PROXY", "false").lower() == "true"
# Crossposting to GoToSocial / Mastodon-API (v2.1.0, ADR-065)
# Enabled when both URL and token are set
app.config["GOTOSOCIAL_URL"] = os.getenv("GOTOSOCIAL_URL", "")
app.config["GOTOSOCIAL_ACCESS_TOKEN"] = os.getenv("GOTOSOCIAL_ACCESS_TOKEN", "")
app.config["GOTOSOCIAL_NAME"] = os.getenv("GOTOSOCIAL_NAME", "GoToSocial")
app.config["GOTOSOCIAL_VISIBILITY"] = os.getenv("GOTOSOCIAL_VISIBILITY", "public")
app.config["GOTOSOCIAL_MAX_CHARS"] = int(os.getenv("GOTOSOCIAL_MAX_CHARS", "5000"))
app.config["GOTOSOCIAL_MAX_MEDIA"] = int(os.getenv("GOTOSOCIAL_MAX_MEDIA", "4"))
app.config["BLUESKY_HANDLE"] = os.getenv("BLUESKY_HANDLE", "")
app.config["BLUESKY_APP_PASSWORD"] = os.getenv("BLUESKY_APP_PASSWORD", "")
app.config["BLUESKY_SERVICE"] = os.getenv("BLUESKY_SERVICE", "https://bsky.social")
app.config["BLUESKY_NAME"] = os.getenv("BLUESKY_NAME", "Bluesky")
app.config["BLUESKY_MAX_CHARS"] = int(os.getenv("BLUESKY_MAX_CHARS", "300"))
# Crossposting to Threads (v2.6.0, ADR-069): long-lived token, refreshed automatically
app.config["THREADS_ACCESS_TOKEN"] = os.getenv("THREADS_ACCESS_TOKEN", "")
app.config["THREADS_USERNAME"] = os.getenv("THREADS_USERNAME", "")
app.config["THREADS_NAME"] = os.getenv("THREADS_NAME", "Threads")
app.config["THREADS_MAX_CHARS"] = int(os.getenv("THREADS_MAX_CHARS", "500"))
app.config["THREADS_MAX_MEDIA"] = int(os.getenv("THREADS_MAX_MEDIA", "4"))
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
# Apply overrides if provided
if config_override:
app.config.update(config_override)
+10
View File
@@ -107,6 +107,16 @@ class ConnectionPool:
if not conn:
return
# A request that failed mid-write must not hand an open transaction
# (and its write lock) to the next request (v2.6.5)
if conn.in_transaction:
try:
conn.rollback()
except Exception:
conn.close()
self._stats['connections_closed'] += 1
return
with self._lock:
if len(self._pool) < self.pool_size:
# Return to pool
+5
View File
@@ -82,6 +82,11 @@ def register_error_handlers(app):
return render_template('404.html'), 404
@app.errorhandler(410)
def gone(error):
"""Handle 410 Gone (deleted notes, Webmention 3.1.5)"""
return render_template('410.html'), 410
@app.errorhandler(405)
def method_not_allowed(error):
"""Handle 405 Method Not Allowed errors"""
+2 -2
View File
@@ -198,7 +198,7 @@ def generate_atom_streaming(
yield f' <link rel="enclosure" type="{_escape_xml(mime_type)}" href="{_escape_xml(media_url)}" length="{size}"{title_attr}/>\n'
# Content - include media as HTML (per Q24)
if note.html:
if note.feed_html:
# Build HTML content with media at top
html_content = ""
@@ -212,7 +212,7 @@ def generate_atom_streaming(
html_content += '</div>'
# Add text content below media
html_content += note.html
html_content += note.feed_html
# HTML content - escaped
yield ' <content type="html">'
+5 -1
View File
@@ -169,7 +169,11 @@ class FeedCache:
- Adds new entry or updates existing entry
"""
cache_key = self._generate_cache_key(format_name, notes_checksum)
etag = self._generate_etag(content)
# From what the feed is made of, not its bytes: the bytes include a
# build time, so each server process made a different ETag and
# conditional requests rarely matched (v2.10.1). Weak ETag: same
# notes, semantically the same feed.
etag = self._generate_etag(f"{format_name}:{notes_checksum}")
timestamp = time.time()
# Evict if cache is full
+2 -2
View File
@@ -267,7 +267,7 @@ def _build_item_object(site_url: str, note: Note) -> Dict[str, Any]:
# Add content (HTML or text)
# Per Q24: Include media as HTML in content_html
if note.html:
if note.feed_html:
content_html = ""
# Add media at top if present (v1.2.0 Phase 3)
@@ -280,7 +280,7 @@ def _build_item_object(site_url: str, note: Note) -> Dict[str, Any]:
content_html += '</div>'
# Add text content below media
content_html += note.html
content_html += note.feed_html
item["content_html"] = content_html
else:
item["content_text"] = note.content
+4 -4
View File
@@ -93,7 +93,7 @@ def generate_rss(
fg.language("en")
# Add self-link for feed discovery (Atom namespace)
fg.link(href=f"{site_url}/feed.xml", rel="self", type="application/rss+xml")
fg.link(href=f"{site_url}/feed.rss", rel="self", type="application/rss+xml")
# Set last build date to now
fg.lastBuildDate(datetime.now(timezone.utc))
@@ -137,7 +137,7 @@ def generate_rss(
html_content += '</div>'
# Add text content below media
html_content += clean_html_for_rss(note.html)
html_content += clean_html_for_rss(note.feed_html)
# feedgen automatically wraps content in CDATA for RSS
fe.description(html_content)
@@ -246,7 +246,7 @@ def generate_rss_streaming(
yield f" <description>{_escape_xml(site_description or site_name)}</description>\n"
yield " <language>en</language>\n"
yield f" <lastBuildDate>{last_build}</lastBuildDate>\n"
yield f' <atom:link href="{_escape_xml(site_url)}/feed.xml" rel="self" type="application/rss+xml"/>\n'
yield f' <atom:link href="{_escape_xml(site_url)}/feed.rss" rel="self" type="application/rss+xml"/>\n'
# Yield items (newest first)
# Notes from database are already in DESC order (newest first)
@@ -278,7 +278,7 @@ def generate_rss_streaming(
html_content += '</div>'
# Add text content below media
html_content += clean_html_for_rss(note.html)
html_content += clean_html_for_rss(note.feed_html)
# Build item XML
item_xml = f""" <item>
+1 -7
View File
@@ -22,6 +22,7 @@ from bs4 import BeautifulSoup
from flask import current_app
from starpunk.database import get_db
from starpunk.utils import absolute_url_for
CODE_LIFETIME_MINUTES = 10
SUPPORTED_SCOPES = ["create", "update", "delete", "media", "profile"]
@@ -55,13 +56,6 @@ def issuer() -> str:
return current_app.config["SITE_URL"].rstrip("/") + "/"
def site_url_for(endpoint: str) -> str:
"""Absolute URL built from SITE_URL (request host/scheme is unreliable behind a proxy)"""
from flask import url_for
return current_app.config["SITE_URL"].rstrip("/") + url_for(endpoint)
def admin_me() -> str:
"""The single identity StarPunk issues tokens for"""
return current_app.config["ADMIN_ME"]
+42
View File
@@ -0,0 +1,42 @@
"""
Markdown extension: turn bare URLs into links (v2.3.0)
Python-Markdown only links <https://...>; notes often contain pasted URLs.
"""
import re
import xml.etree.ElementTree as etree
from markdown.extensions import Extension
from markdown.inlinepatterns import InlineProcessor
from markdown.treeprocessors import Treeprocessor
from markdown.util import AtomicString
# Stop at whitespace and brackets; trailing punctuation is not part of the URL
BARE_URL_RE = r"(?<![\w/\"'=])https?://[^\s<>()\[\]\"']*[^\s<>()\[\]\"'.,;:!?…]"
class _BareUrl(InlineProcessor):
def handleMatch(self, m, data):
url = m.group(0)
el = etree.Element("a")
el.set("href", url)
el.text = AtomicString(url)
return el, m.start(0), m.end(0)
class _UnnestLinks(Treeprocessor):
"""[https://x](https://x) would put a link inside a link; keep the outer one"""
def run(self, root):
for outer in root.iter("a"):
for inner in list(outer.iter("a"))[1:]:
inner.tag = "span"
inner.attrib.clear()
class BareUrlExtension(Extension):
def extendMarkdown(self, md):
# Below links (160) and <autolinks> (120), so those win
md.inlinePatterns.register(_BareUrl(BARE_URL_RE, md), "bare_url", 105)
md.treeprocessors.register(_UnnestLinks(md), "unnest_links", 0)
+31 -14
View File
@@ -93,6 +93,10 @@ def get_optimization_params(file_size: int) -> Tuple[int, int]:
return (1280, 85)
# Shown to clients; the parser's own message names internal objects (v2.9.0)
UNSUPPORTED_IMAGE = "Not a supported image: use JPEG, PNG, GIF, WebP or HEIC"
def validate_image(file_data: bytes, filename: str) -> Tuple[Image.Image, str, int, int]:
"""
Validate image file and return PIL Image for memory-efficient processing
@@ -158,9 +162,9 @@ def validate_image(file_data: bytes, filename: str) -> Tuple[Image.Image, str, i
current_app.logger.info(f'Saved failed upload for analysis: {debug_file}')
except RuntimeError:
pass # Outside app context (e.g., tests)
raise ValueError(f"Invalid or corrupted image: {e}")
raise ValueError(UNSUPPORTED_IMAGE) from e
else:
raise ValueError(f"Invalid or corrupted image: {e}")
raise ValueError(UNSUPPORTED_IMAGE) from e
# Store original format before any conversions
original_format = img.format
@@ -868,20 +872,33 @@ def attach_media_to_note(note_id: int, media_ids: List[int], captions: List[str]
db = get_db(current_app)
# Delete existing associations (for edit case)
db.execute("DELETE FROM note_media WHERE note_id = ?", (note_id,))
try:
# Delete existing associations (for edit case)
db.execute("DELETE FROM note_media WHERE note_id = ?", (note_id,))
# Insert new associations
for i, (media_id, caption) in enumerate(zip(media_ids, captions)):
db.execute(
"""
INSERT INTO note_media (note_id, media_id, display_order, caption)
VALUES (?, ?, ?, ?)
""",
(note_id, media_id, i, caption or None)
)
# Insert new associations
for i, (media_id, caption) in enumerate(zip(media_ids, captions)):
db.execute(
"""
INSERT INTO note_media (note_id, media_id, display_order, caption)
VALUES (?, ?, ?, ?)
""",
(note_id, media_id, i, caption or None)
)
db.commit()
db.commit()
except Exception:
db.rollback()
raise
# Feeds and pages show the photos: refresh caches and tell WebSub (v2.10.1)
try:
from starpunk.notes import _websub_changed, get_note
note = get_note(id=note_id, load_content=False)
if note is not None:
_websub_changed(note, was_public=note.published)
except Exception as e:
current_app.logger.warning(f"Failed to queue WebSub update for note {note_id}: {e}")
def get_note_media(note_id: int) -> List[Dict]:
+168
View File
@@ -0,0 +1,168 @@
"""
Microformats2 consuming helpers (v2.8.0)
Shared by Webmention processing and author discovery:
- representative h-card of a page
- the h-entry on a page that responds to a given URL
- the author of an h-entry (authorship algorithm)
References:
- https://microformats.org/wiki/representative-h-card-parsing
- https://indieweb.org/authorship-spec
"""
from typing import Iterator, Optional
from urllib.parse import urldefrag, urlparse
RESPONSE_PROPERTIES = ('in-reply-to', 'like-of', 'repost-of', 'bookmark-of')
def normalize_url(url: str) -> str:
"""URL for comparison: no fragment, no trailing slash, lowercase scheme/host"""
if not url:
return ''
url = urldefrag(url.strip())[0]
parsed = urlparse(url)
url = parsed._replace(scheme=parsed.scheme.lower(), netloc=parsed.netloc.lower()).geturl()
return url.rstrip('/')
def value_of(value) -> Optional[str]:
"""Plain value of an mf2 property value (strings, {'value':...}, {'html':...})"""
if isinstance(value, dict):
value = value.get('value')
return value if isinstance(value, str) and value else None
def first(item: dict, prop: str) -> Optional[str]:
"""First plain value of an item's property"""
for value in item.get('properties', {}).get(prop, []):
result = value_of(value)
if result:
return result
return None
def values(item: dict, prop: str) -> list[str]:
"""All plain values of an item's property"""
return [v for v in (value_of(x) for x in item.get('properties', {}).get(prop, [])) if v]
def walk(items: list) -> Iterator[dict]:
"""Every microformat in items, depth first: top level, children, nested properties"""
for item in items:
if not isinstance(item, dict) or 'type' not in item:
continue
yield item
yield from walk(item.get('children', []))
for prop_values in item.get('properties', {}).values():
yield from walk([v for v in prop_values if isinstance(v, dict)])
def is_type(item: dict, mf_type: str) -> bool:
return mf_type in item.get('type', [])
def representative_hcard(parsed: dict, page_url: str) -> Optional[dict]:
"""
The page's representative h-card, per the parsing spec, in order:
1. An h-card whose uid and url both match the page URL
2. An h-card whose url matches one of the page's rel=me URLs
3. The only h-card on the page, if its url matches the page URL
Otherwise the page has none.
"""
page = normalize_url(page_url)
cards = [item for item in walk(parsed.get('items', [])) if is_type(item, 'h-card')]
for card in cards:
if page in map(normalize_url, values(card, 'uid')) and \
page in map(normalize_url, values(card, 'url')):
return card
rel_me = {normalize_url(u) for u in parsed.get('rels', {}).get('me', [])}
for card in cards:
if rel_me & set(map(normalize_url, values(card, 'url'))):
return card
if len(cards) == 1 and page in map(normalize_url, values(cards[0], 'url')):
return cards[0]
return None
def responding_entry(parsed: dict, target_url: str) -> Optional[dict]:
"""
The h-entry on a page that refers to target_url
Prefers an entry whose in-reply-to / like-of / repost-of / bookmark-of
is the target, then one whose content links to it, then the first
top-level h-entry.
"""
target = normalize_url(target_url)
entries = [item for item in walk(parsed.get('items', [])) if is_type(item, 'h-entry')]
top = [item for item in parsed.get('items', []) if is_type(item, 'h-entry')]
if not target:
return top[0] if top else (entries[0] if entries else None)
for entry in entries:
for prop in RESPONSE_PROPERTIES:
if target in map(normalize_url, values(entry, prop)):
return entry
for entry in entries:
for content in entry.get('properties', {}).get('content', []):
html = content.get('html', '') if isinstance(content, dict) else str(content)
if target_url in html or target in html:
return entry
return top[0] if top else (entries[0] if entries else None)
def response_type(entry: dict, target_url: str) -> str:
"""reply / like / repost / bookmark if the entry is one to target_url, else mention"""
target = normalize_url(target_url)
for prop, kind in zip(RESPONSE_PROPERTIES, ('reply', 'like', 'repost', 'bookmark')):
if target in map(normalize_url, values(entry, prop)):
return kind
return 'mention'
def entry_author(parsed: dict, entry: Optional[dict], page_url: str) -> dict:
"""
Author of an h-entry, following the authorship algorithm without fetching
1. The entry's author property (an h-card, a URL, or a name)
2. The author of an h-feed that contains the entry
3. The page's rel=author link
4. The page's representative h-card
Returns:
{'name', 'url', 'photo'}; values may be None
"""
author = None
if entry:
author = (entry.get('properties', {}).get('author') or [None])[0]
if author is None:
for feed in walk(parsed.get('items', [])):
if is_type(feed, 'h-feed') and entry in feed.get('children', []):
author = (feed.get('properties', {}).get('author') or [None])[0]
break
if author is None:
rel_author = parsed.get('rels', {}).get('author', [])
author = rel_author[0] if rel_author else representative_hcard(parsed, page_url)
if isinstance(author, dict) and 'type' in author:
return {
'name': first(author, 'name'),
'url': first(author, 'url'),
'photo': first(author, 'photo'),
}
text = value_of(author)
if text and urlparse(text).scheme in ('http', 'https'):
return {'name': None, 'url': text, 'photo': None}
return {'name': text, 'url': None, 'photo': None}
+624 -220
View File
File diff suppressed because it is too large. Load diff
+42 -6
View File
@@ -31,12 +31,47 @@ class MigrationError(Exception):
pass
def create_migrations_table(conn):
def split_sql_statements(script: str) -> list[str]:
"""
Split a SQL script into complete statements (v2.8.1)
Uses sqlite3.complete_statement, so semicolons inside strings and
trigger bodies (BEGIN ... END) don't split a statement.
"""
statements = []
current = ""
for line in script.splitlines(keepends=True):
current += line
if sqlite3.complete_statement(current):
if current.strip().strip(";").strip():
statements.append(current.strip())
current = ""
if current.strip():
statements.append(current.strip())
return statements
def execute_in_transaction(conn, script: str) -> None:
"""
Run a SQL script inside the connection's open transaction
Not conn.executescript(): that COMMITs first, which released the
migration lock mid-migration and let another gunicorn worker apply the
same migration again ("duplicate column name", v2.8.0 deploy).
"""
for statement in split_sql_statements(script):
conn.execute(statement)
def create_migrations_table(conn, commit=True):
"""
Create schema_migrations tracking table if it doesn't exist
Args:
conn: SQLite connection
commit: Commit afterwards. run_migrations passes False: committing
there released its BEGIN IMMEDIATE lock straight away, so
gunicorn workers applied migrations concurrently (v2.8.1)
"""
conn.execute("""
CREATE TABLE IF NOT EXISTS schema_migrations (
@@ -51,7 +86,8 @@ def create_migrations_table(conn):
ON schema_migrations(migration_name)
""")
conn.commit()
if commit:
conn.commit()
def is_schema_current(conn):
@@ -287,7 +323,7 @@ def apply_migration(conn, migration_name, migration_path, logger=None):
# Execute migration in transaction
conn.execute("BEGIN")
conn.executescript(migration_sql)
execute_in_transaction(conn, migration_sql)
# Record migration as applied
conn.execute(
@@ -374,7 +410,7 @@ def run_migrations(db_path, logger=None):
try:
# Ensure migrations tracking table exists
create_migrations_table(conn)
create_migrations_table(conn, commit=False)
# Quick check: have migrations already been applied by another worker?
cursor = conn.execute("SELECT COUNT(*) FROM schema_migrations")
@@ -472,8 +508,8 @@ def run_migrations(db_path, logger=None):
logger.debug(f"Applying migration: {migration_name}")
# Execute migration (already in transaction)
conn.executescript(migration_sql)
# Execute migration inside the locked transaction
execute_in_transaction(conn, migration_sql)
# Record migration as applied
conn.execute(
+118 -9
View File
@@ -11,10 +11,12 @@ from dataclasses import dataclass, field, replace
from datetime import datetime, timedelta
from pathlib import Path
from typing import Any, Optional
import re
import sqlite3
# Third-party imports
import markdown
import nh3
# Local imports
from starpunk.utils import (
@@ -34,13 +36,63 @@ DEFAULT_AUTH_STATE_EXPIRY_MINUTES = 5
DEFAULT_TOKEN_EXPIRY_DAYS = 90
# Constants - Markdown rendering
MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br"]
from starpunk.markdown_ext import BareUrlExtension
MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br", BareUrlExtension()]
# Constants - Content limits
MAX_TITLE_LENGTH = 200
# HTML allowed in rendered notes (v2.9.0). Markdown passes raw HTML through,
# and Micropub clients may send HTML, so scripts, event handlers, iframes and
# javascript: links are stripped. class keeps code highlighting and
# microformats; id keeps footnote links.
ALLOWED_TAGS = set(nh3.ALLOWED_TAGS) | {"video", "audio", "source", "track", "picture"}
ALLOWED_ATTRIBUTES = {
**{tag: set(attrs) for tag, attrs in nh3.ALLOWED_ATTRIBUTES.items()},
"*": {"class", "title", "id", "lang", "dir"},
"img": set(nh3.ALLOWED_ATTRIBUTES.get("img", ())) | {"loading", "srcset", "sizes"},
"video": {"src", "controls", "poster", "width", "height", "loop", "muted", "playsinline", "preload"},
"audio": {"src", "controls", "loop", "muted", "preload"},
"source": {"src", "type", "srcset", "media", "sizes"},
"track": {"src", "kind", "label", "srclang", "default"},
}
def render_markdown(text: str) -> str:
"""Note Markdown (or HTML) to sanitized HTML"""
html = markdown.markdown(text, extensions=MARKDOWN_EXTENSIONS)
return nh3.clean(html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
# Likes, reposts and bookmarks (v2.5.0, v2.9.0)
REACTION_VERBS = {"like": "Liked", "repost": "Reposted", "bookmark": "Bookmarked"}
REACTION_ICONS = {"like": "★", "repost": "🔁", "bookmark": "🔖"}
# A Markdown ATX heading: up to 6 '#', whitespace, then text
TITLE_RE = re.compile(r"#{1,6}[ \t]+(\S.*)$")
EXCERPT_LENGTH = 200
def split_title(content: str) -> tuple[Optional[str], str]:
"""
Split a note's '# Heading' first line (its title, per Q22) off its content
The first line must be a Markdown heading: 1-6 '#' then a space and text.
'#indieweb ...' (a hashtag) or a bare '#' is not a title. This is the one
place that decides whether a note has a title (templates use has_title).
Returns:
(title, remaining content); title is None if the note has no heading
"""
stripped = content.lstrip()
first_line, _, rest = stripped.partition("\n")
match = TITLE_RE.match(first_line)
if not match:
return None, content
return match.group(1).strip().rstrip("#").strip(), rest.strip()
@dataclass(frozen=True)
class Note:
"""
@@ -115,6 +167,10 @@ class Note:
deleted_at: Optional[datetime] = None
content_hash: Optional[str] = None
in_reply_to: Optional[str] = None # v1.6.0: URL being replied to
like_of: Optional[str] = None # v2.5.0: URL liked (this note is a like)
repost_of: Optional[str] = None # v2.5.0: URL reposted (this note is a repost)
bookmark_of: Optional[str] = None # v2.9.0: URL bookmarked (this note is a bookmark)
content_format: Optional[str] = None # v2.9.0: 'html' if sent as HTML via Micropub
_cached_content: Optional[str] = field(
default=None, repr=False, compare=False, init=False
)
@@ -173,6 +229,10 @@ class Note:
_data_dir=data_dir,
content_hash=data.get("content_hash"),
in_reply_to=data.get("in_reply_to"),
like_of=data.get("like_of"),
repost_of=data.get("repost_of"),
bookmark_of=data.get("bookmark_of"),
content_format=data.get("content_format"),
)
@property
@@ -231,17 +291,35 @@ class Note:
<p>This is my note content...</p>
"""
if self._cached_html is None:
# Render markdown to HTML
html = markdown.markdown(
self.content, # This triggers lazy load of content
extensions=MARKDOWN_EXTENSIONS,
)
# Render markdown to sanitized HTML (v2.9.0)
html = render_markdown(self.content) # triggers lazy load of content
# Cache it
object.__setattr__(self, "_cached_html", html)
return self._cached_html
@property
def body_html(self) -> str:
"""
Rendered HTML without the title heading
Pages show a titled note's title as their own heading, so the body
leaves it out to avoid showing it twice. Untitled notes: same as html.
"""
title, body = split_title(self.content)
if title is None:
return self.html
return render_markdown(body)
@property
def has_title(self) -> bool:
"""True if the note starts with a '# Heading' title (an article)"""
try:
return split_title(self.content)[0] is not None
except (FileNotFoundError, OSError):
return False
@property
def title(self) -> str:
"""
@@ -309,10 +387,10 @@ class Note:
'This is my note content. It has some interesting points...'
"""
try:
# Get content
content = self.content
# Leave out the title; previews show it separately
_, content = split_title(self.content)
# Remove markdown heading syntax from first line
# Remove markdown heading syntax
lines = content.split("\n")
cleaned_lines = []
for line in lines:
@@ -335,12 +413,43 @@ class Note:
excerpt = excerpt[:last_space]
excerpt += "..."
if not excerpt and self.reacts_to:
# A like, repost or bookmark with no comment
return f"{REACTION_VERBS[self.kind]} {self.reacts_to}"
return excerpt
except (FileNotFoundError, OSError):
# If file doesn't exist, return slug
return self.slug
@property
def kind(self) -> str:
"""'like', 'repost', 'bookmark' (v2.9.0) or 'note' (v2.5.0)"""
if self.like_of:
return "like"
if self.repost_of:
return "repost"
if self.bookmark_of:
return "bookmark"
return "note"
@property
def reacts_to(self) -> Optional[str]:
"""The URL a like, repost or bookmark points at"""
return self.like_of or self.repost_of or self.bookmark_of
@property
def feed_html(self) -> str:
"""HTML for feeds: a repost/like line, then the note's own HTML"""
from html import escape
if not self.reacts_to:
return self.html
url = escape(self.reacts_to, quote=True)
line = f'<p>{REACTION_ICONS[self.kind]} {REACTION_VERBS[self.kind]} <a href="{url}">{url}</a></p>'
return line + (self.html or "")
@property
def permalink(self) -> str:
"""
+174 -21
View File
@@ -133,13 +133,22 @@ def _get_existing_slugs(db) -> set[str]:
# Core CRUD Functions
# update_note: leave a field as it is
UNCHANGED = object()
def create_note(
content: str,
published: bool = False,
created_at: Optional[datetime] = None,
custom_slug: Optional[str] = None,
tags: Optional[list[str]] = None,
in_reply_to: Optional[str] = None
in_reply_to: Optional[str] = None,
like_of: Optional[str] = None,
repost_of: Optional[str] = None,
has_media: bool = False,
bookmark_of: Optional[str] = None,
content_format: Optional[str] = None,
) -> Note:
"""
Create a new note
@@ -203,7 +212,14 @@ def create_note(
- created_at and updated_at set to same value initially
"""
# 1. VALIDATION (before any changes)
if not content or not content.strip():
# Likes and reposts may have no comment (v2.5.0), photo posts no text (v2.6.5)
# Bookmarks too (v2.9.0)
if sum(bool(u) for u in (like_of, repost_of, bookmark_of)) > 1:
raise InvalidNoteDataError(
"like_of", like_of, "A note can only be one of a like, repost or bookmark"
)
content = content or ""
if not content.strip() and not (like_of or repost_of or bookmark_of or has_media):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
@@ -264,10 +280,12 @@ def create_note(
try:
db.execute(
"""
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash, in_reply_to)
VALUES (?, ?, ?, ?, ?, ?, ?)
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash,
in_reply_to, like_of, repost_of, bookmark_of, content_format)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(slug, file_path_rel, published, created_at, updated_at, content_hash, in_reply_to),
(slug, file_path_rel, published, created_at, updated_at, content_hash,
in_reply_to, like_of, repost_of, bookmark_of, content_format),
)
db.commit()
except Exception as e:
@@ -322,40 +340,65 @@ def create_note(
from starpunk.webmention_sender import queue_webmentions_for_note
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
note_url = f"{site_url}/note/{slug}"
from starpunk.utils import absolute_url_for
note_url = absolute_url_for("public.note", slug=slug)
# Get rendered HTML from note
html_content = note.html
# Queue webmentions (best-effort, failures logged but don't fail note creation)
# The reply/like/repost target is notified first (v2.5.0)
queue_webmentions_for_note(
db_path=db_path,
note_id=note_id,
note_url=note_url,
content_html=html_content,
in_reply_to=in_reply_to
in_reply_to=in_reply_to or like_of or repost_of or bookmark_of
)
except Exception as e:
# Webmention queueing failure should not prevent note creation
current_app.logger.warning(f"Failed to queue webmentions for note {slug}: {e}")
# 12. FETCH REPLY CONTEXT (v1.6.0 Phase 3)
# Fetch context if this is a reply
if in_reply_to:
# Fetch context for the post being replied to, liked or reposted
context_url = in_reply_to or like_of or repost_of or bookmark_of
if context_url:
try:
from starpunk.reply_context import fetch_reply_context
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
# Fetch context (best-effort, failures logged but don't fail note creation)
fetch_reply_context(db_path, in_reply_to, site_url)
fetch_reply_context(db_path, context_url, site_url)
except Exception as e:
# Reply context fetch failure should not prevent note creation
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
# A note with photos is announced when they're attached
# (attach_media_to_note), so subscribers never get it without them
if not has_media:
_websub_changed(note)
return note
def _tag_names(note_id: int) -> list[str]:
"""A note's tag names, for WebSub tag topics (best effort)"""
try:
from starpunk.tags import get_note_tags
return [t["name"] for t in get_note_tags(note_id)]
except Exception:
return []
def _websub_changed(note, was_public: bool = False, old_tags=()) -> None:
"""Clear cached feeds and queue WebSub deliveries (v2.10.0, best effort)"""
try:
from starpunk import websub
websub.note_changed(note, was_public=was_public, old_tags=old_tags)
except Exception as e:
current_app.logger.warning(f"Failed to queue WebSub update for note {note.slug}: {e}")
def get_note(
slug: Optional[str] = None, id: Optional[int] = None, load_content: bool = True
) -> Optional[Note]:
@@ -472,6 +515,7 @@ def list_notes(
offset: int = 0,
order_by: str = "created_at",
order_dir: str = "DESC",
likes: Optional[bool] = None,
) -> list[Note]:
"""
List notes with filtering and pagination
@@ -564,6 +608,11 @@ def list_notes(
params = []
if published_only:
query += " AND published = 1"
# likes: None = all notes, False = leave likes out, True = only likes (v2.5.0)
if likes is True:
query += " AND like_of IS NOT NULL"
elif likes is False:
query += " AND like_of IS NULL"
# Add ordering (safe because order_by validated above)
query += f" ORDER BY {order_by} {order_dir}"
@@ -589,11 +638,15 @@ def update_note(
content: Optional[str] = None,
published: Optional[bool] = None,
tags: Optional[list[str]] = None,
in_reply_to: Optional[str] = None
in_reply_to: Optional[str] = None,
has_media: bool = False,
content_format=UNCHANGED,
) -> Note:
"""
Update a note's content and/or published status
content_format (v2.9.0): 'html' or None (Markdown); left alone unless given
Updates note content and/or metadata, maintaining atomic synchronization
between file and database. At least one of content or published must
be provided.
@@ -663,12 +716,9 @@ def update_note(
if content is None and published is None and tags is None and in_reply_to is None:
raise ValueError("Must provide at least one of content, published, tags, or in_reply_to to update")
# Validate content if provided
if content is not None:
if not content or not content.strip():
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
# Validate content if provided (checked against the note below for
# likes/reposts, whose comment may be empty)
empty_content = content is not None and not content.strip()
# 2. GET EXISTING NOTE
# Phase 6: Load content to get old HTML for webmention URL diffing
@@ -677,6 +727,12 @@ def update_note(
if existing_note is None:
identifier = slug if slug is not None else id
raise NoteNotFoundError(identifier)
old_tag_names = _tag_names(existing_note.id) # for WebSub tag topics (v2.10.0)
if empty_content and not (existing_note.reacts_to or has_media):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
# 3. SETUP
updated_at = datetime.utcnow()
@@ -737,6 +793,10 @@ def update_note(
update_fields.append("in_reply_to = ?")
params.append(in_reply_to)
if content_format is not UNCHANGED:
update_fields.append("content_format = ?")
params.append(content_format)
# Add WHERE clause parameter
if slug is not None:
where_clause = "slug = ?"
@@ -815,13 +875,14 @@ def update_note(
from starpunk.webmention_sender import queue_webmentions_for_note
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
note_url = f"{site_url}/note/{updated_note.slug}"
from starpunk.utils import absolute_url_for
note_url = absolute_url_for("public.note", slug=updated_note.slug)
# Get rendered HTML from updated note (if published)
html_content = updated_note.html if updated_note.published else None
# Determine in_reply_to (use new value if provided, otherwise existing)
reply_to = in_reply_to if in_reply_to is not None else updated_note.in_reply_to
reply_to = (in_reply_to if in_reply_to is not None else updated_note.in_reply_to) or updated_note.reacts_to
# Queue webmentions (best-effort)
# Pass old_html for URL diffing (Phase 6)
@@ -838,8 +899,8 @@ def update_note(
current_app.logger.warning(f"Failed to queue webmentions for note {updated_note.slug}: {e}")
# 9. FETCH REPLY CONTEXT (v1.6.0 Phase 3)
# Fetch context if in_reply_to changed and is not None
if in_reply_to is not None:
# Fetch context if in_reply_to changed and is set ("" clears it)
if in_reply_to:
try:
from starpunk.reply_context import fetch_reply_context
db_path = Path(current_app.config["DATABASE_PATH"])
@@ -851,6 +912,15 @@ def update_note(
# Reply context fetch failure should not prevent note update
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
# Crossposts follow the note (v2.1.0, best-effort)
try:
from starpunk.syndication import note_changed
note_changed(updated_note.id)
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication update: {e}")
_websub_changed(updated_note, was_public=existing_note.published, old_tags=old_tag_names)
# 10. RETURN UPDATED NOTE
return updated_note
@@ -950,6 +1020,10 @@ def delete_note(
# For idempotency, don't raise error - just return
return
# What feeds showed before, for WebSub (v2.10.0)
was_public = bool(existing_note.published) and not getattr(existing_note, "deleted_at", None)
deleted_tags = _tag_names(existing_note.id)
# 4. SETUP
data_dir = Path(current_app.config["DATA_PATH"])
note_path = data_dir / existing_note.file_path
@@ -962,6 +1036,17 @@ def delete_note(
"Path validation failed",
)
# Re-send webmentions so receivers learn the post is gone (spec 3.1.5).
# A hard delete would cascade the queue rows away, so detach them first.
if existing_note.published:
try:
from starpunk.webmention_sender import requeue_previous_webmentions
requeue_previous_webmentions(
Path(current_app.config["DATABASE_PATH"]), existing_note.id, detach=not soft
)
except Exception as e:
current_app.logger.warning(f"Failed to queue webmentions for deleted note: {e}")
# 5. PERFORM DELETION
db = get_db(current_app)
@@ -1018,5 +1103,73 @@ def delete_note(
)
# Don't fail - database record already deleted
# Delete crossposts too (v2.1.0, best-effort)
try:
from starpunk.syndication import note_changed
note_changed(existing_note.id)
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
# Feeds and pages no longer show it (v2.10.0)
object.__setattr__(existing_note, "published", False)
_websub_changed(existing_note, was_public=was_public, old_tags=deleted_tags)
# 6. RETURN (no value)
return None
def undelete_note(slug: str) -> Note:
"""
Restore a soft-deleted note (v2.4.0, Micropub undelete)
Moves the file back from .trash/ and clears deleted_at. Crossposts are
queued again, so a note deleted from them is posted again.
Raises:
NoteNotFoundError: If there is no soft-deleted note with this slug
"""
import shutil
from starpunk.utils import TRASH_DIR_NAME
db = get_db(current_app)
row = db.execute(
"SELECT id, file_path, created_at FROM notes WHERE slug = ? AND deleted_at IS NOT NULL",
(slug,),
).fetchone()
if row is None:
raise NoteNotFoundError(slug)
data_dir = Path(current_app.config["DATA_PATH"])
note_path = data_dir / row["file_path"]
if not note_path.exists():
# delete_note_file put it in .trash/<year>/<month>/<name>
parts = Path(row["file_path"]).parts
candidates = []
if "notes" in parts and len(parts) > parts.index("notes") + 2:
i = parts.index("notes")
candidates.append(data_dir / TRASH_DIR_NAME / parts[i + 1] / parts[i + 2] / note_path.name)
candidates += sorted((data_dir / TRASH_DIR_NAME).glob(f"*/*/{note_path.name}"))
source = next((c for c in candidates if c.exists()), None)
if source is None:
raise NoteSyncError("undelete", f"Trashed file not found for {slug}", "Cannot restore note file")
note_path.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(source), str(note_path))
db.execute("UPDATE notes SET deleted_at = NULL WHERE id = ?", (row["id"],))
db.commit()
try:
from starpunk.syndication import note_changed
note_changed(row["id"])
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication after undelete: {e}")
try:
from starpunk.webmention_sender import requeue_previous_webmentions
requeue_previous_webmentions(Path(current_app.config["DATABASE_PATH"]), row["id"])
except Exception as e:
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
restored = get_note(slug=slug)
_websub_changed(restored)
return restored
+4 -1
View File
@@ -7,7 +7,7 @@ admin, auth, and (conditionally) dev auth routes.
from flask import Flask
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention, websub
def register_routes(app: Flask) -> None:
@@ -44,6 +44,9 @@ def register_routes(app: Flask) -> None:
if app.config.get("WEBMENTION_ENABLED", True):
app.register_blueprint(webmention.bp)
# WebSub hub (v2.10.0); refuses requests when disabled or using another hub
app.register_blueprint(websub.bp)
# Conditionally register dev auth routes
if app.config.get("DEV_MODE"):
app.logger.warning(
+57 -6
View File
@@ -72,12 +72,23 @@ def dashboard():
except Exception:
pass
# WebSub subscribers (v2.10.0)
websub_stats = None
if current_app.config.get('WEBSUB_ENABLED', True):
try:
from starpunk import websub
from starpunk.database import get_db
websub_stats = websub.stats(get_db(current_app))
except Exception:
pass
return render_template(
"admin/dashboard.html",
notes=notes,
page=page,
total_pages=total_pages,
webmention_pending_count=webmention_pending_count,
websub_stats=websub_stats,
user_me=g.me
)
@@ -97,7 +108,9 @@ def new_note_form():
Decorator: @require_auth
Template: templates/admin/new.html
"""
return render_template("admin/new.html")
from starpunk.syndication import get_targets
return render_template("admin/new.html", syndication_targets=get_targets().values())
@bp.route("/new", methods=["POST"])
@@ -137,13 +150,17 @@ def create_note_submit():
# Parse tags (v1.3.0 Phase 3)
tags = parse_tag_input(tags_input)
# With photos, WebSub hears about the note once they're attached (v2.10.1)
has_uploads = any(f.filename for f in request.files.getlist('media_files'))
try:
# Create note first (per Q4)
note = create_note(
content,
published=published,
custom_slug=custom_slug if custom_slug else None,
tags=tags if tags else None
tags=tags if tags else None,
has_media=has_uploads,
)
# Handle media uploads (v1.2.0 Phase 3)
@@ -182,6 +199,16 @@ def create_note_submit():
if errors:
flash(f"Note created, but some images failed: {'; '.join(errors)}", "warning")
if not media_ids:
# No photo made it; announce the note as it is (v2.10.1)
from starpunk.notes import _websub_changed
_websub_changed(note)
# Crosspost after media is attached so images go out too (v2.1.0)
if "syndication_form" in request.form:
from starpunk.syndication import select_targets
select_targets(note.id, request.form.getlist("syndicate_to"))
flash(f"Note created: {note.slug}", "success")
return redirect(url_for("admin.dashboard"))
@@ -219,11 +246,21 @@ def edit_note_form(note_id: int):
flash("Note not found", "error")
return redirect(url_for("admin.dashboard")), 404
# Pre-load tags for the edit form (v1.3.0 Phase 3)
tags = get_note_tags(note.id)
# Pre-load tags for the edit form (v1.3.0 Phase 3), as entered so saving keeps the order
tags = get_note_tags(note.id, entered_order=True)
object.__setattr__(note, '_cached_tags', tags)
return render_template("admin/edit.html", note=note)
from starpunk.syndication import get_note_syndications, get_targets
syndications = get_note_syndications(note.id)
enabled = {s["target"] for s in syndications if s["enabled"]}
return render_template(
"admin/edit.html",
note=note,
syndication_targets=get_targets().values(),
syndications=syndications,
syndication_enabled=enabled,
)
@bp.route("/edit/<int:note_id>", methods=["POST"])
@@ -260,7 +297,7 @@ def update_note_submit(note_id: int):
published = "published" in request.form
tags_input = request.form.get("tags", "")
if not content:
if not content and not existing_note.reacts_to: # likes/reposts may have no comment
flash("Content cannot be empty", "error")
return redirect(url_for("admin.edit_note_form", note_id=note_id))
@@ -274,6 +311,9 @@ def update_note_submit(note_id: int):
published=published,
tags=tags if tags else None
)
if "syndication_form" in request.form:
from starpunk.syndication import select_targets
select_targets(note.id, request.form.getlist("syndicate_to"))
flash(f"Note updated: {note.slug}", "success")
return redirect(url_for("admin.dashboard"))
except ValueError as e:
@@ -838,3 +878,14 @@ def revoke_token(token_id):
revoke_token_by_id(token_id)
flash("App access revoked", "success")
return redirect(url_for("admin.tokens"))
@bp.route("/syndication/<int:syndication_id>/retry", methods=["POST"])
@require_auth
def retry_syndication(syndication_id):
"""Retry a crosspost now"""
from starpunk.syndication import retry
retry(syndication_id)
flash("Crosspost queued", "success")
return redirect(request.referrer or url_for("admin.dashboard"))
+10 -6
View File
@@ -34,11 +34,11 @@ from starpunk.indieauth import (
issuer,
redeem_authorization_code,
revoke_access_token,
site_url_for,
validate_authorization_request,
verify_access_token,
)
from starpunk.micropub import extract_bearer_token
from starpunk.micropub import MicropubError, extract_bearer_token
from starpunk.utils import absolute_url_for
bp = Blueprint("indieauth", __name__)
@@ -89,9 +89,9 @@ def metadata():
return _json(
{
"issuer": issuer(),
"authorization_endpoint": site_url_for("indieauth.authorize"),
"token_endpoint": site_url_for("indieauth.token"),
"revocation_endpoint": site_url_for("indieauth.revoke"),
"authorization_endpoint": absolute_url_for("indieauth.authorize"),
"token_endpoint": absolute_url_for("indieauth.token"),
"revocation_endpoint": absolute_url_for("indieauth.revoke"),
"revocation_endpoint_auth_methods_supported": ["none"],
"scopes_supported": SUPPORTED_SCOPES,
"response_types_supported": ["code"],
@@ -215,7 +215,11 @@ def token():
@bp.route("/auth/token", methods=["GET"])
def token_verify():
"""Legacy token verification for clients that still call it"""
info = verify_access_token(extract_bearer_token(request))
try:
token = extract_bearer_token(request)
except MicropubError as e:
return _oauth_error("invalid_request", e.error_description)
info = verify_access_token(token)
if not info:
return _oauth_error("unauthorized", "Invalid or expired access token", 401)
return _json(info)
+30 -21
View File
@@ -27,6 +27,8 @@ from starpunk.micropub import (
extract_bearer_token,
error_response,
handle_create,
handle_delete,
handle_update,
handle_query,
)
from starpunk.indieauth import verify_access_token, check_scope
@@ -49,8 +51,8 @@ def micropub_endpoint():
POST requests:
Handle action operations (form-encoded or JSON):
- action=create (or no action): Create new post
- action=update: Update existing post (not supported in V1)
- action=delete: Delete post (not supported in V1)
- action=update: Update existing post (JSON; replace/add/delete)
- action=delete / action=undelete: Delete or restore a post
Authentication:
Requires valid bearer token in Authorization header or
@@ -68,7 +70,10 @@ def micropub_endpoint():
}
"""
# Extract and verify token
token = extract_bearer_token(request)
try:
token = extract_bearer_token(request)
except MicropubError as e:
return error_response(e.error, e.error_description, e.status_code)
if not token:
return error_response("unauthorized", "No access token provided", 401)
@@ -79,7 +84,7 @@ def micropub_endpoint():
# Handle query endpoints (GET requests)
if request.method == "GET":
try:
return handle_query(request.args.to_dict(), token_info)
return handle_query(request.args, token_info)
except MicropubError as e:
return error_response(e.error, e.error_description, e.status_code)
except Exception as e:
@@ -92,26 +97,25 @@ def micropub_endpoint():
try:
# Parse request based on content type
if "application/json" in content_type:
data = request.get_json() or {}
# Malformed JSON is the client's error, not ours (v2.9.0)
data = request.get_json(silent=True)
if not isinstance(data, dict):
return error_response("invalid_request", "Request body must be a JSON object", 400)
action = data.get("action", "create")
else:
# Form-encoded or multipart (V1 only supports form-encoded)
# Form-encoded or multipart (files handled separately)
data = request.form.to_dict(flat=False)
action = data.get("action", ["create"])[0]
# Route to appropriate handler
if action == "create":
return handle_create(data, token_info)
return handle_create(data, token_info, files=request.files)
elif action == "update":
# V1: Update not supported
return error_response(
"invalid_request", "Update action not supported in V1", 400
)
elif action == "delete":
# V1: Delete not supported
return error_response(
"invalid_request", "Delete action not supported in V1", 400
)
if "application/json" not in content_type:
return error_response("invalid_request", "Updates must be sent as JSON", 400)
return handle_update(data, token_info)
elif action in ("delete", "undelete"):
return handle_delete(data, token_info, undelete=(action == "undelete"))
else:
return error_response("invalid_request", f"Unknown action: {action}", 400)
@@ -122,7 +126,8 @@ def micropub_endpoint():
return error_response("server_error", "An unexpected error occurred", 500)
@bp.route('/media', methods=['POST'])
@bp.route('/micropub/media', methods=['POST'])
@bp.route('/media', methods=['POST']) # old path, kept for existing clients
def media_endpoint():
"""
Micropub media endpoint for file uploads
@@ -137,7 +142,10 @@ def media_endpoint():
from starpunk.media import save_media
# Extract and verify token
token = extract_bearer_token(request)
try:
token = extract_bearer_token(request)
except MicropubError as e:
return error_response(e.error, e.error_description, e.status_code)
if not token:
return error_response("unauthorized", "No access token provided", 401)
@@ -145,11 +153,12 @@ def media_endpoint():
if not token_info:
return error_response("unauthorized", "Invalid or expired access token", 401)
# Check scope (create scope allows media upload)
if not check_scope("create", token_info.get("scope", "")):
# Check scope: "media", or "create" for older clients
scope = token_info.get("scope", "")
if not (check_scope("media", scope) or check_scope("create", scope)):
return error_response(
"insufficient_scope",
"Token lacks create scope",
"Token lacks media or create scope",
403
)
+87 -13
View File
@@ -11,6 +11,7 @@ from datetime import datetime, timedelta
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
from starpunk.notes import list_notes, get_note
from starpunk.websub import embed_hub
from starpunk.feed import generate_feed_streaming # Legacy RSS
from starpunk.feeds import (
generate_rss,
@@ -55,8 +56,14 @@ def _get_cached_notes():
cache_duration = timedelta(seconds=cache_seconds)
now = datetime.utcnow()
# The cached list is stale once any public note changed, in any process
# (v2.10.0): every change bumps content_version in the database
from starpunk.database import get_db
from starpunk.websub import get_content_version
version = (str(current_app.config["DATABASE_PATH"]), get_content_version(get_db(current_app)))
# Check if note list cache is valid
if _feed_cache["notes"] and _feed_cache["timestamp"]:
if _feed_cache["notes"] and _feed_cache["timestamp"] and _feed_cache.get("version") == version:
cache_age = now - _feed_cache["timestamp"]
if cache_age < cache_duration:
# Use cached note list
@@ -64,7 +71,7 @@ def _get_cached_notes():
# Cache expired or empty, fetch fresh notes
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
notes = list_notes(published_only=True, limit=max_items)
notes = list_notes(published_only=True, limit=max_items, likes=False)
# Batch load media and tags for all notes (v1.5.0 Phase 3)
# Reduces query count from O(n) to O(1) for both media and tags
@@ -82,6 +89,7 @@ def _get_cached_notes():
_feed_cache["notes"] = notes
_feed_cache["timestamp"] = now
_feed_cache["version"] = version
return notes
@@ -122,6 +130,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
notes=notes,
limit=max_items,
)
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
# Create response with proper Content-Type including charset
mime_type = get_mime_type(format_name)
@@ -132,7 +141,12 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
# Caching enabled - use FeedCache
feed_cache = get_cache()
notes_checksum = feed_cache.generate_notes_checksum(notes)
# The content version covers changes the note list doesn't show, such as
# photos attached or captions edited (v2.10.1)
from starpunk.database import get_db
from starpunk.websub import get_content_version
notes_checksum = (feed_cache.generate_notes_checksum(notes)
+ ":" + get_content_version(get_db(current_app)))
# Check If-None-Match header for conditional requests
if_none_match = request.headers.get('If-None-Match')
@@ -170,6 +184,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
notes=notes,
limit=max_items,
)
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
# Store in cache and get ETag
etag = feed_cache.set(format_name, content, notes_checksum)
@@ -249,8 +264,8 @@ def index():
from starpunk.media import get_note_media
from starpunk.tags import get_note_tags
# Get recent published notes (limit 20)
notes = list_notes(published_only=True, limit=20)
# Get recent published notes (limit 20); likes have their own page (v2.5.0)
notes = list_notes(published_only=True, limit=20, likes=False)
# Attach media and tags to each note for display
for note in notes:
@@ -262,9 +277,59 @@ def index():
tags = get_note_tags(note.id)
object.__setattr__(note, '_cached_tags', tags)
attach_contexts(notes)
return render_template("index.html", notes=notes)
def attach_contexts(notes) -> None:
"""Give likes and reposts the fetched context of the post they point at"""
from starpunk.webmention_queries import get_reply_context
for note in notes:
context = None
if note.reacts_to:
try:
context = get_reply_context(note.reacts_to)
except Exception:
context = None
object.__setattr__(note, 'context', context)
@bp.route("/likes")
def likes():
"""Likes, kept off the home page and feeds (v2.5.0)"""
from starpunk.media import get_note_media
from starpunk.tags import get_note_tags
notes = list_notes(published_only=True, limit=50, likes=True)
for note in notes:
object.__setattr__(note, 'media', get_note_media(note.id))
object.__setattr__(note, '_cached_tags', get_note_tags(note.id))
attach_contexts(notes)
return render_template("likes.html", notes=notes)
def _note_was_deleted(slug: str) -> bool:
"""
True if this slug was a note that has been deleted
Soft-deleted notes keep their row; hard-deleted notes that sent
webmentions keep their queue rows (detached from the note).
"""
from starpunk.database import get_db
from starpunk.utils import absolute_url_for
db = get_db(current_app)
if db.execute(
"SELECT 1 FROM notes WHERE slug = ? AND deleted_at IS NOT NULL", (slug,)
).fetchone():
return True
return db.execute(
"SELECT 1 FROM webmention_queue WHERE source_url = ? AND source_note_id IS NULL",
(absolute_url_for("public.note", slug=slug),),
).fetchone() is not None
@bp.route("/note/<slug>")
def note(slug: str):
"""
@@ -290,8 +355,10 @@ def note(slug: str):
# Get note by slug
note_obj = get_note(slug=slug)
# Return 404 if note doesn't exist or isn't published
# Deleted notes are 410 Gone (Webmention 3.1.5); unknown or draft 404
if not note_obj or not note_obj.published:
if not note_obj and _note_was_deleted(slug):
abort(410)
abort(404)
# Get media for note (v1.2.0 Phase 3)
@@ -331,9 +398,15 @@ def note(slug: str):
logger = logging.getLogger(__name__)
logger.error(f"Error loading webmentions for note {slug}: {e}")
from starpunk.syndication import get_note_syndications
syndications = [s for s in get_note_syndications(note_obj.id) if s["state"] == "synced"]
attach_contexts([note_obj])
return render_template(
"note.html",
note=note_obj,
syndications=syndications,
webmentions=webmentions,
reply_context=reply_context,
**grouped if webmentions else {}
@@ -379,6 +452,7 @@ def tag(tag: str):
for note in notes:
media = get_note_media(note.id)
object.__setattr__(note, 'media', media)
attach_contexts(notes)
return render_template(
"tag.html",
@@ -439,15 +513,15 @@ def feed():
)
# Route to appropriate generator
if format_name == 'rss':
return feed_rss()
elif format_name == 'atom':
return feed_atom()
if format_name == 'atom':
response = feed_atom()
elif format_name == 'json':
return feed_json()
response = feed_json()
else:
# Shouldn't reach here, but be defensive
return feed_rss()
response = feed_rss()
# The format (and its rel=self) depends on Accept (v2.10.1)
response.headers.add('Vary', 'Accept')
return response
@bp.route("/feed.rss")
+2 -2
View File
@@ -108,7 +108,7 @@ def api_search():
"title": r["title"] or f"Note from {r['created_at'][:10]}",
"excerpt": r["snippet"], # Already has <mark> tags
"published_at": r["created_at"],
"url": f"/notes/{r['slug']}",
"url": f"/note/{r['slug']}",
}
for r in results
],
@@ -176,7 +176,7 @@ def search_page():
"title": r["title"] or f"Note from {r['created_at'][:10]}",
"excerpt": Markup(safe_snippet), # Mark as safe since we've escaped it ourselves
"published_at": r["created_at"],
"url": f"/notes/{r['slug']}",
"url": f"/note/{r['slug']}",
})
results = formatted_results
except Exception as e:
+21 -13
View File
@@ -11,7 +11,7 @@ References:
"""
from flask import Blueprint, current_app, request, jsonify
from urllib.parse import urlparse
from urllib.parse import urldefrag, urlparse
from starpunk.database import get_db
from starpunk.notes import get_note, NoteNotFoundError
from starpunk.rate_limiter import check_rate_limit
@@ -107,17 +107,24 @@ def validate_url(url: str) -> tuple[bool, str | None]:
return False, "URL is malformed"
def _host_port(url: str) -> tuple[str, int | None]:
parsed = urlparse(url)
default = {'http': 80, 'https': 443}.get(parsed.scheme.lower())
try:
port = parsed.port or default
except ValueError:
port = None
return (parsed.hostname or '').lower(), port
def is_target_on_site(target_url: str) -> bool:
"""
Check if target URL is on this site.
Compares hostname of target_url with SITE_URL configuration.
Compares host (case-insensitively) and effective port with SITE_URL.
"""
site_url = current_app.config.get('SITE_URL', '')
site_parsed = urlparse(site_url)
target_parsed = urlparse(target_url)
return site_parsed.netloc == target_parsed.netloc
return _host_port(site_url) == _host_port(target_url)
def resolve_target_note(target_url: str) -> int | None:
@@ -228,16 +235,17 @@ def receive_webmention():
"error_description": "Target URL is not on this site"
}), 400
# Resolve target to note (may be None for homepage/other pages)
target_note_id = resolve_target_note(target)
# The fragment is ignored (spec 3.2.1) and not stored, so target#a and
# target#b are the same mention
target = urldefrag(target)[0]
# If target looks like a note URL but doesn't resolve, it's not found
target_parsed = urlparse(target)
target_path = target_parsed.path.rstrip('/')
if target_path.startswith('/note/') and target_note_id is None:
# Only published notes accept webmentions (spec 3.2.1: target must be a
# valid resource that accepts them)
target_note_id = resolve_target_note(target)
if target_note_id is None:
return jsonify({
"error": "invalid_target",
"error_description": "Target URL not found"
"error_description": "Target URL does not accept webmentions"
}), 400
# Insert webmention into database (queued for validation)
+57
View File
@@ -0,0 +1,57 @@
"""
WebSub hub endpoint (v2.10.0, ADR-071)
POST /websub with hub.mode subscribe/unsubscribe, hub.topic, hub.callback,
optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
(spec 5.1); errors are 4xx with a plain-text reason.
"""
from flask import Blueprint, current_app, request
from starpunk.database import get_db
from starpunk.rate_limiter import RateLimiter
from starpunk import websub
bp = Blueprint("websub", __name__)
# Subscription requests per client IP per minute
# (per gunicorn process; needs TRUSTED_PROXY=true behind a reverse proxy,
# or every client shares the proxy's address)
_limiter = RateLimiter(limit=30, window_seconds=60)
_requests = 0
def _text(message: str, status: int):
return message + "\n", status, {"Content-Type": "text/plain; charset=utf-8"}
@bp.route("/websub", methods=["POST"])
def hub():
if not websub.enabled():
return _text("WebSub is not enabled on this site", 404)
from starpunk.routes.webmention import get_client_ip
global _requests
_requests += 1
if _requests % 500 == 0:
_limiter.cleanup() # forget idle clients so memory doesn't grow
if not _limiter.is_allowed(get_client_ip()):
return _text("Too many requests", 429)
# Media types are case-insensitive (v2.10.1)
if request.mimetype != "application/x-www-form-urlencoded":
return _text("Send an application/x-www-form-urlencoded body", 400)
try:
websub.accept_subscription_request(request.form, get_db(current_app))
except websub.SubscriptionError as e:
return _text(str(e), e.status)
return _text("Accepted; the subscription will be verified", 202)
@bp.route("/websub", methods=["GET"])
def hub_info():
return _text(
"WebSub hub for this site's own feeds. Subscribe with a POST "
"(https://www.w3.org/TR/websub/).", 200)
File diff suppressed because it is too large. Load diff
+11 -6
View File
@@ -115,27 +115,32 @@ def add_tags_to_note(note_id: int, tags: list[str]) -> None:
db.commit()
def get_note_tags(note_id: int) -> list[dict]:
def get_note_tags(note_id: int, entered_order: bool = False) -> list[dict]:
"""
Get all tags for a note
Args:
note_id: Note database ID
entered_order: In the order they were entered (the first is the
note's main tag, e.g. its Threads topic) instead of alphabetical.
add_tags_to_note() rewrites the links in entry order on every
save, so note_tags.id records it. Use this wherever the tags are
saved back (editor, Micropub update) so the order survives.
Returns:
List of tag dicts with 'name' and 'display_name'
Ordered alphabetically by display_name (case-insensitive)
Query should use: ORDER BY LOWER(tags.display_name) ASC
Ordered alphabetically by display_name (case-insensitive), unless
entered_order
"""
db = get_db(current_app)
order = "note_tags.id" if entered_order else "LOWER(tags.display_name)"
tags = db.execute(
"""
f"""
SELECT tags.name, tags.display_name
FROM tags
JOIN note_tags ON tags.id = note_tags.tag_id
WHERE note_tags.note_id = ?
ORDER BY LOWER(tags.display_name) ASC
ORDER BY {order} ASC
""",
(note_id,)
).fetchall()
+36 -1
View File
@@ -11,7 +11,7 @@ import hashlib
import re
import secrets
import shutil
from datetime import datetime
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
@@ -681,3 +681,38 @@ def parse_iso8601(date_string: str) -> datetime:
# Parse using fromisoformat
return datetime.fromisoformat(date_string)
def absolute_url_for(endpoint: str, **values) -> str:
"""
Absolute URL for an endpoint, built from SITE_URL
Use instead of url_for(..., _external=True): behind a reverse proxy the
request arrives over plain HTTP, so _external URLs would say http://.
"""
from flask import current_app, url_for
return current_app.config["SITE_URL"].rstrip("/") + url_for(endpoint, **values)
def iso_utc(value) -> str:
"""
Format a timestamp as RFC 3339 UTC for dt-* and Micropub (v2.8.0)
Stored times are naive UTC. Without an offset, microformats parsers read
them as local time, so this adds 'Z' and drops microseconds (HTML
<time datetime> allows at most 3 fractional digits).
Args:
value: datetime (naive = UTC) or ISO 8601 string
Returns:
e.g. '2026-11-01T16:25:12Z', or '' for empty values
"""
if not value:
return ''
if isinstance(value, str):
value = datetime.fromisoformat(value.strip().replace('Z', '+00:00'))
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return value.astimezone(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
+171 -200
View File
@@ -14,15 +14,19 @@ References:
- W3C Webmention: https://www.w3.org/TR/webmention/
"""
import json
import logging
import mf2py
import nh3
import httpx
from datetime import datetime
from typing import Dict, Any, Optional
from urllib.parse import urlparse
from urllib.parse import urldefrag, urlparse
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
from starpunk import mf2_utils
from starpunk.utils import iso_utc
from starpunk.webmention_ssrf import (
FetchTooLargeError, SSRFBlockedError, safe_get, user_agent,
)
logger = logging.getLogger(__name__)
@@ -33,205 +37,155 @@ class ValidationError(Exception):
pass
def fetch_source(url: str, site_url: str, timeout: int = 10, max_size: int = 1048576) -> tuple[str, str]:
ACCEPT = ('text/html, application/xhtml+xml;q=0.9, application/json;q=0.8, '
'text/plain;q=0.7, */*;q=0.1')
class TransientError(ValidationError):
"""Source could not be checked right now (network error, 5xx)"""
pass
def fetch_source(url: str, site_url: str) -> tuple[str, str, str]:
"""
Fetch source URL content with SSRF protection.
Per spec 3.2.2: GET following redirects (limited), with an Accept header.
Args:
url: Source URL to fetch
site_url: Site URL for User-Agent header
timeout: Request timeout in seconds (default: 10)
max_size: Maximum content size in bytes (default: 1MB)
Returns:
Tuple of (content, content_type)
Tuple of (content, content_type, final_url)
Raises:
SSRFBlockedError: If URL points to private IP
ValidationError: If fetch fails or content too large
ValidationError: Source is blocked, too large, or returned 4xx
TransientError: Network error, timeout or 5xx
"""
# SSRF protection
try:
validate_url_for_fetch(url)
result = safe_get(url, headers={'User-Agent': user_agent(site_url), 'Accept': ACCEPT})
except (SSRFBlockedError, ValueError) as e:
logger.warning(f"SSRF protection blocked URL {url}: {e}")
raise ValidationError(f"Blocked request: {e}")
# Fetch URL
try:
# Use streaming to check size before downloading entire content
with httpx.stream(
'GET',
url,
timeout=timeout,
headers={'User-Agent': f'StarPunk/1.6.0 (+{site_url})'},
follow_redirects=True
) as response:
response.raise_for_status()
# Check content length if provided
content_length = response.headers.get('Content-Length')
if content_length and int(content_length) > max_size:
raise ValidationError(f"Source exceeds size limit ({max_size} bytes)")
# Read content with size limit
content = b''
for chunk in response.iter_bytes(chunk_size=8192):
content += chunk
if len(content) > max_size:
raise ValidationError(f"Source exceeds size limit ({max_size} bytes)")
content_type = response.headers.get('Content-Type', '')
return content.decode('utf-8', errors='replace'), content_type
except FetchTooLargeError as e:
raise ValidationError(f"Source {e}")
except httpx.TimeoutException:
raise ValidationError("Source fetch timed out")
except httpx.HTTPStatusError as e:
if 400 <= e.response.status_code < 500:
raise ValidationError(f"Source returned {e.response.status_code}")
else:
raise ValidationError(f"Source server error: {e.response.status_code}")
raise TransientError("Source fetch timed out")
except httpx.HTTPError as e:
raise ValidationError(f"Failed to fetch source: {e}")
except UnicodeDecodeError:
raise ValidationError("Source content encoding is not supported")
raise TransientError(f"Failed to fetch source: {e}")
if 400 <= result.status_code < 500:
raise ValidationError(f"Source returned {result.status_code}")
if result.status_code >= 300:
raise TransientError(f"Source server error: {result.status_code}")
return result.text, result.headers.get('Content-Type', ''), result.url
def verify_link_presence(content: str, target_url: str, content_type: str) -> bool:
def _strip_fragment(url: str) -> str:
return urldefrag(url.strip())[0]
def _json_strings(value):
"""Yield every string in a parsed JSON document"""
if isinstance(value, str):
yield value
elif isinstance(value, dict):
for item in value.values():
yield from _json_strings(item)
elif isinstance(value, list):
for item in value:
yield from _json_strings(item)
def verify_link_presence(content: str, target_url: str, content_type: str,
source_url: str = '') -> bool:
"""
Verify that source content contains a link to target URL.
Verify the source links to the target (spec 3.2.2).
The source MUST have an exact match of the target URL, checked per media
type: in HTML, any link-bearing attribute (a/link href, img/video/audio
src, ...) resolved against the source URL; in JSON, any string value; in
other text, the URL anywhere. Fragments are ignored on both sides.
Args:
content: Source content (HTML or text)
content: Source content
target_url: Target URL to find
content_type: Content type of source
source_url: Final source URL, for resolving relative links
Returns:
True if target link found, False otherwise
True if the target is linked, False otherwise
"""
# Normalize URLs for comparison (remove fragments, trailing slashes)
target_normalized = urlparse(target_url)._replace(fragment='').geturl().rstrip('/')
target = _strip_fragment(target_url)
media_type = content_type.split(';')[0].strip().lower()
# For HTML, parse and check all links
if 'html' in content_type.lower():
if 'html' in media_type:
from lxml import html as lxml_html
try:
from lxml import html as lxml_html
doc = lxml_html.fromstring(content)
links = doc.xpath('//a/@href')
for link in links:
# Make absolute if relative
link_normalized = urlparse(link)._replace(fragment='').geturl().rstrip('/')
if link_normalized == target_normalized:
return True
doc = lxml_html.document_fromstring(content)
except Exception as e:
logger.debug(f"HTML parsing failed, falling back to string search: {e}")
logger.debug(f"HTML parsing failed: {e}")
return False
if source_url:
doc.make_links_absolute(source_url, resolve_base_href=True,
handle_failures='ignore')
return any(_strip_fragment(link) == target for _, _, link, _ in doc.iterlinks())
# Fallback: simple string search
return target_normalized in content
if media_type == 'application/json' or media_type.endswith('+json'):
try:
data = json.loads(content)
except ValueError:
return False
return any(_strip_fragment(s) == target for s in _json_strings(data))
if media_type.startswith('text/'):
return target in content
return False
def detect_mention_type(parsed_mf2: Dict, target_url: str) -> str:
"""
Detect mention type from microformats.
Detect mention type from the h-entry that responds to the target
Priority: reply > like > repost > bookmark > mention
Returns:
'reply', 'like', 'repost', 'bookmark', or 'mention'
"""
entry = mf2_utils.responding_entry(parsed_mf2, target_url)
return mf2_utils.response_type(entry, target_url) if entry else 'mention'
def _is_http_url(value: Optional[str]) -> bool:
"""True if value is an absolute http or https URL"""
if not value:
return False
parsed = urlparse(value.strip())
return parsed.scheme.lower() in ('http', 'https') and bool(parsed.netloc)
def extract_author(parsed_mf2: Dict, source_url: str,
entry: Optional[Dict] = None) -> Dict[str, Optional[str]]:
"""
Author of the source's h-entry (authorship algorithm, without fetching)
Args:
parsed_mf2: Parsed microformats data from mf2py
target_url: Target URL being mentioned
source_url: Source URL (fallback for author URL and name)
entry: The h-entry to attribute (default: first top-level h-entry)
Returns:
Mention type: 'reply', 'like', 'repost', 'bookmark', or 'mention'
Dict with author_name, author_url, author_photo; only http(s) URLs
are kept, so they are safe to render as links (XSS)
"""
target_normalized = urlparse(target_url)._replace(fragment='').geturl().rstrip('/')
if entry is None:
entry = mf2_utils.responding_entry(parsed_mf2, '')
author = mf2_utils.entry_author(parsed_mf2, entry, source_url)
# Find h-entry that references target
for item in parsed_mf2.get('items', []):
if 'h-entry' not in item.get('type', []):
continue
props = item.get('properties', {})
# Check in-reply-to (highest priority)
for reply_to in props.get('in-reply-to', []):
url = reply_to if isinstance(reply_to, str) else reply_to.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'reply'
# Check like-of
for like in props.get('like-of', []):
url = like if isinstance(like, str) else like.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'like'
# Check repost-of
for repost in props.get('repost-of', []):
url = repost if isinstance(repost, str) else repost.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'repost'
# Check bookmark-of
for bookmark in props.get('bookmark-of', []):
url = bookmark if isinstance(bookmark, str) else bookmark.get('value', '')
if urlparse(url)._replace(fragment='').geturl().rstrip('/') == target_normalized:
return 'bookmark'
# Default fallback
return 'mention'
def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]]:
"""
Extract author information from h-card.
Args:
parsed_mf2: Parsed microformats data from mf2py
source_url: Source URL (fallback for author URL)
Returns:
Dict with author_name, author_url, author_photo
"""
author_name = None
author_url = None
author_photo = None
# Find h-entry first
entry = None
for item in parsed_mf2.get('items', []):
if 'h-entry' in item.get('type', []):
entry = item
break
# Check for author property in h-entry
if entry:
author_prop = entry.get('properties', {}).get('author', [])
if author_prop:
author_data = author_prop[0]
if isinstance(author_data, dict):
# h-card object
props = author_data.get('properties', {})
author_name = props.get('name', [None])[0]
author_url = props.get('url', [None])[0]
author_photo = props.get('photo', [None])[0]
# If no author in h-entry, find representative h-card
if not author_name:
for item in parsed_mf2.get('items', []):
if 'h-card' in item.get('type', []):
props = item.get('properties', {})
author_name = props.get('name', [None])[0]
author_url = props.get('url', [None])[0]
author_photo = props.get('photo', [None])[0]
break
# Fallback to hostname if no author found
if not author_name:
author_name = urlparse(source_url).netloc
if not author_url:
author_url = source_url
author_url = author['url'] if _is_http_url(author['url']) else source_url
author_photo = author['photo'] if _is_http_url(author['photo']) else None
author_name = author['name'] or urlparse(author_url).netloc
return {
'author_name': author_name,
@@ -240,53 +194,65 @@ def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]
}
def extract_content(parsed_mf2: Dict) -> Optional[str]:
def extract_content(parsed_mf2: Dict, entry: Optional[Dict] = None) -> Optional[str]:
"""
Extract content snippet from h-entry.
Content snippet of an h-entry (e-content, else p-summary)
Args:
parsed_mf2: Parsed microformats data from mf2py
entry: The h-entry (default: first top-level h-entry)
Returns:
Sanitized content snippet (max 500 chars) or None
"""
# Find h-entry
for item in parsed_mf2.get('items', []):
if 'h-entry' not in item.get('type', []):
continue
if entry is None:
entry = mf2_utils.responding_entry(parsed_mf2, '')
if not entry:
return None
props = item.get('properties', {})
props = entry.get('properties', {})
for content in props.get('content', []):
html = (content.get('html') or content.get('value')) if isinstance(content, dict) else content
if html:
return _snippet(html)
# Try e-content first (HTML)
content_prop = props.get('content', [])
if content_prop:
content_data = content_prop[0]
if isinstance(content_data, dict):
content = content_data.get('html', '') or content_data.get('value', '')
else:
content = content_data
summary = mf2_utils.first(entry, 'summary')
return _snippet(summary) if summary else None
if content:
# Sanitize HTML
sanitized = sanitize_content(content)
# Truncate to 500 chars
if len(sanitized) > 500:
sanitized = sanitized[:497] + '...'
return sanitized
# Try p-summary as fallback
summary = props.get('summary', [])
if summary:
content = summary[0]
if isinstance(content, dict):
content = content.get('value', '')
if content:
sanitized = sanitize_content(content)
if len(sanitized) > 500:
sanitized = sanitized[:497] + '...'
return sanitized
def extract_source_details(entry: Optional[Dict]) -> Dict[str, Optional[str]]:
"""
The source post's own URL, uid and published time
return None
Returns:
source_post_url, source_uid (http(s) only) and source_published
(RFC 3339 UTC, or None if missing or unparseable)
"""
if not entry:
return {'source_post_url': None, 'source_uid': None, 'source_published': None}
url = mf2_utils.first(entry, 'url')
uid = mf2_utils.first(entry, 'uid')
published = mf2_utils.first(entry, 'published')
try:
published = iso_utc(published) or None
except (ValueError, TypeError):
published = None
return {
'source_post_url': url if _is_http_url(url) else None,
'source_uid': uid,
'source_published': published,
}
def _snippet(html: str, limit: int = 500) -> str:
"""Sanitized HTML of at most ~limit characters, with tags closed"""
sanitized = sanitize_content(html)
if len(sanitized) > limit:
# Cutting can leave a tag or attribute open; sanitizing again closes it
sanitized = sanitize_content(sanitized[:limit - 3] + '...')
return sanitized
def sanitize_content(html: str) -> str:
@@ -337,17 +303,17 @@ def process_webmention(source_url: str, target_url: str, site_url: str) -> Dict[
ValidationError: If validation fails with specific error
"""
# Fetch source content
content, content_type = fetch_source(source_url, site_url)
content, content_type, final_url = fetch_source(source_url, site_url)
# Verify target link presence
if not verify_link_presence(content, target_url, content_type):
if not verify_link_presence(content, target_url, content_type, final_url):
raise ValidationError("Source does not link to target")
# Parse microformats (only for HTML)
parsed_mf2 = None
if 'html' in content_type.lower():
try:
parsed_mf2 = mf2py.parse(doc=content, url=source_url)
parsed_mf2 = mf2py.parse(doc=content, url=final_url)
except Exception as e:
logger.debug(f"Microformat parsing failed for {source_url}: {e}")
@@ -360,14 +326,19 @@ def process_webmention(source_url: str, target_url: str, site_url: str) -> Dict[
}
content_snippet = None
source_details = extract_source_details(None)
if parsed_mf2:
mention_type = detect_mention_type(parsed_mf2, target_url)
author_info = extract_author(parsed_mf2, source_url)
content_snippet = extract_content(parsed_mf2)
entry = mf2_utils.responding_entry(parsed_mf2, target_url)
mention_type = mf2_utils.response_type(entry, target_url) if entry else 'mention'
author_info = extract_author(parsed_mf2, final_url, entry)
content_snippet = extract_content(parsed_mf2, entry)
source_details = extract_source_details(entry)
return {
'valid': True,
'mention_type': mention_type,
**author_info,
'content_snippet': content_snippet
'content_snippet': content_snippet,
**source_details,
}
+46 -3
View File
@@ -17,7 +17,10 @@ Functions:
import logging
import math
from datetime import datetime
from pathlib import Path
from typing import List, Dict, Any, Optional
from flask import current_app
from starpunk.database import get_db
logger = logging.getLogger(__name__)
@@ -71,9 +74,13 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
for row in rows:
wm = dict(row)
# Parse received_at timestamp
if isinstance(wm.get('received_at'), str):
wm['received_at'] = datetime.fromisoformat(wm['received_at'].replace('Z', '+00:00'))
# Parse timestamps (source_published may be missing or malformed)
for key in ('received_at', 'source_published'):
if isinstance(wm.get(key), str):
try:
wm[key] = datetime.fromisoformat(wm[key].replace('Z', '+00:00'))
except ValueError:
wm[key] = None
mention_type = wm.get('mention_type', 'mention')
if mention_type == 'reply':
@@ -233,10 +240,19 @@ def update_webmention_status(webmention_ids: List[int], status: str) -> int:
WHERE id IN ({placeholders})
"""
# Notes whose displayed mentions change (valid ones switching between
# approved and not)
changed_notes = _visibility_changes(
db, webmention_ids,
"validation_status = 'valid' AND (moderation_status = 'approved') != (? = 'approved')",
[status],
)
params = [status] + webmention_ids
cursor = db.execute(query, params)
db.commit()
_notify_responses_changed(changed_notes)
return cursor.rowcount
@@ -255,15 +271,42 @@ def delete_webmentions(webmention_ids: List[int]) -> int:
db = get_db()
changed_notes = _visibility_changes(
db, webmention_ids, "validation_status = 'valid' AND moderation_status = 'approved'", [],
)
placeholders = ','.join('?' * len(webmention_ids))
query = f"DELETE FROM webmentions WHERE id IN ({placeholders})"
cursor = db.execute(query, webmention_ids)
db.commit()
_notify_responses_changed(changed_notes)
return cursor.rowcount
def _visibility_changes(db, webmention_ids: List[int], condition: str, params: list) -> List[int]:
"""Target note IDs of the given webmentions that match condition"""
placeholders = ','.join('?' * len(webmention_ids))
rows = db.execute(
f"SELECT DISTINCT target_note_id FROM webmentions "
f"WHERE id IN ({placeholders}) AND {condition}",
list(webmention_ids) + params,
).fetchall()
return [row[0] for row in rows]
def _notify_responses_changed(note_ids: List[int]) -> None:
"""Re-send webmentions for notes whose shown responses changed (best effort)"""
if not note_ids:
return
try:
from starpunk.webmention_sender import note_responses_changed
note_responses_changed(Path(current_app.config['DATABASE_PATH']), note_ids)
except Exception as e:
current_app.logger.warning(f"Failed to re-send webmentions after moderation: {e}")
def get_reply_context(url: str) -> Optional[Dict[str, Any]]:
"""
Get cached reply context for a URL.
+111 -61
View File
@@ -14,9 +14,8 @@ References:
import logging
import sqlite3
from datetime import datetime
from pathlib import Path
from typing import Optional, Set
from typing import Optional
from lxml import html as lxml_html
from urllib.parse import urlparse
@@ -45,16 +44,16 @@ def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True)
return []
try:
# Parse HTML
doc = lxml_html.fromstring(html_content)
# Parse HTML inside a wrapper, so a top-level <pre> can be removed too
doc = lxml_html.fragment_fromstring(html_content, create_parent='div')
if exclude_code_blocks:
# Remove <pre> and <code> elements before extracting links
for element in doc.xpath('//pre | //code'):
element.getparent().remove(element)
for element in doc.xpath('.//pre | .//code'):
element.drop_tree()
# Extract all <a href> attributes in document order
links = doc.xpath('//a/@href')
links = doc.xpath('.//a/@href')
# Filter to valid absolute URLs (http/https)
absolute_links = []
@@ -70,6 +69,97 @@ def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True)
return []
def _dedupe(urls: list) -> list:
"""Drop empty and repeated URLs, keeping the first occurrence"""
seen = set()
result = []
for url in urls:
if url and url not in seen:
seen.add(url)
result.append(url)
return result
def previous_targets(db_path: Path, note_id: int) -> list[str]:
"""Targets this note has been queued to send webmentions to before"""
conn = sqlite3.connect(db_path)
try:
rows = conn.execute(
"SELECT target_url FROM webmention_queue WHERE source_note_id = ? ORDER BY id",
(note_id,),
).fetchall()
finally:
conn.close()
return [row[0] for row in rows]
def requeue_previous_webmentions(db_path: Path, note_id: int, detach: bool = False) -> int:
"""
Re-send every webmention this note has sent (spec 3.1.5 deleted posts)
Args:
db_path: Path to SQLite database
note_id: Note being deleted or restored
detach: Clear source_note_id so the rows survive a hard delete
(the foreign key cascades)
Returns:
Number of webmentions queued
"""
conn = sqlite3.connect(db_path)
try:
cursor = conn.execute(f"""
UPDATE webmention_queue
SET status = 'pending',
retry_count = 0,
next_retry_at = NULL,
last_error = NULL,
endpoint_url = NULL,
created_at = CURRENT_TIMESTAMP
{', source_note_id = NULL' if detach else ''}
WHERE source_note_id = ?
""", (note_id,))
conn.commit()
queued = cursor.rowcount
finally:
conn.close()
if queued:
logger.info(f"Re-queued {queued} webmentions for note {note_id}")
return queued
def note_responses_changed(db_path: Path, note_ids) -> int:
"""
Re-send a note's webmentions because the responses it shows changed
Spec 3.1.4: if the source post displays responses (e.g. comments), a
change to them is an update of the post. Only published, undeleted notes.
Args:
db_path: Path to SQLite database
note_ids: IDs of notes whose displayed responses changed
Returns:
Number of webmentions queued
"""
ids = sorted({i for i in note_ids if i is not None})
if not ids:
return 0
conn = sqlite3.connect(db_path)
try:
placeholders = ','.join('?' * len(ids))
live = [row[0] for row in conn.execute(
f"SELECT id FROM notes WHERE id IN ({placeholders}) "
"AND published = 1 AND deleted_at IS NULL", ids,
)]
finally:
conn.close()
return sum(requeue_previous_webmentions(db_path, note_id) for note_id in live)
def queue_webmentions_for_note(
db_path: Path,
note_id: int,
@@ -83,10 +173,9 @@ def queue_webmentions_for_note(
Per architect's answers (Phase 6):
- Called from notes.py AFTER successful database commit
- For updates: diff old vs new URLs, queue for new/existing/removed
- For creates: queue for all URLs in content
- Max 10 webmentions per operation
- Priority: in_reply_to URL first, then document order
- For creates: reply target first, then links in document order (max 10)
- For updates: the same, plus every previously sent target and links
removed by the edit (spec 3.1.4)
- Webmention sending is best-effort; failures do not affect note creation
Args:
@@ -100,58 +189,19 @@ def queue_webmentions_for_note(
Returns:
Number of webmentions queued
"""
# Phase 6: Implement URL diffing for updates
# Targets for the current content: reply target first, then links in
# document order, at most 10
links = extract_links_from_html(content_html) if content_html else []
current = _dedupe(([in_reply_to] if in_reply_to else []) + links)[:10]
if old_content_html is not None:
# This is an update - compute URL diff
old_links = set(extract_links_from_html(old_content_html))
# Handle unpublishing case (content_html is None)
if content_html is None:
# Unpublishing - queue for all old URLs (they're all removed)
new_links = set()
else:
new_links = set(extract_links_from_html(content_html))
# Compute sets for URL diffing
new_urls = new_links - old_links # In new, not in old
existing_urls = new_links & old_links # In both (re-validate)
removed_urls = old_links - new_links # In old, not in new
# Build target list: new + existing + removed
# (All need webmentions per W3C spec)
all_update_urls = list(new_urls) + list(existing_urls) + list(removed_urls)
# Add in_reply_to if changed
target_urls = []
if in_reply_to and in_reply_to not in all_update_urls:
target_urls.append(in_reply_to)
# Add the diffed URLs
target_urls.extend(all_update_urls)
# Limit to 10
target_urls = target_urls[:10]
# Update (spec 3.1.4): also re-send to every target notified before,
# and to links the edit removed, so they can update or remove the
# mention. These are not capped.
removed = [u for u in extract_links_from_html(old_content_html) if u not in links]
target_urls = _dedupe(current + previous_targets(db_path, note_id) + removed)
else:
# This is a create (no old content) - use original logic
links = extract_links_from_html(content_html) if content_html else []
# Build priority-ordered list of target URLs
target_urls = []
# 1. in_reply_to gets highest priority
if in_reply_to:
target_urls.append(in_reply_to)
# 2. Add extracted links in document order (deduplicated)
seen = {in_reply_to} if in_reply_to else set()
for link in links:
if link not in seen:
target_urls.append(link)
seen.add(link)
# 3. Limit to 10 webmentions per operation
target_urls = target_urls[:10]
target_urls = current
if not target_urls:
logger.debug(f"No links to send webmentions for note {note_id}")
+250 -52
View File
@@ -1,113 +1,311 @@
"""
SSRF protection utilities for webmention processing
SSRF protection for webmention fetches
Per ADR-063 Decision 9 and background-worker.md:
- Block private IP address ranges
- Resolve hostname before making requests
Per ADR-063 Decision 9 and Webmention sections 4.2, 4.3 and 4.5:
- Every address a hostname resolves to must be public
- Redirects are followed by hand so each hop is checked
- Each request connects to the address that was checked (no second DNS
lookup that a rebinding DNS server could answer differently)
- Redirect count, response size and total time are limited
- DEV_MODE exception for localhost testing
- Applied to all outgoing HTTP requests
References:
- ADR-063: Webmention Implementation
- Background Worker Design: /docs/design/v1.6.0/background-worker.md
- W3C Webmention: https://www.w3.org/TR/webmention/#security-considerations
"""
import ipaddress
import socket
import threading
import time
from dataclasses import dataclass
from typing import Optional
from urllib.parse import urljoin, urlparse
import httpx
from flask import current_app
MAX_REDIRECTS = 10
MAX_BYTES = 1024 * 1024
TIMEOUT = 10
# Tests can set an httpx transport here (e.g. httpx.MockTransport)
TRANSPORT = None
CGNAT = ipaddress.ip_network('100.64.0.0/10')
THIS_NETWORK = ipaddress.ip_network('0.0.0.0/8')
class SSRFBlockedError(Exception):
"""Raised when SSRF protection blocks a request"""
pass
# Private IP ranges per RFC 1918 and link-local ranges
BLOCKED_NETWORKS = [
ipaddress.ip_network('10.0.0.0/8'), # RFC 1918 private
ipaddress.ip_network('172.16.0.0/12'), # RFC 1918 private
ipaddress.ip_network('192.168.0.0/16'), # RFC 1918 private
ipaddress.ip_network('127.0.0.0/8'), # Localhost
ipaddress.ip_network('169.254.0.0/16'), # Link-local
ipaddress.ip_network('::1/128'), # IPv6 localhost
ipaddress.ip_network('fc00::/7'), # IPv6 unique local
ipaddress.ip_network('fe80::/10'), # IPv6 link-local
]
class ResolutionError(ValueError):
"""The host name could not be resolved (may be temporary; v2.10.1)"""
pass
class FetchTooLargeError(Exception):
"""Raised when a response is larger than the limit"""
pass
def user_agent(site_url: str) -> str:
"""User-Agent for webmention requests (spec 3.1.2 suggests including 'Webmention')"""
from starpunk import __version__
return f"StarPunk/{__version__} Webmention (+{site_url})"
def is_private_ip(ip: str) -> bool:
"""
Check if IP address is in a private/blocked range.
Check if an IP address must not be fetched.
Blocks private, loopback, link-local, multicast, reserved, unspecified,
carrier-grade NAT and IPv4-mapped forms of these.
Args:
ip: IP address string (IPv4 or IPv6)
Returns:
True if IP is private/blocked, False otherwise
True if IP is blocked, False otherwise
"""
try:
addr = ipaddress.ip_address(ip)
return any(addr in network for network in BLOCKED_NETWORKS)
addr = ipaddress.ip_address(ip.split('%')[0])
except ValueError:
# Invalid IP format, treat as blocked
return True
if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped:
addr = addr.ipv4_mapped
if isinstance(addr, ipaddress.IPv4Address) and (addr in CGNAT or addr in THIS_NETWORK):
return True
return (
addr.is_private
or addr.is_loopback
or addr.is_link_local
or addr.is_multicast
or addr.is_reserved
or addr.is_unspecified
)
def safe_resolve(hostname: str) -> str:
"""
Resolve hostname and validate IP is not private.
Resolve hostname and check every address is public.
Per background-worker.md: DEV_MODE exception for localhost testing.
Args:
hostname: Hostname to resolve
Returns:
Resolved IP address if safe
Address to connect to (IPv4 preferred)
Raises:
SSRFBlockedError: If hostname resolves to private IP
SSRFBlockedError: If any address is blocked
socket.gaierror: If hostname cannot be resolved
"""
# DEV_MODE exception for localhost
if current_app.config.get('DEV_MODE') and hostname in ('localhost', '127.0.0.1', '::1'):
return '127.0.0.1'
# Resolve hostname
ip = socket.gethostbyname(hostname)
addresses = sorted({info[4][0] for info in socket.getaddrinfo(hostname, None)},
key=lambda ip: (':' in ip, ip))
for ip in addresses:
if is_private_ip(ip):
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
# Check if resolved IP is private
if is_private_ip(ip):
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
return ip
return addresses[0]
def validate_url_for_fetch(url: str) -> None:
def validate_url_for_fetch(url: str) -> str:
"""
Validate URL is safe to fetch (not private IP).
Validate URL is safe to fetch (http(s), public addresses only).
Args:
url: URL to validate
Returns:
The checked IP address to connect to
Raises:
SSRFBlockedError: If URL points to private IP
ValueError: If URL is malformed
SSRFBlockedError: If URL points to a blocked address
ValueError: If URL is malformed or its host does not resolve
"""
from urllib.parse import urlparse
parsed = urlparse(url)
if not parsed.scheme in ('http', 'https'):
if parsed.scheme not in ('http', 'https'):
raise ValueError(f"Unsupported URL scheme: {parsed.scheme}")
if not parsed.netloc:
if not parsed.hostname:
raise ValueError("URL must include a hostname")
# Extract hostname (remove port if present)
hostname = parsed.hostname
if not hostname:
raise ValueError("Could not extract hostname from URL")
try:
return safe_resolve(parsed.hostname)
except socket.gaierror as e:
raise ResolutionError(f"Could not resolve {parsed.hostname}: {e}")
# Validate hostname resolves to public IP
safe_resolve(hostname)
def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
**kwargs) -> httpx.Request:
"""
Build a request that connects to the checked address for url's host.
The URL's host is replaced by the IP that passed the SSRF check; the
real host goes in the Host header and, for https, in SNI, so virtual
hosting and certificate verification still use the hostname.
"""
ip = validate_url_for_fetch(url)
parsed = urlparse(url)
ip_host = f"[{ip}]" if ':' in ip else ip
port = f":{parsed.port}" if parsed.port else ''
pinned_url = parsed._replace(netloc=f"{ip_host}{port}").geturl()
host_header = f"{parsed.hostname}{port}"
extensions = {'sni_hostname': parsed.hostname} if parsed.scheme == 'https' else {}
return client.build_request(
method, pinned_url, headers={**headers, 'Host': host_header},
extensions=extensions, **kwargs,
)
class _Watchdog:
"""
Close a client when its deadline passes (v2.10.1)
httpx timeouts apply per read, so a server sending one byte at a time
could hold a request for hours; closing the client ends it.
"""
def __init__(self, client: httpx.Client, seconds: float):
self.fired = False
def fire():
self.fired = True
try:
client.close()
except Exception:
pass
self._timer = threading.Timer(max(seconds, 0.01), fire)
self._timer.daemon = True
def __enter__(self):
self._timer.start()
return self
def __exit__(self, exc_type, exc, tb):
self._timer.cancel()
if self.fired and exc_type is not None and not isinstance(exc, httpx.TimeoutException):
raise httpx.TimeoutException("Exceeded time limit") from exc
return False
@dataclass
class FetchResult:
"""A fetched response, after redirects"""
url: str
status_code: int
headers: httpx.Headers
content: bytes
encoding: str
@property
def text(self) -> str:
try:
return self.content.decode(self.encoding, errors='replace')
except LookupError:
return self.content.decode('utf-8', errors='replace')
def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = None,
timeout: float = TIMEOUT, content: Optional[bytes] = None,
max_bytes: int = 4096) -> FetchResult:
"""
POST form data (or a raw body) to a checked, pinned address.
Redirects are not followed. `timeout` bounds the whole exchange, so a
server trickling its response can't hold the caller (v2.10.1). Only the
first max_bytes of the response are read (callers need the status and
maybe an error message).
Raises:
SSRFBlockedError, ValueError: URL is not allowed
httpx.HTTPError: Network errors
"""
kwargs = {'content': content} if content is not None else {'data': data}
deadline = time.monotonic() + timeout
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client, \
_Watchdog(client, timeout):
request = _pinned_request(client, 'POST', url, headers or {}, **kwargs)
response = client.send(request, stream=True)
try:
body = b''
for chunk in response.iter_bytes(chunk_size=1024):
body += chunk
if len(body) >= max_bytes:
body = body[:max_bytes]
break
if time.monotonic() > deadline:
raise httpx.TimeoutException("Response exceeded time limit")
return FetchResult(url=url, status_code=response.status_code, headers=response.headers,
content=body, encoding=response.charset_encoding or 'utf-8')
finally:
response.close()
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
"""
GET a URL, following redirects by hand and checking each hop.
Args:
url: URL to fetch
headers: Request headers
max_bytes: Response body limit
max_redirects: Redirect limit
timeout: Total time limit in seconds
Returns:
FetchResult for the final response (any status code)
Raises:
SSRFBlockedError, ValueError: URL or a redirect target is not allowed
FetchTooLargeError: Body exceeds max_bytes
httpx.HTTPError: Network errors, too many redirects, time limit
"""
deadline = time.monotonic() + timeout
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client, \
_Watchdog(client, timeout):
for _ in range(max_redirects + 1):
remaining = deadline - time.monotonic()
if remaining <= 0:
raise httpx.TimeoutException("Fetch exceeded time limit")
request = _pinned_request(client, 'GET', url, headers, timeout=remaining)
response = client.send(request, stream=True)
try:
location = response.headers.get('Location')
if response.is_redirect and location:
url = urljoin(url, location)
continue
content_length = response.headers.get('Content-Length', '')
if content_length.isdigit() and int(content_length) > max_bytes:
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
content = b''
for chunk in response.iter_bytes(chunk_size=8192):
content += chunk
if len(content) > max_bytes:
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
if time.monotonic() > deadline:
raise httpx.TimeoutException("Fetch exceeded time limit")
return FetchResult(
url=url,
status_code=response.status_code,
headers=response.headers,
content=content,
encoding=response.charset_encoding or 'utf-8',
)
finally:
response.close()
raise httpx.TooManyRedirects(f"More than {max_redirects} redirects")
+248 -151
View File
@@ -12,30 +12,34 @@ References:
- ADR-063: Webmention Implementation
"""
import fcntl
import logging
import re
import threading
import time
import sqlite3
import httpx
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, Any, Optional, TypedDict
from urllib.parse import urlparse
from typing import Dict, Any, Optional
from urllib.parse import urljoin
from lxml import html as lxml_html
from starpunk.webmention_processor import process_webmention, ValidationError
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
from flask import has_app_context
from starpunk.webmention_processor import process_webmention, TransientError, ValidationError
from starpunk.webmention_sender import note_responses_changed
from starpunk.mf2_utils import normalize_url
from starpunk.webmention_ssrf import (
FetchTooLargeError, SSRFBlockedError, safe_get, safe_post, user_agent,
)
logger = logging.getLogger(__name__)
# Link header (RFC 8288): <uri> then ;-separated params; quoted values may
# contain , and ;
LINK_RE = re.compile(r'<([^>]*)>((?:\s*;\s*[^;,"=\s]+\s*(?:=\s*(?:"[^"]*"|[^;,]*))?)*)')
LINK_PARAM_RE = re.compile(r';\s*([^;,"=\s]+)\s*(?:=\s*(?:"([^"]*)"|([^;,]*)))?')
# Discovery cache entry type
class DiscoveryCacheEntry(TypedDict):
"""Cache entry for endpoint discovery results"""
endpoint: Optional[str] # None = negative cache (no endpoint found)
expires_at: datetime
class WebmentionWorker(threading.Thread):
@@ -47,6 +51,7 @@ class WebmentionWorker(threading.Thread):
def __init__(
self,
app,
db_path: Path,
site_url: str,
poll_interval: int = 10,
@@ -58,6 +63,7 @@ class WebmentionWorker(threading.Thread):
Initialize webmention worker thread
Args:
app: Flask app (the SSRF checks read its config)
db_path: Path to SQLite database
site_url: Site URL for User-Agent header
poll_interval: Seconds between queue checks (default: 10)
@@ -66,6 +72,7 @@ class WebmentionWorker(threading.Thread):
auto_approve: Auto-approve valid webmentions (default: True)
"""
super().__init__(daemon=True, name="webmention-worker")
self.app = app
self.db_path = db_path
self.site_url = site_url
self.poll_interval = poll_interval
@@ -74,10 +81,7 @@ class WebmentionWorker(threading.Thread):
self.auto_approve = auto_approve
self._stop_event = threading.Event()
self._last_processed = None
# Phase 3: Discovery cache (worker instance variable)
# Per architect's answer to Q4
self._discovery_cache: Dict[str, DiscoveryCacheEntry] = {}
self._lock_file = None
def run(self):
"""
@@ -92,19 +96,23 @@ class WebmentionWorker(threading.Thread):
try:
logger.info("Webmention worker started")
# Cleanup orphaned items from previous crashes
self._cleanup_orphaned_items()
# Main processing loop
while not self._stop_event.is_set():
# Every gunicorn process starts a worker; only the one holding
# the lock processes queues, so nothing is sent twice
if not self._acquire_lock():
self._stop_event.wait(self.poll_interval)
continue
try:
processed = 0
# Process incoming webmentions
processed += self._process_pending_validations()
with self.app.app_context():
# Process incoming webmentions
processed += self._process_pending_validations()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Update last processed time
if processed > 0:
@@ -121,8 +129,40 @@ class WebmentionWorker(threading.Thread):
except Exception as e:
logger.error(f"Webmention worker thread failed: {e}", exc_info=True)
finally:
self._release_lock()
logger.info("Webmention worker stopped")
def _acquire_lock(self) -> bool:
"""
Take the cross-process worker lock (non-blocking)
On first acquisition, also reset items left mid-send by a previous
holder. The OS releases the lock if this process dies.
Returns:
True if this worker holds the lock
"""
if self._lock_file:
return True
lock_file = open(Path(self.db_path).parent / "webmention-worker.lock", "w")
try:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
lock_file.close()
return False
self._lock_file = lock_file
logger.info("Webmention worker holds the queue lock")
self._cleanup_orphaned_items()
return True
def _release_lock(self):
"""Release the cross-process worker lock if held"""
if self._lock_file:
self._lock_file.close()
self._lock_file = None
def _cleanup_orphaned_items(self):
"""
Clean up orphaned 'sending' status items on startup
@@ -206,6 +246,69 @@ class WebmentionWorker(threading.Thread):
source_url: str,
target_url: str,
target_note_id: Optional[int]
):
"""
Validate a webmention, then re-send the target note's webmentions if
what the note shows changed (spec 3.1.4: displayed responses are an
update of the post). Unchanged re-validations send nothing, so two
sites doing this don't ping-pong.
"""
before = self._display_state(webmention_id)
try:
self._validate_and_store(webmention_id, source_url, target_url, target_note_id)
finally:
after = self._display_state(webmention_id)
if before != after and target_note_id is not None:
try:
note_responses_changed(self.db_path, [target_note_id])
except Exception as e:
logger.warning(f"Failed to re-send webmentions for note {target_note_id}: {e}")
def _is_own_syndicated_copy(self, note_id: Optional[int], source_url: str,
result: Dict[str, Any]) -> bool:
"""True if the source is one of the note's own u-syndication copies"""
if note_id is None:
return False
conn = sqlite3.connect(self.db_path)
try:
copies = {normalize_url(row[0]) for row in conn.execute(
"SELECT remote_url FROM syndications WHERE note_id = ? AND remote_url IS NOT NULL",
(note_id,),
)}
finally:
conn.close()
source = {normalize_url(u) for u in (
source_url, result.get('source_post_url'), result.get('source_uid')) if u}
return bool(copies & source)
def _display_state(self, webmention_id: int) -> Optional[tuple]:
"""
What the note page shows for this webmention, or None if hidden
A row reset to 'pending' by a re-send still counts as shown if it was
valid before (validated with no error), as it was until now.
"""
conn = sqlite3.connect(self.db_path)
try:
row = conn.execute("""
SELECT mention_type, author_name, author_url, author_photo, content_snippet
FROM webmentions
WHERE id = ?
AND moderation_status = 'approved'
AND (validation_status = 'valid'
OR (validation_status = 'pending' AND validated_at IS NOT NULL
AND validation_error IS NULL))
""", (webmention_id,)).fetchone()
finally:
conn.close()
return tuple(row) if row else None
def _validate_and_store(
self,
webmention_id: int,
source_url: str,
target_url: str,
target_note_id: Optional[int]
):
"""
Validate a single webmention
@@ -220,31 +323,49 @@ class WebmentionWorker(threading.Thread):
# Process webmention (fetch, validate, extract)
result = process_webmention(source_url, target_url, self.site_url)
# A copy of this very note on a silo (e.g. backfed by Bridgy) is
# not a response to it
if self._is_own_syndicated_copy(target_note_id, source_url, result):
raise ValidationError("Source is this note's own syndicated copy")
# Update database with successful validation
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
moderation_status = 'approved' if self.auto_approve else 'pending'
# An update to an existing mention (spec 3.2.4) keeps the admin's
# moderation decision; only a first-time valid mention gets the
# default. Previously valid = validated with no error.
cursor.execute("""
UPDATE webmentions
SET validation_status = 'valid',
SET moderation_status = CASE
WHEN moderation_status IN ('hidden', 'deleted') THEN moderation_status
WHEN validated_at IS NOT NULL AND validation_error IS NULL
THEN moderation_status
ELSE ?
END,
validation_status = 'valid',
validation_error = NULL,
validated_at = CURRENT_TIMESTAMP,
mention_type = ?,
author_name = ?,
author_url = ?,
author_photo = ?,
content_snippet = ?,
moderation_status = ?,
source_post_url = ?,
source_published = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (
moderation_status,
result['mention_type'],
result['author_name'],
result['author_url'],
result['author_photo'],
result['content_snippet'],
moderation_status,
result.get('source_post_url'),
result.get('source_published'),
webmention_id
))
@@ -253,11 +374,17 @@ class WebmentionWorker(threading.Thread):
logger.info(
f"Validated webmention: {source_url} -> {target_url} "
f"(type={result['mention_type']}, status={moderation_status})"
f"(type={result['mention_type']})"
)
except TransientError as e:
# Source unreachable right now: keep a previously valid mention
self._mark_validation_failed(webmention_id, source_url, target_url, str(e),
transient=True)
except ValidationError as e:
# Validation failed - update with error
# Validation failed (410/4xx, no link): the mention is removed
# from display (spec 3.2.4)
self._mark_validation_failed(webmention_id, source_url, target_url, str(e))
except Exception as e:
@@ -275,11 +402,15 @@ class WebmentionWorker(threading.Thread):
webmention_id: int,
source_url: str,
target_url: str,
error: str
error: str,
transient: bool = False
):
"""
Mark webmention validation as failed/invalid
A transient failure of a mention that was valid before restores it to
valid with its previous data, rather than hiding it.
Args:
webmention_id: Database ID
source_url: Source URL
@@ -289,7 +420,9 @@ class WebmentionWorker(threading.Thread):
# Determine if error or invalid
# Per background-worker.md: HTTP 4xx and "no link" are invalid
# All other errors are 'error' status
if any(x in error.lower() for x in ['does not link', 'returned 4']):
if transient:
status = 'error'
elif any(x in error.lower() for x in ['does not link', 'returned 4', 'own syndicated copy']):
status = 'invalid'
else:
status = 'error'
@@ -298,6 +431,22 @@ class WebmentionWorker(threading.Thread):
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
if transient:
cursor.execute("""
UPDATE webmentions
SET validation_status = 'valid',
updated_at = CURRENT_TIMESTAMP
WHERE id = ? AND validated_at IS NOT NULL AND validation_error IS NULL
""", (webmention_id,))
if cursor.rowcount:
conn.commit()
conn.close()
logger.warning(
f"Could not re-check webmention, keeping it: "
f"{source_url} -> {target_url}: {error}"
)
return
cursor.execute("""
UPDATE webmentions
SET validation_status = ?,
@@ -418,37 +567,34 @@ class WebmentionWorker(threading.Thread):
conn.commit()
conn.close()
# Send webmention POST
# Send webmention POST (spec 3.1.3)
try:
# SSRF protection for endpoint URL
validate_url_for_fetch(endpoint)
response = httpx.post(
# SSRF-checked and pinned to the checked address (spec 4.3).
# Redirects are not followed: a 301/302 would turn the POST
# into a GET without the body
response = safe_post(
endpoint,
data={
'source': source_url,
'target': target_url
},
timeout=10,
headers={'User-Agent': f'StarPunk/1.6.0 (+{self.site_url})'},
follow_redirects=True
headers={'User-Agent': user_agent(self.site_url)},
)
# Check response
# Any 2xx is success (spec 3.1.3)
if 200 <= response.status_code < 300:
# Success
self._mark_send_success(queue_id, source_url, target_url)
elif 400 <= response.status_code < 500:
# Permanent failure (4xx)
error = f"HTTP {response.status_code}: {response.text[:200]}"
self._mark_send_failed(queue_id, source_url, target_url, error)
else:
# Temporary failure (5xx) - retry
elif response.status_code == 429 or response.status_code >= 500:
# Rate limited or server error - retry
error = f"HTTP {response.status_code}: {response.text[:200]}"
self._mark_send_retry(queue_id, source_url, target_url, retry_count, error)
else:
# Permanent failure (3xx, 4xx)
error = f"HTTP {response.status_code}: {response.text[:200]}"
self._mark_send_failed(queue_id, source_url, target_url, error)
except (httpx.TimeoutException, httpx.HTTPError) as e:
# Connection error - retry
self._mark_send_retry(queue_id, source_url, target_url, retry_count, str(e))
@@ -464,100 +610,81 @@ class WebmentionWorker(threading.Thread):
def _discover_endpoint(self, target_url: str) -> Optional[str]:
"""
Discover webmention endpoint for target URL with caching
Discover the webmention endpoint for a target URL (spec 3.1.2)
Per architect's answers to Q3, Q4, Q10:
- GET only (no HEAD)
- Check Link header → <link rel="webmention"> → <a rel="webmention">
- First match wins
- Cache results: 1h for found, 24h for not found
- SSRF protection on target URL
Args:
target_url: Target URL to discover endpoint for
- GET the target, following redirects (each hop SSRF-checked)
- The first Link header with rel=webmention wins, then the first
<link> or <a> with rel=webmention and an href, in document order
- rel is a space-separated token list, matched case-insensitively
- Relative endpoints resolve against the final URL; an empty href
is the target itself; query strings are kept
- Not cached: an updated post must re-discover (spec 3.1.4)
Returns:
Endpoint URL if found, None otherwise
"""
# Check cache first
cached = self._discovery_cache.get(target_url)
if cached and cached['expires_at'] > datetime.utcnow():
logger.debug(f"Using cached endpoint for {target_url}: {cached['endpoint']}")
return cached['endpoint']
Endpoint URL, or None if the target has none (or is blocked/4xx)
# SSRF protection
Raises:
httpx.HTTPError: Network errors and 5xx, so the caller retries
"""
try:
validate_url_for_fetch(target_url)
result = safe_get(target_url, headers={'User-Agent': user_agent(self.site_url)})
except (SSRFBlockedError, ValueError) as e:
logger.warning(f"SSRF protection blocked target {target_url}: {e}")
# Cache negative result
self._cache_endpoint(target_url, None, hours=24)
return None
except FetchTooLargeError:
logger.info(f"Target too large for discovery: {target_url}")
return None
# Fetch target URL
try:
response = httpx.get(
target_url,
timeout=10,
headers={'User-Agent': f'StarPunk/1.6.0 (+{self.site_url})'},
follow_redirects=True
)
response.raise_for_status()
# 1. Check Link header
link_header = response.headers.get('Link', '')
if link_header:
endpoint = self._parse_link_header(link_header, response.url)
if endpoint:
self._cache_endpoint(target_url, endpoint, hours=1)
return endpoint
# 2. Parse HTML for <link> and <a> elements
content_type = response.headers.get('Content-Type', '')
if 'html' in content_type.lower():
endpoint = self._parse_html_for_endpoint(response.text, response.url)
if endpoint:
self._cache_endpoint(target_url, endpoint, hours=1)
return endpoint
# No endpoint found - cache negative result
logger.info(f"No webmention endpoint found for {target_url}")
self._cache_endpoint(target_url, None, hours=24)
if result.status_code >= 500:
raise httpx.HTTPError(f"Target returned {result.status_code}")
if result.status_code >= 400:
logger.info(f"Target returned {result.status_code}: {target_url}")
return None
except Exception as e:
logger.warning(f"Failed to discover endpoint for {target_url}: {e}")
# Don't cache on error - allow retry
return None
for link_header in result.headers.get_list('Link'):
endpoint = self._parse_link_header(link_header, result.url)
if endpoint is not None:
return endpoint
content_type = result.headers.get('Content-Type', '')
if 'html' in content_type.lower():
endpoint = self._parse_html_for_endpoint(result.text, result.url)
if endpoint is not None:
return endpoint
logger.info(f"No webmention endpoint found for {target_url}")
return None
@staticmethod
def _has_webmention_rel(rel: str) -> bool:
return 'webmention' in rel.lower().split()
def _parse_link_header(self, link_header: str, base_url: str) -> Optional[str]:
"""
Parse Link header for webmention endpoint
Parse a Link header (RFC 8288) for a rel=webmention link
Args:
link_header: Raw Link header value
link_header: Link header value (may hold several links)
base_url: Base URL for resolving relative URLs
Returns:
Absolute endpoint URL if found, None otherwise
Absolute endpoint URL of the first match, None otherwise
"""
# Simple parsing - look for rel="webmention"
# Format: <URL>; rel="webmention"
import re
pattern = r'<([^>]+)>\s*;\s*rel\s*=\s*["\']?webmention["\']?'
match = re.search(pattern, link_header, re.IGNORECASE)
if match:
endpoint = match.group(1)
# Make absolute if relative
from urllib.parse import urljoin
return urljoin(base_url, endpoint)
for match in LINK_RE.finditer(link_header):
uri, params = match.group(1), match.group(2)
for param in LINK_PARAM_RE.finditer(params):
if param.group(1).lower() != 'rel':
continue
rel = param.group(2) if param.group(2) is not None else (param.group(3) or '')
if self._has_webmention_rel(rel):
return urljoin(base_url, uri.strip())
break # only the first rel parameter counts (RFC 8288)
return None
def _parse_html_for_endpoint(self, html_content: str, base_url: str) -> Optional[str]:
"""
Parse HTML for webmention endpoint in <link> or <a> elements
Find the first <link> or <a> with rel=webmention, in document order
Args:
html_content: HTML content
@@ -567,48 +694,18 @@ class WebmentionWorker(threading.Thread):
Absolute endpoint URL if found, None otherwise
"""
try:
doc = lxml_html.fromstring(html_content)
# Check <link rel="webmention"> first
links = doc.xpath('//link[@rel="webmention"]/@href')
if links:
from urllib.parse import urljoin
return urljoin(base_url, links[0])
# Check <a rel="webmention">
anchors = doc.xpath('//a[@rel="webmention"]/@href')
if anchors:
from urllib.parse import urljoin
return urljoin(base_url, anchors[0])
doc = lxml_html.document_fromstring(html_content)
except Exception as e:
logger.debug(f"HTML parsing failed: {e}")
return None
for element in doc.iter('link', 'a'):
href = element.get('href')
if href is not None and self._has_webmention_rel(element.get('rel', '')):
return urljoin(base_url, href.strip())
return None
def _cache_endpoint(self, target_url: str, endpoint: Optional[str], hours: int):
"""
Cache endpoint discovery result
Args:
target_url: Target URL
endpoint: Discovered endpoint (None = negative cache)
hours: TTL in hours
"""
self._discovery_cache[target_url] = {
'endpoint': endpoint,
'expires_at': datetime.utcnow() + timedelta(hours=hours)
}
# Optional: Evict old entries (keep cache bounded)
if len(self._discovery_cache) > 1000:
# Remove expired entries
now = datetime.utcnow()
self._discovery_cache = {
k: v for k, v in self._discovery_cache.items()
if v['expires_at'] > now
}
def _mark_send_success(self, queue_id: int, source_url: str, target_url: str):
"""Mark webmention send as successful"""
try:
+443
View File
@@ -0,0 +1,443 @@
"""
WebSub publishing and the built-in hub (v2.10.0, ADR-071)
Publisher: topic responses advertise the hub and their own URL; a change to a
public note clears cached feeds and queues delivery of every affected topic.
Hub: subscriptions to this site's own topics only. Requests are accepted
here (202) and verified, delivered and retried by the WebSub worker.
References:
- W3C WebSub: https://www.w3.org/TR/websub/
- ADR-071: WebSub publishing with a built-in hub
"""
import json
import secrets
from typing import Iterable, Optional
from urllib.parse import urlparse
from flask import current_app, request
# Topic endpoints and their fixed paths (tag pages are added per tag)
FEED_TOPICS = {
"public.feed_atom": "/feed.atom",
"public.feed_rss": "/feed.rss",
"public.feed_xml_legacy": "/feed.rss", # legacy alias of the RSS topic
"public.feed_json": "/feed.json",
}
PAGE_TOPICS = {"public.index": "/", "public.likes": "/likes"}
NEGOTIATED = { # /feed: the topic of whichever format was served
"application/atom+xml": "/feed.atom",
"application/rss+xml": "/feed.rss",
"application/feed+json": "/feed.json",
"application/json": "/feed.json",
}
LEASE_DEFAULT = 10 * 24 * 3600
LEASE_MIN = 24 * 3600
LEASE_MAX = 30 * 24 * 3600
MAX_SECRET_BYTES = 200
class SubscriptionError(Exception):
"""A subscription request the hub refuses (4xx with a plain-text reason)"""
def __init__(self, message: str, status: int = 400):
super().__init__(message)
self.status = status
# --- Configuration -----------------------------------------------------------
def enabled() -> bool:
return bool(current_app.config.get("WEBSUB_ENABLED", True))
def external_hub() -> Optional[str]:
"""URL of an external hub, if one is configured instead of the built-in one"""
return current_app.config.get("WEBSUB_HUB_URL") or None
def hub_url() -> str:
# Built from SITE_URL so the worker can use it outside a request
return external_hub() or site_url() + "/websub"
def site_url() -> str:
return current_app.config["SITE_URL"].rstrip("/")
# --- Topics ------------------------------------------------------------------
def topic_url(path: str) -> str:
return site_url() + path
def tag_topic(tag_name: str) -> str:
from urllib.parse import quote
return f"{site_url()}/tag/{quote(tag_name, safe='')}"
def topic_for_request(response=None) -> Optional[str]:
"""The topic URL the current request serves, or None if it isn't a topic"""
endpoint = request.endpoint
if endpoint in PAGE_TOPICS:
return topic_url(PAGE_TOPICS[endpoint])
if endpoint in FEED_TOPICS:
return topic_url(FEED_TOPICS[endpoint])
if endpoint == "public.feed" and response is not None:
mime = (response.mimetype or "").lower()
return topic_url(NEGOTIATED[mime]) if mime in NEGOTIATED else None
if endpoint == "public.tag":
from starpunk.tags import normalize_tag
tag = (request.view_args or {}).get("tag", "")
return tag_topic(normalize_tag(tag)[0]) if tag else None
return None
_UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
def decode_unreserved(value: str) -> str:
"""
Decode percent-encoded unreserved characters (%66 -> f), leaving the rest
Spec 5.1.1: hubs MUST always decode non-reserved characters in URL
parameters, so /%66eed.atom and /feed.atom are the same topic.
"""
out, i = [], 0
while i < len(value):
if value[i] == "%" and i + 3 <= len(value):
try:
char = chr(int(value[i + 1:i + 3], 16))
except ValueError:
char = None
if char is not None and char in _UNRESERVED:
out.append(char)
i += 3
continue
out.append(value[i])
i += 1
return "".join(out)
def canonical_topic(url: str) -> Optional[str]:
"""
A subscription's hub.topic as one of this site's topics, or None
Unreserved percent-encodings are decoded and the scheme and host are
compared case-insensitively. Tag topics must name an existing tag.
"""
if not url:
return None
site = urlparse(site_url())
parsed = urlparse(decode_unreserved(url.strip()))
if (parsed.scheme.lower(), parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
return None
path = parsed.path or "/"
if parsed.query or parsed.fragment:
return None
if path in set(PAGE_TOPICS.values()) | set(FEED_TOPICS.values()):
return topic_url(path)
if path.startswith("/tag/"):
from urllib.parse import unquote
from starpunk.tags import get_tag_by_name, normalize_tag
name = normalize_tag(unquote(path[len("/tag/"):]))[0]
if name and get_tag_by_name(name):
return tag_topic(name)
return None
def link_header_values(topic: str) -> list[str]:
return [f'<{hub_url()}>; rel="hub"', f'<{topic}>; rel="self"']
def add_discovery_headers(response):
"""after_request: Link rel=hub and rel=self on topic responses (spec 4)"""
if not enabled() or response.status_code not in (200, 304):
return response
topic = topic_for_request(response)
if topic:
for value in link_header_values(topic):
response.headers.add("Link", value)
return response
def template_links() -> Optional[dict]:
"""For base.html: {'hub', 'self'} on topic pages"""
if not enabled() or not request:
return None
topic = topic_for_request()
return {"hub": hub_url(), "self": topic} if topic else None
def embed_hub(format_name: str, content: str) -> str:
"""Add the hub link inside a generated feed, next to its self link"""
if not enabled():
return content
hub = hub_url()
if format_name == "atom":
marker = '<link rel="self"'
return content.replace(marker, f'<link rel="hub" href="{_xml(hub)}"/>\n {marker}', 1)
if format_name == "rss":
marker = "<atom:link "
return content.replace(marker, f'<atom:link href="{_xml(hub)}" rel="hub"/>\n {marker}', 1)
if format_name == "json":
feed = json.loads(content)
feed["hubs"] = [{"type": "WebSub", "url": hub}]
return json.dumps(feed, ensure_ascii=False, indent=2)
return content
def _xml(value: str) -> str:
return (value.replace("&", "&amp;").replace("<", "&lt;")
.replace(">", "&gt;").replace('"', "&quot;"))
# --- Changes -------------------------------------------------------------------
def get_content_version(db) -> str:
row = db.execute("SELECT value FROM site_state WHERE key = 'content_version'").fetchone()
return row[0] if row else "0"
def bump_content_version(db) -> None:
db.execute(
"INSERT INTO site_state (key, value) VALUES ('content_version', '1') "
"ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1"
)
def note_changed(note, was_public: bool = False, old_tags: Iterable[str] = ()) -> None:
"""
A note was created, edited, deleted, restored or (un)published
Clears cached feeds in every process and queues delivery of the topics
that show it. Changes that are never public (a draft edited) do nothing.
Args:
note: The note (after the change; for a delete, before it)
was_public: Whether it was public before the change
old_tags: Its tag names before the change
"""
from starpunk.database import get_db
is_public = bool(note.published) and not getattr(note, "deleted_at", None)
if not (is_public or was_public):
return
db = get_db(current_app)
bump_content_version(db)
db.commit()
if not enabled():
return
from starpunk.tags import get_note_tags
try:
tags = {t["name"] for t in get_note_tags(note.id)}
except Exception:
tags = set()
tags |= set(old_tags)
# Likes appear on /likes, not on the home page or in the feeds
if getattr(note, "like_of", None):
topics = [topic_url("/likes")]
else:
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
topics += [tag_topic(name) for name in sorted(tags)]
queue_topics(db, topics)
def queue_topics(db, topics: Iterable[str]) -> int:
"""Queue delivery of topics: to their subscribers, or a ping to an external hub"""
topics = list(dict.fromkeys(topics))
if not topics:
return 0
if external_hub():
for topic in topics:
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NULL AND topic = ?", (topic,))
db.execute("INSERT INTO websub_deliveries (topic) VALUES (?)", (topic,))
db.commit()
return len(topics)
placeholders = ",".join("?" * len(topics))
rows = db.execute(
f"SELECT id, topic FROM websub_subscriptions WHERE topic IN ({placeholders}) "
"AND verified_at IS NOT NULL AND expires_at > datetime('now')",
topics,
).fetchall()
for sub_id, topic in rows:
# The newest content replaces anything still waiting (ADR-071 6)
db.execute("DELETE FROM websub_deliveries WHERE subscription_id = ?", (sub_id,))
db.execute("INSERT INTO websub_deliveries (subscription_id, topic) VALUES (?, ?)", (sub_id, topic))
db.commit()
return len(rows)
# --- Hub: subscription requests (spec 5.1) -------------------------------------
MAX_CALLBACK_LENGTH = 2048
MAX_PENDING_INTENTS = 200
ALLOWED_PORTS = {80, 443} # plus any port from 1024 up
def _clean_callback(callback: str) -> str:
"""
Check a hub.callback and return it with an ASCII (IDNA) host
Raises:
SubscriptionError: if it is not an acceptable callback URL
"""
import httpx
if not callback or len(callback) > MAX_CALLBACK_LENGTH:
raise SubscriptionError(f"hub.callback must be a URL of at most {MAX_CALLBACK_LENGTH} characters")
if any(ord(c) < 0x20 or ord(c) == 0x7f for c in callback) or " " in callback:
raise SubscriptionError("hub.callback contains spaces or control characters")
parsed = urlparse(callback)
if parsed.scheme.lower() not in ("http", "https") or not parsed.hostname:
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
try:
port = parsed.port
except ValueError:
raise SubscriptionError("hub.callback has an invalid port")
if port is not None and port not in ALLOWED_PORTS and port < 1024:
raise SubscriptionError("hub.callback may not use that port")
try:
host = parsed.hostname.encode("idna").decode("ascii")
except UnicodeError:
raise SubscriptionError("hub.callback has an invalid host name")
netloc = host + (f":{port}" if port is not None else "")
callback = parsed._replace(scheme=parsed.scheme.lower(), netloc=netloc).geturl()
if host.lower() == (urlparse(site_url()).hostname or "").lower():
raise SubscriptionError("hub.callback may not be on this site")
try:
httpx.URL(callback)
except Exception:
raise SubscriptionError("hub.callback is not a valid URL")
return callback
def accept_subscription_request(form, db) -> None:
"""
Validate a subscription request and queue it for verification
Raises:
SubscriptionError: with the status and plain-text reason to return
"""
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
mode = (form.get("hub.mode") or "").strip()
if mode not in ("subscribe", "unsubscribe"):
raise SubscriptionError("hub.mode must be subscribe or unsubscribe")
if mode == "subscribe" and external_hub():
# Unsubscribes are still taken, so old subscribers can leave
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
callback = _clean_callback((form.get("hub.callback") or "").strip())
try:
callback_ip = validate_url_for_fetch(callback)
except (SSRFBlockedError, ValueError) as e:
raise SubscriptionError(f"hub.callback is not reachable: {e}")
requested_topic = (form.get("hub.topic") or "").strip()
topic = canonical_topic(requested_topic)
if not topic:
raise SubscriptionError("hub.topic is not a topic of this site; use the rel=self URL")
secret, lease = None, None
if mode == "subscribe":
# hub.secret and hub.lease_seconds MUST be ignored for unsubscriptions
secret = form.get("hub.secret")
if secret is not None:
if secret == "":
raise SubscriptionError("hub.secret may not be empty")
if len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
raise SubscriptionError("hub.secret must be less than 200 bytes")
lease = LEASE_DEFAULT
if form.get("hub.lease_seconds"):
try:
lease = int(form["hub.lease_seconds"])
except ValueError:
raise SubscriptionError("hub.lease_seconds must be a number")
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
_check_limits(db, topic, callback, callback_ip)
# Each request gets its own id; the latest replaces any pending one
db.execute(
"""
INSERT INTO websub_intents
(topic, callback, mode, secret, lease_seconds, request_id, requested_topic, callback_ip)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(topic, callback) DO UPDATE SET
mode = excluded.mode, secret = excluded.secret,
lease_seconds = excluded.lease_seconds, request_id = excluded.request_id,
requested_topic = excluded.requested_topic, callback_ip = excluded.callback_ip,
attempts = 0, next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
""",
(topic, callback, mode, secret, lease, secrets.token_hex(16), requested_topic, callback_ip),
)
db.commit()
def _check_limits(db, topic: str, callback: str, callback_ip: str) -> None:
"""
Subscription limits, counting pending requests too (ADR-071 4)
Per callback address rather than host name, so many subdomains on one
server share a limit. Renewing an existing subscription is always fine.
"""
if db.execute("SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?",
(topic, callback)).fetchone():
return
pending = db.execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0]
if pending >= MAX_PENDING_INTENTS:
raise SubscriptionError("Too many subscription requests waiting; try again later", 429)
total = db.execute(
"""
SELECT COUNT(*) FROM (
SELECT topic, callback FROM websub_subscriptions
UNION
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe'
)
"""
).fetchone()[0]
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
raise SubscriptionError("This hub has reached its subscription limit", 429)
per_address = db.execute(
"""
SELECT COUNT(*) FROM (
SELECT topic, callback FROM websub_subscriptions WHERE callback_ip = ?
UNION
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe' AND callback_ip = ?
)
""",
(callback_ip, callback_ip),
).fetchone()[0]
if per_address >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
raise SubscriptionError("Too many subscriptions for that callback's server", 429)
def stats(db) -> dict:
"""Active subscribers per topic and deliveries being retried (admin)"""
by_topic = db.execute(
"SELECT topic, COUNT(*) FROM websub_subscriptions WHERE expires_at > datetime('now') "
"GROUP BY topic ORDER BY COUNT(*) DESC, topic"
).fetchall()
retrying = db.execute(
"SELECT COUNT(*) FROM websub_deliveries WHERE attempts > 0"
).fetchone()[0]
return {"by_topic": [(t, n) for t, n in by_topic], "retrying": retrying,
"external": external_hub()}
def new_challenge() -> str:
return secrets.token_urlsafe(32)
+362
View File
@@ -0,0 +1,362 @@
"""
WebSub worker: verification of intent and content distribution (v2.10.0)
Runs like the Webmention worker: a daemon thread in every gunicorn process,
but only the process holding the file lock works the queues.
- Intents (spec 5.3): GET the callback with a challenge; a 2xx echoing it
verifies. Anything else (including redirects) leaves the subscription as
it was. Network errors, DNS failures and 5xx are retried a few times.
- Deliveries (spec 7): render the topic in-app and POST it with
Link rel=hub/self and, if there is a secret, X-Hub-Signature (sha256).
Failures are retried; the subscription stays until its lease ends.
- External hub: a delivery without a subscription is a publish ping.
v2.10.1: every queue item is handled on its own, so one bad item can't stop
the worker; all outbound requests have a hard time limit.
References:
- https://www.w3.org/TR/websub/
- ADR-071
"""
import fcntl
import hashlib
import hmac
import logging
import threading
from datetime import datetime, timedelta
from pathlib import Path
from typing import Optional
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
import httpx
logger = logging.getLogger(__name__)
DELIVERY_BACKOFF = [60, 300, 1800, 7200, 21600] # then give up on this delivery
INTENT_BACKOFF = [60, 300, 900]
PRUNE_EVERY = timedelta(hours=1)
BATCH = 10
VERIFY_MAX_BYTES = 4096 # a challenge reply is short; larger means not verified
def callback_with_params(callback: str, params: dict) -> str:
"""Append parameters to the callback, keeping its query string (spec 5.3)"""
parts = urlsplit(callback)
query = parts.query + ("&" if parts.query else "") + urlencode(params)
return urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))
def signature(secret: str, body: bytes) -> str:
return "sha256=" + hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
def _now() -> str:
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
def _later(seconds: int) -> str:
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
class WebSubWorker(threading.Thread):
"""Daemon thread that verifies subscriptions and delivers content"""
def __init__(self, app, poll_interval: int = 5):
super().__init__(daemon=True, name="websub-worker")
self.app = app
self.poll_interval = poll_interval
self._stop_event = threading.Event()
self._lock_file = None
self._last_prune = None
# --- thread plumbing ----------------------------------------------------
def run(self):
logger.info("WebSub worker started")
while not self._stop_event.is_set():
if not self._acquire_lock():
self._stop_event.wait(self.poll_interval)
continue
try:
with self.app.app_context():
done = self.process_once()
except Exception as e:
logger.error(f"WebSub worker error: {e}", exc_info=True)
done = 0
if not done:
self._stop_event.wait(self.poll_interval)
self._release_lock()
def stop(self):
self._stop_event.set()
def _acquire_lock(self) -> bool:
if self._lock_file:
return True
path = Path(self.app.config["DATABASE_PATH"]).parent / "websub-worker.lock"
lock_file = open(path, "w")
try:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
lock_file.close()
return False
self._lock_file = lock_file
logger.info("WebSub worker holds the queue lock")
return True
def _release_lock(self):
if self._lock_file:
self._lock_file.close()
self._lock_file = None
# --- one pass (also called directly by tests) ---------------------------
def process_once(self) -> int:
from starpunk.database import get_db
db = get_db(self.app)
done = 0
rendered = {} # topic -> (status, content type, body), once per pass
for row in db.execute(
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
(_now(), BATCH),
).fetchall():
self._guarded(db, row, "verification", self.verify, self._intent_crashed)
done += 1
for row in db.execute(
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
(_now(), BATCH),
).fetchall():
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
done += 1
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
self.prune(db)
return done
def _guarded(self, db, row, what, handler, on_crash) -> None:
"""Handle one queue item; an unexpected error affects only that item"""
try:
handler(db, row)
except Exception as e:
logger.error(f"WebSub {what} {row['id']} failed unexpectedly: {e}", exc_info=True)
try:
db.rollback()
on_crash(db, row, f"{type(e).__name__}: {e}"[:200])
db.commit()
except Exception as cleanup:
logger.error(f"WebSub could not record the failure: {cleanup}")
def _intent_crashed(self, db, intent, error) -> None:
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
(intent["id"], intent["request_id"]))
# --- verification of intent (spec 5.3) ----------------------------------
def verify(self, db, intent) -> None:
from starpunk.websub import new_challenge
from starpunk.webmention_ssrf import (
FetchTooLargeError, ResolutionError, SSRFBlockedError, safe_get, user_agent,
)
challenge = new_challenge()
# The topic exactly as the subscriber sent it (spec 5.3)
params = {"hub.mode": intent["mode"],
"hub.topic": intent["requested_topic"] or intent["topic"],
"hub.challenge": challenge}
if intent["mode"] == "subscribe":
params["hub.lease_seconds"] = intent["lease_seconds"]
url = callback_with_params(intent["callback"], params)
retry = False
try:
result = safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
# Exact match; a trailing newline from a framework is tolerated
verified = 200 <= result.status_code < 300 and result.text.rstrip("\r\n") == challenge
retry = result.status_code >= 500
error = None if verified else f"HTTP {result.status_code}" + (
"" if result.status_code >= 300 else ": challenge not echoed")
except ResolutionError as e:
verified, retry, error = False, True, str(e)
except (SSRFBlockedError, ValueError) as e:
verified, error = False, f"Blocked: {e}"
except FetchTooLargeError:
verified, error = False, "Reply too large to be a challenge"
except httpx.TooManyRedirects:
verified, error = False, "Redirected (not allowed)"
except httpx.HTTPError as e:
verified, retry, error = False, True, (str(e) or type(e).__name__)[:200]
# Only act if no newer request replaced this one meanwhile
current = db.execute("SELECT request_id FROM websub_intents WHERE id = ?", (intent["id"],)).fetchone()
if current is None or current[0] != intent["request_id"]:
return
if verified:
if intent["mode"] == "subscribe":
db.execute(
"""
INSERT INTO websub_subscriptions
(topic, callback, secret, lease_seconds, expires_at, verified_at, callback_ip)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(topic, callback) DO UPDATE SET
secret = excluded.secret, lease_seconds = excluded.lease_seconds,
expires_at = excluded.expires_at, verified_at = excluded.verified_at,
callback_ip = excluded.callback_ip, last_error = NULL
""",
(intent["topic"], intent["callback"], intent["secret"], intent["lease_seconds"],
_later(intent["lease_seconds"]), _now(), intent["callback_ip"]),
)
else:
db.execute("DELETE FROM websub_subscriptions WHERE topic = ? AND callback = ?",
(intent["topic"], intent["callback"]))
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
(intent["id"], intent["request_id"]))
logger.info(f"WebSub {intent['mode']} verified: {intent['callback']} -> {intent['topic']}")
elif retry and intent["attempts"] < len(INTENT_BACKOFF):
db.execute(
"UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? "
"WHERE id = ? AND request_id IS ?",
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"], intent["request_id"]),
)
else:
# Verification failed: the subscription state stays as it was
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
(intent["id"], intent["request_id"]))
logger.info(f"WebSub {intent['mode']} not verified for {intent['callback']}: {error}")
db.commit()
# --- content distribution (spec 7) ---------------------------------------
def render_topic(self, topic: str):
"""(status, content_type, body) of a topic, rendered in-app"""
path = urlparse(topic).path or "/"
response = self.app.test_client().get(path, headers={"Accept": "*/*"})
return response.status_code, response.headers.get("Content-Type", ""), response.get_data()
def deliver(self, db, delivery, rendered: Optional[dict] = None) -> None:
from starpunk.websub import hub_url
from starpunk.webmention_ssrf import ResolutionError, SSRFBlockedError, safe_post, user_agent
rendered = {} if rendered is None else rendered
site = self.app.config["SITE_URL"]
sub = None
if delivery["subscription_id"] is None:
# External hub: ping it (hub.url is the de facto parameter)
target, kwargs = hub_url(), {
"data": {"hub.mode": "publish", "hub.url": delivery["topic"], "hub.topic": delivery["topic"]},
"headers": {"User-Agent": user_agent(site)},
}
else:
sub = db.execute(
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
(delivery["subscription_id"], _now()),
).fetchone()
if sub is None: # unsubscribed or expired meanwhile
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
db.commit()
return
if sub["topic"] not in rendered:
rendered[sub["topic"]] = self.render_topic(sub["topic"])
status, content_type, body = rendered[sub["topic"]]
if status in (404, 410):
# The topic is gone: end the subscription and say so (spec 5.2)
self._deny(sub, "The topic no longer exists")
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
db.commit()
return
if status != 200:
self._retry_or_drop(db, delivery, f"Topic rendered HTTP {status}")
db.commit()
return
headers = {
"Content-Type": content_type,
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
"User-Agent": user_agent(site),
}
if sub["secret"] is not None:
headers["X-Hub-Signature"] = signature(sub["secret"], body)
target, kwargs = sub["callback"], {"content": body, "headers": headers}
permanent = False
try:
result = safe_post(target, **kwargs)
status, error = result.status_code, f"HTTP {result.status_code}"
except ResolutionError as e:
status, error = None, str(e) # DNS can recover: retry
except (SSRFBlockedError, ValueError) as e:
status, error, permanent = None, f"Blocked: {e}", True
except httpx.HTTPError as e:
status, error = None, (str(e) or type(e).__name__)[:200]
succeeded = status is not None and 200 <= status < 300
if succeeded:
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
if sub is not None:
db.execute("UPDATE websub_subscriptions SET last_success_at = ?, last_error = NULL WHERE id = ?",
(_now(), sub["id"]))
elif sub is not None and status == 410:
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
logger.info(f"WebSub subscriber gone (410), removed: {sub['callback']}")
else:
if permanent:
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
logger.warning(f"WebSub delivery of {delivery['topic']} refused: {error}")
else:
self._retry_or_drop(db, delivery, error)
if sub is not None:
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
db.commit()
def _retry_or_drop(self, db, delivery, error: str) -> None:
"""
Retry a failed delivery with backoff, then give up on it
The subscription itself stays active until its lease ends (spec 7);
the next change is delivered again.
"""
if delivery["attempts"] < len(DELIVERY_BACKOFF):
db.execute(
"UPDATE websub_deliveries SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE id = ?",
(_later(DELIVERY_BACKOFF[delivery["attempts"]]), error, delivery["id"]),
)
else:
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
logger.warning(f"WebSub delivery of {delivery['topic']} gave up: {error}")
def _deny(self, sub, reason: str) -> None:
"""Tell a subscriber its subscription has ended (hub.mode=denied), best effort"""
from starpunk.webmention_ssrf import safe_get, user_agent
url = callback_with_params(sub["callback"], {"hub.mode": "denied", "hub.topic": sub["topic"],
"hub.reason": reason})
try:
safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
except Exception as e:
logger.info(f"WebSub denial to {sub['callback']} not delivered: {e}")
# --- housekeeping -------------------------------------------------------
def prune(self, db) -> int:
"""Remove subscriptions whose lease has run out, and their deliveries"""
cursor = db.execute("DELETE FROM websub_subscriptions WHERE expires_at <= ?", (_now(),))
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
db.commit()
self._last_prune = datetime.utcnow()
if cursor.rowcount:
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
return cursor.rowcount
def start_worker(app) -> Optional[WebSubWorker]:
if not app.config.get("WEBSUB_ENABLED", True) or app.config.get("TESTING", False):
return None
worker = WebSubWorker(app, poll_interval=app.config.get("WEBSUB_POLL_INTERVAL", 5))
worker.start()
return worker
+115 -27
View File
@@ -1,36 +1,75 @@
/* StarPunk CSS - Minimal responsive stylesheet */
:root {
--color-text: #333; --color-text-light: #666; --color-bg: #fff; --color-bg-alt: #f5f5f5;
--color-link: #0066cc; --color-link-hover: #004499; --color-border: #ddd;
--color-text: #1f2328; --color-text-light: #59636e; --color-bg: #fff; --color-bg-alt: #f4f5f7;
--color-link: #0a5cc2; --color-link-hover: #083f85; --color-border: #e3e6ea; --color-accent-soft: #e8f0fb;
--color-success: #28a745; --color-error: #dc3545; --color-warning: #ffc107; --color-info: #17a2b8;
--font-body: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Arial, sans-serif;
--font-mono: 'SF Mono', Monaco, Consolas, 'Courier New', monospace;
--font-body: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
--font-reading: Charter, 'Bitstream Charter', 'Sitka Text', Cambria, 'Noto Serif', Georgia, serif;
--font-mono: ui-monospace, 'SF Mono', Monaco, Consolas, 'Liberation Mono', monospace;
--spacing-xs: 0.25rem; --spacing-sm: 0.5rem; --spacing-md: 1rem; --spacing-lg: 2rem; --spacing-xl: 4rem;
--max-width: 42rem; --border-radius: 4px;
--max-width: 40rem; --border-radius: 6px;
}
@media (prefers-color-scheme: dark) {
:root {
--color-text: #e4e6e9; --color-text-light: #9aa3ad; --color-bg: #121417; --color-bg-alt: #1c1f24;
--color-link: #74b0ff; --color-link-hover: #a8ceff; --color-border: #2b3038; --color-accent-soft: #1a2738;
}
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body { font-family: var(--font-body); font-size: 1rem; line-height: 1.6; color: var(--color-text); background: var(--color-bg); padding: var(--spacing-md); }
body {
font-family: var(--font-body); font-size: 1rem; line-height: 1.6; color: var(--color-text); background: var(--color-bg);
padding: var(--spacing-md); min-height: 100vh; display: flex; flex-direction: column;
-webkit-font-smoothing: antialiased;
}
h1, h2, h3 { margin-bottom: var(--spacing-md); line-height: 1.2; font-weight: 600; }
h1 { font-size: 2rem; } h2 { font-size: 1.5rem; } h3 { font-size: 1.25rem; }
h1, h2, h3 { margin-bottom: var(--spacing-md); line-height: 1.25; font-weight: 650; letter-spacing: -0.01em; }
h1 { font-size: 1.75rem; } h2 { font-size: 1.375rem; } h3 { font-size: 1.125rem; }
p { margin-bottom: var(--spacing-md); }
a { color: var(--color-link); text-decoration: none; }
a:hover { color: var(--color-link-hover); text-decoration: underline; }
:focus-visible { outline: 2px solid var(--color-link); outline-offset: 2px; border-radius: 2px; }
code, pre { font-family: var(--font-mono); background: var(--color-bg-alt); padding: 0.125rem 0.25rem; border-radius: var(--border-radius); font-size: 0.875rem; }
code, pre { font-family: var(--font-mono); background: var(--color-bg-alt); padding: 0.125rem 0.3rem; border-radius: var(--border-radius); font-size: 0.875em; }
pre { padding: var(--spacing-md); overflow-x: auto; margin-bottom: var(--spacing-md); }
pre code { padding: 0; background: none; }
header, main, footer { max-width: var(--max-width); margin: 0 auto; }
header { margin-bottom: var(--spacing-lg); padding-bottom: var(--spacing-md); border-bottom: 2px solid var(--color-border); }
header h1 { margin-bottom: var(--spacing-sm); }
header h1 a { color: var(--color-text); text-decoration: none; }
nav { display: flex; gap: var(--spacing-md); flex-wrap: wrap; }
nav a { padding: var(--spacing-sm) var(--spacing-md); border-radius: var(--border-radius); transition: background 0.2s; }
nav a:hover { background: var(--color-bg-alt); text-decoration: none; }
main { min-height: 60vh; margin-bottom: var(--spacing-lg); }
footer { padding-top: var(--spacing-lg); border-top: 1px solid var(--color-border); text-align: center; color: var(--color-text-light); font-size: 0.875rem; }
.visually-hidden { position: absolute !important; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); clip-path: inset(50%); white-space: nowrap; }
/* Page frame */
.site-header, main, .site-footer { width: 100%; max-width: var(--max-width); margin-left: auto; margin-right: auto; }
main { flex: 1; margin-bottom: var(--spacing-xl); }
.site-header {
display: flex; flex-wrap: wrap; align-items: baseline; justify-content: space-between; gap: var(--spacing-sm) var(--spacing-lg);
padding: var(--spacing-md) 0 var(--spacing-md); margin-bottom: var(--spacing-lg); border-bottom: 1px solid var(--color-border);
}
.site-title { font-size: 1.375rem; font-weight: 700; letter-spacing: -0.02em; color: var(--color-text); }
.site-title:hover { color: var(--color-link); text-decoration: none; }
.site-nav { display: flex; align-items: center; flex-wrap: wrap; gap: var(--spacing-md); font-size: 0.9375rem; }
.site-nav > a { color: var(--color-text-light); }
.site-nav > a:hover { color: var(--color-link); text-decoration: none; }
.site-search { display: flex; }
.site-search input[type="search"] {
width: 10rem; padding: 0.3rem 0.6rem; font: inherit; font-size: 0.875rem; color: var(--color-text);
background: var(--color-bg-alt); border: 1px solid var(--color-border); border-right: 0; border-radius: var(--border-radius) 0 0 var(--border-radius);
}
.site-search button {
padding: 0.3rem 0.6rem; font: inherit; font-size: 0.875rem; cursor: pointer; color: var(--color-text-light);
background: var(--color-bg-alt); border: 1px solid var(--color-border); border-radius: 0 var(--border-radius) var(--border-radius) 0;
}
.site-search button:hover { color: var(--color-link); }
.site-footer { padding: var(--spacing-md) 0; border-top: 1px solid var(--color-border); color: var(--color-text-light); font-size: 0.8125rem; }
.site-footer p { margin: 0; }
.site-footer a { color: var(--color-text-light); text-decoration: underline; text-decoration-color: var(--color-border); }
.site-footer a:hover { color: var(--color-link); }
.page-title { font-size: 1.375rem; margin-bottom: var(--spacing-lg); }
.tag-name { color: var(--color-link); }
.tag-name::before { content: "#"; }
.dev-mode-warning { background: var(--color-error); color: white; padding: var(--spacing-md); text-align: center; font-weight: bold; margin: calc(-1 * var(--spacing-md)); margin-bottom: var(--spacing-lg); }
@@ -40,17 +79,46 @@ footer { padding-top: var(--spacing-lg); border-top: 1px solid var(--color-borde
.flash-warning { background: #fff3cd; border-color: var(--color-warning); color: #856404; }
.flash-info { background: #d1ecf1; border-color: var(--color-info); color: #0c5460; }
/* Notes */
.h-entry { margin-bottom: var(--spacing-lg); padding-bottom: var(--spacing-lg); border-bottom: 1px solid var(--color-border); }
.h-entry:last-child { border-bottom: none; }
.note-preview .e-content { margin-bottom: var(--spacing-md); }
.note-meta { color: var(--color-text-light); font-size: 0.875rem; }
.h-entry > .p-name a { color: var(--color-text); }
.h-entry > .p-name a:hover { color: var(--color-link); }
.e-content, .p-summary { font-family: var(--font-reading); font-size: 1.125rem; line-height: 1.65; overflow-wrap: anywhere; }
.e-content > :last-child, .p-summary > :last-child { margin-bottom: 0; }
.e-content a { text-decoration: underline; text-decoration-thickness: 1px; text-underline-offset: 2px; text-decoration-color: color-mix(in srgb, currentColor 40%, transparent); }
.e-content a:hover { text-decoration-color: currentColor; }
.e-content blockquote { margin: 0 0 var(--spacing-md); padding-left: var(--spacing-md); border-left: 3px solid var(--color-border); color: var(--color-text-light); }
.e-content ul, .e-content ol { margin: 0 0 var(--spacing-md) 1.25rem; }
.e-content img { max-width: 100%; height: auto; }
.e-content h1, .e-content h2, .e-content h3 { font-family: var(--font-body); }
.reaction { margin-bottom: var(--spacing-sm); font-size: 0.9375rem; color: var(--color-text-light); overflow-wrap: anywhere; }
.reaction-verb { font-weight: 600; margin-right: 0.25rem; }
.reaction-like .reaction-verb { color: #c2410c; }
.reaction-repost .reaction-verb { color: #15803d; }
.reaction a { color: var(--color-text); font-weight: 500; }
.reaction a:hover { color: var(--color-link); }
.reaction-by { margin-left: 0.25rem; }
@media (prefers-color-scheme: dark) {
.reaction-like .reaction-verb { color: #fb923c; }
.reaction-repost .reaction-verb { color: #4ade80; }
}
.read-more { margin: var(--spacing-sm) 0 0; font-size: 0.9375rem; }
.note-meta { display: flex; flex-wrap: wrap; align-items: baseline; gap: 0 0.4rem; margin-top: var(--spacing-sm); color: var(--color-text-light); font-size: 0.8125rem; }
.note-meta a { color: var(--color-text-light); }
.note-nav { margin-top: var(--spacing-md); }
.note-meta a:hover { color: var(--color-link); }
.note-meta > :not(.visually-hidden) + :not(.visually-hidden)::before { content: "·"; margin-right: 0.4rem; color: var(--color-text-light); }
.note-tags { display: inline-flex; flex-wrap: wrap; gap: 0 0.4rem; }
.note-meta > .note-tags::before { margin-right: 0; }
.note-tags .p-category::before { content: "#"; opacity: 0.7; }
.h-entry:not(.note-preview) .note-meta { margin-top: var(--spacing-md); }
.note-nav { margin-top: var(--spacing-lg); font-size: 0.9375rem; }
.empty-state { text-align: center; padding: var(--spacing-xl); color: var(--color-text-light); }
.form-group { margin-bottom: var(--spacing-md); }
label { display: block; margin-bottom: var(--spacing-sm); font-weight: 600; }
input[type="text"], input[type="url"], input[type="email"], textarea { width: 100%; padding: var(--spacing-sm); border: 1px solid var(--color-border); border-radius: var(--border-radius); font-family: inherit; font-size: 1rem; }
input[type="text"], input[type="url"], input[type="email"], input[type="search"], textarea { color: var(--color-text); background: var(--color-bg); width: 100%; padding: var(--spacing-sm); border: 1px solid var(--color-border); border-radius: var(--border-radius); font-family: inherit; font-size: 1rem; }
textarea { font-family: var(--font-mono); resize: vertical; min-height: 10rem; }
small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-light); font-size: 0.875rem; }
.form-checkbox { display: flex; align-items: center; gap: var(--spacing-sm); }
@@ -68,7 +136,8 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.button-warning { background: var(--color-warning); color: #333; border-color: var(--color-warning); }
.button-small { padding: 0.25rem 0.5rem; font-size: 0.875rem; }
.admin-container { max-width: 60rem; }
.admin-page { --max-width: 60rem; }
.table-scroll { overflow-x: auto; }
.admin-nav { display: flex; gap: var(--spacing-md); padding: var(--spacing-md); background: var(--color-bg-alt); border-radius: var(--border-radius); margin-bottom: var(--spacing-lg); flex-wrap: wrap; align-items: center; }
.admin-nav .logout-form { margin-left: auto; }
.admin-nav .logout-form button { margin: 0; }
@@ -79,7 +148,7 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.note-table { width: 100%; border-collapse: collapse; margin-top: var(--spacing-md); }
.note-table th, .note-table td { padding: var(--spacing-md); text-align: left; border-bottom: 1px solid var(--color-border); }
.note-table th { background: var(--color-bg-alt); font-weight: 600; }
.note-content-preview { max-width: 30rem; }
.note-content-preview { min-width: 14rem; overflow-wrap: anywhere; }
.note-excerpt { margin-bottom: var(--spacing-xs); }
.note-slug { color: var(--color-text-light); font-family: var(--font-mono); font-size: 0.75rem; }
.note-date { white-space: nowrap; color: var(--color-text-light); }
@@ -171,11 +240,14 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
}
@media (min-width: 768px) {
body { padding: var(--spacing-lg); }
h1 { font-size: 2.5rem; } h2 { font-size: 2rem; } h3 { font-size: 1.5rem; }
body { padding: var(--spacing-md) var(--spacing-lg); }
h1 { font-size: 2rem; } h2 { font-size: 1.5rem; } h3 { font-size: 1.25rem; }
}
@media (max-width: 767px) {
.site-nav { width: 100%; }
.site-search { flex: 1; justify-content: flex-end; }
.site-search input[type="search"] { width: 100%; max-width: 12rem; }
.note-table { font-size: 0.875rem; }
.note-table th, .note-table td { padding: var(--spacing-sm); }
.note-actions .button { font-size: 0.75rem; padding: 0.25rem 0.5rem; }
@@ -260,6 +332,9 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
}
.webmention-author a {
display: inline-flex;
align-items: center;
gap: var(--spacing-sm);
font-weight: bold;
}
@@ -289,6 +364,13 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
margin-left: 0;
}
.facepile .face > .h-card,
.facepile .face > .h-card > a {
display: block;
width: 100%;
height: 100%;
}
.facepile .face img {
width: 100%;
height: 100%;
@@ -320,6 +402,12 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
border-radius: 4px;
}
.webmention-mention .h-card img {
border-radius: 50%;
vertical-align: middle;
margin-right: var(--spacing-xs);
}
.webmention-mention p {
margin: var(--spacing-xs) 0 0 0;
font-size: 0.875rem;
@@ -426,7 +514,7 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.content-preview {
padding: var(--spacing-sm);
background: white;
background: var(--color-bg);
border: 1px solid var(--color-border);
border-radius: 4px;
margin-top: var(--spacing-xs);
+11
View File
@@ -0,0 +1,11 @@
{% extends "base.html" %}
{% block title %}Post Deleted - {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<article class="error-page">
<h1>410 - Post Deleted</h1>
<p>This post has been deleted.</p>
<p><a href="/">Return to homepage</a></p>
</article>
{% endblock %}
+3
View File
@@ -1,5 +1,8 @@
{% extends "base.html" %}
{# Admin pages use a wider frame so tables fit under the header #}
{% block body_class %} class="admin-page"{% endblock %}
{% block content %}
<div class="admin-container">
<nav class="admin-nav">
+17
View File
@@ -20,7 +20,23 @@
{% endif %}
</div>
{# WebSub subscribers (v2.10.0) #}
{% if websub_stats %}
<p class="note-meta websub-stats">
{% if websub_stats.external %}
WebSub: using {{ websub_stats.external }}
{% elif websub_stats.by_topic %}
WebSub subscribers:
{% for topic, count in websub_stats.by_topic %}{{ topic|replace(config.SITE_URL.rstrip('/'), '') or '/' }} {{ count }}{% if not loop.last %}, {% endif %}{% endfor %}
{% if websub_stats.retrying %} · {{ websub_stats.retrying }} deliveries being retried{% endif %}
{% else %}
WebSub: no subscribers yet
{% endif %}
</p>
{% endif %}
{% if notes %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -67,6 +83,7 @@
{% endfor %}
</tbody>
</table>
</div>
{# Pagination (v1.6.0) #}
{% if total_pages > 1 %}
+53 -7
View File
@@ -12,12 +12,15 @@
<form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form">
<div class="form-group">
<label for="content">Content (Markdown)</label>
{% if note.reacts_to %}
<p class="note-meta">{{ {'like': '★ Like of', 'repost': '🔁 Repost of', 'bookmark': '🔖 Bookmark of'}[note.kind] }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
{% endif %}
<label for="content">{% if note.reacts_to %}Comment (optional, Markdown){% else %}Content (Markdown){% endif %}</label>
<textarea
id="content"
name="content"
rows="20"
required
rows="{{ 6 if note.reacts_to else 20 }}"
{% if not note.reacts_to %}required{% endif %}
autofocus
>{{ note.content }}</textarea>
<small>Use Markdown syntax for formatting</small>
@@ -52,15 +55,58 @@
<label for="published">Published</label>
</div>
{% if syndication_targets %}
<input type="hidden" name="syndication_form" value="1">
<fieldset class="form-group">
<legend>Crosspost</legend>
{% for target in syndication_targets %}
<div class="form-checkbox">
<input type="checkbox" id="syndicate-{{ target.id }}" name="syndicate_to" value="{{ target.id }}"{% if target.id in syndication_enabled %} checked{% endif %}>
<label for="syndicate-{{ target.id }}">{{ target.name }}</label>
</div>
{% endfor %}
</fieldset>
{% endif %}
<div class="form-actions">
<button type="submit" class="button button-primary">Update Note</button>
<a href="{{ url_for('admin.dashboard') }}" class="button button-secondary">Cancel</a>
<form action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" style="display: inline;" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');">
<input type="hidden" name="confirm" value="yes">
<button type="submit" class="button button-danger">Delete Note</button>
</form>
<button type="submit" form="delete-note-form" class="button button-danger">Delete Note</button>
</div>
</form>
{# Outside the edit form: nested forms are not allowed in HTML #}
<form id="delete-note-form" action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');">
<input type="hidden" name="confirm" value="yes">
</form>
{% if syndications %}
<section class="syndication-status">
<h3>Crossposts</h3>
<ul>
{% for s in syndications %}
<li>
<strong>{{ s.name }}</strong>:
{% if s.state == 'synced' and s.remote_url %}
<a href="{{ s.remote_url }}" rel="noopener">posted</a>
{% elif s.state == 'synced' %}
removed
{% elif s.state == 'pending' %}
pending{% if s.attempts %} (retry {{ s.attempts }}){% endif %}
{% else %}
failed
{% endif %}
{% if s.last_error %}<br><small>{{ s.last_error }}</small>{% endif %}
{% if s.state != 'synced' %}
<form action="{{ url_for('admin.retry_syndication', syndication_id=s.id) }}" method="POST" style="display: inline;">
<button type="submit" class="button button-secondary">Retry now</button>
</form>
{% endif %}
</li>
{% endfor %}
</ul>
</section>
{% endif %}
</div>
{% endblock %}
+13
View File
@@ -65,6 +65,19 @@
<label for="published">Publish immediately</label>
</div>
{% if syndication_targets %}
<input type="hidden" name="syndication_form" value="1">
<fieldset class="form-group">
<legend>Crosspost</legend>
{% for target in syndication_targets %}
<div class="form-checkbox">
<input type="checkbox" id="syndicate-{{ target.id }}" name="syndicate_to" value="{{ target.id }}" checked>
<label for="syndicate-{{ target.id }}">{{ target.name }}</label>
</div>
{% endfor %}
</fieldset>
{% endif %}
<div class="form-actions">
<button type="submit" class="button button-primary">Create Note</button>
<a href="{{ url_for('admin.dashboard') }}" class="button button-secondary">Cancel</a>
+2
View File
@@ -8,6 +8,7 @@
<p>Micropub apps you have approved. Revoke any you no longer use.</p>
{% if tokens %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -37,6 +38,7 @@
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p>No apps connected.</p>
{% endif %}
+2
View File
@@ -43,6 +43,7 @@
</button>
</div>
<div class="table-scroll">
<table class="webmention-table">
<thead>
<tr>
@@ -85,6 +86,7 @@
{% endfor %}
</tbody>
</table>
</div>
</form>
{# Pagination #}
+41 -29
View File
@@ -3,59 +3,69 @@
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{% block title %}StarPunk{% endblock %}</title>
<title>{% block title %}{{ config.SITE_NAME }}{% endblock %}</title>
<meta name="color-scheme" content="light dark">
<link rel="stylesheet" href="{{ url_for('static', filename='css/style.css') }}">
<link rel="alternate" type="application/rss+xml" title="{{ config.SITE_NAME }} RSS Feed" href="{{ url_for('public.feed', _external=True) }}">
<link rel="alternate" type="application/xml+opml" title="{{ config.SITE_NAME }} Feed Subscription List" href="{{ url_for('public.opml', _external=True) }}">
<link rel="alternate" type="application/rss+xml" title="{{ config.SITE_NAME }} RSS Feed" href="{{ absolute_url_for('public.feed') }}">
<link rel="alternate" type="application/xml+opml" title="{{ config.SITE_NAME }} Feed Subscription List" href="{{ absolute_url_for('public.opml') }}">
{# Webmention endpoint (v1.6.0) #}
{% if config.WEBMENTION_ENABLED %}
<link rel="webmention" href="/webmention">
<link rel="micropub" href="{{ config.SITE_URL.rstrip('/') }}{{ url_for('micropub.micropub_endpoint') }}">
<link rel="indieauth-metadata" href="{{ config.SITE_URL.rstrip('/') }}{{ url_for('indieauth.metadata') }}">
<link rel="authorization_endpoint" href="{{ config.SITE_URL.rstrip('/') }}{{ url_for('indieauth.authorize') }}">
<link rel="token_endpoint" href="{{ config.SITE_URL.rstrip('/') }}{{ url_for('indieauth.token') }}">
{% endif %}
{# WebSub hub and topic on feed pages (v2.10.0) #}
{% set websub = websub_links() %}
{% if websub %}
<link rel="hub" href="{{ websub.hub }}">
<link rel="self" href="{{ websub.self }}">
{% endif %}
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
<link rel="authorization_endpoint" href="{{ absolute_url_for('indieauth.authorize') }}">
<link rel="token_endpoint" href="{{ absolute_url_for('indieauth.token') }}">
{# Author's home page: rel=me makes the h-card linking to it this site's
representative h-card; rel=author is the authorship fallback (v2.8.0) #}
{% set author_home = (author.url or author.me) if author else config.ADMIN_ME %}
{% if author_home %}
<link rel="me" href="{{ author_home }}">
<link rel="author" href="{{ author_home }}">
{% endif %}
{# rel-me links from discovered author profile (v1.2.0 Phase 2) #}
{% if author and author.rel_me_links %}
{% for profile_url in author.rel_me_links %}
{% for profile_url in author.rel_me_links if profile_url != author_home %}
<link rel="me" href="{{ profile_url }}">
{% endfor %}
{% endif %}
{% block head %}{% endblock %}
</head>
<body>
<body{% block body_class %}{% endblock %}>
{% if config.DEV_MODE %}
<div class="dev-mode-warning">
WARNING: DEVELOPMENT MODE - Authentication bypassed
</div>
{% endif %}
<header>
<h1><a href="/">StarPunk</a></h1>
<nav>
<header class="site-header">
<a class="site-title" href="/">{{ config.SITE_NAME }}</a>
<nav class="site-nav">
<a href="/">Home</a>
<a href="{{ url_for('public.likes') }}">Likes</a>
<a href="{{ url_for('public.feed') }}">RSS</a>
{% if g.me %}
<a href="{{ url_for('admin.dashboard') }}">Dashboard</a>
<a href="{{ url_for('admin.dashboard') }}">Admin</a>
{% else %}
<a href="{{ url_for('auth.login_form') }}">Login</a>
<a href="{{ url_for('auth.login_form') }}">Log in</a>
{% endif %}
<form action="/search" method="get" role="search" style="margin-left: auto; display: flex; gap: var(--spacing-sm);">
<input
type="search"
name="q"
placeholder="Search..."
aria-label="Search"
value="{{ request.args.get('q', '') }}"
minlength="2"
maxlength="100"
required
style="width: 200px; padding: var(--spacing-xs) var(--spacing-sm);"
>
<button type="submit" class="button button-small" style="padding: var(--spacing-xs) var(--spacing-sm);">🔍</button>
<form class="site-search" action="/search" method="get" role="search">
<input type="search" name="q" placeholder="Search" aria-label="Search"
value="{{ request.args.get('q', '') }}" minlength="2" maxlength="100" required>
<button type="submit" aria-label="Search">Search</button>
</form>
</nav>
</header>
@@ -72,8 +82,10 @@
{% block content %}{% endblock %}
</main>
<footer>
<p>StarPunk v{{ config.get('VERSION', '0.5.0') }}</p>
<footer class="site-footer">
<p>
Powered by <a href="https://git.thesatelliteoflove.com/phil/StarPunk">StarPunk</a> v{{ config.get('VERSION', '0.5.0') }}
</p>
</footer>
</body>
</html>
+5 -48
View File
@@ -1,15 +1,15 @@
{% extends "base.html" %}
{% from "partials/media.html" import display_media %}
{% from "partials/note_preview.html" import note_preview %}
{% block title %}StarPunk - Home{% endblock %}
{% block title %}{{ config.SITE_NAME }}{% endblock %}
{% block content %}
<div class="h-feed">
{# h-feed required properties per microformats.org/wiki/h-feed #}
<h2 class="p-name">{{ config.SITE_NAME or 'Recent Notes' }}</h2>
<h1 class="p-name visually-hidden">{{ config.SITE_NAME or 'Recent Notes' }}</h1>
{# u-url for feed (self-reference) #}
<a class="u-url" href="{{ url_for('public.index', _external=True) }}" style="display: none;">Feed URL</a>
<a class="u-url" href="{{ absolute_url_for('public.index') }}" style="display: none;">Feed URL</a>
{# Feed-level author h-card with all properties #}
{# Hidden because it's semantic-only markup for parsers, not visual content #}
@@ -34,50 +34,7 @@
{% if notes %}
{% for note in notes %}
<article class="h-entry note-preview">
{# Detect if note has explicit title (starts with # heading) - per Q22 #}
{% set has_explicit_title = note.content.strip().startswith('#') %}
{# p-name only if note has explicit title (per Q22) #}
{% if has_explicit_title %}
<h3 class="p-name">{{ note.title }}</h3>
{% endif %}
{# Media preview (if available) #}
{{ display_media(note.media) }}
{# e-content: note content (preview) #}
<div class="e-content">
{{ note.html[:300]|safe }}{% if note.html|length > 300 %}...{% endif %}
</div>
<footer class="note-meta">
{# u-url for permalink #}
<a class="u-url" href="{{ url_for('public.note', slug=note.slug, _external=True) }}">
<time class="dt-published" datetime="{{ note.created_at.isoformat() }}">
{{ note.created_at.strftime('%B %d, %Y') }}
</time>
</a>
{# Tags in preview #}
{% if note.tags %}
<span class="note-tags">
{% for tag in note.tags %}
<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>
{% endfor %}
</span>
{% endif %}
{# Author h-card nested in each h-entry (per Q20) #}
{% if author %}
<div class="p-author h-card">
<a class="p-name u-url" href="{{ author.url or author.me }}">
{{ author.name or author.url or author.me }}
</a>
</div>
{% endif %}
</footer>
</article>
{{ note_preview(note, author) }}
{% endfor %}
{% else %}
<p class="empty-state">No notes published yet. Check back soon!</p>
+28
View File
@@ -0,0 +1,28 @@
{% extends "base.html" %}
{% from "partials/note_preview.html" import note_preview %}
{% block title %}Likes · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<div class="h-feed">
<h1 class="p-name page-title">Likes</h1>
{# Feed URL and author, as on the home page (v2.8.0) #}
<a class="u-url" href="{{ absolute_url_for(request.endpoint, **request.view_args) }}" hidden>Feed URL</a>
{% if author %}
<span class="p-author h-card" hidden><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
{% endif %}
{% if notes %}
{% for note in notes %}
{{ note_preview(note, author) }}
{% endfor %}
{% else %}
<p class="empty-state">Nothing liked yet.</p>
{% endif %}
<nav class="note-nav">
<a href="/">← All notes</a>
</nav>
</div>
{% endblock %}
+35 -118
View File
@@ -1,12 +1,14 @@
{% extends "base.html" %}
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% import "partials/responses.html" as responses with context %}
{% block title %}{{ note.slug }} - StarPunk{% endblock %}
{% block title %}{% if note.has_title %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.bookmark_of %}Bookmark{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<article class="h-entry">
{# Detect if note has explicit title (starts with # heading) - per Q22 #}
{% set has_explicit_title = note.content.strip().startswith('#') %}
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
{% set has_explicit_title = note.has_title %}
{# p-name only if note has explicit title (per Q22) #}
{% if has_explicit_title %}
@@ -17,13 +19,14 @@
{% if note.in_reply_to %}
<div class="reply-context">
<span class="reply-label">In reply to:</span>
<blockquote class="h-cite">
{# u-in-reply-to on the h-cite itself makes it the entry's property (Post Type Discovery) #}
<blockquote class="u-in-reply-to h-cite">
{% if reply_context and reply_context.fetch_status == 'fetched' %}
{% if reply_context.content_snippet %}
<p class="p-content">{{ reply_context.content_snippet }}</p>
{% endif %}
<cite>
<a class="u-in-reply-to" href="{{ note.in_reply_to }}">
<a class="u-url" href="{{ note.in_reply_to }}">
{% if reply_context.author_name %}{{ reply_context.author_name }}{% endif %}
{% if reply_context.site_name %} on {{ reply_context.site_name }}{% endif %}
{% if not reply_context.author_name and not reply_context.site_name %}{{ note.in_reply_to }}{% endif %}
@@ -31,67 +34,52 @@
</cite>
{% else %}
<cite>
<a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
<a class="u-url" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
</cite>
{% endif %}
</blockquote>
</div>
{% endif %}
{{ reaction(note) }}
{# u-photo placement: Per draft spec, u-photo must be direct child of h-entry, #}
{# NOT inside e-content. Media is rendered ABOVE e-content to meet this requirement. #}
{{ display_media(note.media) }}
{{ display_media(note.media, note.excerpt|truncate(150)) }}
{# e-content: note content BELOW media (per ADR-057) #}
<div class="e-content">
{{ note.html|safe }}
{{ note.body_html|safe }}
</div>
<footer class="note-meta">
{# u-url and u-uid same for notes (per Q23) #}
<a class="u-url u-uid" href="{{ url_for('public.note', slug=note.slug, _external=True) }}">
<time class="dt-published" datetime="{{ note.created_at.isoformat() }}">
{{ note.created_at.strftime('%B %d, %Y at %I:%M %p') }}
</time>
</a>
{# dt-updated if note was modified #}
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at|iso_utc }}">{{ note.created_at.strftime('%B %-d, %Y · %-I:%M %p') }}</time></a>
{% if note.updated_at and note.updated_at != note.created_at %}
<span class="updated">
(Updated: <time class="dt-updated" datetime="{{ note.updated_at.isoformat() }}">{{ note.updated_at.strftime('%B %d, %Y') }}</time>)
<span class="updated">Updated <time class="dt-updated" datetime="{{ note.updated_at|iso_utc }}">{{ note.updated_at.strftime('%b %-d, %Y') }}</time></span>
{% endif %}
{# rel="tag" per microformats2 p-category #}
{% if note.tags %}
<span class="note-tags">
{% for tag in note.tags %}<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>{% endfor %}
</span>
{% endif %}
{# Tags / Categories #}
{# rel="tag" per microformats2 p-category specification #}
{% if note.tags %}
<div class="note-tags">
{% for tag in note.tags %}
<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">
{{ tag.display_name }}
</a>
{% endfor %}
</div>
{% set synced = (syndications or [])|selectattr('remote_url')|list %}
{% if synced %}
<span class="syndication">Also on {% for s in synced %}<a class="u-syndication" href="{{ s.remote_url }}" rel="syndication">{{ s.name }}</a>{% if not loop.last %}, {% endif %}{% endfor %}</span>
{% endif %}
{# Author h-card (nested within h-entry per Q20) #}
{# Author h-card (nested within h-entry per Q20); hidden on a single-author site #}
{% if author %}
<div class="p-author h-card">
{% if author.photo %}
<img class="u-photo" src="{{ author.photo }}" alt="{{ author.name or 'Author' }}" width="48" height="48">
{% endif %}
<a class="p-name u-url" href="{{ author.url or author.me }}">
{{ author.name or author.url or author.me }}
</a>
{% if author.note %}
<span class="p-note" style="display: none;">{{ author.note }}</span>
{% endif %}
</div>
<span class="p-author h-card visually-hidden">
{% if author.photo %}<img class="u-photo" src="{{ author.photo }}" alt="">{% endif %}
<a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a>
{% if author.note %}<span class="p-note">{{ author.note }}</span>{% endif %}
</span>
{% endif %}
</footer>
<nav class="note-nav">
<a href="/">Back to all notes</a>
<a href="/">← All notes</a>
</nav>
{# Webmentions section (v1.6.0 Phase 5) - only shown if approved webmentions exist #}
@@ -102,106 +90,35 @@
{% if replies %}
<div class="webmention-group webmention-replies">
<h3>{{ replies|length }} {{ 'Reply' if replies|length == 1 else 'Replies' }}</h3>
{% for reply in replies %}
<article class="webmention webmention-reply h-cite">
<div class="webmention-author">
{% if reply.author_photo %}
<img src="{{ reply.author_photo }}" alt="" width="48" height="48" loading="lazy">
{% endif %}
<a class="p-author" href="{{ reply.author_url or reply.source_url }}">
{{ reply.author_name or 'Anonymous' }}
</a>
</div>
{% if reply.content_snippet %}
<div class="p-content">{{ reply.content_snippet|safe }}</div>
{% endif %}
<a class="u-url webmention-date" href="{{ reply.source_url }}">
<time class="dt-published" datetime="{{ reply.received_at.isoformat() }}">
{{ reply.received_at.strftime('%B %d, %Y') }}
</time>
</a>
</article>
{% endfor %}
{% for wm in replies %}{{ responses.reply(wm) }}{% endfor %}
</div>
{% endif %}
{% if likes %}
<div class="webmention-group webmention-likes">
<h3>{{ likes|length }} {{ 'Like' if likes|length == 1 else 'Likes' }}</h3>
<div class="facepile">
{% for wm in likes[:8] %}
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
{% if wm.author_photo %}
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
{% else %}
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
{{ (wm.author_name or '?')[0]|upper }}
</span>
{% endif %}
</a>
{% endfor %}
{% if likes|length > 8 %}
<span class="face-overflow">+{{ likes|length - 8 }} others</span>
{% endif %}
</div>
{{ responses.facepile(likes, 'u-like') }}
</div>
{% endif %}
{% if reposts %}
<div class="webmention-group webmention-reposts">
<h3>{{ reposts|length }} {{ 'Repost' if reposts|length == 1 else 'Reposts' }}</h3>
<div class="facepile">
{% for wm in reposts[:8] %}
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
{% if wm.author_photo %}
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
{% else %}
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
{{ (wm.author_name or '?')[0]|upper }}
</span>
{% endif %}
</a>
{% endfor %}
{% if reposts|length > 8 %}
<span class="face-overflow">+{{ reposts|length - 8 }} others</span>
{% endif %}
</div>
{{ responses.facepile(reposts, 'u-repost') }}
</div>
{% endif %}
{% if bookmarks %}
<div class="webmention-group webmention-bookmarks">
<h3>{{ bookmarks|length }} {{ 'Bookmark' if bookmarks|length == 1 else 'Bookmarks' }}</h3>
<div class="facepile">
{% for wm in bookmarks[:8] %}
<a href="{{ wm.author_url or wm.source_url }}" class="face" title="{{ wm.author_name or 'Anonymous' }}">
{% if wm.author_photo %}
<img src="{{ wm.author_photo }}" alt="{{ wm.author_name or '' }}" width="32" height="32" loading="lazy">
{% else %}
<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}">
{{ (wm.author_name or '?')[0]|upper }}
</span>
{% endif %}
</a>
{% endfor %}
{% if bookmarks|length > 8 %}
<span class="face-overflow">+{{ bookmarks|length - 8 }} others</span>
{% endif %}
</div>
{{ responses.facepile(bookmarks, 'u-bookmark') }}
</div>
{% endif %}
{% if mentions %}
<div class="webmention-group webmention-mentions">
<h3>{{ mentions|length }} {{ 'Mention' if mentions|length == 1 else 'Mentions' }}</h3>
{% for mention in mentions %}
<div class="webmention webmention-mention">
<a href="{{ mention.source_url }}">{{ mention.author_name or mention.source_url }}</a>
{% if mention.content_snippet %}
<p>{{ mention.content_snippet|truncate(200) }}</p>
{% endif %}
</div>
{% endfor %}
{% for wm in mentions %}{{ responses.mention(wm) }}{% endfor %}
</div>
{% endif %}
</section>
+3 -2
View File
@@ -1,11 +1,12 @@
{# Reusable media display macro #}
{% macro display_media(media_items) %}
{# fallback_alt: for photos without a caption, usually the post's text (v2.6.5) #}
{% macro display_media(media_items, fallback_alt='') %}
{% if media_items %}
<div class="note-media">
{% for item in media_items %}
<figure class="media-item">
<img src="{{ url_for('public.media_file', path=item.path) }}"
alt="{{ item.caption or 'Image' }}"
alt="{{ item.caption or fallback_alt or 'Image' }}"
class="u-photo"
loading="lazy">
{# No figcaption - caption is for alt text only #}
+46
View File
@@ -0,0 +1,46 @@
{# One note in a list (home, tag pages). v2.3.0 #}
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% macro note_preview(note, author) %}
<article class="h-entry note-preview">
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
{% set has_explicit_title = note.has_title %}
{% if has_explicit_title %}
<h2 class="p-name"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">{{ note.title }}</a></h2>
{% endif %}
{% if note.in_reply_to %}
<p class="reaction reply-to"><span class="reaction-verb">↩ In reply to</span> <a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a></p>
{% endif %}
{{ reaction(note) }}
{{ display_media(note.media, note.excerpt|truncate(150)) }}
{% if has_explicit_title %}
{# Titled notes are articles: show a summary and link through #}
<div class="p-summary">
<p>{{ note.excerpt }}</p>
</div>
<p class="read-more"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">Read more →</a></p>
{% else %}
{# Always present, even empty: without it a text-free photo post gets an implied name (mf2) #}
<div class="e-content">
{{ note.html|safe }}
</div>
{% endif %}
<footer class="note-meta">
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}"><time class="dt-published" datetime="{{ note.created_at|iso_utc }}">{{ note.created_at.strftime('%b %-d, %Y') }}</time></a>
{% if note.tags %}
<span class="note-tags">
{% for tag in note.tags %}<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>{% endfor %}
</span>
{% endif %}
{% if author %}
{# Single-author site: keep the h-card for parsers, hide it from readers #}
<span class="p-author h-card visually-hidden"><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
{% endif %}
</footer>
</article>
{% endmacro %}
+12
View File
@@ -0,0 +1,12 @@
{# "Liked" / "Reposted" line for likes and reposts (v2.5.0) #}
{% macro reaction(note) %}
{% if note.reacts_to %}
{% set ctx = note.context if note.context is defined else None %}
{% if ctx and ctx.title %}{% set label = ctx.title %}{% elif ctx and ctx.content_snippet %}{% set label = ctx.content_snippet|striptags|truncate(90) %}{% elif ctx and (ctx.author_name or ctx.site_name) %}{% set label = 'a post' %}{% else %}{% set label = note.reacts_to %}{% endif %}
<p class="reaction reaction-{{ note.kind }}">
<span class="reaction-verb">{{ {'like': '★ Liked', 'repost': '🔁 Reposted', 'bookmark': '🔖 Bookmarked'}[note.kind] }}</span>
<a class="u-{{ note.kind }}-of" href="{{ note.reacts_to }}">{{ label }}</a>
{% if ctx and (ctx.author_name or ctx.site_name) %}<span class="reaction-by">by {{ ctx.author_name or ctx.site_name }}</span>{% endif %}
</p>
{% endif %}
{% endmacro %}
+62
View File
@@ -0,0 +1,62 @@
{# Received webmentions as microformats2 responses (v2.8.0)
Each response is an h-cite in a property of the note's h-entry:
u-comment (replies), u-like, u-repost, u-bookmark, u-mention.
See https://indieweb.org/comments-presentation and microformats.org/wiki/h-entry #}
{# When the source said when it was posted, use that; else when we received it #}
{% macro response_time(wm, fmt='%B %-d, %Y') %}
{% set when = wm.source_published or wm.received_at %}
<time class="dt-published" datetime="{{ when|iso_utc }}">{{ when.strftime(fmt) if when.strftime is defined else when }}</time>
{%- endmacro %}
{# The response's own URL: the source's u-url if it gave one #}
{% macro response_url(wm) %}{{ wm.source_post_url or wm.source_url }}{% endmacro %}
{# Author h-card: name, URL and photo. u-url is on an inner link: on the
h-card element itself it would belong to the enclosing h-cite #}
{% macro author_card(wm, class='', photo_size=48, show_name=True) %}
<span class="p-author h-card {{ class }}"><a class="u-url" href="{{ wm.author_url or wm.source_url }}" title="{{ wm.author_name or 'Anonymous' }}">
{%- if wm.author_photo %}<img class="u-photo" src="{{ wm.author_photo }}" alt="" width="{{ photo_size }}" height="{{ photo_size }}" loading="lazy">
{%- elif not show_name %}<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}" aria-hidden="true">{{ (wm.author_name or '?')[0]|upper }}</span>
{%- endif %}
<span class="p-name{{ '' if show_name else ' visually-hidden' }}">{{ wm.author_name or 'Anonymous' }}</span>
</a></span>
{%- endmacro %}
{# A reply, shown as a comment #}
{% macro reply(wm) %}
<article class="webmention webmention-reply u-comment h-cite">
<div class="webmention-author">{{ author_card(wm) }}</div>
{% if wm.content_snippet %}
<div class="e-content">{{ wm.content_snippet|safe }}</div>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
</article>
{%- endmacro %}
{# Likes, reposts or bookmarks as a facepile; prop is u-like / u-repost / u-bookmark.
Faces past `shown` are hidden on screen but kept for parsers. #}
{% macro facepile(items, prop, shown=8) %}
<div class="facepile">
{% for wm in items %}
<span class="face {{ prop }} h-cite"{% if loop.index > shown %} hidden{% endif %}>
{{ author_card(wm, photo_size=32, show_name=False) }}
<a class="u-url" href="{{ response_url(wm) }}" hidden>{{ response_time(wm) }}</a>
</span>
{% endfor %}
{% if items|length > shown %}
<span class="face-overflow">+{{ items|length - shown }} others</span>
{% endif %}
</div>
{%- endmacro %}
{# A post that mentions this note without replying #}
{% macro mention(wm) %}
<div class="webmention webmention-mention u-mention h-cite">
{{ author_card(wm, photo_size=24) }}
{% if wm.content_snippet %}
<p class="p-content">{{ wm.content_snippet|striptags|truncate(200) }}</p>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
</div>
{%- endmacro %}
+1 -1
View File
@@ -1,6 +1,6 @@
{% extends "base.html" %}
{% block title %}{% if query %}Search: {{ query }}{% else %}Search{% endif %} - StarPunk{% endblock %}
{% block title %}{% if query %}Search: {{ query }}{% else %}Search{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<div class="search-container">
+11 -48
View File
@@ -1,65 +1,28 @@
{% extends "base.html" %}
{% from "partials/media.html" import display_media %}
{% from "partials/note_preview.html" import note_preview %}
{% block title %}{{ tag.display_name }} - StarPunk{% endblock %}
{% block title %}{{ tag.display_name }} · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<div class="h-feed">
<h1 class="p-name">Notes tagged "{{ tag.display_name }}"</h1>
<h1 class="p-name page-title">Notes tagged <span class="tag-name">{{ tag.display_name }}</span></h1>
{# Feed URL and author, as on the home page (v2.8.0) #}
<a class="u-url" href="{{ absolute_url_for(request.endpoint, **request.view_args) }}" hidden>Feed URL</a>
{% if author %}
<span class="p-author h-card" hidden><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
{% endif %}
{% if notes %}
{% for note in notes %}
<article class="h-entry note-preview">
{# Detect if note has explicit title (starts with # heading) - per Q22 #}
{% set has_explicit_title = note.content.strip().startswith('#') %}
{# p-name only if note has explicit title (per Q22) #}
{% if has_explicit_title %}
<h3 class="p-name">{{ note.title }}</h3>
{% endif %}
{# Media preview (if available) #}
{{ display_media(note.media) }}
{# e-content: note content (preview) #}
<div class="e-content">
{{ note.html[:300]|safe }}{% if note.html|length > 300 %}...{% endif %}
</div>
<footer class="note-meta">
{# u-url for permalink #}
<a class="u-url" href="{{ url_for('public.note', slug=note.slug, _external=True) }}">
<time class="dt-published" datetime="{{ note.created_at.isoformat() }}">
{{ note.created_at.strftime('%B %d, %Y') }}
</time>
</a>
{# Tags in preview #}
{% if note.tags %}
<span class="note-tags">
{% for tag in note.tags %}
<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>
{% endfor %}
</span>
{% endif %}
{# Author h-card nested in each h-entry (per Q20) #}
{% if author %}
<div class="p-author h-card">
<a class="p-name u-url" href="{{ author.url or author.me }}">
{{ author.name or author.url or author.me }}
</a>
</div>
{% endif %}
</footer>
</article>
{{ note_preview(note, author) }}
{% endfor %}
{% else %}
<p class="empty-state">No notes with this tag.</p>
{% endif %}
<nav class="note-nav">
<a href="/">Back to all notes</a>
<a href="/">← All notes</a>
</nav>
</div>
{% endblock %}
+48
View File
@@ -0,0 +1,48 @@
"""
Absolute URLs come from SITE_URL, not the request (v2.0.1)
Behind a reverse proxy the app is reached over plain HTTP, so URLs built
from the request said http:// on an https site.
"""
import pytest
from starpunk import create_app
from starpunk.notes import create_note
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
return create_app(
{
"TESTING": True,
"SITE_URL": "https://blog.example.com",
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
}
)
def test_pages_use_site_url_behind_http_proxy(app):
with app.app_context():
note = create_note(content="Hello over the proxy", published=True, tags=["proxy"])
client = app.test_client()
pages = ["/", f"/note/{note.slug}", "/tag/proxy"]
for path in pages:
# The test client makes plain http://localhost requests, like Caddy does
response = client.get(path)
assert response.status_code == 200, path
html = response.data.decode()
assert "http://localhost" not in html, path
home = client.get("/").data.decode()
assert 'href="https://blog.example.com/feed"' in home
assert 'href="https://blog.example.com/opml.xml"' in home
note_html = client.get(f"/note/{note.slug}").data.decode()
assert f'href="https://blog.example.com/note/{note.slug}"' in note_html
+1 -1
View File
@@ -230,7 +230,7 @@ class TestGenerateFeed:
# Check for Atom namespace and self-link
assert "atom" in feed_xml
assert "feed.xml" in feed_xml
assert "feed.rss" in feed_xml # self link (v2.10.0: /feed.rss is the RSS topic)
assert 'rel="self"' in feed_xml
def test_generate_feed_item_structure(self, app, sample_notes):
+228
View File
@@ -0,0 +1,228 @@
"""
Likes and reposts (v2.5.0, ADR-068)
Fake GoToSocial and Bluesky servers check that likes/reposts of their own
posts become native favourites/boosts/likes/reposts, and everything else
falls back (reposts) or is skipped (likes).
"""
import json
import httpx
import pytest
from starpunk import create_app, syndication
from starpunk.indieauth import create_access_token
from starpunk.notes import delete_note, get_note
from starpunk.syndication import BlueskyClient, MastodonClient, get_note_syndications, process_due
SITE = "https://blog.example.com"
GTS = "https://social.example.com"
FEDI_POST = "https://mastodon.example/@alice/111"
BSKY_POST = "https://bsky.app/profile/alice.example/post/3abc"
WEB_PAGE = "https://someblog.example/2026/a-post"
class FakeGTS:
def __init__(self):
self.calls = []
self.statuses = {}
def __call__(self, request):
path = request.url.path
self.calls.append((request.method, path))
if path == "/api/v2/search":
q = request.url.params["q"]
return httpx.Response(200, json={"statuses": [{"id": "s1"}] if q == FEDI_POST else []})
if path in ("/api/v1/statuses/s1/favourite", "/api/v1/statuses/s1/unfavourite",
"/api/v1/statuses/s1/reblog", "/api/v1/statuses/s1/unreblog"):
return httpx.Response(200, json={"id": "s1"})
if path == "/api/v1/statuses" and request.method == "POST":
sid = f"p{len(self.statuses) + 1}"
self.statuses[sid] = json.loads(request.content)
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
if path.startswith("/api/v1/statuses/") and request.method == "DELETE":
self.statuses.pop(path.rsplit("/", 1)[1], None)
return httpx.Response(200, json={})
return httpx.Response(500)
class FakePDS:
def __init__(self):
self.records = {}
self.calls = []
def __call__(self, request):
nsid = request.url.path.removeprefix("/xrpc/")
self.calls.append(nsid)
if nsid == "com.atproto.server.createSession":
return httpx.Response(200, json={"did": "did:plc:me", "handle": "me.example",
"accessJwt": "a", "refreshJwt": "r"})
if nsid == "com.atproto.identity.resolveHandle":
return httpx.Response(200, json={"did": "did:plc:alice"})
if nsid == "app.bsky.feed.getPosts":
return httpx.Response(200, json={"posts": [{"uri": request.url.params["uris"], "cid": "cid1"}]})
if nsid == "com.atproto.repo.createRecord":
body = json.loads(request.content)
uri = f"at://did:plc:me/{body['collection']}/r{len(self.records) + 1}"
self.records[uri] = body["record"]
return httpx.Response(200, json={"uri": uri, "cid": "c"})
if nsid == "com.atproto.repo.deleteRecord":
body = json.loads(request.content)
self.records.pop(f"at://{body['repo']}/{body['collection']}/{body['rkey']}", None)
return httpx.Response(200, json={})
return httpx.Response(500, json={"error": nsid})
@pytest.fixture
def gts():
return FakeGTS()
@pytest.fixture
def pds():
return FakePDS()
@pytest.fixture
def app(tmp_path, gts, pds):
data = tmp_path / "data"
app = create_app({
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com", "SESSION_SECRET": "x",
"DATA_PATH": data, "NOTES_PATH": data / "notes", "DATABASE_PATH": data / "starpunk.db",
"GOTOSOCIAL_URL": GTS, "GOTOSOCIAL_ACCESS_TOKEN": "t",
"BLUESKY_HANDLE": "me.example", "BLUESKY_APP_PASSWORD": "p", "BLUESKY_SERVICE": "https://pds.example",
"FEED_CACHE_ENABLED": False,
})
# The feed's note list is cached at module level across apps
from starpunk.routes import public
public._feed_cache["notes"] = None
public._feed_cache["timestamp"] = None
syndication._client_cache[GTS] = MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(gts)))
syndication._client_cache["bluesky:https://pds.example:me.example"] = BlueskyClient(
"https://pds.example", "me.example", "p", http=httpx.Client(transport=httpx.MockTransport(pds)))
yield app
syndication._client_cache.clear()
@pytest.fixture
def post(app, monkeypatch):
# Don't fetch contexts from the network
monkeypatch.setattr("starpunk.reply_context.fetch_reply_context", lambda *a, **k: None)
with app.test_request_context():
token = create_access_token("https://quill.example/", "create update delete")["access_token"]
client = app.test_client()
def _post(**props):
# Crosspost to every configured target; absent means none (v2.9.0).
# Sent inside properties, as JSON clients like Quill do.
with app.app_context():
from starpunk.syndication import get_targets
props["mp-syndicate-to"] = [t.uid for t in get_targets().values()]
response = client.post("/micropub", json={"type": ["h-entry"], "properties": props},
headers={"Authorization": f"Bearer {token}"})
assert response.status_code == 201, response.get_json()
location = response.headers["Location"]
with app.app_context():
note = get_note(slug=location.rsplit("/", 1)[1])
process_due(limit=10)
rows = {r["target"]: r for r in get_note_syndications(note.id)}
return note, rows
return _post
def page(app, path):
return app.test_client().get(path).data.decode()
class TestSite:
def test_like_without_comment(self, app, post):
note, _ = post(**{"like-of": [WEB_PAGE]})
assert note.kind == "like" and note.content == ""
assert note.slug not in page(app, "/")
likes = page(app, "/likes")
assert f'class="u-like-of" href="{WEB_PAGE}"' in likes and "★ Liked" in likes
assert note.slug not in page(app, "/feed.json")
def test_repost_in_stream_and_feed(self, app, post):
note, _ = post(**{"repost-of": [WEB_PAGE], "content": ["Worth a read"]})
home = page(app, "/")
assert f'class="u-repost-of" href="{WEB_PAGE}"' in home and "Worth a read" in home
feed = json.loads(page(app, "/feed.json"))
item = next(i for i in feed["items"] if note.slug in i["url"])
assert "Reposted" in item["content_html"] and WEB_PAGE in item["content_html"]
def test_webmention_goes_to_target_with_clean_url(self, app, post, monkeypatch):
sent = {}
monkeypatch.setattr("starpunk.webmention_sender.queue_webmentions_for_note",
lambda **kw: sent.update(kw) or 0)
app.config["WEBMENTION_ENABLED"] = True
note, _ = post(**{"like-of": [WEB_PAGE]})
assert sent["in_reply_to"] == WEB_PAGE
assert sent["note_url"] == f"{SITE}/note/{note.slug}" # no double slash
def test_both_kinds_rejected(self, app, post):
with pytest.raises(AssertionError):
post(**{"like-of": [WEB_PAGE], "repost-of": [WEB_PAGE]})
class TestGoToSocial:
def test_like_of_fediverse_post_favourites(self, app, post, gts):
note, rows = post(**{"like-of": [FEDI_POST]})
assert rows["gotosocial"]["remote_id"] == "favourite:s1"
assert rows["gotosocial"]["remote_url"] is None
with app.test_request_context():
delete_note(id=note.id)
process_due(limit=10)
assert ("POST", "/api/v1/statuses/s1/unfavourite") in gts.calls
def test_like_of_web_page_is_not_crossposted(self, app, post, gts):
_, rows = post(**{"like-of": [WEB_PAGE]})
assert rows["gotosocial"]["state"] == "synced" and rows["gotosocial"]["remote_id"] is None
assert gts.statuses == {}
def test_repost_boosts(self, app, post, gts):
_, rows = post(**{"repost-of": [FEDI_POST]})
assert rows["gotosocial"]["remote_id"] == "reblog:s1"
def test_repost_with_comment_posts_status(self, app, post, gts):
_, rows = post(**{"repost-of": [FEDI_POST], "content": ["So true"]})
(status,) = gts.statuses.values()
assert status["status"].startswith("So true") and FEDI_POST in status["status"]
assert ("POST", "/api/v1/statuses/s1/reblog") not in gts.calls
def test_repost_of_web_page_posts_link(self, app, post, gts):
post(**{"repost-of": [WEB_PAGE]})
(status,) = gts.statuses.values()
assert status["status"].startswith("🔁") and WEB_PAGE in status["status"]
class TestBluesky:
def test_like_of_bluesky_post(self, app, post, pds):
note, rows = post(**{"like-of": [BSKY_POST]})
uri = rows["bluesky"]["remote_id"]
assert "/app.bsky.feed.like/" in uri
assert pds.records[uri]["subject"] == {"uri": "at://did:plc:alice/app.bsky.feed.post/3abc", "cid": "cid1"}
with app.test_request_context():
delete_note(id=note.id)
process_due(limit=10)
assert uri not in pds.records
def test_like_of_web_page_skipped(self, app, post, pds):
_, rows = post(**{"like-of": [WEB_PAGE]})
assert rows["bluesky"]["remote_id"] is None and pds.records == {}
def test_repost_and_quote(self, app, post, pds):
_, rows = post(**{"repost-of": [BSKY_POST]})
assert "/app.bsky.feed.repost/" in rows["bluesky"]["remote_id"]
_, rows = post(**{"repost-of": [BSKY_POST], "content": ["Great thread"]})
record = pds.records[rows["bluesky"]["remote_id"]]
assert record["text"] == "Great thread"
assert record["embed"] == {"$type": "app.bsky.embed.record",
"record": {"uri": "at://did:plc:alice/app.bsky.feed.post/3abc", "cid": "cid1"}}
def test_repost_of_web_page_posts_link_card(self, app, post, pds):
_, rows = post(**{"repost-of": [WEB_PAGE]})
record = pds.records[rows["bluesky"]["remote_id"]]
assert record["embed"]["external"]["uri"] == WEB_PAGE
assert record["text"].startswith("🔁")
+10 -15
View File
@@ -110,7 +110,7 @@ class TestImageValidation:
assert mime_type == 'image/jpeg'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_valid_png(self):
"""Test validation of valid PNG image"""
@@ -120,7 +120,7 @@ class TestImageValidation:
assert mime_type == 'image/png'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_valid_gif(self):
"""Test validation of valid GIF image"""
@@ -130,7 +130,7 @@ class TestImageValidation:
assert mime_type == 'image/gif'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_valid_webp(self):
"""Test validation of valid WebP image"""
@@ -140,7 +140,7 @@ class TestImageValidation:
assert mime_type == 'image/webp'
assert width == 800
assert height == 600
assert file_data == image_data # No conversion
assert isinstance(file_data, Image.Image) # validate_image returns a PIL Image (v1.5.0)
def test_file_too_large(self):
"""Test rejection of >10MB file (per Q6)"""
@@ -168,7 +168,7 @@ class TestImageValidation:
with pytest.raises(ValueError) as exc_info:
validate_image(corrupted_data, 'corrupt.jpg')
assert "Invalid or corrupted" in str(exc_info.value)
assert "Not a supported image" in str(exc_info.value)
class TestHEICSupport:
@@ -183,11 +183,8 @@ class TestHEICSupport:
assert mime_type == 'image/jpeg'
assert width == 800
assert height == 600
# file_data should be different (converted to JPEG)
assert file_data != heic_data
# Verify it's actually JPEG by opening it
img = Image.open(io.BytesIO(file_data))
assert img.format == 'JPEG'
# Converted to JPEG (validate_image returns a PIL Image)
assert file_data.format == 'JPEG'
def test_heic_with_rgba_mode(self):
"""Test HEIC with alpha channel is converted to RGB JPEG"""
@@ -209,8 +206,7 @@ class TestHEICSupport:
# Should be converted to JPEG (no alpha)
assert mime_type == 'image/jpeg'
# Verify it's RGB (no alpha)
img = Image.open(io.BytesIO(file_data))
assert img.mode == 'RGB'
assert file_data.mode == 'RGB'
def test_heic_dimensions_preserved(self):
"""Test HEIC conversion preserves dimensions"""
@@ -273,9 +269,8 @@ class TestMPOSupport:
assert mime_type == 'image/jpeg'
assert width == 800
assert height == 600
# Verify it's actually JPEG by opening it
img = Image.open(io.BytesIO(file_data))
assert img.format == 'JPEG'
# Converted to JPEG (validate_image returns a PIL Image)
assert file_data.format == 'JPEG'
# Note: MPO with single frame may have identical bytes to JPEG
# The important part is the format detection and MIME type correction
+8 -4
View File
@@ -289,16 +289,20 @@ class TestRelMe:
assert 'https://github.com/testuser' in rels['me']
assert 'https://mastodon.social/@testuser' in rels['me']
def test_no_relme_without_author(self, client, app):
"""No rel=me links if no author profile"""
def test_relme_to_admin_me_without_author(self, client, app):
"""Without a discovered profile, rel=me and rel=author still point at ADMIN_ME (v2.8.0)
The home page's p-author h-card links to the author's site; rel=me to
the same URL makes it the representative h-card.
"""
with patch('starpunk.author_discovery.get_author_profile', return_value=None):
response = client.get('/')
parsed = mf2py.parse(doc=response.data.decode(), url='http://localhost/')
rels = parsed.get('rels', {})
# Should not have rel=me, or it should be empty
assert len(rels.get('me', [])) == 0, "Should not have rel=me without author"
assert rels.get('me') == [app.config['ADMIN_ME']]
assert rels.get('author') == [app.config['ADMIN_ME']]
class TestV130Microformats:
+268
View File
@@ -0,0 +1,268 @@
"""
Microformats2 fixes (v2.8.0)
Pages are rendered and parsed with mf2py; assertions are on the parsed data,
which is what IndieWeb readers and Webmention receivers see.
"""
import re
import sqlite3
from unittest.mock import patch
import mf2py
import pytest
from starpunk import mf2_utils
from starpunk.models import split_title
from starpunk.webmention_processor import extract_author, extract_source_details
from starpunk.webmention_worker import WebmentionWorker
BASE = 'http://localhost:5000'
def parse(client, path):
response = client.get(path)
assert response.status_code == 200
return mf2py.parse(doc=response.data.decode(), url=BASE + path)
def entry_of(parsed):
return next(i for i in parsed['items'] if 'h-entry' in i['type'])
def make_note(app, content, **kw):
from starpunk.notes import create_note
with app.test_request_context():
return create_note(content, published=True, **kw)
def add_mention(app, note, kind, n, photo=True, **cols):
db = sqlite3.connect(app.config['DATABASE_PATH'])
fields = {
'source_url': f'https://src.example/{kind}/{n}',
'target_url': f'{BASE}/note/{note.slug}',
'target_note_id': note.id,
'mention_type': kind,
'author_name': f'{kind.title()} Person {n}',
'author_url': f'https://{kind}{n}.example/',
'author_photo': f'https://{kind}{n}.example/me.jpg' if photo else None,
'content_snippet': f'<p>{kind} text <a href="https://x.example/">link</a></p>',
'validation_status': 'valid',
'moderation_status': 'approved',
**cols,
}
db.execute(f"INSERT INTO webmentions ({', '.join(fields)}) VALUES ({', '.join('?' * len(fields))})",
tuple(fields.values()))
db.commit()
db.close()
# A1/A2: received responses are properties of the entry
def test_responses_are_entry_properties(app, client):
note = make_note(app, "A note")
add_mention(app, note, 'reply', 1, source_post_url='https://src.example/reply/1#canonical',
source_published='2026-11-01T10:00:00Z')
for n in range(10):
add_mention(app, note, 'like', n, photo=n % 2 == 0)
add_mention(app, note, 'repost', 1)
add_mention(app, note, 'bookmark', 1)
add_mention(app, note, 'mention', 1)
props = entry_of(parse(client, f'/note/{note.slug}'))['properties']
assert 'children' not in entry_of(parse(client, f'/note/{note.slug}'))
(comment,) = props['comment']
assert comment['type'] == ['h-cite']
author = comment['properties']['author'][0]
assert author['type'] == ['h-card']
assert author['properties']['name'] == ['Reply Person 1']
assert author['properties']['url'] == ['https://reply1.example/']
assert author['properties']['photo'][0] in ('https://reply1.example/me.jpg',
{'value': 'https://reply1.example/me.jpg', 'alt': ''})
assert '<a href="https://x.example/"' in comment['properties']['content'][0]['html']
# the source's own URL and published time, not ours
assert comment['properties']['url'] == ['https://src.example/reply/1#canonical']
assert comment['properties']['published'] == ['2026-11-01T10:00:00Z']
# all ten likes, including the ones past the visible eight
assert len(props['like']) == 10
names = {like['properties']['author'][0]['properties']['name'][0] for like in props['like']}
assert names == {f'Like Person {n}' for n in range(10)}
assert props['like'][0]['properties']['url'] == ['https://src.example/like/0']
assert props['repost'][0]['properties']['author'][0]['properties']['name'] == ['Repost Person 1']
assert props['bookmark'][0]['type'] == ['h-cite']
assert props['mention'][0]['properties']['author'][0]['properties']['url'] == ['https://mention1.example/']
# A3: dt-* are RFC 3339 UTC
UTC = re.compile(r'^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$')
def test_dates_are_utc(app, client):
from starpunk.notes import update_note
note = make_note(app, "Dated")
with app.test_request_context():
update_note(id=note.id, content="Dated, edited")
add_mention(app, note, 'reply', 1)
props = entry_of(parse(client, f'/note/{note.slug}'))['properties']
assert UTC.match(props['published'][0]), props['published']
assert UTC.match(props['updated'][0]), props['updated']
assert UTC.match(props['comment'][0]['properties']['published'][0])
home = parse(client, '/')['items'][0]['children'][0]['properties']
assert UTC.match(home['published'][0])
def test_micropub_source_published_is_utc():
from datetime import datetime, timezone, timedelta
from starpunk.utils import iso_utc
assert iso_utc(datetime(2026, 11, 1, 16, 25, 12, 921768)) == '2026-11-01T16:25:12Z'
assert iso_utc(datetime(2026, 11, 1, 9, 0, tzinfo=timezone(timedelta(hours=-7)))) == '2026-11-01T16:00:00Z'
assert iso_utc('2026-11-01T16:25:15+0000') == '2026-11-01T16:25:15Z'
# A4: only a real Markdown heading is a title
@pytest.mark.parametrize("content,title", [
("# Title\n\nBody", "Title"),
("## Sub title ##\n\nBody", "Sub title"),
("#indieweb is great", None),
("#\n\nbody text", None),
("####### seven", None),
("Plain note", None),
])
def test_split_title(content, title):
assert split_title(content)[0] == title
@pytest.mark.parametrize("content,kind", [
("#indieweb is great, posting from my phone", "note"),
("#\n\nbody text", "note"),
("# A Title\n\nThe body", "article"),
])
def test_post_type_of_hash_notes(app, client, content, kind):
note = make_note(app, content)
for path in (f'/note/{note.slug}', '/'):
parsed = parse(client, path)
entry = entry_of(parsed) if path != '/' else parsed['items'][0]['children'][0]
props = entry['properties']
if kind == 'note':
assert 'name' not in props
assert props['content'][0]['value'].strip()
else:
assert props['name'] == ['A Title']
# A5: the note's own syndicated copy is not a response
def test_own_syndicated_copy_rejected(app):
note = make_note(app, "Crossposted")
db = sqlite3.connect(app.config['DATABASE_PATH'])
db.execute("INSERT INTO syndications (note_id, target, remote_url) VALUES (?, 'gotosocial', ?)",
(note.id, 'https://social.example/@me/statuses/1'))
db.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) VALUES (?, ?, ?)",
('https://brid.gy/post/mastodon/@someone/1', f'{BASE}/note/{note.slug}', note.id))
db.commit()
result = {'valid': True, 'mention_type': 'mention', 'author_name': 'Me', 'author_url': 'https://me.example/',
'author_photo': None, 'content_snippet': 'x',
'source_post_url': 'https://social.example/@me/statuses/1', 'source_uid': None,
'source_published': None}
worker = WebmentionWorker(app, db_path=app.config['DATABASE_PATH'], site_url=BASE)
with patch("starpunk.webmention_worker.process_webmention", return_value=result):
worker._validate_webmention(1, 'https://brid.gy/post/mastodon/@someone/1', 'x', note.id)
status, error = db.execute("SELECT validation_status, validation_error FROM webmentions").fetchone()
assert status == 'invalid' and 'own syndicated copy' in error
# B1/B3: discovery links
def test_head_links_without_webmention(app, client):
app.config['WEBMENTION_ENABLED'] = False
rels = parse(client, '/')['rels']
assert 'webmention' not in rels
for rel in ('micropub', 'indieauth-metadata', 'authorization_endpoint', 'token_endpoint'):
assert rels.get(rel), rel
def test_home_has_representative_hcard(app, client):
# Fresh install: discovery fails, the fallback profile is used
parsed = parse(client, '/')
card = mf2_utils.representative_hcard(parsed, BASE + '/')
assert card and card['properties']['url'] == [app.config['ADMIN_ME']]
@pytest.mark.parametrize("path", ['/likes', '/tag/python'])
def test_list_feeds_have_url_and_author(app, client, path):
make_note(app, "Tagged", tags=["python"])
feed = parse(client, path)['items'][0]
assert feed['type'] == ['h-feed']
assert feed['properties']['url'] == [BASE + path]
assert feed['properties']['author'][0]['type'] == ['h-card']
# B2: representative h-card
def _cards(html, url='https://me.example/'):
return mf2py.parse(doc=html, url=url)
def test_representative_hcard_rules():
other = '<div class="h-card"><a class="p-name u-url" href="https://bob.example/">Bob</a></div>'
# someone else's h-card only: none
assert mf2_utils.representative_hcard(_cards(other), 'https://me.example/') is None
# uid + url
me = '<div class="h-card"><a class="p-name u-url u-uid" href="/">Me</a></div>'
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(other + me), 'https://me.example/'), 'name') == 'Me'
# rel=me
relme = ('<a rel="me" href="https://social.example/@me">x</a>'
'<div class="h-card"><a class="p-name u-url" href="https://social.example/@me">Me</a></div>')
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(other + relme), 'https://me.example/'), 'name') == 'Me'
# only one h-card, url = page
single = '<div class="h-card"><a class="p-name u-url" href="/">Me</a></div>'
assert mf2_utils.first(mf2_utils.representative_hcard(_cards(single), 'https://me.example/'), 'name') == 'Me'
# B4: the responding entry and its author
FEED_PAGE = '''
<div class="h-feed">
<div class="p-author h-card"><a class="p-name u-url" href="https://feed.example/">Feed Owner</a></div>
<div class="h-entry"><a class="u-url" href="https://feed.example/1">one</a>
<div class="e-content">unrelated</div></div>
<div class="h-entry"><a class="u-url" href="https://feed.example/2">two</a>
<time class="dt-published" datetime="2026-11-01T09:00:00-07:00">x</time>
<a class="u-in-reply-to" href="https://me.example/note/a">re</a>
<div class="e-content">my reply</div></div>
</div>'''
def test_responding_entry_and_feed_author():
parsed = mf2py.parse(doc=FEED_PAGE, url='https://feed.example/')
entry = mf2_utils.responding_entry(parsed, 'https://me.example/note/a#frag')
assert mf2_utils.first(entry, 'url') == 'https://feed.example/2'
assert mf2_utils.response_type(entry, 'https://me.example/note/a') == 'reply'
author = extract_author(parsed, 'https://feed.example/', entry)
assert author['author_name'] == 'Feed Owner'
assert author['author_url'] == 'https://feed.example/'
details = extract_source_details(entry)
assert details['source_post_url'] == 'https://feed.example/2'
assert details['source_published'] == '2026-11-01T16:00:00Z'
def test_author_as_url_or_name():
url_author = '<div class="h-entry"><a class="u-author" href="https://al.example/">al</a></div>'
parsed = mf2py.parse(doc=url_author, url='https://s.example/')
assert extract_author(parsed, 'https://s.example/')['author_url'] == 'https://al.example/'
name_author = '<div class="h-entry"><span class="p-author">Al Smith</span></div>'
parsed = mf2py.parse(doc=name_author, url='https://s.example/')
author = extract_author(parsed, 'https://s.example/')
assert author['author_name'] == 'Al Smith' and author['author_url'] == 'https://s.example/'
+77 -29
View File
@@ -167,6 +167,34 @@ def test_micropub_create_with_name(client, app, mock_valid_token):
assert response.status_code == 201
assert 'Location' in response.headers
# StarPunk titles are a '# Heading' first line, so name must land there
slug = response.headers['Location'].rstrip('/').split('/')[-1]
with app.app_context():
note = get_note(slug)
assert note.content == '# My Test Title\n\nContent goes here'
assert note.title == 'My Test Title'
def test_micropub_create_with_name_already_heading(client, app, mock_valid_token):
"""A name that matches the content's heading is not added twice"""
with patch('starpunk.routes.micropub.verify_access_token', mock_valid_token):
response = client.post(
'/micropub',
json={
'type': ['h-entry'],
'properties': {
'name': ['Same Title'],
'content': ['# Same Title\n\nBody']
}
},
headers={'Authorization': 'Bearer valid_token'}
)
assert response.status_code == 201
slug = response.headers['Location'].rstrip('/').split('/')[-1]
with app.app_context():
assert get_note(slug).content == '# Same Title\n\nBody'
def test_micropub_create_with_categories(client, app, mock_valid_token):
"""Test creating a note with tags (category property)"""
@@ -206,7 +234,7 @@ def test_micropub_create_with_custom_slug_form(client, app, mock_valid_token):
# Verify the custom slug was used
location = response.headers['Location']
assert location.endswith('/notes/my-custom-slug')
assert location.endswith('/note/my-custom-slug')
# Verify note exists with the custom slug
with app.app_context():
@@ -236,7 +264,7 @@ def test_micropub_create_with_custom_slug_json(client, app, mock_valid_token):
# Verify the custom slug was used
location = response.headers['Location']
assert location.endswith('/notes/json-custom-slug')
assert location.endswith('/note/json-custom-slug')
# Verify note exists with the custom slug
with app.app_context():
@@ -290,6 +318,38 @@ def test_micropub_query_source(client, mock_valid_token):
data = response.get_json()
assert 'properties' in data
assert data['properties']['content'][0] == 'Source test'
# url must be the real permalink (was /notes/<slug>, a 404)
assert data['properties']['url'] == [location]
# A note without a heading has no name
assert 'name' not in data['properties']
def test_micropub_query_source_with_name(client, mock_valid_token):
"""q=source returns name and content the way they were posted"""
with patch('starpunk.routes.micropub.verify_access_token', mock_valid_token):
create_response = client.post(
'/micropub',
json={
'type': ['h-entry'],
'properties': {
'name': ['Round Trip'],
'content': ['Body text']
}
},
headers={'Authorization': 'Bearer valid_token'}
)
location = create_response.headers['Location']
response = client.get(
f'/micropub?q=source&url={location}',
headers={'Authorization': 'Bearer valid_token'}
)
assert response.status_code == 200
properties = response.get_json()['properties']
assert properties['name'] == ['Round Trip']
assert properties['content'] == ['Body text']
assert properties['url'] == [location]
# Error Handling Tests
@@ -312,33 +372,21 @@ def test_micropub_missing_content(client, mock_valid_token):
assert data['error'] == 'invalid_request'
def test_micropub_unsupported_action(client, mock_valid_token):
"""Test unsupported actions (update, delete) return error"""
def test_micropub_update_delete_need_scope(client, mock_valid_token):
"""A create-only token can't update or delete (v2.4.0)"""
with patch('starpunk.routes.micropub.verify_access_token', mock_valid_token):
# Test update
response = client.post(
'/micropub',
json={
'action': 'update',
'url': 'https://example.com/note/123'
},
headers={'Authorization': 'Bearer valid_token'}
)
for action in ('update', 'delete', 'undelete'):
response = client.post(
'/micropub',
json={'action': action, 'url': 'https://example.com/note/123', 'replace': {}},
headers={'Authorization': 'Bearer valid_token'}
)
assert response.status_code == 403, action
assert response.get_json()['error'] == 'insufficient_scope'
def test_micropub_unknown_action(client, mock_valid_token):
with patch('starpunk.routes.micropub.verify_access_token', mock_valid_token):
response = client.post('/micropub', json={'action': 'frobnicate'},
headers={'Authorization': 'Bearer valid_token'})
assert response.status_code == 400
data = response.get_json()
assert 'not supported' in data['error_description']
# Test delete
response = client.post(
'/micropub',
json={
'action': 'delete',
'url': 'https://example.com/note/123'
},
headers={'Authorization': 'Bearer valid_token'}
)
assert response.status_code == 400
data = response.get_json()
assert 'not supported' in data['error_description']
+209
View File
@@ -0,0 +1,209 @@
"""
Micropub round 1 (v2.4.0): update, delete/undelete, drafts, replies,
multipart and remote photos, q=source lists and filters, q=category.
Uses real tokens from StarPunk's own IndieAuth server.
"""
import io
import httpx
import pytest
from PIL import Image
from starpunk import create_app
from starpunk.indieauth import create_access_token
from starpunk.notes import get_note
SITE = "https://blog.example.com"
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
return create_app({
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
})
def auth(app, scope="create update delete media"):
with app.test_request_context():
token = create_access_token("https://quill.example/", scope)["access_token"]
return {"Authorization": f"Bearer {token}"}
@pytest.fixture
def client(app):
return app.test_client()
@pytest.fixture
def h(app):
return auth(app)
def jpeg(color="red"):
buf = io.BytesIO()
Image.new("RGB", (64, 48), color).save(buf, format="JPEG")
return buf.getvalue()
def create(client, h, **props):
body = {"type": ["h-entry"], "properties": {"content": ["Hello"], **props}}
response = client.post("/micropub", json=body, headers=h)
assert response.status_code == 201, response.get_json()
return response.headers["Location"]
def note_at(app, url):
with app.app_context():
return get_note(slug=url.rstrip("/").rsplit("/", 1)[1])
def source(client, h, url, *props):
query = f"/micropub?q=source&url={url}" + "".join(f"&properties[]={p}" for p in props)
return client.get(query, headers=h).get_json()
class TestCreate:
def test_draft(self, app, client, h):
url = create(client, h, **{"post-status": ["draft"]})
assert note_at(app, url).published is False
assert client.get(url.replace(SITE, "")).status_code == 404
def test_reply(self, app, client, h):
url = create(client, h, **{"in-reply-to": ["https://other.example/post/1"]})
assert note_at(app, url).in_reply_to == "https://other.example/post/1"
def test_multipart_photo(self, app, client, h):
response = client.post(
"/micropub",
data={"h": "entry", "content": "With a file", "photo": (io.BytesIO(jpeg()), "pic.jpg")},
headers=h, content_type="multipart/form-data",
)
assert response.status_code == 201
assert "photo" in source(client, h, response.headers["Location"])["properties"]
def test_remote_photo_is_downloaded(self, app, client, h, monkeypatch):
# Fetched through the SSRF-checked, address-pinned fetcher (v2.9.0)
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT",
httpx.MockTransport(lambda req: httpx.Response(200, content=jpeg("blue"))))
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
lambda *a, **k: [(2, 1, 6, "", ("93.184.216.34", 0))])
url = create(client, h, photo=[{"value": "https://photos.example/cat.jpg", "alt": "A cat"}])
(photo,) = source(client, h, url)["properties"]["photo"]
assert photo["value"].startswith(f"{SITE}/media/") and photo["alt"] == "A cat"
def test_only_h_entry(self, client, h):
response = client.post("/micropub", data={"h": "event", "content": "x"}, headers=h)
assert response.status_code == 400
class TestUpdate:
def update(self, client, h, url, **ops):
return client.post("/micropub", json={"action": "update", "url": url, **ops}, headers=h)
def test_replace_content_keeps_title(self, app, client, h):
url = create(client, h, name=["My Title"])
assert self.update(client, h, url, replace={"content": ["New body"]}).status_code == 204
props = source(client, h, url)["properties"]
assert props["name"] == ["My Title"] and props["content"] == ["New body"]
def test_replace_and_delete_name(self, app, client, h):
url = create(client, h)
self.update(client, h, url, replace={"name": ["Added"]})
assert note_at(app, url).content.startswith("# Added")
self.update(client, h, url, delete=["name"])
assert "name" not in source(client, h, url)["properties"]
def test_categories(self, app, client, h):
url = create(client, h, category=["one", "two"])
self.update(client, h, url, add={"category": ["three", "One"]})
assert source(client, h, url)["properties"]["category"] == ["one", "two", "three"] # entered order (v2.6.4)
self.update(client, h, url, delete={"category": ["two"]})
assert source(client, h, url)["properties"]["category"] == ["one", "three"]
self.update(client, h, url, delete=["category"])
assert "category" not in source(client, h, url)["properties"]
def test_post_status_and_reply(self, app, client, h):
url = create(client, h)
self.update(client, h, url, replace={"post-status": ["draft"], "in-reply-to": ["https://x.example/a"]})
note = note_at(app, url)
assert note.published is False and note.in_reply_to == "https://x.example/a"
self.update(client, h, url, delete=["in-reply-to", "post-status"])
note = note_at(app, url)
assert note.published is True and not note.in_reply_to
def test_photos(self, app, client, h):
uploaded = client.post("/micropub/media", data={"file": (io.BytesIO(jpeg()), "a.jpg")},
headers=h, content_type="multipart/form-data").headers["Location"]
url = create(client, h)
self.update(client, h, url, add={"photo": [{"value": uploaded, "alt": "Red"}]})
assert source(client, h, url)["properties"]["photo"] == [{"value": uploaded, "alt": "Red"}]
self.update(client, h, url, delete={"photo": [uploaded]})
assert "photo" not in source(client, h, url)["properties"]
def test_unsupported_property(self, client, h):
url = create(client, h)
response = self.update(client, h, url, replace={"published": ["2020-01-01T00:00:00Z"]})
assert response.status_code == 400 and "published" in response.get_json()["error_description"]
def test_unknown_post(self, client, h):
response = self.update(client, h, f"{SITE}/note/nope", replace={"content": ["x"]})
assert response.status_code == 404
def test_form_update_rejected(self, client, h):
url = create(client, h)
assert client.post("/micropub", data={"action": "update", "url": url}, headers=h).status_code == 400
class TestDelete:
def test_delete_and_undelete(self, app, client, h):
url = create(client, h, category=["kept"])
path = url.replace(SITE, "")
assert client.post("/micropub", data={"action": "delete", "url": url}, headers=h).status_code == 204
assert client.get(path).status_code == 410 # Gone (Webmention 3.1.5)
response = client.post("/micropub", json={"action": "undelete", "url": url}, headers=h)
assert response.status_code == 204
assert client.get(path).status_code == 200
assert source(client, h, url)["properties"]["category"] == ["kept"]
def test_undelete_needs_a_deleted_post(self, client, h):
url = create(client, h)
assert client.post("/micropub", json={"action": "undelete", "url": url}, headers=h).status_code == 404
def test_delete_scope(self, app, client):
h = auth(app, "create")
url = create(client, h)
assert client.post("/micropub", data={"action": "delete", "url": url}, headers=h).status_code == 403
class TestQueries:
def test_source_properties_filter(self, client, h):
url = create(client, h, name=["T"], category=["x"])
data = source(client, h, url, "name", "category")
assert data == {"properties": {"name": ["T"], "category": ["x"]}}
def test_source_list_includes_drafts(self, client, h):
create(client, h, content=["first"])
create(client, h, content=["second"], **{"post-status": ["draft"]})
items = client.get("/micropub?q=source&limit=5", headers=h).get_json()["items"]
assert {i["properties"]["post-status"][0] for i in items} == {"published", "draft"}
def test_category(self, client, h):
create(client, h, category=["IndieWeb", "python"])
create(client, h, category=["IndieWeb"])
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["IndieWeb", "python"]
assert client.get("/micropub?q=category&filter=py", headers=h).get_json()["categories"] == ["python"]
def test_config_lists_queries(self, client, h):
config = client.get("/micropub?q=config", headers=h).get_json()
assert "category" in config["q"] and "media-endpoint" in config
def test_media_scope_alone_can_upload(app, client):
response = client.post("/micropub/media", data={"file": (io.BytesIO(jpeg()), "a.jpg")},
headers=auth(app, "media"), content_type="multipart/form-data")
assert response.status_code == 201
+105
View File
@@ -0,0 +1,105 @@
"""
Micropub media fixes (v2.6.5): photo posts without text, the same photo
twice, and failed creates that left a post (and a database lock) behind.
"""
import io
from starpunk.database.pool import ConnectionPool
from starpunk.notes import list_notes
from starpunk.syndication import compose_text
from tests.test_micropub_v240 import SITE, app, client, h, jpeg, note_at, source # noqa: F401
def upload(client, h, color="red"):
response = client.post("/micropub/media", data={"file": (io.BytesIO(jpeg(color)), "a.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 201
return response.headers["Location"]
def post(client, h, props):
return client.post("/micropub", json={"type": ["h-entry"], "properties": props}, headers=h)
def update(client, h, url, **ops):
return client.post("/micropub", json={"action": "update", "url": url, **ops}, headers=h)
class TestPhotoWithoutText:
def test_json_photo_only(self, app, client, h):
photo = upload(client, h)
response = post(client, h, {"photo": [{"value": photo, "alt": "Red"}]})
assert response.status_code == 201, response.get_json()
url = response.headers["Location"]
assert note_at(app, url).content == ""
assert source(client, h, url, "photo")["properties"]["photo"] == [{"value": photo, "alt": "Red"}]
page = client.get(url.replace(SITE, ""))
assert page.status_code == 200 and b"<title>Photo \xc2\xb7 " in page.data
def test_multipart_photo_only(self, client, h):
response = client.post("/micropub", data={"h": "entry", "photo": (io.BytesIO(jpeg()), "p.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 201, response.get_json()
def test_nothing_at_all_is_still_rejected(self, client, h):
assert post(client, h, {}).status_code == 400
# A photo that can't be found doesn't count
assert post(client, h, {"photo": [f"{SITE}/media/nope.jpg"]}).status_code == 400
def test_text_can_be_cleared_while_photos_remain(self, app, client, h):
photo = upload(client, h)
url = post(client, h, {"content": ["Hi"], "photo": [photo]}).headers["Location"]
assert update(client, h, url, replace={"content": [""]}).status_code == 204
assert note_at(app, url).content == ""
assert update(client, h, url, replace={"content": ["Back"]}).status_code == 204
assert update(client, h, url, delete=["content"]).status_code == 204
assert note_at(app, url).content == ""
# Without photos the text has to stay
response = update(client, h, url, delete=["content", "photo"])
assert response.status_code == 400
def test_text_is_required_without_photos(self, client, h):
url = post(client, h, {"content": ["Hi"]}).headers["Location"]
assert update(client, h, url, replace={"content": [""]}).status_code == 400
assert update(client, h, url, delete=["content"]).status_code == 400
def test_crosspost_text_starts_with_the_link(self, app, client, h):
url = post(client, h, {"photo": [upload(client, h)]}).headers["Location"]
with app.app_context():
note = note_at(app, url)
assert compose_text(note, url, ["cats"], 500) == f"{url}\n\n#cats"
class TestFailures:
def test_same_photo_twice_is_shown_once(self, client, h):
photo = upload(client, h)
response = post(client, h, {"content": ["Twice"], "photo": [photo, photo]})
assert response.status_code == 201, response.get_json()
assert source(client, h, response.headers["Location"], "photo")["properties"]["photo"] == [{"value": photo, "alt": ""}]
def test_failed_create_leaves_no_post_and_no_lock(self, app, client, h, monkeypatch):
import starpunk.syndication as syndication
def boom(*args, **kwargs):
raise RuntimeError("crosspost store down")
monkeypatch.setattr(syndication, "select_targets", boom)
response = post(client, h, {"content": ["Doomed"], "photo": [upload(client, h)]})
assert response.status_code == 500
with app.app_context():
assert list_notes(published_only=False) == []
monkeypatch.undo()
# The database is usable straight away
assert post(client, h, {"content": ["Next"]}).status_code == 201
def test_pool_rolls_back_open_transactions(self, tmp_path):
pool = ConnectionPool(str(tmp_path / "x.db"), pool_size=1, timeout=1)
conn = pool.get_connection()
conn.execute("CREATE TABLE t (x)")
conn.commit()
conn.execute("INSERT INTO t VALUES (1)")
assert conn.in_transaction
pool.return_connection(conn)
assert not conn.in_transaction
assert pool.get_connection().execute("SELECT count(*) FROM t").fetchone()[0] == 0
+249
View File
@@ -0,0 +1,249 @@
"""
Micropub compliance fixes (v2.9.0)
Section numbers refer to the W3C Micropub Recommendation.
"""
import io
import sqlite3
import httpx
import pytest
from tests.test_micropub_v240 import SITE, app, auth, client, create, h, jpeg, note_at, source # noqa: F401
def syndications(app, url):
note = note_at(app, url)
db = sqlite3.connect(app.config["DATABASE_PATH"])
return sorted(r[0] for r in db.execute("SELECT target FROM syndications WHERE note_id = ?", (note.id,)))
@pytest.fixture
def targets(app):
app.config.update(BLUESKY_HANDLE="me.example", BLUESKY_APP_PASSWORD="p")
from starpunk import syndication
syndication._client_cache.clear()
with app.app_context():
return {t.uid: name for name, t in syndication.get_targets().items()}
# Crossposting: mp-* in JSON properties; absent means none
def test_mp_syndicate_to_inside_json_properties(app, client, h, targets):
bsky_uid = next(uid for uid, name in targets.items() if name == "bluesky")
url = create(client, h, **{"mp-syndicate-to": [bsky_uid]})
assert syndications(app, url) == ["bluesky"]
@pytest.mark.parametrize("extra", [{}, {"mp-syndicate-to": []}])
def test_no_choice_means_no_crossposting(app, client, h, targets, extra):
url = create(client, h, **extra)
assert syndications(app, url) == []
def test_form_mp_syndicate_to(app, client, h, targets):
bsky_uid = next(uid for uid, name in targets.items() if name == "bluesky")
response = client.post("/micropub", data={"content": "x", "mp-syndicate-to[]": bsky_uid}, headers=h)
assert syndications(app, response.headers["Location"]) == ["bluesky"]
def test_mp_slug_inside_json_properties(app, client, h):
url = create(client, h, **{"mp-slug": ["my-slug"]})
assert url.endswith("/note/my-slug")
assert "mp-slug" not in source(client, h, url)["properties"]
# Scopes
def test_source_needs_a_posting_scope(app, client, h):
create(client, h, **{"post-status": ["draft"]})
profile_only = auth(app, "profile")
assert client.get("/micropub?q=source", headers=profile_only).status_code == 403
assert client.get("/micropub?q=category", headers=profile_only).status_code == 403
assert client.get("/micropub?q=config", headers=profile_only).status_code == 200
def test_draft_scope_only_creates_drafts(app, client):
url = create(client, auth(app, "draft"))
assert note_at(app, url).published is False
# Visibility
@pytest.mark.parametrize("visibility,published", [("public", True), ("unlisted", False), ("private", False)])
def test_visibility(app, client, h, visibility, published):
url = create(client, h, visibility=[visibility])
assert note_at(app, url).published is published
def test_unknown_visibility_rejected(client, h):
body = {"type": ["h-entry"], "properties": {"content": ["x"], "visibility": ["friends"]}}
assert client.post("/micropub", json=body, headers=h).status_code == 400
# Photos that can't be used fail the request
def test_unreachable_photo_is_an_error(app, client, h, monkeypatch):
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(lambda r: httpx.Response(404)))
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
lambda *a, **k: [(2, 1, 6, "", ("93.184.216.34", 0))])
body = {"type": ["h-entry"], "properties": {"content": ["x"], "photo": ["https://p.example/404.jpg"]}}
assert client.post("/micropub", json=body, headers=h).status_code == 400
assert client.get("/micropub?q=source", headers=h).get_json()["items"] == []
def test_unknown_local_media_is_an_error_on_update(app, client, h):
url = create(client, h)
response = client.post("/micropub", json={"action": "update", "url": url,
"replace": {"photo": [f"{SITE}/media/2026/11/nope.jpg"]}}, headers=h)
assert response.status_code == 400
def test_redirect_to_private_address_is_not_fetched(app, client, h, monkeypatch):
def resolve(host, *a, **k):
return [(2, 1, 6, "", ("10.0.0.5" if host == "internal.example" else "93.184.216.34", 0))]
def site(request):
return httpx.Response(302, headers={"Location": "http://internal.example/secret.jpg"})
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(site))
body = {"type": ["h-entry"], "properties": {"content": ["x"], "photo": ["https://p.example/a.jpg"]}}
assert client.post("/micropub", json=body, headers=h).status_code == 400
# JSON values must be arrays (3.3.2)
def test_scalar_json_values_are_not_split(app, client, h):
body = {"type": ["h-entry"], "properties": {"content": "whole text", "category": "solo", "name": "Title"}}
response = client.post("/micropub", json=body, headers=h)
props = source(client, h, response.headers["Location"])["properties"]
assert props["content"] == ["whole text"]
assert props["category"] == ["solo"]
assert props["name"] == ["Title"]
def test_update_delete_values_must_be_arrays(app, client, h):
url = create(client, h, category=["a", "b"])
response = client.post("/micropub", json={"action": "update", "url": url, "delete": {"category": "a"}}, headers=h)
assert response.status_code == 400
def test_update_needs_an_operation(app, client, h):
url = create(client, h)
assert client.post("/micropub", json={"action": "update", "url": url}, headers=h).status_code == 400
def test_person_tag_categories_are_ignored(app, client, h):
url = create(client, h, category=["kept", {"type": ["h-card"], "properties": {"url": ["https://a.example/"]}}])
assert source(client, h, url)["properties"]["category"] == ["kept"]
# Properties that used to be dropped
def test_bookmark_of(app, client, h):
body = {"type": ["h-entry"], "properties": {"bookmark-of": ["https://read.example/post"]}}
url = client.post("/micropub", json=body, headers=h).headers["Location"]
note = note_at(app, url)
assert note.kind == "bookmark" and note.bookmark_of == "https://read.example/post"
props = source(client, h, url)["properties"]
assert props["bookmark-of"] == ["https://read.example/post"] and "content" not in props
page = client.get(url.replace(SITE, "")).data.decode()
assert 'class="u-bookmark-of" href="https://read.example/post"' in page
def test_summary_or_name_without_content(app, client, h):
response = client.post("/micropub", json={"type": ["h-entry"], "properties": {"summary": ["Just a summary"]}},
headers=h)
assert source(client, h, response.headers["Location"])["properties"]["content"] == ["Just a summary"]
response = client.post("/micropub", json={"type": ["h-entry"], "properties": {"name": ["Only a title"]}},
headers=h)
assert source(client, h, response.headers["Location"])["properties"]["name"] == ["Only a title"]
@pytest.mark.parametrize("prop", ["video", "audio"])
def test_unsupported_media_is_refused(client, h, prop):
body = {"type": ["h-entry"], "properties": {"content": ["x"], prop: ["https://m.example/clip"]}}
assert client.post("/micropub", json=body, headers=h).status_code == 400
upload = {"content": "x", prop: (io.BytesIO(b"data"), "clip.mp4")}
assert client.post("/micropub", data=upload, headers=h, content_type="multipart/form-data").status_code == 400
# HTML content (3.7.2) and sanitizing
def test_html_content_comes_back_as_html(app, client, h):
url = create(client, h, content=[{"html": "<p>Some <b>bold</b> text</p>"}])
assert source(client, h, url)["properties"]["content"] == [{"html": "<p>Some <b>bold</b> text</p>"}]
url = create(client, h, content=["plain *markdown*"])
assert source(client, h, url)["properties"]["content"] == ["plain *markdown*"]
def test_update_with_html_content(app, client, h):
url = create(client, h)
client.post("/micropub", json={"action": "update", "url": url,
"replace": {"content": [{"html": "<p>new</p>"}]}}, headers=h)
assert source(client, h, url)["properties"]["content"] == [{"html": "<p>new</p>"}]
def test_html_is_sanitized_on_the_page(app, client, h):
url = create(client, h, content=[{"html": '<p>hi<script>alert(1)</script><img src="/x.jpg" onerror="evil()">'
'<a href="javascript:alert(2)">j</a><iframe src="https://e.example"></iframe></p>'}])
page = client.get(url.replace(SITE, "")).data.decode()
for bad in ("<script>alert(1)", "onerror", "javascript:alert", "<iframe"):
assert bad not in page
assert '<img src="/x.jpg">' in page
# published (3.3)
def test_published_is_stored_as_utc(app, client, h):
url = create(client, h, published=["2020-02-03T04:05:06-08:00"])
assert source(client, h, url)["properties"]["published"] == ["2020-02-03T12:05:06Z"]
created = note_at(app, url).created_at
assert created.tzinfo is None and (created.hour, created.minute) == (12, 5)
def test_unparseable_published_is_an_error(client, h):
body = {"type": ["h-entry"], "properties": {"content": ["x"], "published": ["yesterday"]}}
assert client.post("/micropub", json=body, headers=h).status_code == 400
# Errors (3.8) and authentication (5)
@pytest.mark.parametrize("payload", ["{not json", "[1, 2]"])
def test_malformed_json_is_400(client, h, payload):
response = client.post("/micropub", data=payload, headers={**h, "Content-Type": "application/json"})
assert response.status_code == 400
assert response.get_json()["error"] == "invalid_request"
def test_bearer_scheme_is_case_insensitive(client, h):
lower = {"Authorization": h["Authorization"].replace("Bearer", "bearer")}
assert client.get("/micropub?q=config", headers=lower).status_code == 200
def test_token_in_header_and_body_is_400(client, h):
response = client.post("/micropub", data={"content": "x", "access_token": "other"}, headers=h)
assert response.status_code == 400
def test_401_has_www_authenticate(client):
response = client.get("/micropub?q=config")
assert response.status_code == 401
assert response.headers["WWW-Authenticate"].startswith("Bearer")
response = client.get("/micropub?q=config", headers={"Authorization": "Bearer nope"})
assert 'error="invalid_token"' in response.headers["WWW-Authenticate"]
def test_media_error_does_not_leak_internals(client, h):
response = client.post("/micropub/media", data={"file": (io.BytesIO(b"not an image"), "a.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 400
assert "BytesIO" not in response.get_json()["error_description"]
def test_config_lists_bookmarks(client, h):
types = {t["type"] for t in client.get("/micropub?q=config", headers=h).get_json()["post-types"]}
assert "bookmark" in types
+86
View File
@@ -0,0 +1,86 @@
"""
Migrations under concurrent gunicorn workers (v2.8.1)
The v2.8.0 deploy failed to boot: one worker applied migration 014 while
another applied it again ("duplicate column name"), because
conn.executescript() committed and released the migration lock early.
"""
import multiprocessing
import sqlite3
from pathlib import Path
from starpunk.database.schema import INITIAL_SCHEMA_SQL
from starpunk.migrations import execute_in_transaction, run_migrations, split_sql_statements
MIGRATIONS = Path(__file__).parent.parent / "migrations"
def test_script_runs_inside_the_transaction(tmp_path):
conn = sqlite3.connect(tmp_path / "t.db")
conn.execute("CREATE TABLE t (a INTEGER)")
conn.commit()
conn.execute("BEGIN IMMEDIATE")
execute_in_transaction(conn, "ALTER TABLE t ADD COLUMN b TEXT;\nALTER TABLE t ADD COLUMN c TEXT;")
conn.rollback()
assert [row[1] for row in conn.execute("PRAGMA table_info(t)")] == ["a"]
def test_split_keeps_trigger_bodies_together():
script = """
-- comment; with a semicolon
CREATE TABLE t (a TEXT DEFAULT 'x;y');
CREATE TRIGGER tr AFTER DELETE ON t BEGIN
DELETE FROM t WHERE a = old.a;
SELECT 1;
END;
"""
statements = split_sql_statements(script)
assert len(statements) == 2
assert statements[1].startswith("CREATE TRIGGER") and statements[1].endswith("END;")
def _migrate(db_path, barrier, errors):
try:
barrier.wait()
run_migrations(db_path)
except Exception as e: # report to the parent
errors.put(repr(e))
import pytest
@pytest.mark.parametrize("trial", range(5)) # the race hit ~1 in 5 runs
def test_concurrent_workers_apply_each_migration_once(tmp_path, trial):
# A database at migration 013, like production before v2.8.0
db_path = tmp_path / "starpunk.db"
conn = sqlite3.connect(db_path)
conn.executescript(INITIAL_SCHEMA_SQL) # as init_db does
conn.close()
run_migrations(db_path)
conn = sqlite3.connect(db_path)
conn.execute("DELETE FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'")
conn.commit()
columns = [row[1] for row in conn.execute("PRAGMA table_info(webmentions)")]
if "source_post_url" in columns:
conn.execute("ALTER TABLE webmentions DROP COLUMN source_post_url")
conn.execute("ALTER TABLE webmentions DROP COLUMN source_published")
conn.commit()
conn.close()
ctx = multiprocessing.get_context("fork")
barrier, errors = ctx.Barrier(4), ctx.Queue()
workers = [ctx.Process(target=_migrate, args=(db_path, barrier, errors)) for _ in range(4)]
for w in workers:
w.start()
for w in workers:
w.join(timeout=60)
assert errors.empty(), errors.get()
conn = sqlite3.connect(db_path)
assert conn.execute(
"SELECT COUNT(*) FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'"
).fetchone()[0] == 1
+4 -3
View File
@@ -43,7 +43,7 @@ class TestRetryLogic:
from starpunk.database import init_db
from starpunk import create_app
app = create_app({'DATABASE_PATH': str(temp_db)})
app = create_app({'DATABASE_PATH': str(temp_db), 'TESTING': True})
init_db(app)
# Verify migrations table exists and has records
@@ -302,12 +302,13 @@ class TestErrorHandling:
def test_rollback_failure_causes_system_exit(self, temp_db):
"""Test that rollback failure raises SystemExit"""
with patch('sqlite3.connect') as mock_connect:
with patch('sqlite3.connect') as mock_connect, \
patch('starpunk.migrations.execute_in_transaction',
side_effect=Exception("Migration failed")):
mock_conn = MagicMock()
mock_connect.return_value = mock_conn
# Make both migration and rollback fail
mock_conn.executescript.side_effect = Exception("Migration failed")
mock_conn.rollback.side_effect = Exception("Rollback failed")
mock_conn.execute.return_value.fetchone.return_value = (0,)
+38 -1
View File
@@ -11,7 +11,7 @@ Organized by model:
import pytest
from datetime import datetime, timedelta
from pathlib import Path
from starpunk.models import Note, Session, Token, AuthState
from starpunk.models import Note, Session, Token, AuthState, split_title
from starpunk.utils import calculate_content_hash
@@ -245,6 +245,43 @@ class TestNoteModel:
assert note.excerpt == "my-note"
def _note_with(self, tmp_path, content):
note_file = tmp_path / "notes" / "2024" / "11" / "test.md"
note_file.parent.mkdir(parents=True, exist_ok=True)
note_file.write_text(content)
return Note(
id=1,
slug="test",
file_path="notes/2024/11/test.md",
published=True,
created_at=datetime.utcnow(),
updated_at=datetime.utcnow(),
_data_dir=tmp_path,
)
def test_split_title(self):
"""A '# Heading' first line is split off as the title"""
assert split_title("# My Title\n\nBody") == ("My Title", "Body")
assert split_title("\n## Sub\nBody") == ("Sub", "Body")
assert split_title("No heading\n# later") == (None, "No heading\n# later")
assert split_title("#\nBody") == (None, "#\nBody")
def test_body_html_leaves_out_title(self, tmp_path):
"""Pages show the title themselves, so the body must not repeat it"""
note = self._note_with(tmp_path, "# My Title\n\nBody text.")
assert "My Title" not in note.body_html
assert "<p>Body text.</p>" in note.body_html
assert "<h1>My Title</h1>" in note.html # html is unchanged
def test_body_html_untitled_matches_html(self, tmp_path):
note = self._note_with(tmp_path, "Plain note.")
assert note.body_html == note.html
def test_excerpt_leaves_out_title(self, tmp_path):
"""Previews show the title above the excerpt, so it is not repeated"""
note = self._note_with(tmp_path, "# Title\n\nShort content.")
assert note.excerpt == "Short content."
def test_permalink(self):
"""Test permalink generation"""
note = Note(
+98
View File
@@ -0,0 +1,98 @@
"""
Rendered posts read as the right post type (v2.6.5).
Parses note pages and the home page with mf2py, then runs W3C Post Type
Discovery (https://www.w3.org/TR/post-type-discovery/) on what a consumer
sees. A photo post is any post with a photo, unless it's first a repost,
like or reply.
"""
import pytest
import mf2py
import starpunk.reply_context as reply_context
from tests.test_micropub_v240 import SITE, app, client, h, jpeg # noqa: F401
from tests.test_micropub_v265 import upload
def post_type(props):
"""W3C Post Type Discovery, section 5 (no events or RSVPs here)."""
for prop, kind in [("repost-of", "repost"), ("like-of", "like"), ("in-reply-to", "reply"),
("video", "video"), ("photo", "photo")]:
if props.get(prop):
return kind
def first(prop):
for v in props.get(prop, []):
v = v.get("value", "") if isinstance(v, dict) else v
if v.strip():
return " ".join(v.split())
return ""
content = first("content") or first("summary")
name = first("name")
if not content or not name or content.startswith(name):
return "note"
return "article"
def entries(items):
for item in items:
if "h-entry" in item.get("type", []):
yield item
yield from entries(item.get("children", []))
def value(v):
"""A u-* value, or the url of a nested microformat (h-cite)."""
return v.get("value") or v.get("properties", {}).get("url", [None])[0] if isinstance(v, dict) else v
CASES = {
"note": ({"content": ["Just a note"]}, "note"),
"article": ({"name": ["A Title"], "content": ["Body text here"]}, "article"),
"photo without text": ({"photo": ["PHOTO"]}, "photo"),
"photo with caption": ({"content": ["Sunset"], "photo": [{"value": "PHOTO", "alt": "Orange sky"}]}, "photo"),
"photo with title": ({"name": ["Trip"], "content": ["Day one"], "photo": ["PHOTO"]}, "photo"),
"reply": ({"content": ["Agreed"], "in-reply-to": ["https://other.example/post/1"]}, "reply"),
"reply with photo": ({"content": ["Mine"], "in-reply-to": ["https://other.example/post/2"],
"photo": ["PHOTO"]}, "reply"),
"repost": ({"repost-of": ["https://other.example/post/4"]}, "repost"),
}
@pytest.mark.parametrize("case", CASES)
def test_post_type(app, client, h, monkeypatch, case):
monkeypatch.setattr(reply_context, "fetch_reply_context", lambda *a, **k: None, raising=False)
props, want = CASES[case]
photo = upload(client, h) if "PHOTO" in str(props) else None
props = {k: [photo if v == "PHOTO" else ({**v, "value": photo} if isinstance(v, dict) else v)
for v in vs] for k, vs in props.items()}
response = client.post("/micropub", json={"type": ["h-entry"], "properties": props}, headers=h)
assert response.status_code == 201, response.get_json()
url = response.headers["Location"]
(entry,) = entries(mf2py.parse(doc=client.get(url.replace(SITE, "")).data.decode(), url=url)["items"])
home = mf2py.parse(doc=client.get("/").data.decode(), url=f"{SITE}/")
(listed,) = [e for e in entries(home["items"]) if e["properties"].get("url") == [url]]
for e in (entry, listed):
assert post_type(e["properties"]) == want
if want == "reply":
assert value(e["properties"]["in-reply-to"][0]) == props["in-reply-to"][0]
if want == "photo" and "name" not in props:
assert "name" not in e["properties"], "no implied name to show as a caption"
def test_photo_alt_text(app, client, h):
"""Alt text is the caption, else the post's text, never made up (v2.6.5)."""
def alts(props):
url = client.post("/micropub", json={"type": ["h-entry"], "properties": props},
headers=h).headers["Location"]
(entry,) = entries(mf2py.parse(doc=client.get(url.replace(SITE, "")).data.decode(), url=url)["items"])
return [p["alt"] for p in entry["properties"]["photo"]]
assert alts({"content": ["Beach day"], "photo": [{"value": upload(client, h), "alt": "Waves"}]}) == ["Waves"]
assert alts({"content": ["Beach day"], "photo": [upload(client, h, "blue")]}) == ["Beach day"]
assert alts({"name": ["Trip"], "content": ["Day one"], "photo": [upload(client, h, "green")]}) == ["Day one"]
assert alts({"photo": [upload(client, h, "gray")]}) == ["Image"]
+69
View File
@@ -0,0 +1,69 @@
"""Public page markup from the v2.3.0 style pass"""
import markdown
import pytest
from starpunk import create_app
from starpunk.models import MARKDOWN_EXTENSIONS
from starpunk.notes import create_note
def md(text):
return markdown.markdown(text, extensions=MARKDOWN_EXTENSIONS)
@pytest.mark.parametrize("text, expected", [
("see https://sqlite.org/x.html.", '<a href="https://sqlite.org/x.html">https://sqlite.org/x.html</a>.'),
("(https://a.example/p)", '(<a href="https://a.example/p">https://a.example/p</a>)'),
("[label](https://a.example)", '<a href="https://a.example">label</a>'),
("`https://code.example`", "<code>https://code.example</code>"),
])
def test_bare_urls_are_linked(text, expected):
assert expected in md(text)
def test_url_as_link_label_is_not_nested():
assert md("[https://b.example](https://b.example)").count("<a ") == 1
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
return create_app({
"TESTING": True, "SITE_URL": "https://blog.example.com", "SITE_NAME": "My Notes",
"ADMIN_ME": "https://example.com", "SESSION_SECRET": "x",
"DATA_PATH": data, "NOTES_PATH": data / "notes", "DATABASE_PATH": data / "starpunk.db",
})
def test_home_shows_full_notes_and_site_name(app):
long_text = "word " * 120 + "THE-END"
with app.app_context():
create_note(content=long_text, published=True, tags=["thoughts"])
html = app.test_client().get("/").data.decode()
assert "THE-END" in html # not cut at 300 characters
assert '<a class="site-title" href="/">My Notes</a>' in html
assert 'class="p-category" rel="tag"' in html
# Notes' metadata is not styled as the page footer
assert html.count('<footer class="site-footer">') == 1
assert '<footer class="note-meta">' in html
nav = html.split('class="site-nav"', 1)[1].split("</nav>", 1)[0]
assert 'href="/auth/login"' in nav # login is in the top bar
def test_note_page_meta_line(app):
with app.app_context():
note = create_note(content="Hello", published=True)
html = app.test_client().get(f"/note/{note.slug}").data.decode()
assert "<title>Hello · My Notes</title>" in html
assert 'class="u-url u-uid"' in html
def test_nav_shows_admin_when_logged_in(app):
from starpunk.auth import create_session
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://example.com"))
nav = client.get("/").data.decode().split('class="site-nav"', 1)[1].split("</nav>", 1)[0]
assert ">Admin</a>" in nav
assert "/auth/login" not in nav
+2 -1
View File
@@ -274,7 +274,8 @@ class TestVersionDisplay:
"""Test version number appears in footer"""
response = client.get("/")
assert response.status_code == 200
assert b"0.5.0" in response.data or b"StarPunk v" in response.data
from starpunk import __version__
assert f"v{__version__}".encode() in response.data
# OAuth metadata endpoint tests removed in Phase 1 of IndieAuth server removal
+2 -2
View File
@@ -220,8 +220,8 @@ def test_search_api_generates_correct_urls(client, test_notes):
if data["count"] > 0:
result = data["results"][0]
assert result["url"].startswith("/notes/")
assert result["url"] == f"/notes/{result['slug']}"
assert result["url"].startswith("/note/")
assert result["url"] == f"/note/{result['slug']}"
def test_search_api_provides_fallback_title(client, app):
+2 -2
View File
@@ -185,8 +185,8 @@ def test_search_page_links_to_notes(client, test_notes):
response = client.get("/search?q=test")
assert response.status_code == 200
# Should contain links to /notes/
assert b'href="/notes/' in response.data
# Should contain links to /note/
assert b'href="/note/' in response.data
def test_search_form_validation(client):
+397
View File
@@ -0,0 +1,397 @@
"""
Tests for crossposting to GoToSocial (starpunk/syndication.py, v2.1.0)
A fake Mastodon-API server (httpx.MockTransport) records what StarPunk sends.
"""
import io
import json
import httpx
import pytest
from PIL import Image
from starpunk import create_app, syndication
from starpunk.auth import create_session
from starpunk.indieauth import create_access_token
from starpunk.notes import create_note, delete_note, update_note
from starpunk.syndication import (
MastodonClient,
compose_text,
get_note_syndications,
note_changed,
process_due,
select_targets,
)
GTS = "https://social.example.com"
class FakeServer:
def __init__(self):
self.statuses = {}
self.calls = []
self.media = 0
self.uploads = [] # raw multipart bodies
self.fail_next = None # status code to return once
def __call__(self, request: httpx.Request) -> httpx.Response:
self.calls.append((request.method, request.url.path))
assert request.headers["Authorization"] == "Bearer secret-token"
if self.fail_next:
code, self.fail_next = self.fail_next, None
return httpx.Response(code, json={"error": "nope"})
path = request.url.path
if path == "/api/v2/media":
self.media += 1
self.uploads.append(request.read())
return httpx.Response(200, json={"id": f"m{self.media}"})
if path == "/api/v1/statuses" and request.method == "POST":
body = json.loads(request.content)
sid = str(len(self.statuses) + 1)
self.statuses[sid] = body
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
if path.startswith("/api/v1/statuses/"):
sid = path.rsplit("/", 1)[1]
if sid not in self.statuses:
return httpx.Response(404, json={"error": "not found"})
if request.method == "PUT":
self.statuses[sid] = json.loads(request.content)
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
if request.method == "DELETE":
del self.statuses[sid]
return httpx.Response(200, json={"id": sid})
return httpx.Response(500)
@pytest.fixture
def server():
return FakeServer()
@pytest.fixture
def app(tmp_path, server):
data = tmp_path / "data"
app = create_app(
{
"TESTING": True,
"SITE_URL": "https://blog.example.com",
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"GOTOSOCIAL_URL": GTS,
"GOTOSOCIAL_ACCESS_TOKEN": "secret-token",
}
)
syndication._client_cache[GTS] = MastodonClient(
GTS, "secret-token", http=httpx.Client(transport=httpx.MockTransport(server))
)
yield app
syndication._client_cache.clear()
def rows(app, note_id):
with app.app_context():
return get_note_syndications(note_id)
def publish(app, content="Hello fediverse", tags=None, targets=("gotosocial",)):
with app.test_request_context():
note = create_note(content=content, published=True, tags=tags)
select_targets(note.id, list(targets))
process_due()
return note
class TestCompose:
def test_full_note_with_link_and_hashtags(self, app):
with app.app_context():
note = create_note(content="# Title\n\nSome *markdown*", published=True)
text = compose_text(note, "https://blog.example.com/note/x", ["IndieWeb", "Self Hosting"], 5000)
assert text == "# Title\n\nSome *markdown*\n\nhttps://blog.example.com/note/x\n\n#IndieWeb #SelfHosting"
def test_long_note_is_shortened_but_keeps_link(self, app):
with app.app_context():
note = create_note(content="word " * 2000, published=True)
text = compose_text(note, "https://blog.example.com/note/x", [], 500)
assert len(text) <= 500
assert text.endswith("https://blog.example.com/note/x")
assert "…" in text
class TestLifecycle:
def test_publish_creates_status(self, app, server):
note = publish(app, tags=["Test"])
(row,) = rows(app, note.id)
assert row["state"] == "synced"
assert row["remote_url"] == f"{GTS}/@phil/statuses/1"
body = server.statuses["1"]
assert body["content_type"] == "text/markdown"
assert body["visibility"] == "public"
assert f"https://blog.example.com/note/{note.slug}" in body["status"]
assert "#Test" in body["status"]
def test_edit_updates_status(self, app, server):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, content="Edited text")
process_due()
assert server.statuses["1"]["status"].startswith("Edited text")
assert ("PUT", "/api/v1/statuses/1") in server.calls
def test_delete_deletes_status(self, app, server):
note = publish(app)
with app.test_request_context():
delete_note(id=note.id)
process_due()
assert server.statuses == {}
(row,) = rows(app, note.id)
assert row["remote_id"] is None and row["state"] == "synced"
def test_hard_delete_deletes_status(self, app, server):
note = publish(app)
with app.test_request_context():
delete_note(id=note.id, soft=False)
process_due()
assert server.statuses == {}
def test_unpublish_then_republish(self, app, server):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, published=False)
process_due()
assert server.statuses == {}
update_note(id=note.id, published=True)
process_due()
assert len(server.statuses) == 1
def test_opt_out_sends_nothing(self, app, server):
note = publish(app, targets=())
assert server.calls == []
assert rows(app, note.id) == []
def test_unticking_later_deletes_status(self, app, server):
note = publish(app)
with app.test_request_context():
select_targets(note.id, [])
process_due()
assert server.statuses == {}
def test_drafts_are_not_sent(self, app, server):
with app.test_request_context():
note = create_note(content="draft", published=False)
select_targets(note.id, ["gotosocial"])
process_due()
assert server.statuses == {}
def test_remote_deleted_by_hand_is_reposted_on_edit(self, app, server):
note = publish(app)
server.statuses.clear()
with app.test_request_context():
note_changed(note.id)
process_due()
assert len(server.statuses) == 1
def test_edit_during_send_is_not_lost(self, app, server):
with app.test_request_context():
note = create_note(content="v1", published=True)
select_targets(note.id, ["gotosocial"])
# Simulate an edit arriving while the first send is in flight
original = syndication._reconcile
def edit_mid_send(row, target):
result = original(row, target)
update_note(id=note.id, content="v2")
return result
syndication._reconcile = edit_mid_send
try:
process_due()
finally:
syndication._reconcile = original
assert rows(app, note.id)[0]["state"] == "pending"
process_due()
assert server.statuses["1"]["status"].startswith("v2")
class TestFailures:
def test_server_error_retries_later(self, app, server):
server.fail_next = 503
note = publish(app)
(row,) = rows(app, note.id)
assert row["state"] == "pending"
assert row["attempts"] == 1
assert "503" in row["last_error"]
def test_client_error_fails_permanently(self, app, server):
server.fail_next = 422
note = publish(app)
(row,) = rows(app, note.id)
assert row["state"] == "failed"
def test_retry_button(self, app, server):
server.fail_next = 422
note = publish(app)
(row,) = rows(app, note.id)
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://example.com"))
client.post(f"/admin/syndication/{row['id']}/retry")
with app.test_request_context():
process_due()
assert rows(app, note.id)[0]["state"] == "synced"
def test_no_targets_configured_does_nothing(self, tmp_path):
data = tmp_path / "d"
app = create_app({
"TESTING": True, "SITE_URL": "https://b.example", "ADMIN_ME": "https://e.example",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "s.db",
})
with app.test_request_context():
note = create_note(content="x", published=True)
select_targets(note.id, ["gotosocial"])
assert process_due() == 0
assert get_note_syndications(note.id) == []
class TestMedia:
def test_images_are_uploaded_with_alt_text(self, app, server):
from starpunk.media import attach_media_to_note, save_media
buf = io.BytesIO()
Image.new("RGB", (800, 600), "red").save(buf, format="JPEG")
with app.test_request_context():
note = create_note(content="With a picture", published=True)
media = save_media(buf.getvalue(), "red.jpg")
attach_media_to_note(note.id, [media["id"]], ["A red square"])
select_targets(note.id, ["gotosocial"])
process_due()
assert ("POST", "/api/v2/media") in server.calls
assert server.statuses["1"]["media_ids"] == ["m1"]
class TestInlineImages:
"""Clients like OutPost upload a photo and embed it in the content (v2.6.6)"""
def _upload(self, app):
from starpunk.media import save_media
buf = io.BytesIO()
Image.new("RGB", (800, 600), "orange").save(buf, format="JPEG")
with app.test_request_context():
return save_media(buf.getvalue(), "cat.jpg")
def test_embedded_own_image_is_attached(self, app, server):
media = self._upload(app)
url = f"https://blog.example.com/media/{media['path']}"
publish(app, content=f"![Tortie cat on a chair]({url})")
status = server.statuses["1"]
assert status["media_ids"] == ["m1"]
assert b"Tortie cat on a chair" in server.uploads[0]
assert "![" not in status["status"] and url not in status["status"]
assert status["status"].startswith("https://blog.example.com/note/")
def test_text_around_image_is_kept(self, app, server):
media = self._upload(app)
publish(app, content=f"Look\n\n<img src=\"/media/{media['path']}\" alt=\"cat\">\n\nNice")
status = server.statuses["1"]
assert status["media_ids"] == ["m1"]
assert status["status"].startswith("Look\n\nNice")
def test_image_both_attached_and_embedded_is_sent_once(self, app, server):
from starpunk.media import attach_media_to_note
media = self._upload(app)
with app.test_request_context():
note = create_note(content=f"![x](/media/{media['path']})", published=True)
attach_media_to_note(note.id, [media["id"]], ["caption"])
select_targets(note.id, ["gotosocial"])
process_due()
assert server.statuses["1"]["media_ids"] == ["m1"]
def test_other_sites_images_are_ignored(self, app, server):
publish(app, content="![x](https://elsewhere.example/cat.jpg) hi")
assert server.statuses["1"]["media_ids"] == []
assert server.uploads == []
class TestEntryPoints:
@pytest.fixture
def admin(self, app):
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://example.com"))
return client
def test_new_note_form_ticks_target_by_default(self, admin):
html = admin.get("/admin/new").data.decode()
assert 'name="syndicate_to" value="gotosocial" checked' in html
def test_admin_create_queues(self, app, admin):
admin.post("/admin/new", data={
"content": "From the editor", "published": "on",
"syndication_form": "1", "syndicate_to": "gotosocial",
})
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 1
def test_admin_create_opt_out(self, app, admin):
admin.post("/admin/new", data={"content": "Not this one", "published": "on", "syndication_form": "1"})
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 0
def _token(self, app):
with app.test_request_context():
return create_access_token("https://quill.example/", "create")["access_token"]
def test_micropub_lists_target(self, app):
token = self._token(app)
data = app.test_client().get(
"/micropub?q=syndicate-to", headers={"Authorization": f"Bearer {token}"}
).get_json()
assert data["syndicate-to"] == [{"uid": f"{GTS}/", "name": "GoToSocial"}]
def test_micropub_default_is_no_targets(self, app):
"""Absent mp-syndicate-to means no crossposting (v2.9.0)"""
token = self._token(app)
response = app.test_client().post(
"/micropub", data={"h": "entry", "content": "via quill"},
headers={"Authorization": f"Bearer {token}"},
)
assert response.status_code == 201
assert "/note/" in response.headers["Location"]
assert response.headers["Location"].startswith("https://blog.example.com/note/")
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 0
def test_micropub_explicit_empty_choice(self, app):
token = self._token(app)
app.test_client().post(
"/micropub", json={"type": ["h-entry"], "properties": {"content": ["no crosspost"]},
"mp-syndicate-to": []},
headers={"Authorization": f"Bearer {token}"},
)
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 0
def test_note_page_links_to_crosspost(self, app, server):
note = publish(app)
html = app.test_client().get(f"/note/{note.slug}").data.decode()
assert f'class="u-syndication" href="{GTS}/@phil/statuses/1"' in html
def test_edit_page_shows_status_and_separate_delete_form(self, app, admin, server):
note = publish(app)
html = admin.get(f"/admin/edit/{note.id}").data.decode()
assert 'value="gotosocial" checked' in html
assert f"{GTS}/@phil/statuses/1" in html
# Delete button belongs to its own form, not the edit form
assert 'form="delete-note-form"' in html
edit_form = html.split('class="note-form"', 1)[1].split("</form>", 1)[0]
assert "<form" not in edit_form
+392
View File
@@ -0,0 +1,392 @@
"""
Tests for crossposting to Bluesky (starpunk/syndication.py, v2.2.0)
A fake PDS (httpx.MockTransport) records what StarPunk sends.
"""
import io
import json
import httpx
import pytest
from PIL import Image
from starpunk import create_app, syndication
from starpunk.notes import create_note, delete_note, update_note
from starpunk.syndication import (
BlueskyClient,
bluesky_facets,
get_note_syndications,
plain_text,
process_due,
select_targets,
)
SERVICE = "https://bsky.example"
PDS = "https://pds.example"
DID = "did:plc:abc123"
class FakePDS:
def __init__(self):
self.records = {}
self.blobs = []
self.calls = []
self.logins = 0
self.expire_next = False
self.lose_next_reply = False
self.fail_next = None
def _json(self, code, body):
return httpx.Response(code, json=body)
def __call__(self, request: httpx.Request) -> httpx.Response:
nsid = request.url.path.removeprefix("/xrpc/")
self.calls.append((request.url.host, nsid))
if nsid == "com.atproto.server.createSession":
body = json.loads(request.content)
if body["password"] != "app-pass":
return self._json(401, {"error": "AuthenticationRequired"})
self.logins += 1
return self._json(200, {
"did": DID, "handle": "phil.example", "accessJwt": f"access{self.logins}",
"refreshJwt": "refresh",
"didDoc": {"service": [{"id": "#atproto_pds", "serviceEndpoint": PDS}]},
})
if nsid == "com.atproto.server.refreshSession":
self.logins += 1
return self._json(200, {"did": DID, "handle": "phil.example",
"accessJwt": f"access{self.logins}", "refreshJwt": "refresh"})
assert request.url.host == "pds.example"
if request.headers.get("Authorization") != f"Bearer access{self.logins}" or self.expire_next:
self.expire_next = False
return self._json(400, {"error": "ExpiredToken"})
if self.fail_next:
code, self.fail_next = self.fail_next, None
return self._json(code, {"error": "Nope"})
if nsid == "com.atproto.repo.uploadBlob":
self.blobs.append(len(request.content))
assert len(request.content) <= 1_000_000
return self._json(200, {"blob": {"$type": "blob", "ref": {"$link": f"b{len(self.blobs)}"},
"mimeType": request.headers["Content-Type"],
"size": len(request.content)}})
if nsid == "com.atproto.repo.createRecord":
body = json.loads(request.content)
uri = f"at://{DID}/app.bsky.feed.post/r{len(self.records) + 1}"
self.records[uri] = body["record"]
if self.lose_next_reply:
self.lose_next_reply = False
raise httpx.ReadTimeout("lost reply")
return self._json(200, {"uri": uri, "cid": "c"})
if nsid == "com.atproto.repo.listRecords":
return self._json(200, {"records": [{"uri": u, "value": v} for u, v in self.records.items()]})
if nsid == "com.atproto.repo.deleteRecord":
body = json.loads(request.content)
self.records.pop(f"at://{body['repo']}/{body['collection']}/{body['rkey']}", None)
return self._json(200, {})
return self._json(500, {"error": "unknown"})
@pytest.fixture
def pds():
return FakePDS()
class FakeWeb:
"""Other people's pages, for link cards: url -> (status, headers, body)"""
def __init__(self):
self.pages = {}
self.fetched = []
def __call__(self, request):
url = str(request.url)
self.fetched.append(url)
status, headers, body = self.pages.get(url, (404, {}, b"nope"))
return httpx.Response(status, headers=headers, content=body)
@pytest.fixture
def web():
return FakeWeb()
def routed(pds, web):
return httpx.MockTransport(
lambda r: pds(r) if r.url.host in ("bsky.example", "pds.example") else web(r))
@pytest.fixture
def app(tmp_path, pds, web):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
"SITE_URL": "https://blog.example.com",
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"BLUESKY_HANDLE": "phil.example",
"BLUESKY_APP_PASSWORD": "app-pass",
"BLUESKY_SERVICE": SERVICE,
})
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"] = BlueskyClient(
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=routed(pds, web))
)
yield app
syndication._client_cache.clear()
def publish(app, content="Hello Bluesky", tags=None):
with app.test_request_context():
note = create_note(content=content, published=True, tags=tags)
select_targets(note.id, ["bluesky"])
process_due()
return note
def only_record(pds):
(record,) = pds.records.values()
return record
def row(app, note):
with app.app_context():
(r,) = get_note_syndications(note.id)
return r
class TestText:
def test_markdown_becomes_plain_text(self):
md = "# Title\n\nSome *emphasis* and a [link](https://x.example).\n\n- one\n- two\n\n![pic](/m.jpg)"
assert plain_text(md) == "Title\n\nSome emphasis and a link.\n\n• one\n• two"
def test_facets_use_utf8_byte_offsets(self):
text = "Café ☕ https://blog.example.com/note/x #IndieWeb"
link, tag = bluesky_facets(text)
raw = text.encode()
assert raw[link["index"]["byteStart"]:link["index"]["byteEnd"]] == b"https://blog.example.com/note/x"
assert raw[tag["index"]["byteStart"]:tag["index"]["byteEnd"]] == b"#IndieWeb"
assert tag["features"][0]["tag"] == "IndieWeb"
def test_fragment_in_url_is_not_a_tag(self):
facets = bluesky_facets("see https://x.example/page#section and #42 too")
assert [f["features"][0]["$type"] for f in facets] == ["app.bsky.richtext.facet#link"]
class TestLifecycle:
def test_publish_posts_with_link_card(self, app, pds):
note = publish(app, content="Hello **Bluesky**", tags=["IndieWeb"])
record = only_record(pds)
permalink = f"https://blog.example.com/note/{note.slug}"
assert record["text"] == f"Hello Bluesky\n\n{permalink}\n\n#IndieWeb"
assert record["embed"]["external"]["uri"] == permalink
kinds = {f["features"][0]["$type"] for f in record["facets"]}
assert kinds == {"app.bsky.richtext.facet#link", "app.bsky.richtext.facet#tag"}
r = row(app, note)
assert r["state"] == "synced"
assert r["remote_id"] == f"at://{DID}/app.bsky.feed.post/r1"
assert r["remote_url"] == "https://bsky.app/profile/phil.example/post/r1"
def test_long_note_fits_300(self, app, pds):
note = publish(app, content="word " * 500)
text = only_record(pds)["text"]
assert len(text) <= 300
assert text.endswith(f"https://blog.example.com/note/{note.slug}")
def test_edit_leaves_post_alone(self, app, pds):
note = publish(app, content="first")
with app.test_request_context():
update_note(id=note.id, content="second")
process_due()
assert only_record(pds)["text"].startswith("first")
assert row(app, note)["state"] == "synced"
def test_delete_and_unpublish_remove_post(self, app, pds):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, published=False)
process_due()
assert pds.records == {}
update_note(id=note.id, published=True)
process_due()
assert len(pds.records) == 1
delete_note(id=note.id)
process_due()
assert pds.records == {}
class TestSession:
def test_logs_in_once_and_uses_pds(self, app, pds):
publish(app, content="one")
publish(app, content="two")
assert pds.logins == 1
assert ("bsky.example", "com.atproto.server.createSession") in pds.calls
assert ("pds.example", "com.atproto.repo.createRecord") in pds.calls
def test_expired_token_is_refreshed(self, app, pds):
publish(app, content="one")
pds.expire_next = True
publish(app, content="two")
assert len(pds.records) == 2
assert ("bsky.example", "com.atproto.server.refreshSession") in pds.calls
def test_bad_password_fails_permanently(self, app, pds):
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"].password = "wrong"
note = publish(app)
r = row(app, note)
assert r["state"] == "failed" and "401" in r["last_error"]
def test_lost_reply_does_not_double_post(self, app, pds):
pds.lose_next_reply = True
note = publish(app)
assert row(app, note)["state"] == "pending"
with app.test_request_context():
from starpunk.syndication import retry
retry(row(app, note)["id"])
# retry() resets attempts; simulate the scheduled retry instead
from starpunk.database import get_db
get_db(app).execute("UPDATE syndications SET attempts = 1")
get_db(app).commit()
process_due()
assert len(pds.records) == 1
assert row(app, note)["state"] == "synced"
def test_server_error_retries(self, app, pds):
publish(app, content="warm up") # log in first
pds.fail_next = 502
note = publish(app, content="later")
assert row(app, note)["state"] == "pending"
class TestMedia:
def test_large_image_is_shrunk_with_alt_and_aspect_ratio(self, app, pds):
from starpunk.media import attach_media_to_note, save_media
import os
img = Image.frombytes("RGB", (2400, 1800), os.urandom(2400 * 1800 * 3))
buf = io.BytesIO()
img.save(buf, format="PNG")
assert buf.tell() > 1_000_000
with app.test_request_context():
note = create_note(content="noisy", published=True)
media = save_media(buf.getvalue(), "noise.png")
attach_media_to_note(note.id, [media["id"]], ["Static"])
select_targets(note.id, ["bluesky"])
process_due()
record = only_record(pds)
(image,) = record["embed"]["images"]
assert record["embed"]["$type"] == "app.bsky.embed.images"
assert image["alt"] == "Static"
assert image["aspectRatio"]["width"] > 0
ARTICLE = "https://news.example/story"
ARTICLE_HTML = b"""<html><head><title>Fallback title</title>
<meta property="og:title" content="A Great Story">
<meta property="og:description" content="What happened, in short.">
<meta property="og:image" content="/img/story.png">
</head><body>...</body></html>"""
def png_bytes():
buf = io.BytesIO()
Image.new("RGB", (60, 40), (10, 120, 200)).save(buf, format="PNG")
return buf.getvalue()
class TestLinkCards:
def test_note_link_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
web.pages["https://news.example/img/story.png"] = (200, {"content-type": "image/png"}, png_bytes())
note = publish(app, content=f"Read [this story]({ARTICLE}) now")
record = only_record(pds)
external = record["embed"]["external"]
assert external["uri"] == ARTICLE
assert external["title"] == "A Great Story"
assert external["description"] == "What happened, in short."
assert external["thumb"]["$type"] == "blob"
# The link text is clickable and the note's permalink is still in the text
permalink = f"https://blog.example.com/note/{note.slug}"
assert record["text"].startswith("Read this story now") and permalink in record["text"]
raw = record["text"].encode()
links = {raw[f["index"]["byteStart"]:f["index"]["byteEnd"]].decode(): f["features"][0]["uri"]
for f in record["facets"]}
assert links["this story"] == ARTICLE
assert links[permalink] == permalink
def test_bare_url_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
publish(app, content=f"See {ARTICLE}.")
assert only_record(pds)["embed"]["external"]["uri"] == ARTICLE
def test_unreachable_page_still_gets_a_card(self, app, pds, web):
note = publish(app, content=f"See {ARTICLE}")
external = only_record(pds)["embed"]["external"]
assert external == {"uri": ARTICLE, "title": ARTICLE, "description": ""}
assert row(app, note)["state"] == "synced"
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
monkeypatch.setattr("starpunk.micropub._public_address", lambda host: host != "internal.example")
app.config["TESTING"] = False
try:
publish(app, content=f"See {ARTICLE}")
finally:
app.config["TESTING"] = True
assert "http://internal.example/admin" not in web.fetched
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE
def test_images_win_over_link_card(self, app, pds, web):
from starpunk.media import attach_media_to_note, save_media
with app.test_request_context():
note = create_note(content=f"Look {ARTICLE}", published=True)
media = save_media(png_bytes(), "a.png")
attach_media_to_note(note.id, [media["id"]], ["pic"])
select_targets(note.id, ["bluesky"])
process_due()
assert only_record(pds)["embed"]["$type"] == "app.bsky.embed.images"
assert web.fetched == []
class TestLinkText:
def test_note_links_in_reading_order(self):
from starpunk.syndication import note_links
md = "a [x](https://one.example) b https://two.example/p. c <https://three.example> [rel](/local)"
assert note_links(md) == ["https://one.example", "https://two.example/p", "https://three.example"]
def test_link_modes(self):
from starpunk.syndication import plain_text_links
md = "[a story](https://s.example) and [https://u.example](https://u.example)"
assert plain_text_links(md)[0] == "a story and https://u.example"
assert plain_text_links(md, "inline")[0] == "a story (https://s.example) and https://u.example"
assert plain_text_links(md, "anchor") == ("a story and https://u.example",
[("a story", "https://s.example")])
def test_inline_url_budget(self):
from starpunk.syndication import plain_text_links
md = "https://bare.example then [one](https://1.example) [two](https://2.example) [one again](https://1.example)"
text = plain_text_links(md, "inline", max_urls=2)[0]
assert text == "https://bare.example then one (https://1.example) two one again (https://1.example)"
def test_anchor_facet_byte_offsets(self):
text = "Café ☕ this story and more"
(facet,) = bluesky_facets(text, [("this story", "https://s.example")])
raw = text.encode()
assert raw[facet["index"]["byteStart"]:facet["index"]["byteEnd"]] == b"this story"
def test_both_targets_listed_for_micropub(tmp_path):
data = tmp_path / "d"
app = create_app({
"TESTING": True, "SITE_URL": "https://b.example", "ADMIN_ME": "https://e.example",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "s.db",
"GOTOSOCIAL_URL": "https://social.example", "GOTOSOCIAL_ACCESS_TOKEN": "t",
"BLUESKY_HANDLE": "phil.example", "BLUESKY_APP_PASSWORD": "p",
})
with app.app_context():
from starpunk.syndication import get_targets, resolve_targets
assert set(get_targets()) == {"gotosocial", "bluesky"}
assert resolve_targets(["https://bsky.app/profile/phil.example"]) == ["bluesky"]
syndication._client_cache.clear()
Loaded 100 of 106 files, more files were not shown because too many files have changed in this diff. Show more