Compare commits

...
25 Commits
Author SHA1 Message Date
Phil 43ec7cb8be Merge branch 'feat/v2.7.0-webmention-spec' for v2.7.0 release
Build Container / build (push) Successful in 11s
2026-11-03 08:07:12 -07:00
PhilandClaude Opus 5.5 4aaa7a8122 feat: W3C Webmention compliance (v2.7.0)
Sending: RFC 8288 Link header parsing, rel token matching and document
order for <link>/<a>, endpoint re-discovery on every send, re-send to
previous targets on update/delete/undelete, 410 Gone for deleted notes,
no redirect following on the POST, 429 retried.

Receiving: exact target match per media type, Accept header, final URL
for relative links, only published notes accepted, fragments ignored,
moderation kept on updates, transient errors keep valid mentions.

Security: every redirect hop SSRF-checked, all resolved addresses
checked, broader blocklist, total time and size limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:07:04 -07:00
Phil 0cfc9965df Merge branch 'fix/v2.6.9-webmention-processing' for v2.6.9 release
Build Container / build (push) Successful in 12s
2026-11-03 07:53:19 -07:00
PhilandClaude Opus 5.5 106495d114 fix: Webmention processing works end to end (v2.6.9)
- Normalise mf2 author values: u-photo with alt text is a dict in mf2py and
  broke the database update for most real senders
- Pass str(response.url) to urljoin during endpoint discovery; the TypeError
  marked every target as having no endpoint. Retry on discovery errors
- Hold a cross-process flock so only one gunicorn process runs the worker
- Keep author URL/photo only if http(s) (XSS via javascript: URLs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:53:19 -07:00
Phil 3c1e109358 Merge branch 'fix/v2.6.8-webmention-worker' for v2.6.8 release
Build Container / build (push) Successful in 20s
2026-11-03 07:41:11 -07:00
PhilandClaude Opus 5.5 d43361d505 fix: Webmention worker runs inside the app context (v2.6.8)
The worker thread called the SSRF checks, which read current_app.config,
without an app context, so every incoming and outgoing webmention failed with
'Working outside of application context'. A teardown_appcontext hook also
stopped the worker whenever a request ended.

Pass the app to WebmentionWorker and run each poll in app.app_context(), like
SyndicationWorker, and drop the teardown hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:41:06 -07:00
Phil f924413934 Merge branch 'fix/v2.6.7-admin-layout' for v2.6.7 release
Build Container / build (push) Successful in 7s
2026-11-02 18:24:15 -07:00
PhilandClaude Opus 5.5 4b2ae4c7fc fix: Admin pages line up with the header (v2.6.7)
Every page sat in the 40rem reading column, so the admin wrapper's 60rem
never applied and the notes table, which can't shrink that far, ran past
the header's right edge. Admin pages now set a 60rem frame for header,
content and footer; post previews wrap, and tables scroll sideways on
narrow screens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:24:14 -07:00
Phil d3148063e3 Merge branch 'fix/v2.6.6-inline-image-crosspost' for v2.6.6 release
Build Container / build (push) Successful in 15s
2026-11-02 18:12:51 -07:00
PhilandClaude Opus 5.5 39c8259f3f fix: Crossposts include photos embedded in the post text (v2.6.6)
Some Micropub clients (OutPost) upload a photo to the media endpoint and
then embed it in content instead of sending it as `photo`. Those posts
have no attached media, so crossposts went out without the image.

Images in the text that are in this site's media library are now sent
like attached photos, with their alt text, and GoToSocial's text no
longer carries the image Markdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:11:27 -07:00
Phil 1ce88b38a5 Merge branch 'fix/v2.6.5-micropub-media' for v2.6.5 release
Build Container / build (push) Successful in 6s
2026-11-01 10:53:37 -07:00
PhilandClaude Opus 5.5 4ffe36f9d3 fix: Photos without alt text use the post's text (v2.6.5)
Uncaptioned photos rendered alt="Image", which mf2 parsers report as
the photo's alt text. They now take the post's text (excerpt, up to 150
characters), falling back to "Image" only for text-free posts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 10:44:54 -07:00
PhilandClaude Opus 5.5 a551770c76 fix: Replies and photo posts read as the right post type (v2.6.5)
Checked rendered pages with mf2py against W3C Post Type Discovery:

- Note pages put u-in-reply-to inside a bare h-cite, making it the
  h-cite's property, so replies parsed as notes (or photo posts). The
  reply context is now the entry's u-in-reply-to h-cite.
- Home page previews had no reply markup; they now show "In reply to"
  with u-in-reply-to.
- Previews always emit e-content, so a text-free photo post can't get
  an implied name.
- Text-free page titles follow PTD order (Repost, Like, Reply, Photo).

tests/test_post_types.py runs PTD on parsed note pages and the home page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 10:06:44 -07:00
PhilandClaude Opus 5.5 d0bfc80c2e fix: Micropub photo posts and failed creates (v2.6.5)
- A create that failed after the note was written (e.g. the same photo
  twice) answered 500 but left the post published without photos.
  Photos are now resolved first, and a failed note is removed.
- Pooled connections roll back unfinished writes, so one failure no
  longer locks the database for the next ~10 s.
- The same photo twice in one post is shown once.
- Photo posts no longer need text (Micropub doesn't require content,
  and q=config already advertised a photo-only post type). Updates may
  clear the text while photos remain. Crossposts start with the link;
  the page is titled "Photo".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 09:50:30 -07:00
PhilandClaude Opus 5.5 d16463ef8a Merge branch 'fix/v2.6.4-tag-order' for v2.6.4 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:59:19 -07:00
PhilandClaude Opus 5.5 4b56f7cd4b fix: Tags keep the order they were entered in (v2.6.4)
add_tags_to_note() rewrites note_tags in entry order on every save, so
note_tags.id already records it; get_note_tags(entered_order=True) reads
it back. Crossposts (Threads topic, which hashtags fit on Bluesky), the
editor and Micropub q=source/update use it, so saving no longer
reorders tags alphabetically. Public pages stay alphabetical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:59:18 -07:00
PhilandClaude Opus 5.5 2474628646 Merge branch 'fix/v2.6.3-link-cards' for v2.6.3 release
Build Container / build (push) Successful in 8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:25 -07:00
PhilandClaude Opus 5.5 0fe341161e feat: Crosspost cards show the note's own link (v2.6.3)
When a note links somewhere (an article, say), the first link gets the
preview card instead of the note (ADR-070). Bluesky cards are built from
the page's OG/Twitter/HTML title, description and image (thumb shrunk
under 1 MB); Threads previews the first link in the text. Markdown links
no longer lose their URL on Bluesky (link text becomes a facet) or Threads
("text (url)"); Threads keeps to Meta's 5-link limit. Page fetches are
public-address only, checked at every redirect, size-capped, and never
fail the post.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:24 -07:00
PhilandClaude Opus 5.5 a62077ab30 Merge branch 'fix/v2.6.2-threads-link-preview' for v2.6.2 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 fade75fd2d fix: Let Threads build the link preview itself (v2.6.2)
An explicit link_attachment for a URL Meta hasn't fetched yet fails at
threads_publish with "Invalid Link Attachment" (subcode 4279047), which
is every new note; it only went out on the retry. Tested live with fresh
URLs: with link_attachment it fails, without it it publishes first time
and Threads still sets link_attachment_url from the first link in the
text. Reposts likewise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 0dd59fbc18 Merge branch 'fix/v2.6.1-threads-publish-retry' for v2.6.1 release
Build Container / build (push) Successful in 6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:18:11 -06:00
PhilandClaude Opus 5.5 11e09a0d02 fix: Retry Threads publish errors, show full error detail (v2.6.1)
Meta's threads_publish returned a one-off HTTP 400 "Fatal" for a plain
text post; the same post published fine on retry. Publish-step errors
are now retryable unless they are auth/permission errors (codes 10, 190,
200); the retry's existing-post check prevents doubles. Errors carry the
Graph code, subcode, user message and fbtrace id. A short pause before
the first container status poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:18:11 -06:00
PhilandClaude Opus 5.5 565adaeecf Merge branch 'feature/v2.6.0-threads' for v2.6.0 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:54:15 -06:00
PhilandClaude Opus 5.5 53d10ed271 feat: Crosspost notes to Threads (v2.6.0)
Third crossposting target on the ADR-065 loop (ADR-069). Text posts get a
link preview, 1 image posts as IMAGE and 2-4 as a CAROUSEL (Meta fetches
them from /media/), with alt text and the first tag as topic_tag. Posts are
shortened to 500 characters, counting non-ASCII as UTF-8 bytes. Edits don't
follow (the API has no edit); unpublish and delete remove the post. Retried
creates check for an existing post first.

The 60-day token is refreshed daily from the worker loop and kept in
DATA_PATH/threads_token.json under a file lock; a new configured token
replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:46:56 -06:00
PhilandClaude Opus 5.5 e8c1b03980 feat: Likes and reposts (v2.5.0)
Build Container / build (push) Successful in 6s
Micropub like-of/repost-of with optional comment; likes on /likes, off the
home page and feeds; reposts in the stream. Native favourites/boosts on
GoToSocial and likes/reposts/quotes on Bluesky when the target is on that
network, link posts for other reposts. Fix double slash in outgoing
webmention source URLs. ADR-068.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:16:21 -06:00
54 changed files with 3706 additions and 488 deletions

No files matched your search

+8
View File
@@ -63,6 +63,14 @@ GOTOSOCIAL_NAME=GoToSocial
# public, unlisted, private (default: public)
GOTOSOCIAL_VISIBILITY=public
# Threads account (v2.6.0). A long-lived user token with threads_basic,
# threads_content_publish and threads_delete; StarPunk refreshes it daily and
# keeps the current one in DATA_PATH/threads_token.json.
# How to get one: docs/design/v2.6.0/2026-10-31-threads-crossposting.md
THREADS_ACCESS_TOKEN=
# Your Threads username (for the Micropub syndicate-to uid)
THREADS_USERNAME=
# =============================================================================
# DATA STORAGE
# =============================================================================
+135
View File
@@ -7,6 +7,141 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.7.0] - 2026-11-03
Webmention now follows the [W3C Recommendation](https://www.w3.org/TR/webmention/) throughout.
### Changed
- Webmentions are only accepted for published notes. Mentions of the home page or other pages were stored but never shown; they're now refused with 400
- Deleted notes answer 410 Gone with a "This post has been deleted" page instead of 404
- A received mention that is sent again keeps its moderation: one you hid stays hidden, one you approved stays approved
### Fixed
- Sending: endpoint discovery now handles `rel` values with several words (`rel="webmention other"`), any capitalisation, `rel` after other Link parameters, and several Link headers, and takes the first `<link>` or `<a>` in page order
- Sending: the endpoint is looked up again every time (the 1–24 hour cache broke the spec's rule for updated posts)
- Sending: editing a note notifies every page it mentioned before, including an old reply target and links the edit removed. Deleting or restoring a note notifies them too, so they can remove or restore the mention
- Sending: a redirect from the endpoint is no longer treated as success (the POST was turned into a GET); "429 Too Many Requests" is retried
- Receiving: the source must link to the exact target URL (in a link, image, or other HTML attribute, any JSON value, or plain text). A longer URL that starts with the target, or the target mentioned in plain text or an HTML comment, no longer counts
- Receiving: relative links in the source are resolved, using the URL after redirects
- Receiving: the source is fetched with an `Accept` header and decoded in its declared character set
- Receiving: if the source can't be reached when a mention is sent again, the existing mention stays up instead of disappearing
- Receiving: the target's host is compared without regard to case or a default port, and `#fragment`s are dropped, so one mention isn't stored twice
- Mention snippets cut at 500 characters no longer leave HTML tags open, and the "Mentions" list no longer shows raw HTML tags
- The `Link: rel="webmention"` header is added alongside other Link headers instead of replacing them
- Links at the top level of a code block were still sent webmentions
### Security
- Fetches for webmentions check every redirect, not just the first URL, so a public page can't redirect StarPunk to a private address. Every address a hostname resolves to (IPv4 and IPv6) must be public; carrier-grade NAT, `0.0.0.0/8`, multicast, reserved and IPv4-mapped addresses are blocked too. Each fetch has a total time limit and a 1 MB size limit, including endpoint discovery
## [2.6.9] - 2026-11-03
### Fixed
- Received webmentions failed when the sender's profile photo had alt text (Bridgy, Mastodon and most IndieWeb sites do this). The photo is now stored as its URL
- Outgoing webmentions were never sent: endpoint discovery crashed on every target that had an endpoint, and the target was then marked as having none. Discovery now works, and network errors during discovery are retried instead of giving up
- Each server process ran its own webmention worker, so a webmention could be processed or sent more than once. Only one process handles the queues at a time now; if it stops, another takes over
### Security
- A webmention sender could set its author link to a `javascript:` URL, which was shown as a clickable link on the note page. Author links and photos are now only kept if they are http or https URLs
## [2.6.8] - 2026-11-03
### Fixed
- Webmentions were never processed. The background worker checked URLs against this site's settings without the app loaded, so every incoming webmention was marked "error" ("Working outside of application context") and every outgoing one kept failing and retrying. The worker now runs with the app loaded. A received webmention showed up in the admin list but never on the note
- The webmention worker also stopped as soon as any request finished, so after startup it often wasn't running at all. It now runs for the life of the server process
## [2.6.7] - 2026-11-02
### Fixed
- Admin pages were misaligned: the notes table ran past the right edge of the header and admin menu, because every page sat in the 40rem reading column and the table couldn't shrink to fit it. Admin pages now use a 60rem frame for the header, menu, content and footer alike, long post previews wrap, and on narrow screens tables scroll sideways instead of spilling out
## [2.6.6] - 2026-11-02
### Fixed
- Crossposts left out photos that were embedded in the post text instead of attached. Some Micropub clients (OutPost, for one) upload a photo to the media endpoint and then put it in `content` as an image rather than sending it as `photo`; both are valid Micropub. Images in the text that are in this site's media library are now sent to GoToSocial, Bluesky and Threads like attached photos, with their alt text, after any attached ones and within each target's photo limit. GoToSocial's text no longer includes the image Markdown. Images hosted on other sites are still left out
## [2.6.5] - 2026-11-01
### Fixed
- A Micropub post that failed partway (for example, the same photo listed twice) answered "Failed to create post" but stayed published without its photos, so a client retrying would post it twice. Photos are now looked up before the post is made, and a post that fails after that is removed again
- After such a failure, every request failed with "database is locked" for about 10 seconds: database connections now roll back unfinished writes when they're returned to the pool
- The same photo listed twice in one post is shown once instead of failing the post
- Replies didn't read as replies to IndieWeb readers and webmention receivers: on note pages `u-in-reply-to` sat inside an unrelated `h-cite`, and the home page didn't mark replies at all, so [Post Type Discovery](https://www.w3.org/TR/post-type-discovery/) saw notes (or photo posts). The reply context is now the entry's `u-in-reply-to h-cite`, and home page entries show "↩ In reply to"
- Photos without alt text used "Image" as their description, which IndieWeb readers passed on as if you'd written it. They now use the post's text (up to 150 characters), and fall back to "Image" only when the post has no text
- Home page entries always have an `e-content`, so a photo post without text never gets an implied name (which readers would show as its caption)
### Changed
- Photo posts no longer need text. Micropub doesn't require `content`, and `q=config` already offered a Photo post type with only `photo`. A Micropub update may also clear the text of a post that keeps its photos. Crossposts of a photo post without text start with its link, and its page is titled "Photo"
## [2.6.4] - 2026-11-01
### Fixed
- Tags keep the order you entered them in. The first tag is the note's main tag: it's the Threads topic, and Bluesky keeps the first tags when space runs out. Before, both used the alphabetically first tag. Public pages still list tags alphabetically
- Opening a note in the editor or updating it via Micropub reordered its tags alphabetically on save; the editor and Micropub `q=source` now show, and keep, the entered order
## [2.6.3] - 2026-11-01
### Changed
- **A note's own link gets the preview card** (ADR-070). When a note links somewhere (say, an article), the crosspost's card shows that page instead of the note. Bluesky cards are built from the page's title, description and image; Threads previews it from the text. Notes without links keep their own card
### Fixed
- Markdown links (`[text](url)`) lost their URL on Bluesky and Threads. On Bluesky the link text is now clickable; on Threads it's written "text (url)"
- Notes with many links failed on Threads (Meta allows 5 per post); extra links now keep just their text
## [2.6.2] - 2026-11-01
### Fixed
- New notes failed their first Threads crosspost with "Invalid Link Attachment" (Meta rejects an explicit link preview for a page it hasn't fetched yet) and only went out on the retry a minute later. StarPunk no longer sends the explicit link preview; Threads still shows one, built from the note's link in the text
## [2.6.1] - 2026-11-01
### Fixed
- Threads crossposts gave up after Meta's publish step returned a one-off `HTTP 400 Fatal`; publish errors are now retried (a retry checks for an existing post first, so nothing is posted twice). Only a bad token or missing permission stops retries
- Threads errors now include Meta's error code, subcode, user message and trace id, so the editor shows why a post failed
- Wait a moment before the first readiness check on a Threads post, since Meta can report it ready too early
## [2.6.0] - 2026-11-01
### Added
- **Crossposting to Threads** (ADR-069): set `THREADS_ACCESS_TOKEN` (and optionally `THREADS_USERNAME`) to add a Threads target. Text posts get a link preview; 1 image posts as an image, 2–4 as a carousel, with alt text; the note's first tag becomes the post's topic. Shortened to 500 characters. Edits don't follow (the API can't edit); unpublish and delete remove the post. Likes aren't crossposted to Threads; reposts post "🔁 title url" (with the comment, if any)
- The Threads token (60-day lifetime) is refreshed automatically once a day and kept in `data/threads_token.json`, so it survives restarts; setting a new token in the config replaces it. Setup steps: `docs/design/v2.6.0/`
## [2.5.0] - 2026-11-01
### Added
- **Likes and reposts** via Micropub `like-of` / `repost-of` (ADR-068); a comment is optional
- Likes are kept off the home page and feeds and listed on a new `/likes` page (linked in the header)
- Reposts appear in the stream and feeds with a "🔁 Reposted" line
- The liked/reposted post's title and author are fetched and shown; it is sent a webmention
- GoToSocial: likes/reposts of fediverse posts become native favourites/boosts; reposts of other pages (or with a comment) post "🔁 title url"
- Bluesky: likes/reposts of Bluesky posts become native likes/reposts; a repost with a comment is a quote post; reposts of other pages post with a link card
- Likes of ordinary web pages are not crossposted; deleting a like/repost undoes it everywhere
- Migration 013: `notes.like_of`, `notes.repost_of`
### Fixed
- Outgoing webmentions named posts `https://site//note/<slug>` (double slash)
### Notes
- The GoToSocial token needs the extra scopes `read:search` and `write:favourites`
## [2.4.0] - 2026-10-31
### Added
@@ -0,0 +1,32 @@
# ADR-068: Likes and reposts
## Status
Accepted (v2.5.0, 2026-11-01). Bookmarks were considered and declined.
## Decision
1. **Storage.** `notes.like_of` / `notes.repost_of` (migration 013). The note's
content is an optional comment and may be empty.
2. **Display.** Likes stay off the home page and feeds and are listed on
`/likes`. Reposts appear in the stream and feeds. Both show the target's
fetched title and author (the reply-context cache) and carry
`u-like-of` / `u-repost-of`. The target is sent a webmention.
3. **Crossposting, per target.**
| | Target is a post on that network | Anything else |
|---|---|---|
| Like | native favourite / like | nothing |
| Repost, no comment | native boost / repost | "🔁 title url" post |
| Repost, with comment | Bluesky quote post; GoToSocial status | comment + "🔁 title url" post |
GoToSocial finds fediverse posts with `/api/v2/search?resolve=true`
(needs `read:search`; favourites need `write:favourites`). Bluesky
recognises `bsky.app/profile/<actor>/post/<rkey>` URLs. Native actions
are stored as `favourite:<id>` / `reblog:<id>` or the like/repost record
URI, and deleting the note undoes them.
## Consequences
- Editing a like/repost doesn't change a native favourite/boost.
- A repost with a comment can't be a GoToSocial boost (no quote posts there).
@@ -0,0 +1,57 @@
# ADR-069: Crossposting to Threads
## Status
Accepted (v2.6.0, 2026-10-31)
## Context
ADR-065 and ADR-066 crosspost notes to GoToSocial and Bluesky. Threads
(Meta) has a publishing API, so it can be a third target on the same loop.
## Decision
1. **Third client, same loop.** `ThreadsClient` is target id `threads`
behind the `syndications` table and reconcile loop from ADR-065.
2. **Auth.** A long-lived Threads user token (`THREADS_ACCESS_TOKEN`) with
`threads_basic`, `threads_content_publish` and `threads_delete`. These
tokens expire after 60 days, so StarPunk refreshes the token about once a
day (`th_refresh_token`) from the worker loop, even when nothing is being
posted. The current token is kept in `DATA_PATH/threads_token.json`
(mode 0600) with a hash of the configured token. A different configured
token replaces the saved one. A file lock stops gunicorn workers from
refreshing at the same time. The file avoids a migration for one value.
3. **Posting.** Create a container, poll its status until `FINISHED`, then
`threads_publish`. Text-only notes are `TEXT` posts. Threads previews
the first link in the text (the permalink) by itself. An explicit
`link_attachment` is not sent, because Meta rejects one for a page it
hasn't fetched yet ("Invalid Link Attachment", subcode 4279047), which
is always the case for a new note (v2.6.2). One image is an `IMAGE` post and 2–4 images are a
`CAROUSEL`. Meta fetches images from their public `/media/` URL. Only
JPEG and PNG up to 8 MB are allowed, so GIF and WebP images are left off.
Alt text goes in `alt_text`.
4. **Content.** Plain text (as for Bluesky), then the permalink, shortened
to 500 characters. Non-ASCII characters are counted as UTF-8 bytes,
because Threads counts emoji that way. Threads allows one tag per post,
so the note's first tag is sent as `topic_tag` and no hashtags are added
to the text.
5. **Edits don't follow.** The API has no way to edit a post. As with
Bluesky, the post stays as first sent, and unpublishing, unticking the
target or deleting the note removes it.
6. **Likes and reposts (ADR-068).** The API can't like or repost outside
content, so likes are not crossposted. Reposts are text posts with
"🔁 title" and the URL, plus the comment if there is one, and the
reposted page as the link preview.
7. **No double posts.** A retried create first looks for a recent post that
already contains the permalink.
## Consequences
- Meta limits accounts to 250 posts and 100 deletes per day. Rate-limit
errors (Graph codes 4, 17, 32, 613) are retried.
- If StarPunk is down for more than 60 days, the token expires and a new
one is needed.
- Getting the first token requires a Meta developer app with the Threads
use case. See the setup notes in `docs/design/v2.6.0/`.
- Notes with more than five links fail on Threads (a Meta limit). The
error is shown in the editor.
@@ -0,0 +1,44 @@
# ADR-070: Links and link cards in crossposts
## Status
Accepted (v2.6.3, 2026-11-01)
## Context
Every crosspost carries the note's permalink. Notes often link elsewhere
too, usually to the article being discussed. Before this change:
- Bluesky and Threads get plain text, and a Markdown link `[text](url)`
lost its URL, so readers there couldn't follow it.
- The Bluesky card always showed the note, and the Threads preview showed
whichever link came first.
- Threads rejects posts with more than 5 different links.
## Decision
1. **The card shows what the note links to.** If the note has its own link
(a Markdown link, `<url>` or a bare URL), the first one gets the
preview card. A note without links gets a card for itself, as before.
Images still take the place of a card on Bluesky.
2. **Bluesky.** The card is built by StarPunk: it fetches the page's Open
Graph, Twitter or plain HTML title, description and image, and uploads
the image (shrunk to under 1 MB) as the thumbnail. If the page can't
be fetched, the card has just the URL. Markdown link text becomes a
clickable link facet, so no characters are spent on the URL.
3. **Threads.** It has no clickable link text, so Markdown links are
written "text (url)". Threads previews the first link in the text, which
is now the linked page. To stay within Meta's 5-link limit, after 4
different links in the body a Markdown link keeps only its text.
4. **GoToSocial** already gets Markdown, so its links are unchanged.
Receiving servers build their own cards.
5. **Fetching is careful.** Only http(s) URLs on public addresses are
fetched, and this is checked again at every redirect. At most 512 KB of
a page and 5 MB of an image are read. A failed fetch never fails the
post.
## Consequences
- One or two extra requests to the linked site when a note is crossposted
to Bluesky.
- Reposts (ADR-068) are unchanged: their card is the reposted page.
@@ -0,0 +1,62 @@
# v2.6.0: Crossposting to Threads
Implements ADR-069.
## What changed
- `starpunk/syndication.py`: `ThreadsClient` (target `threads`); media
entries now carry their public `url`; `process_due()` calls each client's
optional `maintain()` hook, which Threads uses to refresh its token.
- `starpunk/config.py`: `THREADS_ACCESS_TOKEN`, `THREADS_USERNAME`,
`THREADS_NAME`, `THREADS_MAX_CHARS` (500), `THREADS_MAX_MEDIA` (4).
- `tests/test_syndication_threads.py`: a fake Graph API covering text,
image and carousel posts; container polling; edits not following; deletes
(including posts already deleted by hand); a lost publish reply;
permanent and rate-limit errors; and token refresh, persistence and
replacement.
## Configuration
| Variable | Meaning |
|---|---|
| `THREADS_ACCESS_TOKEN` | Long-lived Threads user token. Enables the target. |
| `THREADS_USERNAME` | Optional. Used for the Micropub `syndicate-to` uid (`https://www.threads.com/@name`) and for fallback links. |
| `THREADS_NAME` | Label in the editor (default `Threads`) |
After the first daily refresh, the working token lives in
`DATA_PATH/threads_token.json`. The configured value only seeds it.
Changing `THREADS_ACCESS_TOKEN` switches to the new token.
## Getting a token (one time)
1. At <https://developers.facebook.com/apps>, create an app with the
**Access the Threads API** use case.
2. Under the use case's permissions, add `threads_basic`,
`threads_content_publish` and `threads_delete`.
3. Under **App roles → Roles**, add your Threads account as a Threads
tester. Accept the invite in Threads (Settings → Account → Website
permissions → Invites). The app can stay in development mode.
4. In the use case settings, add `https://<site>/` as a redirect URI. Then
open:
https://threads.net/oauth/authorize?client_id=APP_ID&redirect_uri=https://<site>/&scope=threads_basic,threads_content_publish,threads_delete&response_type=code
After you approve, the browser lands on `https://<site>/?code=...`. Copy
the code (drop a trailing `#_`).
5. Exchange the code for a short-lived token, then exchange that for a
long-lived one:
curl -s -X POST https://graph.threads.net/oauth/access_token \
-F client_id=APP_ID -F client_secret=APP_SECRET \
-F grant_type=authorization_code -F redirect_uri=https://<site>/ -F code=CODE
curl -s "https://graph.threads.net/access_token?grant_type=th_exchange_token&client_secret=APP_SECRET&access_token=SHORT_TOKEN"
6. Set `THREADS_ACCESS_TOKEN` to the long-lived `access_token`.
The app secret is only needed for these steps. StarPunk doesn't use it.
## Not done
- Video, GIF and WebP images (Threads takes video by URL. StarPunk only
stores still images, and Threads rejects GIF and WebP).
- Edits (no API for them).
@@ -0,0 +1,89 @@
# v2.6.5: Micropub media fixes
Found while adding media uploads to tuipub, a Micropub client written to the
spec rather than to StarPunk.
## Problems
1. **Failed creates left a post behind.** `handle_create` made the note, then
attached photos. Listing one photo twice broke the `note_media` unique key,
so the client got a 500 while the note stayed published without photos.
A retry would post it twice.
2. **Then the database stayed locked.** `attach_media_to_note` didn't roll
back, and `ConnectionPool.return_connection` pooled the connection with
its write transaction still open. Other requests got "database is locked"
until the 10 s timeout.
3. **Photo posts needed text.** The Micropub spec doesn't require `content`,
and `q=config` advertised a `photo` post type with only `photo`, yet
creates without content got 400 "Content is required".
## Changes
- `database/pool.py`: `return_connection` rolls back an open transaction
(or closes the connection if that fails).
- `media.py`: `attach_media_to_note` rolls back on error.
- `micropub.py`:
- `handle_create` resolves photos (by media URL, downloading remote ones)
before `create_note`. If attaching or crosspost selection fails, it
hard-deletes the note before answering 500.
- `_resolve_photos` skips repeated media.
- Content is optional when photos resolve. Content-less posts whose photos
all fail still get a 400 from `create_note`.
- `handle_update` allows `replace: {content: [""]}` and `delete: ["content"]`
while the note keeps photos (or is a like or repost). Otherwise it answers
400 "A post needs text or a photo".
- `notes.py`: `create_note` and `update_note` take `has_media`, so
empty content is allowed for photo posts.
- `syndication.py`: `compose_text` drops the leading blank line when there's
no body.
## Post types (IndieWeb standards check)
Post Type Discovery (W3C Note) has no separate "photo" type to store. A post
with a valid `photo` is a photo post, unless it's first a repost, like or
reply (step order: repost, like, reply, video, photo). Otherwise it's a note,
or an article when its `name` isn't a prefix of its content. h-entry treats
`u-photo` as the primary content, with `name` as a caption. mf2 implies a
`name` only when an entry has no other `p-*`/`e-*` properties and no nested
microformats.
StarPunk already stores photo posts as notes with media and renders
`u-photo` outside `e-content`, which is what the standards ask for. Checking
the rendered pages with mf2py found:
- **Replies weren't replies.** In `note.html`, `u-in-reply-to` was inside a
bare `h-cite`, so it was the h-cite's property, not the entry's.
`note_preview.html` had no reply markup at all. Fixed: the blockquote is
`u-in-reply-to h-cite` (with `u-url` inside), and previews show
"↩ In reply to" with `u-in-reply-to`.
- **Implied names were possible.** Previews left out `e-content` for posts
without text. They worked only because the author h-card blocked an
implied name. Previews now always emit `e-content`.
- Page title for text-free posts follows PTD order: Repost, Like, Reply,
Photo.
Photos without a caption take the post's text as alt (`partials/media.html`
`fallback_alt`) instead of the made-up "Image". The Micropub `q=source`
still reports the stored caption (`""`).
`tests/test_post_types.py` parses note pages and the home page with mf2py and
runs PTD for notes, articles, photo posts (no text, caption, title), replies
(with and without a photo) and reposts.
## Not changed (noted for later)
- Text-free photos without alt text still render `alt="Image"`. Captioned
photos use their caption, and others now use the post's text (`display_media`
`fallback_alt`, the excerpt cut to 150 characters).
- More than 4 photos are still cut to 4 without an error (ADR-057).
- `video` and `audio` are ignored. This is allowed by the spec, and they
aren't in `q=config` post-types.
- Media endpoint errors for non-images expose Python text
("cannot identify image file <_io.BytesIO …>").
## Tests
`tests/test_micropub_v265.py`. The full suite runs with
`SESSION_SECRET=test`. 8 tests in `test_media_upload.py` and
`test_migration_race_condition.py` already fail on main, and still fail here.
@@ -0,0 +1,37 @@
# v2.6.8: Webmention worker app context
## Problem
Production webmention #1 (Bridgy, reply from Mastodon to note 20261101162512)
was stored with `validation_status = 'error'`:
"Working outside of application context". Both outgoing queue items had the
same error and were on their last retry.
## Causes
1. `webmention_ssrf.safe_resolve()` reads `current_app.config['DEV_MODE']`.
`WebmentionWorker` is a plain thread with no Flask app context, so every
`validate_url_for_fetch()` call raised `RuntimeError`, for both
validation (incoming) and endpoint discovery and sending (outgoing).
2. `create_app()` registered a `teardown_appcontext` handler that called
`webmention_worker.stop()`. Teardown runs whenever any app context ends
(every request), so the worker stopped soon after startup.
## Fix
- `WebmentionWorker` takes the Flask app and runs each poll inside
`app.app_context()`, as `SyndicationWorker` already does.
- The teardown handler is removed. The worker is a daemon thread and ends with
the process.
## Tests
`tests/test_webmention_worker.py`: an outgoing send completes through the
worker thread, and the worker is still alive after an app context and a
request end. Both fail without the fix.
## Production data
After deploying, reset webmention 1 to `validation_status = 'pending'` and the
two queue items to `retry_count = 0, next_retry_at = NULL` so the worker
processes them again.
@@ -0,0 +1,36 @@
# v2.6.9: Webmention processing fixes
Found after v2.6.8 let the worker run in production, and by a review of the
implementation against the W3C Webmention Recommendation.
## Fixes
1. **Author photo with alt text** (`webmention_processor.extract_author`).
mf2py returns `u-photo` with an `alt` attribute as
`{'value': url, 'alt': ...}`; the dict was bound to SQLite and the update
failed. `_first_str()` reduces mf2 values to their `value`.
2. **Endpoint discovery** (`WebmentionWorker._discover_endpoint`).
`response.url` is an `httpx.URL`; `urljoin(str, URL)` raised `TypeError`,
which was caught and returned as "no endpoint", so every target was marked
`no_endpoint`. Pass `str(response.url)`. Fetch errors now propagate so the
caller schedules a retry, instead of being treated as "no endpoint".
3. **Duplicate processing.** gunicorn runs 4 processes, each with a worker,
and queue items were claimed by SELECT then UPDATE. The worker now takes a
non-blocking `flock` on `<data>/webmention-worker.lock` before processing;
others wait and retry each poll, so one takes over if the holder dies.
Orphan cleanup runs when the lock is acquired.
4. **XSS (spec 4.1).** `author_url` and `author_photo` are kept only if they
are absolute http(s) URLs; otherwise the source URL / no photo is used.
## Tests
`tests/test_webmention_worker.py`: the send test uses a real `httpx.Response`
(the old `MagicMock` with a string URL hid fix 2), plus tests for the lock,
an alt-text photo and a `javascript:` author URL. All fail on v2.6.8.
## Remaining spec gaps
Tracked separately: rel-token and document-order endpoint discovery, exact
target match during verification, Accept header, re-sending on delete, 410
for deleted notes, SSRF re-validation on redirects, and the others listed in
the compliance review.
@@ -0,0 +1,50 @@
# v2.7.0: Webmention W3C compliance
Review of StarPunk's Webmention support against the W3C Recommendation
(https://www.w3.org/TR/webmention/, 12 January 2017), and the changes made.
v2.6.8 and v2.6.9 fixed the bugs that stopped webmentions working at all.
## Sending
| Section | Requirement | Change |
|---|---|---|
| 3.1.2 | Link header with rel=webmention first (MUST) | RFC 8288 parsing: several links per header, several headers, `rel` in any position, token list, case-insensitive |
| 3.1.2 | Then first `<link>`/`<a>` rel=webmention in document order (MUST) | Single pass over `link` and `a` in document order; rel token match; elements without `href` skipped |
| 3.1.2 | Relative URL resolved against target; empty href is the target; query kept (MUST) | `urljoin` against the final URL after redirects |
| 3.1.2 | Follow redirects | `safe_get` follows up to 10, checking each hop |
| 3.1.2 | User-Agent with "Webmention" (MAY) | `StarPunk/<version> Webmention (+site)` |
| 3.1.3 | Form-encoded POST; any 2xx success (MUST) | Unchanged. Redirects not followed (httpx turned 301/302 POST into GET); 429 retried |
| 3.1.4 | Re-discover endpoint on update (MUST) | Discovery cache removed |
| 3.1.4 | Re-send to previously sent targets, including removed links (SHOULD) | `previous_targets()` from the queue, plus removed links; not capped |
| 3.1.5 | Deleted post returns 410 (SHOULD), tombstone (SHOULD) | `public.note` aborts 410 for soft-deleted notes and hard-deleted notes that sent webmentions; `410.html` |
| 3.1.5 | Re-send on delete (SHOULD) | `requeue_previous_webmentions()`; for a hard delete the rows are detached (`source_note_id = NULL`) first, since the foreign key cascades. Also on undelete |
## Receiving
| Section | Requirement | Change |
|---|---|---|
| 3.2 | 202 for async processing (MUST) | Unchanged |
| 3.2.1 | Valid http(s) URLs; source ≠ target; 400 on error (MUST) | Unchanged |
| 3.2.1 | Target is a valid resource (SHOULD) | Only published notes; others 400 |
| 3.2.1 | Fragment ignored | Stripped before storing (no duplicate rows) |
| 3.2.2 | GET source, follow redirects, limit redirects | `safe_get` |
| 3.2.2 | Accept header (SHOULD) | Added |
| 3.2.2 | Exact match of target, per media type (MUST) | HTML: every link attribute (`lxml iterlinks`) resolved against the final URL; JSON: any string value; `text/*`: substring; other types: no match. Fragments ignored |
| 3.2.4 | Update existing data; 410 / no link removes it (SHOULD) | Unchanged in effect. Moderation decisions kept on update; transient errors keep a valid mention |
## Security (section 4)
- 4.1 XSS: author URL/photo must be http(s) (v2.6.9); snippets re-sanitized after truncation; Mentions list uses `striptags`.
- 4.2 / 4.5: `safe_get` checks every redirect hop, checks all `getaddrinfo` addresses, blocks private, loopback, link-local, multicast, reserved, unspecified, CGNAT, `0.0.0.0/8` and IPv4-mapped forms; 1 MB and 10 s total limits.
- 4.3: endpoint URL checked before POST, no redirects followed.
## Known limits
- DNS is checked before connecting, not pinned for the connection, so a
rebinding DNS server could still answer differently on connect.
- Homepage mentions are refused rather than stored.
- `error` rows (unexpected failures) are not retried automatically.
## Tests
`tests/test_webmention_spec.py` (section-numbered), `tests/test_webmention_worker.py`.
+8
View File
@@ -0,0 +1,8 @@
-- Migration 013: Likes and reposts (v2.5.0, ADR-068)
-- A note with like_of is a like; with repost_of, a repost. Content is the
-- optional comment and may be empty.
ALTER TABLE notes ADD COLUMN like_of TEXT;
ALTER TABLE notes ADD COLUMN repost_of TEXT;
CREATE INDEX IF NOT EXISTS idx_notes_like_of ON notes(like_of);
+4 -9
View File
@@ -302,7 +302,7 @@ def create_app(config=None):
if (app.config.get('WEBMENTION_ENABLED', True) and
response.content_type and
'text/html' in response.content_type):
response.headers['Link'] = '</webmention>; rel="webmention"'
response.headers.add('Link', '</webmention>; rel="webmention"')
return response
# Register centralized error handlers
@@ -330,6 +330,7 @@ def create_app(config=None):
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
from starpunk.webmention_worker import WebmentionWorker
webmention_worker = WebmentionWorker(
app,
db_path=app.config['DATABASE_PATH'],
site_url=app.config['SITE_URL'],
poll_interval=app.config.get('WEBMENTION_POLL_INTERVAL', 10),
@@ -345,12 +346,6 @@ def create_app(config=None):
f"batch_size={webmention_worker.batch_size})"
)
# Register cleanup handler
@app.teardown_appcontext
def cleanup_webmention_worker(error=None):
if hasattr(app, 'webmention_worker') and app.webmention_worker.is_alive():
app.webmention_worker.stop()
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
@@ -464,5 +459,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.4.0"
__version_info__ = (2, 1, 0)
__version__ = "2.7.0"
__version_info__ = (2, 7, 0)
+6
View File
@@ -137,6 +137,12 @@ def load_config(app, config_override=None):
app.config["BLUESKY_SERVICE"] = os.getenv("BLUESKY_SERVICE", "https://bsky.social")
app.config["BLUESKY_NAME"] = os.getenv("BLUESKY_NAME", "Bluesky")
app.config["BLUESKY_MAX_CHARS"] = int(os.getenv("BLUESKY_MAX_CHARS", "300"))
# Crossposting to Threads (v2.6.0, ADR-069): long-lived token, refreshed automatically
app.config["THREADS_ACCESS_TOKEN"] = os.getenv("THREADS_ACCESS_TOKEN", "")
app.config["THREADS_USERNAME"] = os.getenv("THREADS_USERNAME", "")
app.config["THREADS_NAME"] = os.getenv("THREADS_NAME", "Threads")
app.config["THREADS_MAX_CHARS"] = int(os.getenv("THREADS_MAX_CHARS", "500"))
app.config["THREADS_MAX_MEDIA"] = int(os.getenv("THREADS_MAX_MEDIA", "4"))
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
# Apply overrides if provided
+10
View File
@@ -107,6 +107,16 @@ class ConnectionPool:
if not conn:
return
# A request that failed mid-write must not hand an open transaction
# (and its write lock) to the next request (v2.6.5)
if conn.in_transaction:
try:
conn.rollback()
except Exception:
conn.close()
self._stats['connections_closed'] += 1
return
with self._lock:
if len(self._pool) < self.pool_size:
# Return to pool
+5
View File
@@ -82,6 +82,11 @@ def register_error_handlers(app):
return render_template('404.html'), 404
@app.errorhandler(410)
def gone(error):
"""Handle 410 Gone (deleted notes, Webmention 3.1.5)"""
return render_template('410.html'), 410
@app.errorhandler(405)
def method_not_allowed(error):
"""Handle 405 Method Not Allowed errors"""
+2 -2
View File
@@ -198,7 +198,7 @@ def generate_atom_streaming(
yield f' <link rel="enclosure" type="{_escape_xml(mime_type)}" href="{_escape_xml(media_url)}" length="{size}"{title_attr}/>\n'
# Content - include media as HTML (per Q24)
if note.html:
if note.feed_html:
# Build HTML content with media at top
html_content = ""
@@ -212,7 +212,7 @@ def generate_atom_streaming(
html_content += '</div>'
# Add text content below media
html_content += note.html
html_content += note.feed_html
# HTML content - escaped
yield ' <content type="html">'
+2 -2
View File
@@ -267,7 +267,7 @@ def _build_item_object(site_url: str, note: Note) -> Dict[str, Any]:
# Add content (HTML or text)
# Per Q24: Include media as HTML in content_html
if note.html:
if note.feed_html:
content_html = ""
# Add media at top if present (v1.2.0 Phase 3)
@@ -280,7 +280,7 @@ def _build_item_object(site_url: str, note: Note) -> Dict[str, Any]:
content_html += '</div>'
# Add text content below media
content_html += note.html
content_html += note.feed_html
item["content_html"] = content_html
else:
item["content_text"] = note.content
+2 -2
View File
@@ -137,7 +137,7 @@ def generate_rss(
html_content += '</div>'
# Add text content below media
html_content += clean_html_for_rss(note.html)
html_content += clean_html_for_rss(note.feed_html)
# feedgen automatically wraps content in CDATA for RSS
fe.description(html_content)
@@ -278,7 +278,7 @@ def generate_rss_streaming(
html_content += '</div>'
# Add text content below media
html_content += clean_html_for_rss(note.html)
html_content += clean_html_for_rss(note.feed_html)
# Build item XML
item_xml = f""" <item>
+16 -12
View File
@@ -868,20 +868,24 @@ def attach_media_to_note(note_id: int, media_ids: List[int], captions: List[str]
db = get_db(current_app)
# Delete existing associations (for edit case)
db.execute("DELETE FROM note_media WHERE note_id = ?", (note_id,))
try:
# Delete existing associations (for edit case)
db.execute("DELETE FROM note_media WHERE note_id = ?", (note_id,))
# Insert new associations
for i, (media_id, caption) in enumerate(zip(media_ids, captions)):
db.execute(
"""
INSERT INTO note_media (note_id, media_id, display_order, caption)
VALUES (?, ?, ?, ?)
""",
(note_id, media_id, i, caption or None)
)
# Insert new associations
for i, (media_id, caption) in enumerate(zip(media_ids, captions)):
db.execute(
"""
INSERT INTO note_media (note_id, media_id, display_order, caption)
VALUES (?, ?, ?, ?)
""",
(note_id, media_id, i, caption or None)
)
db.commit()
db.commit()
except Exception:
db.rollback()
raise
def get_note_media(note_id: int) -> List[Dict]:
+71 -14
View File
@@ -195,6 +195,16 @@ def extract_content(properties: dict) -> str:
return content.strip()
def _has_text(content) -> bool:
"""Whether a content property has any text. Likes, reposts (v2.5.0) and
photo posts (v2.6.5) may come without."""
values = content if isinstance(content, list) else [content]
return any(
(c.strip() if isinstance(c, str) else (c.get("text") or c.get("html") or "").strip())
for c in values if c
)
def extract_title(properties: dict) -> Optional[str]:
"""
Extract the explicit title (Micropub 'name' property)
@@ -296,16 +306,21 @@ def extract_post_status(properties: dict) -> bool:
raise MicropubValidationError(f"Unsupported post-status: {status}")
def extract_in_reply_to(properties: dict) -> Optional[str]:
value = _first(properties.get("in-reply-to"))
def extract_url_property(properties: dict, name: str) -> Optional[str]:
"""A URL property (in-reply-to, like-of, repost-of): string or h-cite"""
value = _first(properties.get(name))
if isinstance(value, dict): # h-cite
value = _first((value.get("properties") or {}).get("url")) or value.get("value")
value = (value or "").strip() if isinstance(value, str) else ""
if value and not value.startswith(("http://", "https://")):
raise MicropubValidationError("in-reply-to must be an http(s) URL")
raise MicropubValidationError(f"{name} must be an http(s) URL")
return value or None
def extract_in_reply_to(properties: dict) -> Optional[str]:
return extract_url_property(properties, "in-reply-to")
def extract_photos(properties: dict) -> list[dict[str, str]]:
"""
Extract photo URLs and alt text from Micropub properties
@@ -415,7 +430,8 @@ def _resolve_photos(photos: list[dict]) -> tuple[list[int], list[str]]:
if len(media_ids) == MAX_PHOTOS:
break
media_id = photo.get("media_id") or _media_id_for_url(photo["url"])
if media_id:
# The same photo twice is shown once (v2.6.5)
if media_id and media_id not in media_ids:
media_ids.append(media_id)
captions.append(photo.get("alt", ""))
return media_ids, captions
@@ -516,19 +532,33 @@ def handle_create(data: dict, token_info: dict, files=None):
try:
properties = normalize_properties(data)
content = extract_content(properties)
like_of = extract_url_property(properties, "like-of")
repost_of = extract_url_property(properties, "repost-of")
if like_of and repost_of:
raise MicropubValidationError("A post can't be both a like and a repost")
has_content = _has_text(properties.get("content"))
photos = extract_photos(properties) + _save_uploads(files)
# Photo posts may have no text (v2.6.5): Micropub doesn't require content
content = (
extract_content(properties)
if has_content or not (like_of or repost_of or photos)
else ""
)
title = extract_title(properties)
tags = extract_tags(properties)
published_date = extract_published_date(properties)
published = extract_post_status(properties)
in_reply_to = extract_in_reply_to(properties)
photos = extract_photos(properties) + _save_uploads(files)
# Look up and download photos before the note exists, so a bad photo
# can't leave a half-made post behind (v2.6.5)
media_ids, captions = _resolve_photos(photos)
except MicropubError:
raise
except Exception as e:
current_app.logger.error(f"Property extraction failed: {e}")
raise MicropubValidationError(f"Failed to parse request: {str(e)}")
note = None
try:
note = create_note(
content=add_title_heading(content, title),
@@ -537,10 +567,14 @@ def handle_create(data: dict, token_info: dict, files=None):
custom_slug=custom_slug,
tags=tags if tags else None,
in_reply_to=in_reply_to,
like_of=like_of,
repost_of=repost_of,
has_media=bool(media_ids),
)
if photos:
_attach_photos_to_note(note.id, photos)
if media_ids:
from starpunk.media import attach_media_to_note
attach_media_to_note(note.id, media_ids, captions)
# Crosspost after photos are attached (v2.1.0); drafts wait until published
from starpunk.syndication import get_targets, resolve_targets, select_targets
@@ -557,6 +591,13 @@ def handle_create(data: dict, token_info: dict, files=None):
raise
except Exception as e:
current_app.logger.error(f"Failed to create note via Micropub: {e}")
# The client is told it failed, so don't leave the post up (v2.6.5)
if note is not None:
try:
from starpunk.notes import delete_note
delete_note(id=note.id, soft=False)
except Exception as cleanup:
current_app.logger.error(f"Could not remove failed note {note.slug}: {cleanup}")
raise MicropubError("server_error", "Failed to create post", status_code=500)
@@ -587,8 +628,10 @@ def handle_update(data: dict, token_info: dict):
if set(add) - {"category", "photo"}:
raise MicropubValidationError("Only category and photo can be added to")
from starpunk.tags import get_note_tags
title, body = split_title(note.content)
tags = [t["display_name"] for t in note.tags]
tags = [t["display_name"] for t in get_note_tags(note.id, entered_order=True)]
photos = _note_photos(note)
changes = {}
@@ -598,7 +641,7 @@ def handle_update(data: dict, token_info: dict):
if prop == "name":
title = None
elif prop == "content":
raise MicropubValidationError("content can't be deleted")
body = ""
elif prop == "category":
tags = []
elif prop == "photo":
@@ -619,7 +662,8 @@ def handle_update(data: dict, token_info: dict):
raise MicropubValidationError(f"Can't delete individual values of {prop}")
if "content" in replace:
body = extract_content(replace)
# Empty text is fine if the note keeps its photos; update_note checks (v2.6.5)
body = extract_content(replace) if _has_text(replace["content"]) else ""
if "name" in replace:
title = extract_title(replace)
if "category" in replace:
@@ -637,6 +681,10 @@ def handle_update(data: dict, token_info: dict):
if "photo" in add:
photos += extract_photos(add)
# Text may go only if photos stay (or it's a like or repost) (v2.6.5)
if not (body or "").strip() and not (photos or note.like_of or note.repost_of):
raise MicropubValidationError("A post needs text or a photo")
new_content = add_title_heading(body, title)
if new_content != note.content:
changes["content"] = new_content
@@ -645,7 +693,7 @@ def handle_update(data: dict, token_info: dict):
try:
if changes:
update_note(id=note.id, **changes)
update_note(id=note.id, has_media=bool(photos), **changes)
if "photo" in touched:
_attach_photos_to_note(note.id, photos)
note_changed(note.id)
@@ -701,10 +749,17 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
}
if title:
props["name"] = [title]
if note.tags:
props["category"] = [tag["display_name"] for tag in note.tags]
from starpunk.tags import get_note_tags
tags = get_note_tags(note.id, entered_order=True)
if tags:
props["category"] = [tag["display_name"] for tag in tags]
if getattr(note, "in_reply_to", None):
props["in-reply-to"] = [note.in_reply_to]
if getattr(note, "like_of", None):
props["like-of"] = [note.like_of]
if getattr(note, "repost_of", None):
props["repost-of"] = [note.repost_of]
photos = _note_photos(note)
if photos:
props["photo"] = [{"value": p["url"], "alt": p["alt"]} for p in photos]
@@ -768,6 +823,8 @@ def handle_query(args, token_info: dict):
{"type": "article", "name": "Article", "properties": ["name", "content"]},
{"type": "photo", "name": "Photo", "properties": ["photo"]},
{"type": "reply", "name": "Reply", "properties": ["in-reply-to", "content"]},
{"type": "like", "name": "Like", "properties": ["like-of"]},
{"type": "repost", "name": "Repost", "properties": ["repost-of", "content"]},
],
}), 200
+35
View File
@@ -134,6 +134,8 @@ class Note:
deleted_at: Optional[datetime] = None
content_hash: Optional[str] = None
in_reply_to: Optional[str] = None # v1.6.0: URL being replied to
like_of: Optional[str] = None # v2.5.0: URL liked (this note is a like)
repost_of: Optional[str] = None # v2.5.0: URL reposted (this note is a repost)
_cached_content: Optional[str] = field(
default=None, repr=False, compare=False, init=False
)
@@ -192,6 +194,8 @@ class Note:
_data_dir=data_dir,
content_hash=data.get("content_hash"),
in_reply_to=data.get("in_reply_to"),
like_of=data.get("like_of"),
repost_of=data.get("repost_of"),
)
@property
@@ -367,12 +371,43 @@ class Note:
excerpt = excerpt[:last_space]
excerpt += "..."
if not excerpt and self.reacts_to:
# A like or repost with no comment
verb = "Liked" if self.kind == "like" else "Reposted"
return f"{verb} {self.reacts_to}"
return excerpt
except (FileNotFoundError, OSError):
# If file doesn't exist, return slug
return self.slug
@property
def kind(self) -> str:
"""'like', 'repost' or 'note' (v2.5.0)"""
if self.like_of:
return "like"
if self.repost_of:
return "repost"
return "note"
@property
def reacts_to(self) -> Optional[str]:
"""The URL a like or repost points at"""
return self.like_of or self.repost_of
@property
def feed_html(self) -> str:
"""HTML for feeds: a repost/like line, then the note's own HTML"""
from html import escape
if not self.reacts_to:
return self.html
verb = "Liked" if self.kind == "like" else "Reposted"
url = escape(self.reacts_to, quote=True)
line = f'<p>{"★" if self.kind == "like" else "🔁"} {verb} <a href="{url}">{url}</a></p>'
return line + (self.html or "")
@property
def permalink(self) -> str:
"""
+58 -21
View File
@@ -139,7 +139,10 @@ def create_note(
created_at: Optional[datetime] = None,
custom_slug: Optional[str] = None,
tags: Optional[list[str]] = None,
in_reply_to: Optional[str] = None
in_reply_to: Optional[str] = None,
like_of: Optional[str] = None,
repost_of: Optional[str] = None,
has_media: bool = False,
) -> Note:
"""
Create a new note
@@ -203,7 +206,11 @@ def create_note(
- created_at and updated_at set to same value initially
"""
# 1. VALIDATION (before any changes)
if not content or not content.strip():
# Likes and reposts may have no comment (v2.5.0), photo posts no text (v2.6.5)
if like_of and repost_of:
raise InvalidNoteDataError("like_of", like_of, "A note can't be both a like and a repost")
content = content or ""
if not content.strip() and not (like_of or repost_of or has_media):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
@@ -264,10 +271,10 @@ def create_note(
try:
db.execute(
"""
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash, in_reply_to)
VALUES (?, ?, ?, ?, ?, ?, ?)
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash, in_reply_to, like_of, repost_of)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(slug, file_path_rel, published, created_at, updated_at, content_hash, in_reply_to),
(slug, file_path_rel, published, created_at, updated_at, content_hash, in_reply_to, like_of, repost_of),
)
db.commit()
except Exception as e:
@@ -322,33 +329,36 @@ def create_note(
from starpunk.webmention_sender import queue_webmentions_for_note
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
note_url = f"{site_url}/note/{slug}"
from starpunk.utils import absolute_url_for
note_url = absolute_url_for("public.note", slug=slug)
# Get rendered HTML from note
html_content = note.html
# Queue webmentions (best-effort, failures logged but don't fail note creation)
# The reply/like/repost target is notified first (v2.5.0)
queue_webmentions_for_note(
db_path=db_path,
note_id=note_id,
note_url=note_url,
content_html=html_content,
in_reply_to=in_reply_to
in_reply_to=in_reply_to or like_of or repost_of
)
except Exception as e:
# Webmention queueing failure should not prevent note creation
current_app.logger.warning(f"Failed to queue webmentions for note {slug}: {e}")
# 12. FETCH REPLY CONTEXT (v1.6.0 Phase 3)
# Fetch context if this is a reply
if in_reply_to:
# Fetch context for the post being replied to, liked or reposted
context_url = in_reply_to or like_of or repost_of
if context_url:
try:
from starpunk.reply_context import fetch_reply_context
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
# Fetch context (best-effort, failures logged but don't fail note creation)
fetch_reply_context(db_path, in_reply_to, site_url)
fetch_reply_context(db_path, context_url, site_url)
except Exception as e:
# Reply context fetch failure should not prevent note creation
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
@@ -472,6 +482,7 @@ def list_notes(
offset: int = 0,
order_by: str = "created_at",
order_dir: str = "DESC",
likes: Optional[bool] = None,
) -> list[Note]:
"""
List notes with filtering and pagination
@@ -564,6 +575,11 @@ def list_notes(
params = []
if published_only:
query += " AND published = 1"
# likes: None = all notes, False = leave likes out, True = only likes (v2.5.0)
if likes is True:
query += " AND like_of IS NOT NULL"
elif likes is False:
query += " AND like_of IS NULL"
# Add ordering (safe because order_by validated above)
query += f" ORDER BY {order_by} {order_dir}"
@@ -589,7 +605,8 @@ def update_note(
content: Optional[str] = None,
published: Optional[bool] = None,
tags: Optional[list[str]] = None,
in_reply_to: Optional[str] = None
in_reply_to: Optional[str] = None,
has_media: bool = False,
) -> Note:
"""
Update a note's content and/or published status
@@ -663,12 +680,9 @@ def update_note(
if content is None and published is None and tags is None and in_reply_to is None:
raise ValueError("Must provide at least one of content, published, tags, or in_reply_to to update")
# Validate content if provided
if content is not None:
if not content or not content.strip():
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
# Validate content if provided (checked against the note below for
# likes/reposts, whose comment may be empty)
empty_content = content is not None and not content.strip()
# 2. GET EXISTING NOTE
# Phase 6: Load content to get old HTML for webmention URL diffing
@@ -678,6 +692,11 @@ def update_note(
identifier = slug if slug is not None else id
raise NoteNotFoundError(identifier)
if empty_content and not (existing_note.like_of or existing_note.repost_of or has_media):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
# 3. SETUP
updated_at = datetime.utcnow()
data_dir = Path(current_app.config["DATA_PATH"])
@@ -815,13 +834,14 @@ def update_note(
from starpunk.webmention_sender import queue_webmentions_for_note
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
note_url = f"{site_url}/note/{updated_note.slug}"
from starpunk.utils import absolute_url_for
note_url = absolute_url_for("public.note", slug=updated_note.slug)
# Get rendered HTML from updated note (if published)
html_content = updated_note.html if updated_note.published else None
# Determine in_reply_to (use new value if provided, otherwise existing)
reply_to = in_reply_to if in_reply_to is not None else updated_note.in_reply_to
reply_to = (in_reply_to if in_reply_to is not None else updated_note.in_reply_to) or updated_note.like_of or updated_note.repost_of
# Queue webmentions (best-effort)
# Pass old_html for URL diffing (Phase 6)
@@ -838,8 +858,8 @@ def update_note(
current_app.logger.warning(f"Failed to queue webmentions for note {updated_note.slug}: {e}")
# 9. FETCH REPLY CONTEXT (v1.6.0 Phase 3)
# Fetch context if in_reply_to changed and is not None
if in_reply_to is not None:
# Fetch context if in_reply_to changed and is set ("" clears it)
if in_reply_to:
try:
from starpunk.reply_context import fetch_reply_context
db_path = Path(current_app.config["DATABASE_PATH"])
@@ -969,6 +989,17 @@ def delete_note(
"Path validation failed",
)
# Re-send webmentions so receivers learn the post is gone (spec 3.1.5).
# A hard delete would cascade the queue rows away, so detach them first.
if existing_note.published:
try:
from starpunk.webmention_sender import requeue_previous_webmentions
requeue_previous_webmentions(
Path(current_app.config["DATABASE_PATH"]), existing_note.id, detach=not soft
)
except Exception as e:
current_app.logger.warning(f"Failed to queue webmentions for deleted note: {e}")
# 5. PERFORM DELETION
db = get_db(current_app)
@@ -1082,4 +1113,10 @@ def undelete_note(slug: str) -> Note:
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication after undelete: {e}")
try:
from starpunk.webmention_sender import requeue_previous_webmentions
requeue_previous_webmentions(Path(current_app.config["DATABASE_PATH"]), row["id"])
except Exception as e:
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
return get_note(slug=slug)
+3 -3
View File
@@ -226,8 +226,8 @@ def edit_note_form(note_id: int):
flash("Note not found", "error")
return redirect(url_for("admin.dashboard")), 404
# Pre-load tags for the edit form (v1.3.0 Phase 3)
tags = get_note_tags(note.id)
# Pre-load tags for the edit form (v1.3.0 Phase 3), as entered so saving keeps the order
tags = get_note_tags(note.id, entered_order=True)
object.__setattr__(note, '_cached_tags', tags)
from starpunk.syndication import get_note_syndications, get_targets
@@ -277,7 +277,7 @@ def update_note_submit(note_id: int):
published = "published" in request.form
tags_input = request.form.get("tags", "")
if not content:
if not content and not existing_note.reacts_to: # likes/reposts may have no comment
flash("Content cannot be empty", "error")
return redirect(url_for("admin.edit_note_form", note_id=note_id))
+58 -4
View File
@@ -64,7 +64,7 @@ def _get_cached_notes():
# Cache expired or empty, fetch fresh notes
max_items = current_app.config.get("FEED_MAX_ITEMS", 50)
notes = list_notes(published_only=True, limit=max_items)
notes = list_notes(published_only=True, limit=max_items, likes=False)
# Batch load media and tags for all notes (v1.5.0 Phase 3)
# Reduces query count from O(n) to O(1) for both media and tags
@@ -249,8 +249,8 @@ def index():
from starpunk.media import get_note_media
from starpunk.tags import get_note_tags
# Get recent published notes (limit 20)
notes = list_notes(published_only=True, limit=20)
# Get recent published notes (limit 20); likes have their own page (v2.5.0)
notes = list_notes(published_only=True, limit=20, likes=False)
# Attach media and tags to each note for display
for note in notes:
@@ -262,9 +262,59 @@ def index():
tags = get_note_tags(note.id)
object.__setattr__(note, '_cached_tags', tags)
attach_contexts(notes)
return render_template("index.html", notes=notes)
def attach_contexts(notes) -> None:
"""Give likes and reposts the fetched context of the post they point at"""
from starpunk.webmention_queries import get_reply_context
for note in notes:
context = None
if note.reacts_to:
try:
context = get_reply_context(note.reacts_to)
except Exception:
context = None
object.__setattr__(note, 'context', context)
@bp.route("/likes")
def likes():
"""Likes, kept off the home page and feeds (v2.5.0)"""
from starpunk.media import get_note_media
from starpunk.tags import get_note_tags
notes = list_notes(published_only=True, limit=50, likes=True)
for note in notes:
object.__setattr__(note, 'media', get_note_media(note.id))
object.__setattr__(note, '_cached_tags', get_note_tags(note.id))
attach_contexts(notes)
return render_template("likes.html", notes=notes)
def _note_was_deleted(slug: str) -> bool:
"""
True if this slug was a note that has been deleted
Soft-deleted notes keep their row; hard-deleted notes that sent
webmentions keep their queue rows (detached from the note).
"""
from starpunk.database import get_db
from starpunk.utils import absolute_url_for
db = get_db(current_app)
if db.execute(
"SELECT 1 FROM notes WHERE slug = ? AND deleted_at IS NOT NULL", (slug,)
).fetchone():
return True
return db.execute(
"SELECT 1 FROM webmention_queue WHERE source_url = ? AND source_note_id IS NULL",
(absolute_url_for("public.note", slug=slug),),
).fetchone() is not None
@bp.route("/note/<slug>")
def note(slug: str):
"""
@@ -290,8 +340,10 @@ def note(slug: str):
# Get note by slug
note_obj = get_note(slug=slug)
# Return 404 if note doesn't exist or isn't published
# Deleted notes are 410 Gone (Webmention 3.1.5); unknown or draft 404
if not note_obj or not note_obj.published:
if not note_obj and _note_was_deleted(slug):
abort(410)
abort(404)
# Get media for note (v1.2.0 Phase 3)
@@ -334,6 +386,7 @@ def note(slug: str):
from starpunk.syndication import get_note_syndications
syndications = [s for s in get_note_syndications(note_obj.id) if s["state"] == "synced"]
attach_contexts([note_obj])
return render_template(
"note.html",
@@ -384,6 +437,7 @@ def tag(tag: str):
for note in notes:
media = get_note_media(note.id)
object.__setattr__(note, 'media', media)
attach_contexts(notes)
return render_template(
"tag.html",
+21 -13
View File
@@ -11,7 +11,7 @@ References:
"""
from flask import Blueprint, current_app, request, jsonify
from urllib.parse import urlparse
from urllib.parse import urldefrag, urlparse
from starpunk.database import get_db
from starpunk.notes import get_note, NoteNotFoundError
from starpunk.rate_limiter import check_rate_limit
@@ -107,17 +107,24 @@ def validate_url(url: str) -> tuple[bool, str | None]:
return False, "URL is malformed"
def _host_port(url: str) -> tuple[str, int | None]:
parsed = urlparse(url)
default = {'http': 80, 'https': 443}.get(parsed.scheme.lower())
try:
port = parsed.port or default
except ValueError:
port = None
return (parsed.hostname or '').lower(), port
def is_target_on_site(target_url: str) -> bool:
"""
Check if target URL is on this site.
Compares hostname of target_url with SITE_URL configuration.
Compares host (case-insensitively) and effective port with SITE_URL.
"""
site_url = current_app.config.get('SITE_URL', '')
site_parsed = urlparse(site_url)
target_parsed = urlparse(target_url)
return site_parsed.netloc == target_parsed.netloc
return _host_port(site_url) == _host_port(target_url)
def resolve_target_note(target_url: str) -> int | None:
@@ -228,16 +235,17 @@ def receive_webmention():
"error_description": "Target URL is not on this site"
}), 400
# Resolve target to note (may be None for homepage/other pages)
target_note_id = resolve_target_note(target)
# The fragment is ignored (spec 3.2.1) and not stored, so target#a and
# target#b are the same mention
target = urldefrag(target)[0]
# If target looks like a note URL but doesn't resolve, it's not found
target_parsed = urlparse(target)
target_path = target_parsed.path.rstrip('/')
if target_path.startswith('/note/') and target_note_id is None:
# Only published notes accept webmentions (spec 3.2.1: target must be a
# valid resource that accepts them)
target_note_id = resolve_target_note(target)
if target_note_id is None:
return jsonify({
"error": "invalid_target",
"error_description": "Target URL not found"
"error_description": "Target URL does not accept webmentions"
}), 400
# Insert webmention into database (queued for validation)
+710 -33
View File
@@ -1,10 +1,10 @@
"""
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066)
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066, v2.6.0 ADR-069, v2.6.3 ADR-070)
Published notes are copied to syndication targets, and later edits and
deletes follow them. Two target kinds exist: a GoToSocial (or any
Mastodon-API) account, and a Bluesky account. Bluesky posts can't be edited,
so there only deletes follow.
deletes follow them. Three target kinds exist: a GoToSocial (or any
Mastodon-API) account, a Bluesky account and a Threads account. Bluesky and
Threads posts can't be edited, so there only deletes follow.
Each (note, target) pair has a row in `syndications`. Changing a note marks
its rows pending; a background worker then reconciles the remote post with
@@ -17,9 +17,15 @@ the note:
Rows carry a version so an edit made while a send is in flight is not lost.
"""
import fcntl
import hashlib
import io
import json
import logging
import os
import re
import threading
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
@@ -28,6 +34,9 @@ from typing import Dict, List, Optional
import httpx
from flask import current_app
# httpx logs request URLs at INFO, and Threads takes its token in the URL
logging.getLogger("httpx").setLevel(logging.WARNING)
from starpunk.database import get_db
MAX_ATTEMPTS = 8
@@ -113,21 +122,58 @@ class MastodonClient:
# Interface used by the reconcile loop
can_edit = True
NATIVE = ("favourite", "reblog") # remote_id prefixes for likes/boosts (v2.5.0)
def find_status(self, url: str) -> Optional[str]:
"""Local id of a fediverse post, fetching it if needed; None if the URL isn't one"""
try:
response = self._call("GET", "/api/v2/search", params={
"q": url, "resolve": "true", "type": "statuses", "limit": 1,
})
except RemoteGone:
return None
statuses = response.json().get("statuses") or []
return statuses[0]["id"] if statuses else None
def _prepare(self, post: "Post") -> tuple:
text = compose_text(post.note, post.permalink, post.tags, post.max_chars)
if post.kind == "repost":
return compose_repost_text(post, post.note.content, post.max_chars), []
# Images in the text are sent as attachments instead (v2.6.6)
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=strip_images(post.note.content))
media_ids = [self.upload_media(f["path"], f["mime_type"], f["description"]) for f in post.media]
return text, media_ids
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
if post.kind in ("like", "repost") and post.target_url:
# Native favourite/boost when the target is a fediverse post; a
# repost with a comment is posted as a status so the comment stays
commented = bool(post.note.content.strip())
if post.kind == "like" or not commented:
status_id = self.find_status(post.target_url)
if status_id:
action = "favourite" if post.kind == "like" else "reblog"
self._call("POST", f"/api/v1/statuses/{status_id}/{action}")
return {"id": f"{action}:{status_id}", "url": None}
if post.kind == "like":
return {"id": None, "url": None} # nothing to like here
text, media_ids = self._prepare(post)
return self.create_status(text, media_ids, idempotency_key=key)
def update(self, remote_id: str, post: "Post") -> dict:
if remote_id.split(":", 1)[0] in self.NATIVE:
return {"id": remote_id, "url": None} # a favourite/boost has nothing to edit
text, media_ids = self._prepare(post)
return self.update_status(remote_id, text, media_ids)
def delete(self, remote_id: str) -> None:
action, _, status_id = remote_id.partition(":")
if action in self.NATIVE and status_id:
try:
self._call("POST", f"/api/v1/statuses/{status_id}/un{action}")
except RemoteGone:
pass
return
self.delete_status(remote_id)
@@ -229,20 +275,42 @@ class BlueskyClient:
image["aspectRatio"] = {"width": width, "height": height}
return image
def _record(self, post: "Post") -> dict:
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=plain_text(post.note.content))
record = {
"$type": self.POST,
"text": text,
"createdAt": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z",
}
facets = bluesky_facets(text)
@staticmethod
def _now() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
def _link_card(self, url: str) -> dict:
"""External embed for a page, with its title, description and image when we can get them"""
card = page_card(self.http, url)
external = {"uri": url, "title": card["title"][:300], "description": card["description"][:1000]}
if card["image"]:
try:
data, mime_type, _ = _fetch_public(self.http, card["image"], CARD_IMAGE_LIMIT,
truncate=False)
data, mime_type, _, _ = _fit_image(data, mime_type.split(";")[0] or "image/jpeg",
BLUESKY_BLOB_LIMIT)
external["thumb"] = self.upload_blob(data, mime_type)
except Exception as e: # a card without a picture is fine
current_app.logger.info(f"No link card image for {url}: {e}")
return {"$type": "app.bsky.embed.external", "external": external}
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None,
anchors: List[tuple] = ()) -> dict:
if text is None:
body, anchors = plain_text_links(post.note.content, links="anchor")
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body)
record = {"$type": self.POST, "text": text, "createdAt": self._now()}
facets = bluesky_facets(text, anchors)
if facets:
record["facets"] = facets
if post.media:
if embed is not None:
record["embed"] = embed
elif post.media:
record["embed"] = {"$type": "app.bsky.embed.images",
"images": [self._image(m) for m in post.media]}
elif note_links(post.note.content):
# The note links somewhere: the card shows that page, not the note
record["embed"] = self._link_card(note_links(post.note.content)[0])
else:
record["embed"] = {"$type": "app.bsky.embed.external", "external": {
"uri": post.permalink,
@@ -251,6 +319,56 @@ class BlueskyClient:
}}
return record
_POST_URL = re.compile(r"^https://bsky\.app/profile/([^/]+)/post/([^/?#]+)")
def resolve_post(self, url: str) -> Optional[dict]:
"""{'uri', 'cid'} of a Bluesky post URL; None if the URL isn't one"""
m = self._POST_URL.match(url or "")
if not m:
return None
actor, rkey = m.groups()
if not actor.startswith("did:"):
actor = self._call("GET", "com.atproto.identity.resolveHandle",
params={"handle": actor}).json()["did"]
uri = f"at://{actor}/{self.POST}/{rkey}"
posts = self._call("GET", "app.bsky.feed.getPosts", params={"uris": uri}).json().get("posts") or []
return {"uri": uri, "cid": posts[0]["cid"]} if posts else None
def _create(self, collection: str, record: dict) -> str:
response = self._call("POST", "com.atproto.repo.createRecord",
json={"repo": self.did or self.identifier, "collection": collection,
"record": record})
return response.json()["uri"]
def _react(self, post: "Post") -> dict:
"""Like, repost or quote a Bluesky post; fall back to a link post for reposts"""
comment, anchors = (plain_text_links(post.note.content, links="anchor")
if post.note.content.strip() else ("", []))
subject = self.resolve_post(post.target_url)
if subject and post.kind == "like":
uri = self._create("app.bsky.feed.like", {
"$type": "app.bsky.feed.like", "subject": subject, "createdAt": self._now()})
return {"id": uri, "url": None}
if subject and not comment:
uri = self._create("app.bsky.feed.repost", {
"$type": "app.bsky.feed.repost", "subject": subject, "createdAt": self._now()})
return {"id": uri, "url": None}
if subject: # repost with a comment -> quote post
text = comment if len(comment) <= post.max_chars else \
comment[:post.max_chars - 1].rsplit(" ", 1)[0].rstrip() + "…"
uri = self._create(self.POST, self._record(
post, text=text, embed={"$type": "app.bsky.embed.record", "record": subject},
anchors=anchors))
return {"id": uri, "url": self._web_url(uri)}
if post.kind == "like":
return {"id": None, "url": None} # not a Bluesky post: nothing to like
text = compose_repost_text(post, comment, post.max_chars)
embed = {"$type": "app.bsky.embed.external", "external": {
"uri": post.target_url, "title": (post.target_title or post.target_url)[:300],
"description": ""}}
uri = self._create(self.POST, self._record(post, text=text, embed=embed))
return {"id": uri, "url": self._web_url(uri)}
def _web_url(self, uri: str) -> str:
rkey = uri.rsplit("/", 1)[1]
return f"https://bsky.app/profile/{self.handle or self.did}/post/{rkey}"
@@ -265,6 +383,11 @@ class BlueskyClient:
return None
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
if post.kind in ("like", "repost") and post.target_url:
if not self.did:
with self._lock:
self._login()
return self._react(post)
if retrying:
if not self.did:
with self._lock:
@@ -289,11 +412,15 @@ class BlueskyClient:
json={"repo": repo, "collection": collection, "rkey": rkey})
def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
"""Image bytes no bigger than `limit`, re-encoded as JPEG if needed. Returns (data, mime, w, h)."""
def _fit_image(path, mime_type: str, limit: int) -> tuple:
"""
Image (a path, or bytes) no bigger than `limit`, re-encoded as JPEG if
needed. Returns (data, mime, w, h).
"""
from PIL import Image
data = Path(path).read_bytes()
data = path if isinstance(path, bytes) else Path(path).read_bytes()
name = "image" if isinstance(path, bytes) else Path(path).name
with Image.open(io.BytesIO(data)) as img:
width, height = img.size
if len(data) <= limit:
@@ -306,7 +433,355 @@ def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
copy.save(buf, format="JPEG", quality=quality, optimize=True)
if buf.tell() <= limit:
return buf.getvalue(), "image/jpeg", copy.size[0], copy.size[1]
raise SyndicationError(f"{Path(path).name}: image too large for Bluesky", permanent=True)
raise SyndicationError(f"{name}: image too large for Bluesky", permanent=True)
# --- Link cards ---------------------------------------------------------------
CARD_PAGE_LIMIT = 512 * 1024 # the <head> is near the start; read no more
CARD_IMAGE_LIMIT = 5 * 1024 * 1024
def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple:
"""
GET a public web URL, checking every redirect hop (no requests into the
LAN). Reads at most `limit` bytes: stops there if `truncate`, else fails.
Returns (data, content_type, final_url).
"""
from urllib.parse import urljoin, urlparse
from starpunk.micropub import _public_address
site = current_app.config.get("SITE_URL", "")
for _ in range(5):
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise SyndicationError(f"not a web URL: {url}", permanent=True)
if not current_app.config.get("TESTING") and not _public_address(parsed.hostname):
raise SyndicationError(f"not a public address: {url}", permanent=True)
try:
with http.stream("GET", url, follow_redirects=False, timeout=10.0,
headers={"User-Agent": f"StarPunk (+{site})"}) as response:
if response.is_redirect and "location" in response.headers:
url = urljoin(url, response.headers["location"])
continue
if response.status_code >= 400:
raise SyndicationError(f"{url}: HTTP {response.status_code}")
data = bytearray()
for chunk in response.iter_bytes():
data += chunk
if len(data) > limit:
if truncate:
break
raise SyndicationError(f"{url}: too large")
return bytes(data), response.headers.get("content-type", ""), url
except httpx.HTTPError as e:
raise SyndicationError(f"{url}: {type(e).__name__}")
raise SyndicationError(f"{url}: too many redirects")
def page_card(http: httpx.Client, url: str) -> dict:
"""
A page's title, description and image URL (Open Graph, Twitter or plain
HTML tags), for a link card. Missing parts come back empty; a page that
can't be fetched gives just the URL as the title.
"""
from urllib.parse import urljoin
from bs4 import BeautifulSoup
card = {"title": url, "description": "", "image": ""}
try:
data, content_type, final = _fetch_public(http, url, CARD_PAGE_LIMIT, truncate=True)
except SyndicationError as e:
current_app.logger.info(f"No link card details for {url}: {e}")
return card
if "html" not in content_type.lower():
return card
soup = BeautifulSoup(data, "html.parser")
def meta(*names):
for name in names:
tag = soup.find("meta", attrs={"property": name}) or soup.find("meta", attrs={"name": name})
if tag and (tag.get("content") or "").strip():
return " ".join(tag["content"].split())
return ""
title = meta("og:title", "twitter:title")
if not title and soup.title:
title = " ".join(soup.title.get_text(" ").split())
image = meta("og:image:secure_url", "og:image", "og:image:url", "twitter:image")
card.update(title=title or url,
description=meta("og:description", "twitter:description", "description"),
image=urljoin(final, image) if image else "")
return card
THREADS_API = "https://graph.threads.net/v1.0"
THREADS_IMAGE_TYPES = {"image/jpeg", "image/png"}
THREADS_IMAGE_LIMIT = 8_000_000
THREADS_MAX_LINKS = 5
THREADS_REFRESH_EVERY = 24 * 3600 # tokens last 60 days; Meta allows a refresh once one is a day old
# Graph API error codes that are temporary (rate limits, service hiccups)
_GRAPH_TRANSIENT = {1, 2, 4, 17, 32, 341, 613}
_GRAPH_AUTH = {10, 190, 200} # bad token or missing permission: retrying won't help
def threads_length(text: str) -> int:
"""Length as Threads counts it: emoji (and, to be safe, all non-ASCII) by UTF-8 bytes"""
return sum(1 if ord(ch) < 128 else len(ch.encode()) for ch in text)
def threads_topic(tags: List[str]) -> str:
"""The first tag usable as a Threads topic (one per post; no '.' or '&', 1-50 chars)"""
for tag in tags:
topic = re.sub(r"[.&]", "", tag).strip()[:50]
if topic:
return topic
return ""
class ThreadsClient:
"""
Minimal Threads API client (Meta Graph API)
Posting is two steps: create a container, wait for it to be ready, then
publish it. Meta fetches images from their public URL on this site.
Posts can't be edited through the API, so only deletes follow.
Long-lived tokens expire after 60 days unless refreshed. `maintain()`
refreshes the token once a day and keeps it in `token_file`, so it
survives restarts. A different token in the config replaces it.
"""
can_edit = False
def __init__(self, token: str, token_file: Optional[Path] = None, api: str = THREADS_API,
http: Optional[httpx.Client] = None, sleep=time.sleep):
self.config_token = token
self.token_file = Path(token_file) if token_file else None
self.api = api.rstrip("/")
self.http = http or httpx.Client(timeout=30.0)
self.sleep = sleep
self.user_id: Optional[str] = None
self.username: Optional[str] = None
self._lock = threading.Lock()
self._token = token
self._checked_at = 0.0
saved = self._read_file()
if saved:
self._token = saved["token"]
# --- Token ---
def _seed(self) -> str:
return hashlib.sha256(self.config_token.encode()).hexdigest()
def _read_file(self) -> Optional[dict]:
"""The saved token, if it descends from the configured one"""
if not self.token_file or not self.token_file.exists():
return None
try:
data = json.loads(self.token_file.read_text())
except (OSError, ValueError):
return None
return data if data.get("seed") == self._seed() and data.get("token") else None
def _write_file(self, token: str, refreshed_at: float) -> None:
tmp = self.token_file.with_suffix(".tmp")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f:
json.dump({"seed": self._seed(), "token": token, "refreshed_at": refreshed_at}, f)
os.replace(tmp, self.token_file)
def maintain(self) -> None:
"""Refresh the token if a day has passed since the last try (cheap to call often)"""
now = time.time()
if now - self._checked_at < THREADS_REFRESH_EVERY:
return
self._checked_at = now
if not self.token_file:
return
# Gunicorn workers share the file: one refreshes, the others pick up its result
self.token_file.parent.mkdir(parents=True, exist_ok=True)
with open(self.token_file.with_suffix(".lock"), "w") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
saved = self._read_file()
if saved:
self._token = saved["token"]
if now - saved.get("refreshed_at", 0) < THREADS_REFRESH_EVERY:
self._checked_at = saved["refreshed_at"]
return
base = self.api.rsplit("/", 1)[0] # refresh lives outside the versioned path
try:
response = self.http.get(f"{base}/refresh_access_token", params={
"grant_type": "th_refresh_token", "access_token": self._token})
except httpx.HTTPError as e:
current_app.logger.warning(f"Threads token refresh failed: {type(e).__name__}")
self._checked_at = now - THREADS_REFRESH_EVERY + 3600 # try again in an hour
return
if response.status_code < 400:
self._token = response.json()["access_token"]
current_app.logger.info("Threads access token refreshed")
else:
# e.g. a token less than a day old: keep it, try again tomorrow
current_app.logger.warning(
f"Threads token refresh failed: HTTP {response.status_code} {response.text[:200]}")
self._write_file(self._token, now)
# --- Graph plumbing ---
def _call(self, method: str, path: str, **params) -> dict:
params["access_token"] = self._token
kwargs = {"params": params} if method in ("GET", "DELETE") else {"data": params}
try:
response = self.http.request(method, f"{self.api}/{path}", **kwargs)
except httpx.HTTPError as e:
raise SyndicationError(f"{method} {path}: {type(e).__name__}") # never echo the URL/token
try:
body = response.json()
except ValueError:
body = {}
error = body.get("error") if isinstance(body, dict) else None
if response.status_code >= 400 or error:
error = error if isinstance(error, dict) else {}
code = error.get("code")
detail = " / ".join(str(error[k]) for k in ("error_user_title", "error_user_msg")
if error.get(k))
message = (f"{method} {path}: HTTP {response.status_code} "
f"{error.get('message') or response.text[:200]}"
f" (code {code}, subcode {error.get('error_subcode')}"
f"{', ' + detail if detail else ''}, fbtrace {error.get('fbtrace_id')})")
if response.status_code == 404 or (code == 100 and error.get("error_subcode") == 33):
raise RemoteGone(message, permanent=True)
permanent = (400 <= response.status_code < 500 and response.status_code != 429
and code not in _GRAPH_TRANSIENT)
exc = SyndicationError(message, permanent=permanent)
exc.code = code
raise exc
return body
def _me(self) -> str:
with self._lock:
if not self.user_id:
me = self._call("GET", "me", fields="id,username")
self.user_id, self.username = me["id"], me.get("username")
return self.user_id
def _wait(self, container_id: str) -> None:
"""Wait until Meta has processed a container (images take a few seconds)"""
delay = 1
self.sleep(2) # Meta can report a container ready a moment too early
for _ in range(8):
status = self._call("GET", container_id, fields="status,error_message")
state = status.get("status")
if state in ("FINISHED", None):
return
if state in ("ERROR", "EXPIRED"):
raise SyndicationError(
f"Threads container {state}: {status.get('error_message') or 'no detail'}",
permanent=True)
self.sleep(delay)
delay = min(delay * 2, 15)
raise SyndicationError("Threads container still processing")
# --- Posts ---
def _text(self, post: "Post") -> str:
# Links written "text (url)"; the first one gets Threads' preview card.
# Meta allows 5 different links per post and the permalink is one.
body = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
limit = post.max_chars
while True:
text = compose_text(post.note, post.permalink, [], limit, body=body)
over = threads_length(text) - post.max_chars
if over <= 0:
return text
limit -= over
def _image_container(self, uid: str, media: dict, **extra) -> str:
params = {"media_type": "IMAGE", "image_url": media["url"], **extra}
if media["description"]:
params["alt_text"] = media["description"][:1000]
return self._call("POST", f"{uid}/threads", **params)["id"]
def _repost_text(self, post: "Post") -> str:
comment = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
limit = post.max_chars
while True:
text = compose_repost_text(post, comment, limit)
over = threads_length(text) - post.max_chars
if over <= 0:
return text
limit -= over
def _container(self, uid: str, post: "Post") -> str:
if post.kind == "repost":
# No native repost of outside content: post "🔁 title url"
return self._call("POST", f"{uid}/threads", media_type="TEXT",
text=self._repost_text(post))["id"]
common = {"text": self._text(post)}
topic = threads_topic(post.tags)
if topic:
common["topic_tag"] = topic
# Threads takes JPEG/PNG up to 8 MB; others (GIF, WebP) are left off
images = [m for m in post.media if m["mime_type"] in THREADS_IMAGE_TYPES
and Path(m["path"]).stat().st_size <= THREADS_IMAGE_LIMIT]
if not images:
# No explicit link_attachment: Meta rejects one for a page it hasn't
# seen yet ("Invalid Link Attachment"), but previews the text's
# first link by itself without failing
return self._call("POST", f"{uid}/threads", media_type="TEXT", **common)["id"]
if len(images) == 1:
return self._image_container(uid, images[0], **common)
children = [self._image_container(uid, m, is_carousel_item="true") for m in images]
for child in children:
self._wait(child)
return self._call("POST", f"{uid}/threads", media_type="CAROUSEL",
children=",".join(children), **common)["id"]
def _profile_url(self) -> str:
return f"https://www.threads.com/@{self.username}" if self.username else "https://www.threads.com/"
def _find_existing(self, uid: str, link: str) -> Optional[dict]:
"""A recent post that already has this link (an earlier send whose reply was lost)"""
listing = self._call("GET", f"{uid}/threads", fields="id,text,permalink", limit=25)
for item in listing.get("data", []):
if link in (item.get("text") or ""):
return {"id": item["id"], "url": item.get("permalink") or self._profile_url()}
return None
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
if post.kind == "like" and post.target_url:
return {"id": None, "url": None} # the API can't like things; likes stay on the site
uid = self._me()
if retrying:
marker = post.target_url if post.kind == "repost" and post.target_url else post.permalink
existing = self._find_existing(uid, marker)
if existing:
return existing
container = self._container(uid, post)
self._wait(container)
try:
media_id = self._call("POST", f"{uid}/threads_publish", creation_id=container)["id"]
except SyndicationError as e:
if getattr(e, "code", None) in _GRAPH_AUTH:
raise
# Meta's publish step fails now and then ("Fatal") and works on a
# later try; the retry checks for an existing post first
raise SyndicationError(str(e))
try:
url = self._call("GET", media_id, fields="permalink").get("permalink")
except SyndicationError:
url = None # it's published; don't fail (and post twice) over the link
return {"id": media_id, "url": url or self._profile_url()}
def update(self, remote_id: str, post: "Post") -> dict:
raise NotImplementedError("Threads posts can't be edited through the API")
def delete(self, remote_id: str) -> None:
try:
self._call("DELETE", remote_id)
except RemoteGone:
pass # already gone
@dataclass
@@ -327,6 +802,24 @@ class Post:
tags: List[str]
max_chars: int
media: List[dict] = field(default_factory=list)
kind: str = "note" # note, like or repost (v2.5.0)
target_url: Optional[str] = None # what a like/repost points at
target_title: Optional[str] = None # its fetched title, if any
def compose_repost_text(post: "Post", comment: str, max_chars: int) -> str:
"""'comment' then '🔁 title' and the URL; the comment is shortened to fit"""
title = post.target_title or "Reposted"
tail = f"🔁 {title}\n{post.target_url}"
if len(tail) > max_chars:
tail = f"🔁 {post.target_url}"
comment = (comment or "").strip()
if not comment:
return tail
room = max_chars - len(tail) - 2
if len(comment) > room:
comment = comment[:max(room - 1, 0)].rsplit(" ", 1)[0].rstrip() + "…"
return f"{comment}\n\n{tail}" if room > 1 else tail
_client_cache: Dict[str, object] = {}
@@ -337,6 +830,7 @@ def targets_configured(config) -> bool:
return bool(
(config.get("GOTOSOCIAL_URL") and config.get("GOTOSOCIAL_ACCESS_TOKEN"))
or (config.get("BLUESKY_HANDLE") and config.get("BLUESKY_APP_PASSWORD"))
or config.get("THREADS_ACCESS_TOKEN")
)
@@ -378,6 +872,23 @@ def get_targets() -> Dict[str, Target]:
max_chars=int(config.get("BLUESKY_MAX_CHARS", 300)),
max_media=4,
)
token = config.get("THREADS_ACCESS_TOKEN") or ""
if token:
client = _client_cache.get("threads")
if client is None or client.config_token != token:
client = ThreadsClient(token, Path(config["DATA_PATH"]) / "threads_token.json",
api=config.get("THREADS_API") or THREADS_API)
_client_cache["threads"] = client
username = (config.get("THREADS_USERNAME") or "").lstrip("@")
targets["threads"] = Target(
id="threads",
uid=f"https://www.threads.com/@{username}" if username else "https://www.threads.com/",
name=config.get("THREADS_NAME", "Threads"),
client=client,
max_chars=int(config.get("THREADS_MAX_CHARS", 500)),
max_media=int(config.get("THREADS_MAX_MEDIA", 4)),
)
return targets
@@ -494,20 +1005,82 @@ def compose_text(note, permalink: str, tags: List[str], max_chars: int,
if len(body) + len(tail) > max_chars:
room = max_chars - len(tail) - 1
body = body[:room].rsplit(" ", 1)[0].rstrip() + "…"
return body + tail
# A photo post may have no text: start with the link (v2.6.5)
return (body + tail).lstrip()
_BLOCKS = ["p", "h1", "h2", "h3", "h4", "h5", "h6", "pre", "blockquote", "ul", "ol", "hr", "table"]
def plain_text(markdown_text: str) -> str:
"""Markdown as readable plain text (for targets without Markdown)"""
def _note_soup(markdown_text: str):
import markdown
from bs4 import BeautifulSoup
soup = BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
return BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
def _clean_url(url: str) -> str:
return url.rstrip(".,;:!?)\"'…")
def _is_web(url: str) -> bool:
return url.startswith(("http://", "https://"))
def note_links(markdown_text: str) -> List[str]:
"""Web links in a note, in reading order: Markdown links, <autolinks> and bare URLs"""
from bs4 import NavigableString
links = []
for node in _note_soup(markdown_text).descendants:
if getattr(node, "name", None) == "a" and _is_web(node.get("href", "")):
links.append(node["href"])
elif isinstance(node, NavigableString) and not node.find_parent("a"):
links.extend(_clean_url(m.group(0)) for m in _URL_RE.finditer(str(node)))
return list(dict.fromkeys(links))
def plain_text(markdown_text: str, links: str = "text", max_urls: Optional[int] = None) -> str:
"""Markdown as readable plain text (for targets without Markdown). See `plain_text_links`."""
return plain_text_links(markdown_text, links, max_urls)[0]
def plain_text_links(markdown_text: str, links: str = "text",
max_urls: Optional[int] = None) -> tuple:
"""
Markdown as plain text, and the links whose text stands in for a URL.
A Markdown link `[text](url)` becomes:
links="text" just its text
links="inline" "text (url)", for targets with no clickable link text;
after `max_urls` different URLs, just the text
links="anchor" its text, listed in the returned anchors as (text, url)
so the target can make that text clickable
A link whose text is the URL itself stays a bare URL.
"""
from bs4 import NavigableString
soup = _note_soup(markdown_text)
for img in soup.find_all("img"):
img.decompose()
anchors = []
seen = set()
for node in list(soup.descendants): # bare URLs count towards max_urls first
if isinstance(node, NavigableString) and not node.find_parent("a"):
seen.update(_clean_url(m.group(0)) for m in _URL_RE.finditer(str(node)))
for a in soup.find_all("a"):
href, label = a.get("href", ""), a.get_text()
if not _is_web(href):
continue
bare = re.sub(r"^https?://", "", href).rstrip("/")
if label.strip().rstrip("/") in (href.rstrip("/"), bare):
a.replace_with(href)
seen.add(href)
elif links == "inline" and (href in seen or max_urls is None or len(seen) < max_urls):
a.replace_with(f"{label} ({href})")
seen.add(href)
elif links == "anchor":
anchors.append((label, href))
for node in soup.find_all(string=True):
if not node.strip() and "\n" in node and node.parent.name != "pre":
node.extract() # layout newlines between tags; we add our own
@@ -520,24 +1093,41 @@ def plain_text(markdown_text: str) -> str:
block.append("\n\n")
text = soup.get_text()
text = re.sub(r"[ \t]+\n", "\n", text)
return re.sub(r"\n{3,}", "\n\n", text).strip()
return re.sub(r"\n{3,}", "\n\n", text).strip(), anchors
_URL_RE = re.compile(r"https?://[^\s<>\"]+")
_TAG_RE = re.compile(r"(?<![\w#])#(\w*[^\W\d]\w*)")
def bluesky_facets(text: str) -> List[dict]:
"""Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post"""
def bluesky_facets(text: str, anchors: List[tuple] = ()) -> List[dict]:
"""
Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post.
`anchors` are (text, url) pairs: that text, where it appears, links to url.
"""
def span(start: int, end: int) -> dict:
return {"byteStart": len(text[:start].encode()), "byteEnd": len(text[:end].encode())}
facets = []
for m in _URL_RE.finditer(text):
url = m.group(0).rstrip(".,;:!?)\"'…")
url = _clean_url(m.group(0))
facets.append({"index": span(m.start(), m.start() + len(url)),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
taken = [(m.start(), m.end()) for m in _URL_RE.finditer(text)]
pos = 0
for label, url in anchors:
start = text.find(label, pos) if label.strip() else -1
if start < 0:
continue # shortened away
end = start + len(label)
if any(a < end and start < b for a, b in taken):
continue
taken.append((start, end))
pos = end
facets.append({"index": span(start, end),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
urls = [(f["index"]["byteStart"], f["index"]["byteEnd"]) for f in facets]
facets.sort(key=lambda f: f["index"]["byteStart"])
for m in _TAG_RE.finditer(text):
index = span(m.start(), m.end())
if any(a <= index["byteStart"] < b for a, b in urls):
@@ -547,23 +1137,84 @@ def bluesky_facets(text: str) -> List[dict]:
return facets
def _media_files(note_id: int, limit: int) -> List[dict]:
"""Image files for a note: the 'large' variant when there is one"""
_MD_IMAGE_RE = re.compile(r"!\[[^\]]*\]\([^)]*\)")
_HTML_IMAGE_RE = re.compile(r"<img\b[^>]*>", re.IGNORECASE)
def inline_images(markdown_text: str) -> List[tuple]:
"""(url, alt) for each image embedded in a note's text, in reading order"""
images = []
for img in _note_soup(markdown_text).find_all("img"):
if img.get("src"):
images.append((img["src"], img.get("alt") or ""))
return images
def strip_images(markdown_text: str) -> str:
"""A note's Markdown without its embedded images"""
text = _HTML_IMAGE_RE.sub("", _MD_IMAGE_RE.sub("", markdown_text))
return re.sub(r"\n{3,}", "\n\n", text).strip()
def _own_media_path(url: str) -> Optional[str]:
"""Path in the media library for one of this site's /media/ URLs"""
site = current_app.config["SITE_URL"].rstrip("/")
if url.startswith(site):
url = url[len(site):]
return url[len("/media/"):] if url.startswith("/media/") else None
def _media_files(note_id: int, limit: int, content: str = "") -> List[dict]:
"""
Image files for a note: the 'large' variant when there is one.
Attached photos come first, then images embedded in the text that are in
this site's media library. Some Micropub clients (OutPost, for one)
upload a photo and put it in the content rather than in `photo` (v2.6.6).
"""
from starpunk.media import get_note_media
media_dir = Path(current_app.config["DATA_PATH"]) / "media"
site = current_app.config["SITE_URL"].rstrip("/")
items = [(m, m.get("caption") or "") for m in get_note_media(note_id)]
seen = {m["path"] for m, _ in items}
for url, alt in inline_images(content) if content else []:
path = _own_media_path(url)
if not path or path in seen:
continue
media = _media_by_path(path)
if media:
seen.add(path)
items.append((media, alt or media.get("caption") or ""))
files = []
for media in get_note_media(note_id)[:limit]:
for media, description in items[:limit]:
variant = (media.get("variants") or {}).get("large")
rel = variant["path"] if variant else media["path"]
files.append({
"path": media_dir / rel,
"url": f"{site}/media/{rel}", # for targets that fetch images themselves
"mime_type": media.get("mime_type") or "image/jpeg",
"description": media.get("caption") or "",
"description": description,
})
return files
def _media_by_path(path: str) -> Optional[dict]:
"""A media row with its variants, found by its path in the library"""
db = get_db(current_app)
row = db.execute("SELECT id, path, mime_type FROM media WHERE path = ?", (path,)).fetchone()
if not row:
return None
variants = {
v["variant_type"]: {"path": v["path"]}
for v in db.execute(
"SELECT variant_type, path FROM media_variants WHERE media_id = ?", (row["id"],)
)
}
return {"path": row["path"], "mime_type": row["mime_type"], "variants": variants, "caption": ""}
# --- Worker -----------------------------------------------------------------
def _claim(db, row_id: int) -> bool:
@@ -615,6 +1266,21 @@ def _fail(db, row, error: SyndicationError) -> None:
)
def _target_title(url: Optional[str]) -> Optional[str]:
"""Fetched title of a liked/reposted post, if we have one"""
if not url:
return None
try:
from starpunk.webmention_queries import get_reply_context
context = get_reply_context(url) or {}
except Exception:
return None
title = context.get("title") or ""
if not title and context.get("author_name"):
title = f"Post by {context['author_name']}"
return " ".join(title.split())[:200] or None
def _reconcile(row, target: Target) -> tuple:
"""Bring the remote post in line with the note. Returns (remote_id, remote_url)."""
from starpunk.notes import get_note
@@ -635,9 +1301,13 @@ def _reconcile(row, target: Target) -> tuple:
post = Post(
note=note,
permalink=f"{site}/note/{note.slug}",
tags=[t["display_name"] for t in get_note_tags(note.id)],
tags=[t["display_name"] for t in get_note_tags(note.id, entered_order=True)],
max_chars=target.max_chars,
media=_media_files(note.id, target.max_media),
media=_media_files(note.id, target.max_media, note.content)
if note.kind == "note" else [],
kind=note.kind,
target_url=note.reacts_to,
target_title=_target_title(note.reacts_to),
)
if row["remote_id"]:
@@ -658,6 +1328,13 @@ def process_due(limit: int = 5) -> int:
targets = get_targets()
if not targets:
return 0
for target in targets.values():
maintain = getattr(target.client, "maintain", None)
if maintain:
try:
maintain()
except Exception as e:
current_app.logger.error(f"Syndication {target.id} upkeep failed: {e}")
db = get_db(current_app)
rows = db.execute(
"""
+11 -6
View File
@@ -115,27 +115,32 @@ def add_tags_to_note(note_id: int, tags: list[str]) -> None:
db.commit()
def get_note_tags(note_id: int) -> list[dict]:
def get_note_tags(note_id: int, entered_order: bool = False) -> list[dict]:
"""
Get all tags for a note
Args:
note_id: Note database ID
entered_order: In the order they were entered (the first is the
note's main tag, e.g. its Threads topic) instead of alphabetical.
add_tags_to_note() rewrites the links in entry order on every
save, so note_tags.id records it. Use this wherever the tags are
saved back (editor, Micropub update) so the order survives.
Returns:
List of tag dicts with 'name' and 'display_name'
Ordered alphabetically by display_name (case-insensitive)
Query should use: ORDER BY LOWER(tags.display_name) ASC
Ordered alphabetically by display_name (case-insensitive), unless
entered_order
"""
db = get_db(current_app)
order = "note_tags.id" if entered_order else "LOWER(tags.display_name)"
tags = db.execute(
"""
f"""
SELECT tags.name, tags.display_name
FROM tags
JOIN note_tags ON tags.id = note_tags.tag_id
WHERE note_tags.note_id = ?
ORDER BY LOWER(tags.display_name) ASC
ORDER BY {order} ASC
""",
(note_id,)
).fetchall()
+127 -88
View File
@@ -14,15 +14,18 @@ References:
- W3C Webmention: https://www.w3.org/TR/webmention/
"""
import json
import logging
import mf2py
import nh3
import httpx
from datetime import datetime
from typing import Dict, Any, Optional
from urllib.parse import urlparse
from urllib.parse import urldefrag, urlparse
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
from starpunk.webmention_ssrf import (
FetchTooLargeError, SSRFBlockedError, safe_get, user_agent,
)
logger = logging.getLogger(__name__)
@@ -33,103 +36,113 @@ class ValidationError(Exception):
pass
def fetch_source(url: str, site_url: str, timeout: int = 10, max_size: int = 1048576) -> tuple[str, str]:
ACCEPT = ('text/html, application/xhtml+xml;q=0.9, application/json;q=0.8, '
'text/plain;q=0.7, */*;q=0.1')
class TransientError(ValidationError):
"""Source could not be checked right now (network error, 5xx)"""
pass
def fetch_source(url: str, site_url: str) -> tuple[str, str, str]:
"""
Fetch source URL content with SSRF protection.
Per spec 3.2.2: GET following redirects (limited), with an Accept header.
Args:
url: Source URL to fetch
site_url: Site URL for User-Agent header
timeout: Request timeout in seconds (default: 10)
max_size: Maximum content size in bytes (default: 1MB)
Returns:
Tuple of (content, content_type)
Tuple of (content, content_type, final_url)
Raises:
SSRFBlockedError: If URL points to private IP
ValidationError: If fetch fails or content too large
ValidationError: Source is blocked, too large, or returned 4xx
TransientError: Network error, timeout or 5xx
"""
# SSRF protection
try:
validate_url_for_fetch(url)
result = safe_get(url, headers={'User-Agent': user_agent(site_url), 'Accept': ACCEPT})
except (SSRFBlockedError, ValueError) as e:
logger.warning(f"SSRF protection blocked URL {url}: {e}")
raise ValidationError(f"Blocked request: {e}")
# Fetch URL
try:
# Use streaming to check size before downloading entire content
with httpx.stream(
'GET',
url,
timeout=timeout,
headers={'User-Agent': f'StarPunk/1.6.0 (+{site_url})'},
follow_redirects=True
) as response:
response.raise_for_status()
# Check content length if provided
content_length = response.headers.get('Content-Length')
if content_length and int(content_length) > max_size:
raise ValidationError(f"Source exceeds size limit ({max_size} bytes)")
# Read content with size limit
content = b''
for chunk in response.iter_bytes(chunk_size=8192):
content += chunk
if len(content) > max_size:
raise ValidationError(f"Source exceeds size limit ({max_size} bytes)")
content_type = response.headers.get('Content-Type', '')
return content.decode('utf-8', errors='replace'), content_type
except FetchTooLargeError as e:
raise ValidationError(f"Source {e}")
except httpx.TimeoutException:
raise ValidationError("Source fetch timed out")
except httpx.HTTPStatusError as e:
if 400 <= e.response.status_code < 500:
raise ValidationError(f"Source returned {e.response.status_code}")
else:
raise ValidationError(f"Source server error: {e.response.status_code}")
raise TransientError("Source fetch timed out")
except httpx.HTTPError as e:
raise ValidationError(f"Failed to fetch source: {e}")
except UnicodeDecodeError:
raise ValidationError("Source content encoding is not supported")
raise TransientError(f"Failed to fetch source: {e}")
if 400 <= result.status_code < 500:
raise ValidationError(f"Source returned {result.status_code}")
if result.status_code >= 300:
raise TransientError(f"Source server error: {result.status_code}")
return result.text, result.headers.get('Content-Type', ''), result.url
def verify_link_presence(content: str, target_url: str, content_type: str) -> bool:
def _strip_fragment(url: str) -> str:
return urldefrag(url.strip())[0]
def _json_strings(value):
"""Yield every string in a parsed JSON document"""
if isinstance(value, str):
yield value
elif isinstance(value, dict):
for item in value.values():
yield from _json_strings(item)
elif isinstance(value, list):
for item in value:
yield from _json_strings(item)
def verify_link_presence(content: str, target_url: str, content_type: str,
source_url: str = '') -> bool:
"""
Verify that source content contains a link to target URL.
Verify the source links to the target (spec 3.2.2).
The source MUST have an exact match of the target URL, checked per media
type: in HTML, any link-bearing attribute (a/link href, img/video/audio
src, ...) resolved against the source URL; in JSON, any string value; in
other text, the URL anywhere. Fragments are ignored on both sides.
Args:
content: Source content (HTML or text)
content: Source content
target_url: Target URL to find
content_type: Content type of source
source_url: Final source URL, for resolving relative links
Returns:
True if target link found, False otherwise
True if the target is linked, False otherwise
"""
# Normalize URLs for comparison (remove fragments, trailing slashes)
target_normalized = urlparse(target_url)._replace(fragment='').geturl().rstrip('/')
target = _strip_fragment(target_url)
media_type = content_type.split(';')[0].strip().lower()
# For HTML, parse and check all links
if 'html' in content_type.lower():
if 'html' in media_type:
from lxml import html as lxml_html
try:
from lxml import html as lxml_html
doc = lxml_html.fromstring(content)
links = doc.xpath('//a/@href')
for link in links:
# Make absolute if relative
link_normalized = urlparse(link)._replace(fragment='').geturl().rstrip('/')
if link_normalized == target_normalized:
return True
doc = lxml_html.document_fromstring(content)
except Exception as e:
logger.debug(f"HTML parsing failed, falling back to string search: {e}")
logger.debug(f"HTML parsing failed: {e}")
return False
if source_url:
doc.make_links_absolute(source_url, resolve_base_href=True,
handle_failures='ignore')
return any(_strip_fragment(link) == target for _, _, link, _ in doc.iterlinks())
# Fallback: simple string search
return target_normalized in content
if media_type == 'application/json' or media_type.endswith('+json'):
try:
data = json.loads(content)
except ValueError:
return False
return any(_strip_fragment(s) == target for s in _json_strings(data))
if media_type.startswith('text/'):
return target in content
return False
def detect_mention_type(parsed_mf2: Dict, target_url: str) -> str:
@@ -182,6 +195,28 @@ def detect_mention_type(parsed_mf2: Dict, target_url: str) -> str:
return 'mention'
def _first_str(props: Dict, name: str) -> Optional[str]:
"""
First value of an mf2 property as a string
mf2py gives u-photo with alt text as {'value': url, 'alt': ...} and nested
microformats as dicts with a 'value'; reduce those to the plain value.
"""
values = props.get(name) or [None]
value = values[0]
if isinstance(value, dict):
value = value.get('value')
return value if isinstance(value, str) and value else None
def _is_http_url(value: Optional[str]) -> bool:
"""True if value is an absolute http or https URL"""
if not value:
return False
parsed = urlparse(value.strip())
return parsed.scheme.lower() in ('http', 'https') and bool(parsed.netloc)
def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]]:
"""
Extract author information from h-card.
@@ -212,26 +247,29 @@ def extract_author(parsed_mf2: Dict, source_url: str) -> Dict[str, Optional[str]
if isinstance(author_data, dict):
# h-card object
props = author_data.get('properties', {})
author_name = props.get('name', [None])[0]
author_url = props.get('url', [None])[0]
author_photo = props.get('photo', [None])[0]
author_name = _first_str(props, 'name')
author_url = _first_str(props, 'url')
author_photo = _first_str(props, 'photo')
# If no author in h-entry, find representative h-card
if not author_name:
for item in parsed_mf2.get('items', []):
if 'h-card' in item.get('type', []):
props = item.get('properties', {})
author_name = props.get('name', [None])[0]
author_url = props.get('url', [None])[0]
author_photo = props.get('photo', [None])[0]
author_name = _first_str(props, 'name')
author_url = _first_str(props, 'url')
author_photo = _first_str(props, 'photo')
break
# Fallback to hostname if no author found
if not author_name:
author_name = urlparse(source_url).netloc
if not author_url:
# Only http(s) URLs are rendered as links (blocks javascript: and data:)
if not _is_http_url(author_url):
author_url = source_url
if not _is_http_url(author_photo):
author_photo = None
return {
'author_name': author_name,
@@ -267,12 +305,7 @@ def extract_content(parsed_mf2: Dict) -> Optional[str]:
content = content_data
if content:
# Sanitize HTML
sanitized = sanitize_content(content)
# Truncate to 500 chars
if len(sanitized) > 500:
sanitized = sanitized[:497] + '...'
return sanitized
return _snippet(content)
# Try p-summary as fallback
summary = props.get('summary', [])
@@ -281,14 +314,20 @@ def extract_content(parsed_mf2: Dict) -> Optional[str]:
if isinstance(content, dict):
content = content.get('value', '')
if content:
sanitized = sanitize_content(content)
if len(sanitized) > 500:
sanitized = sanitized[:497] + '...'
return sanitized
return _snippet(content)
return None
def _snippet(html: str, limit: int = 500) -> str:
"""Sanitized HTML of at most ~limit characters, with tags closed"""
sanitized = sanitize_content(html)
if len(sanitized) > limit:
# Cutting can leave a tag or attribute open; sanitizing again closes it
sanitized = sanitize_content(sanitized[:limit - 3] + '...')
return sanitized
def sanitize_content(html: str) -> str:
"""
Sanitize HTML content using nh3.
@@ -337,17 +376,17 @@ def process_webmention(source_url: str, target_url: str, site_url: str) -> Dict[
ValidationError: If validation fails with specific error
"""
# Fetch source content
content, content_type = fetch_source(source_url, site_url)
content, content_type, final_url = fetch_source(source_url, site_url)
# Verify target link presence
if not verify_link_presence(content, target_url, content_type):
if not verify_link_presence(content, target_url, content_type, final_url):
raise ValidationError("Source does not link to target")
# Parse microformats (only for HTML)
parsed_mf2 = None
if 'html' in content_type.lower():
try:
parsed_mf2 = mf2py.parse(doc=content, url=source_url)
parsed_mf2 = mf2py.parse(doc=content, url=final_url)
except Exception as e:
logger.debug(f"Microformat parsing failed for {source_url}: {e}")
+80 -61
View File
@@ -14,9 +14,8 @@ References:
import logging
import sqlite3
from datetime import datetime
from pathlib import Path
from typing import Optional, Set
from typing import Optional
from lxml import html as lxml_html
from urllib.parse import urlparse
@@ -45,16 +44,16 @@ def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True)
return []
try:
# Parse HTML
doc = lxml_html.fromstring(html_content)
# Parse HTML inside a wrapper, so a top-level <pre> can be removed too
doc = lxml_html.fragment_fromstring(html_content, create_parent='div')
if exclude_code_blocks:
# Remove <pre> and <code> elements before extracting links
for element in doc.xpath('//pre | //code'):
element.getparent().remove(element)
for element in doc.xpath('.//pre | .//code'):
element.drop_tree()
# Extract all <a href> attributes in document order
links = doc.xpath('//a/@href')
links = doc.xpath('.//a/@href')
# Filter to valid absolute URLs (http/https)
absolute_links = []
@@ -70,6 +69,66 @@ def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True)
return []
def _dedupe(urls: list) -> list:
"""Drop empty and repeated URLs, keeping the first occurrence"""
seen = set()
result = []
for url in urls:
if url and url not in seen:
seen.add(url)
result.append(url)
return result
def previous_targets(db_path: Path, note_id: int) -> list[str]:
"""Targets this note has been queued to send webmentions to before"""
conn = sqlite3.connect(db_path)
try:
rows = conn.execute(
"SELECT target_url FROM webmention_queue WHERE source_note_id = ? ORDER BY id",
(note_id,),
).fetchall()
finally:
conn.close()
return [row[0] for row in rows]
def requeue_previous_webmentions(db_path: Path, note_id: int, detach: bool = False) -> int:
"""
Re-send every webmention this note has sent (spec 3.1.5 deleted posts)
Args:
db_path: Path to SQLite database
note_id: Note being deleted or restored
detach: Clear source_note_id so the rows survive a hard delete
(the foreign key cascades)
Returns:
Number of webmentions queued
"""
conn = sqlite3.connect(db_path)
try:
cursor = conn.execute(f"""
UPDATE webmention_queue
SET status = 'pending',
retry_count = 0,
next_retry_at = NULL,
last_error = NULL,
endpoint_url = NULL,
created_at = CURRENT_TIMESTAMP
{', source_note_id = NULL' if detach else ''}
WHERE source_note_id = ?
""", (note_id,))
conn.commit()
queued = cursor.rowcount
finally:
conn.close()
if queued:
logger.info(f"Re-queued {queued} webmentions for note {note_id}")
return queued
def queue_webmentions_for_note(
db_path: Path,
note_id: int,
@@ -83,10 +142,9 @@ def queue_webmentions_for_note(
Per architect's answers (Phase 6):
- Called from notes.py AFTER successful database commit
- For updates: diff old vs new URLs, queue for new/existing/removed
- For creates: queue for all URLs in content
- Max 10 webmentions per operation
- Priority: in_reply_to URL first, then document order
- For creates: reply target first, then links in document order (max 10)
- For updates: the same, plus every previously sent target and links
removed by the edit (spec 3.1.4)
- Webmention sending is best-effort; failures do not affect note creation
Args:
@@ -100,58 +158,19 @@ def queue_webmentions_for_note(
Returns:
Number of webmentions queued
"""
# Phase 6: Implement URL diffing for updates
# Targets for the current content: reply target first, then links in
# document order, at most 10
links = extract_links_from_html(content_html) if content_html else []
current = _dedupe(([in_reply_to] if in_reply_to else []) + links)[:10]
if old_content_html is not None:
# This is an update - compute URL diff
old_links = set(extract_links_from_html(old_content_html))
# Handle unpublishing case (content_html is None)
if content_html is None:
# Unpublishing - queue for all old URLs (they're all removed)
new_links = set()
else:
new_links = set(extract_links_from_html(content_html))
# Compute sets for URL diffing
new_urls = new_links - old_links # In new, not in old
existing_urls = new_links & old_links # In both (re-validate)
removed_urls = old_links - new_links # In old, not in new
# Build target list: new + existing + removed
# (All need webmentions per W3C spec)
all_update_urls = list(new_urls) + list(existing_urls) + list(removed_urls)
# Add in_reply_to if changed
target_urls = []
if in_reply_to and in_reply_to not in all_update_urls:
target_urls.append(in_reply_to)
# Add the diffed URLs
target_urls.extend(all_update_urls)
# Limit to 10
target_urls = target_urls[:10]
# Update (spec 3.1.4): also re-send to every target notified before,
# and to links the edit removed, so they can update or remove the
# mention. These are not capped.
removed = [u for u in extract_links_from_html(old_content_html) if u not in links]
target_urls = _dedupe(current + previous_targets(db_path, note_id) + removed)
else:
# This is a create (no old content) - use original logic
links = extract_links_from_html(content_html) if content_html else []
# Build priority-ordered list of target URLs
target_urls = []
# 1. in_reply_to gets highest priority
if in_reply_to:
target_urls.append(in_reply_to)
# 2. Add extracted links in document order (deduplicated)
seen = {in_reply_to} if in_reply_to else set()
for link in links:
if link not in seen:
target_urls.append(link)
seen.add(link)
# 3. Limit to 10 webmentions per operation
target_urls = target_urls[:10]
target_urls = current
if not target_urls:
logger.debug(f"No links to send webmentions for note {note_id}")
+142 -52
View File
@@ -1,113 +1,203 @@
"""
SSRF protection utilities for webmention processing
SSRF protection for webmention fetches
Per ADR-063 Decision 9 and background-worker.md:
- Block private IP address ranges
- Resolve hostname before making requests
Per ADR-063 Decision 9 and Webmention sections 4.2, 4.3 and 4.5:
- Every address a hostname resolves to must be public
- Redirects are followed by hand so each hop is checked
- Redirect count, response size and total time are limited
- DEV_MODE exception for localhost testing
- Applied to all outgoing HTTP requests
References:
- ADR-063: Webmention Implementation
- Background Worker Design: /docs/design/v1.6.0/background-worker.md
- W3C Webmention: https://www.w3.org/TR/webmention/#security-considerations
"""
import ipaddress
import socket
import time
from dataclasses import dataclass
from urllib.parse import urljoin, urlparse
import httpx
from flask import current_app
MAX_REDIRECTS = 10
MAX_BYTES = 1024 * 1024
TIMEOUT = 10
# Tests can set an httpx transport here (e.g. httpx.MockTransport)
TRANSPORT = None
CGNAT = ipaddress.ip_network('100.64.0.0/10')
THIS_NETWORK = ipaddress.ip_network('0.0.0.0/8')
class SSRFBlockedError(Exception):
"""Raised when SSRF protection blocks a request"""
pass
# Private IP ranges per RFC 1918 and link-local ranges
BLOCKED_NETWORKS = [
ipaddress.ip_network('10.0.0.0/8'), # RFC 1918 private
ipaddress.ip_network('172.16.0.0/12'), # RFC 1918 private
ipaddress.ip_network('192.168.0.0/16'), # RFC 1918 private
ipaddress.ip_network('127.0.0.0/8'), # Localhost
ipaddress.ip_network('169.254.0.0/16'), # Link-local
ipaddress.ip_network('::1/128'), # IPv6 localhost
ipaddress.ip_network('fc00::/7'), # IPv6 unique local
ipaddress.ip_network('fe80::/10'), # IPv6 link-local
]
class FetchTooLargeError(Exception):
"""Raised when a response is larger than the limit"""
pass
def user_agent(site_url: str) -> str:
"""User-Agent for webmention requests (spec 3.1.2 suggests including 'Webmention')"""
from starpunk import __version__
return f"StarPunk/{__version__} Webmention (+{site_url})"
def is_private_ip(ip: str) -> bool:
"""
Check if IP address is in a private/blocked range.
Check if an IP address must not be fetched.
Blocks private, loopback, link-local, multicast, reserved, unspecified,
carrier-grade NAT and IPv4-mapped forms of these.
Args:
ip: IP address string (IPv4 or IPv6)
Returns:
True if IP is private/blocked, False otherwise
True if IP is blocked, False otherwise
"""
try:
addr = ipaddress.ip_address(ip)
return any(addr in network for network in BLOCKED_NETWORKS)
addr = ipaddress.ip_address(ip.split('%')[0])
except ValueError:
# Invalid IP format, treat as blocked
return True
if isinstance(addr, ipaddress.IPv6Address) and addr.ipv4_mapped:
addr = addr.ipv4_mapped
if isinstance(addr, ipaddress.IPv4Address) and (addr in CGNAT or addr in THIS_NETWORK):
return True
return (
addr.is_private
or addr.is_loopback
or addr.is_link_local
or addr.is_multicast
or addr.is_reserved
or addr.is_unspecified
)
def safe_resolve(hostname: str) -> str:
"""
Resolve hostname and validate IP is not private.
Resolve hostname and check every address is public.
Per background-worker.md: DEV_MODE exception for localhost testing.
Args:
hostname: Hostname to resolve
Returns:
Resolved IP address if safe
First resolved IP address
Raises:
SSRFBlockedError: If hostname resolves to private IP
SSRFBlockedError: If any address is blocked
socket.gaierror: If hostname cannot be resolved
"""
# DEV_MODE exception for localhost
if current_app.config.get('DEV_MODE') and hostname in ('localhost', '127.0.0.1', '::1'):
return '127.0.0.1'
# Resolve hostname
ip = socket.gethostbyname(hostname)
addresses = sorted({info[4][0] for info in socket.getaddrinfo(hostname, None)})
for ip in addresses:
if is_private_ip(ip):
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
# Check if resolved IP is private
if is_private_ip(ip):
raise SSRFBlockedError(f"Blocked request to private IP: {ip}")
return ip
return addresses[0]
def validate_url_for_fetch(url: str) -> None:
"""
Validate URL is safe to fetch (not private IP).
Args:
url: URL to validate
Validate URL is safe to fetch (http(s), public addresses only).
Raises:
SSRFBlockedError: If URL points to private IP
ValueError: If URL is malformed
SSRFBlockedError: If URL points to a blocked address
ValueError: If URL is malformed or its host does not resolve
"""
from urllib.parse import urlparse
parsed = urlparse(url)
if not parsed.scheme in ('http', 'https'):
if parsed.scheme not in ('http', 'https'):
raise ValueError(f"Unsupported URL scheme: {parsed.scheme}")
if not parsed.netloc:
if not parsed.hostname:
raise ValueError("URL must include a hostname")
# Extract hostname (remove port if present)
hostname = parsed.hostname
if not hostname:
raise ValueError("Could not extract hostname from URL")
try:
safe_resolve(parsed.hostname)
except socket.gaierror as e:
raise ValueError(f"Could not resolve {parsed.hostname}: {e}")
# Validate hostname resolves to public IP
safe_resolve(hostname)
@dataclass
class FetchResult:
"""A fetched response, after redirects"""
url: str
status_code: int
headers: httpx.Headers
content: bytes
encoding: str
@property
def text(self) -> str:
try:
return self.content.decode(self.encoding, errors='replace')
except LookupError:
return self.content.decode('utf-8', errors='replace')
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
"""
GET a URL, following redirects by hand and checking each hop.
Args:
url: URL to fetch
headers: Request headers
max_bytes: Response body limit
max_redirects: Redirect limit
timeout: Total time limit in seconds
Returns:
FetchResult for the final response (any status code)
Raises:
SSRFBlockedError, ValueError: URL or a redirect target is not allowed
FetchTooLargeError: Body exceeds max_bytes
httpx.HTTPError: Network errors, too many redirects, time limit
"""
deadline = time.monotonic() + timeout
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client:
for _ in range(max_redirects + 1):
validate_url_for_fetch(url)
remaining = deadline - time.monotonic()
if remaining <= 0:
raise httpx.TimeoutException("Fetch exceeded time limit")
with client.stream('GET', url, headers=headers, timeout=remaining) as response:
location = response.headers.get('Location')
if response.is_redirect and location:
url = urljoin(url, location)
continue
content_length = response.headers.get('Content-Length', '')
if content_length.isdigit() and int(content_length) > max_bytes:
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
content = b''
for chunk in response.iter_bytes(chunk_size=8192):
content += chunk
if len(content) > max_bytes:
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
if time.monotonic() > deadline:
raise httpx.TimeoutException("Fetch exceeded time limit")
return FetchResult(
url=url,
status_code=response.status_code,
headers=response.headers,
content=content,
encoding=response.charset_encoding or 'utf-8',
)
raise httpx.TooManyRedirects(f"More than {max_redirects} redirects")
+174 -147
View File
@@ -12,30 +12,32 @@ References:
- ADR-063: Webmention Implementation
"""
import fcntl
import logging
import re
import threading
import time
import sqlite3
import httpx
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, Any, Optional, TypedDict
from urllib.parse import urlparse
from typing import Dict, Any, Optional
from urllib.parse import urljoin
from lxml import html as lxml_html
from starpunk.webmention_processor import process_webmention, ValidationError
from starpunk.webmention_ssrf import validate_url_for_fetch, SSRFBlockedError
from flask import has_app_context
from starpunk.webmention_processor import process_webmention, TransientError, ValidationError
from starpunk.webmention_ssrf import (
FetchTooLargeError, SSRFBlockedError, safe_get, user_agent, validate_url_for_fetch,
)
logger = logging.getLogger(__name__)
# Link header (RFC 8288): <uri> then ;-separated params; quoted values may
# contain , and ;
LINK_RE = re.compile(r'<([^>]*)>((?:\s*;\s*[^;,"=\s]+\s*(?:=\s*(?:"[^"]*"|[^;,]*))?)*)')
LINK_PARAM_RE = re.compile(r';\s*([^;,"=\s]+)\s*(?:=\s*(?:"([^"]*)"|([^;,]*)))?')
# Discovery cache entry type
class DiscoveryCacheEntry(TypedDict):
"""Cache entry for endpoint discovery results"""
endpoint: Optional[str] # None = negative cache (no endpoint found)
expires_at: datetime
class WebmentionWorker(threading.Thread):
@@ -47,6 +49,7 @@ class WebmentionWorker(threading.Thread):
def __init__(
self,
app,
db_path: Path,
site_url: str,
poll_interval: int = 10,
@@ -58,6 +61,7 @@ class WebmentionWorker(threading.Thread):
Initialize webmention worker thread
Args:
app: Flask app (the SSRF checks read its config)
db_path: Path to SQLite database
site_url: Site URL for User-Agent header
poll_interval: Seconds between queue checks (default: 10)
@@ -66,6 +70,7 @@ class WebmentionWorker(threading.Thread):
auto_approve: Auto-approve valid webmentions (default: True)
"""
super().__init__(daemon=True, name="webmention-worker")
self.app = app
self.db_path = db_path
self.site_url = site_url
self.poll_interval = poll_interval
@@ -74,10 +79,7 @@ class WebmentionWorker(threading.Thread):
self.auto_approve = auto_approve
self._stop_event = threading.Event()
self._last_processed = None
# Phase 3: Discovery cache (worker instance variable)
# Per architect's answer to Q4
self._discovery_cache: Dict[str, DiscoveryCacheEntry] = {}
self._lock_file = None
def run(self):
"""
@@ -92,19 +94,23 @@ class WebmentionWorker(threading.Thread):
try:
logger.info("Webmention worker started")
# Cleanup orphaned items from previous crashes
self._cleanup_orphaned_items()
# Main processing loop
while not self._stop_event.is_set():
# Every gunicorn process starts a worker; only the one holding
# the lock processes queues, so nothing is sent twice
if not self._acquire_lock():
self._stop_event.wait(self.poll_interval)
continue
try:
processed = 0
# Process incoming webmentions
processed += self._process_pending_validations()
with self.app.app_context():
# Process incoming webmentions
processed += self._process_pending_validations()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Update last processed time
if processed > 0:
@@ -121,8 +127,40 @@ class WebmentionWorker(threading.Thread):
except Exception as e:
logger.error(f"Webmention worker thread failed: {e}", exc_info=True)
finally:
self._release_lock()
logger.info("Webmention worker stopped")
def _acquire_lock(self) -> bool:
"""
Take the cross-process worker lock (non-blocking)
On first acquisition, also reset items left mid-send by a previous
holder. The OS releases the lock if this process dies.
Returns:
True if this worker holds the lock
"""
if self._lock_file:
return True
lock_file = open(Path(self.db_path).parent / "webmention-worker.lock", "w")
try:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
lock_file.close()
return False
self._lock_file = lock_file
logger.info("Webmention worker holds the queue lock")
self._cleanup_orphaned_items()
return True
def _release_lock(self):
"""Release the cross-process worker lock if held"""
if self._lock_file:
self._lock_file.close()
self._lock_file = None
def _cleanup_orphaned_items(self):
"""
Clean up orphaned 'sending' status items on startup
@@ -226,25 +264,34 @@ class WebmentionWorker(threading.Thread):
moderation_status = 'approved' if self.auto_approve else 'pending'
# An update to an existing mention (spec 3.2.4) keeps the admin's
# moderation decision; only a first-time valid mention gets the
# default. Previously valid = validated with no error.
cursor.execute("""
UPDATE webmentions
SET validation_status = 'valid',
SET moderation_status = CASE
WHEN moderation_status IN ('hidden', 'deleted') THEN moderation_status
WHEN validated_at IS NOT NULL AND validation_error IS NULL
THEN moderation_status
ELSE ?
END,
validation_status = 'valid',
validation_error = NULL,
validated_at = CURRENT_TIMESTAMP,
mention_type = ?,
author_name = ?,
author_url = ?,
author_photo = ?,
content_snippet = ?,
moderation_status = ?,
updated_at = CURRENT_TIMESTAMP
WHERE id = ?
""", (
moderation_status,
result['mention_type'],
result['author_name'],
result['author_url'],
result['author_photo'],
result['content_snippet'],
moderation_status,
webmention_id
))
@@ -253,11 +300,17 @@ class WebmentionWorker(threading.Thread):
logger.info(
f"Validated webmention: {source_url} -> {target_url} "
f"(type={result['mention_type']}, status={moderation_status})"
f"(type={result['mention_type']})"
)
except TransientError as e:
# Source unreachable right now: keep a previously valid mention
self._mark_validation_failed(webmention_id, source_url, target_url, str(e),
transient=True)
except ValidationError as e:
# Validation failed - update with error
# Validation failed (410/4xx, no link): the mention is removed
# from display (spec 3.2.4)
self._mark_validation_failed(webmention_id, source_url, target_url, str(e))
except Exception as e:
@@ -275,11 +328,15 @@ class WebmentionWorker(threading.Thread):
webmention_id: int,
source_url: str,
target_url: str,
error: str
error: str,
transient: bool = False
):
"""
Mark webmention validation as failed/invalid
A transient failure of a mention that was valid before restores it to
valid with its previous data, rather than hiding it.
Args:
webmention_id: Database ID
source_url: Source URL
@@ -289,7 +346,9 @@ class WebmentionWorker(threading.Thread):
# Determine if error or invalid
# Per background-worker.md: HTTP 4xx and "no link" are invalid
# All other errors are 'error' status
if any(x in error.lower() for x in ['does not link', 'returned 4']):
if transient:
status = 'error'
elif any(x in error.lower() for x in ['does not link', 'returned 4']):
status = 'invalid'
else:
status = 'error'
@@ -298,6 +357,22 @@ class WebmentionWorker(threading.Thread):
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
if transient:
cursor.execute("""
UPDATE webmentions
SET validation_status = 'valid',
updated_at = CURRENT_TIMESTAMP
WHERE id = ? AND validated_at IS NOT NULL AND validation_error IS NULL
""", (webmention_id,))
if cursor.rowcount:
conn.commit()
conn.close()
logger.warning(
f"Could not re-check webmention, keeping it: "
f"{source_url} -> {target_url}: {error}"
)
return
cursor.execute("""
UPDATE webmentions
SET validation_status = ?,
@@ -418,11 +493,13 @@ class WebmentionWorker(threading.Thread):
conn.commit()
conn.close()
# Send webmention POST
# Send webmention POST (spec 3.1.3)
try:
# SSRF protection for endpoint URL
# SSRF protection for endpoint URL (spec 4.3)
validate_url_for_fetch(endpoint)
# Redirects are not followed: httpx would turn a 301/302 POST
# into a GET without the body
response = httpx.post(
endpoint,
data={
@@ -430,25 +507,24 @@ class WebmentionWorker(threading.Thread):
'target': target_url
},
timeout=10,
headers={'User-Agent': f'StarPunk/1.6.0 (+{self.site_url})'},
follow_redirects=True
headers={'User-Agent': user_agent(self.site_url)},
follow_redirects=False
)
# Check response
# Any 2xx is success (spec 3.1.3)
if 200 <= response.status_code < 300:
# Success
self._mark_send_success(queue_id, source_url, target_url)
elif 400 <= response.status_code < 500:
# Permanent failure (4xx)
error = f"HTTP {response.status_code}: {response.text[:200]}"
self._mark_send_failed(queue_id, source_url, target_url, error)
else:
# Temporary failure (5xx) - retry
elif response.status_code == 429 or response.status_code >= 500:
# Rate limited or server error - retry
error = f"HTTP {response.status_code}: {response.text[:200]}"
self._mark_send_retry(queue_id, source_url, target_url, retry_count, error)
else:
# Permanent failure (3xx, 4xx)
error = f"HTTP {response.status_code}: {response.text[:200]}"
self._mark_send_failed(queue_id, source_url, target_url, error)
except (httpx.TimeoutException, httpx.HTTPError) as e:
# Connection error - retry
self._mark_send_retry(queue_id, source_url, target_url, retry_count, str(e))
@@ -464,100 +540,81 @@ class WebmentionWorker(threading.Thread):
def _discover_endpoint(self, target_url: str) -> Optional[str]:
"""
Discover webmention endpoint for target URL with caching
Discover the webmention endpoint for a target URL (spec 3.1.2)
Per architect's answers to Q3, Q4, Q10:
- GET only (no HEAD)
- Check Link header → <link rel="webmention"> → <a rel="webmention">
- First match wins
- Cache results: 1h for found, 24h for not found
- SSRF protection on target URL
Args:
target_url: Target URL to discover endpoint for
- GET the target, following redirects (each hop SSRF-checked)
- The first Link header with rel=webmention wins, then the first
<link> or <a> with rel=webmention and an href, in document order
- rel is a space-separated token list, matched case-insensitively
- Relative endpoints resolve against the final URL; an empty href
is the target itself; query strings are kept
- Not cached: an updated post must re-discover (spec 3.1.4)
Returns:
Endpoint URL if found, None otherwise
"""
# Check cache first
cached = self._discovery_cache.get(target_url)
if cached and cached['expires_at'] > datetime.utcnow():
logger.debug(f"Using cached endpoint for {target_url}: {cached['endpoint']}")
return cached['endpoint']
Endpoint URL, or None if the target has none (or is blocked/4xx)
# SSRF protection
Raises:
httpx.HTTPError: Network errors and 5xx, so the caller retries
"""
try:
validate_url_for_fetch(target_url)
result = safe_get(target_url, headers={'User-Agent': user_agent(self.site_url)})
except (SSRFBlockedError, ValueError) as e:
logger.warning(f"SSRF protection blocked target {target_url}: {e}")
# Cache negative result
self._cache_endpoint(target_url, None, hours=24)
return None
except FetchTooLargeError:
logger.info(f"Target too large for discovery: {target_url}")
return None
# Fetch target URL
try:
response = httpx.get(
target_url,
timeout=10,
headers={'User-Agent': f'StarPunk/1.6.0 (+{self.site_url})'},
follow_redirects=True
)
response.raise_for_status()
# 1. Check Link header
link_header = response.headers.get('Link', '')
if link_header:
endpoint = self._parse_link_header(link_header, response.url)
if endpoint:
self._cache_endpoint(target_url, endpoint, hours=1)
return endpoint
# 2. Parse HTML for <link> and <a> elements
content_type = response.headers.get('Content-Type', '')
if 'html' in content_type.lower():
endpoint = self._parse_html_for_endpoint(response.text, response.url)
if endpoint:
self._cache_endpoint(target_url, endpoint, hours=1)
return endpoint
# No endpoint found - cache negative result
logger.info(f"No webmention endpoint found for {target_url}")
self._cache_endpoint(target_url, None, hours=24)
if result.status_code >= 500:
raise httpx.HTTPError(f"Target returned {result.status_code}")
if result.status_code >= 400:
logger.info(f"Target returned {result.status_code}: {target_url}")
return None
except Exception as e:
logger.warning(f"Failed to discover endpoint for {target_url}: {e}")
# Don't cache on error - allow retry
return None
for link_header in result.headers.get_list('Link'):
endpoint = self._parse_link_header(link_header, result.url)
if endpoint is not None:
return endpoint
content_type = result.headers.get('Content-Type', '')
if 'html' in content_type.lower():
endpoint = self._parse_html_for_endpoint(result.text, result.url)
if endpoint is not None:
return endpoint
logger.info(f"No webmention endpoint found for {target_url}")
return None
@staticmethod
def _has_webmention_rel(rel: str) -> bool:
return 'webmention' in rel.lower().split()
def _parse_link_header(self, link_header: str, base_url: str) -> Optional[str]:
"""
Parse Link header for webmention endpoint
Parse a Link header (RFC 8288) for a rel=webmention link
Args:
link_header: Raw Link header value
link_header: Link header value (may hold several links)
base_url: Base URL for resolving relative URLs
Returns:
Absolute endpoint URL if found, None otherwise
Absolute endpoint URL of the first match, None otherwise
"""
# Simple parsing - look for rel="webmention"
# Format: <URL>; rel="webmention"
import re
pattern = r'<([^>]+)>\s*;\s*rel\s*=\s*["\']?webmention["\']?'
match = re.search(pattern, link_header, re.IGNORECASE)
if match:
endpoint = match.group(1)
# Make absolute if relative
from urllib.parse import urljoin
return urljoin(base_url, endpoint)
for match in LINK_RE.finditer(link_header):
uri, params = match.group(1), match.group(2)
for param in LINK_PARAM_RE.finditer(params):
if param.group(1).lower() != 'rel':
continue
rel = param.group(2) if param.group(2) is not None else (param.group(3) or '')
if self._has_webmention_rel(rel):
return urljoin(base_url, uri.strip())
break # only the first rel parameter counts (RFC 8288)
return None
def _parse_html_for_endpoint(self, html_content: str, base_url: str) -> Optional[str]:
"""
Parse HTML for webmention endpoint in <link> or <a> elements
Find the first <link> or <a> with rel=webmention, in document order
Args:
html_content: HTML content
@@ -567,48 +624,18 @@ class WebmentionWorker(threading.Thread):
Absolute endpoint URL if found, None otherwise
"""
try:
doc = lxml_html.fromstring(html_content)
# Check <link rel="webmention"> first
links = doc.xpath('//link[@rel="webmention"]/@href')
if links:
from urllib.parse import urljoin
return urljoin(base_url, links[0])
# Check <a rel="webmention">
anchors = doc.xpath('//a[@rel="webmention"]/@href')
if anchors:
from urllib.parse import urljoin
return urljoin(base_url, anchors[0])
doc = lxml_html.document_fromstring(html_content)
except Exception as e:
logger.debug(f"HTML parsing failed: {e}")
return None
for element in doc.iter('link', 'a'):
href = element.get('href')
if href is not None and self._has_webmention_rel(element.get('rel', '')):
return urljoin(base_url, href.strip())
return None
def _cache_endpoint(self, target_url: str, endpoint: Optional[str], hours: int):
"""
Cache endpoint discovery result
Args:
target_url: Target URL
endpoint: Discovered endpoint (None = negative cache)
hours: TTL in hours
"""
self._discovery_cache[target_url] = {
'endpoint': endpoint,
'expires_at': datetime.utcnow() + timedelta(hours=hours)
}
# Optional: Evict old entries (keep cache bounded)
if len(self._discovery_cache) > 1000:
# Remove expired entries
now = datetime.utcnow()
self._discovery_cache = {
k: v for k, v in self._discovery_cache.items()
if v['expires_at'] > now
}
def _mark_send_success(self, queue_id: int, source_url: str, target_url: str):
"""Mark webmention send as successful"""
try:
+14 -2
View File
@@ -92,6 +92,17 @@ main { flex: 1; margin-bottom: var(--spacing-xl); }
.e-content ul, .e-content ol { margin: 0 0 var(--spacing-md) 1.25rem; }
.e-content img { max-width: 100%; height: auto; }
.e-content h1, .e-content h2, .e-content h3 { font-family: var(--font-body); }
.reaction { margin-bottom: var(--spacing-sm); font-size: 0.9375rem; color: var(--color-text-light); overflow-wrap: anywhere; }
.reaction-verb { font-weight: 600; margin-right: 0.25rem; }
.reaction-like .reaction-verb { color: #c2410c; }
.reaction-repost .reaction-verb { color: #15803d; }
.reaction a { color: var(--color-text); font-weight: 500; }
.reaction a:hover { color: var(--color-link); }
.reaction-by { margin-left: 0.25rem; }
@media (prefers-color-scheme: dark) {
.reaction-like .reaction-verb { color: #fb923c; }
.reaction-repost .reaction-verb { color: #4ade80; }
}
.read-more { margin: var(--spacing-sm) 0 0; font-size: 0.9375rem; }
.note-meta { display: flex; flex-wrap: wrap; align-items: baseline; gap: 0 0.4rem; margin-top: var(--spacing-sm); color: var(--color-text-light); font-size: 0.8125rem; }
@@ -125,7 +136,8 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.button-warning { background: var(--color-warning); color: #333; border-color: var(--color-warning); }
.button-small { padding: 0.25rem 0.5rem; font-size: 0.875rem; }
.admin-container { max-width: 60rem; }
.admin-page { --max-width: 60rem; }
.table-scroll { overflow-x: auto; }
.admin-nav { display: flex; gap: var(--spacing-md); padding: var(--spacing-md); background: var(--color-bg-alt); border-radius: var(--border-radius); margin-bottom: var(--spacing-lg); flex-wrap: wrap; align-items: center; }
.admin-nav .logout-form { margin-left: auto; }
.admin-nav .logout-form button { margin: 0; }
@@ -136,7 +148,7 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.note-table { width: 100%; border-collapse: collapse; margin-top: var(--spacing-md); }
.note-table th, .note-table td { padding: var(--spacing-md); text-align: left; border-bottom: 1px solid var(--color-border); }
.note-table th { background: var(--color-bg-alt); font-weight: 600; }
.note-content-preview { max-width: 30rem; }
.note-content-preview { min-width: 14rem; overflow-wrap: anywhere; }
.note-excerpt { margin-bottom: var(--spacing-xs); }
.note-slug { color: var(--color-text-light); font-family: var(--font-mono); font-size: 0.75rem; }
.note-date { white-space: nowrap; color: var(--color-text-light); }
+11
View File
@@ -0,0 +1,11 @@
{% extends "base.html" %}
{% block title %}Post Deleted - {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<article class="error-page">
<h1>410 - Post Deleted</h1>
<p>This post has been deleted.</p>
<p><a href="/">Return to homepage</a></p>
</article>
{% endblock %}
+3
View File
@@ -1,5 +1,8 @@
{% extends "base.html" %}
{# Admin pages use a wider frame so tables fit under the header #}
{% block body_class %} class="admin-page"{% endblock %}
{% block content %}
<div class="admin-container">
<nav class="admin-nav">
+2
View File
@@ -21,6 +21,7 @@
</div>
{% if notes %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -67,6 +68,7 @@
{% endfor %}
</tbody>
</table>
</div>
{# Pagination (v1.6.0) #}
{% if total_pages > 1 %}
+6 -3
View File
@@ -12,12 +12,15 @@
<form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form">
<div class="form-group">
<label for="content">Content (Markdown)</label>
{% if note.reacts_to %}
<p class="note-meta">{{ '★ Like of' if note.kind == 'like' else '🔁 Repost of' }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
{% endif %}
<label for="content">{% if note.reacts_to %}Comment (optional, Markdown){% else %}Content (Markdown){% endif %}</label>
<textarea
id="content"
name="content"
rows="20"
required
rows="{{ 6 if note.reacts_to else 20 }}"
{% if not note.reacts_to %}required{% endif %}
autofocus
>{{ note.content }}</textarea>
<small>Use Markdown syntax for formatting</small>
+2
View File
@@ -8,6 +8,7 @@
<p>Micropub apps you have approved. Revoke any you no longer use.</p>
{% if tokens %}
<div class="table-scroll">
<table class="note-table">
<thead>
<tr>
@@ -37,6 +38,7 @@
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p>No apps connected.</p>
{% endif %}
+2
View File
@@ -43,6 +43,7 @@
</button>
</div>
<div class="table-scroll">
<table class="webmention-table">
<thead>
<tr>
@@ -85,6 +86,7 @@
{% endfor %}
</tbody>
</table>
</div>
</form>
{# Pagination #}
+2 -1
View File
@@ -27,7 +27,7 @@
{% block head %}{% endblock %}
</head>
<body>
<body{% block body_class %}{% endblock %}>
{% if config.DEV_MODE %}
<div class="dev-mode-warning">
WARNING: DEVELOPMENT MODE - Authentication bypassed
@@ -38,6 +38,7 @@
<a class="site-title" href="/">{{ config.SITE_NAME }}</a>
<nav class="site-nav">
<a href="/">Home</a>
<a href="{{ url_for('public.likes') }}">Likes</a>
<a href="{{ url_for('public.feed') }}">RSS</a>
{% if g.me %}
<a href="{{ url_for('admin.dashboard') }}">Admin</a>
+22
View File
@@ -0,0 +1,22 @@
{% extends "base.html" %}
{% from "partials/note_preview.html" import note_preview %}
{% block title %}Likes · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<div class="h-feed">
<h1 class="p-name page-title">Likes</h1>
{% if notes %}
{% for note in notes %}
{{ note_preview(note, author) }}
{% endfor %}
{% else %}
<p class="empty-state">Nothing liked yet.</p>
{% endif %}
<nav class="note-nav">
<a href="/">← All notes</a>
</nav>
</div>
{% endblock %}
+10 -6
View File
@@ -1,7 +1,8 @@
{% extends "base.html" %}
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% block title %}{% if note.content.strip().startswith('#') %}{{ note.title }}{% else %}{{ note.excerpt|truncate(60) }}{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block title %}{% if note.content.strip().startswith('#') %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<article class="h-entry">
@@ -17,13 +18,14 @@
{% if note.in_reply_to %}
<div class="reply-context">
<span class="reply-label">In reply to:</span>
<blockquote class="h-cite">
{# u-in-reply-to on the h-cite itself makes it the entry's property (Post Type Discovery) #}
<blockquote class="u-in-reply-to h-cite">
{% if reply_context and reply_context.fetch_status == 'fetched' %}
{% if reply_context.content_snippet %}
<p class="p-content">{{ reply_context.content_snippet }}</p>
{% endif %}
<cite>
<a class="u-in-reply-to" href="{{ note.in_reply_to }}">
<a class="u-url" href="{{ note.in_reply_to }}">
{% if reply_context.author_name %}{{ reply_context.author_name }}{% endif %}
{% if reply_context.site_name %} on {{ reply_context.site_name }}{% endif %}
{% if not reply_context.author_name and not reply_context.site_name %}{{ note.in_reply_to }}{% endif %}
@@ -31,16 +33,18 @@
</cite>
{% else %}
<cite>
<a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
<a class="u-url" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
</cite>
{% endif %}
</blockquote>
</div>
{% endif %}
{{ reaction(note) }}
{# u-photo placement: Per draft spec, u-photo must be direct child of h-entry, #}
{# NOT inside e-content. Media is rendered ABOVE e-content to meet this requirement. #}
{{ display_media(note.media) }}
{{ display_media(note.media, note.excerpt|truncate(150)) }}
{# e-content: note content BELOW media (per ADR-057) #}
<div class="e-content">
@@ -181,7 +185,7 @@
<div class="webmention webmention-mention">
<a href="{{ mention.source_url }}">{{ mention.author_name or mention.source_url }}</a>
{% if mention.content_snippet %}
<p>{{ mention.content_snippet|truncate(200) }}</p>
<p>{{ mention.content_snippet|striptags|truncate(200) }}</p>
{% endif %}
</div>
{% endfor %}
+3 -2
View File
@@ -1,11 +1,12 @@
{# Reusable media display macro #}
{% macro display_media(media_items) %}
{# fallback_alt: for photos without a caption, usually the post's text (v2.6.5) #}
{% macro display_media(media_items, fallback_alt='') %}
{% if media_items %}
<div class="note-media">
{% for item in media_items %}
<figure class="media-item">
<img src="{{ url_for('public.media_file', path=item.path) }}"
alt="{{ item.caption or 'Image' }}"
alt="{{ item.caption or fallback_alt or 'Image' }}"
class="u-photo"
loading="lazy">
{# No figcaption - caption is for alt text only #}
+9 -1
View File
@@ -1,5 +1,6 @@
{# One note in a list (home, tag pages). v2.3.0 #}
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% macro note_preview(note, author) %}
<article class="h-entry note-preview">
{# Notes starting with a # heading have a title (Q22) #}
@@ -8,7 +9,13 @@
<h2 class="p-name"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">{{ note.title }}</a></h2>
{% endif %}
{{ display_media(note.media) }}
{% if note.in_reply_to %}
<p class="reaction reply-to"><span class="reaction-verb">↩ In reply to</span> <a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a></p>
{% endif %}
{{ reaction(note) }}
{{ display_media(note.media, note.excerpt|truncate(150)) }}
{% if has_explicit_title %}
{# Titled notes are articles: show a summary and link through #}
@@ -17,6 +24,7 @@
</div>
<p class="read-more"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">Read more →</a></p>
{% else %}
{# Always present, even empty: without it a text-free photo post gets an implied name (mf2) #}
<div class="e-content">
{{ note.html|safe }}
</div>
+12
View File
@@ -0,0 +1,12 @@
{# "Liked" / "Reposted" line for likes and reposts (v2.5.0) #}
{% macro reaction(note) %}
{% if note.reacts_to %}
{% set ctx = note.context if note.context is defined else None %}
{% if ctx and ctx.title %}{% set label = ctx.title %}{% elif ctx and ctx.content_snippet %}{% set label = ctx.content_snippet|striptags|truncate(90) %}{% elif ctx and (ctx.author_name or ctx.site_name) %}{% set label = 'a post' %}{% else %}{% set label = note.reacts_to %}{% endif %}
<p class="reaction reaction-{{ note.kind }}">
<span class="reaction-verb">{% if note.kind == 'like' %}★ Liked{% else %}🔁 Reposted{% endif %}</span>
<a class="{{ 'u-like-of' if note.kind == 'like' else 'u-repost-of' }}" href="{{ note.reacts_to }}">{{ label }}</a>
{% if ctx and (ctx.author_name or ctx.site_name) %}<span class="reaction-by">by {{ ctx.author_name or ctx.site_name }}</span>{% endif %}
</p>
{% endif %}
{% endmacro %}
+223
View File
@@ -0,0 +1,223 @@
"""
Likes and reposts (v2.5.0, ADR-068)
Fake GoToSocial and Bluesky servers check that likes/reposts of their own
posts become native favourites/boosts/likes/reposts, and everything else
falls back (reposts) or is skipped (likes).
"""
import json
import httpx
import pytest
from starpunk import create_app, syndication
from starpunk.indieauth import create_access_token
from starpunk.notes import delete_note, get_note
from starpunk.syndication import BlueskyClient, MastodonClient, get_note_syndications, process_due
SITE = "https://blog.example.com"
GTS = "https://social.example.com"
FEDI_POST = "https://mastodon.example/@alice/111"
BSKY_POST = "https://bsky.app/profile/alice.example/post/3abc"
WEB_PAGE = "https://someblog.example/2026/a-post"
class FakeGTS:
def __init__(self):
self.calls = []
self.statuses = {}
def __call__(self, request):
path = request.url.path
self.calls.append((request.method, path))
if path == "/api/v2/search":
q = request.url.params["q"]
return httpx.Response(200, json={"statuses": [{"id": "s1"}] if q == FEDI_POST else []})
if path in ("/api/v1/statuses/s1/favourite", "/api/v1/statuses/s1/unfavourite",
"/api/v1/statuses/s1/reblog", "/api/v1/statuses/s1/unreblog"):
return httpx.Response(200, json={"id": "s1"})
if path == "/api/v1/statuses" and request.method == "POST":
sid = f"p{len(self.statuses) + 1}"
self.statuses[sid] = json.loads(request.content)
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
if path.startswith("/api/v1/statuses/") and request.method == "DELETE":
self.statuses.pop(path.rsplit("/", 1)[1], None)
return httpx.Response(200, json={})
return httpx.Response(500)
class FakePDS:
def __init__(self):
self.records = {}
self.calls = []
def __call__(self, request):
nsid = request.url.path.removeprefix("/xrpc/")
self.calls.append(nsid)
if nsid == "com.atproto.server.createSession":
return httpx.Response(200, json={"did": "did:plc:me", "handle": "me.example",
"accessJwt": "a", "refreshJwt": "r"})
if nsid == "com.atproto.identity.resolveHandle":
return httpx.Response(200, json={"did": "did:plc:alice"})
if nsid == "app.bsky.feed.getPosts":
return httpx.Response(200, json={"posts": [{"uri": request.url.params["uris"], "cid": "cid1"}]})
if nsid == "com.atproto.repo.createRecord":
body = json.loads(request.content)
uri = f"at://did:plc:me/{body['collection']}/r{len(self.records) + 1}"
self.records[uri] = body["record"]
return httpx.Response(200, json={"uri": uri, "cid": "c"})
if nsid == "com.atproto.repo.deleteRecord":
body = json.loads(request.content)
self.records.pop(f"at://{body['repo']}/{body['collection']}/{body['rkey']}", None)
return httpx.Response(200, json={})
return httpx.Response(500, json={"error": nsid})
@pytest.fixture
def gts():
return FakeGTS()
@pytest.fixture
def pds():
return FakePDS()
@pytest.fixture
def app(tmp_path, gts, pds):
data = tmp_path / "data"
app = create_app({
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com", "SESSION_SECRET": "x",
"DATA_PATH": data, "NOTES_PATH": data / "notes", "DATABASE_PATH": data / "starpunk.db",
"GOTOSOCIAL_URL": GTS, "GOTOSOCIAL_ACCESS_TOKEN": "t",
"BLUESKY_HANDLE": "me.example", "BLUESKY_APP_PASSWORD": "p", "BLUESKY_SERVICE": "https://pds.example",
"FEED_CACHE_ENABLED": False,
})
# The feed's note list is cached at module level across apps
from starpunk.routes import public
public._feed_cache["notes"] = None
public._feed_cache["timestamp"] = None
syndication._client_cache[GTS] = MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(gts)))
syndication._client_cache["bluesky:https://pds.example:me.example"] = BlueskyClient(
"https://pds.example", "me.example", "p", http=httpx.Client(transport=httpx.MockTransport(pds)))
yield app
syndication._client_cache.clear()
@pytest.fixture
def post(app, monkeypatch):
# Don't fetch contexts from the network
monkeypatch.setattr("starpunk.reply_context.fetch_reply_context", lambda *a, **k: None)
with app.test_request_context():
token = create_access_token("https://quill.example/", "create update delete")["access_token"]
client = app.test_client()
def _post(**props):
response = client.post("/micropub", json={"type": ["h-entry"], "properties": props},
headers={"Authorization": f"Bearer {token}"})
assert response.status_code == 201, response.get_json()
location = response.headers["Location"]
with app.app_context():
note = get_note(slug=location.rsplit("/", 1)[1])
process_due(limit=10)
rows = {r["target"]: r for r in get_note_syndications(note.id)}
return note, rows
return _post
def page(app, path):
return app.test_client().get(path).data.decode()
class TestSite:
def test_like_without_comment(self, app, post):
note, _ = post(**{"like-of": [WEB_PAGE]})
assert note.kind == "like" and note.content == ""
assert note.slug not in page(app, "/")
likes = page(app, "/likes")
assert f'class="u-like-of" href="{WEB_PAGE}"' in likes and "★ Liked" in likes
assert note.slug not in page(app, "/feed.json")
def test_repost_in_stream_and_feed(self, app, post):
note, _ = post(**{"repost-of": [WEB_PAGE], "content": ["Worth a read"]})
home = page(app, "/")
assert f'class="u-repost-of" href="{WEB_PAGE}"' in home and "Worth a read" in home
feed = json.loads(page(app, "/feed.json"))
item = next(i for i in feed["items"] if note.slug in i["url"])
assert "Reposted" in item["content_html"] and WEB_PAGE in item["content_html"]
def test_webmention_goes_to_target_with_clean_url(self, app, post, monkeypatch):
sent = {}
monkeypatch.setattr("starpunk.webmention_sender.queue_webmentions_for_note",
lambda **kw: sent.update(kw) or 0)
app.config["WEBMENTION_ENABLED"] = True
note, _ = post(**{"like-of": [WEB_PAGE]})
assert sent["in_reply_to"] == WEB_PAGE
assert sent["note_url"] == f"{SITE}/note/{note.slug}" # no double slash
def test_both_kinds_rejected(self, app, post):
with pytest.raises(AssertionError):
post(**{"like-of": [WEB_PAGE], "repost-of": [WEB_PAGE]})
class TestGoToSocial:
def test_like_of_fediverse_post_favourites(self, app, post, gts):
note, rows = post(**{"like-of": [FEDI_POST]})
assert rows["gotosocial"]["remote_id"] == "favourite:s1"
assert rows["gotosocial"]["remote_url"] is None
with app.test_request_context():
delete_note(id=note.id)
process_due(limit=10)
assert ("POST", "/api/v1/statuses/s1/unfavourite") in gts.calls
def test_like_of_web_page_is_not_crossposted(self, app, post, gts):
_, rows = post(**{"like-of": [WEB_PAGE]})
assert rows["gotosocial"]["state"] == "synced" and rows["gotosocial"]["remote_id"] is None
assert gts.statuses == {}
def test_repost_boosts(self, app, post, gts):
_, rows = post(**{"repost-of": [FEDI_POST]})
assert rows["gotosocial"]["remote_id"] == "reblog:s1"
def test_repost_with_comment_posts_status(self, app, post, gts):
_, rows = post(**{"repost-of": [FEDI_POST], "content": ["So true"]})
(status,) = gts.statuses.values()
assert status["status"].startswith("So true") and FEDI_POST in status["status"]
assert ("POST", "/api/v1/statuses/s1/reblog") not in gts.calls
def test_repost_of_web_page_posts_link(self, app, post, gts):
post(**{"repost-of": [WEB_PAGE]})
(status,) = gts.statuses.values()
assert status["status"].startswith("🔁") and WEB_PAGE in status["status"]
class TestBluesky:
def test_like_of_bluesky_post(self, app, post, pds):
note, rows = post(**{"like-of": [BSKY_POST]})
uri = rows["bluesky"]["remote_id"]
assert "/app.bsky.feed.like/" in uri
assert pds.records[uri]["subject"] == {"uri": "at://did:plc:alice/app.bsky.feed.post/3abc", "cid": "cid1"}
with app.test_request_context():
delete_note(id=note.id)
process_due(limit=10)
assert uri not in pds.records
def test_like_of_web_page_skipped(self, app, post, pds):
_, rows = post(**{"like-of": [WEB_PAGE]})
assert rows["bluesky"]["remote_id"] is None and pds.records == {}
def test_repost_and_quote(self, app, post, pds):
_, rows = post(**{"repost-of": [BSKY_POST]})
assert "/app.bsky.feed.repost/" in rows["bluesky"]["remote_id"]
_, rows = post(**{"repost-of": [BSKY_POST], "content": ["Great thread"]})
record = pds.records[rows["bluesky"]["remote_id"]]
assert record["text"] == "Great thread"
assert record["embed"] == {"$type": "app.bsky.embed.record",
"record": {"uri": "at://did:plc:alice/app.bsky.feed.post/3abc", "cid": "cid1"}}
def test_repost_of_web_page_posts_link_card(self, app, post, pds):
_, rows = post(**{"repost-of": [WEB_PAGE]})
record = pds.records[rows["bluesky"]["remote_id"]]
assert record["embed"]["external"]["uri"] == WEB_PAGE
assert record["text"].startswith("🔁")
+2 -2
View File
@@ -118,7 +118,7 @@ class TestUpdate:
def test_categories(self, app, client, h):
url = create(client, h, category=["one", "two"])
self.update(client, h, url, add={"category": ["three", "One"]})
assert source(client, h, url)["properties"]["category"] == ["one", "three", "two"]
assert source(client, h, url)["properties"]["category"] == ["one", "two", "three"] # entered order (v2.6.4)
self.update(client, h, url, delete={"category": ["two"]})
assert source(client, h, url)["properties"]["category"] == ["one", "three"]
self.update(client, h, url, delete=["category"])
@@ -161,7 +161,7 @@ class TestDelete:
url = create(client, h, category=["kept"])
path = url.replace(SITE, "")
assert client.post("/micropub", data={"action": "delete", "url": url}, headers=h).status_code == 204
assert client.get(path).status_code == 404
assert client.get(path).status_code == 410 # Gone (Webmention 3.1.5)
response = client.post("/micropub", json={"action": "undelete", "url": url}, headers=h)
assert response.status_code == 204
assert client.get(path).status_code == 200
+105
View File
@@ -0,0 +1,105 @@
"""
Micropub media fixes (v2.6.5): photo posts without text, the same photo
twice, and failed creates that left a post (and a database lock) behind.
"""
import io
from starpunk.database.pool import ConnectionPool
from starpunk.notes import list_notes
from starpunk.syndication import compose_text
from tests.test_micropub_v240 import SITE, app, client, h, jpeg, note_at, source # noqa: F401
def upload(client, h, color="red"):
response = client.post("/micropub/media", data={"file": (io.BytesIO(jpeg(color)), "a.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 201
return response.headers["Location"]
def post(client, h, props):
return client.post("/micropub", json={"type": ["h-entry"], "properties": props}, headers=h)
def update(client, h, url, **ops):
return client.post("/micropub", json={"action": "update", "url": url, **ops}, headers=h)
class TestPhotoWithoutText:
def test_json_photo_only(self, app, client, h):
photo = upload(client, h)
response = post(client, h, {"photo": [{"value": photo, "alt": "Red"}]})
assert response.status_code == 201, response.get_json()
url = response.headers["Location"]
assert note_at(app, url).content == ""
assert source(client, h, url, "photo")["properties"]["photo"] == [{"value": photo, "alt": "Red"}]
page = client.get(url.replace(SITE, ""))
assert page.status_code == 200 and b"<title>Photo \xc2\xb7 " in page.data
def test_multipart_photo_only(self, client, h):
response = client.post("/micropub", data={"h": "entry", "photo": (io.BytesIO(jpeg()), "p.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 201, response.get_json()
def test_nothing_at_all_is_still_rejected(self, client, h):
assert post(client, h, {}).status_code == 400
# A photo that can't be found doesn't count
assert post(client, h, {"photo": [f"{SITE}/media/nope.jpg"]}).status_code == 400
def test_text_can_be_cleared_while_photos_remain(self, app, client, h):
photo = upload(client, h)
url = post(client, h, {"content": ["Hi"], "photo": [photo]}).headers["Location"]
assert update(client, h, url, replace={"content": [""]}).status_code == 204
assert note_at(app, url).content == ""
assert update(client, h, url, replace={"content": ["Back"]}).status_code == 204
assert update(client, h, url, delete=["content"]).status_code == 204
assert note_at(app, url).content == ""
# Without photos the text has to stay
response = update(client, h, url, delete=["content", "photo"])
assert response.status_code == 400
def test_text_is_required_without_photos(self, client, h):
url = post(client, h, {"content": ["Hi"]}).headers["Location"]
assert update(client, h, url, replace={"content": [""]}).status_code == 400
assert update(client, h, url, delete=["content"]).status_code == 400
def test_crosspost_text_starts_with_the_link(self, app, client, h):
url = post(client, h, {"photo": [upload(client, h)]}).headers["Location"]
with app.app_context():
note = note_at(app, url)
assert compose_text(note, url, ["cats"], 500) == f"{url}\n\n#cats"
class TestFailures:
def test_same_photo_twice_is_shown_once(self, client, h):
photo = upload(client, h)
response = post(client, h, {"content": ["Twice"], "photo": [photo, photo]})
assert response.status_code == 201, response.get_json()
assert source(client, h, response.headers["Location"], "photo")["properties"]["photo"] == [{"value": photo, "alt": ""}]
def test_failed_create_leaves_no_post_and_no_lock(self, app, client, h, monkeypatch):
import starpunk.syndication as syndication
def boom(*args, **kwargs):
raise RuntimeError("crosspost store down")
monkeypatch.setattr(syndication, "select_targets", boom)
response = post(client, h, {"content": ["Doomed"], "photo": [upload(client, h)]})
assert response.status_code == 500
with app.app_context():
assert list_notes(published_only=False) == []
monkeypatch.undo()
# The database is usable straight away
assert post(client, h, {"content": ["Next"]}).status_code == 201
def test_pool_rolls_back_open_transactions(self, tmp_path):
pool = ConnectionPool(str(tmp_path / "x.db"), pool_size=1, timeout=1)
conn = pool.get_connection()
conn.execute("CREATE TABLE t (x)")
conn.commit()
conn.execute("INSERT INTO t VALUES (1)")
assert conn.in_transaction
pool.return_connection(conn)
assert not conn.in_transaction
assert pool.get_connection().execute("SELECT count(*) FROM t").fetchone()[0] == 0
+98
View File
@@ -0,0 +1,98 @@
"""
Rendered posts read as the right post type (v2.6.5).
Parses note pages and the home page with mf2py, then runs W3C Post Type
Discovery (https://www.w3.org/TR/post-type-discovery/) on what a consumer
sees. A photo post is any post with a photo, unless it's first a repost,
like or reply.
"""
import pytest
import mf2py
import starpunk.reply_context as reply_context
from tests.test_micropub_v240 import SITE, app, client, h, jpeg # noqa: F401
from tests.test_micropub_v265 import upload
def post_type(props):
"""W3C Post Type Discovery, section 5 (no events or RSVPs here)."""
for prop, kind in [("repost-of", "repost"), ("like-of", "like"), ("in-reply-to", "reply"),
("video", "video"), ("photo", "photo")]:
if props.get(prop):
return kind
def first(prop):
for v in props.get(prop, []):
v = v.get("value", "") if isinstance(v, dict) else v
if v.strip():
return " ".join(v.split())
return ""
content = first("content") or first("summary")
name = first("name")
if not content or not name or content.startswith(name):
return "note"
return "article"
def entries(items):
for item in items:
if "h-entry" in item.get("type", []):
yield item
yield from entries(item.get("children", []))
def value(v):
"""A u-* value, or the url of a nested microformat (h-cite)."""
return v.get("value") or v.get("properties", {}).get("url", [None])[0] if isinstance(v, dict) else v
CASES = {
"note": ({"content": ["Just a note"]}, "note"),
"article": ({"name": ["A Title"], "content": ["Body text here"]}, "article"),
"photo without text": ({"photo": ["PHOTO"]}, "photo"),
"photo with caption": ({"content": ["Sunset"], "photo": [{"value": "PHOTO", "alt": "Orange sky"}]}, "photo"),
"photo with title": ({"name": ["Trip"], "content": ["Day one"], "photo": ["PHOTO"]}, "photo"),
"reply": ({"content": ["Agreed"], "in-reply-to": ["https://other.example/post/1"]}, "reply"),
"reply with photo": ({"content": ["Mine"], "in-reply-to": ["https://other.example/post/2"],
"photo": ["PHOTO"]}, "reply"),
"repost": ({"repost-of": ["https://other.example/post/4"]}, "repost"),
}
@pytest.mark.parametrize("case", CASES)
def test_post_type(app, client, h, monkeypatch, case):
monkeypatch.setattr(reply_context, "fetch_reply_context", lambda *a, **k: None, raising=False)
props, want = CASES[case]
photo = upload(client, h) if "PHOTO" in str(props) else None
props = {k: [photo if v == "PHOTO" else ({**v, "value": photo} if isinstance(v, dict) else v)
for v in vs] for k, vs in props.items()}
response = client.post("/micropub", json={"type": ["h-entry"], "properties": props}, headers=h)
assert response.status_code == 201, response.get_json()
url = response.headers["Location"]
(entry,) = entries(mf2py.parse(doc=client.get(url.replace(SITE, "")).data.decode(), url=url)["items"])
home = mf2py.parse(doc=client.get("/").data.decode(), url=f"{SITE}/")
(listed,) = [e for e in entries(home["items"]) if e["properties"].get("url") == [url]]
for e in (entry, listed):
assert post_type(e["properties"]) == want
if want == "reply":
assert value(e["properties"]["in-reply-to"][0]) == props["in-reply-to"][0]
if want == "photo" and "name" not in props:
assert "name" not in e["properties"], "no implied name to show as a caption"
def test_photo_alt_text(app, client, h):
"""Alt text is the caption, else the post's text, never made up (v2.6.5)."""
def alts(props):
url = client.post("/micropub", json={"type": ["h-entry"], "properties": props},
headers=h).headers["Location"]
(entry,) = entries(mf2py.parse(doc=client.get(url.replace(SITE, "")).data.decode(), url=url)["items"])
return [p["alt"] for p in entry["properties"]["photo"]]
assert alts({"content": ["Beach day"], "photo": [{"value": upload(client, h), "alt": "Waves"}]}) == ["Waves"]
assert alts({"content": ["Beach day"], "photo": [upload(client, h, "blue")]}) == ["Beach day"]
assert alts({"name": ["Trip"], "content": ["Day one"], "photo": [upload(client, h, "green")]}) == ["Day one"]
assert alts({"photo": [upload(client, h, "gray")]}) == ["Image"]
+47
View File
@@ -32,6 +32,7 @@ class FakeServer:
self.statuses = {}
self.calls = []
self.media = 0
self.uploads = [] # raw multipart bodies
self.fail_next = None # status code to return once
def __call__(self, request: httpx.Request) -> httpx.Response:
@@ -43,6 +44,7 @@ class FakeServer:
path = request.url.path
if path == "/api/v2/media":
self.media += 1
self.uploads.append(request.read())
return httpx.Response(200, json={"id": f"m{self.media}"})
if path == "/api/v1/statuses" and request.method == "POST":
body = json.loads(request.content)
@@ -271,6 +273,51 @@ class TestMedia:
assert server.statuses["1"]["media_ids"] == ["m1"]
class TestInlineImages:
"""Clients like OutPost upload a photo and embed it in the content (v2.6.6)"""
def _upload(self, app):
from starpunk.media import save_media
buf = io.BytesIO()
Image.new("RGB", (800, 600), "orange").save(buf, format="JPEG")
with app.test_request_context():
return save_media(buf.getvalue(), "cat.jpg")
def test_embedded_own_image_is_attached(self, app, server):
media = self._upload(app)
url = f"https://blog.example.com/media/{media['path']}"
publish(app, content=f"![Tortie cat on a chair]({url})")
status = server.statuses["1"]
assert status["media_ids"] == ["m1"]
assert b"Tortie cat on a chair" in server.uploads[0]
assert "![" not in status["status"] and url not in status["status"]
assert status["status"].startswith("https://blog.example.com/note/")
def test_text_around_image_is_kept(self, app, server):
media = self._upload(app)
publish(app, content=f"Look\n\n<img src=\"/media/{media['path']}\" alt=\"cat\">\n\nNice")
status = server.statuses["1"]
assert status["media_ids"] == ["m1"]
assert status["status"].startswith("Look\n\nNice")
def test_image_both_attached_and_embedded_is_sent_once(self, app, server):
from starpunk.media import attach_media_to_note
media = self._upload(app)
with app.test_request_context():
note = create_note(content=f"![x](/media/{media['path']})", published=True)
attach_media_to_note(note.id, [media["id"]], ["caption"])
select_targets(note.id, ["gotosocial"])
process_due()
assert server.statuses["1"]["media_ids"] == ["m1"]
def test_other_sites_images_are_ignored(self, app, server):
publish(app, content="![x](https://elsewhere.example/cat.jpg) hi")
assert server.statuses["1"]["media_ids"] == []
assert server.uploads == []
class TestEntryPoints:
@pytest.fixture
def admin(self, app):
+120 -2
View File
@@ -92,8 +92,31 @@ def pds():
return FakePDS()
class FakeWeb:
"""Other people's pages, for link cards: url -> (status, headers, body)"""
def __init__(self):
self.pages = {}
self.fetched = []
def __call__(self, request):
url = str(request.url)
self.fetched.append(url)
status, headers, body = self.pages.get(url, (404, {}, b"nope"))
return httpx.Response(status, headers=headers, content=body)
@pytest.fixture
def app(tmp_path, pds):
def web():
return FakeWeb()
def routed(pds, web):
return httpx.MockTransport(
lambda r: pds(r) if r.url.host in ("bsky.example", "pds.example") else web(r))
@pytest.fixture
def app(tmp_path, pds, web):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
@@ -108,7 +131,7 @@ def app(tmp_path, pds):
"BLUESKY_SERVICE": SERVICE,
})
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"] = BlueskyClient(
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=httpx.MockTransport(pds))
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=routed(pds, web))
)
yield app
syndication._client_cache.clear()
@@ -258,6 +281,101 @@ class TestMedia:
assert image["aspectRatio"]["width"] > 0
ARTICLE = "https://news.example/story"
ARTICLE_HTML = b"""<html><head><title>Fallback title</title>
<meta property="og:title" content="A Great Story">
<meta property="og:description" content="What happened, in short.">
<meta property="og:image" content="/img/story.png">
</head><body>...</body></html>"""
def png_bytes():
buf = io.BytesIO()
Image.new("RGB", (60, 40), (10, 120, 200)).save(buf, format="PNG")
return buf.getvalue()
class TestLinkCards:
def test_note_link_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
web.pages["https://news.example/img/story.png"] = (200, {"content-type": "image/png"}, png_bytes())
note = publish(app, content=f"Read [this story]({ARTICLE}) now")
record = only_record(pds)
external = record["embed"]["external"]
assert external["uri"] == ARTICLE
assert external["title"] == "A Great Story"
assert external["description"] == "What happened, in short."
assert external["thumb"]["$type"] == "blob"
# The link text is clickable and the note's permalink is still in the text
permalink = f"https://blog.example.com/note/{note.slug}"
assert record["text"].startswith("Read this story now") and permalink in record["text"]
raw = record["text"].encode()
links = {raw[f["index"]["byteStart"]:f["index"]["byteEnd"]].decode(): f["features"][0]["uri"]
for f in record["facets"]}
assert links["this story"] == ARTICLE
assert links[permalink] == permalink
def test_bare_url_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
publish(app, content=f"See {ARTICLE}.")
assert only_record(pds)["embed"]["external"]["uri"] == ARTICLE
def test_unreachable_page_still_gets_a_card(self, app, pds, web):
note = publish(app, content=f"See {ARTICLE}")
external = only_record(pds)["embed"]["external"]
assert external == {"uri": ARTICLE, "title": ARTICLE, "description": ""}
assert row(app, note)["state"] == "synced"
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
monkeypatch.setattr("starpunk.micropub._public_address", lambda host: host != "internal.example")
app.config["TESTING"] = False
try:
publish(app, content=f"See {ARTICLE}")
finally:
app.config["TESTING"] = True
assert "http://internal.example/admin" not in web.fetched
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE
def test_images_win_over_link_card(self, app, pds, web):
from starpunk.media import attach_media_to_note, save_media
with app.test_request_context():
note = create_note(content=f"Look {ARTICLE}", published=True)
media = save_media(png_bytes(), "a.png")
attach_media_to_note(note.id, [media["id"]], ["pic"])
select_targets(note.id, ["bluesky"])
process_due()
assert only_record(pds)["embed"]["$type"] == "app.bsky.embed.images"
assert web.fetched == []
class TestLinkText:
def test_note_links_in_reading_order(self):
from starpunk.syndication import note_links
md = "a [x](https://one.example) b https://two.example/p. c <https://three.example> [rel](/local)"
assert note_links(md) == ["https://one.example", "https://two.example/p", "https://three.example"]
def test_link_modes(self):
from starpunk.syndication import plain_text_links
md = "[a story](https://s.example) and [https://u.example](https://u.example)"
assert plain_text_links(md)[0] == "a story and https://u.example"
assert plain_text_links(md, "inline")[0] == "a story (https://s.example) and https://u.example"
assert plain_text_links(md, "anchor") == ("a story and https://u.example",
[("a story", "https://s.example")])
def test_inline_url_budget(self):
from starpunk.syndication import plain_text_links
md = "https://bare.example then [one](https://1.example) [two](https://2.example) [one again](https://1.example)"
text = plain_text_links(md, "inline", max_urls=2)[0]
assert text == "https://bare.example then one (https://1.example) two one again (https://1.example)"
def test_anchor_facet_byte_offsets(self):
text = "Café ☕ this story and more"
(facet,) = bluesky_facets(text, [("this story", "https://s.example")])
raw = text.encode()
assert raw[facet["index"]["byteStart"]:facet["index"]["byteEnd"]] == b"this story"
def test_both_targets_listed_for_micropub(tmp_path):
data = tmp_path / "d"
app = create_app({
+418
View File
@@ -0,0 +1,418 @@
"""
Tests for crossposting to Threads (starpunk/syndication.py, v2.6.0)
A fake Graph API (httpx.MockTransport) records what StarPunk sends.
"""
import io
import json
import time
from urllib.parse import parse_qsl
import httpx
import pytest
from PIL import Image
from starpunk import create_app, syndication
from starpunk.notes import create_note, delete_note, update_note
from starpunk.syndication import (
THREADS_REFRESH_EVERY,
ThreadsClient,
get_note_syndications,
process_due,
select_targets,
threads_length,
threads_topic,
)
API = "https://graph.threads.example/v1.0"
UID = "1789"
class FakeThreads:
def __init__(self):
self.containers = {}
self.posts = {}
self.calls = []
self.token = "tok1"
self.refreshes = 0
self.refresh_fails = False
self.pending_polls = 0
self.lose_next_publish = False
self.fail_next = None
self.fail_next_publish = False
def _json(self, code, body):
return httpx.Response(code, json=body)
def _error(self, code, message, err_code, subcode=None):
error = {"message": message, "type": "OAuthException", "code": err_code}
if subcode:
error["error_subcode"] = subcode
return self._json(code, {"error": error})
def __call__(self, request: httpx.Request) -> httpx.Response:
params = dict(request.url.params)
if request.content:
params.update(parse_qsl(request.content.decode()))
path = request.url.path
self.calls.append((request.method, path, params))
if path == "/refresh_access_token":
assert params["grant_type"] == "th_refresh_token"
if self.refresh_fails or params.get("access_token") != self.token:
return self._error(400, "token too new", 190)
self.refreshes += 1
self.token = f"tok{self.refreshes + 1}"
return self._json(200, {"access_token": self.token, "token_type": "bearer",
"expires_in": 5184000})
if params.get("access_token") != self.token:
return self._error(400, "Invalid OAuth access token", 190)
if self.fail_next:
code, err, self.fail_next = *self.fail_next, None
return self._error(code, "nope", err)
path = path.removeprefix("/v1.0/")
if path == "me":
return self._json(200, {"id": UID, "username": "phil"})
if path == f"{UID}/threads" and request.method == "POST":
cid = f"c{len(self.containers) + 1}"
self.containers[cid] = params
return self._json(200, {"id": cid})
if path == f"{UID}/threads" and request.method == "GET":
return self._json(200, {"data": [
{"id": pid, "text": p["text"], "permalink": f"https://www.threads.com/@phil/post/{pid}"}
for pid, p in self.posts.items()]})
if path == f"{UID}/threads_publish" and self.fail_next_publish:
self.fail_next_publish = False
return self._error(400, "Fatal", 24, 4279009)
if path == f"{UID}/threads_publish":
container = self.containers[params["creation_id"]]
pid = f"p{len(self.posts) + 1}"
self.posts[pid] = container
if self.lose_next_publish:
self.lose_next_publish = False
raise httpx.ReadTimeout("lost reply")
return self._json(200, {"id": pid})
if path in self.containers and request.method == "GET":
if self.pending_polls:
self.pending_polls -= 1
return self._json(200, {"status": "IN_PROGRESS"})
return self._json(200, {"status": "FINISHED"})
if path in self.posts and request.method == "GET":
return self._json(200, {"id": path, "permalink": f"https://www.threads.com/@phil/post/{path}"})
if request.method == "DELETE":
if self.posts.pop(path, None) is None:
return self._error(400, "Object does not exist", 100, 33)
return self._json(200, {"success": True, "deleted_id": path})
return self._json(500, {"error": {"message": "unknown", "code": 2}})
@pytest.fixture
def api():
return FakeThreads()
def make_client(app, api, token="tok1"):
return ThreadsClient(token, app.config["DATA_PATH"] / "threads_token.json", api=API,
http=httpx.Client(transport=httpx.MockTransport(api)), sleep=lambda s: None)
@pytest.fixture
def app(tmp_path, api):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
"SITE_URL": "https://blog.example.com",
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"THREADS_ACCESS_TOKEN": "tok1",
"THREADS_USERNAME": "phil",
})
syndication._client_cache["threads"] = make_client(app, api)
yield app
syndication._client_cache.clear()
def publish(app, content="Hello Threads", tags=None):
with app.test_request_context():
note = create_note(content=content, published=True, tags=tags)
select_targets(note.id, ["threads"])
process_due()
return note
def only_post(api):
(post,) = api.posts.values()
return post
def row(app, note):
with app.app_context():
(r,) = get_note_syndications(note.id)
return r
def image_bytes(fmt="JPEG", size=(800, 600)):
buf = io.BytesIO()
Image.new("RGB", size, (200, 30, 30)).save(buf, format=fmt)
return buf.getvalue()
class TestText:
def test_emoji_count_as_bytes(self):
assert threads_length("abc") == 3
assert threads_length("☕") == 3
def test_topic_is_first_usable_tag(self):
assert threads_topic(["...", "Indie.Web & Co", "x"]) == "IndieWeb Co"
assert threads_topic([]) == ""
class TestLifecycle:
def test_publish_text_post(self, app, api):
note = publish(app, content="Hello **Threads**", tags=["IndieWeb", "Second"])
post = only_post(api)
permalink = f"https://blog.example.com/note/{note.slug}"
assert post["media_type"] == "TEXT"
assert post["text"] == f"Hello Threads\n\n{permalink}"
assert "link_attachment" not in post # Threads previews the first link itself
assert post["topic_tag"] == "IndieWeb"
r = row(app, note)
assert r["state"] == "synced"
assert r["remote_id"] == "p1"
assert r["remote_url"] == "https://www.threads.com/@phil/post/p1"
def test_long_note_fits_500(self, app, api):
note = publish(app, content="☕ word " * 300)
text = only_post(api)["text"]
assert threads_length(text) <= 500
assert text.endswith(f"https://blog.example.com/note/{note.slug}")
def test_waits_for_container(self, app, api):
api.pending_polls = 2
note = publish(app)
assert row(app, note)["state"] == "synced"
def test_edit_leaves_post_alone(self, app, api):
note = publish(app, content="first")
with app.test_request_context():
update_note(id=note.id, content="second")
process_due()
assert only_post(api)["text"].startswith("first")
assert row(app, note)["state"] == "synced"
def test_delete_and_unpublish_remove_post(self, app, api):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, published=False)
process_due()
assert api.posts == {}
update_note(id=note.id, published=True)
process_due()
assert len(api.posts) == 1
delete_note(id=note.id)
process_due()
assert api.posts == {}
def test_post_deleted_on_threads_is_fine(self, app, api):
note = publish(app)
api.posts.clear()
with app.test_request_context():
delete_note(id=note.id)
process_due()
assert row(app, note)["state"] == "synced"
class TestErrors:
def test_lost_reply_does_not_double_post(self, app, api):
api.lose_next_publish = True
note = publish(app)
assert row(app, note)["state"] == "pending"
with app.test_request_context():
from starpunk.database import get_db
get_db(app).execute("UPDATE syndications SET next_attempt_at = datetime('now')")
get_db(app).commit()
process_due()
assert len(api.posts) == 1
assert row(app, note)["state"] == "synced"
def test_publish_fatal_is_retried(self, app, api):
api.fail_next_publish = True
note = publish(app)
r = row(app, note)
assert r["state"] == "pending" and "Fatal" in r["last_error"] and "subcode 4279009" in r["last_error"]
with app.test_request_context():
from starpunk.database import get_db
get_db(app).execute("UPDATE syndications SET next_attempt_at = datetime('now')")
get_db(app).commit()
process_due()
assert len(api.posts) == 1
assert row(app, note)["state"] == "synced"
def test_bad_token_fails_permanently(self, app, api):
api.token = "other"
note = publish(app)
r = row(app, note)
assert r["state"] == "failed" and "Invalid OAuth" in r["last_error"]
assert "tok1" not in r["last_error"]
def test_rate_limit_retries(self, app, api):
publish(app, content="warm up")
api.fail_next = (400, 4) # Graph "application request limit reached"
note = publish(app, content="later")
assert row(app, note)["state"] == "pending"
class TestLinks:
def test_markdown_link_comes_first_for_the_preview(self, app, api):
note = publish(app, content="Read [this story](https://news.example/story) now")
text = only_post(api)["text"]
assert text.startswith("Read this story (https://news.example/story) now")
assert text.index("https://news.example/story") < text.index(f"/note/{note.slug}")
def test_stays_within_five_links(self, app, api):
import re
links = " ".join(f"[l{i}](https://l{i}.example)" for i in range(7))
publish(app, content=f"Lots: {links}")
text = only_post(api)["text"]
assert len(set(re.findall(r"https?://[^\s)]+", text))) == 5
assert "l6" in text # the extra links keep their text
class TestMedia:
def _note_with(self, app, images):
from starpunk.media import attach_media_to_note, save_media
with app.test_request_context():
note = create_note(content="pics", published=True)
ids = [save_media(data, name)["id"] for data, name in images]
attach_media_to_note(note.id, ids, [f"alt {i}" for i in range(len(ids))])
select_targets(note.id, ["threads"])
process_due()
return note
def test_single_image(self, app, api):
self._note_with(app, [(image_bytes(), "a.jpg")])
post = only_post(api)
assert post["media_type"] == "IMAGE"
assert post["image_url"].startswith("https://blog.example.com/media/")
assert post["alt_text"] == "alt 0"
assert "link_attachment" not in post
def test_carousel(self, app, api):
self._note_with(app, [(image_bytes(), "a.jpg"), (image_bytes("PNG"), "b.png")])
post = only_post(api)
assert post["media_type"] == "CAROUSEL"
children = [api.containers[c] for c in post["children"].split(",")]
assert [c["is_carousel_item"] for c in children] == ["true", "true"]
assert [c["alt_text"] for c in children] == ["alt 0", "alt 1"]
def test_unsupported_format_is_left_off(self, app, api):
self._note_with(app, [(image_bytes("WEBP"), "a.webp")])
assert only_post(api)["media_type"] == "TEXT"
class TestToken:
def test_refreshes_daily_and_saves(self, app, api):
client = syndication._client_cache["threads"]
with app.app_context():
client.maintain()
assert api.refreshes == 1
client.maintain() # same day: nothing
assert api.refreshes == 1
saved = json.loads((app.config["DATA_PATH"] / "threads_token.json").read_text())
assert saved["token"] == "tok2"
# A restart picks up the saved token
assert make_client(app, api)._token == "tok2"
publish(app)
assert len(api.posts) == 1
def test_other_worker_refresh_is_reused(self, app, api):
first, second = make_client(app, api), make_client(app, api)
with app.app_context():
first.maintain()
second.maintain()
assert api.refreshes == 1
assert second._token == "tok2"
def test_refresh_due_after_a_day(self, app, api, monkeypatch):
client = syndication._client_cache["threads"]
with app.app_context():
client.maintain()
later = time.time() + THREADS_REFRESH_EVERY + 1
monkeypatch.setattr(syndication.time, "time", lambda: later)
client.maintain()
assert api.refreshes == 2
def test_failed_refresh_keeps_token(self, app, api):
api.refresh_fails = True
client = syndication._client_cache["threads"]
with app.app_context():
client.maintain()
assert client._token == "tok1"
publish(app)
assert len(api.posts) == 1
def test_new_config_token_replaces_saved_one(self, app, api):
with app.app_context():
syndication._client_cache["threads"].maintain()
api.token = "fresh"
assert make_client(app, api, token="fresh")._token == "fresh"
def test_worker_runs_upkeep(self, app, api):
with app.app_context():
process_due()
assert api.refreshes == 1
def test_all_targets_listed_for_micropub(tmp_path):
data = tmp_path / "d"
app = create_app({
"TESTING": True, "SITE_URL": "https://b.example", "ADMIN_ME": "https://e.example",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "s.db",
"GOTOSOCIAL_URL": "https://social.example", "GOTOSOCIAL_ACCESS_TOKEN": "t",
"BLUESKY_HANDLE": "phil.example", "BLUESKY_APP_PASSWORD": "p",
"THREADS_ACCESS_TOKEN": "x", "THREADS_USERNAME": "@phil",
})
with app.app_context():
from starpunk.syndication import get_targets, resolve_targets
assert set(get_targets()) == {"gotosocial", "bluesky", "threads"}
assert resolve_targets(["https://www.threads.com/@phil"]) == ["threads"]
syndication._client_cache.clear()
class TestLikesReposts:
PAGE = "https://other.example/post/1"
def _post(self, app, monkeypatch, content="", **kind):
monkeypatch.setattr("starpunk.reply_context.fetch_reply_context", lambda *a, **k: None)
with app.test_request_context():
note = create_note(content=content, published=True, **kind)
select_targets(note.id, ["threads"])
process_due()
return note
def test_like_is_not_crossposted(self, app, api, monkeypatch):
note = self._post(app, monkeypatch, like_of=self.PAGE)
assert api.posts == {}
r = row(app, note)
assert r["state"] == "synced" and r["remote_id"] is None
def test_repost_posts_link(self, app, api, monkeypatch):
self._post(app, monkeypatch, repost_of=self.PAGE)
post = only_post(api)
assert post["text"].startswith("🔁") and self.PAGE in post["text"]
assert "link_attachment" not in post
def test_repost_with_comment_keeps_comment(self, app, api, monkeypatch):
note = self._post(app, monkeypatch, content="So **true**", repost_of=self.PAGE)
assert only_post(api)["text"].startswith("So true")
with app.test_request_context():
delete_note(id=note.id)
process_due()
assert api.posts == {}
+86
View File
@@ -0,0 +1,86 @@
"""
Tags keep the order they were entered in (v2.6.4)
Public pages list tags alphabetically; crossposts (the Threads topic, which
tags fit on Bluesky), the editor and Micropub use the entered order, and
saving from the editor or Micropub keeps it.
"""
import pytest
from starpunk import create_app
from starpunk.auth import create_session
from starpunk.indieauth import create_access_token
from starpunk.notes import create_note, get_note
from starpunk.tags import get_note_tags
ENTERED = ["Zebra", "apple", "Mango"]
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
data.mkdir()
yield create_app(config={
"TESTING": True, "DATABASE_PATH": data / "starpunk.db", "DATA_PATH": data,
"NOTES_PATH": data / "notes", "SESSION_SECRET": "s", "ADMIN_ME": "https://test.example.com",
"SITE_URL": "http://localhost:5000", "DEV_MODE": False,
})
@pytest.fixture
def note(app):
with app.test_request_context():
return create_note(content="Tagged", published=True, tags=ENTERED)
def names(app, note_id, **kw):
with app.app_context():
return [t["display_name"] for t in get_note_tags(note_id, **kw)]
def test_alphabetical_by_default_entered_on_request(app, note):
assert names(app, note.id) == ["apple", "Mango", "Zebra"]
assert names(app, note.id, entered_order=True) == ENTERED
def test_public_page_stays_alphabetical(app, note):
html = app.test_client().get(f"/note/{note.slug}").data.decode()
assert html.index(">apple<") < html.index(">Mango<") < html.index(">Zebra<")
def test_editor_shows_and_keeps_entered_order(app, note):
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://test.example.com"))
html = client.get(f"/admin/edit/{note.id}").data.decode()
assert 'value="Zebra, apple, Mango"' in html
# Saving the form unchanged keeps the order
response = client.post(f"/admin/edit/{note.id}", data={
"content": "Tagged, edited", "published": "on", "tags": "Zebra, apple, Mango"})
assert response.status_code in (200, 302)
assert names(app, note.id, entered_order=True) == ENTERED
def test_micropub_source_and_update_keep_entered_order(app, note):
with app.test_request_context():
token = create_access_token("https://quill.example/", "create update")["access_token"]
client = app.test_client()
auth = {"Authorization": f"Bearer {token}"}
url = f"http://localhost:5000/note/{note.slug}"
source = client.get(f"/micropub?q=source&url={url}", headers=auth).get_json()
assert source["properties"]["category"] == ENTERED
response = client.post("/micropub", headers=auth, json={
"action": "update", "url": url, "add": {"category": ["Kiwi"]}})
assert response.status_code in (200, 204), response.get_data(as_text=True)
assert names(app, note.id, entered_order=True) == ENTERED + ["Kiwi"]
def test_crossposts_use_entered_order(app, note):
from starpunk.syndication import compose_text, threads_topic
tags = names(app, note.id, entered_order=True)
assert threads_topic(tags) == "Zebra"
with app.app_context():
text = compose_text(get_note(id=note.id, load_content=True),
"https://x.example/note/1", tags, 25)
assert "#Zebra" in text and "#Mango" not in text # room for the first tags only
+283
View File
@@ -0,0 +1,283 @@
"""
W3C Webmention conformance tests (v2.7.0)
https://www.w3.org/TR/webmention/ — section numbers in test names.
"""
import sqlite3
from unittest.mock import patch
import httpx
import pytest
from starpunk import webmention_ssrf
from starpunk.webmention_processor import (
TransientError, ValidationError, fetch_source, verify_link_presence,
)
from starpunk.webmention_worker import WebmentionWorker
PUBLIC = [(2, 1, 6, '', ('93.184.216.34', 0))]
BASE = 'https://t.example/test/1?x=1'
@pytest.fixture
def worker(app):
return WebmentionWorker(app, db_path=app.config["DATABASE_PATH"],
site_url=app.config["SITE_URL"])
@pytest.fixture
def mock_web():
"""Route safe_get through a MockTransport; all hosts resolve public"""
routes = {}
def handler(request):
return routes[str(request.url)](request)
with patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC), \
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
yield routes
# 3.1.2 Sender discovers receiver Webmention endpoint
@pytest.mark.parametrize("header,expected", [
('<https://e.example/wm>; rel="webmention"', 'https://e.example/wm'),
('</wm>; rel=webmention', 'https://t.example/wm'),
('<wm>; rel="other webmention"', 'https://t.example/test/wm'),
('<wm>; title="a;b,c"; rel="webmention"', 'https://t.example/test/wm'),
('<wm>; REL="WebMention"', 'https://t.example/test/wm'),
('<a>; rel="other", </wm?q=1>; rel="webmention"', 'https://t.example/wm?q=1'),
('<x>; rel="webmention-not"', None),
('<x>; rel="not-webmention"', None),
])
def test_3_1_2_link_header(worker, header, expected):
assert worker._parse_link_header(header, BASE) == expected
@pytest.mark.parametrize("html,expected", [
# <a> before <link>: first in document order wins
('<a rel="webmention" href="/a">x</a><link rel="webmention" href="/l">', 'https://t.example/a'),
('<link rel="webmention somethingelse" href="/l">', 'https://t.example/l'),
('<link rel="Webmention" href="/l">', 'https://t.example/l'),
# empty href is the target itself
('<link rel="webmention" href="">', BASE),
# no href: skipped
('<link rel="webmention"><a rel="webmention" href="/a">', 'https://t.example/a'),
# comments, escaped markup and other rels are ignored
('<!-- <link rel="webmention" href="/bad"> -->'
'<a rel="not-webmention" href="/bad"></a>'
'<code>&lt;a rel="webmention" href="/bad"&gt;</code>'
'<link rel="webmention" href="/ok">', 'https://t.example/ok'),
('<p>nothing</p>', None),
])
def test_3_1_2_html(worker, html, expected):
assert worker._parse_html_for_endpoint(html, BASE) == expected
def test_3_1_2_header_before_html_and_relative_to_final_url(app, worker, mock_web):
mock_web['https://t.example/old'] = lambda r: httpx.Response(
301, headers={'Location': 'https://t.example/new/page'})
mock_web['https://t.example/new/page'] = lambda r: httpx.Response(
200, headers={'Link': '<wm>; rel="webmention"', 'Content-Type': 'text/html'},
text='<link rel="webmention" href="/html">')
with app.app_context():
assert worker._discover_endpoint('https://t.example/old') == 'https://t.example/new/wm'
def test_3_1_2_multiple_link_headers(app, worker, mock_web):
mock_web['https://t.example/p'] = lambda r: httpx.Response(
200, headers=[('Link', '</other>; rel="other"'), ('Link', '</wm>; rel="webmention"')])
with app.app_context():
assert worker._discover_endpoint('https://t.example/p') == 'https://t.example/wm'
def test_3_1_2_server_error_is_retried_not_final(app, worker, mock_web):
mock_web['https://t.example/p'] = lambda r: httpx.Response(503)
with app.app_context(), pytest.raises(httpx.HTTPError):
worker._discover_endpoint('https://t.example/p')
# 4.5 SSRF: every redirect hop is checked
def test_4_5_redirect_to_private_address_blocked(app):
def resolve(host, *a, **k):
ip = '169.254.169.254' if host == 'metadata.internal' else '93.184.216.34'
return [(2, 1, 6, '', (ip, 0))]
def handler(request):
return httpx.Response(302, headers={'Location': 'http://metadata.internal/'})
with app.app_context(), \
patch("starpunk.webmention_ssrf.socket.getaddrinfo", side_effect=resolve), \
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler)):
with pytest.raises(ValidationError, match="Blocked"):
fetch_source('https://evil.example/', 'http://localhost:5000')
@pytest.mark.parametrize("ip", [
'127.0.0.1', '10.1.2.3', '169.254.169.254', '0.0.0.0', '100.64.0.1',
'::1', '::ffff:127.0.0.1', 'fe80::1', '224.0.0.1',
])
def test_4_5_blocked_addresses(ip):
assert webmention_ssrf.is_private_ip(ip)
def test_4_5_any_blocked_address_blocks(app):
mixed = PUBLIC + [(10, 1, 6, '', ('::1', 0, 0, 0))]
with app.app_context(), patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=mixed):
with pytest.raises(webmention_ssrf.SSRFBlockedError):
webmention_ssrf.validate_url_for_fetch('https://x.example/')
# 3.2.2 Webmention verification
TARGET = 'https://me.example/note/abc'
@pytest.mark.parametrize("content,content_type,expected", [
('<a href="/note/abc#c">x</a>', 'text/html', True),
('<img src="https://me.example/note/abc">', 'text/html; charset=utf-8', True),
('<a href="https://me.example/note/abcdef">x</a>', 'text/html', False),
('<p>https://me.example/note/abc</p>', 'text/html', False),
('<!-- <a href="https://me.example/note/abc"> -->', 'text/html', False),
('{"a": ["https://me.example/note/abc"]}', 'application/json', True),
('{"a": "https://me.example/note/abcdef"}', 'application/json', False),
('see https://me.example/note/abc', 'text/plain', True),
])
def test_3_2_2_exact_match(content, content_type, expected):
assert verify_link_presence(content, TARGET, content_type, 'https://me.example/x') is expected
def test_3_2_2_accept_header_and_status(app, mock_web):
seen = {}
def page(request):
seen['accept'] = request.headers.get('Accept')
return httpx.Response(410)
mock_web['https://s.example/p'] = page
with app.app_context():
with pytest.raises(ValidationError, match="410"):
fetch_source('https://s.example/p', 'http://localhost:5000')
assert 'text/html' in seen['accept']
mock_web['https://s.example/p'] = lambda r: httpx.Response(500)
with app.app_context(), pytest.raises(TransientError):
fetch_source('https://s.example/p', 'http://localhost:5000')
# 3.2.1 Request verification
def _post(client, source, target):
return client.post('/webmention', data={'source': source, 'target': target})
def test_3_2_1_target_must_be_a_note(app, client, sample_note):
note_url = f"http://localhost:5000/note/{sample_note.slug}"
assert _post(client, 'https://s.example/a', note_url).status_code == 202
assert _post(client, 'https://s.example/a', 'http://localhost:5000/nope').status_code == 400
assert _post(client, 'https://s.example/a', 'http://localhost:5000/').status_code == 400
assert _post(client, 'https://s.example/a', 'https://other.example/note/x').status_code == 400
assert _post(client, note_url, note_url).status_code == 400
assert _post(client, 'ftp://s.example/a', note_url).status_code == 400
def test_3_2_1_host_case_and_fragment(app, client, sample_note):
note_url = f"http://LOCALHOST:5000/note/{sample_note.slug}"
assert _post(client, 'https://s.example/a', note_url + '#one').status_code == 202
assert _post(client, 'https://s.example/a', note_url + '#two').status_code == 202
db = sqlite3.connect(app.config['DATABASE_PATH'])
rows = db.execute("SELECT target_url FROM webmentions").fetchall()
assert rows == [(note_url,)]
# 3.2.4 Updating existing Webmentions
def _insert_mention(app, note_id, **cols):
db = sqlite3.connect(app.config['DATABASE_PATH'])
fields = {'source_url': 'https://s.example/a', 'target_url': 'http://localhost:5000/n',
'target_note_id': note_id, **cols}
db.execute(f"INSERT INTO webmentions ({', '.join(fields)}) VALUES ({', '.join('?' * len(fields))})",
tuple(fields.values()))
db.commit()
return db
VALID = {'valid': True, 'mention_type': 'reply', 'author_name': 'A', 'author_url': 'https://a.example/',
'author_photo': None, 'content_snippet': 'hi'}
def test_3_2_4_update_keeps_moderation(app, worker, sample_note):
db = _insert_mention(app, sample_note.id, validation_status='valid', moderation_status='hidden',
validated_at='2026-01-01')
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
assert db.execute("SELECT validation_status, moderation_status FROM webmentions").fetchone() == ('valid', 'hidden')
def test_3_2_4_new_mention_gets_default_moderation(app, worker, sample_note):
db = _insert_mention(app, sample_note.id)
with patch("starpunk.webmention_worker.process_webmention", return_value=VALID):
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
assert db.execute("SELECT validation_status, moderation_status FROM webmentions").fetchone() == ('valid', 'approved')
def test_3_2_4_transient_error_keeps_valid_mention(app, worker, sample_note):
db = _insert_mention(app, sample_note.id, validation_status='pending', moderation_status='approved',
validated_at='2026-01-01', content_snippet='kept')
with patch("starpunk.webmention_worker.process_webmention", side_effect=TransientError("timeout")):
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
assert db.execute("SELECT validation_status, content_snippet FROM webmentions").fetchone() == ('valid', 'kept')
def test_3_2_4_gone_source_removes_mention(app, worker, sample_note):
db = _insert_mention(app, sample_note.id, validation_status='pending', moderation_status='approved',
validated_at='2026-01-01')
with patch("starpunk.webmention_worker.process_webmention",
side_effect=ValidationError("Source returned 410")):
worker._validate_webmention(1, 'https://s.example/a', 'http://localhost:5000/n', sample_note.id)
assert db.execute("SELECT validation_status FROM webmentions").fetchone() == ('invalid',)
# 3.1.4 Updated posts / 3.1.5 Deleted posts
def _queue(app):
db = sqlite3.connect(app.config['DATABASE_PATH'])
return db.execute(
"SELECT target_url, status, source_note_id FROM webmention_queue ORDER BY target_url"
).fetchall()
def _sent(app):
db = sqlite3.connect(app.config['DATABASE_PATH'])
db.execute("UPDATE webmention_queue SET status = 'sent'")
db.commit()
def test_3_1_4_update_resends_previous_targets(app):
from starpunk.notes import create_note, update_note
with app.test_request_context():
note = create_note("Reply", published=True, in_reply_to="https://a.example/post")
_sent(app)
update_note(id=note.id, content="Now about [b](https://b.example/x)", in_reply_to="")
targets = {t: s for t, s, _ in _queue(app)}
assert targets == {'https://a.example/post': 'pending', 'https://b.example/x': 'pending'}
@pytest.mark.parametrize("soft", [True, False])
def test_3_1_5_delete_resends_and_returns_410(app, client, soft):
from starpunk.notes import create_note, delete_note
with app.test_request_context():
note = create_note("See [a](https://a.example/x)", published=True)
_sent(app)
delete_note(id=note.id, soft=soft)
rows = _queue(app)
assert [(t, s) for t, s, _ in rows] == [('https://a.example/x', 'pending')]
if not soft:
assert rows[0][2] is None # detached, so it survives the delete
assert client.get(f"/note/{note.slug}").status_code == 410
assert client.get("/note/never-existed").status_code == 404
+131
View File
@@ -0,0 +1,131 @@
"""
Webmention worker regression tests (v2.6.8)
The worker thread used Flask's current_app (via the SSRF checks) without an
app context, so every send and validation failed. It was also stopped
whenever any app context ended.
"""
import sqlite3
import httpx
import time
from unittest.mock import MagicMock, patch
from starpunk.webmention_worker import WebmentionWorker
PUBLIC = [(2, 1, 6, '', ('93.184.216.34', 0))]
def _wait_for(predicate, timeout=5):
deadline = time.time() + timeout
while time.time() < deadline:
if predicate():
return True
time.sleep(0.05)
return False
def test_worker_sends_queued_webmention(app):
db_path = app.config["DATABASE_PATH"]
conn = sqlite3.connect(db_path)
conn.execute(
"INSERT INTO webmention_queue (source_url, target_url, status) "
"VALUES (?, ?, 'pending')",
("http://localhost:5000/note/1", "https://example.org/post"),
)
conn.commit()
conn.close()
def target_site(request):
return httpx.Response(200, headers={"Link": '</wm>; rel="webmention"'})
accepted = MagicMock(status_code=202, text="")
worker = WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"],
poll_interval=0.05)
with patch("starpunk.webmention_ssrf.socket.getaddrinfo", return_value=PUBLIC), \
patch("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(target_site)), \
patch("starpunk.webmention_worker.httpx.post", return_value=accepted) as post:
worker.start()
try:
def status():
c = sqlite3.connect(db_path)
row = c.execute("SELECT status, last_error FROM webmention_queue").fetchone()
c.close()
return row
assert _wait_for(lambda: status()[0] != "pending"), status()
assert status() == ("sent", None)
post.assert_called_once()
assert post.call_args.args[0] == "https://example.org/wm"
finally:
worker.stop()
worker.join(timeout=2)
def test_worker_survives_app_context_teardown(app):
worker = WebmentionWorker(app, db_path=app.config["DATABASE_PATH"],
site_url=app.config["SITE_URL"], poll_interval=0.05)
worker.start()
try:
with app.app_context():
pass
app.test_client().get("/")
time.sleep(0.2)
assert worker.is_alive()
finally:
worker.stop()
worker.join(timeout=2)
def test_only_one_worker_processes_queues(app):
"""Each gunicorn process starts a worker; only one may hold the queue lock"""
kwargs = dict(db_path=app.config["DATABASE_PATH"], site_url=app.config["SITE_URL"])
first = WebmentionWorker(app, **kwargs)
second = WebmentionWorker(app, **kwargs)
try:
assert first._acquire_lock()
assert not second._acquire_lock()
first._release_lock()
assert second._acquire_lock()
finally:
first._release_lock()
second._release_lock()
def test_author_photo_with_alt_text_is_a_string():
"""mf2py returns u-photo with alt as a dict; the database needs a URL string"""
import mf2py
from starpunk.webmention_processor import extract_author
html = """
<div class="h-entry">
<span class="p-author h-card">
<a class="u-url p-name" href="https://a.example/">A Person</a>
<img class="u-photo" src="https://a.example/me.jpg" alt=""></span>
<a href="https://site.example/note/1">target</a>
</div>"""
author = extract_author(mf2py.parse(doc=html, url="https://src.example/"),
"https://src.example/")
assert author == {
"author_name": "A Person",
"author_url": "https://a.example/",
"author_photo": "https://a.example/me.jpg",
}
def test_author_urls_must_be_http():
"""A javascript: author URL would be rendered as a clickable link (XSS)"""
import mf2py
from starpunk.webmention_processor import extract_author
html = """
<div class="h-entry">
<span class="p-author h-card">
<a class="u-url p-name" href="javascript:alert(1)">Evil</a>
<img class="u-photo" src="data:image/svg+xml,x" alt=""></span>
</div>"""
author = extract_author(mf2py.parse(doc=html, url="https://src.example/"),
"https://src.example/")
assert author["author_url"] == "https://src.example/"
assert author["author_photo"] is None