Files
StarPunk/docs/architecture/syndication-architecture.md
T
PhilandClaude Opus 5.5 aa88db50b2 feat: admin Log and Settings pages, decluttered Webmentions page (v2.18.0)
Notifications move to /admin/settings. /admin/log lists everything failing
now (crossposts, response checks, outgoing webmentions, reply contexts,
WebSub, notifications, refused silo credentials) with the affected post,
the error and Retry/Check now. Back to Dashboard button removed; nav gains
Webmentions, Log (with error count) and Settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:05:26 -07:00

17 KiB
Raw Permalink Blame History

Syndication (POSSE) Architecture

Current as of v2.14.0. Code: starpunk/syndication.py, starpunk/backfeed.py.

StarPunk publishes on its own site first and then copies each note to other networks (POSSE: Publish on your Own Site, Syndicate Elsewhere). Later edits, unpublishing and deletes follow the copies.

Feeds (RSS, Atom, JSON Feed, OPML) and WebSub are pull and push distribution of the site itself, not crossposting. They are described in indieweb-protocols.md. This document covered feeds before v2.1.0.

Decisions: ADR-065 (GoToSocial, queue model), ADR-066 (Bluesky), ADR-068 (likes and reposts), ADR-069 (Threads), ADR-070 (links and link cards), ADR-073 (backfeed), ADR-074 (notifications, including crossposting problems), and the v2.9.0 Micropub review (bookmarks, mp-syndicate-to default).

Targets

A target exists when its credentials are configured. get_targets() builds the list from config on each call and caches one client object per target in the process.

Target id Client Enabled by Micropub uid Text limit Images Edits follow?
gotosocial MastodonClient (any Mastodon-API server) GOTOSOCIAL_URL + GOTOSOCIAL_ACCESS_TOKEN GOTOSOCIAL_URL/ GOTOSOCIAL_MAX_CHARS (5000) GOTOSOCIAL_MAX_MEDIA (4) Yes
bluesky BlueskyClient (AT Protocol, app password) BLUESKY_HANDLE + BLUESKY_APP_PASSWORD (BLUESKY_SERVICE, default https://bsky.social) https://bsky.app/profile/<handle> BLUESKY_MAX_CHARS (300) 4 No
threads ThreadsClient (Meta Graph API) THREADS_ACCESS_TOKEN (THREADS_USERNAME for links) https://www.threads.com/@<username> THREADS_MAX_CHARS (500) THREADS_MAX_MEDIA (4) No

Display names come from GOTOSOCIAL_NAME, BLUESKY_NAME and THREADS_NAME. GoToSocial statuses use GOTOSOCIAL_VISIBILITY (default public). The GoToSocial token needs write:statuses and write:media, plus read:search and write:favourites for likes and boosts, and read:statuses and read:accounts (or read) for backfeed. Backfeed on Threads needs threads_read_replies and threads_manage_insights. The Threads client uses the API host graph.threads.com (v2.13.0).

Bluesky and Threads can't edit posts through their APIs. Once a copy exists there, edits to the note don't change it, but unpublish and delete still remove it.

Choosing targets

select_targets(note_id, target_ids) records which targets a note goes to. Every configured target not in the list is disabled; if it already has a copy, that copy is deleted. Targets the note never went to get no row.

Origin Targets
Admin editor, new note One checkbox per target, all checked by default
Admin editor, edit Checkboxes reflect the targets currently enabled; saving applies the selection
Micropub create Only the targets in mp-syndicate-to (form field, mp-syndicate-to[], or inside JSON properties). Values may be a target's uid (trailing slash ignored) or its id. No mp-syndicate-to means no crossposting (v2.9.0; before, all targets)
Micropub update / delete Target selection is unchanged; the change follows existing copies

Micropub clients learn the targets from q=syndicate-to and q=config ([{"uid", "name"}]). Selecting targets for a draft records them; nothing is sent until the note is published.

Data: the syndications table (migration 012)

Migration 020 adds the backfeed columns; see Backfeed and database.md.

One row per (note, target), unique on (note_id, target):

Column Meaning
enabled Whether the note should have a copy on this target
remote_id, remote_url The copy: status id, at:// URI, Threads media id, or a favourite:<id> / reblog:<id> marker for native likes and boosts
state pending, synced or failed
version Incremented on every change that needs a reconcile
attempts, last_error, next_attempt_at Retry bookkeeping
claimed_until Lease held by the worker processing the row

note_id deliberately has no foreign key, so the row outlives a hard delete and the worker can still remove the copy.

Change hooks

Event Call Effect
Admin create/edit select_targets() Rows set to pending, version bumped
Micropub create select_targets(resolve_targets(mp-syndicate-to)) After photos are attached
update_note() (admin or Micropub, including publish/unpublish) note_changed(id) Every row of the note set to pending, version bumped, attempts reset
Micropub photo changes note_changed(id) Same
delete_note() (soft or hard), undelete_note() note_changed(id) Same
Retry button on the edit page retry(id) That row set to pending now, attempts reset
Check now (POST /admin/syndication/<id>/check) backfeed.check_now(id) That copy's responses checked at the next worker pass
Backfill responses (POST /admin/backfeed/backfill) backfeed.start_backfill() Every eligible copy flagged for one check

Worker

SyndicationWorker is a daemon thread started in every gunicorn process when at least one target is configured (not under TESTING). It runs process_due() every SYNDICATION_POLL_INTERVAL seconds (default 10), and loops again immediately while there is work.

process_due():

  1. Calls each client's maintain() (the Threads token refresh).
  2. Selects up to 5 rows that are pending, due, and not claimed.
  3. Claims each row with an atomic UPDATE ... SET claimed_until = now + 5 min WHERE state = 'pending' AND (claimed_until IS NULL OR claimed_until < now). The other processes skip it. Unlike the Webmention and WebSub workers, there is no file lock: the row claim is the only coordination.
  4. Reconciles the row (below), then records the result.

Reconcile

note published and row enabled?
  no  → remote copy exists? delete it            → (None, None)
  yes → copy exists and target can't edit?        → keep it as is
        copy exists                                → update (if it's gone remotely: create again)
        no copy                                    → create

Deleted notes are not found by get_note(), so they count as "not published".

Versions

The row's version is read when the row is picked up. _finish() sets state = 'synced' only if the version is unchanged. If the note was edited while the send was in flight, the row stays pending and is reconciled again with the new content. _fail() does the same.

Retries and idempotency

  • SyndicationError(permanent=True) fails the row immediately. This covers HTTP 4xx other than 429, a missing target, images too large for Bluesky, a Threads container in ERROR/EXPIRED, and Graph API auth/permission errors (codes 10, 190, 200).
  • Other errors retry after min(30 s × 2^attempts, 6 h). After 8 attempts the row becomes failed. The error is shown on the edit page and on the admin Log page (v2.18.0), both with a Retry button. When a row finally fails, the owner also gets a problems notification (ADR-074).
  • RemoteGone (404, or Graph error 100/33) during an update means the copy was deleted remotely, so a new one is created. During a delete it counts as success.
  • Duplicate posts after a lost response are prevented as follows:
    • GoToSocial: an Idempotency-Key: starpunk-<row id>-<version> header.
    • Bluesky and Threads: on a retry (attempts > 0), the last 25 posts are searched for one containing the permalink (for Threads reposts, the reposted URL). A match is adopted instead of posting again.
  • Threads' publish step sometimes fails transiently ("Fatal"); it is retried unless the error is an auth error.

Composing posts

Post carries the note, permalink, tags, the target's character limit, media files, and for likes, reposts and bookmarks, the URL they point at and its fetched title (from the reply-context cache).

Text

compose_text() builds: body, a blank line, the permalink, a blank line, the hashtags. If that is too long, the body is cut at a word boundary with "…"; the permalink always survives. Hashtags (# plus the tag's display name without non-word characters) may take at most a third of the limit. A bookmark's body starts with 🔖 <bookmarked URL> (v2.9.0). A photo-only note starts with the permalink.

Target Body format Links Hashtags
GoToSocial The note's Markdown, sent with content_type: text/markdown; embedded images removed from the text (sent as attachments) As written In the text
Bluesky Plain text from the rendered Markdown Markdown link text becomes a clickable link facet; bare URLs and #tags get link/tag facets (UTF-8 byte offsets) In the text, as tag facets
Threads Plain text Markdown links are written text (url), up to 4 different URLs plus the permalink (Threads allows 5); the length counts non-ASCII characters as UTF-8 bytes None in the text; the first usable tag becomes the post's topic_tag
  • Bluesky: if the note has images, they are the embed. Otherwise, if the note links somewhere, StarPunk builds an app.bsky.embed.external card for the first link. page_card() fetches the page (at most 512 KB, every redirect hop checked against private addresses, 10 s per request, 5 redirects) and reads the Open Graph, Twitter or HTML title, description and image. The image (at most 5 MB) is shrunk under 1 MB and uploaded as the thumbnail. If the page can't be fetched, the card has just the URL. A note without links gets a card for its own permalink, built from the note without fetching anything.
  • Threads: no explicit link_attachment (Meta rejects one for a page it hasn't crawled yet). Threads previews the first link in the text by itself.
  • GoToSocial: the server makes its own previews.

Media

_media_files() collects the note's attached photos first, then images embedded in the note text that are in this site's media library (some Micropub clients put uploaded photos in the content). It stops at the target's image limit. The large variant is used when one exists. Alt text comes from the caption, or from the embedded image's alt.

Target Handling
GoToSocial Uploaded via POST /api/v2/media with description; re-uploaded on each edit
Bluesky uploadBlob; images over 1,000,000 bytes are re-encoded as JPEG at decreasing size and quality, and fail permanently if they still don't fit. Aspect ratio is sent
Threads Meta fetches each image from its public /media/ URL. Only JPEG and PNG up to 8 MB are used; others are left off. 1 image → IMAGE container, 2 or more → CAROUSEL of child containers. Alt text up to 1000 characters

Only notes of kind note carry media; likes, reposts and bookmarks don't.

Likes, reposts and bookmarks

Kind GoToSocial Bluesky Threads
Like of a post on that network Native favourite (favourite:<id>) Native app.bsky.feed.like Not crossposted (the API can't like)
Like of anything else Not crossposted Not crossposted Not crossposted
Repost without comment, of a post on that network Native boost (reblog:<id>) Native app.bsky.feed.repost Not native: posts 🔁 title + URL
Repost with comment, of a post on that network Status: comment, 🔁 title, URL Quote post (app.bsky.embed.record) Comment, 🔁 title, URL
Repost of another page Status: (comment,) 🔁 title, URL Post with an external card for the URL (Comment,) 🔁 title, URL
Bookmark Ordinary status starting 🔖 <url> Ordinary post starting 🔖 <url> Ordinary post starting 🔖 <url>

"On that network" is decided by the client. GoToSocial resolves the URL through /api/v2/search?resolve=true. Bluesky matches https://bsky.app/profile/<actor>/post/<rkey> and resolves it to a URI and CID. Native likes and boosts have no URL of their own and nothing to edit. Deleting the note undoes them (unfavourite / unreblog, or deleting the like or repost record). A like that isn't crossposted is recorded as synced with no remote copy.

u-syndication

templates/note.html lists every row that has a remote_url as "Also on ", using <a class="u-syndication" rel="syndication">. Micropub q=source returns the same URLs as the syndication property. The Webmention worker rejects a webmention whose source is one of the note's own copies (e.g. backfed by Bridgy), so a copy is never shown as a response to itself (v2.8.0).

Backfeed (v2.13.0, ADR-073)

starpunk/backfeed.py brings responses to the copies back to the note. After each crosspost pass, the worker loop calls backfeed.process_due(), which does nothing unless BACKFEED_ENABLED; otherwise it checks up to three synced copies of published notes (not likes or reposts) that are due, each under a lease (backfeed_claimed_until):

  • Reading. Each client has responses(remote_id, remote_url), which returns a Harvest: GoToSocial favourited_by, reblogged_by and public direct replies from context; Bluesky getLikes, getRepostedBy, getQuotes and getPostThread (depth 1), plus getProfiles for reply and quote authors' bios; Threads replies and insights (like, repost and quote counts). Lists are paged up to 1,000 items; a list cut short makes the harvest incomplete.
  • Storing. Responses are upserted into webmentions with origin (the target id) and remote_id, validation_status = 'valid', keyed by (source_url, target_url). Likes and reposts use the copy's URL plus #liked-by-<actor> / #reposted-by-<actor>. A new row gets the WEBMENTION_AUTO_APPROVE default; an existing row keeps its moderation. A complete harvest marks that silo's rows for the note that weren't found invalid ("No longer on "). Authors with #nobridge or #nobot in their bio, or Bluesky's !no-unauthenticated label, are not stored and their rows are deleted; since v2.13.3 their likes, reposts and quotes are counted anonymously instead (hidden_counts()). If the note's shown responses changed, its webmentions are re-sent (spec 3.1.4). syndications.likes_count, reposts_count and quotes_count hold the silo-only counts: all of Threads; elsewhere people who opted out. New responses found by a backfill, or by a copy's first check on a note older than a day, are stored as already notified (v2.14.0).
  • Schedule. backfeed_next_at follows the note's age: 15 minutes (day 1), 1 hour (week 1), 1 day (to 30 days), 1 week (to BACKFEED_MAX_AGE_DAYS), with up to 10% jitter; none after. Failures back off like crossposts; after 8 they wait a month.
  • Backfill. start_backfill() flags every eligible copy (backfeed_backfill = 1). Flagged copies are checked only when no routine check is due, at most one every 5 seconds across workers (site_state.backfeed_backfill_last). Start and finish times are in site_state.
  • Refused credentials. A 401/403 (or Graph auth error) puts the target in backfeed.denied until restart; the admin Log page says which scopes are needed, and a problems notification is sent. Crossposting is unaffected.

Avatars of all responses (received and backfed) are saved by starpunk/avatars.py and served from /avatars/<sha256>.webp; the webmention worker fills in missing ones every 10 minutes and removes unused ones weekly.

Client sessions and tokens

  • Bluesky: createSession with the app password, then refreshSession when the access JWT expires (a 401 or ExpiredToken triggers one re-login and retry). The PDS endpoint is taken from the DID document. Calls are serialized per client with a lock.
  • Threads: long-lived tokens expire after 60 days. maintain() tries a refresh (refresh_access_token, th_refresh_token) at most once a day. The current token is kept in DATA_PATH/threads_token.json (mode 0600, written atomically), together with a SHA-256 "seed" of the configured token. A different THREADS_ACCESS_TOKEN in the config therefore replaces the saved one. An exclusive flock on threads_token.lock makes one gunicorn process refresh while the others pick up its result. A failed refresh is retried an hour later (network error) or the next day (HTTP error, e.g. a token less than a day old). The httpx logger is raised to WARNING because Threads takes the token in the URL, and errors never echo the URL.
  • GoToSocial: a static bearer token.

Known limits

  • Bluesky and Threads copies don't follow edits (platform limitation). Unticking and re-ticking a target makes a new copy.
  • No per-target choice for Micropub updates: mp-syndicate-to is read only on create.
  • Likes of ordinary web pages, and all likes on Threads, stay on the site.