Files
StarPunk/docs/architecture/syndication-architecture.md
T
PhilandClaude Opus 5.5 75946be7b2 maint: Dependency upgrades and High backlog items (v2.11.0)
- All dependencies to current releases (97 known vulnerabilities -> 0),
  Python 3.14, uv pinned, requirements-test.txt, lxml/dateutil pinned
- CSRF tokens on admin forms, logout and IndieAuth consent
- IndieAuth client and crosspost link-card fetches via safe_get
- FLASK_DEBUG/FLASK_ENV default off/production; image sets them
- Foreign keys enforced on pooled connections; migration 018 removes orphans
- Rate limits shared across processes (rate_limits, migration 019)
- datetime.utcnow() and sqlite3 default adapters replaced
- compose.yaml fixed; duplicate dev_auth.py and test.ini removed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:09:52 -07:00

13 KiB
Raw Blame History

Syndication (POSSE) Architecture

Current as of v2.10.1. Code: starpunk/syndication.py.

StarPunk publishes on its own site first and then copies each note to other networks (POSSE: Publish on your Own Site, Syndicate Elsewhere). Later edits, unpublishing and deletes follow the copies.

Feeds (RSS, Atom, JSON Feed, OPML) and WebSub are pull and push distribution of the site itself, not crossposting. They are described in indieweb-protocols.md. This document covered feeds before v2.1.0.

Decisions: ADR-065 (GoToSocial, queue model), ADR-066 (Bluesky), ADR-068 (likes and reposts), ADR-069 (Threads), ADR-070 (links and link cards), and the v2.9.0 Micropub review (bookmarks, mp-syndicate-to default).

Targets

A target exists when its credentials are configured. get_targets() builds the list from config on each call and caches one client object per target in the process.

Target id Client Enabled by Micropub uid Text limit Images Edits follow?
gotosocial MastodonClient (any Mastodon-API server) GOTOSOCIAL_URL + GOTOSOCIAL_ACCESS_TOKEN GOTOSOCIAL_URL/ GOTOSOCIAL_MAX_CHARS (5000) GOTOSOCIAL_MAX_MEDIA (4) Yes
bluesky BlueskyClient (AT Protocol, app password) BLUESKY_HANDLE + BLUESKY_APP_PASSWORD (BLUESKY_SERVICE, default https://bsky.social) https://bsky.app/profile/<handle> BLUESKY_MAX_CHARS (300) 4 No
threads ThreadsClient (Meta Graph API) THREADS_ACCESS_TOKEN (THREADS_USERNAME for links) https://www.threads.com/@<username> THREADS_MAX_CHARS (500) THREADS_MAX_MEDIA (4) No

Display names come from GOTOSOCIAL_NAME, BLUESKY_NAME and THREADS_NAME. GoToSocial statuses use GOTOSOCIAL_VISIBILITY (default public). The GoToSocial token needs write:statuses and write:media, plus read:search and write:favourites for likes and boosts.

Bluesky and Threads can't edit posts through their APIs. Once a copy exists there, edits to the note don't change it, but unpublish and delete still remove it.

Choosing targets

select_targets(note_id, target_ids) records which targets a note goes to. Every configured target not in the list is disabled; if it already has a copy, that copy is deleted. Targets the note never went to get no row.

Origin Targets
Admin editor, new note One checkbox per target, all checked by default
Admin editor, edit Checkboxes reflect the targets currently enabled; saving applies the selection
Micropub create Only the targets in mp-syndicate-to (form field, mp-syndicate-to[], or inside JSON properties). Values may be a target's uid (trailing slash ignored) or its id. No mp-syndicate-to means no crossposting (v2.9.0; before, all targets)
Micropub update / delete Target selection is unchanged; the change follows existing copies

Micropub clients learn the targets from q=syndicate-to and q=config ([{"uid", "name"}]). Selecting targets for a draft records them; nothing is sent until the note is published.

Data: the syndications table (migration 012)

One row per (note, target), unique on (note_id, target):

Column Meaning
enabled Whether the note should have a copy on this target
remote_id, remote_url The copy: status id, at:// URI, Threads media id, or a favourite:<id> / reblog:<id> marker for native likes and boosts
state pending, synced or failed
version Incremented on every change that needs a reconcile
attempts, last_error, next_attempt_at Retry bookkeeping
claimed_until Lease held by the worker processing the row

note_id deliberately has no foreign key, so the row outlives a hard delete and the worker can still remove the copy.

Change hooks

Event Call Effect
Admin create/edit select_targets() Rows set to pending, version bumped
Micropub create select_targets(resolve_targets(mp-syndicate-to)) After photos are attached
update_note() (admin or Micropub, including publish/unpublish) note_changed(id) Every row of the note set to pending, version bumped, attempts reset
Micropub photo changes note_changed(id) Same
delete_note() (soft or hard), undelete_note() note_changed(id) Same
Retry button on the edit page retry(id) That row set to pending now, attempts reset

Worker

SyndicationWorker is a daemon thread started in every gunicorn process when at least one target is configured (not under TESTING). It runs process_due() every SYNDICATION_POLL_INTERVAL seconds (default 10), and loops again immediately while there is work.

process_due():

  1. Calls each client's maintain() (the Threads token refresh).
  2. Selects up to 5 rows that are pending, due, and not claimed.
  3. Claims each row with an atomic UPDATE ... SET claimed_until = now + 5 min WHERE state = 'pending' AND (claimed_until IS NULL OR claimed_until < now). The other processes skip it. Unlike the Webmention and WebSub workers, there is no file lock: the row claim is the only coordination.
  4. Reconciles the row (below), then records the result.

Reconcile

note published and row enabled?
  no  → remote copy exists? delete it            → (None, None)
  yes → copy exists and target can't edit?        → keep it as is
        copy exists                                → update (if it's gone remotely: create again)
        no copy                                    → create

Deleted notes are not found by get_note(), so they count as "not published".

Versions

The row's version is read when the row is picked up. _finish() sets state = 'synced' only if the version is unchanged. If the note was edited while the send was in flight, the row stays pending and is reconciled again with the new content. _fail() does the same.

Retries and idempotency

  • SyndicationError(permanent=True) fails the row immediately. This covers HTTP 4xx other than 429, a missing target, images too large for Bluesky, a Threads container in ERROR/EXPIRED, and Graph API auth/permission errors (codes 10, 190, 200).
  • Other errors retry after min(30 s × 2^attempts, 6 h). After 8 attempts the row becomes failed. The error is shown on the edit page with a Retry button.
  • RemoteGone (404, or Graph error 100/33) during an update means the copy was deleted remotely, so a new one is created. During a delete it counts as success.
  • Duplicate posts after a lost response are prevented as follows:
    • GoToSocial: an Idempotency-Key: starpunk-<row id>-<version> header.
    • Bluesky and Threads: on a retry (attempts > 0), the last 25 posts are searched for one containing the permalink (for Threads reposts, the reposted URL). A match is adopted instead of posting again.
  • Threads' publish step sometimes fails transiently ("Fatal"); it is retried unless the error is an auth error.

Composing posts

Post carries the note, permalink, tags, the target's character limit, media files, and for likes, reposts and bookmarks, the URL they point at and its fetched title (from the reply-context cache).

Text

compose_text() builds: body, a blank line, the permalink, a blank line, the hashtags. If that is too long, the body is cut at a word boundary with "…"; the permalink always survives. Hashtags (# plus the tag's display name without non-word characters) may take at most a third of the limit. A bookmark's body starts with 🔖 <bookmarked URL> (v2.9.0). A photo-only note starts with the permalink.

Target Body format Links Hashtags
GoToSocial The note's Markdown, sent with content_type: text/markdown; embedded images removed from the text (sent as attachments) As written In the text
Bluesky Plain text from the rendered Markdown Markdown link text becomes a clickable link facet; bare URLs and #tags get link/tag facets (UTF-8 byte offsets) In the text, as tag facets
Threads Plain text Markdown links are written text (url), up to 4 different URLs plus the permalink (Threads allows 5); the length counts non-ASCII characters as UTF-8 bytes None in the text; the first usable tag becomes the post's topic_tag
  • Bluesky: if the note has images, they are the embed. Otherwise, if the note links somewhere, StarPunk builds an app.bsky.embed.external card for the first link. page_card() fetches the page (at most 512 KB, every redirect hop checked against private addresses, 10 s per request, 5 redirects) and reads the Open Graph, Twitter or HTML title, description and image. The image (at most 5 MB) is shrunk under 1 MB and uploaded as the thumbnail. If the page can't be fetched, the card has just the URL. A note without links gets a card for its own permalink, built from the note without fetching anything.
  • Threads: no explicit link_attachment (Meta rejects one for a page it hasn't crawled yet). Threads previews the first link in the text by itself.
  • GoToSocial: the server makes its own previews.

Media

_media_files() collects the note's attached photos first, then images embedded in the note text that are in this site's media library (some Micropub clients put uploaded photos in the content). It stops at the target's image limit. The large variant is used when one exists. Alt text comes from the caption, or from the embedded image's alt.

Target Handling
GoToSocial Uploaded via POST /api/v2/media with description; re-uploaded on each edit
Bluesky uploadBlob; images over 1,000,000 bytes are re-encoded as JPEG at decreasing size and quality, and fail permanently if they still don't fit. Aspect ratio is sent
Threads Meta fetches each image from its public /media/ URL. Only JPEG and PNG up to 8 MB are used; others are left off. 1 image → IMAGE container, 2 or more → CAROUSEL of child containers. Alt text up to 1000 characters

Only notes of kind note carry media; likes, reposts and bookmarks don't.

Likes, reposts and bookmarks

Kind GoToSocial Bluesky Threads
Like of a post on that network Native favourite (favourite:<id>) Native app.bsky.feed.like Not crossposted (the API can't like)
Like of anything else Not crossposted Not crossposted Not crossposted
Repost without comment, of a post on that network Native boost (reblog:<id>) Native app.bsky.feed.repost Not native: posts 🔁 title + URL
Repost with comment, of a post on that network Status: comment, 🔁 title, URL Quote post (app.bsky.embed.record) Comment, 🔁 title, URL
Repost of another page Status: (comment,) 🔁 title, URL Post with an external card for the URL (Comment,) 🔁 title, URL
Bookmark Ordinary status starting 🔖 <url> Ordinary post starting 🔖 <url> Ordinary post starting 🔖 <url>

"On that network" is decided by the client. GoToSocial resolves the URL through /api/v2/search?resolve=true. Bluesky matches https://bsky.app/profile/<actor>/post/<rkey> and resolves it to a URI and CID. Native likes and boosts have no URL of their own and nothing to edit. Deleting the note undoes them (unfavourite / unreblog, or deleting the like or repost record). A like that isn't crossposted is recorded as synced with no remote copy.

u-syndication

templates/note.html lists every row that has a remote_url as "Also on ", using <a class="u-syndication" rel="syndication">. Micropub q=source returns the same URLs as the syndication property. The Webmention worker rejects a webmention whose source is one of the note's own copies (e.g. backfed by Bridgy), so a copy is never shown as a response to itself (v2.8.0).

Client sessions and tokens

  • Bluesky: createSession with the app password, then refreshSession when the access JWT expires (a 401 or ExpiredToken triggers one re-login and retry). The PDS endpoint is taken from the DID document. Calls are serialized per client with a lock.
  • Threads: long-lived tokens expire after 60 days. maintain() tries a refresh (refresh_access_token, th_refresh_token) at most once a day. The current token is kept in DATA_PATH/threads_token.json (mode 0600, written atomically), together with a SHA-256 "seed" of the configured token. A different THREADS_ACCESS_TOKEN in the config therefore replaces the saved one. An exclusive flock on threads_token.lock makes one gunicorn process refresh while the others pick up its result. A failed refresh is retried an hour later (network error) or the next day (HTTP error, e.g. a token less than a day old). The httpx logger is raised to WARNING because Threads takes the token in the URL, and errors never echo the URL.
  • GoToSocial: a static bearer token.

Known limits

  • Bluesky and Threads copies don't follow edits (platform limitation). Unticking and re-ticking a target makes a new copy.
  • No per-target choice for Micropub updates: mp-syndicate-to is read only on create.
  • Likes of ordinary web pages, and all likes on Threads, stay on the site.