Likes, reposts, quotes and replies to a note's GoToSocial, Bluesky and Threads copies are read back from the silos and stored as webmentions (ADR-073): age-based schedule to 90 days, pruning of withdrawn responses, #nobridge/#nobot opt-outs, admin backfill and Check now, Threads counts, a Quotes group, local avatar copies, and delete tombstones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 KiB
Syndication (POSSE) Architecture
Current as of v2.10.1. Code: starpunk/syndication.py.
StarPunk publishes on its own site first and then copies each note to other networks (POSSE: Publish on your Own Site, Syndicate Elsewhere). Later edits, unpublishing and deletes follow the copies.
Feeds (RSS, Atom, JSON Feed, OPML) and WebSub are pull and push distribution of the site itself, not crossposting. They are described in indieweb-protocols.md. This document covered feeds before v2.1.0.
Decisions: ADR-065
(GoToSocial, queue model), ADR-066
(Bluesky), ADR-068 (likes and
reposts), ADR-069 (Threads),
ADR-070 (links and link
cards), and the v2.9.0 Micropub review
(bookmarks, mp-syndicate-to default).
Targets
A target exists when its credentials are configured. get_targets() builds
the list from config on each call and caches one client object per target
in the process.
| Target id | Client | Enabled by | Micropub uid |
Text limit | Images | Edits follow? |
|---|---|---|---|---|---|---|
gotosocial |
MastodonClient (any Mastodon-API server) |
GOTOSOCIAL_URL + GOTOSOCIAL_ACCESS_TOKEN |
GOTOSOCIAL_URL/ |
GOTOSOCIAL_MAX_CHARS (5000) |
GOTOSOCIAL_MAX_MEDIA (4) |
Yes |
bluesky |
BlueskyClient (AT Protocol, app password) |
BLUESKY_HANDLE + BLUESKY_APP_PASSWORD (BLUESKY_SERVICE, default https://bsky.social) |
https://bsky.app/profile/<handle> |
BLUESKY_MAX_CHARS (300) |
4 | No |
threads |
ThreadsClient (Meta Graph API) |
THREADS_ACCESS_TOKEN (THREADS_USERNAME for links) |
https://www.threads.com/@<username> |
THREADS_MAX_CHARS (500) |
THREADS_MAX_MEDIA (4) |
No |
Display names come from GOTOSOCIAL_NAME, BLUESKY_NAME and
THREADS_NAME. GoToSocial statuses use GOTOSOCIAL_VISIBILITY (default
public). The GoToSocial token needs write:statuses and write:media,
plus read:search and write:favourites for likes and boosts.
Bluesky and Threads can't edit posts through their APIs. Once a copy exists there, edits to the note don't change it, but unpublish and delete still remove it.
Choosing targets
select_targets(note_id, target_ids) records which targets a note goes to.
Every configured target not in the list is disabled; if it already has a
copy, that copy is deleted. Targets the note never went to get no row.
| Origin | Targets |
|---|---|
| Admin editor, new note | One checkbox per target, all checked by default |
| Admin editor, edit | Checkboxes reflect the targets currently enabled; saving applies the selection |
| Micropub create | Only the targets in mp-syndicate-to (form field, mp-syndicate-to[], or inside JSON properties). Values may be a target's uid (trailing slash ignored) or its id. No mp-syndicate-to means no crossposting (v2.9.0; before, all targets) |
| Micropub update / delete | Target selection is unchanged; the change follows existing copies |
Micropub clients learn the targets from q=syndicate-to and q=config
([{"uid", "name"}]). Selecting targets for a draft records them; nothing
is sent until the note is published.
Data: the syndications table (migration 012)
One row per (note, target), unique on (note_id, target):
| Column | Meaning |
|---|---|
enabled |
Whether the note should have a copy on this target |
remote_id, remote_url |
The copy: status id, at:// URI, Threads media id, or a favourite:<id> / reblog:<id> marker for native likes and boosts |
state |
pending, synced or failed |
version |
Incremented on every change that needs a reconcile |
attempts, last_error, next_attempt_at |
Retry bookkeeping |
claimed_until |
Lease held by the worker processing the row |
note_id deliberately has no foreign key, so the row outlives a hard
delete and the worker can still remove the copy.
Change hooks
| Event | Call | Effect |
|---|---|---|
| Admin create/edit | select_targets() |
Rows set to pending, version bumped |
| Micropub create | select_targets(resolve_targets(mp-syndicate-to)) |
After photos are attached |
update_note() (admin or Micropub, including publish/unpublish) |
note_changed(id) |
Every row of the note set to pending, version bumped, attempts reset |
| Micropub photo changes | note_changed(id) |
Same |
delete_note() (soft or hard), undelete_note() |
note_changed(id) |
Same |
| Retry button on the edit page | retry(id) |
That row set to pending now, attempts reset |
Worker
SyndicationWorker is a daemon thread started in every gunicorn process
when at least one target is configured (not under TESTING). It runs
process_due() every SYNDICATION_POLL_INTERVAL seconds (default 10), and
loops again immediately while there is work.
process_due():
- Calls each client's
maintain()(the Threads token refresh). - Selects up to 5 rows that are
pending, due, and not claimed. - Claims each row with an atomic
UPDATE ... SET claimed_until = now + 5 min WHERE state = 'pending' AND (claimed_until IS NULL OR claimed_until < now). The other processes skip it. Unlike the Webmention and WebSub workers, there is no file lock: the row claim is the only coordination. - Reconciles the row (below), then records the result.
Reconcile
note published and row enabled?
no → remote copy exists? delete it → (None, None)
yes → copy exists and target can't edit? → keep it as is
copy exists → update (if it's gone remotely: create again)
no copy → create
Deleted notes are not found by get_note(), so they count as "not
published".
Versions
The row's version is read when the row is picked up. _finish() sets
state = 'synced' only if the version is unchanged. If the note was edited
while the send was in flight, the row stays pending and is reconciled
again with the new content. _fail() does the same.
Retries and idempotency
SyndicationError(permanent=True)fails the row immediately. This covers HTTP 4xx other than 429, a missing target, images too large for Bluesky, a Threads container inERROR/EXPIRED, and Graph API auth/permission errors (codes 10, 190, 200).- Other errors retry after
min(30 s × 2^attempts, 6 h). After 8 attempts the row becomesfailed. The error is shown on the edit page with a Retry button. RemoteGone(404, or Graph error 100/33) during an update means the copy was deleted remotely, so a new one is created. During a delete it counts as success.- Duplicate posts after a lost response are prevented as follows:
- GoToSocial: an
Idempotency-Key: starpunk-<row id>-<version>header. - Bluesky and Threads: on a retry (
attempts > 0), the last 25 posts are searched for one containing the permalink (for Threads reposts, the reposted URL). A match is adopted instead of posting again.
- GoToSocial: an
- Threads' publish step sometimes fails transiently ("Fatal"); it is retried unless the error is an auth error.
Composing posts
Post carries the note, permalink, tags, the target's character limit,
media files, and for likes, reposts and bookmarks, the URL they point at and
its fetched title (from the reply-context cache).
Text
compose_text() builds: body, a blank line, the permalink, a blank line,
the hashtags. If that is too long, the body is cut at a word boundary with
"…"; the permalink always survives. Hashtags (# plus the tag's display
name without non-word characters) may take at most a third of the limit. A
bookmark's body starts with 🔖 <bookmarked URL> (v2.9.0). A photo-only
note starts with the permalink.
| Target | Body format | Links | Hashtags |
|---|---|---|---|
| GoToSocial | The note's Markdown, sent with content_type: text/markdown; embedded images removed from the text (sent as attachments) |
As written | In the text |
| Bluesky | Plain text from the rendered Markdown | Markdown link text becomes a clickable link facet; bare URLs and #tags get link/tag facets (UTF-8 byte offsets) |
In the text, as tag facets |
| Threads | Plain text | Markdown links are written text (url), up to 4 different URLs plus the permalink (Threads allows 5); the length counts non-ASCII characters as UTF-8 bytes |
None in the text; the first usable tag becomes the post's topic_tag |
Link cards (ADR-070)
- Bluesky: if the note has images, they are the embed. Otherwise, if
the note links somewhere, StarPunk builds an
app.bsky.embed.externalcard for the first link.page_card()fetches the page (at most 512 KB, every redirect hop checked against private addresses, 10 s per request, 5 redirects) and reads the Open Graph, Twitter or HTML title, description and image. The image (at most 5 MB) is shrunk under 1 MB and uploaded as the thumbnail. If the page can't be fetched, the card has just the URL. A note without links gets a card for its own permalink, built from the note without fetching anything. - Threads: no explicit
link_attachment(Meta rejects one for a page it hasn't crawled yet). Threads previews the first link in the text by itself. - GoToSocial: the server makes its own previews.
Media
_media_files() collects the note's attached photos first, then images
embedded in the note text that are in this site's media library (some
Micropub clients put uploaded photos in the content). It stops at the
target's image limit. The large variant is used when one exists. Alt text
comes from the caption, or from the embedded image's alt.
| Target | Handling |
|---|---|
| GoToSocial | Uploaded via POST /api/v2/media with description; re-uploaded on each edit |
| Bluesky | uploadBlob; images over 1,000,000 bytes are re-encoded as JPEG at decreasing size and quality, and fail permanently if they still don't fit. Aspect ratio is sent |
| Threads | Meta fetches each image from its public /media/ URL. Only JPEG and PNG up to 8 MB are used; others are left off. 1 image → IMAGE container, 2 or more → CAROUSEL of child containers. Alt text up to 1000 characters |
Only notes of kind note carry media; likes, reposts and bookmarks don't.
Likes, reposts and bookmarks
| Kind | GoToSocial | Bluesky | Threads |
|---|---|---|---|
| Like of a post on that network | Native favourite (favourite:<id>) |
Native app.bsky.feed.like |
Not crossposted (the API can't like) |
| Like of anything else | Not crossposted | Not crossposted | Not crossposted |
| Repost without comment, of a post on that network | Native boost (reblog:<id>) |
Native app.bsky.feed.repost |
Not native: posts 🔁 title + URL |
| Repost with comment, of a post on that network | Status: comment, 🔁 title, URL |
Quote post (app.bsky.embed.record) |
Comment, 🔁 title, URL |
| Repost of another page | Status: (comment,) 🔁 title, URL |
Post with an external card for the URL | (Comment,) 🔁 title, URL |
| Bookmark | Ordinary status starting 🔖 <url> |
Ordinary post starting 🔖 <url> |
Ordinary post starting 🔖 <url> |
"On that network" is decided by the client. GoToSocial resolves the URL
through /api/v2/search?resolve=true. Bluesky matches
https://bsky.app/profile/<actor>/post/<rkey> and resolves it to a URI and
CID. Native likes and boosts have no URL of their own and nothing to edit.
Deleting the note undoes them (unfavourite / unreblog, or deleting the
like or repost record). A like that isn't crossposted is recorded as synced
with no remote copy.
u-syndication
templates/note.html lists every row that has a remote_url as "Also on
", using <a class="u-syndication" rel="syndication">. Micropub
q=source returns the same URLs as the syndication property. The
Webmention worker rejects a webmention whose source is one of the note's
own copies (e.g. backfed by Bridgy), so a copy is never shown as a response
to itself (v2.8.0).
Backfeed (v2.13.0, ADR-073)
starpunk/backfeed.py brings responses to the copies back to the note.
After each crosspost pass, the worker loop calls backfeed.process_due(),
which checks up to three synced copies of published notes (not likes or
reposts) that are due, each under a lease (backfeed_claimed_until):
- Reading. Each client has
responses(remote_id, remote_url), which returns aHarvest: GoToSocialfavourited_by,reblogged_byand public direct replies fromcontext; BlueskygetLikes,getRepostedBy,getQuotesandgetPostThread(depth 1), plusgetProfilesfor reply and quote authors' bios; Threadsrepliesandinsights(like, repost and quote counts). Lists are paged up to 1,000 items; a list cut short makes the harvest incomplete. - Storing. Responses are upserted into
webmentionswithorigin(the target id) andremote_id,validation_status = 'valid', keyed by(source_url, target_url). Likes and reposts use the copy's URL plus#liked-by-<actor>/#reposted-by-<actor>. A new row gets theWEBMENTION_AUTO_APPROVEdefault; an existing row keeps its moderation. A complete harvest marks that silo's rows for the note that weren't foundinvalid("No longer on "). Authors with#nobridgeor#nobotin their bio, or Bluesky's!no-unauthenticatedlabel, are skipped and their rows deleted. If the note's shown responses changed, its webmentions are re-sent (spec 3.1.4). Threads counts go insyndications.likes_count,reposts_count,quotes_count. - Schedule.
backfeed_next_atfollows the note's age: 15 minutes (day 1), 1 hour (week 1), 1 day (to 30 days), 1 week (toBACKFEED_MAX_AGE_DAYS), with up to 10% jitter; none after. Failures back off like crossposts; after 8 they wait a month. - Backfill.
start_backfill()flags every eligible copy (backfeed_backfill = 1). Flagged copies are checked only when no routine check is due, at most one every 5 seconds across workers (site_state.backfeed_backfill_last). Start and finish times are insite_state. - Refused credentials. A 401/403 (or Graph auth error) puts the target
in
backfeed.denieduntil restart; the admin Webmentions page says which scopes are needed. Crossposting is unaffected.
Avatars of all responses (received and backfed) are saved by
starpunk/avatars.py and served from /avatars/<sha256>.webp; the
webmention worker fills in missing ones every 10 minutes and removes unused
ones weekly.
Client sessions and tokens
- Bluesky:
createSessionwith the app password, thenrefreshSessionwhen the access JWT expires (a 401 orExpiredTokentriggers one re-login and retry). The PDS endpoint is taken from the DID document. Calls are serialized per client with a lock. - Threads: long-lived tokens expire after 60 days.
maintain()tries a refresh (refresh_access_token,th_refresh_token) at most once a day. The current token is kept inDATA_PATH/threads_token.json(mode 0600, written atomically), together with a SHA-256 "seed" of the configured token. A differentTHREADS_ACCESS_TOKENin the config therefore replaces the saved one. An exclusiveflockonthreads_token.lockmakes one gunicorn process refresh while the others pick up its result. A failed refresh is retried an hour later (network error) or the next day (HTTP error, e.g. a token less than a day old). The httpx logger is raised to WARNING because Threads takes the token in the URL, and errors never echo the URL. - GoToSocial: a static bearer token.
Known limits
- Bluesky and Threads copies don't follow edits (platform limitation). Unticking and re-ticking a target makes a new copy.
- No per-target choice for Micropub updates:
mp-syndicate-tois read only on create. - Likes of ordinary web pages, and all likes on Threads, stay on the site.