13 KiB
ADR-073: Backfeed of responses to crossposts
Status
Accepted (v2.13.0, 2026-11-10)
Implementation notes (v2.13.0):
- Until an avatar has a local copy, pages show the initial and keep the
original URL only as
<data class="u-photo">for microformats parsers, so browsers don't load it. - Bluesky post views carry no bio, so reply and quote authors' profiles
are fetched (
getProfiles, 25 at a time) to honour#nobridge. - A Threads token without
threads_manage_insightsstops Threads backfeed as a whole, as for any refused credential. - Backfill checks take a shared slot in
site_state(backfeed_backfill_last), so all gunicorn workers together do at most one every 5 seconds.
Context
StarPunk crossposts notes to GoToSocial, Bluesky and Threads (ADR-065, ADR-066, ADR-069). People like, repost, quote and reply to those copies, and none of that reaches the note. Backfeed brings those responses back to the original post.
The usual tool is Bridgy, which polls the silos and sends webmentions. It is a poor fit here:
- It supports Mastodon and Bluesky, not Threads.
- It only checks the 30 most recent posts, about every 30 minutes, and says nothing when a like is withdrawn or a reply deleted.
- Since v2.8.0 StarPunk rejects webmentions from a note's own crossposts
(
_is_own_syndicated_copy,webmention_worker.py). Bridgy's like and repost pages most likely carry the crosspost's URL plus a fragment asu-url, so they would be rejected. Replies have their own URL and get through (production webmention #1 was a Bridgy reply from Mastodon). - Every Bridgy source is on
brid.gy, which shares one source-domain rate limit of 10 webmentions a minute (ADR-063, Decision 10). - Every response passes through a third party.
StarPunk already holds a credential for each silo and knows each copy
(syndications.remote_id, remote_url). What the silos let it read:
| Silo | Likes | Reposts | Quotes | Replies | Credential change |
|---|---|---|---|---|---|
| Bluesky | getLikes |
getRepostedBy |
getQuotes |
getPostThread |
none (app password) |
| GoToSocial | favourited_by |
reblogged_by |
not supported | context |
read scopes |
| Threads | count only | count only | count only | replies |
threads_read_replies, threads_manage_insights |
The Threads API never says who liked, reposted or quoted a post; the Insights API gives lifetime counts only.
The production GoToSocial prunes remote posts older than 90 days, but never a thread that a local account started or took part in (GoToSocial's "Post Caching and Pruning" docs), so replies to StarPunk's copies are kept. On 2026-11-10 the production database still had 106 remote replies to the account older than 90 days, back to August 2024.
Decision
1. Native polling, no Bridgy
StarPunk polls each silo itself for responses to its own copies. Bridgy is switched off for the Mastodon and Bluesky accounts once this ships. The own-copy rule stays as it is.
2. Backfed responses are webmentions
Responses are stored in the webmentions table, so moderation, the
Responses section, and re-sending webmentions when a note's responses
change (spec 3.1.4, note_responses_changed) work as they do now. A
backfed row:
source_url: the response's own URL where it has one (a reply or quote post). Likes and reposts have no page, so they get the copy's URL plus#liked-by-<actor id>or#reposted-by-<actor id>. The(source_url, target_url)key makes repeated checks update the same row.target_url: the note's permalink.target_note_id: the note.origin: new column.webmentionfor received webmentions (the default), elsegotosocial,blueskyorthreads.remote_id: new column. The silo's id for the response (status id, AT-URI, Threads media id), or the actor's id for a like or repost.mention_type:like,repost,replyor the newquote.- Author name, profile URL and avatar from the silo's account data;
content_snippetfrom the reply or quote text, sanitised the same way as received webmentions (Mastodon HTML through nh3, Bluesky and Threads text escaped with links kept);source_publishedfrom the silo's timestamp. validation_status = 'valid': StarPunk read it from the silo's API itself, so there's nothing to fetch and verify. Moderation followsWEBMENTION_AUTO_APPROVElike any first-time webmention.
Only replies directly to the copy are taken, not deeper replies in the thread. Responses from the accounts StarPunk posts as are skipped.
3. Schedule
Each synced copy of a published note is checked on a schedule that slows as the note ages:
| Note age | Check every |
|---|---|
| under 1 day | 15 minutes |
| 1 to 7 days | 1 hour |
| 7 to 30 days | 1 day |
| 30 to 90 days | 1 week |
| over 90 days | not checked |
A random jitter of up to 10% spreads the checks. The cutoff is
BACKFEED_MAX_AGE_DAYS (default 90). Copies of likes and reposts
(ADR-068) aren't checked: they have no responses of their own. A note that
is unpublished or deleted is not checked, and its rows stay as they are.
4. Withdrawn likes and deleted replies
A check that read the whole list (every page, under a cap of 1,000 items
per list) marks that note's rows from that silo that are no longer there
as validation_status = 'invalid' with the error "No longer on
". They drop off the page but keep their moderation decision, and
come back if they reappear. A check that stopped early (an error, or the
cap) removes nothing.
5. Privacy
- Only public responses: GoToSocial
visibilitypublic(not unlisted, followers-only or direct); on Bluesky, nothing from accounts with the!no-unauthenticatedlabel. - People who opt out are skipped:
#nobridgeor#nobotin their bio (Mastodonnote, Blueskydescription), the same signal Bridgy honours. A row already stored for them is removed at the next check. - Admin Delete stops being a real
DELETEand setsmoderation_status = 'deleted'(the value exists but is unused), so a later check or re-sent webmention can't bring the response back. This applies to all webmentions.
6. Avatars are cached locally
Received webmentions hotlink author photos today. That leaks visitors'
addresses to other sites, and Threads and Bluesky avatar URLs change or
expire. Avatars are fetched once with safe_get (1 MB limit), resized to
96 px WebP with Pillow and saved as DATA_PATH/avatars/<sha256 of URL>.webp,
served from /avatars/. They're refetched when the URL changes, and files
no row uses are removed weekly. Received webmentions use the same cache
from their validation step. A failed fetch falls back to the coloured
initial.
7. Threads counts
Threads likes, reposts and quotes are shown as anonymous counts, not as
responses. The worker reads
/{media-id}/insights?metric=likes,reposts,quotes with each Threads check
and stores the numbers on the copy's syndications row. The note shows a
plain line such as "On Threads: 12 likes · 3 reposts" after the
Responses, with no u-like or h-cite markup, because these aren't
people. Threads replies counts aren't used: the replies themselves are
shown. views and shares are left out (Meta marks them "in
development").
8. Backfill
An admin action, Backfill responses, checks every synced copy once, whatever the note's age. Nothing runs by itself on deploy; the owner runs it after the first deploy (and may run it again later). It:
- marks every synced copy due now, flagged as a backfill;
- is worked through by the same worker, at most one copy every 5 seconds, ahead of the routine schedule only when nothing routine is due;
- records when it started and finished; the admin page shows this, and the button is disabled while one is running.
Upserts make it safe to repeat.
9. Worker
The checks run in the syndication worker's loop (ADR-065) after crossposts,
reusing its clients, token refresh (maintain()) and the lease pattern:
a copy is claimed with backfeed_claimed_until so only one gunicorn worker
checks it. A failed check backs off like a failed crosspost
(min(30 s × 2^attempts, 6 h)) without changing the copy's crosspost
state. If a silo answers with a permission error, backfeed for that silo
stops until restart and the admin page says which credential needs which
scope; crossposting is unaffected.
Per check: Bluesky 4 calls (likes, reposts, quotes, thread), GoToSocial 3, Threads 2 (replies, insights), plus extra pages.
10. Data (migration 020)
webmentions: rebuilt to addquoteto themention_typeCHECK, with new columnsorigin TEXT NOT NULL DEFAULT 'webmention'andremote_id TEXT, and an index on(target_note_id, origin).syndications: new columnsbackfeed_next_at,backfeed_checked_at,backfeed_claimed_until,backfeed_attempts,backfeed_error,backfeed_backfill(flag), andlikes_count,reposts_count,quotes_count(Threads only).- The backfill's start and finish times go in the existing
site_statekey/value table (migration 016):backfeed_backfill_started,backfeed_backfill_finished.
11. Display and admin
- Each backfed response shows a small "via Bluesky" (or GoToSocial,
Threads) label. Quotes get their own group, rendered like mentions with
the quote text, as
u-mention h-cite. - The note editor shows when responses were last checked, with a Check now button for each copy.
- The webmentions list can be filtered by origin.
- The dashboard shows backfeed errors next to crosspost errors.
12. Configuration and credentials
BACKFEED_ENABLED(defaultfalse; production setstrue) andBACKFEED_MAX_AGE_DAYS(default 90).- GoToSocial: a new token with read access added to the current scopes
(
read:statusesandread:accounts, or plainread; checked against GoToSocial's scope list when the token is issued). - Threads: the Meta app is re-authorised with
threads_read_repliesandthreads_manage_insightsadded. The newTHREADS_ACCESS_TOKENreplaces the saved token file automatically (ADR-069, Decision 2). - Bluesky: no change.
- Production tokens go in the Ansible vault (
bin/vault-run).
While at it, the Threads client moves from graph.threads.net to
graph.threads.com, the host Meta's current documentation uses.
Amendments (v2.13.2, 2026-11-10)
- Display. Likes, reposts and bookmarks are a line of counts under the
note instead of facepiles in the Responses section. Hovering or focusing
a count lists the people (as
u-like/u-repost/u-bookmarkh-cites) and the silo each came from. Threads' anonymous counts are added to the totals ("+N on Threads") instead of a separate line (Decision 7). - Web actions. Each like, repost and reply count is an
<indie-action do="…" with="<permalink>">whose fallback is a link to the same action on each crosspost. No web-action polyfill is loaded; handlers such as browser extensions can use the markup.
Amendments (v2.13.3, 2026-11-10)
- Opted-out people are counted. Their likes, reposts and quotes are
still not stored or named (Decision 5), but each check records how many
there were on the copy's
syndicationsrow (likes_count,reposts_count,quotes_count, the columns Threads' counts use), and the counts line adds them to the totals as "N on ". Totals then match the silo.
Consequences
- Responses on all three silos show on the note within 15 minutes for new notes, with no third party involved.
- Withdrawn likes and deleted replies disappear by themselves for 90 days; after that a note's responses are frozen.
- Threads shows replies in full but likes, reposts and quotes only as numbers.
- Backfeed stores other people's public posts and avatars. The privacy rules (Decision 5) and moderation are the controls; there is no consent step.
- More API calls. At about 50 new notes a month the routine schedule is roughly 5,000 calls a month across the three silos, well inside their limits. The backfill makes one burst of a few calls per copy.
- Deleting a webmention in the admin now leaves a tombstone row.
- A crosspost removed and posted again gets a new copy; responses to the old copy are marked gone at the next check.
Alternatives considered
- Bridgy. No Threads, 30-post window, no withdrawals, a third party in the path, and it needs the own-copy rule and rate limit loosened (see Context). Rejected.
- A separate responses table. Would duplicate the display, moderation
and 3.1.4 re-sending code that
webmentionsalready has. Rejected. - Notification feeds (Mastodon
/api/v1/notifications, BlueskylistNotifications): one call covers every post, but they only report new events, so withdrawals are never seen and a backfill can't use them. Could be added later as a fast path in front of per-post checks. - Streaming (Bluesky Jetstream, Mastodon streaming API): real time, but needs a long-lived connection per gunicorn worker. Not worth it for a personal site.
- Keep the Threads counts out. They aren't IndieWeb responses, but they are the only signal Threads gives, and showing them anonymously costs one call and identifies no one. Kept.