Files
StarPunk/docs/decisions/ADR-073-backfeed.md
T

13 KiB
Raw Blame History

ADR-073: Backfeed of responses to crossposts

Status

Accepted (v2.13.0, 2026-11-10)

Implementation notes (v2.13.0):

  • Until an avatar has a local copy, pages show the initial and keep the original URL only as <data class="u-photo"> for microformats parsers, so browsers don't load it.
  • Bluesky post views carry no bio, so reply and quote authors' profiles are fetched (getProfiles, 25 at a time) to honour #nobridge.
  • A Threads token without threads_manage_insights stops Threads backfeed as a whole, as for any refused credential.
  • Backfill checks take a shared slot in site_state (backfeed_backfill_last), so all gunicorn workers together do at most one every 5 seconds.

Context

StarPunk crossposts notes to GoToSocial, Bluesky and Threads (ADR-065, ADR-066, ADR-069). People like, repost, quote and reply to those copies, and none of that reaches the note. Backfeed brings those responses back to the original post.

The usual tool is Bridgy, which polls the silos and sends webmentions. It is a poor fit here:

  • It supports Mastodon and Bluesky, not Threads.
  • It only checks the 30 most recent posts, about every 30 minutes, and says nothing when a like is withdrawn or a reply deleted.
  • Since v2.8.0 StarPunk rejects webmentions from a note's own crossposts (_is_own_syndicated_copy, webmention_worker.py). Bridgy's like and repost pages most likely carry the crosspost's URL plus a fragment as u-url, so they would be rejected. Replies have their own URL and get through (production webmention #1 was a Bridgy reply from Mastodon).
  • Every Bridgy source is on brid.gy, which shares one source-domain rate limit of 10 webmentions a minute (ADR-063, Decision 10).
  • Every response passes through a third party.

StarPunk already holds a credential for each silo and knows each copy (syndications.remote_id, remote_url). What the silos let it read:

Silo Likes Reposts Quotes Replies Credential change
Bluesky getLikes getRepostedBy getQuotes getPostThread none (app password)
GoToSocial favourited_by reblogged_by not supported context read scopes
Threads count only count only count only replies threads_read_replies, threads_manage_insights

The Threads API never says who liked, reposted or quoted a post; the Insights API gives lifetime counts only.

The production GoToSocial prunes remote posts older than 90 days, but never a thread that a local account started or took part in (GoToSocial's "Post Caching and Pruning" docs), so replies to StarPunk's copies are kept. On 2026-11-10 the production database still had 106 remote replies to the account older than 90 days, back to August 2024.

Decision

1. Native polling, no Bridgy

StarPunk polls each silo itself for responses to its own copies. Bridgy is switched off for the Mastodon and Bluesky accounts once this ships. The own-copy rule stays as it is.

2. Backfed responses are webmentions

Responses are stored in the webmentions table, so moderation, the Responses section, and re-sending webmentions when a note's responses change (spec 3.1.4, note_responses_changed) work as they do now. A backfed row:

  • source_url: the response's own URL where it has one (a reply or quote post). Likes and reposts have no page, so they get the copy's URL plus #liked-by-<actor id> or #reposted-by-<actor id>. The (source_url, target_url) key makes repeated checks update the same row.
  • target_url: the note's permalink. target_note_id: the note.
  • origin: new column. webmention for received webmentions (the default), else gotosocial, bluesky or threads.
  • remote_id: new column. The silo's id for the response (status id, AT-URI, Threads media id), or the actor's id for a like or repost.
  • mention_type: like, repost, reply or the new quote.
  • Author name, profile URL and avatar from the silo's account data; content_snippet from the reply or quote text, sanitised the same way as received webmentions (Mastodon HTML through nh3, Bluesky and Threads text escaped with links kept); source_published from the silo's timestamp.
  • validation_status = 'valid': StarPunk read it from the silo's API itself, so there's nothing to fetch and verify. Moderation follows WEBMENTION_AUTO_APPROVE like any first-time webmention.

Only replies directly to the copy are taken, not deeper replies in the thread. Responses from the accounts StarPunk posts as are skipped.

3. Schedule

Each synced copy of a published note is checked on a schedule that slows as the note ages:

Note age Check every
under 1 day 15 minutes
1 to 7 days 1 hour
7 to 30 days 1 day
30 to 90 days 1 week
over 90 days not checked

A random jitter of up to 10% spreads the checks. The cutoff is BACKFEED_MAX_AGE_DAYS (default 90). Copies of likes and reposts (ADR-068) aren't checked: they have no responses of their own. A note that is unpublished or deleted is not checked, and its rows stay as they are.

4. Withdrawn likes and deleted replies

A check that read the whole list (every page, under a cap of 1,000 items per list) marks that note's rows from that silo that are no longer there as validation_status = 'invalid' with the error "No longer on ". They drop off the page but keep their moderation decision, and come back if they reappear. A check that stopped early (an error, or the cap) removes nothing.

5. Privacy

  • Only public responses: GoToSocial visibility public (not unlisted, followers-only or direct); on Bluesky, nothing from accounts with the !no-unauthenticated label.
  • People who opt out are skipped: #nobridge or #nobot in their bio (Mastodon note, Bluesky description), the same signal Bridgy honours. A row already stored for them is removed at the next check.
  • Admin Delete stops being a real DELETE and sets moderation_status = 'deleted' (the value exists but is unused), so a later check or re-sent webmention can't bring the response back. This applies to all webmentions.

6. Avatars are cached locally

Received webmentions hotlink author photos today. That leaks visitors' addresses to other sites, and Threads and Bluesky avatar URLs change or expire. Avatars are fetched once with safe_get (1 MB limit), resized to 96 px WebP with Pillow and saved as DATA_PATH/avatars/<sha256 of URL>.webp, served from /avatars/. They're refetched when the URL changes, and files no row uses are removed weekly. Received webmentions use the same cache from their validation step. A failed fetch falls back to the coloured initial.

7. Threads counts

Threads likes, reposts and quotes are shown as anonymous counts, not as responses. The worker reads /{media-id}/insights?metric=likes,reposts,quotes with each Threads check and stores the numbers on the copy's syndications row. The note shows a plain line such as "On Threads: 12 likes · 3 reposts" after the Responses, with no u-like or h-cite markup, because these aren't people. Threads replies counts aren't used: the replies themselves are shown. views and shares are left out (Meta marks them "in development").

8. Backfill

An admin action, Backfill responses, checks every synced copy once, whatever the note's age. Nothing runs by itself on deploy; the owner runs it after the first deploy (and may run it again later). It:

  • marks every synced copy due now, flagged as a backfill;
  • is worked through by the same worker, at most one copy every 5 seconds, ahead of the routine schedule only when nothing routine is due;
  • records when it started and finished; the admin page shows this, and the button is disabled while one is running.

Upserts make it safe to repeat.

9. Worker

The checks run in the syndication worker's loop (ADR-065) after crossposts, reusing its clients, token refresh (maintain()) and the lease pattern: a copy is claimed with backfeed_claimed_until so only one gunicorn worker checks it. A failed check backs off like a failed crosspost (min(30 s × 2^attempts, 6 h)) without changing the copy's crosspost state. If a silo answers with a permission error, backfeed for that silo stops until restart and the admin page says which credential needs which scope; crossposting is unaffected.

Per check: Bluesky 4 calls (likes, reposts, quotes, thread), GoToSocial 3, Threads 2 (replies, insights), plus extra pages.

10. Data (migration 020)

  • webmentions: rebuilt to add quote to the mention_type CHECK, with new columns origin TEXT NOT NULL DEFAULT 'webmention' and remote_id TEXT, and an index on (target_note_id, origin).
  • syndications: new columns backfeed_next_at, backfeed_checked_at, backfeed_claimed_until, backfeed_attempts, backfeed_error, backfeed_backfill (flag), and likes_count, reposts_count, quotes_count (Threads only).
  • The backfill's start and finish times go in the existing site_state key/value table (migration 016): backfeed_backfill_started, backfeed_backfill_finished.

11. Display and admin

  • Each backfed response shows a small "via Bluesky" (or GoToSocial, Threads) label. Quotes get their own group, rendered like mentions with the quote text, as u-mention h-cite.
  • The note editor shows when responses were last checked, with a Check now button for each copy.
  • The webmentions list can be filtered by origin.
  • The dashboard shows backfeed errors next to crosspost errors.

12. Configuration and credentials

  • BACKFEED_ENABLED (default false; production sets true) and BACKFEED_MAX_AGE_DAYS (default 90).
  • GoToSocial: a new token with read access added to the current scopes (read:statuses and read:accounts, or plain read; checked against GoToSocial's scope list when the token is issued).
  • Threads: the Meta app is re-authorised with threads_read_replies and threads_manage_insights added. The new THREADS_ACCESS_TOKEN replaces the saved token file automatically (ADR-069, Decision 2).
  • Bluesky: no change.
  • Production tokens go in the Ansible vault (bin/vault-run).

While at it, the Threads client moves from graph.threads.net to graph.threads.com, the host Meta's current documentation uses.

Amendments (v2.13.2, 2026-11-10)

  • Display. Likes, reposts and bookmarks are a line of counts under the note instead of facepiles in the Responses section. Hovering or focusing a count lists the people (as u-like / u-repost / u-bookmark h-cites) and the silo each came from. Threads' anonymous counts are added to the totals ("+N on Threads") instead of a separate line (Decision 7).
  • Web actions. Each like, repost and reply count is an <indie-action do="…" with="<permalink>"> whose fallback is a link to the same action on each crosspost. No web-action polyfill is loaded; handlers such as browser extensions can use the markup.

Amendments (v2.13.3, 2026-11-10)

  • Opted-out people are counted. Their likes, reposts and quotes are still not stored or named (Decision 5), but each check records how many there were on the copy's syndications row (likes_count, reposts_count, quotes_count, the columns Threads' counts use), and the counts line adds them to the totals as "N on ". Totals then match the silo.

Consequences

  • Responses on all three silos show on the note within 15 minutes for new notes, with no third party involved.
  • Withdrawn likes and deleted replies disappear by themselves for 90 days; after that a note's responses are frozen.
  • Threads shows replies in full but likes, reposts and quotes only as numbers.
  • Backfeed stores other people's public posts and avatars. The privacy rules (Decision 5) and moderation are the controls; there is no consent step.
  • More API calls. At about 50 new notes a month the routine schedule is roughly 5,000 calls a month across the three silos, well inside their limits. The backfill makes one burst of a few calls per copy.
  • Deleting a webmention in the admin now leaves a tombstone row.
  • A crosspost removed and posted again gets a new copy; responses to the old copy are marked gone at the next check.

Alternatives considered

  • Bridgy. No Threads, 30-post window, no withdrawals, a third party in the path, and it needs the own-copy rule and rate limit loosened (see Context). Rejected.
  • A separate responses table. Would duplicate the display, moderation and 3.1.4 re-sending code that webmentions already has. Rejected.
  • Notification feeds (Mastodon /api/v1/notifications, Bluesky listNotifications): one call covers every post, but they only report new events, so withdrawals are never seen and a backfill can't use them. Could be added later as a fast path in front of per-post checks.
  • Streaming (Bluesky Jetstream, Mastodon streaming API): real time, but needs a long-lived connection per gunicorn worker. Not worth it for a personal site.
  • Keep the Threads counts out. They aren't IndieWeb responses, but they are the only signal Threads gives, and showing them anonymously costs one call and identifies no one. Kept.