docs: ADR-073 backfeed of responses to crossposts (proposed)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
PhilandClaude Opus 5.5 committed 2026-11-10 11:59:37 -07:00
1 parent afef1fc9a5
commit f8a429cdc1
3 files changed
+256 -6

No files matched your search

+249
View File
@@ -0,0 +1,249 @@
# ADR-073: Backfeed of responses to crossposts
## Status
Proposed (2026-11-10, for v2.13.0)
## Context
StarPunk crossposts notes to GoToSocial, Bluesky and Threads (ADR-065,
ADR-066, ADR-069). People like, repost, quote and reply to those copies,
and none of that reaches the note. [Backfeed](https://indieweb.org/backfeed)
brings those responses back to the original post.
The usual tool is [Bridgy](https://brid.gy/about), which polls the silos and
sends webmentions. It is a poor fit here:
- It supports Mastodon and Bluesky, not Threads.
- It only checks the 30 most recent posts, about every 30 minutes, and
says nothing when a like is withdrawn or a reply deleted.
- Since v2.8.0 StarPunk rejects webmentions from a note's own crossposts
(`_is_own_syndicated_copy`, `webmention_worker.py`). Bridgy's like and
repost pages most likely carry the crosspost's URL plus a fragment as
`u-url`, so they would be rejected. Replies have their own URL and get
through (production webmention #1 was a Bridgy reply from Mastodon).
- Every Bridgy source is on `brid.gy`, which shares one source-domain rate
limit of 10 webmentions a minute (ADR-063, Decision 10).
- Every response passes through a third party.
StarPunk already holds a credential for each silo and knows each copy
(`syndications.remote_id`, `remote_url`). What the silos let it read:
| Silo | Likes | Reposts | Quotes | Replies | Credential change |
|---|---|---|---|---|---|
| Bluesky | `getLikes` | `getRepostedBy` | `getQuotes` | `getPostThread` | none (app password) |
| GoToSocial | `favourited_by` | `reblogged_by` | not supported | `context` | read scopes |
| Threads | count only | count only | count only | `replies` | `threads_read_replies`, `threads_manage_insights` |
The Threads API never says who liked, reposted or quoted a post; the
Insights API gives lifetime counts only.
The production GoToSocial deletes cached remote posts older than 90 days,
so replies from other servers to older notes may no longer be there.
## Decision
### 1. Native polling, no Bridgy
StarPunk polls each silo itself for responses to its own copies. Bridgy is
switched off for the Mastodon and Bluesky accounts once this ships. The
own-copy rule stays as it is.
### 2. Backfed responses are webmentions
Responses are stored in the `webmentions` table, so moderation, the
Responses section, and re-sending webmentions when a note's responses
change (spec 3.1.4, `note_responses_changed`) work as they do now. A
backfed row:
- `source_url`: the response's own URL where it has one (a reply or quote
post). Likes and reposts have no page, so they get the copy's URL plus
`#liked-by-<actor id>` or `#reposted-by-<actor id>`. The
`(source_url, target_url)` key makes repeated checks update the same row.
- `target_url`: the note's permalink. `target_note_id`: the note.
- `origin`: new column. `webmention` for received webmentions (the
default), else `gotosocial`, `bluesky` or `threads`.
- `remote_id`: new column. The silo's id for the response (status id,
AT-URI, Threads media id), or the actor's id for a like or repost.
- `mention_type`: `like`, `repost`, `reply` or the new `quote`.
- Author name, profile URL and avatar from the silo's account data;
`content_snippet` from the reply or quote text, sanitised the same way as
received webmentions (Mastodon HTML through nh3, Bluesky and Threads text
escaped with links kept); `source_published` from the silo's timestamp.
- `validation_status = 'valid'`: StarPunk read it from the silo's API
itself, so there's nothing to fetch and verify. Moderation follows
`WEBMENTION_AUTO_APPROVE` like any first-time webmention.
Only replies directly to the copy are taken, not deeper replies in the
thread. Responses from the accounts StarPunk posts as are skipped.
### 3. Schedule
Each synced copy of a published note is checked on a schedule that slows
as the note ages:
| Note age | Check every |
|---|---|
| under 1 day | 15 minutes |
| 1 to 7 days | 1 hour |
| 7 to 30 days | 1 day |
| 30 to 90 days | 1 week |
| over 90 days | not checked |
A random jitter of up to 10% spreads the checks. The cutoff is
`BACKFEED_MAX_AGE_DAYS` (default 90). Copies of likes and reposts
(ADR-068) aren't checked: they have no responses of their own. A note that
is unpublished or deleted is not checked, and its rows stay as they are.
### 4. Withdrawn likes and deleted replies
A check that read the whole list (every page, under a cap of 1,000 items
per list) marks that note's rows from that silo that are no longer there
as `validation_status = 'invalid'` with the error "No longer on
<silo>". They drop off the page but keep their moderation decision, and
come back if they reappear. A check that stopped early (an error, or the
cap) removes nothing. Checks of notes older than the cutoff (the backfill,
Decision 8) only add and update, never remove, because GoToSocial may have
pruned old replies.
### 5. Privacy
- Only public responses: GoToSocial `visibility` `public` (not unlisted,
followers-only or direct); on Bluesky, nothing from accounts with the
`!no-unauthenticated` label.
- People who opt out are skipped: `#nobridge` or `#nobot` in their bio
(Mastodon `note`, Bluesky `description`), the same signal Bridgy honours.
A row already stored for them is removed at the next check.
- Admin **Delete** stops being a real `DELETE` and sets
`moderation_status = 'deleted'` (the value exists but is unused), so a
later check or re-sent webmention can't bring the response back. This
applies to all webmentions.
### 6. Avatars are cached locally
Received webmentions hotlink author photos today. That leaks visitors'
addresses to other sites, and Threads and Bluesky avatar URLs change or
expire. Avatars are fetched once with `safe_get` (1 MB limit), resized to
96 px WebP with Pillow and saved as `DATA_PATH/avatars/<sha256 of URL>.webp`,
served from `/avatars/`. They're refetched when the URL changes, and files
no row uses are removed weekly. Received webmentions use the same cache
from their validation step. A failed fetch falls back to the coloured
initial.
### 7. Threads counts
Threads likes, reposts and quotes are shown as anonymous counts, not as
responses. The worker reads
`/{media-id}/insights?metric=likes,reposts,quotes` with each Threads check
and stores the numbers on the copy's `syndications` row. The note shows a
plain line such as "On Threads: 12 likes · 3 reposts" after the
Responses, with no `u-like` or `h-cite` markup, because these aren't
people. Threads `replies` counts aren't used: the replies themselves are
shown. `views` and `shares` are left out (Meta marks them "in
development").
### 8. Backfill
An admin action, **Backfill responses**, checks every synced copy once,
whatever the note's age. Nothing runs by itself on deploy; the owner runs
it after the first deploy (and may run it again later). It:
- marks every synced copy due now, flagged as a backfill;
- is worked through by the same worker, at most one copy every 5 seconds,
ahead of the routine schedule only when nothing routine is due;
- only adds and updates (Decision 4);
- records when it started and finished; the admin page shows this, and
the button is disabled while one is running.
Upserts make it safe to repeat.
### 9. Worker
The checks run in the syndication worker's loop (ADR-065) after crossposts,
reusing its clients, token refresh (`maintain()`) and the lease pattern:
a copy is claimed with `backfeed_claimed_until` so only one gunicorn worker
checks it. A failed check backs off like a failed crosspost
(`min(30 s × 2^attempts, 6 h)`) without changing the copy's crosspost
state. If a silo answers with a permission error, backfeed for that silo
stops until restart and the admin page says which credential needs which
scope; crossposting is unaffected.
Per check: Bluesky 4 calls (likes, reposts, quotes, thread), GoToSocial 3,
Threads 2 (replies, insights), plus extra pages.
### 10. Data (migration 020)
- `webmentions`: rebuilt to add `quote` to the `mention_type` CHECK, with
new columns `origin TEXT NOT NULL DEFAULT 'webmention'` and
`remote_id TEXT`, and an index on `(target_note_id, origin)`.
- `syndications`: new columns `backfeed_next_at`, `backfeed_checked_at`,
`backfeed_claimed_until`, `backfeed_attempts`, `backfeed_error`,
`backfeed_backfill` (flag), and `likes_count`, `reposts_count`,
`quotes_count` (Threads only).
- The backfill's start and finish times go in the existing `site_state`
key/value table (migration 016): `backfeed_backfill_started`,
`backfeed_backfill_finished`.
### 11. Display and admin
- Each backfed response shows a small "via Bluesky" (or GoToSocial,
Threads) label. Quotes get their own group, rendered like mentions with
the quote text, as `u-mention h-cite`.
- The note editor shows when responses were last checked, with a
**Check now** button for each copy.
- The webmentions list can be filtered by origin.
- The dashboard shows backfeed errors next to crosspost errors.
### 12. Configuration and credentials
- `BACKFEED_ENABLED` (default `false`; production sets `true`) and
`BACKFEED_MAX_AGE_DAYS` (default 90).
- GoToSocial: a new token with read access added to the current scopes
(`read:statuses` and `read:accounts`, or plain `read`; checked against
GoToSocial's scope list when the token is issued).
- Threads: the Meta app is re-authorised with `threads_read_replies` and
`threads_manage_insights` added. The new `THREADS_ACCESS_TOKEN` replaces
the saved token file automatically (ADR-069, Decision 2).
- Bluesky: no change.
- Production tokens go in the Ansible vault (`bin/vault-run`).
While at it, the Threads client moves from `graph.threads.net` to
`graph.threads.com`, the host Meta's current documentation uses.
## Consequences
- Responses on all three silos show on the note within 15 minutes for new
notes, with no third party involved.
- Withdrawn likes and deleted replies disappear by themselves for 90 days;
after that a note's responses are frozen.
- Threads shows replies in full but likes, reposts and quotes only as
numbers.
- Replies from other servers to GoToSocial copies older than 90 days may
be missing: GoToSocial pruned them before StarPunk could read them.
- Backfeed stores other people's public posts and avatars. The privacy
rules (Decision 5) and moderation are the controls; there is no consent
step.
- More API calls. At about 50 new notes a month the routine schedule is
roughly 5,000 calls a month across the three silos, well inside their
limits. The backfill makes one burst of a few calls per copy.
- Deleting a webmention in the admin now leaves a tombstone row.
- A crosspost removed and posted again gets a new copy; responses to the
old copy are marked gone at the next check.
## Alternatives considered
- **Bridgy.** No Threads, 30-post window, no withdrawals, a third party in
the path, and it needs the own-copy rule and rate limit loosened (see
Context). Rejected.
- **A separate responses table.** Would duplicate the display, moderation
and 3.1.4 re-sending code that `webmentions` already has. Rejected.
- **Notification feeds** (Mastodon `/api/v1/notifications`, Bluesky
`listNotifications`): one call covers every post, but they only report
new events, so withdrawals are never seen and a backfill can't use them.
Could be added later as a fast path in front of per-post checks.
- **Streaming** (Bluesky Jetstream, Mastodon streaming API): real time, but
needs a long-lived connection per gunicorn worker. Not worth it for a
personal site.
- **Keep the Threads counts out.** They aren't IndieWeb responses, but they
are the only signal Threads gives, and showing them anonymously costs one
call and identifies no one. Kept.
+4 -3
View File
@@ -5,8 +5,8 @@ records one decision: its context, what was decided, and the consequences.
Superseded ADRs are kept as history; their Status line names the ADR that
replaced them.
**Total ADR files**: 68 (67 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-072, with two files numbered ADR-012). See [Numbering](#numbering).
**Total ADR files**: 69 (68 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-073, with two files numbered ADR-012). See [Numbering](#numbering).
Status values: **Accepted** (in force), **Superseded** (replaced by a later
ADR), **Deprecated** (withdrawn, nothing replaces it; none at present),
@@ -85,6 +85,7 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
| [066](ADR-066-bluesky-syndication.md) | Crossposting to Bluesky | Accepted | Bluesky as a second crosspost target (v2.2.0). |
| [069](ADR-069-threads-syndication.md) | Crossposting to Threads | Accepted | Threads as a third target, with automatic token refresh (v2.6.0). |
| [070](ADR-070-crosspost-links-and-cards.md) | Links and link cards in crossposts | Accepted | Keep note links in crossposts and build link cards for the linked page (v2.6.3). |
| [073](ADR-073-backfeed.md) | Backfeed of responses to crossposts | Proposed | Poll GoToSocial, Bluesky and Threads for responses to crossposts and store them as webmentions; admin backfill (planned v2.13.0). |
### Authentication (current)
@@ -167,7 +168,7 @@ was carried out and is not itself superseded.
## How to create a new ADR
1. Use the next number after the highest existing one (currently ADR-073).
1. Use the next number after the highest existing one (currently ADR-074).
2. Name it `ADR-NNN-brief-descriptive-title.md`.
3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071):
+3 -3
View File
@@ -75,9 +75,9 @@ None open.
- **Source**: Documentation refresh 2026-11-03
### Backfeed responses to crossposts
- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy.
- **Where**: `starpunk/syndication.py`, `starpunk/webmention_worker.py`
- **Source**: v2.8.0 microformats2 review discussion
- **Description**: Bring likes, reposts, quotes and replies to StarPunk's own crossposts back to the note as responses. Decided in ADR-073 (Proposed): native polling of GoToSocial, Bluesky and Threads into the `webmentions` table, Threads likes/reposts/quotes as counts, local avatar cache, admin backfill. Needs new GoToSocial and Threads tokens with read scopes.
- **Where**: `starpunk/syndication.py`, `starpunk/webmention_*.py`, `templates/partials/responses.html`, `starpunk/routes/admin.py`
- **Source**: v2.8.0 microformats2 review discussion; ADR-073 (2026-11-10)
### Tag-filtered feeds
- **Description**: Tag pages are h-feeds and WebSub topics, but there are no RSS, Atom or JSON feeds per tag (for example `/tag/python/feed.atom`). Tag pages also have no pagination, and the editor has no tag autocomplete.