From f8a429cdc17701c8b2b3d1aceec2d0c455db4f58 Mon Sep 17 00:00:00 2001 From: Phil Date: Tue, 10 Nov 2026 11:59:37 -0700 Subject: [PATCH 1/2] docs: ADR-073 backfeed of responses to crossposts (proposed) Co-Authored-By: Claude Opus 5.5 --- docs/decisions/ADR-073-backfeed.md | 249 +++++++++++++++++++++++++++++ docs/decisions/INDEX.md | 7 +- docs/projectplan/BACKLOG.md | 6 +- 3 files changed, 256 insertions(+), 6 deletions(-) create mode 100644 docs/decisions/ADR-073-backfeed.md diff --git a/docs/decisions/ADR-073-backfeed.md b/docs/decisions/ADR-073-backfeed.md new file mode 100644 index 0000000..eff18ff --- /dev/null +++ b/docs/decisions/ADR-073-backfeed.md @@ -0,0 +1,249 @@ +# ADR-073: Backfeed of responses to crossposts + +## Status + +Proposed (2026-11-10, for v2.13.0) + +## Context + +StarPunk crossposts notes to GoToSocial, Bluesky and Threads (ADR-065, +ADR-066, ADR-069). People like, repost, quote and reply to those copies, +and none of that reaches the note. [Backfeed](https://indieweb.org/backfeed) +brings those responses back to the original post. + +The usual tool is [Bridgy](https://brid.gy/about), which polls the silos and +sends webmentions. It is a poor fit here: + +- It supports Mastodon and Bluesky, not Threads. +- It only checks the 30 most recent posts, about every 30 minutes, and + says nothing when a like is withdrawn or a reply deleted. +- Since v2.8.0 StarPunk rejects webmentions from a note's own crossposts + (`_is_own_syndicated_copy`, `webmention_worker.py`). Bridgy's like and + repost pages most likely carry the crosspost's URL plus a fragment as + `u-url`, so they would be rejected. Replies have their own URL and get + through (production webmention #1 was a Bridgy reply from Mastodon). +- Every Bridgy source is on `brid.gy`, which shares one source-domain rate + limit of 10 webmentions a minute (ADR-063, Decision 10). +- Every response passes through a third party. + +StarPunk already holds a credential for each silo and knows each copy +(`syndications.remote_id`, `remote_url`). What the silos let it read: + +| Silo | Likes | Reposts | Quotes | Replies | Credential change | +|---|---|---|---|---|---| +| Bluesky | `getLikes` | `getRepostedBy` | `getQuotes` | `getPostThread` | none (app password) | +| GoToSocial | `favourited_by` | `reblogged_by` | not supported | `context` | read scopes | +| Threads | count only | count only | count only | `replies` | `threads_read_replies`, `threads_manage_insights` | + +The Threads API never says who liked, reposted or quoted a post; the +Insights API gives lifetime counts only. + +The production GoToSocial deletes cached remote posts older than 90 days, +so replies from other servers to older notes may no longer be there. + +## Decision + +### 1. Native polling, no Bridgy + +StarPunk polls each silo itself for responses to its own copies. Bridgy is +switched off for the Mastodon and Bluesky accounts once this ships. The +own-copy rule stays as it is. + +### 2. Backfed responses are webmentions + +Responses are stored in the `webmentions` table, so moderation, the +Responses section, and re-sending webmentions when a note's responses +change (spec 3.1.4, `note_responses_changed`) work as they do now. A +backfed row: + +- `source_url`: the response's own URL where it has one (a reply or quote + post). Likes and reposts have no page, so they get the copy's URL plus + `#liked-by-` or `#reposted-by-`. The + `(source_url, target_url)` key makes repeated checks update the same row. +- `target_url`: the note's permalink. `target_note_id`: the note. +- `origin`: new column. `webmention` for received webmentions (the + default), else `gotosocial`, `bluesky` or `threads`. +- `remote_id`: new column. The silo's id for the response (status id, + AT-URI, Threads media id), or the actor's id for a like or repost. +- `mention_type`: `like`, `repost`, `reply` or the new `quote`. +- Author name, profile URL and avatar from the silo's account data; + `content_snippet` from the reply or quote text, sanitised the same way as + received webmentions (Mastodon HTML through nh3, Bluesky and Threads text + escaped with links kept); `source_published` from the silo's timestamp. +- `validation_status = 'valid'`: StarPunk read it from the silo's API + itself, so there's nothing to fetch and verify. Moderation follows + `WEBMENTION_AUTO_APPROVE` like any first-time webmention. + +Only replies directly to the copy are taken, not deeper replies in the +thread. Responses from the accounts StarPunk posts as are skipped. + +### 3. Schedule + +Each synced copy of a published note is checked on a schedule that slows +as the note ages: + +| Note age | Check every | +|---|---| +| under 1 day | 15 minutes | +| 1 to 7 days | 1 hour | +| 7 to 30 days | 1 day | +| 30 to 90 days | 1 week | +| over 90 days | not checked | + +A random jitter of up to 10% spreads the checks. The cutoff is +`BACKFEED_MAX_AGE_DAYS` (default 90). Copies of likes and reposts +(ADR-068) aren't checked: they have no responses of their own. A note that +is unpublished or deleted is not checked, and its rows stay as they are. + +### 4. Withdrawn likes and deleted replies + +A check that read the whole list (every page, under a cap of 1,000 items +per list) marks that note's rows from that silo that are no longer there +as `validation_status = 'invalid'` with the error "No longer on +". They drop off the page but keep their moderation decision, and +come back if they reappear. A check that stopped early (an error, or the +cap) removes nothing. Checks of notes older than the cutoff (the backfill, +Decision 8) only add and update, never remove, because GoToSocial may have +pruned old replies. + +### 5. Privacy + +- Only public responses: GoToSocial `visibility` `public` (not unlisted, + followers-only or direct); on Bluesky, nothing from accounts with the + `!no-unauthenticated` label. +- People who opt out are skipped: `#nobridge` or `#nobot` in their bio + (Mastodon `note`, Bluesky `description`), the same signal Bridgy honours. + A row already stored for them is removed at the next check. +- Admin **Delete** stops being a real `DELETE` and sets + `moderation_status = 'deleted'` (the value exists but is unused), so a + later check or re-sent webmention can't bring the response back. This + applies to all webmentions. + +### 6. Avatars are cached locally + +Received webmentions hotlink author photos today. That leaks visitors' +addresses to other sites, and Threads and Bluesky avatar URLs change or +expire. Avatars are fetched once with `safe_get` (1 MB limit), resized to +96 px WebP with Pillow and saved as `DATA_PATH/avatars/.webp`, +served from `/avatars/`. They're refetched when the URL changes, and files +no row uses are removed weekly. Received webmentions use the same cache +from their validation step. A failed fetch falls back to the coloured +initial. + +### 7. Threads counts + +Threads likes, reposts and quotes are shown as anonymous counts, not as +responses. The worker reads +`/{media-id}/insights?metric=likes,reposts,quotes` with each Threads check +and stores the numbers on the copy's `syndications` row. The note shows a +plain line such as "On Threads: 12 likes · 3 reposts" after the +Responses, with no `u-like` or `h-cite` markup, because these aren't +people. Threads `replies` counts aren't used: the replies themselves are +shown. `views` and `shares` are left out (Meta marks them "in +development"). + +### 8. Backfill + +An admin action, **Backfill responses**, checks every synced copy once, +whatever the note's age. Nothing runs by itself on deploy; the owner runs +it after the first deploy (and may run it again later). It: + +- marks every synced copy due now, flagged as a backfill; +- is worked through by the same worker, at most one copy every 5 seconds, + ahead of the routine schedule only when nothing routine is due; +- only adds and updates (Decision 4); +- records when it started and finished; the admin page shows this, and + the button is disabled while one is running. + +Upserts make it safe to repeat. + +### 9. Worker + +The checks run in the syndication worker's loop (ADR-065) after crossposts, +reusing its clients, token refresh (`maintain()`) and the lease pattern: +a copy is claimed with `backfeed_claimed_until` so only one gunicorn worker +checks it. A failed check backs off like a failed crosspost +(`min(30 s × 2^attempts, 6 h)`) without changing the copy's crosspost +state. If a silo answers with a permission error, backfeed for that silo +stops until restart and the admin page says which credential needs which +scope; crossposting is unaffected. + +Per check: Bluesky 4 calls (likes, reposts, quotes, thread), GoToSocial 3, +Threads 2 (replies, insights), plus extra pages. + +### 10. Data (migration 020) + +- `webmentions`: rebuilt to add `quote` to the `mention_type` CHECK, with + new columns `origin TEXT NOT NULL DEFAULT 'webmention'` and + `remote_id TEXT`, and an index on `(target_note_id, origin)`. +- `syndications`: new columns `backfeed_next_at`, `backfeed_checked_at`, + `backfeed_claimed_until`, `backfeed_attempts`, `backfeed_error`, + `backfeed_backfill` (flag), and `likes_count`, `reposts_count`, + `quotes_count` (Threads only). +- The backfill's start and finish times go in the existing `site_state` + key/value table (migration 016): `backfeed_backfill_started`, + `backfeed_backfill_finished`. + +### 11. Display and admin + +- Each backfed response shows a small "via Bluesky" (or GoToSocial, + Threads) label. Quotes get their own group, rendered like mentions with + the quote text, as `u-mention h-cite`. +- The note editor shows when responses were last checked, with a + **Check now** button for each copy. +- The webmentions list can be filtered by origin. +- The dashboard shows backfeed errors next to crosspost errors. + +### 12. Configuration and credentials + +- `BACKFEED_ENABLED` (default `false`; production sets `true`) and + `BACKFEED_MAX_AGE_DAYS` (default 90). +- GoToSocial: a new token with read access added to the current scopes + (`read:statuses` and `read:accounts`, or plain `read`; checked against + GoToSocial's scope list when the token is issued). +- Threads: the Meta app is re-authorised with `threads_read_replies` and + `threads_manage_insights` added. The new `THREADS_ACCESS_TOKEN` replaces + the saved token file automatically (ADR-069, Decision 2). +- Bluesky: no change. +- Production tokens go in the Ansible vault (`bin/vault-run`). + +While at it, the Threads client moves from `graph.threads.net` to +`graph.threads.com`, the host Meta's current documentation uses. + +## Consequences + +- Responses on all three silos show on the note within 15 minutes for new + notes, with no third party involved. +- Withdrawn likes and deleted replies disappear by themselves for 90 days; + after that a note's responses are frozen. +- Threads shows replies in full but likes, reposts and quotes only as + numbers. +- Replies from other servers to GoToSocial copies older than 90 days may + be missing: GoToSocial pruned them before StarPunk could read them. +- Backfeed stores other people's public posts and avatars. The privacy + rules (Decision 5) and moderation are the controls; there is no consent + step. +- More API calls. At about 50 new notes a month the routine schedule is + roughly 5,000 calls a month across the three silos, well inside their + limits. The backfill makes one burst of a few calls per copy. +- Deleting a webmention in the admin now leaves a tombstone row. +- A crosspost removed and posted again gets a new copy; responses to the + old copy are marked gone at the next check. + +## Alternatives considered + +- **Bridgy.** No Threads, 30-post window, no withdrawals, a third party in + the path, and it needs the own-copy rule and rate limit loosened (see + Context). Rejected. +- **A separate responses table.** Would duplicate the display, moderation + and 3.1.4 re-sending code that `webmentions` already has. Rejected. +- **Notification feeds** (Mastodon `/api/v1/notifications`, Bluesky + `listNotifications`): one call covers every post, but they only report + new events, so withdrawals are never seen and a backfill can't use them. + Could be added later as a fast path in front of per-post checks. +- **Streaming** (Bluesky Jetstream, Mastodon streaming API): real time, but + needs a long-lived connection per gunicorn worker. Not worth it for a + personal site. +- **Keep the Threads counts out.** They aren't IndieWeb responses, but they + are the only signal Threads gives, and showing them anonymously costs one + call and identifies no one. Kept. diff --git a/docs/decisions/INDEX.md b/docs/decisions/INDEX.md index 14159da..fee6e6e 100644 --- a/docs/decisions/INDEX.md +++ b/docs/decisions/INDEX.md @@ -5,8 +5,8 @@ records one decision: its context, what was decided, and the consequences. Superseded ADRs are kept as history; their Status line names the ADR that replaced them. -**Total ADR files**: 68 (67 numbers: ADR-001 to ADR-044 and ADR-050 to -ADR-072, with two files numbered ADR-012). See [Numbering](#numbering). +**Total ADR files**: 69 (68 numbers: ADR-001 to ADR-044 and ADR-050 to +ADR-073, with two files numbered ADR-012). See [Numbering](#numbering). Status values: **Accepted** (in force), **Superseded** (replaced by a later ADR), **Deprecated** (withdrawn, nothing replaces it; none at present), @@ -85,6 +85,7 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md). | [066](ADR-066-bluesky-syndication.md) | Crossposting to Bluesky | Accepted | Bluesky as a second crosspost target (v2.2.0). | | [069](ADR-069-threads-syndication.md) | Crossposting to Threads | Accepted | Threads as a third target, with automatic token refresh (v2.6.0). | | [070](ADR-070-crosspost-links-and-cards.md) | Links and link cards in crossposts | Accepted | Keep note links in crossposts and build link cards for the linked page (v2.6.3). | +| [073](ADR-073-backfeed.md) | Backfeed of responses to crossposts | Proposed | Poll GoToSocial, Bluesky and Threads for responses to crossposts and store them as webmentions; admin backfill (planned v2.13.0). | ### Authentication (current) @@ -167,7 +168,7 @@ was carried out and is not itself superseded. ## How to create a new ADR -1. Use the next number after the highest existing one (currently ADR-073). +1. Use the next number after the highest existing one (currently ADR-074). 2. Name it `ADR-NNN-brief-descriptive-title.md`. 3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071): diff --git a/docs/projectplan/BACKLOG.md b/docs/projectplan/BACKLOG.md index cfe242a..5cea18b 100644 --- a/docs/projectplan/BACKLOG.md +++ b/docs/projectplan/BACKLOG.md @@ -75,9 +75,9 @@ None open. - **Source**: Documentation refresh 2026-11-03 ### Backfeed responses to crossposts -- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy. -- **Where**: `starpunk/syndication.py`, `starpunk/webmention_worker.py` -- **Source**: v2.8.0 microformats2 review discussion +- **Description**: Bring likes, reposts, quotes and replies to StarPunk's own crossposts back to the note as responses. Decided in ADR-073 (Proposed): native polling of GoToSocial, Bluesky and Threads into the `webmentions` table, Threads likes/reposts/quotes as counts, local avatar cache, admin backfill. Needs new GoToSocial and Threads tokens with read scopes. +- **Where**: `starpunk/syndication.py`, `starpunk/webmention_*.py`, `templates/partials/responses.html`, `starpunk/routes/admin.py` +- **Source**: v2.8.0 microformats2 review discussion; ADR-073 (2026-11-10) ### Tag-filtered feeds - **Description**: Tag pages are h-feeds and WebSub topics, but there are no RSS, Atom or JSON feeds per tag (for example `/tag/python/feed.atom`). Tag pages also have no pagination, and the editor has no tag autocomplete. From 2654692d151f18b08e6c48ece433ba2175a39781 Mon Sep 17 00:00:00 2001 From: Phil Date: Tue, 10 Nov 2026 12:03:40 -0700 Subject: [PATCH 2/2] docs: ADR-073: GoToSocial keeps threads with local posts, so backfill can reconcile Co-Authored-By: Claude Opus 5.5 --- docs/decisions/ADR-073-backfeed.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/decisions/ADR-073-backfeed.md b/docs/decisions/ADR-073-backfeed.md index eff18ff..a6e93bf 100644 --- a/docs/decisions/ADR-073-backfeed.md +++ b/docs/decisions/ADR-073-backfeed.md @@ -38,8 +38,11 @@ StarPunk already holds a credential for each silo and knows each copy The Threads API never says who liked, reposted or quoted a post; the Insights API gives lifetime counts only. -The production GoToSocial deletes cached remote posts older than 90 days, -so replies from other servers to older notes may no longer be there. +The production GoToSocial prunes remote posts older than 90 days, but never +a thread that a local account started or took part in (GoToSocial's "Post +Caching and Pruning" docs), so replies to StarPunk's copies are kept. On +2026-11-10 the production database still had 106 remote replies to the +account older than 90 days, back to August 2024. ## Decision @@ -102,9 +105,7 @@ per list) marks that note's rows from that silo that are no longer there as `validation_status = 'invalid'` with the error "No longer on ". They drop off the page but keep their moderation decision, and come back if they reappear. A check that stopped early (an error, or the -cap) removes nothing. Checks of notes older than the cutoff (the backfill, -Decision 8) only add and update, never remove, because GoToSocial may have -pruned old replies. +cap) removes nothing. ### 5. Privacy @@ -151,7 +152,6 @@ it after the first deploy (and may run it again later). It: - marks every synced copy due now, flagged as a backfill; - is worked through by the same worker, at most one copy every 5 seconds, ahead of the routine schedule only when nothing routine is due; -- only adds and updates (Decision 4); - records when it started and finished; the admin page shows this, and the button is disabled while one is running. @@ -218,8 +218,6 @@ While at it, the Threads client moves from `graph.threads.net` to after that a note's responses are frozen. - Threads shows replies in full but likes, reposts and quotes only as numbers. -- Replies from other servers to GoToSocial copies older than 90 days may - be missing: GoToSocial pruned them before StarPunk could read them. - Backfeed stores other people's public posts and avatars. The privacy rules (Decision 5) and moderation are the controls; there is no consent step.