Compare commits

...
22 Commits
Author SHA1 Message Date
Phil caa66f7bb1 Merge branch 'feat/v2.14.0-notifications' for v2.14.0 release
Build Container / build (push) Successful in 9m15s
2026-11-10 14:30:10 -07:00
PhilandClaude Opus 5.5 0465ff924b feat: notifications through Apprise (v2.14.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:30:01 -07:00
Phil 815aff5443 Merge branch 'docs/v2.14.0-notifications-adr': ADR-074 notifications 2026-11-10 14:25:58 -07:00
PhilandClaude Opus 5.5 4306633a42 docs: ADR-074 accepted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:25:57 -07:00
PhilandClaude Opus 5.5 c5c2d9832e docs: ADR-074: one configurable priority for all messages
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:24:43 -07:00
PhilandClaude Opus 5.5 483c24b565 docs: ADR-074 notifications through Apprise (proposed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:17:30 -07:00
Phil d312fa0074 Merge branch 'fix/v2.13.4-list-counts' for v2.13.4 release
Build Container / build (push) Successful in 8m56s
2026-11-10 14:05:05 -07:00
PhilandClaude Opus 5.5 647f6c723b feat: counts on lists of notes, reply cards link to the reply, fix class clash (v2.13.4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:05:03 -07:00
Phil 81c730813c Merge branch 'fix/v2.13.3-count-opted-out' for v2.13.3 release
Build Container / build (push) Successful in 8m35s
2026-11-10 13:19:18 -07:00
PhilandClaude Opus 5.5 113f28aff1 fix: count opted-out likes and reposts without naming them (v2.13.3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:16:19 -07:00
Phil 76b81d6ad9 Merge branch 'fix/v2.13.2-reaction-counts' for v2.13.2 release
Build Container / build (push) Successful in 8m34s
2026-11-10 13:11:30 -07:00
PhilandClaude Opus 5.5 44f77a3adf feat: likes and reposts as a counts line with hover lists, web actions (v2.13.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:11:29 -07:00
Phil b247de011d Merge branch 'fix/v2.13.1-avatar-fetch' for v2.13.1 release
Build Container / build (push) Successful in 8m33s
2026-11-10 12:56:11 -07:00
PhilandClaude Opus 5.5 65fc246d55 fix: save avatars from own GoToSocial and Bluesky PDSes (v2.13.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:56:04 -07:00
Phil f3ed1b3488 Merge branch 'feat/v2.13.0-backfeed' for v2.13.0 release
Build Container / build (push) Successful in 8m33s
2026-11-10 12:42:24 -07:00
PhilandClaude Opus 5.5 d400e4d838 fix: don't refetch Threads avatars whose signed URL changed (v2.13.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:21:03 -07:00
PhilandClaude Opus 5.5 55ce6a1fdd feat: backfeed of responses to crossposts (v2.13.0)
Likes, reposts, quotes and replies to a note's GoToSocial, Bluesky and
Threads copies are read back from the silos and stored as webmentions
(ADR-073): age-based schedule to 90 days, pruning of withdrawn responses,
#nobridge/#nobot opt-outs, admin backfill and Check now, Threads counts,
a Quotes group, local avatar copies, and delete tombstones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:19:57 -07:00
Phil 6aaac982d5 Merge branch 'docs/v2.13.0-backfeed-adr': ADR-073 backfeed (proposed) 2026-11-10 12:04:22 -07:00
PhilandClaude Opus 5.5 2654692d15 docs: ADR-073: GoToSocial keeps threads with local posts, so backfill can reconcile
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 12:03:40 -07:00
PhilandClaude Opus 5.5 f8a429cdc1 docs: ADR-073 backfeed of responses to crossposts (proposed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:59:37 -07:00
Phil afef1fc9a5 Merge branch 'fix/v2.12.1-category-unused-tags' for v2.12.1 release
Build Container / build (push) Successful in 8m3s
2026-11-10 11:28:42 -07:00
PhilandClaude Opus 5.5 b450c0c648 fix: q=category lists only tags on existing posts (v2.12.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:28:32 -07:00
43 changed files with 3416 additions and 129 deletions

No files matched your search

+26 -2
View File
@@ -165,7 +165,8 @@ WEBSUB_ENABLED=true
# GoToSocial or any Mastodon-API server (v2.1.0, ADR-065). Create an app
# token with scopes: write:statuses write:media, plus read:search
# write:favourites for native likes and boosts (v2.5.0)
# write:favourites for native likes and boosts (v2.5.0), plus read:statuses
# read:accounts (or read) for backfeed (v2.13.0)
GOTOSOCIAL_URL=
GOTOSOCIAL_ACCESS_TOKEN=
# Name shown in the editor and Micropub apps (default: GoToSocial)
@@ -185,7 +186,8 @@ BLUESKY_APP_PASSWORD=
# BLUESKY_MAX_CHARS=300
# Threads (v2.6.0, ADR-069). A long-lived user token with threads_basic,
# threads_content_publish and threads_delete; StarPunk refreshes it daily and
# threads_content_publish and threads_delete (plus threads_read_replies and
# threads_manage_insights for backfeed, v2.13.0); StarPunk refreshes it daily and
# keeps the current one in DATA_PATH/threads_token.json.
# How to get one: docs/design/v2.6.0/2026-10-31-threads-crossposting.md
THREADS_ACCESS_TOKEN=
@@ -198,6 +200,28 @@ THREADS_USERNAME=
# Seconds between syndication worker passes (default: 10)
# SYNDICATION_POLL_INTERVAL=10
# Backfeed (v2.13.0, ADR-073): bring likes, reposts, quotes and replies to the
# crossposts back to the note as responses. Needs the read scopes above.
# BACKFEED_ENABLED=false
# Stop checking a note's responses after this many days (default: 90)
# BACKFEED_MAX_AGE_DAYS=90
# =============================================================================
# NOTIFICATIONS (v2.14.0, ADR-074; optional)
# =============================================================================
# Apprise URLs to notify, space- or comma-separated
# (https://github.com/caronc/apprise/wiki). Empty: notifications off.
# Examples: gotifys://gotify.example.com/APPTOKEN ntfys://my-topic
# apprises://apprise.example.com/KEY (an apprise-api server)
# NOTIFY_URLS=
# What to notify: reply, mention, quote, like, repost, bookmark, problems
# (a crosspost that failed for good, a silo refusing backfeed)
# NOTIFY_EVENTS=reply,mention,quote,problems
# Priority of every message, added to URLs that don't set one:
# low, moderate, normal, high, emergency (Gotify also takes 0-10)
# NOTIFY_PRIORITY=normal
# =============================================================================
# DEBUGGING
# =============================================================================
+81
View File
@@ -7,6 +7,87 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.14.0] - 2026-11-10
Notifications (ADR-074).
### Added
- StarPunk can notify you of new responses through [Apprise](https://github.com/caronc/apprise): Gotify, ntfy, Pushover, email, Matrix, Discord and about 100 more, or an apprise-api server. Set `NOTIFY_URLS` to one or more Apprise URLs (e.g. `gotifys://gotify.example.com/APPTOKEN`)
- `NOTIFY_EVENTS` chooses what you hear about: `reply`, `mention`, `quote`, `like`, `repost`, `bookmark`, and `problems` (a crosspost that failed for good, or a silo refusing backfeed). Default `reply,mention,quote,problems`. Replies, mentions and quotes come one at a time with their text; likes, reposts and bookmarks are grouped per note
- `NOTIFY_PRIORITY` (default `normal`) sets the priority of every message
- Received webmentions and backfed responses are notified alike, each once. Responses waiting for moderation say so and link to the admin
- Admin → Webmentions shows the notification settings and the last send error, with **Send test notification**
### Changed
- New dependency: `apprise` 2.2. Migration 021 marks every existing response as already notified, and responses found by a backfill (or on a crosspost's first check of an older note) are never notified
## [2.13.4] - 2026-11-10
### Added
- The home page, tag pages and the Likes page show each note's like, repost and reply counts next to its date (♥ 3 🔁 1 💬 2), linking to the note's responses. Notes with none show nothing
### Changed
- Clicking anywhere on a reply opens it where it was posted (the reply on Bluesky, GoToSocial or Threads, or the page that sent the webmention). The author link and links in the reply still go to their own pages
### Fixed
- The "Liked", "Reposted" and "In reply to" lines on notes picked up styles meant for v2.13.2's counts line (both used the class `reaction`). The counts line's classes are now `response-*`
## [2.13.3] - 2026-11-10
### Changed
- Likes and reposts by people who opted out of backfeed (`#nobridge` or `#nobot` in their profile, or Bluesky's "don't show me to logged-out users" label) are counted in a note's totals, without their name or avatar ("+1 on Bluesky"). Before, they were left out entirely, so totals were lower than on the silo
## [2.13.2] - 2026-11-10
### Changed
- Likes, reposts and bookmarks are no longer rows of faces under "Responses". A note shows one line of counts under the post (♥ likes, 🔁 reposts, 💬 replies, bookmarks), and hovering or tapping a count lists who, with where each came from. Threads' anonymous counts are part of the totals ("+1 on Threads"), so the separate "On Threads" line is gone. The Responses section now holds replies, quotes and mentions
- Web actions (https://indieweb.org/webactions): each like, repost and reply count is an `<indie-action>` for the note, with links to do the same on the note's crossposts ("Like on Bluesky · GoToSocial · Threads"). The icons show even before anyone has responded, when the note has crossposts
### Fixed
- Avatars that GoToSocial hasn't cached (it redirects to the original server) are saved; the redirect is followed through the safe fetcher
## [2.13.1] - 2026-11-10
### Fixed
- Avatars from your own GoToSocial server are saved again. When StarPunk reaches it through the same proxy, its name resolves to an internal address, which the safe fetcher refused. That one server (`GOTOSOCIAL_URL`) is now fetched directly, as crossposting already does
- Bluesky avatars are saved again. Bluesky's image CDN refuses some servers' IPv4 addresses (HTTP 403), so avatars are now downloaded from the account's own server (`com.atproto.sync.getBlob`)
## [2.13.0] - 2026-11-10
Backfeed: responses to crossposts come back to the note (ADR-073).
### Added
- Likes, reposts, quotes and replies to a note's crossposts on GoToSocial and Bluesky, and replies on Threads, are shown on the note as responses, labelled "via Bluesky" (or GoToSocial, Threads). StarPunk checks each crosspost itself: every 15 minutes on a note's first day, hourly in its first week, daily until 30 days, weekly until `BACKFEED_MAX_AGE_DAYS` (90). Withdrawn likes and deleted replies drop off at the next check
- Threads like, repost and quote counts, shown as a line such as "On Threads: 12 likes · 3 reposts" (the Threads API doesn't say who)
- A Quotes group in a note's responses
- **Backfill responses** on the admin Webmentions page checks every crosspost once, however old the note; each note's edit page shows when its responses were last checked, with **Check now**
- The admin Webmentions list can be filtered by where a response came from, and shows deleted ones under **Deleted**
- Settings `BACKFEED_ENABLED` (default off) and `BACKFEED_MAX_AGE_DAYS`. Backfeed needs read access: a GoToSocial token with `read:statuses` and `read:accounts` (or `read`), and a Threads token with `threads_read_replies` and `threads_manage_insights`. Bluesky app passwords already work
### Changed
- Response authors' avatars are saved on this site (96 px WebP in `DATA_PATH/avatars`) and shown from there, instead of being loaded from other sites. Until a copy is saved the initial is shown
- Deleting a webmention in the admin keeps it as a deleted record, so a re-sent webmention or the next check doesn't bring it back
- People with `#nobridge` or `#nobot` in their profile, and Bluesky accounts that ask not to be shown to logged-out visitors, are not backfed; only public responses are
- A webmention for a response that was already backfed (for example a reply also sent by Bridgy) is ignored, so it isn't shown twice
- The Threads client uses `graph.threads.com`, the host in Meta's current documentation
## [2.12.1] - 2026-11-10
### Fixed
- Micropub `q=category` (the tag list apps use for autocomplete) only lists tags on posts that still exist. It used to include tags that an edit had removed from every post, and tags only used on deleted posts. Drafts still count; a deleted post's tags come back if it is undeleted
## [2.12.0] - 2026-11-03
The metrics dashboard is gone (ADR-072).
+2 -2
View File
@@ -202,9 +202,9 @@ coordinated through the data volume:
| Thread | Started when | Coordination | Work |
|--------|--------------|--------------|------|
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries |
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries; saves response avatars (v2.13.0); sends notifications of new responses (v2.14.0, `notifications.py`, ADR-074) |
| WebSub worker (`websub_worker.py`) | `WEBSUB_ENABLED` and not testing | Same pattern with `websub-worker.lock` | Verifies subscription intents, delivers topic content (signed with `X-Hub-Signature` when the subscriber gave a secret), retries, expires leases; pings an external hub instead if `WEBSUB_HUB_URL` is set |
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts |
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts. Then backfeed: checks copies for responses (v2.13.0, `backfeed.py`, ADR-073) |
All workers run inside an app context, are daemon threads (they end with the
process) and sleep for their poll interval when there is nothing to do. Under
@@ -223,6 +223,49 @@ Webmention worker rejects a webmention whose source is one of the note's
own copies (e.g. backfed by Bridgy), so a copy is never shown as a response
to itself (v2.8.0).
## Backfeed (v2.13.0, ADR-073)
`starpunk/backfeed.py` brings responses to the copies back to the note.
After each crosspost pass, the worker loop calls `backfeed.process_due()`,
which checks up to three synced copies of published notes (not likes or
reposts) that are due, each under a lease (`backfeed_claimed_until`):
- **Reading.** Each client has `responses(remote_id, remote_url)`, which
returns a `Harvest`: GoToSocial `favourited_by`, `reblogged_by` and
public direct replies from `context`; Bluesky `getLikes`,
`getRepostedBy`, `getQuotes` and `getPostThread` (depth 1), plus
`getProfiles` for reply and quote authors' bios; Threads `replies` and
`insights` (like, repost and quote counts). Lists are paged up to 1,000
items; a list cut short makes the harvest incomplete.
- **Storing.** Responses are upserted into `webmentions` with `origin` (the
target id) and `remote_id`, `validation_status = 'valid'`, keyed by
`(source_url, target_url)`. Likes and reposts use the copy's URL plus
`#liked-by-<actor>` / `#reposted-by-<actor>`. A new row gets the
`WEBMENTION_AUTO_APPROVE` default; an existing row keeps its moderation.
A complete harvest marks that silo's rows for the note that weren't
found `invalid` ("No longer on <silo>"). Authors with `#nobridge` or
`#nobot` in their bio, or Bluesky's `!no-unauthenticated` label, are
skipped and their rows deleted. If the note's shown responses changed,
its webmentions are re-sent (spec 3.1.4). Threads counts go in
`syndications.likes_count`, `reposts_count`, `quotes_count`.
- **Schedule.** `backfeed_next_at` follows the note's age: 15 minutes (day
1), 1 hour (week 1), 1 day (to 30 days), 1 week (to
`BACKFEED_MAX_AGE_DAYS`), with up to 10% jitter; none after. Failures back
off like crossposts; after 8 they wait a month.
- **Backfill.** `start_backfill()` flags every eligible copy
(`backfeed_backfill = 1`). Flagged copies are checked only when no
routine check is due, at most one every 5 seconds across workers
(`site_state.backfeed_backfill_last`). Start and finish times are in
`site_state`.
- **Refused credentials.** A 401/403 (or Graph auth error) puts the target
in `backfeed.denied` until restart; the admin Webmentions page says which
scopes are needed. Crossposting is unaffected.
Avatars of all responses (received and backfed) are saved by
`starpunk/avatars.py` and served from `/avatars/<sha256>.webp`; the
webmention worker fills in missing ones every 10 minutes and removes unused
ones weekly.
## Client sessions and tokens
- **Bluesky**: `createSession` with the app password, then
+1
View File
@@ -43,6 +43,7 @@ vulnerabilities (`pip-audit`); check again before each release.
| python-dateutil | `4.0.*` | Dates in reply contexts |
| feedgen | `1.0.*` | RSS 2.0 feed (`starpunk/feeds/rss.py`); Atom and JSON Feed are generated by StarPunk's own code |
| httpx | `0.28.*` | All outbound HTTP; requests to user-supplied URLs go through `webmention_ssrf.safe_get`/`safe_post` |
| apprise | `2.2.*` | Notifications to the owner's services (`starpunk/notifications.py`, ADR-074); sends to the URLs the owner configures, outside the SSRF-safe fetcher |
| PyJWT[crypto] | `2.15.*` | Verifying OIDC ID tokens (JWKS, RS/ES signatures via `cryptography`) |
| python-dotenv | `1.2.*` | Loading `.env` |
| Pillow | `13.0.*` | Image validation, resizing and variants |
+281
View File
@@ -0,0 +1,281 @@
# ADR-073: Backfeed of responses to crossposts
## Status
Accepted (v2.13.0, 2026-11-10)
Implementation notes (v2.13.0):
- Until an avatar has a local copy, pages show the initial and keep the
original URL only as `<data class="u-photo">` for microformats parsers,
so browsers don't load it.
- Bluesky post views carry no bio, so reply and quote authors' profiles
are fetched (`getProfiles`, 25 at a time) to honour `#nobridge`.
- A Threads token without `threads_manage_insights` stops Threads
backfeed as a whole, as for any refused credential.
- Backfill checks take a shared slot in `site_state`
(`backfeed_backfill_last`), so all gunicorn workers together do at most
one every 5 seconds.
## Context
StarPunk crossposts notes to GoToSocial, Bluesky and Threads (ADR-065,
ADR-066, ADR-069). People like, repost, quote and reply to those copies,
and none of that reaches the note. [Backfeed](https://indieweb.org/backfeed)
brings those responses back to the original post.
The usual tool is [Bridgy](https://brid.gy/about), which polls the silos and
sends webmentions. It is a poor fit here:
- It supports Mastodon and Bluesky, not Threads.
- It only checks the 30 most recent posts, about every 30 minutes, and
says nothing when a like is withdrawn or a reply deleted.
- Since v2.8.0 StarPunk rejects webmentions from a note's own crossposts
(`_is_own_syndicated_copy`, `webmention_worker.py`). Bridgy's like and
repost pages most likely carry the crosspost's URL plus a fragment as
`u-url`, so they would be rejected. Replies have their own URL and get
through (production webmention #1 was a Bridgy reply from Mastodon).
- Every Bridgy source is on `brid.gy`, which shares one source-domain rate
limit of 10 webmentions a minute (ADR-063, Decision 10).
- Every response passes through a third party.
StarPunk already holds a credential for each silo and knows each copy
(`syndications.remote_id`, `remote_url`). What the silos let it read:
| Silo | Likes | Reposts | Quotes | Replies | Credential change |
|---|---|---|---|---|---|
| Bluesky | `getLikes` | `getRepostedBy` | `getQuotes` | `getPostThread` | none (app password) |
| GoToSocial | `favourited_by` | `reblogged_by` | not supported | `context` | read scopes |
| Threads | count only | count only | count only | `replies` | `threads_read_replies`, `threads_manage_insights` |
The Threads API never says who liked, reposted or quoted a post; the
Insights API gives lifetime counts only.
The production GoToSocial prunes remote posts older than 90 days, but never
a thread that a local account started or took part in (GoToSocial's "Post
Caching and Pruning" docs), so replies to StarPunk's copies are kept. On
2026-11-10 the production database still had 106 remote replies to the
account older than 90 days, back to August 2024.
## Decision
### 1. Native polling, no Bridgy
StarPunk polls each silo itself for responses to its own copies. Bridgy is
switched off for the Mastodon and Bluesky accounts once this ships. The
own-copy rule stays as it is.
### 2. Backfed responses are webmentions
Responses are stored in the `webmentions` table, so moderation, the
Responses section, and re-sending webmentions when a note's responses
change (spec 3.1.4, `note_responses_changed`) work as they do now. A
backfed row:
- `source_url`: the response's own URL where it has one (a reply or quote
post). Likes and reposts have no page, so they get the copy's URL plus
`#liked-by-<actor id>` or `#reposted-by-<actor id>`. The
`(source_url, target_url)` key makes repeated checks update the same row.
- `target_url`: the note's permalink. `target_note_id`: the note.
- `origin`: new column. `webmention` for received webmentions (the
default), else `gotosocial`, `bluesky` or `threads`.
- `remote_id`: new column. The silo's id for the response (status id,
AT-URI, Threads media id), or the actor's id for a like or repost.
- `mention_type`: `like`, `repost`, `reply` or the new `quote`.
- Author name, profile URL and avatar from the silo's account data;
`content_snippet` from the reply or quote text, sanitised the same way as
received webmentions (Mastodon HTML through nh3, Bluesky and Threads text
escaped with links kept); `source_published` from the silo's timestamp.
- `validation_status = 'valid'`: StarPunk read it from the silo's API
itself, so there's nothing to fetch and verify. Moderation follows
`WEBMENTION_AUTO_APPROVE` like any first-time webmention.
Only replies directly to the copy are taken, not deeper replies in the
thread. Responses from the accounts StarPunk posts as are skipped.
### 3. Schedule
Each synced copy of a published note is checked on a schedule that slows
as the note ages:
| Note age | Check every |
|---|---|
| under 1 day | 15 minutes |
| 1 to 7 days | 1 hour |
| 7 to 30 days | 1 day |
| 30 to 90 days | 1 week |
| over 90 days | not checked |
A random jitter of up to 10% spreads the checks. The cutoff is
`BACKFEED_MAX_AGE_DAYS` (default 90). Copies of likes and reposts
(ADR-068) aren't checked: they have no responses of their own. A note that
is unpublished or deleted is not checked, and its rows stay as they are.
### 4. Withdrawn likes and deleted replies
A check that read the whole list (every page, under a cap of 1,000 items
per list) marks that note's rows from that silo that are no longer there
as `validation_status = 'invalid'` with the error "No longer on
<silo>". They drop off the page but keep their moderation decision, and
come back if they reappear. A check that stopped early (an error, or the
cap) removes nothing.
### 5. Privacy
- Only public responses: GoToSocial `visibility` `public` (not unlisted,
followers-only or direct); on Bluesky, nothing from accounts with the
`!no-unauthenticated` label.
- People who opt out are skipped: `#nobridge` or `#nobot` in their bio
(Mastodon `note`, Bluesky `description`), the same signal Bridgy honours.
A row already stored for them is removed at the next check.
- Admin **Delete** stops being a real `DELETE` and sets
`moderation_status = 'deleted'` (the value exists but is unused), so a
later check or re-sent webmention can't bring the response back. This
applies to all webmentions.
### 6. Avatars are cached locally
Received webmentions hotlink author photos today. That leaks visitors'
addresses to other sites, and Threads and Bluesky avatar URLs change or
expire. Avatars are fetched once with `safe_get` (1 MB limit), resized to
96 px WebP with Pillow and saved as `DATA_PATH/avatars/<sha256 of URL>.webp`,
served from `/avatars/`. They're refetched when the URL changes, and files
no row uses are removed weekly. Received webmentions use the same cache
from their validation step. A failed fetch falls back to the coloured
initial.
### 7. Threads counts
Threads likes, reposts and quotes are shown as anonymous counts, not as
responses. The worker reads
`/{media-id}/insights?metric=likes,reposts,quotes` with each Threads check
and stores the numbers on the copy's `syndications` row. The note shows a
plain line such as "On Threads: 12 likes · 3 reposts" after the
Responses, with no `u-like` or `h-cite` markup, because these aren't
people. Threads `replies` counts aren't used: the replies themselves are
shown. `views` and `shares` are left out (Meta marks them "in
development").
### 8. Backfill
An admin action, **Backfill responses**, checks every synced copy once,
whatever the note's age. Nothing runs by itself on deploy; the owner runs
it after the first deploy (and may run it again later). It:
- marks every synced copy due now, flagged as a backfill;
- is worked through by the same worker, at most one copy every 5 seconds,
ahead of the routine schedule only when nothing routine is due;
- records when it started and finished; the admin page shows this, and
the button is disabled while one is running.
Upserts make it safe to repeat.
### 9. Worker
The checks run in the syndication worker's loop (ADR-065) after crossposts,
reusing its clients, token refresh (`maintain()`) and the lease pattern:
a copy is claimed with `backfeed_claimed_until` so only one gunicorn worker
checks it. A failed check backs off like a failed crosspost
(`min(30 s × 2^attempts, 6 h)`) without changing the copy's crosspost
state. If a silo answers with a permission error, backfeed for that silo
stops until restart and the admin page says which credential needs which
scope; crossposting is unaffected.
Per check: Bluesky 4 calls (likes, reposts, quotes, thread), GoToSocial 3,
Threads 2 (replies, insights), plus extra pages.
### 10. Data (migration 020)
- `webmentions`: rebuilt to add `quote` to the `mention_type` CHECK, with
new columns `origin TEXT NOT NULL DEFAULT 'webmention'` and
`remote_id TEXT`, and an index on `(target_note_id, origin)`.
- `syndications`: new columns `backfeed_next_at`, `backfeed_checked_at`,
`backfeed_claimed_until`, `backfeed_attempts`, `backfeed_error`,
`backfeed_backfill` (flag), and `likes_count`, `reposts_count`,
`quotes_count` (Threads only).
- The backfill's start and finish times go in the existing `site_state`
key/value table (migration 016): `backfeed_backfill_started`,
`backfeed_backfill_finished`.
### 11. Display and admin
- Each backfed response shows a small "via Bluesky" (or GoToSocial,
Threads) label. Quotes get their own group, rendered like mentions with
the quote text, as `u-mention h-cite`.
- The note editor shows when responses were last checked, with a
**Check now** button for each copy.
- The webmentions list can be filtered by origin.
- The dashboard shows backfeed errors next to crosspost errors.
### 12. Configuration and credentials
- `BACKFEED_ENABLED` (default `false`; production sets `true`) and
`BACKFEED_MAX_AGE_DAYS` (default 90).
- GoToSocial: a new token with read access added to the current scopes
(`read:statuses` and `read:accounts`, or plain `read`; checked against
GoToSocial's scope list when the token is issued).
- Threads: the Meta app is re-authorised with `threads_read_replies` and
`threads_manage_insights` added. The new `THREADS_ACCESS_TOKEN` replaces
the saved token file automatically (ADR-069, Decision 2).
- Bluesky: no change.
- Production tokens go in the Ansible vault (`bin/vault-run`).
While at it, the Threads client moves from `graph.threads.net` to
`graph.threads.com`, the host Meta's current documentation uses.
## Amendments (v2.13.2, 2026-11-10)
- **Display.** Likes, reposts and bookmarks are a line of counts under the
note instead of facepiles in the Responses section. Hovering or focusing
a count lists the people (as `u-like` / `u-repost` / `u-bookmark`
h-cites) and the silo each came from. Threads' anonymous counts are added
to the totals ("+N on Threads") instead of a separate line (Decision 7).
- **Web actions.** Each like, repost and reply count is an
`<indie-action do="…" with="<permalink>">` whose fallback is a link to
the same action on each crosspost. No web-action polyfill is loaded;
handlers such as browser extensions can use the markup.
## Amendments (v2.13.3, 2026-11-10)
- **Opted-out people are counted.** Their likes, reposts and quotes are
still not stored or named (Decision 5), but each check records how many
there were on the copy's `syndications` row (`likes_count`,
`reposts_count`, `quotes_count`, the columns Threads' counts use), and
the counts line adds them to the totals as "N on <silo>". Totals then
match the silo.
## Consequences
- Responses on all three silos show on the note within 15 minutes for new
notes, with no third party involved.
- Withdrawn likes and deleted replies disappear by themselves for 90 days;
after that a note's responses are frozen.
- Threads shows replies in full but likes, reposts and quotes only as
numbers.
- Backfeed stores other people's public posts and avatars. The privacy
rules (Decision 5) and moderation are the controls; there is no consent
step.
- More API calls. At about 50 new notes a month the routine schedule is
roughly 5,000 calls a month across the three silos, well inside their
limits. The backfill makes one burst of a few calls per copy.
- Deleting a webmention in the admin now leaves a tombstone row.
- A crosspost removed and posted again gets a new copy; responses to the
old copy are marked gone at the next check.
## Alternatives considered
- **Bridgy.** No Threads, 30-post window, no withdrawals, a third party in
the path, and it needs the own-copy rule and rate limit loosened (see
Context). Rejected.
- **A separate responses table.** Would duplicate the display, moderation
and 3.1.4 re-sending code that `webmentions` already has. Rejected.
- **Notification feeds** (Mastodon `/api/v1/notifications`, Bluesky
`listNotifications`): one call covers every post, but they only report
new events, so withdrawals are never seen and a backfill can't use them.
Could be added later as a fast path in front of per-post checks.
- **Streaming** (Bluesky Jetstream, Mastodon streaming API): real time, but
needs a long-lived connection per gunicorn worker. Not worth it for a
personal site.
- **Keep the Threads counts out.** They aren't IndieWeb responses, but they
are the only signal Threads gives, and showing them anonymously costs one
call and identifies no one. Kept.
+155
View File
@@ -0,0 +1,155 @@
# ADR-074: Notifications through Apprise
## Status
Accepted (v2.14.0, 2026-11-10)
## Context
Responses now reach a note from two places: received webmentions
(ADR-063) and backfeed from the crossposts (ADR-073). The owner only finds
out about them by opening the site or the admin. Crossposts that finally
fail, and silos that refuse a backfeed credential, are just as invisible.
Owners use different notification services. The production site uses
Gotify (read on iOS through iGotify). Others use ntfy, Pushover, email,
Matrix, Discord and so on. [Apprise](https://github.com/caronc/apprise)
covers about 100 of them behind one URL syntax (`gotifys://host/token`,
`ntfys://topic`, `mailtos://…`), both as a Python library and as a
separate HTTP server (apprise-api).
Options considered:
1. **Gotify only.** Simplest for this site, no use to anyone else.
2. **Post to an apprise-api server.** Standard, but every owner needs one
running, and messages take an extra hop.
3. **Embed the Apprise library.** Sends straight to each service; an owner
who prefers a central apprise-api can still use its `apprise://` /
`apprises://` URL.
4. **Gotify plus a generic JSON webhook, hand-written.** No dependency, but
every other service needs glue the owner has to write.
## Decision
### 1. Embed Apprise
StarPunk depends on the `apprise` package and sends with it directly.
`NOTIFY_URLS` holds one or more Apprise URLs (space- or comma-separated);
each message goes to all of them. With no URLs, notifications are off and
nothing else changes.
### 2. What can be notified
`NOTIFY_EVENTS` is a comma-separated list. Default:
`reply,mention,quote,problems`.
| Event | Sent when |
|---|---|
| `reply`, `mention`, `quote`, `like`, `repost`, `bookmark` | a response of that type is first shown on a note, or first awaits moderation (received webmention or backfeed alike) |
| `problems` | a crosspost fails for good (permanent error or last attempt), or a silo refuses the backfeed credential (ADR-073 Decision 9) |
### 3. Messages
Markdown, so services that render it (Gotify, iGotify, ntfy, Matrix…)
show links; others get the text.
- **Replies, mentions, quotes:** one message each. Title "↩ Reply from Ana
Lima via Bluesky" (or "💬 Mention from …", "❝ Quote from …"); body the
response text (plain, up to 300 characters), then a link to the note's
responses (`<permalink>#responses`).
- **Likes, reposts, bookmarks:** grouped per note and type in each pass:
"♥ 3 new likes on 'Starpunk now supports…'", with up to five names and a
link to the note.
- **Awaiting moderation:** when `WEBMENTION_AUTO_APPROVE` is off, the title
says so and the link goes to the admin webmention page instead.
- **Problems:** "⚠ Crosspost to Bluesky failed" / "⚠ Backfeed for Threads
stopped" with the error and a link to the note's edit page or the
Webmentions page.
Apprise's Gotify plugin has no click-through link, so the link is in the
text.
### 3a. Priority
Every message is sent with the same priority, `NOTIFY_PRIORITY` (default
`normal`). Apprise takes priority as a `priority` argument in each service
URL, with values that differ by service, so StarPunk adds
`priority=<NOTIFY_PRIORITY>` to each URL in `NOTIFY_URLS` that doesn't set
one itself; a URL can still set its own. Apprise's names (`low`,
`moderate`, `normal`, `high`, `emergency`) work for most services; Gotify
also takes 0–10 (it maps the names to 0, 3, 5, 8, 10). Different
priorities per event can come later.
### 4. Delivery
- New column `webmentions.notified_at`. A sweep finds responses with
`notified_at IS NULL`, `validation_status = 'valid'`, a moderation status
of `approved` or `pending`, and a type in `NOTIFY_EVENTS`, sends them,
and sets `notified_at`. Types not selected are marked without sending.
- The sweep runs in the webmention worker loop, which already runs in one
process at a time (its file lock), so nothing is sent twice by several
gunicorn workers. It handles received and backfed responses alike.
- A failed send leaves `notified_at` empty, so the next pass (about 10
seconds later) retries, with backoff up to 15 minutes. Responses older
than 24 hours that still haven't gone out are marked and dropped, with a
warning, so an outage doesn't end in a flood. The last error is shown on
the admin Webmentions page.
- Problems are sent at the moment they happen (from the syndication and
backfeed workers) and not retried: they are also in the log and admin.
- Sending never blocks or breaks validation, crossposting or backfeed.
### 5. No floods from history
- Migration 021 sets `notified_at` on every existing response.
- Responses stored by a backfill check (ADR-073 Decision 8) are stored with
`notified_at` set.
- So are responses found on a crosspost's first check when its note is more
than a day old (copies linked after the fact).
- Grouping (Decision 3) keeps a busy note to one like message per pass.
### 6. Admin
- **Send test notification** on the admin Webmentions page sends a test
message to every URL and reports each result.
- The page shows whether notifications are on, which events are selected,
and the last send error.
### 7. Configuration
| Setting | Default | Meaning |
|---|---|---|
| `NOTIFY_URLS` | empty (off) | Apprise URLs, space- or comma-separated |
| `NOTIFY_EVENTS` | `reply,mention,quote,problems` | which events are sent |
| `NOTIFY_PRIORITY` | `normal` | priority of every message, added to URLs that don't set one |
Settings stay in the environment, like the rest of StarPunk's (ADR-052).
URLs contain tokens and belong with the other secrets.
## Consequences
- One new dependency, `apprise`, with its own (requests, PyYAML, click,
Markdown, certifi). It must install on Python 3.14 in the container
image; this is checked before the release.
- Owners can send to almost any service without writing glue, directly or
through apprise-api.
- Each response is notified at most once, a few seconds after it appears,
and only if its type is selected.
- Apprise opens outbound connections to whatever URLs the owner configures,
outside the SSRF-safe fetcher. That is intended: the owner chooses these
URLs, as with the crosspost targets. A service behind the same reverse
proxy as StarPunk may need the deployment's internal-address entry
(`extra_hosts`), as GoToSocial does.
- Notification text includes other people's names and words, sent to the
owner's own services only.
## Alternatives considered
- **Gotify only / hand-written webhook.** See Context, options 1 and 4: no
dependency, but each service beyond Gotify needs glue the owner writes.
- **apprise-api only.** See option 2. Still possible through an
`apprises://` URL.
- **Sending from where responses are stored** (webmention worker,
backfeed) instead of a sweep. Two code paths, no retry, and backfeed
runs in every gunicorn worker. Rejected.
- **Choosing events in the admin UI.** Possible later; the environment is
where StarPunk's settings live today.
+5 -3
View File
@@ -5,8 +5,8 @@ records one decision: its context, what was decided, and the consequences.
Superseded ADRs are kept as history; their Status line names the ADR that
replaced them.
**Total ADR files**: 68 (67 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-072, with two files numbered ADR-012). See [Numbering](#numbering).
**Total ADR files**: 70 (69 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-074, with two files numbered ADR-012). See [Numbering](#numbering).
Status values: **Accepted** (in force), **Superseded** (replaced by a later
ADR), **Deprecated** (withdrawn, nothing replaces it; none at present),
@@ -85,6 +85,8 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
| [066](ADR-066-bluesky-syndication.md) | Crossposting to Bluesky | Accepted | Bluesky as a second crosspost target (v2.2.0). |
| [069](ADR-069-threads-syndication.md) | Crossposting to Threads | Accepted | Threads as a third target, with automatic token refresh (v2.6.0). |
| [070](ADR-070-crosspost-links-and-cards.md) | Links and link cards in crossposts | Accepted | Keep note links in crossposts and build link cards for the linked page (v2.6.3). |
| [074](ADR-074-notifications.md) | Notifications through Apprise | Accepted | Notify the owner of new responses and crosspost/backfeed problems via the embedded Apprise library (v2.14.0). |
| [073](ADR-073-backfeed.md) | Backfeed of responses to crossposts | Accepted | Poll GoToSocial, Bluesky and Threads for responses to crossposts and store them as webmentions; admin backfill (v2.13.0). |
### Authentication (current)
@@ -167,7 +169,7 @@ was carried out and is not itself superseded.
## How to create a new ADR
1. Use the next number after the highest existing one (currently ADR-073).
1. Use the next number after the highest existing one (currently ADR-075).
2. Name it `ADR-NNN-brief-descriptive-title.md`.
3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071):
+29 -1
View File
@@ -21,7 +21,7 @@ is in `../architecture/`; decisions are in `../decisions/` (ADRs).
## Contents
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [v2.12.0](#v2120) · [hotfix](#hotfix)
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [v2.12.0](#v2120) · [v2.12.1](#v2121) · [v2.13.0](#v2130) · [v2.13.1](#v2131) · [v2.13.2](#v2132) · [v2.13.3](#v2133) · [v2.13.4](#v2134) · [v2.14.0](#v2140) · [hotfix](#hotfix)
## v1.0.0
@@ -418,6 +418,34 @@ OIDC admin login and built-in IndieAuth server.
- [2026-11-03-remove-metrics.md](v2.12.0/2026-11-03-remove-metrics.md) - Metrics dashboard and `starpunk/monitoring` removed (ADR-072)
## v2.12.1
- [2026-11-10-category-unused-tags.md](v2.12.1/2026-11-10-category-unused-tags.md) - q=category lists only tags on posts that exist
## v2.13.0
- [2026-11-10-backfeed.md](v2.13.0/2026-11-10-backfeed.md) - Backfeed of responses to crossposts (ADR-073), avatar cache, delete tombstones
## v2.13.1
- [2026-11-10-avatar-fetch.md](v2.13.1/2026-11-10-avatar-fetch.md) - Avatars from the own GoToSocial server and from Bluesky PDSes
## v2.13.2
- [2026-11-10-reaction-counts.md](v2.13.2/2026-11-10-reaction-counts.md) - Likes/reposts as a counts line with hover lists; web actions
## v2.13.3
- [2026-11-10-count-opted-out.md](v2.13.3/2026-11-10-count-opted-out.md) - Opted-out people's likes and reposts counted without names
## v2.13.4
- [2026-11-10-list-counts.md](v2.13.4/2026-11-10-list-counts.md) - Counts on lists of notes; whole reply card links to the reply; class clash fixed
## v2.14.0
- [2026-11-10-notifications.md](v2.14.0/2026-11-10-notifications.md) - Notifications through Apprise (ADR-074)
## hotfix
Not a release folder.
@@ -0,0 +1,28 @@
# v2.12.1: q=category lists only tags in use
## Problem
`q=category` (added in v2.4.0) is what Micropub clients use for tag
autocomplete. It read every row of `tags` with a `LEFT JOIN` on
`note_tags`, so it also returned:
- tags no post uses any more (a `tags` row stays after an edit removes
the tag from its last post), and
- tags only used on soft-deleted posts (`note_tags` rows stay so that
undelete can restore them).
Clients then offered tags that appear nowhere on the site.
## Fix
`_all_categories()` in `starpunk/micropub.py` inner-joins `note_tags` and
`notes` with `deleted_at IS NULL`. Order is unchanged: most used first,
then by name. Drafts count, since they are the owner's posts. Unused
`tags` rows are left in place; nothing else reads them as a list.
## Verification
`tests/test_micropub_v240.py::TestQueries::test_category_skips_unused_tags`
retags one post and deletes another, checks only the live tag is listed,
then undeletes and checks the deleted post's tag returns. It fails on
v2.12.0.
@@ -0,0 +1,53 @@
# v2.13.0: Backfeed (implementation report)
Implements ADR-073.
## What changed
- `migrations/020_backfeed.sql`: `webmentions` rebuilt with `origin`,
`remote_id` and the `quote` type (rows copied, indexes recreated, plus
`(target_note_id, origin)`); `syndications` gains the `backfeed_*`
columns and Threads counts.
- `starpunk/backfeed.py` (new): `Response`/`Harvest`, schedule, lease,
storing and pruning, opt-outs, backfill, `check_now`, `status`.
- `starpunk/syndication.py`: `responses()` on `MastodonClient`,
`BlueskyClient` and `ThreadsClient`; HTTP status kept on errors
(`exc.status`) so refused credentials can be told apart; Mastodon `_call`
accepts absolute URLs (pagination links); the worker loop runs backfeed
after crossposts; Threads host is `graph.threads.com`.
- `starpunk/avatars.py` (new) and `/avatars/<name>`: local avatar copies
for every response; `local_avatar` template filter.
- Webmentions: Quotes group and "via" label; admin delete is a tombstone
(`moderation_status = 'deleted'`, hidden from the list unless filtered);
origin filter; a webmention can't reset a backfed row, and one whose
source is an already backfed response is rejected ("Already backfed").
- Admin: Backfeed panel with **Backfill responses** on the Webmentions
page; **Check now** and last-checked time per crosspost on the edit page.
- Settings `BACKFEED_ENABLED` (default false), `BACKFEED_MAX_AGE_DAYS` (90).
## Verification
- `tests/test_backfeed.py` (22 tests): schedule; text/facet HTML; storing,
updating and moderation kept; pruning only on complete harvests and
restoring; opt-outs; tombstones; backfill and its pacing; likes/reposts
and drafts skipped; refused credentials; backoff; note page (via label,
quotes, Threads counts, local avatars); avatar save/serve/cleanup;
webmention can't overwrite a backfed row; the three clients against fake
APIs; migration 020 on a pre-020 database keeps rows and foreign keys;
the admin Backfeed panel, backfill and Check now; Threads avatar URLs
(signed, changing per request) aren't refetched while a copy exists.
- Tests for the touched areas (webmentions, syndication, microformats,
migrations, templates, routes): all pass. The full suite runs in CI.
- `test_microformats_v280` expects response authors' `u-photo`: kept via
`<data class="u-photo">` until the avatar is saved.
## Deploying
1. Issue a new GoToSocial token with `read:statuses read:accounts` added
to the current scopes, and re-authorise the Threads app with
`threads_read_replies` and `threads_manage_insights`. Put both in the
Ansible vault.
2. Set `BACKFEED_ENABLED=true` in the StarPunk environment and deploy
(migration 020 runs at start: back up the database first).
3. Run **Backfill responses** once from Admin → Webmentions. Turn Bridgy
off for the Mastodon and Bluesky accounts.
@@ -0,0 +1,33 @@
# v2.13.1: Avatar fetching in production
## Problem
After deploying v2.13.0, backfeed stored responses but no avatar was saved:
- GoToSocial avatars (`https://social.../fileserver/...`):
`SSRFBlockedError: Blocked request to private IP: 172.20.0.5`. Inside the
container the GoToSocial host resolves to the reverse proxy's internal
address (the deployment's hairpin `extra_hosts` entry).
- Bluesky avatars (`https://cdn.bsky.app/img/avatar/...`): HTTP 403 from
BunnyCDN. From the server, the CDN answers 403 over IPv4 and 200 over
IPv6; the container has no IPv6 route. From elsewhere IPv4 works.
## Fix (`starpunk/avatars.py`, `_download`)
- The host of `GOTOSOCIAL_URL` is fetched with a plain HTTP client (no
redirects, 1 MB limit), as `MastodonClient` already talks to it. Every
other host still goes through `safe_get`, so private addresses stay
blocked.
- `cdn.bsky.app/img/<kind>/plain/<did>/<cid>` URLs are fetched from the
account's PDS with `com.atproto.sync.getBlob`; the PDS comes from the DID
document (`plc.directory` for `did:plc`, `/.well-known/did.json` for
`did:web`). Both requests use `safe_get`. The cache file is still named
after the original URL.
## Verification
- `tests/test_backfeed.py::test_avatars_from_own_gotosocial_and_bluesky_pds`:
own GoToSocial on a private address is fetched, a Bluesky CDN URL is
fetched from its PDS, another private address is still refused.
- Checked against the real PLC directory and a bsky.network PDS.
- From the production server: getBlob over IPv4 returns the image (200).
@@ -0,0 +1,32 @@
# v2.13.2: Reaction counts and web actions
## Why
With backfeed live, notes showed "5 Likes" and "1 Repost" headings over
rows of faces (blank until avatars were saved), plus a separate
"On Threads: 1 like" line. The owner asked for the usual presentation: a
flat count under the post, with who on hover. Web actions
(https://indieweb.org/webactions) fit the same line.
## What changed
- `templates/partials/responses.html`: `facepile` and `silo_counts` are
replaced by `reactions` / `reaction`. Each count is a focusable
`.reaction` with an SVG icon; its `.reaction-who` card lists the people
(avatar, name, silo) inside a scrollable `.reaction-people`, Threads'
anonymous counts, and an `<indie-action>` with links to the crossposts.
The reply count has no people list (replies are listed below).
- `templates/note.html`: the counts line follows the note's footer; the
Responses section shows only when there are replies, quotes or mentions.
- `static/css/style.css`: facepile styles replaced by `.reactions`; the card
opens on `:hover` and `:focus-within` (tap on touch screens).
- `starpunk/avatars.py`: a redirect from the own GoToSocial server (media it
hasn't cached) is followed through `safe_get`.
## Verification
- `tests/test_backfeed.py` (24 tests): totals include Threads' anonymous
count, `<indie-action>` and crosspost links present; redirected avatar
saved. Microformats, template and route tests pass.
- Rendered locally in headless Chromium in dark and light themes with the
like and reply cards open.
@@ -0,0 +1,22 @@
# v2.13.3: Count opted-out people without naming them
## Why
A note's Bluesky copy had 1 like on Bluesky and 0 on the note. The liker's
account has the `!no-unauthenticated` label, so backfeed skipped them
(ADR-073 Decision 5). Every other Bluesky copy's like and repost counts
matched what StarPunk had stored.
## Change
`backfeed.hidden_counts(harvest)`: the silo's own counts when it gives no
names (Threads), else the number of opted-out likes, reposts and quotes in
the harvest. `check()` stores it on the copy's `syndications` row, and the
counts line (v2.13.2) already adds those columns to the totals as
"N on <silo>". Opted-out people are still not stored.
## Verification
`tests/test_backfeed.py::test_people_who_opt_out_are_counted_not_named`:
label, `#nobot` and `#nobridge` opt-outs counted (2 likes, 1 repost), not
stored, not named on the page; totals include them.
@@ -0,0 +1,24 @@
# v2.13.4: Counts on lists of notes, reply cards as links
## What changed
- `webmention_queries.response_counts(note_ids)`: likes, reposts and
replies per note in two grouped queries (shown webmentions, plus the
anonymous counts on synced `syndications` rows). `routes/public.py`
`attach_response_counts()` runs after `attach_contexts()` on the home,
tag and Likes pages.
- `partials/responses.html` `list_counts`: icons and numbers in the note's
meta line, linking to `<permalink>#responses` (an anchor on the note page).
- Reply cards: the date link (already the reply's own URL) stretches over
the card with `::after`; author and content links sit above it.
- v2.13.2's `.reaction*` classes clashed with the older `.reaction` lines
(Liked / Reposted / In reply to, `partials/reaction.html`): the counts
line now uses `.response-count*` / `.response-card*` / `.response-*`.
## Verification
- `tests/test_backfeed.py::test_home_page_shows_counts`; template, route,
microformats and likes/reposts tests pass.
- Rendered the home page, a note page and the Likes page in headless
Chromium: counts beside the date, quiet notes unchanged, the Liked line
laid out as before v2.13.2.
@@ -0,0 +1,38 @@
# v2.14.0: Notifications (implementation report)
Implements ADR-074.
## What changed
- `starpunk/notifications.py` (new): settings (`urls`, `events`,
`with_priority`), `send` (Apprise, Markdown, 15 s timeout, never raises,
errors captured with `apprise.LogCapture`), `problem`, and `Sweeper`,
which sends due responses, groups likes/reposts/bookmarks per note,
marks `notified_at`, backs off after a failure (10 s doubling to 15 min),
drops responses older than 24 hours, and records the last error in
`site_state.notify_last_error`.
- `migrations/021_notifications.sql`: `webmentions.notified_at`, set on all
existing rows.
- `webmention_worker.py`: runs the sweep each pass (one process holds the
worker lock).
- `backfeed.py`: `store(..., quiet=True)` inserts new rows as notified;
`check()` is quiet for backfill checks and a copy's first check of a note
older than a day. A refused credential sends a `problems` message.
- `syndication.py`: `_fail` sends a `problems` message when it gives up.
- Admin Webmentions page: status (events, services, priority, last error)
and **Send test notification** (`POST /admin/notifications/test`).
- Settings `NOTIFY_URLS`, `NOTIFY_EVENTS`, `NOTIFY_PRIORITY`; dependency
`apprise==2.2.*`.
## Verification
- `tests/test_notifications.py` (11 tests): settings parsing and priority
injection; one message per reply/mention, likes grouped, unselected types
marked, pending moderation linked to the admin; nothing sent twice;
failure backoff and retry, last error set and cleared; 24-hour drop; off
without URLs; quiet backfill and late first checks; problems (selected or
not) and the final crosspost failure; admin status and test button; a
real Apprise send (`json://`) to a local server; migration 021.
- Webmention, syndication, backfeed, admin and migration tests pass.
- Apprise 2.2.0 installs and imports on Python 3.14.
- Production Gotify (3.1.1) accepted a message with the StarPunk app token.
-5
View File
@@ -74,11 +74,6 @@ None open.
- **Where**: `templates/base.html`
- **Source**: Documentation refresh 2026-11-03
### Backfeed responses to crossposts
- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy.
- **Where**: `starpunk/syndication.py`, `starpunk/webmention_worker.py`
- **Source**: v2.8.0 microformats2 review discussion
### Tag-filtered feeds
- **Description**: Tag pages are h-feeds and WebSub topics, but there are no RSS, Atom or JSON feeds per tag (for example `/tag/python/feed.atom`). Tag pages also have no pagination, and the editor has no tag autocomplete.
- **Where**: `starpunk/routes/public.py`, `starpunk/feeds/`, `templates/admin/`
+1 -1
View File
@@ -4,7 +4,7 @@
**Last Updated**: 2026-11-03 (v2.10.1)
StarPunk uses [Semantic Versioning 2.0.0](https://semver.org/):
`MAJOR.MINOR.PATCH`, currently **2.10.1**. Every release is a git tag
`MAJOR.MINOR.PATCH`, currently **2.14.0**. Every release is a git tag
`vX.Y.Z` and a `CHANGELOG.md` entry. Where the number is stored and how to
write the changelog entry is in
[version-implementation-guide.md](version-implementation-guide.md); the
+66
View File
@@ -0,0 +1,66 @@
-- Migration 020: Backfeed of responses to crossposts (v2.13.0, ADR-073)
--
-- webmentions: where a response came from (origin, remote_id) and a new
-- 'quote' type. SQLite can't change a CHECK constraint, so the table is
-- rebuilt (as in migration 003). Nothing references webmentions.
--
-- syndications: when each copy's responses were last checked and are next
-- due, a lease so one gunicorn worker checks a copy, and Threads counts.
CREATE TABLE webmentions_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_url TEXT NOT NULL,
target_url TEXT NOT NULL,
target_note_id INTEGER,
mention_type TEXT NOT NULL DEFAULT 'mention'
CHECK (mention_type IN ('reply', 'like', 'repost', 'bookmark', 'mention', 'quote')),
author_name TEXT,
author_url TEXT,
author_photo TEXT,
content_snippet TEXT,
validation_status TEXT NOT NULL DEFAULT 'pending'
CHECK (validation_status IN ('pending', 'valid', 'invalid', 'error')),
moderation_status TEXT NOT NULL DEFAULT 'pending'
CHECK (moderation_status IN ('pending', 'approved', 'hidden', 'deleted')),
validation_error TEXT,
validated_at TIMESTAMP,
received_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
source_post_url TEXT,
source_published TIMESTAMP,
origin TEXT NOT NULL DEFAULT 'webmention',
remote_id TEXT,
FOREIGN KEY (target_note_id) REFERENCES notes(id) ON DELETE SET NULL,
UNIQUE(source_url, target_url)
);
INSERT INTO webmentions_new (
id, source_url, target_url, target_note_id, mention_type, author_name, author_url,
author_photo, content_snippet, validation_status, moderation_status, validation_error,
validated_at, received_at, updated_at, source_post_url, source_published
)
SELECT
id, source_url, target_url, target_note_id, mention_type, author_name, author_url,
author_photo, content_snippet, validation_status, moderation_status, validation_error,
validated_at, received_at, updated_at, source_post_url, source_published
FROM webmentions;
DROP TABLE webmentions;
ALTER TABLE webmentions_new RENAME TO webmentions;
CREATE INDEX IF NOT EXISTS idx_webmentions_target_note ON webmentions(target_note_id);
CREATE INDEX IF NOT EXISTS idx_webmentions_moderation ON webmentions(moderation_status);
CREATE INDEX IF NOT EXISTS idx_webmentions_validation ON webmentions(validation_status);
CREATE INDEX IF NOT EXISTS idx_webmentions_received ON webmentions(received_at DESC);
CREATE INDEX IF NOT EXISTS idx_webmentions_note_origin ON webmentions(target_note_id, origin);
ALTER TABLE syndications ADD COLUMN backfeed_next_at TIMESTAMP;
ALTER TABLE syndications ADD COLUMN backfeed_checked_at TIMESTAMP;
ALTER TABLE syndications ADD COLUMN backfeed_claimed_until TIMESTAMP;
ALTER TABLE syndications ADD COLUMN backfeed_attempts INTEGER NOT NULL DEFAULT 0;
ALTER TABLE syndications ADD COLUMN backfeed_error TEXT;
ALTER TABLE syndications ADD COLUMN backfeed_backfill INTEGER NOT NULL DEFAULT 0;
ALTER TABLE syndications ADD COLUMN likes_count INTEGER;
ALTER TABLE syndications ADD COLUMN reposts_count INTEGER;
ALTER TABLE syndications ADD COLUMN quotes_count INTEGER;
+6
View File
@@ -0,0 +1,6 @@
-- Migration 021: Notifications (v2.14.0, ADR-074)
-- When the owner was notified of a response. Everything already stored
-- counts as notified, so turning notifications on doesn't send history.
ALTER TABLE webmentions ADD COLUMN notified_at TIMESTAMP;
UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP;
+3
View File
@@ -19,6 +19,9 @@ httpx==0.28.*
PyJWT[crypto]==2.15.*
python-dotenv==1.2.*
# Notifications (v2.14.0, ADR-074)
apprise==2.2.*
# Images
Pillow==13.0.*
pillow-heif==1.8.*
+10 -2
View File
@@ -217,6 +217,14 @@ def create_app(config=None):
from starpunk.utils import iso_utc
app.add_template_filter(iso_utc, 'iso_utc')
# Template filter - local copy of a response author's avatar, or None (v2.13.0)
@app.template_filter('local_avatar')
def local_avatar_filter(url):
from flask import url_for
from starpunk.avatars import cached_name
name = cached_name(app.config.get('DATA_PATH', 'data'), url)
return url_for('public.avatar', name=name) if name else None
# Template filter - URL to color for facepile (v1.6.0)
@app.template_filter('url_to_color')
def url_to_color_filter(url):
@@ -457,5 +465,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.12.0"
__version_info__ = (2, 12, 0)
__version__ = "2.14.0"
__version_info__ = (2, 14, 0)
+217
View File
@@ -0,0 +1,217 @@
"""
Local copies of response authors' avatars (v2.13.0, ADR-073 Decision 6)
Pages used to show other sites' avatar URLs directly, which tells those
sites who is reading, and Bluesky and Threads avatar URLs change or expire.
Each avatar is fetched once (SSRF-safe), shrunk to a 96 px WebP and saved
as DATA_PATH/avatars/<sha256 of its URL>.webp. Pages show the saved copy,
or the coloured initial until there is one.
Paths and URLs are passed in, so the webmention worker can use it. Fetching
needs an app context (the SSRF-safe fetcher reads DEV_MODE).
"""
import hashlib
import io
import logging
import re
import sqlite3
import time
from pathlib import Path
from typing import Optional
logger = logging.getLogger(__name__)
SIZE = 96
MAX_BYTES = 1024 * 1024
# Bluesky avatar on its CDN: .../img/<kind>/plain/<did>/<cid>[@jpeg]
BSKY_CDN_RE = re.compile(
r"^https://cdn\.bsky\.app/img/[a-z_]+/plain/(did:(?:plc|web):[A-Za-z0-9._:%-]+)/([a-z2-7]+)(?:@\w+)?$")
NAME_RE = re.compile(r"^[0-9a-f]{64}\.webp$")
CLEANUP_EVERY = 7 * 24 * 3600
def name_for(url: str) -> str:
return hashlib.sha256(url.encode()).hexdigest() + ".webp"
def folder(data_path) -> Path:
return Path(data_path) / "avatars"
def cached_name(data_path, url: Optional[str]) -> Optional[str]:
"""File name of the saved copy of this avatar, if there is one"""
if not url or not url.startswith(("http://", "https://")):
return None
name = name_for(url)
return name if (folder(data_path) / name).is_file() else None
def fetch(data_path, url: Optional[str], site_url: str) -> bool:
"""
Save a copy of an avatar unless there is one. Returns True if one exists
afterwards. Failures are logged and the page keeps showing the initial.
"""
if not url or not url.startswith(("http://", "https://")):
return False
path = folder(data_path) / name_for(url)
if path.is_file():
return True
from PIL import Image
try:
with Image.open(io.BytesIO(_download(url, site_url))) as image:
image.load()
image = image.convert("RGBA" if "A" in image.getbands() else "RGB")
side = min(image.size)
left, top = (image.width - side) // 2, (image.height - side) // 2
image = image.crop((left, top, left + side, top + side))
image = image.resize((SIZE, SIZE), Image.LANCZOS)
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
image.save(tmp, "WEBP", quality=80)
tmp.replace(path)
return True
except Exception as e:
logger.info(f"Could not save avatar {url}: {type(e).__name__}: {e}")
return False
def _own_instance_host() -> Optional[str]:
"""Host of the GoToSocial server StarPunk posts to (it may resolve to an internal address)"""
from urllib.parse import urlparse
from flask import current_app
return urlparse(current_app.config.get("GOTOSOCIAL_URL") or "").hostname
def _get(url: str, headers: dict) -> bytes:
"""GET a public URL through the SSRF-safe fetcher"""
from starpunk.webmention_ssrf import safe_get
result = safe_get(url, headers=headers, max_bytes=MAX_BYTES, max_redirects=5,
timeout=10.0, truncate=False)
if result.status_code >= 400:
raise ValueError(f"HTTP {result.status_code}")
return result.content
def _pds(did: str, headers: dict) -> str:
"""The PDS that hosts a Bluesky account, from its DID document"""
if did.startswith("did:plc:"):
doc_url = f"https://plc.directory/{did}"
else: # did:web:<host>
doc_url = f"https://{did[len('did:web:'):]}/.well-known/did.json"
import json
for service in json.loads(_get(doc_url, headers)).get("service") or []:
if service.get("id", "").endswith("#atproto_pds") and service.get("serviceEndpoint"):
return service["serviceEndpoint"].rstrip("/")
raise ValueError(f"no PDS for {did}")
def _download(url: str, site_url: str) -> bytes:
"""
The avatar's bytes (v2.13.1):
- from StarPunk's own GoToSocial directly, as the crossposting client
does: behind the same proxy it resolves to an internal address, which
the SSRF-safe fetcher refuses;
- Bluesky CDN avatars from the account's PDS (com.atproto.sync.getBlob),
because the CDN refuses some servers' IPv4 addresses;
- anything else through the SSRF-safe fetcher.
"""
from urllib.parse import quote, urlparse
import httpx
from starpunk import webmention_ssrf
from starpunk.webmention_ssrf import user_agent
headers = {"User-Agent": user_agent(site_url)}
own = _own_instance_host()
if own and urlparse(url).hostname == own:
with httpx.Client(timeout=10.0, transport=webmention_ssrf.TRANSPORT) as http:
with http.stream("GET", url, headers=headers) as response:
if response.is_redirect:
# GoToSocial sends media it hasn't cached to the original
# server: that one is fetched like any outside URL (v2.13.2)
from urllib.parse import urljoin
return _get(urljoin(url, response.headers["location"]), headers)
if response.status_code >= 400:
raise ValueError(f"HTTP {response.status_code}")
data = b""
for chunk in response.iter_bytes():
data += chunk
if len(data) > MAX_BYTES:
raise ValueError("too large")
return data
match = BSKY_CDN_RE.match(url)
if match:
did, cid = match.groups()
url = (f"{_pds(did, headers)}/xrpc/com.atproto.sync.getBlob"
f"?did={quote(did, safe=':')}&cid={cid}")
return _get(url, headers)
def fill_missing(db_path, data_path, site_url: str, limit: int = 20) -> int:
"""Save avatars of shown responses that have no copy yet. Returns how many were tried."""
conn = sqlite3.connect(db_path)
try:
urls = [row[0] for row in conn.execute(
"""
SELECT DISTINCT author_photo FROM webmentions
WHERE author_photo IS NOT NULL AND author_photo != ''
AND validation_status = 'valid' AND moderation_status = 'approved'
ORDER BY id DESC
"""
)]
finally:
conn.close()
tried = 0
for url in urls:
if tried >= limit:
break
if cached_name(data_path, url) or not url.startswith(("http://", "https://")):
continue
fetch(data_path, url, site_url)
tried += 1
return tried
def remove_unused(db_path, data_path) -> int:
"""Delete saved avatars no stored response uses. Returns how many were removed."""
directory = folder(data_path)
if not directory.is_dir():
return 0
conn = sqlite3.connect(db_path)
try:
used = {name_for(row[0]) for row in conn.execute(
"SELECT DISTINCT author_photo FROM webmentions WHERE author_photo IS NOT NULL")}
finally:
conn.close()
removed = 0
for path in directory.iterdir():
if NAME_RE.match(path.name) and path.name not in used:
path.unlink(missing_ok=True)
removed += 1
return removed
class Upkeep:
"""Runs fill_missing every `fill_every` seconds and remove_unused weekly"""
def __init__(self, fill_every: int = 600):
self.fill_every = fill_every
self._filled_at = 0.0
self._cleaned_at = 0.0
def run(self, db_path, data_path, site_url: str) -> None:
now = time.time()
if now - self._filled_at >= self.fill_every:
self._filled_at = now
fill_missing(db_path, data_path, site_url)
if now - self._cleaned_at >= CLEANUP_EVERY:
self._cleaned_at = now
removed = remove_unused(db_path, data_path)
if removed:
logger.info(f"Removed {removed} unused avatars")
+485
View File
@@ -0,0 +1,485 @@
"""
Backfeed of responses to crossposts (v2.13.0, ADR-073)
StarPunk asks each silo for the responses to its own copies of a note
(likes, reposts, quotes and replies) and stores them in `webmentions`, so
moderation, the Responses section and re-sending webmentions when a note's
responses change (spec 3.1.4) work as they do for received webmentions.
Each synced copy of a published note is checked on a schedule that slows as
the note ages, and not at all after BACKFEED_MAX_AGE_DAYS. A backfill
checks every copy once, whatever its age. Checks run in the syndication
worker's loop (`process_due`), one copy at a time under a lease.
A check that read the whole list marks responses that are no longer there
as invalid, so withdrawn likes and deleted replies drop off the page.
"""
import html
import random
import re
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Dict, List, Optional
from flask import current_app
from starpunk.database import get_db
DAY = 24 * 3600
SCHEDULE = [ # (note younger than, check every), in seconds
(DAY, 15 * 60),
(7 * DAY, 3600),
(30 * DAY, DAY),
]
LATE_INTERVAL = 7 * DAY # from 30 days until the cutoff
CLAIM_MINUTES = 5
MAX_ATTEMPTS = 8
BACKFILL_GAP = 5 # seconds between backfill checks, across workers
SILO_NAMES = {"gotosocial": "GoToSocial", "bluesky": "Bluesky", "threads": "Threads"}
OPT_OUT_RE = re.compile(r"#no(bridge|bot)\b", re.IGNORECASE)
# Silos whose credential was refused this run: target id -> message
denied: Dict[str, str] = {}
@dataclass
class Response:
"""One like, repost, quote or reply on a silo"""
kind: str # like, repost, quote, reply
source_url: str # its URL (likes/reposts: the copy + #liked-by-<id>)
remote_id: str # silo id of the response, or of the actor
author_id: str = ""
author_name: str = ""
author_url: str = ""
author_photo: Optional[str] = None
author_bio: str = ""
opted_out: bool = False # set by the client from silo-specific signals
content_html: Optional[str] = None
published: Optional[str] = None
@dataclass
class Harvest:
"""What one check found. complete=False: some list was cut short."""
responses: List[Response]
complete: bool = True
# Likes/reposts/quotes shown only as numbers: all of them on Threads
# (its API doesn't say who); elsewhere, those by people who opted out
counts: Optional[Dict[str, int]] = None
# --- Text -----------------------------------------------------------------
def text_html(text: str) -> str:
"""Plain text as HTML: escaped, URLs linked, line breaks kept"""
escaped = html.escape(text)
linked = re.sub(r"https?://[^\s<>&\"]+",
lambda m: f'<a href="{m.group(0)}">{m.group(0)}</a>', escaped)
return "<p>" + linked.replace("\n", "<br>") + "</p>" if text.strip() else ""
def facets_html(text: str, facets: list) -> str:
"""Bluesky post text as HTML, with its link facets (byte ranges) as links"""
data = text.encode()
links = []
for facet in facets:
index = facet.get("index") or {}
for feature in facet.get("features") or []:
uri = feature.get("uri") or ""
if feature.get("$type") == "app.bsky.richtext.facet#link" and uri.startswith(("http://", "https://")):
links.append((index.get("byteStart", 0), index.get("byteEnd", 0), uri))
links.sort()
parts, pos = [], 0
for start, end, uri in links:
if start < pos or end > len(data) or start >= end:
continue
parts.append(html.escape(data[pos:start].decode(errors="ignore")))
label = html.escape(data[start:end].decode(errors="ignore"))
parts.append(f'<a href="{html.escape(uri, quote=True)}">{label}</a>')
pos = end
parts.append(html.escape(data[pos:].decode(errors="ignore")))
body = "".join(parts)
return "<p>" + body.replace("\n", "<br>") + "</p>" if text.strip() else ""
# --- Scheduling --------------------------------------------------------------
def interval_for(age_seconds: float, max_age_days: int) -> Optional[int]:
"""Seconds until a note of this age is checked again; None past the cutoff"""
if age_seconds >= max_age_days * DAY:
return None
for younger_than, every in SCHEDULE:
if age_seconds < younger_than:
return every
return LATE_INTERVAL
def _age_seconds(created_at) -> float:
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
if created_at.tzinfo is None:
created_at = created_at.replace(tzinfo=timezone.utc)
return (datetime.now(timezone.utc) - created_at).total_seconds()
_ELIGIBLE = """
FROM syndications s JOIN notes n ON n.id = s.note_id
WHERE s.state = 'synced' AND s.enabled = 1 AND s.remote_id IS NOT NULL
AND n.published = 1 AND n.deleted_at IS NULL
AND n.like_of IS NULL AND n.repost_of IS NULL
"""
def _claim(db, row_id: int) -> bool:
cursor = db.execute(
f"""
UPDATE syndications SET backfeed_claimed_until = datetime('now', '+{CLAIM_MINUTES} minutes')
WHERE id = ? AND (backfeed_claimed_until IS NULL OR backfeed_claimed_until < datetime('now'))
""",
(row_id,),
)
db.commit()
return cursor.rowcount == 1
def _backfill_turn(db) -> bool:
"""Take the next backfill slot: at most one check every BACKFILL_GAP seconds"""
cursor = db.execute(
f"""
UPDATE site_state SET value = datetime('now')
WHERE key = 'backfeed_backfill_last' AND value <= datetime('now', '-{BACKFILL_GAP} seconds')
"""
)
if cursor.rowcount == 0:
cursor = db.execute(
"INSERT OR IGNORE INTO site_state (key, value) VALUES ('backfeed_backfill_last', datetime('now'))")
db.commit()
return cursor.rowcount == 1
def _due(db, limit: int) -> list:
"""Routine checks due now; a backfill check only when none are"""
max_age = int(current_app.config.get("BACKFEED_MAX_AGE_DAYS", 90))
lease = "(s.backfeed_claimed_until IS NULL OR s.backfeed_claimed_until < datetime('now'))"
rows = db.execute(
f"""
SELECT s.*, n.created_at AS note_created, n.slug AS note_slug {_ELIGIBLE}
AND {lease}
AND n.created_at >= datetime('now', ?)
AND (s.backfeed_next_at IS NULL OR s.backfeed_next_at <= datetime('now'))
ORDER BY s.backfeed_next_at IS NOT NULL, s.backfeed_next_at LIMIT ?
""",
(f"-{max_age} days", limit),
).fetchall()
if rows:
return rows
backfill = db.execute(
f"""
SELECT s.*, n.created_at AS note_created, n.slug AS note_slug {_ELIGIBLE}
AND {lease} AND s.backfeed_backfill = 1
AND (s.backfeed_next_at IS NULL OR s.backfeed_next_at <= datetime('now'))
ORDER BY s.id LIMIT 1
"""
).fetchall()
if backfill and _backfill_turn(db):
return backfill
return []
def _finish(db, row) -> None:
max_age = int(current_app.config.get("BACKFEED_MAX_AGE_DAYS", 90))
every = interval_for(_age_seconds(row["note_created"]), max_age)
if every is None:
next_at = None
else:
next_at = f"+{int(every * random.uniform(1.0, 1.1))} seconds"
db.execute(
"""
UPDATE syndications
SET backfeed_checked_at = datetime('now'), backfeed_claimed_until = NULL,
backfeed_attempts = 0, backfeed_error = NULL, backfeed_backfill = 0,
backfeed_next_at = CASE WHEN ? IS NULL THEN NULL ELSE datetime('now', ?) END
WHERE id = ?
""",
(next_at, next_at, row["id"]),
)
db.commit()
def _fail(db, row, message: str) -> None:
"""Back off; after MAX_ATTEMPTS drop any backfill and try again in a month"""
attempts = row["backfeed_attempts"] + 1
stop = attempts >= MAX_ATTEMPTS
delay = 30 * DAY if stop else min(30 * (2 ** attempts), 6 * 3600)
db.execute(
f"""
UPDATE syndications
SET backfeed_attempts = ?, backfeed_error = ?, backfeed_claimed_until = NULL,
backfeed_backfill = CASE WHEN ? THEN 0 ELSE backfeed_backfill END,
backfeed_next_at = datetime('now', '+{delay} seconds')
WHERE id = ?
""",
(attempts, message[:500], 1 if stop else 0, row["id"]),
)
db.commit()
current_app.logger.warning(
f"Backfeed {row['target']} for note {row['note_id']} failed (attempt {attempts}): {message}")
# --- Storing -----------------------------------------------------------------
def _opted_out(response: Response) -> bool:
return response.opted_out or bool(OPT_OUT_RE.search(html.unescape(response.author_bio or "")))
def _display_state(db, note_id: int) -> list:
return [tuple(r) for r in db.execute(
"""
SELECT id, mention_type, author_name, author_url, author_photo, content_snippet
FROM webmentions
WHERE target_note_id = ? AND validation_status = 'valid' AND moderation_status = 'approved'
ORDER BY id
""",
(note_id,),
)]
def hidden_counts(harvest: Harvest) -> Dict[str, int]:
"""
Likes, reposts and quotes to show only as numbers: the silo's own counts
when it gives no names (Threads), else those by people who opted out
(v2.13.3: they're counted, not named)
"""
if harvest.counts is not None:
return harvest.counts
hidden = [r.kind for r in harvest.responses if _opted_out(r)]
return {k: hidden.count(kind) for k, kind in
(("likes", "like"), ("reposts", "repost"), ("quotes", "quote"))}
def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
quiet: bool = False) -> None:
"""
Upsert a check's responses for one note and silo. With a complete
harvest, that silo's responses for the note that weren't found are
marked invalid. Responses from people who opted out are removed (they
are counted instead, see `hidden_counts`). With `quiet`, new responses
are stored as already notified (ADR-074 Decision 5).
"""
from starpunk.webmention_processor import _snippet
moderation = "approved" if current_app.config.get("WEBMENTION_AUTO_APPROVE", True) else "pending"
silo = SILO_NAMES.get(origin, origin)
keep = [r for r in harvest.responses if not _opted_out(r)]
dropped = [r.source_url for r in harvest.responses if _opted_out(r)]
from starpunk.avatars import cached_name
before = _display_state(db, note_id)
for r in keep:
snippet = _snippet(r.content_html) if r.content_html else None
# Threads avatar URLs are signed and change on every request: keep
# the stored URL while its saved copy exists, so it isn't refetched
old = db.execute("SELECT author_photo FROM webmentions WHERE source_url = ? AND target_url = ?",
(r.source_url, target_url)).fetchone()
if old and old["author_photo"] and origin == "threads" \
and cached_name(current_app.config["DATA_PATH"], old["author_photo"]):
r.author_photo = old["author_photo"]
db.execute(
"""
INSERT INTO webmentions (
source_url, target_url, target_note_id, mention_type, author_name, author_url,
author_photo, content_snippet, validation_status, moderation_status,
validated_at, source_published, origin, remote_id, notified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'valid', ?, CURRENT_TIMESTAMP, ?, ?, ?,
CASE WHEN ? THEN CURRENT_TIMESTAMP END)
ON CONFLICT(source_url, target_url) DO UPDATE SET
target_note_id = excluded.target_note_id,
mention_type = excluded.mention_type,
author_name = excluded.author_name,
author_url = excluded.author_url,
author_photo = excluded.author_photo,
content_snippet = excluded.content_snippet,
source_published = COALESCE(excluded.source_published, webmentions.source_published),
origin = excluded.origin,
remote_id = excluded.remote_id,
validation_status = 'valid',
validation_error = NULL,
validated_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP
""",
(r.source_url, target_url, note_id, r.kind, r.author_name or None, r.author_url or None,
r.author_photo or None, snippet, moderation, r.published, origin, r.remote_id,
1 if quiet else 0),
)
if dropped:
marks = ",".join("?" * len(dropped))
db.execute(f"DELETE FROM webmentions WHERE origin = ? AND target_note_id = ? AND source_url IN ({marks})",
[origin, note_id] + dropped)
if harvest.complete:
found = [r.source_url for r in keep]
marks = ",".join("?" * len(found)) or "''"
db.execute(
f"""
UPDATE webmentions
SET validation_status = 'invalid', validation_error = ?, updated_at = CURRENT_TIMESTAMP
WHERE target_note_id = ? AND origin = ? AND validation_status = 'valid'
AND source_url NOT IN ({marks})
""",
[f"No longer on {silo}", note_id, origin] + found,
)
db.commit()
if _display_state(db, note_id) != before:
try:
from pathlib import Path
from starpunk.webmention_sender import note_responses_changed
note_responses_changed(Path(current_app.config["DATABASE_PATH"]), [note_id])
except Exception as e:
current_app.logger.warning(f"Failed to re-send webmentions for note {note_id}: {e}")
from starpunk import avatars
for r in keep:
avatars.fetch(current_app.config["DATA_PATH"], r.author_photo,
current_app.config.get("SITE_URL", ""))
def _store_counts(db, row_id: int, counts: Optional[dict]) -> None:
if counts is None:
return
db.execute(
"UPDATE syndications SET likes_count = ?, reposts_count = ?, quotes_count = ? WHERE id = ?",
(counts.get("likes"), counts.get("reposts"), counts.get("quotes"), row_id),
)
db.commit()
# --- Checking ---------------------------------------------------------------
def _is_denied(error) -> bool:
"""The silo refused the credential (missing scope or bad token)"""
from starpunk.syndication import _GRAPH_AUTH
return getattr(error, "status", None) in (401, 403) or getattr(error, "code", None) in _GRAPH_AUTH
SCOPES = {
"gotosocial": "a GoToSocial token with read:statuses and read:accounts (or read)",
"bluesky": "a working Bluesky app password",
"threads": "a Threads token with threads_read_replies and threads_manage_insights",
}
def check(row, target) -> None:
"""Check one copy now and store what it found (raises SyndicationError)"""
db = get_db(current_app)
site = current_app.config["SITE_URL"].rstrip("/")
harvest = target.client.responses(row["remote_id"], row["remote_url"] or "")
# No notifications for history: a backfill, or a copy's first check on an
# older note (e.g. one linked after the fact). ADR-074 Decision 5.
quiet = bool(row["backfeed_backfill"]) or (
row["backfeed_checked_at"] is None and _age_seconds(row["note_created"]) > DAY)
store(db, row["note_id"], f"{site}/note/{row['note_slug']}", row["target"], harvest, quiet=quiet)
_store_counts(db, row["id"], hidden_counts(harvest))
def process_due(limit: int = 3) -> int:
"""Check up to `limit` copies that are due. Returns how many were handled."""
if not current_app.config.get("BACKFEED_ENABLED", False):
return 0
from starpunk.syndication import SyndicationError, get_targets
targets = get_targets()
db = get_db(current_app)
handled = 0
for row in _due(db, limit):
if row["target"] in denied or row["target"] not in targets:
continue
if not _claim(db, row["id"]):
continue
handled += 1
try:
check(row, targets[row["target"]])
except SyndicationError as e:
if _is_denied(e):
denied[row["target"]] = f"Needs {SCOPES.get(row['target'], 'read access')}: {e}"
current_app.logger.error(f"Backfeed for {row['target']} stopped: {denied[row['target']]}")
from starpunk import notifications
site = current_app.config["SITE_URL"].rstrip("/")
notifications.problem(current_app.config, current_app.config["DATABASE_PATH"],
f"Backfeed for {SILO_NAMES.get(row['target'], row['target'])} stopped",
denied[row["target"]], f"{site}/admin/webmentions")
db.execute("UPDATE syndications SET backfeed_claimed_until = NULL WHERE id = ?", (row["id"],))
db.commit()
continue
_fail(db, row, str(e))
continue
except Exception as e: # one bad copy mustn't stop the worker
_fail(db, row, f"unexpected error: {e}")
continue
_finish(db, row)
_note_backfill_done(db)
return handled
# --- Backfill and admin ---------------------------------------------------------
def _state(db, key: str) -> Optional[str]:
row = db.execute("SELECT value FROM site_state WHERE key = ?", (key,)).fetchone()
return row["value"] if row and row["value"] else None
def _set_state(db, key: str, value: str) -> None:
db.execute("INSERT INTO site_state (key, value) VALUES (?, ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value", (key, value))
def start_backfill() -> int:
"""Mark every eligible copy for one check, whatever its age. Returns how many."""
db = get_db(current_app)
cursor = db.execute(
f"""
UPDATE syndications
SET backfeed_backfill = 1, backfeed_attempts = 0, backfeed_next_at = datetime('now')
WHERE id IN (SELECT s.id {_ELIGIBLE})
"""
)
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
_set_state(db, "backfeed_backfill_started", now)
_set_state(db, "backfeed_backfill_finished", "")
db.commit()
return cursor.rowcount
def _note_backfill_done(db) -> None:
if _state(db, "backfeed_backfill_started") and not _state(db, "backfeed_backfill_finished"):
left = db.execute("SELECT COUNT(*) FROM syndications WHERE backfeed_backfill = 1").fetchone()[0]
if left == 0:
_set_state(db, "backfeed_backfill_finished",
datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S"))
db.commit()
def check_now(syndication_id: int) -> None:
"""Check this copy's responses at the next worker pass"""
db = get_db(current_app)
db.execute("UPDATE syndications SET backfeed_next_at = datetime('now'), backfeed_attempts = 0 "
"WHERE id = ?", (syndication_id,))
db.commit()
def status() -> dict:
"""For the admin pages: backfill progress and silos whose credential was refused"""
db = get_db(current_app)
return {
"enabled": bool(current_app.config.get("BACKFEED_ENABLED", False)),
"started": _state(db, "backfeed_backfill_started"),
"finished": _state(db, "backfeed_backfill_finished"),
"remaining": db.execute(
"SELECT COUNT(*) FROM syndications WHERE backfeed_backfill = 1").fetchone()[0],
"denied": dict(denied),
"errors": db.execute(
"SELECT COUNT(*) FROM syndications WHERE backfeed_error IS NOT NULL").fetchone()[0],
}
+7
View File
@@ -146,6 +146,13 @@ def load_config(app, config_override=None):
app.config["THREADS_MAX_CHARS"] = int(os.getenv("THREADS_MAX_CHARS", "500"))
app.config["THREADS_MAX_MEDIA"] = int(os.getenv("THREADS_MAX_MEDIA", "4"))
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
# Backfeed of responses to crossposts (v2.13.0, ADR-073)
app.config["BACKFEED_ENABLED"] = os.getenv("BACKFEED_ENABLED", "false").lower() == "true"
app.config["BACKFEED_MAX_AGE_DAYS"] = int(os.getenv("BACKFEED_MAX_AGE_DAYS", "90"))
# Notifications through Apprise (v2.14.0, ADR-074)
app.config["NOTIFY_URLS"] = os.getenv("NOTIFY_URLS", "")
app.config["NOTIFY_EVENTS"] = os.getenv("NOTIFY_EVENTS", "reply,mention,quote,problems")
app.config["NOTIFY_PRIORITY"] = os.getenv("NOTIFY_PRIORITY", "normal")
# Apply overrides if provided
# CSRF protection for admin and consent forms (v2.11.0); on by default.
+6 -3
View File
@@ -842,13 +842,16 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
def _all_categories(prefix: str = "") -> list:
"""Tags in use, most used first (q=category)"""
"""Tags on posts that exist (drafts included), most used first (q=category)"""
from starpunk.database import get_db
# Inner joins: tags left behind by edits or soft-deleted posts don't count
rows = get_db(current_app).execute(
"""
SELECT t.display_name, COUNT(nt.note_id) AS uses
FROM tags t LEFT JOIN note_tags nt ON nt.tag_id = t.id
SELECT t.display_name, COUNT(*) AS uses
FROM tags t
JOIN note_tags nt ON nt.tag_id = t.id
JOIN notes n ON n.id = nt.note_id AND n.deleted_at IS NULL
GROUP BY t.id ORDER BY uses DESC, t.display_name
"""
).fetchall()
+236
View File
@@ -0,0 +1,236 @@
"""
Notifications through Apprise (v2.14.0, ADR-074)
The owner hears about new responses (received webmentions and backfeed
alike) and about crossposting problems, on whatever service the Apprise
URLs in NOTIFY_URLS point at (Gotify, ntfy, Pushover, email, an
apprise-api server…).
Responses are sent by a sweep (`sweep`) in the webmention worker, which
runs in one process at a time: each response with `notified_at` empty is
sent once and marked. Likes, reposts and bookmarks are grouped per note.
Failed sends are retried with backoff; responses still unsent after 24
hours are dropped. Problems (`problem`) are sent at once, without retry.
Nothing here may break validation, crossposting or backfeed: every send is
wrapped, and failures are only logged and shown in the admin.
"""
import html
import logging
import re
import sqlite3
import time
from typing import Dict, List, Optional, Tuple
from urllib.parse import parse_qsl, urlsplit
logger = logging.getLogger(__name__)
RESPONSE_EVENTS = ("reply", "mention", "quote", "like", "repost", "bookmark")
EVENTS = RESPONSE_EVENTS + ("problems",)
DEFAULT_EVENTS = "reply,mention,quote,problems"
GROUPED = ("like", "repost", "bookmark")
DROP_AFTER_HOURS = 24
MAX_BACKOFF = 15 * 60
SEND_TIMEOUT = 15
HEADINGS = {
"reply": "↩ Reply from {who}",
"mention": "💬 Mention from {who}",
"quote": "❝ Quote from {who}",
}
GROUP_HEADINGS = {
"like": ("♥", "like", "likes"),
"repost": ("🔁", "repost", "reposts"),
"bookmark": ("🔖", "bookmark", "bookmarks"),
}
# --- Settings ---------------------------------------------------------------------
def urls(config) -> List[str]:
"""NOTIFY_URLS as a list (space- or comma-separated)"""
return [u for u in re.split(r"[\s,]+", config.get("NOTIFY_URLS") or "") if u]
def events(config) -> set:
"""NOTIFY_EVENTS as a set of known event names"""
chosen = {e.strip().lower() for e in (config.get("NOTIFY_EVENTS") or DEFAULT_EVENTS).split(",")}
return chosen & set(EVENTS)
def enabled(config) -> bool:
return bool(urls(config))
def with_priority(url: str, priority: str) -> str:
"""The URL with `priority=` added, unless it sets one itself"""
if not priority or any(k == "priority" for k, _ in parse_qsl(urlsplit(url).query)):
return url
return url + ("&" if "?" in url else "?") + "priority=" + priority
# --- Sending ----------------------------------------------------------------------------
def _deliver(targets: List[str], title: str, body: str) -> Tuple[bool, str]:
"""Send one message to every URL. Returns (all sent, error text)."""
import apprise
sender = apprise.Apprise()
bad = [u.split("://", 1)[0] + "://…" for u in targets if not sender.add(u)]
if bad:
return False, f"Not a usable Apprise URL: {', '.join(bad)}"
with apprise.LogCapture(level=logging.WARNING, fmt="%(message)s") as log:
result = sender.notify(body, title=title, body_format=apprise.NotifyFormat.MARKDOWN,
timeout=SEND_TIMEOUT)
return bool(result), log.getvalue().strip()[:500]
def send(config, title: str, body: str) -> Tuple[bool, str]:
"""Send to all NOTIFY_URLS at NOTIFY_PRIORITY. Never raises."""
targets = [with_priority(u, config.get("NOTIFY_PRIORITY") or "normal") for u in urls(config)]
if not targets:
return False, "No NOTIFY_URLS set"
try:
ok, error = _deliver(targets, title, body)
except Exception as e: # a broken service mustn't break the caller
ok, error = False, f"{type(e).__name__}: {e}"
if not ok:
logger.warning(f"Notification '{title}' not sent: {error or 'unknown error'}")
return ok, error
def problem(config, db_path, title: str, detail: str, link: str) -> None:
"""A crosspost or backfeed problem, sent at once if 'problems' is selected"""
if not enabled(config) or "problems" not in events(config):
return
ok, error = send(config, f"⚠ {title}", f"{detail}\n\n[Open in StarPunk]({link})")
_record(db_path, ok, error)
def _record(db_path, ok: bool, error: str) -> None:
"""Remember the last failure for the admin page (cleared by a success)"""
try:
conn = sqlite3.connect(db_path)
try:
conn.execute("INSERT INTO site_state (key, value) VALUES ('notify_last_error', ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
("" if ok else time.strftime("%Y-%m-%d %H:%M:%S UTC ", time.gmtime())
+ (error or "unknown error"),))
conn.commit()
finally:
conn.close()
except sqlite3.Error as e:
logger.warning(f"Could not record notification result: {e}")
def last_error(db) -> str:
row = db.execute("SELECT value FROM site_state WHERE key = 'notify_last_error'").fetchone()
return row[0] if row else ""
# --- Messages -----------------------------------------------------------------------------
def _plain(snippet: Optional[str], limit: int = 300) -> str:
text = html.unescape(re.sub(r"<[^>]+>", " ", snippet or ""))
text = re.sub(r"\s+", " ", text).strip()
return text if len(text) <= limit else text[:limit - 1].rsplit(" ", 1)[0] + "…"
def _note_label(conn, note_id: int) -> str:
from starpunk.notes import get_note
try:
note = get_note(id=note_id, load_content=True)
label = " ".join((note.title or "").split()) if note else ""
except Exception:
label = ""
return (label[:57] + "…") if len(label) > 60 else (label or "a note")
def _messages(rows, site_url: str, conn) -> List[Tuple[str, str, List[int]]]:
"""(title, body, webmention ids) for a batch of responses"""
site = site_url.rstrip("/")
names = {"gotosocial": "GoToSocial", "bluesky": "Bluesky", "threads": "Threads"}
messages, groups = [], {}
for row in rows:
via = names.get(row["origin"]) if row["origin"] != "webmention" else None
who = (row["author_name"] or "someone") + (f" via {via}" if via else "")
pending = row["moderation_status"] == "pending"
link = (f"{site}/admin/webmentions/{row['id']}" if pending
else f"{site}/note/{row['slug']}#responses")
if row["mention_type"] in GROUPED:
groups.setdefault((row["target_note_id"], row["mention_type"], row["slug"], pending), []).append(
(row["id"], row["author_name"] or "someone", link))
continue
title = HEADINGS.get(row["mention_type"], "Response from {who}").format(who=who)
if pending:
title += " (awaiting moderation)"
text = _plain(row["content_snippet"])
body = (f"{text}\n\n" if text else "") + f"On “{_note_label(conn, row['target_note_id'])}”: [open]({link})"
messages.append((title, body, [row["id"]]))
for (note_id, kind, slug, pending), items in groups.items():
icon, one, many = GROUP_HEADINGS[kind]
n = len(items)
title = f"{icon} {n} new {one if n == 1 else many} on “{_note_label(conn, note_id)}”"
if pending:
title += " (awaiting moderation)"
people = ", ".join(name for _, name, _ in items[:5]) + (f" and {n - 5} more" if n > 5 else "")
messages.append((title, f"{people}\n\n[open]({items[0][2]})", [i for i, _, _ in items]))
return messages
# --- Sweep ----------------------------------------------------------------------------------
class Sweeper:
"""Sends new responses; one instance per webmention worker"""
def __init__(self):
self._failures = 0
self._next_try = 0.0
def run(self, config, db_path, site_url: str) -> int:
"""Send what's due. Returns how many responses were sent."""
if not enabled(config) or time.time() < self._next_try:
return 0
chosen = events(config)
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
try:
# Too old to be news (an outage, say): drop quietly
dropped = conn.execute(
f"""UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP
WHERE notified_at IS NULL
AND received_at < datetime('now', '-{DROP_AFTER_HOURS} hours')""").rowcount
if dropped:
logger.warning(f"Dropped {dropped} notification(s) older than {DROP_AFTER_HOURS} hours")
rows = conn.execute(
"""
SELECT w.*, n.slug FROM webmentions w JOIN notes n ON n.id = w.target_note_id
WHERE w.notified_at IS NULL AND w.validation_status = 'valid'
AND w.moderation_status IN ('approved', 'pending')
ORDER BY w.id LIMIT 100
"""
).fetchall()
skip = [r["id"] for r in rows if r["mention_type"] not in chosen]
if skip:
conn.execute(f"UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP "
f"WHERE id IN ({','.join('?' * len(skip))})", skip)
conn.commit()
due = [r for r in rows if r["mention_type"] in chosen]
sent = 0
for title, body, ids in _messages(due, site_url, conn):
ok, error = send(config, title, body)
_record(db_path, ok, error)
if not ok:
self._failures += 1
self._next_try = time.time() + min(10 * 2 ** self._failures, MAX_BACKOFF)
break
self._failures = 0
conn.execute(f"UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP "
f"WHERE id IN ({','.join('?' * len(ids))})", ids)
conn.commit()
sent += len(ids)
return sent
finally:
conn.close()
+60 -2
View File
@@ -5,7 +5,7 @@ Handles authenticated admin functionality including dashboard, note creation,
editing, and deletion. All routes require authentication.
"""
from flask import Blueprint, flash, g, jsonify, redirect, render_template, request, url_for
from flask import Blueprint, current_app, flash, g, jsonify, redirect, render_template, request, url_for
import logging
import os
import shutil
@@ -259,6 +259,7 @@ def edit_note_form(note_id: int):
syndication_targets=get_targets().values(),
syndications=syndications,
syndication_enabled=enabled,
backfeed_enabled=current_app.config.get("BACKFEED_ENABLED", False),
)
@@ -496,18 +497,25 @@ def webmentions():
from starpunk.webmention_queries import list_webmentions
status_filter = request.args.get('status')
origin_filter = request.args.get('origin') or None
page = int(request.args.get('page', 1))
webmentions, total_pages = list_webmentions(
status_filter=status_filter,
page=page,
per_page=25
per_page=25,
origin_filter=origin_filter,
)
from starpunk import backfeed
return render_template(
"admin/webmentions.html",
webmentions=webmentions,
status_filter=status_filter,
origin_filter=origin_filter,
backfeed=backfeed.status(),
notify=_notify_status(),
page=page,
total_pages=total_pages,
user_me=g.me
@@ -674,6 +682,56 @@ def revoke_token(token_id):
return redirect(url_for("admin.tokens"))
def _notify_status() -> dict:
"""Notification settings and last error for the Webmentions page (v2.14.0)"""
from starpunk import notifications
from starpunk.database import get_db
return {
"enabled": notifications.enabled(current_app.config),
"events": sorted(notifications.events(current_app.config)),
"priority": current_app.config.get("NOTIFY_PRIORITY") or "normal",
"services": [u.split("://", 1)[0] for u in notifications.urls(current_app.config)],
"last_error": notifications.last_error(get_db(current_app)),
}
@bp.route("/notifications/test", methods=["POST"])
@require_auth
def notifications_test():
"""Send a test notification to every NOTIFY_URLS service (v2.14.0)"""
from starpunk import notifications
ok, error = notifications.send(current_app.config, "StarPunk test notification",
"Notifications from StarPunk are working.")
notifications._record(current_app.config["DATABASE_PATH"], ok, error)
flash("Test notification sent" if ok else f"Test notification failed: {error or 'unknown error'}",
"success" if ok else "error")
return redirect(url_for("admin.webmentions"))
@bp.route("/backfeed/backfill", methods=["POST"])
@require_auth
def backfeed_backfill():
"""Check every crosspost once for responses, whatever its age (v2.13.0)"""
from starpunk import backfeed
count = backfeed.start_backfill()
flash(f"Backfill started: {count} crosspost(s) to check", "success")
return redirect(url_for("admin.webmentions"))
@bp.route("/syndication/<int:syndication_id>/check", methods=["POST"])
@require_auth
def check_syndication_responses(syndication_id):
"""Check one crosspost's responses at the next worker pass (v2.13.0)"""
from starpunk import backfeed
backfeed.check_now(syndication_id)
flash("Responses will be checked shortly", "success")
return redirect(request.referrer or url_for("admin.dashboard"))
@bp.route("/syndication/<int:syndication_id>/retry", methods=["POST"])
@require_auth
def retry_syndication(syndication_id):
+30
View File
@@ -250,6 +250,20 @@ def media_file(path):
return response
@bp.route("/avatars/<name>")
def avatar(name: str):
"""A saved copy of a response author's avatar (v2.13.0, ADR-073)"""
from pathlib import Path
from starpunk.avatars import NAME_RE, folder
if not NAME_RE.match(name):
abort(404)
response = send_from_directory(folder(Path(current_app.config.get('DATA_PATH', 'data'))), name)
# Named by the hash of the original URL; a changed avatar gets a new name
response.headers['Cache-Control'] = 'public, max-age=604800'
return response
@bp.route("/")
def index():
"""
@@ -278,9 +292,23 @@ def index():
object.__setattr__(note, '_cached_tags', tags)
attach_contexts(notes)
attach_response_counts(notes)
return render_template("index.html", notes=notes)
def attach_response_counts(notes) -> None:
"""Give listed notes their like, repost and reply counts (v2.13.4)"""
counts = {}
if current_app.config.get('WEBMENTION_ENABLED', False) and notes:
try:
from starpunk.webmention_queries import response_counts
counts = response_counts([n.id for n in notes])
except Exception as e:
current_app.logger.warning(f"Could not count responses: {e}")
for note in notes:
object.__setattr__(note, 'response_counts', counts.get(note.id))
def attach_contexts(notes) -> None:
"""Give likes and reposts the fetched context of the post they point at"""
from starpunk.webmention_queries import get_reply_context
@@ -306,6 +334,7 @@ def likes():
object.__setattr__(note, 'media', get_note_media(note.id))
object.__setattr__(note, '_cached_tags', get_note_tags(note.id))
attach_contexts(notes)
attach_response_counts(notes)
return render_template("likes.html", notes=notes)
@@ -453,6 +482,7 @@ def tag(tag: str):
media = get_note_media(note.id)
object.__setattr__(note, 'media', media)
attach_contexts(notes)
attach_response_counts(notes)
return render_template(
"tag.html",
+3
View File
@@ -258,7 +258,10 @@ def receive_webmention():
ON CONFLICT(source_url, target_url) DO UPDATE SET
validation_status = 'pending',
updated_at = CURRENT_TIMESTAMP
WHERE webmentions.origin = 'webmention'
""", (source, target, target_note_id))
# (A backfed response with the same URLs is StarPunk's own copy of
# what the silo said; a webmention doesn't overwrite it. v2.13.0)
db.commit()
current_app.logger.info(
+218 -5
View File
@@ -67,8 +67,9 @@ class MastodonClient:
def _call(self, method: str, path: str, **kwargs) -> httpx.Response:
headers = {**self.headers, **kwargs.pop("headers", {})}
url = path if path.startswith(("http://", "https://")) else self.base_url + path
try:
response = self.http.request(method, self.base_url + path, headers=headers, **kwargs)
response = self.http.request(method, url, headers=headers, **kwargs)
except httpx.HTTPError as e:
raise SyndicationError(f"{method} {path}: {e}")
if response.status_code == 404:
@@ -76,10 +77,12 @@ class MastodonClient:
if response.status_code >= 400:
# 4xx other than rate limiting won't fix itself
permanent = 400 <= response.status_code < 500 and response.status_code != 429
raise SyndicationError(
exc = SyndicationError(
f"{method} {path}: HTTP {response.status_code} {response.text[:200]}",
permanent=permanent,
)
exc.status = response.status_code
raise exc
return response
def upload_media(self, path: Path, mime_type: str, description: str) -> str:
@@ -166,6 +169,61 @@ class MastodonClient:
text, media_ids = self._prepare(post)
return self.update_status(remote_id, text, media_ids)
# --- Backfeed (v2.13.0, ADR-073) ---
def _account_id(self) -> str:
"""Id of the account StarPunk posts as (its own responses are skipped)"""
if not getattr(self, "account_id", None):
self.account_id = self._call("GET", "/api/v1/accounts/verify_credentials").json()["id"]
return self.account_id
def _list(self, path: str, cap: int) -> tuple:
"""Items of a paginated list (Link: rel="next"). Returns (items, complete)."""
items, url, params = [], path, {"limit": 80}
while url:
response = self._call("GET", url, params=params)
items.extend(response.json() or [])
if len(items) >= cap:
return items[:cap], False
url, params = (response.links.get("next") or {}).get("url"), None
return items, True
@staticmethod
def _author(account: dict) -> dict:
return {
"author_id": account.get("id") or "",
"author_name": account.get("display_name") or account.get("username") or account.get("acct") or "",
"author_url": account.get("url") or "",
"author_photo": account.get("avatar_static") or account.get("avatar"),
"author_bio": account.get("note") or "",
}
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
"""Likes, boosts and public direct replies of one of our statuses"""
from starpunk.backfeed import Harvest, Response
me = self._account_id()
found, complete = [], True
for kind, path, fragment in (("like", "favourited_by", "liked-by"),
("repost", "reblogged_by", "reposted-by")):
accounts, done = self._list(f"/api/v1/statuses/{remote_id}/{path}", cap)
complete = complete and done
for account in accounts:
if account.get("id") == me:
continue
found.append(Response(kind=kind, source_url=f"{remote_url}#{fragment}-{account['id']}",
remote_id=account["id"], **self._author(account)))
context = self._call("GET", f"/api/v1/statuses/{remote_id}/context").json() or {}
for status in context.get("descendants") or []:
account = status.get("account") or {}
if (status.get("in_reply_to_id") != remote_id or status.get("visibility") != "public"
or account.get("id") == me):
continue
found.append(Response(kind="reply", source_url=status.get("url") or status["uri"],
remote_id=status["id"], content_html=status.get("content") or "",
published=status.get("created_at"), **self._author(account)))
return Harvest(found, complete)
def delete(self, remote_id: str) -> None:
action, _, status_id = remote_id.partition(":")
if action in self.NATIVE and status_id:
@@ -227,9 +285,11 @@ class BlueskyClient:
if response.status_code < 400:
return
permanent = 400 <= response.status_code < 500 and response.status_code != 429
raise SyndicationError(
exc = SyndicationError(
f"{nsid}: HTTP {response.status_code} {response.text[:200]}", permanent=permanent
)
exc.status = response.status_code
raise exc
def _login(self) -> None:
response = None
@@ -405,6 +465,94 @@ class BlueskyClient:
def update(self, remote_id: str, post: "Post") -> dict:
raise NotImplementedError("Bluesky posts can't be edited")
# --- Backfeed (v2.13.0, ADR-073) ---
def _paged(self, nsid: str, params: dict, key: str, cap: int) -> tuple:
"""Items of a cursor-paginated XRPC list. Returns (items, complete)."""
items, cursor = [], None
while True:
query = {**params, "limit": 100}
if cursor:
query["cursor"] = cursor
data = self._call("GET", nsid, params=query).json()
page = data.get(key) or []
items.extend(page)
if len(items) >= cap:
return items[:cap], False
cursor = data.get("cursor")
if not cursor or not page:
return items, True
@staticmethod
def _author(profile: dict) -> dict:
handle = profile.get("handle") or profile.get("did") or ""
return {
"author_id": profile.get("did") or "",
"author_name": profile.get("displayName") or handle,
"author_url": f"https://bsky.app/profile/{handle}",
"author_photo": profile.get("avatar"),
"author_bio": profile.get("description") or "",
# The author asked not to be shown to people who aren't logged in
"opted_out": any(label.get("val") == "!no-unauthenticated"
for label in profile.get("labels") or []),
}
def _post_response(self, kind: str, view: dict):
from starpunk.backfeed import Response, facets_html
author = view.get("author") or {}
record = view.get("record") or {}
rkey = view["uri"].rsplit("/", 1)[1]
return Response(
kind=kind,
source_url=f"https://bsky.app/profile/{author.get('handle') or author.get('did')}/post/{rkey}",
remote_id=view["uri"],
content_html=facets_html(record.get("text") or "", record.get("facets") or []),
published=record.get("createdAt") or view.get("indexedAt"),
**self._author(author),
)
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
"""Likes, reposts, quotes and direct replies of one of our posts"""
from starpunk.backfeed import Harvest, Response
if not self.did:
with self._lock:
self._login()
found, complete = [], True
likes, done = self._paged("app.bsky.feed.getLikes", {"uri": remote_id}, "likes", cap)
complete = complete and done
for like in likes:
actor = like.get("actor") or {}
found.append(Response(kind="like", source_url=f"{remote_url}#liked-by-{actor.get('did')}",
remote_id=actor.get("did") or "", published=like.get("createdAt"),
**self._author(actor)))
reposters, done = self._paged("app.bsky.feed.getRepostedBy", {"uri": remote_id},
"repostedBy", cap)
complete = complete and done
for actor in reposters:
found.append(Response(kind="repost", source_url=f"{remote_url}#reposted-by-{actor.get('did')}",
remote_id=actor.get("did") or "", **self._author(actor)))
quotes, done = self._paged("app.bsky.feed.getQuotes", {"uri": remote_id}, "posts", cap)
complete = complete and done
posts = [self._post_response("quote", view) for view in quotes]
thread = self._call("GET", "app.bsky.feed.getPostThread",
params={"uri": remote_id, "depth": 1, "parentHeight": 0}).json()
for item in (thread.get("thread") or {}).get("replies") or []:
if item.get("post"): # skips blocked and deleted replies
posts.append(self._post_response("reply", item["post"]))
# Post views carry no bio; fetch them to honour #nobridge
dids = list({p.author_id for p in posts if p.author_id})
bios = {}
for i in range(0, len(dids), 25):
profiles = self._call("GET", "app.bsky.actor.getProfiles",
params={"actors": dids[i:i + 25]}).json().get("profiles") or []
bios.update({p["did"]: p.get("description") or "" for p in profiles})
for post in posts:
post.author_bio = bios.get(post.author_id, "")
found.extend(posts)
return Harvest([r for r in found if r.author_id != self.did], complete)
def delete(self, remote_id: str) -> None:
# at://<did>/app.bsky.feed.post/<rkey>; deleting a missing record succeeds
repo, collection, rkey = remote_id[len("at://"):].split("/", 2)
@@ -513,7 +661,7 @@ def page_card(http: httpx.Client, url: str) -> dict:
return card
THREADS_API = "https://graph.threads.net/v1.0"
THREADS_API = "https://graph.threads.com/v1.0" # Meta's current host (v2.13.0)
THREADS_IMAGE_TYPES = {"image/jpeg", "image/png"}
THREADS_IMAGE_LIMIT = 8_000_000
THREADS_MAX_LINKS = 5
@@ -775,6 +923,57 @@ class ThreadsClient:
def update(self, remote_id: str, post: "Post") -> dict:
raise NotImplementedError("Threads posts can't be edited through the API")
# --- Backfeed (v2.13.0, ADR-073) ---
REPLY_FIELDS = "id,text,username,permalink,timestamp,profile_picture_url,hide_status"
SHOWN = (None, "NOT_HUSHED", "UNHUSHED") # hide_status values of replies nobody hid
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
"""
Direct replies to one of our posts, and its like/repost/quote counts.
The API doesn't say who liked, reposted or quoted (ADR-073 Decision 7).
"""
from starpunk.backfeed import Harvest, Response, text_html
self._me()
replies, after, complete = [], None, True
while True:
params = {"fields": self.REPLY_FIELDS}
if after:
params["after"] = after
data = self._call("GET", f"{remote_id}/replies", **params)
page = data.get("data") or []
replies.extend(page)
if len(replies) >= cap:
replies, complete = replies[:cap], False
break
paging = data.get("paging") or {}
after = (paging.get("cursors") or {}).get("after")
if not page or not after or not paging.get("next"):
break
found = []
for reply in replies:
username = reply.get("username")
# Private accounts have no username or permalink: nothing public to show
if (not username or not reply.get("permalink") or username == self.username
or reply.get("hide_status") not in self.SHOWN):
continue
found.append(Response(
kind="reply", source_url=reply["permalink"], remote_id=reply["id"],
author_id=username, author_name=username,
author_url=f"https://www.threads.com/@{username}",
author_photo=reply.get("profile_picture_url"),
content_html=text_html(reply.get("text") or ""),
published=reply.get("timestamp"),
))
insights = self._call("GET", f"{remote_id}/insights", metric="likes,reposts,quotes")
counts = {}
for metric in insights.get("data") or []:
values = metric.get("values") or [{}]
counts[metric.get("name")] = int((values[0] or {}).get("value") or 0)
return Harvest(found, complete, counts={k: counts.get(k, 0)
for k in ("likes", "reposts", "quotes")})
def delete(self, remote_id: str) -> None:
try:
self._call("DELETE", remote_id)
@@ -1265,6 +1464,13 @@ def _fail(db, row, error: SyndicationError) -> None:
f"Syndication {row['target']} for note {row['note_id']} failed "
f"(attempt {attempts}{', giving up' if failed else ''}): {error}"
)
if failed: # ADR-074: tell the owner, once, when we give up
from starpunk import notifications
target = get_targets().get(row["target"])
site = current_app.config["SITE_URL"].rstrip("/")
notifications.problem(current_app.config, current_app.config["DATABASE_PATH"],
f"Crosspost to {target.name if target else row['target']} failed",
str(error)[:300], f"{site}/admin/edit/{row['note_id']}")
def _target_title(url: Optional[str]) -> Optional[str]:
@@ -1381,12 +1587,19 @@ class SyndicationWorker(threading.Thread):
def run(self):
while not self._stop_event.is_set():
handled = 0
try:
with self.app.app_context():
handled = process_due()
except Exception as e:
self.app.logger.error(f"Syndication worker error: {e}", exc_info=True)
handled = 0
# Responses to the copies (v2.13.0, ADR-073), after any crossposts
try:
from starpunk import backfeed
with self.app.app_context():
handled += backfeed.process_due()
except Exception as e:
self.app.logger.error(f"Backfeed error: {e}", exc_info=True)
if handled == 0:
self._stop_event.wait(self.poll_interval)
+75 -10
View File
@@ -39,7 +39,8 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
note_id: Note database ID
Returns:
Dict with keys: 'replies', 'likes', 'reposts', 'bookmarks', 'mentions'
Dict with keys: 'replies', 'likes', 'reposts', 'bookmarks', 'quotes'
(backfed quote posts, v2.13.0), 'mentions'
Each value is a list of webmention dicts
"""
db = get_db()
@@ -56,24 +57,34 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
WHEN 'like' THEN 2
WHEN 'repost' THEN 3
WHEN 'bookmark' THEN 4
WHEN 'mention' THEN 5
WHEN 'quote' THEN 5
WHEN 'mention' THEN 6
END,
received_at ASC
"""
rows = db.execute(query, (note_id,)).fetchall()
# Backfed responses say which silo they came from (v2.13.0)
silo_names = {}
if any(row['origin'] != 'webmention' for row in rows):
from starpunk.backfeed import SILO_NAMES
from starpunk.syndication import get_targets
silo_names = {**SILO_NAMES, **{t.id: t.name for t in get_targets().values()}}
# Group by type
grouped = {
'replies': [],
'likes': [],
'reposts': [],
'bookmarks': [],
'quotes': [],
'mentions': []
}
for row in rows:
wm = dict(row)
wm['via'] = silo_names.get(wm['origin']) if wm['origin'] != 'webmention' else None
# Parse timestamps (source_published may be missing or malformed)
for key in ('received_at', 'source_published'):
if isinstance(wm.get(key), str):
@@ -91,24 +102,70 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
grouped['reposts'].append(wm)
elif mention_type == 'bookmark':
grouped['bookmarks'].append(wm)
elif mention_type == 'quote':
grouped['quotes'].append(wm)
else:
grouped['mentions'].append(wm)
return grouped
def response_counts(note_ids: List[int]) -> Dict[int, Dict[str, int]]:
"""
Likes, reposts and replies per note, for lists of notes (v2.13.4).
Shown responses (valid and approved) plus the counts silos give without
names (Threads; people who opted out of backfeed) on synced crossposts.
Notes with none are left out.
"""
if not note_ids:
return {}
db = get_db()
marks = ','.join('?' * len(note_ids))
counts: Dict[int, Dict[str, int]] = {}
def add(note_id, key, n):
if n:
counts.setdefault(note_id, {'likes': 0, 'reposts': 0, 'replies': 0})[key] += n
for row in db.execute(
f"""
SELECT target_note_id, mention_type, COUNT(*) FROM webmentions
WHERE target_note_id IN ({marks}) AND validation_status = 'valid'
AND moderation_status = 'approved' AND mention_type IN ('like', 'repost', 'reply')
GROUP BY target_note_id, mention_type
""",
note_ids,
):
add(row[0], {'like': 'likes', 'repost': 'reposts', 'reply': 'replies'}[row[1]], row[2])
for row in db.execute(
f"""
SELECT note_id, SUM(COALESCE(likes_count, 0)), SUM(COALESCE(reposts_count, 0))
FROM syndications WHERE note_id IN ({marks}) AND state = 'synced'
GROUP BY note_id
""",
note_ids,
):
add(row[0], 'likes', row[1])
add(row[0], 'reposts', row[2])
return counts
def list_webmentions(
status_filter: Optional[str] = None,
page: int = 1,
per_page: int = 25
per_page: int = 25,
origin_filter: Optional[str] = None,
) -> tuple[List[Dict[str, Any]], int]:
"""
List webmentions for admin with pagination and filtering.
Args:
status_filter: Filter by moderation_status ('pending', 'approved', 'hidden')
status_filter: Filter by moderation_status ('pending', 'approved', 'hidden',
'deleted'). Deleted ones are left out unless asked for (v2.13.0).
page: Page number (1-indexed)
per_page: Items per page (default: 25)
origin_filter: 'webmention' or a backfeed silo id (v2.13.0)
Returns:
Tuple of (webmentions list, total_pages)
@@ -124,12 +181,16 @@ def list_webmentions(
count_query = "SELECT COUNT(*) FROM webmentions"
params = []
where_clause = ""
conditions = ["moderation_status = ?" if status_filter else "moderation_status != 'deleted'"]
if status_filter:
where_clause = " WHERE w.moderation_status = ?"
count_query += " WHERE moderation_status = ?"
params.append(status_filter)
if origin_filter:
conditions.append("origin = ?")
params.append(origin_filter)
where = " AND ".join(conditions)
where_clause = " WHERE " + where.replace("moderation_status", "w.moderation_status").replace(
"origin", "w.origin")
count_query += " WHERE " + where
# Count total
total_count = db.execute(count_query, params).fetchone()[0]
@@ -206,7 +267,7 @@ def count_webmentions(status_filter: Optional[str] = None) -> int:
query = "SELECT COUNT(*) FROM webmentions WHERE moderation_status = ?"
result = db.execute(query, (status_filter,)).fetchone()
else:
query = "SELECT COUNT(*) FROM webmentions"
query = "SELECT COUNT(*) FROM webmentions WHERE moderation_status != 'deleted'"
result = db.execute(query).fetchone()
return result[0] if result else 0
@@ -260,6 +321,9 @@ def delete_webmentions(webmention_ids: List[int]) -> int:
"""
Delete webmentions by IDs.
The row stays as a tombstone (moderation_status 'deleted', v2.13.0), so
a re-sent webmention or the next backfeed check doesn't bring it back.
Args:
webmention_ids: List of webmention IDs to delete
@@ -276,7 +340,8 @@ def delete_webmentions(webmention_ids: List[int]) -> int:
)
placeholders = ','.join('?' * len(webmention_ids))
query = f"DELETE FROM webmentions WHERE id IN ({placeholders})"
query = (f"UPDATE webmentions SET moderation_status = 'deleted', updated_at = CURRENT_TIMESTAMP "
f"WHERE id IN ({placeholders}) AND moderation_status != 'deleted'")
cursor = db.execute(query, webmention_ids)
db.commit()
+39 -1
View File
@@ -82,6 +82,10 @@ class WebmentionWorker(threading.Thread):
self._stop_event = threading.Event()
self._last_processed = None
self._lock_file = None
from starpunk.avatars import Upkeep
from starpunk.notifications import Sweeper
self._avatar_upkeep = Upkeep()
self._notifier = Sweeper()
def run(self):
"""
@@ -113,6 +117,14 @@ class WebmentionWorker(threading.Thread):
# Phase 3: Process outgoing webmentions
processed += self._process_pending_sends()
# Local copies of response avatars (v2.13.0)
self._avatar_upkeep.run(self.db_path, self.app.config['DATA_PATH'],
self.site_url)
# New responses to the owner (v2.14.0, ADR-074)
try:
self._notifier.run(self.app.config, self.db_path, self.site_url)
except Exception as e:
logger.error(f"Notification sweep failed: {e}", exc_info=True)
# Update last processed time
if processed > 0:
@@ -281,6 +293,22 @@ class WebmentionWorker(threading.Thread):
source_url, result.get('source_post_url'), result.get('source_uid')) if u}
return bool(copies & source)
def _is_backfed(self, note_id: Optional[int], source_url: str, result: Dict[str, Any]) -> bool:
"""True if the source is a response already stored by backfeed for this note"""
if note_id is None:
return False
conn = sqlite3.connect(self.db_path)
try:
backfed = {normalize_url(row[0]) for row in conn.execute(
"SELECT source_url FROM webmentions WHERE target_note_id = ? AND origin != 'webmention'",
(note_id,),
)}
finally:
conn.close()
source = {normalize_url(u) for u in (
source_url, result.get('source_post_url'), result.get('source_uid')) if u}
return bool(backfed & source)
def _display_state(self, webmention_id: int) -> Optional[tuple]:
"""
What the note page shows for this webmention, or None if hidden
@@ -328,6 +356,11 @@ class WebmentionWorker(threading.Thread):
if self._is_own_syndicated_copy(target_note_id, source_url, result):
raise ValidationError("Source is this note's own syndicated copy")
# A response StarPunk already backfed from the silo (e.g. a reply
# also sent by Bridgy) would show twice (v2.13.0, ADR-073)
if self._is_backfed(target_note_id, source_url, result):
raise ValidationError("Already backfed from the silo")
# Update database with successful validation
conn = sqlite3.connect(self.db_path)
cursor = conn.cursor()
@@ -372,6 +405,10 @@ class WebmentionWorker(threading.Thread):
conn.commit()
conn.close()
# Pages show a local copy of the avatar (v2.13.0, ADR-073)
from starpunk import avatars
avatars.fetch(self.app.config['DATA_PATH'], result['author_photo'], self.site_url)
logger.info(
f"Validated webmention: {source_url} -> {target_url} "
f"(type={result['mention_type']})"
@@ -422,7 +459,8 @@ class WebmentionWorker(threading.Thread):
# All other errors are 'error' status
if transient:
status = 'error'
elif any(x in error.lower() for x in ['does not link', 'returned 4', 'own syndicated copy']):
elif any(x in error.lower() for x in ['does not link', 'returned 4', 'own syndicated copy',
'already backfed']):
status = 'invalid'
else:
status = 'error'
+42 -47
View File
@@ -343,57 +343,52 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
color: var(--color-text-light);
}
/* Facepile */
.facepile {
display: flex;
align-items: center;
flex-wrap: wrap;
}
.facepile .face {
display: block;
width: 32px;
height: 32px;
border-radius: 50%;
overflow: hidden;
border: 2px solid var(--color-bg);
margin-left: -8px;
}
.facepile .face:first-child {
margin-left: 0;
}
.facepile .face > .h-card,
.facepile .face > .h-card > a {
display: block;
width: 100%;
height: 100%;
}
.facepile .face img {
width: 100%;
height: 100%;
object-fit: cover;
}
.facepile .face-initial {
display: flex;
align-items: center;
justify-content: center;
width: 100%;
height: 100%;
font-size: 14px;
font-weight: bold;
color: white;
}
.facepile .face-overflow {
margin-left: var(--spacing-xs);
/* Backfed responses (v2.13.0) */
.webmention-via {
font-size: 0.875rem;
color: var(--color-text-light);
}
/* Likes, reposts and bookmarks: counts with a hover list (v2.13.2) */
.response-counts { display: flex; flex-wrap: wrap; gap: var(--spacing-md); margin-top: var(--spacing-sm); color: var(--color-text-light); font-size: 0.875rem; }
.response-count { position: relative; display: inline-flex; align-items: center; gap: 0.3rem; padding-bottom: 0.4rem; cursor: default; }
.response-count > svg { width: 1rem; height: 1rem; flex: none; }
.response-count:hover, .response-count:focus-within { color: var(--color-text); }
.response-count-like:hover > svg, .response-count-like:focus-within > svg { color: #e0245e; }
.response-count-repost:hover > svg, .response-count-repost:focus-within > svg { color: #17bf63; }
.response-card {
display: none; position: absolute; left: 0; top: 100%; z-index: 10;
flex-direction: column; gap: 0.4rem; min-width: 13rem; max-width: 20rem;
padding: var(--spacing-sm); background: var(--color-bg-alt); border: 1px solid var(--color-border);
border-radius: var(--border-radius); box-shadow: 0 6px 20px rgba(0, 0, 0, 0.25);
}
.response-count:hover .response-card, .response-count:focus-within .response-card { display: flex; }
.response-card-title { font-weight: 600; color: var(--color-text); }
.response-people { display: flex; flex-direction: column; gap: 0.35rem; max-height: 14rem; overflow-y: auto; }
.response-person { display: flex; align-items: center; gap: 0.4rem; min-width: 0; line-height: 1.3; }
.response-person .h-card { display: flex; min-width: 0; }
.response-person .h-card a { display: inline-flex; align-items: center; gap: 0.4rem; color: var(--color-text); min-width: 0; }
.response-person .p-name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.response-person img { width: 20px; height: 20px; border-radius: 50%; object-fit: cover; flex: none; }
.response-person .webmention-via { margin-left: auto; padding-left: 0.5rem; white-space: nowrap; }
.response-anon { color: var(--color-text-light); }
.response-elsewhere { display: block; margin-top: 0.15rem; padding-top: 0.4rem; border-top: 1px solid var(--color-border); color: var(--color-text-light); }
.response-elsewhere:first-child { margin-top: 0; padding-top: 0; border-top: 0; }
.response-elsewhere a { color: var(--color-link); }
/* A reply card opens the reply where it was posted (v2.13.4): its date link
stretches over the card; the author link and links in the text stay on top */
.webmention-reply { position: relative; }
.webmention-reply .webmention-date::after { content: ""; position: absolute; inset: 0; }
.webmention-reply .webmention-author a, .webmention-reply .e-content a { position: relative; z-index: 1; }
.webmention-reply:hover { cursor: pointer; }
/* Counts in lists of notes (v2.13.4) */
.list-counts { display: inline-flex; gap: 0.6rem; color: var(--color-text-light); }
.note-meta a.list-counts:hover { color: var(--color-text); text-decoration: none; }
.list-count { display: inline-flex; align-items: center; gap: 0.2rem; }
.list-count > svg { width: 0.85rem; height: 0.85rem; flex: none; }
/* Generic Mentions */
.webmention-mention {
margin-bottom: var(--spacing-sm);
+6
View File
@@ -102,6 +102,12 @@
<button type="submit" class="button button-secondary">Retry now</button>
</form>
{% endif %}
{% if backfeed_enabled and s.state == 'synced' and s.remote_id %}
<br><small>Responses {% if s.backfeed_checked_at %}checked {{ s.backfeed_checked_at }} UTC{% else %}not checked yet{% endif %}{% if s.backfeed_error %}: {{ s.backfeed_error }}{% endif %}</small>
<form action="{{ url_for('admin.check_syndication_responses', syndication_id=s.id) }}" method="POST" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button button-secondary">Check now</button>
</form>
{% endif %}
</li>
{% endfor %}
</ul>
+4 -1
View File
@@ -16,6 +16,7 @@
<div class="detail-row">
<strong>Source:</strong>
<a href="{{ webmention.source_url }}" target="_blank">{{ webmention.source_url }}</a>
{% if webmention.origin and webmention.origin != 'webmention' %}<small>(backfed from {{ webmention.origin }})</small>{% endif %}
</div>
<div class="detail-row">
@@ -43,7 +44,9 @@
{% if webmention.author_photo %}
<div class="detail-row">
<strong>Author Photo:</strong>
<img src="{{ webmention.author_photo }}" alt="{{ webmention.author_name or 'Author' }}" width="48" height="48" loading="lazy">
{% set photo = webmention.author_photo|local_avatar %}
{% if photo %}<img src="{{ photo }}" alt="{{ webmention.author_name or 'Author' }}" width="48" height="48" loading="lazy">{% endif %}
<a href="{{ webmention.author_photo }}" rel="noopener" target="_blank">{{ webmention.author_photo|truncate(60) }}</a>
</div>
{% endif %}
+47 -2
View File
@@ -12,6 +12,43 @@
</a>
</div>
{# Notifications (v2.14.0, ADR-074) #}
{% if notify %}
<section class="notify-status">
<h3>Notifications</h3>
{% if notify.enabled %}
<p>Sending {{ notify.events|join(', ') }} to {{ notify.services|join(', ') }} at {{ notify.priority }} priority.</p>
{% if notify.last_error %}<p class="flash flash-error">Last send failed: {{ notify.last_error }}</p>{% endif %}
<form method="POST" action="{{ url_for('admin.notifications_test') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button">Send test notification</button>
</form>
{% else %}
<p>Off. Set <code>NOTIFY_URLS</code> to one or more Apprise URLs to be notified of new responses.</p>
{% endif %}
</section>
{% endif %}
{# Backfeed (v2.13.0, ADR-073) #}
{% if backfeed and backfeed.enabled %}
<section class="backfeed-status">
<h3>Backfeed</h3>
{% for silo, message in backfeed.denied.items() %}
<p class="flash flash-error">{{ silo }}: {{ message }}</p>
{% endfor %}
{% if backfeed.remaining %}
<p>Backfill started {{ backfeed.started }} UTC: {{ backfeed.remaining }} crosspost(s) left to check.</p>
{% elif backfeed.finished %}
<p>Last backfill: started {{ backfeed.started }}, finished {{ backfeed.finished }} UTC.</p>
{% else %}
<p>No backfill has been run. It checks every crosspost once for responses, however old the note.</p>
{% endif %}
{% if backfeed.errors %}<p>{{ backfeed.errors }} crosspost(s) had an error at their last check (see each note's edit page).</p>{% endif %}
<form method="POST" action="{{ url_for('admin.backfeed_backfill') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button"{% if backfeed.remaining %} disabled{% endif %}>Backfill responses</button>
</form>
</section>
{% endif %}
{# Filter tabs #}
<div class="filter-tabs">
<a href="{{ url_for('admin.webmentions') }}" class="{% if not status_filter %}active{% endif %}">
@@ -26,6 +63,14 @@
<a href="{{ url_for('admin.webmentions', status='hidden') }}" class="{% if status_filter == 'hidden' %}active{% endif %}">
Hidden
</a>
<a href="{{ url_for('admin.webmentions', status='deleted') }}" class="{% if status_filter == 'deleted' %}active{% endif %}">
Deleted
</a>
</div>
<div class="filter-tabs">
{% for value, label in [(None, 'Any source'), ('webmention', 'Webmentions'), ('gotosocial', 'GoToSocial'), ('bluesky', 'Bluesky'), ('threads', 'Threads')] %}
<a href="{{ url_for('admin.webmentions', status=status_filter, origin=value) }}" class="{% if origin_filter == value %}active{% endif %}">{{ label }}</a>
{% endfor %}
</div>
{% if webmentions %}
@@ -93,11 +138,11 @@
{% if total_pages > 1 %}
<nav class="pagination">
{% if page > 1 %}
<a href="?page={{ page - 1 }}{% if status_filter %}&status={{ status_filter }}{% endif %}" class="button">Previous</a>
<a href="?page={{ page - 1 }}{% if status_filter %}&status={{ status_filter }}{% endif %}{% if origin_filter %}&origin={{ origin_filter }}{% endif %}" class="button">Previous</a>
{% endif %}
<span>Page {{ page }} of {{ total_pages }}</span>
{% if page < total_pages %}
<a href="?page={{ page + 1 }}{% if status_filter %}&status={{ status_filter }}{% endif %}" class="button">Next</a>
<a href="?page={{ page + 1 }}{% if status_filter %}&status={{ status_filter }}{% endif %}{% if origin_filter %}&origin={{ origin_filter }}{% endif %}" class="button">Next</a>
{% endif %}
</nav>
{% endif %}
+10 -20
View File
@@ -78,12 +78,16 @@
{% endif %}
</footer>
<div id="responses"></div>
{{ responses.reactions(likes, reposts, replies, bookmarks, syndications, absolute_url_for('public.note', slug=note.slug)) }}
<nav class="note-nav">
<a href="/">← All notes</a>
</nav>
{# Webmentions section (v1.6.0 Phase 5) - only shown if approved webmentions exist #}
{% if webmentions %}
{# Webmentions section (v1.6.0 Phase 5): replies, quotes and mentions.
Likes, reposts and bookmarks are the counts line above (v2.13.2). #}
{% if replies or quotes or mentions %}
<section class="webmentions">
<h2>Responses</h2>
@@ -94,24 +98,10 @@
</div>
{% endif %}
{% if likes %}
<div class="webmention-group webmention-likes">
<h3>{{ likes|length }} {{ 'Like' if likes|length == 1 else 'Likes' }}</h3>
{{ responses.facepile(likes, 'u-like') }}
</div>
{% endif %}
{% if reposts %}
<div class="webmention-group webmention-reposts">
<h3>{{ reposts|length }} {{ 'Repost' if reposts|length == 1 else 'Reposts' }}</h3>
{{ responses.facepile(reposts, 'u-repost') }}
</div>
{% endif %}
{% if bookmarks %}
<div class="webmention-group webmention-bookmarks">
<h3>{{ bookmarks|length }} {{ 'Bookmark' if bookmarks|length == 1 else 'Bookmarks' }}</h3>
{{ responses.facepile(bookmarks, 'u-bookmark') }}
{% if quotes %}
<div class="webmention-group webmention-quotes">
<h3>{{ quotes|length }} {{ 'Quote' if quotes|length == 1 else 'Quotes' }}</h3>
{% for wm in quotes %}{{ responses.mention(wm) }}{% endfor %}
</div>
{% endif %}
+2
View File
@@ -1,6 +1,7 @@
{# One note in a list (home, tag pages). v2.3.0 #}
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% from "partials/responses.html" import list_counts %}
{% macro note_preview(note, author) %}
<article class="h-entry note-preview">
{# Notes starting with a '# Heading' have a title (Q22; Note.has_title) #}
@@ -37,6 +38,7 @@
{% for tag in note.tags %}<a class="p-category" rel="tag" href="{{ url_for('public.tag', tag=tag.name) }}">{{ tag.display_name }}</a>{% endfor %}
</span>
{% endif %}
{{ list_counts(note.response_counts if note.response_counts is defined else None, absolute_url_for('public.note', slug=note.slug)) }}
{% if author %}
{# Single-author site: keep the h-card for parsers, hide it from readers #}
<span class="p-author h-card visually-hidden"><a class="p-name u-url" href="{{ author.url or author.me }}">{{ author.name or author.url or author.me }}</a></span>
+86 -22
View File
@@ -12,11 +12,18 @@
{# The response's own URL: the source's u-url if it gave one #}
{% macro response_url(wm) %}{{ wm.source_post_url or wm.source_url }}{% endmacro %}
{# Where a backfed response came from, e.g. "via Bluesky" (v2.13.0) #}
{% macro via(wm) %}{% if wm.via %}<span class="webmention-via">via {{ wm.via }}</span>{% endif %}{% endmacro %}
{# Author h-card: name, URL and photo. u-url is on an inner link: on the
h-card element itself it would belong to the enclosing h-cite #}
h-card element itself it would belong to the enclosing h-cite.
The photo is this site's saved copy (v2.13.0); until there is one, the
initial is shown instead of loading it from the other site. #}
{% macro author_card(wm, class='', photo_size=48, show_name=True) %}
<span class="p-author h-card {{ class }}"><a class="u-url" href="{{ wm.author_url or wm.source_url }}" title="{{ wm.author_name or 'Anonymous' }}">
{%- if wm.author_photo %}<img class="u-photo" src="{{ wm.author_photo }}" alt="" width="{{ photo_size }}" height="{{ photo_size }}" loading="lazy">
{% set photo = wm.author_photo|local_avatar %}
<span class="p-author h-card {{ class }}"><a class="u-url" href="{{ wm.author_url or wm.source_url }}" title="{{ wm.author_name or 'Anonymous' }}{% if wm.via %} via {{ wm.via }}{% endif %}">
{%- if photo %}<img class="u-photo" src="{{ photo }}" alt="" width="{{ photo_size }}" height="{{ photo_size }}" loading="lazy">
{%- elif wm.author_photo %}<data class="u-photo" value="{{ wm.author_photo }}"></data>{% if not show_name %}<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}" aria-hidden="true">{{ (wm.author_name or '?')[0]|upper }}</span>{% endif %}
{%- elif not show_name %}<span class="face-initial" style="background-color: {{ wm.author_url|url_to_color }}" aria-hidden="true">{{ (wm.author_name or '?')[0]|upper }}</span>
{%- endif %}
<span class="p-name{{ '' if show_name else ' visually-hidden' }}">{{ wm.author_name or 'Anonymous' }}</span>
@@ -30,33 +37,90 @@
{% if wm.content_snippet %}
<div class="e-content">{{ wm.content_snippet|safe }}</div>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a> {{ via(wm) }}
</article>
{%- endmacro %}
{# Likes, reposts or bookmarks as a facepile; prop is u-like / u-repost / u-bookmark.
Faces past `shown` are hidden on screen but kept for parsers. #}
{% macro facepile(items, prop, shown=8) %}
<div class="facepile">
{% for wm in items %}
<span class="face {{ prop }} h-cite"{% if loop.index > shown %} hidden{% endif %}>
{{ author_card(wm, photo_size=32, show_name=False) }}
<a class="u-url" href="{{ response_url(wm) }}" hidden>{{ response_time(wm) }}</a>
</span>
{% endfor %}
{% if items|length > shown %}
<span class="face-overflow">+{{ items|length - shown }} others</span>
{% endif %}
</div>
{%- endmacro %}
{# A post that mentions this note without replying #}
{# A post that mentions this note without replying; also quote posts (v2.13.0) #}
{% macro mention(wm) %}
<div class="webmention webmention-mention u-mention h-cite">
{{ author_card(wm, photo_size=24) }}
{% if wm.content_snippet %}
<p class="p-content">{{ wm.content_snippet|striptags|truncate(200) }}</p>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a>
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a> {{ via(wm) }}
</div>
{%- endmacro %}
{# Likes, reposts, replies and bookmarks: one line of counts under the note,
like other sites (v2.13.2). Hovering or focusing a count lists who, and
offers the same action on the note's crossposts. Each action is an
<indie-action> (https://indieweb.org/webactions) with those silo links as
its fallback, so web action handlers can take over. Counts without names
(all of Threads; elsewhere people who opted out, v2.13.3) are added to the
total. The people stay in the
markup as u-like / u-repost / u-bookmark h-cites. #}
{% set ICONS = {
'like': '<svg viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M12 21.35l-1.45-1.32C5.4 15.36 2 12.28 2 8.5 2 5.42 4.42 3 7.5 3c1.74 0 3.41.81 4.5 2.09C13.09 3.81 14.76 3 16.5 3 19.58 3 22 5.42 22 8.5c0 3.78-3.4 6.86-8.55 11.54L12 21.35z"/></svg>',
'repost': '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M17 1l4 4-4 4"/><path d="M3 11V9a4 4 0 0 1 4-4h14"/><path d="M7 23l-4-4 4-4"/><path d="M21 13v2a4 4 0 0 1-4 4H3"/></svg>',
'reply': '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M21 11.5a8.4 8.4 0 0 1-9 8.4 8.6 8.6 0 0 1-3.9-.9L3 21l1.9-5.1A8.4 8.4 0 0 1 12 3.1a8.4 8.4 0 0 1 9 8.4z"/></svg>',
'bookmark': '<svg viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M6 2h12a1 1 0 0 1 1 1v19l-7-4-7 4V3a1 1 0 0 1 1-1z"/></svg>',
} %}
{% set VERBS = {'like': 'Like', 'repost': 'Repost', 'reply': 'Reply'} %}
{% macro reaction(kind, items, prop, anon, one, many, copies=[], permalink='', people=True) %}
{% set ns = namespace(total=items|length) %}
{% for name, n in anon %}{% set ns.total = ns.total + n %}{% endfor %}
{% set verb = VERBS.get(kind) %}
{% if ns.total or (verb and copies) %}
<span class="response-count response-count-{{ kind }}" tabindex="0">
{{ ICONS[kind]|safe }}{% if ns.total %}<span class="response-count-n">{{ ns.total }}</span>{% endif %}<span class="visually-hidden"> {{ ns.total }} {{ one if ns.total == 1 else many }}</span>
<span class="response-card" role="tooltip">
{% if ns.total %}<span class="response-card-title">{{ ns.total }} {{ one if ns.total == 1 else many }}</span>{% endif %}
{% if (people and items) or anon %}
<span class="response-people">
{% if people %}
{% for wm in items %}
<span class="response-person {{ prop }} h-cite">{{ author_card(wm, photo_size=20) }}{% if wm.via %}<span class="webmention-via">{{ wm.via }}</span>{% endif %}<a class="u-url" href="{{ response_url(wm) }}" hidden>{{ response_time(wm) }}</a></span>
{% endfor %}
{% endif %}
{% for name, n in anon %}<span class="response-anon">{{ '+' if items and people else '' }}{{ n }} on {{ name }}</span>{% endfor %}
</span>
{% endif %}
{% if verb and copies %}
<indie-action class="response-elsewhere" do="{{ kind }}" with="{{ permalink }}">{{ verb }} on {% for s in copies %}<a href="{{ s.remote_url }}" rel="noopener">{{ s.name }}</a>{% if not loop.last %} · {% endif %}{% endfor %}</indie-action>
{% endif %}
</span>
</span>
{% endif %}
{%- endmacro %}
{% macro reactions(likes, reposts, replies, bookmarks, syndications, permalink) %}
{% set anon_likes, anon_reposts = [], [] %}
{% for s in syndications or [] %}
{% if s.likes_count %}{% set _ = anon_likes.append((s.name, s.likes_count)) %}{% endif %}
{% if s.reposts_count %}{% set _ = anon_reposts.append((s.name, s.reposts_count)) %}{% endif %}
{% endfor %}
{% set copies = (syndications or [])|selectattr('remote_url')|list %}
{% if copies or likes or reposts or replies or bookmarks or anon_likes or anon_reposts %}
<div class="response-counts">
{{ reaction('like', likes or [], 'u-like', anon_likes, 'like', 'likes', copies, permalink) }}
{{ reaction('repost', reposts or [], 'u-repost', anon_reposts, 'repost', 'reposts', copies, permalink) }}
{# Replies are listed in full below; here just the count and where to reply #}
{{ reaction('reply', replies or [], '', [], 'reply', 'replies', copies, permalink, people=False) }}
{{ reaction('bookmark', bookmarks or [], 'u-bookmark', [], 'bookmark', 'bookmarks') }}
</div>
{% endif %}
{%- endmacro %}
{# Compact like / repost / reply counts for lists of notes (v2.13.4); links
to the note's responses. `counts` is from webmention_queries.response_counts. #}
{% macro list_counts(counts, permalink) %}
{% if counts %}
<a class="list-counts" href="{{ permalink }}#responses">
{%- for key, kind, one, many in [('likes', 'like', 'like', 'likes'), ('reposts', 'repost', 'repost', 'reposts'), ('replies', 'reply', 'reply', 'replies')] if counts[key] -%}
<span class="list-count list-count-{{ kind }}" title="{{ counts[key] }} {{ one if counts[key] == 1 else many }}">{{ ICONS[kind]|safe }}{{ counts[key] }}<span class="visually-hidden"> {{ one if counts[key] == 1 else many }}</span></span>
{%- endfor -%}
</a>
{% endif %}
{%- endmacro %}
+641
View File
@@ -0,0 +1,641 @@
"""
Tests for backfeed of responses to crossposts (v2.13.0, ADR-073)
Silo clients are tested against fake APIs (httpx.MockTransport). The
worker, storage and pages are tested with a fake client that returns
whatever a test puts in it.
"""
import io
import json
from urllib.parse import parse_qs
import httpx
import pytest
from PIL import Image
from starpunk import avatars, backfeed, create_app, syndication
from starpunk.backfeed import Harvest, Response, facets_html, interval_for, text_html
from starpunk.database import get_db
from starpunk.notes import create_note
from starpunk.syndication import BlueskyClient, MastodonClient, SyndicationError, ThreadsClient
SITE = "https://blog.example.com"
GTS = "https://social.example"
COPY = f"{GTS}/@phil/statuses/01COPY"
# --- Fakes --------------------------------------------------------------------
class FakeClient:
"""Stands in for a silo client: returns the harvest a test sets"""
config_token = "token"
def __init__(self):
self.harvest = Harvest([])
self.error = None
self.calls = []
def responses(self, remote_id, remote_url, cap=1000):
self.calls.append(remote_id)
if self.error:
raise self.error
return self.harvest
def maintain(self):
pass
def like(n, bio="", **kw):
return Response(kind="like", source_url=f"{COPY}#liked-by-{n}", remote_id=str(n),
author_id=str(n), author_name=f"Liker {n}", author_url=f"{GTS}/@l{n}",
author_bio=bio, **kw)
def reply(n, text="Nice!"):
return Response(kind="reply", source_url=f"{GTS}/@r{n}/statuses/{n}", remote_id=str(n),
author_id=f"r{n}", author_name=f"Replier {n}", author_url=f"{GTS}/@r{n}",
content_html=f"<p>{text}</p>", published="2026-11-10T12:00:00Z")
def resolve(host, *args, **kwargs):
return [(2, 1, 6, "", ("93.184.216.34", 0))]
def png(color="red", size=(200, 120)):
buf = io.BytesIO()
Image.new("RGB", size, color).save(buf, "PNG")
return buf.getvalue()
class FakeWeb:
"""Avatar images by URL (requests are pinned to an IP; Host names the site)"""
def __init__(self):
self.files = {}
def __call__(self, request):
url = f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
if url in self.files:
return httpx.Response(200, headers={"content-type": "image/png"}, content=self.files[url])
return httpx.Response(404)
@pytest.fixture
def web(monkeypatch):
fake = FakeWeb()
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(fake))
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
return fake
@pytest.fixture
def client():
return FakeClient()
@pytest.fixture
def app(tmp_path, client, web):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
"SITE_URL": SITE,
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"GOTOSOCIAL_URL": GTS,
"GOTOSOCIAL_ACCESS_TOKEN": "token",
"BACKFEED_ENABLED": True,
})
syndication._client_cache[GTS] = client
backfeed.denied.clear()
yield app
syndication._client_cache.clear()
backfeed.denied.clear()
def crossposted(app, age_days=0, content="Hello"):
"""A published note with a synced GoToSocial copy; returns (note, syndication id)"""
with app.app_context():
note = create_note(content=content, published=True)
db = get_db(app)
if age_days:
db.execute("UPDATE notes SET created_at = datetime('now', ?) WHERE id = ?",
(f"-{age_days} days", note.id))
cursor = db.execute(
"INSERT INTO syndications (note_id, target, enabled, remote_id, remote_url, state) "
"VALUES (?, 'gotosocial', 1, '01COPY', ?, 'synced')", (note.id, COPY))
db.commit()
return note, cursor.lastrowid
def rows(app, note_id):
with app.app_context():
return [dict(r) for r in get_db(app).execute(
"SELECT * FROM webmentions WHERE target_note_id = ? ORDER BY source_url", (note_id,))]
def run(app):
with app.app_context():
return backfeed.process_due()
# --- Schedule and text ------------------------------------------------------------
def test_schedule_slows_with_age_and_stops_at_cutoff():
day = 24 * 3600
assert interval_for(3600, 90) == 15 * 60
assert interval_for(2 * day, 90) == 3600
assert interval_for(10 * day, 90) == day
assert interval_for(60 * day, 90) == 7 * day
assert interval_for(91 * day, 90) is None
def test_text_and_facets_become_safe_html():
assert text_html("hi <b>\nsee https://a.example/x") == \
'<p>hi &lt;b&gt;<br>see <a href="https://a.example/x">https://a.example/x</a></p>'
text = "café: example.com/long…"
start = len("café: ".encode())
facets = [{"index": {"byteStart": start, "byteEnd": len(text.encode())},
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": "https://example.com/long/path"}]}]
assert facets_html(text, facets) == \
'<p>café: <a href="https://example.com/long/path">example.com/long…</a></p>'
assert facets_html("", []) == ""
# --- Worker and storage -------------------------------------------------------------
def test_check_stores_responses_as_webmentions(app, client):
note, sid = crossposted(app)
client.harvest = Harvest([like(1), reply(2, "Great <script>x</script>post")])
assert run(app) == 1
stored = rows(app, note.id)
assert [r["mention_type"] for r in stored] == ["like", "reply"]
assert all(r["origin"] == "gotosocial" and r["validation_status"] == "valid"
and r["moderation_status"] == "approved" for r in stored)
assert stored[1]["content_snippet"] == "<p>Great post</p>"
assert stored[0]["target_url"] == f"{SITE}/note/{note.slug}"
with app.app_context():
s = get_db(app).execute("SELECT * FROM syndications WHERE id = ?", (sid,)).fetchone()
assert s["backfeed_checked_at"] and s["backfeed_next_at"] and s["backfeed_error"] is None
assert run(app) == 0, "not due again for 15 minutes"
def test_repeat_check_updates_rows_and_keeps_moderation(app, client):
note, sid = crossposted(app)
client.harvest = Harvest([like(1), like(2)])
run(app)
with app.app_context():
db = get_db(app)
db.execute("UPDATE webmentions SET moderation_status = 'hidden' WHERE source_url LIKE '%-1'")
db.execute("UPDATE syndications SET backfeed_next_at = datetime('now')")
db.commit()
client.harvest = Harvest([like(1), like(2)])
run(app)
stored = rows(app, note.id)
assert len(stored) == 2
assert [r["moderation_status"] for r in stored] == ["hidden", "approved"]
def test_withdrawn_responses_drop_off_and_come_back(app, client):
note, _ = crossposted(app)
client.harvest = Harvest([like(1), reply(2)])
run(app)
def recheck(harvest):
with app.app_context():
get_db(app).execute("UPDATE syndications SET backfeed_next_at = datetime('now')")
get_db(app).commit()
client.harvest = harvest
run(app)
return {r["source_url"]: (r["validation_status"], r["validation_error"]) for r in rows(app, note.id)}
# An incomplete list (cap reached, say) removes nothing
assert all(v == ("valid", None) for v in recheck(Harvest([like(1)], complete=False)).values())
state = recheck(Harvest([like(1)]))
assert state[reply(2).source_url] == ("invalid", "No longer on GoToSocial")
assert recheck(Harvest([like(1), reply(2)]))[reply(2).source_url] == ("valid", None)
def test_people_who_opt_out_are_skipped_and_removed(app, client):
note, _ = crossposted(app)
client.harvest = Harvest([like(1), like(2)])
run(app)
with app.app_context():
get_db(app).execute("UPDATE syndications SET backfeed_next_at = datetime('now')")
get_db(app).commit()
client.harvest = Harvest([like(1, bio="<p>Please #NoBridge thanks</p>"), like(2),
like(3, opted_out=True)])
run(app)
assert [r["source_url"] for r in rows(app, note.id)] == [like(2).source_url]
def test_admin_delete_is_a_tombstone_the_next_check_keeps(app, client):
from starpunk.webmention_queries import count_webmentions, delete_webmentions, list_webmentions
note, _ = crossposted(app)
client.harvest = Harvest([like(1)])
run(app)
with app.test_request_context():
wm_id = rows(app, note.id)[0]["id"]
assert delete_webmentions([wm_id]) == 1
get_db(app).execute("UPDATE syndications SET backfeed_next_at = datetime('now')")
get_db(app).commit()
run(app)
assert rows(app, note.id)[0]["moderation_status"] == "deleted"
with app.test_request_context():
assert list_webmentions()[0] == [] and count_webmentions() == 0
assert len(list_webmentions(status_filter="deleted")[0]) == 1
def test_old_notes_wait_for_a_backfill(app, client):
old, sid = crossposted(app, age_days=200)
client.harvest = Harvest([reply(1)])
assert run(app) == 0
with app.app_context():
assert backfeed.start_backfill() == 1
assert backfeed.status()["remaining"] == 1
assert run(app) == 1
assert len(rows(app, old.id)) == 1
with app.app_context():
status = backfeed.status()
s = get_db(app).execute("SELECT * FROM syndications WHERE id = ?", (sid,)).fetchone()
assert status["remaining"] == 0 and status["finished"]
assert s["backfeed_backfill"] == 0 and s["backfeed_next_at"] is None, "past the cutoff: no more checks"
def test_backfill_is_paced(app, client):
crossposted(app, age_days=200)
crossposted(app, age_days=300)
with app.app_context():
backfeed.start_backfill()
assert run(app) == 1
assert run(app) == 0, "the next backfill check waits a few seconds"
with app.app_context():
get_db(app).execute("UPDATE site_state SET value = datetime('now', '-10 seconds') "
"WHERE key = 'backfeed_backfill_last'")
get_db(app).commit()
assert run(app) == 1
def test_routine_checks_skip_likes_reposts_and_unpublished(app, client):
with app.app_context():
liked = create_note(content="", published=True, like_of="https://other.example/post")
draft = create_note(content="Draft", published=False)
db = get_db(app)
for note in (liked, draft):
db.execute("INSERT INTO syndications (note_id, target, enabled, remote_id, remote_url, state) "
"VALUES (?, 'gotosocial', 1, 'x', 'https://s.example/x', 'synced')", (note.id,))
db.commit()
assert run(app) == 0 and client.calls == []
def test_refused_credential_stops_that_silo(app, client):
crossposted(app)
error = SyndicationError("GET /api/v1/statuses/01COPY/favourited_by: HTTP 403 forbidden", permanent=True)
error.status = 403
client.error = error
run(app)
assert "gotosocial" in backfeed.denied and "read:statuses" in backfeed.denied["gotosocial"]
client.error = None
assert run(app) == 0, "not retried until restart"
def test_failures_back_off(app, client):
_, sid = crossposted(app)
client.error = SyndicationError("HTTP 502 bad gateway")
run(app)
with app.app_context():
s = get_db(app).execute("SELECT * FROM syndications WHERE id = ?", (sid,)).fetchone()
assert s["backfeed_attempts"] == 1 and "502" in s["backfeed_error"]
assert run(app) == 0
def test_disabled_does_nothing(app, client):
crossposted(app)
app.config["BACKFEED_ENABLED"] = False
assert run(app) == 0 and client.calls == []
# --- Pages ------------------------------------------------------------------------------
def test_note_page_shows_backfed_responses_quotes_and_counts(app, client, web):
note, sid = crossposted(app)
web.files["https://cdn.example/a.png"] = png()
quote = Response(kind="quote", source_url="https://bsky.app/profile/q.example/post/3k",
remote_id="at://did:plc:q/app.bsky.feed.post/3k", author_name="Quoter",
author_url="https://bsky.app/profile/q.example", content_html="<p>Look at this</p>")
client.harvest = Harvest([like(1, author_photo="https://cdn.example/a.png"), reply(2), quote])
run(app)
with app.app_context():
get_db(app).execute("UPDATE syndications SET likes_count = 12, reposts_count = 1 WHERE id = ?", (sid,))
get_db(app).commit()
page = app.test_client().get(f"/note/{note.slug}").get_data(as_text=True)
assert "via GoToSocial" in page
assert "1 Quote" in page and "Look at this" in page
# Likes and reposts are a counts line (v2.13.2): named likes plus the silo's anonymous count
assert '<span class="response-count-n">13</span>' in page and "+12 on GoToSocial" in page
assert '<span class="response-count-n">1</span>' in page
assert f'<indie-action class="response-elsewhere" do="like" with="{SITE}/note/{note.slug}">' in page
assert f'Like on <a href="{COPY}"' in page
assert 'src="/avatars/' in page and 'src="https://cdn.example/a.png"' not in page
def test_avatars_are_saved_served_and_cleaned_up(app, web):
url = "https://cdn.example/face.png"
web.files[url] = png(size=(300, 200))
data = app.config["DATA_PATH"]
assert avatars.cached_name(data, url) is None
with app.app_context():
assert avatars.fetch(data, url, SITE)
assert not avatars.fetch(data, "https://cdn.example/missing.png", SITE)
name = avatars.cached_name(data, url)
with Image.open(avatars.folder(data) / name) as image:
assert image.size == (96, 96) and image.format == "WEBP"
response = app.test_client().get(f"/avatars/{name}")
assert response.status_code == 200 and response.mimetype == "image/webp"
assert app.test_client().get("/avatars/../starpunk.db").status_code == 404
# Nothing uses it: the weekly cleanup removes it
assert avatars.remove_unused(app.config["DATABASE_PATH"], data) == 1
def test_webmention_cannot_overwrite_a_backfed_response(app, client):
note, _ = crossposted(app)
client.harvest = Harvest([reply(7)])
run(app)
response = app.test_client().post("/webmention", data={
"source": reply(7).source_url, "target": f"{SITE}/note/{note.slug}"})
assert response.status_code == 202
assert rows(app, note.id)[0]["validation_status"] == "valid"
# --- Silo clients ---------------------------------------------------------------------------
def mastodon_api(request):
path, query = request.url.path, parse_qs(request.url.query.decode())
account = lambda n, **kw: {"id": n, "display_name": f"User {n}", "url": f"{GTS}/@u{n}",
"avatar": f"{GTS}/a/{n}.png", "note": "", **kw}
if path == "/api/v1/accounts/verify_credentials":
return httpx.Response(200, json={"id": "me"})
if path == "/api/v1/statuses/01COPY/favourited_by":
if "max_id" in query:
return httpx.Response(200, json=[account("3")])
return httpx.Response(200, json=[account("1"), account("me")],
headers={"Link": f'<{GTS}/api/v1/statuses/01COPY/favourited_by?max_id=9>; rel="next"'})
if path == "/api/v1/statuses/01COPY/reblogged_by":
return httpx.Response(200, json=[account("2")])
if path == "/api/v1/statuses/01COPY/context":
status = lambda n, parent="01COPY", vis="public", who=None: {
"id": n, "url": f"{GTS}/@u/statuses/{n}", "uri": f"{GTS}/u/{n}", "in_reply_to_id": parent,
"visibility": vis, "content": f"<p>reply {n}</p>", "created_at": "2026-11-10T00:00:00Z",
"account": who or account("r" + n)}
return httpx.Response(200, json={"ancestors": [], "descendants": [
status("10"), status("11", vis="unlisted"), status("12", parent="10"),
status("13", who=account("me"))]})
return httpx.Response(404)
def test_mastodon_client_reads_likes_boosts_and_public_direct_replies():
gts = MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(mastodon_api)))
harvest = gts.responses("01COPY", COPY)
assert harvest.complete
got = [(r.kind, r.source_url) for r in harvest.responses]
assert got == [("like", f"{COPY}#liked-by-1"), ("like", f"{COPY}#liked-by-3"),
("repost", f"{COPY}#reposted-by-2"), ("reply", f"{GTS}/@u/statuses/10")]
assert harvest.responses[-1].content_html == "<p>reply 10</p>"
assert gts.responses("01COPY", COPY, cap=1).complete is False
def test_bluesky_client_reads_likes_reposts_quotes_and_replies():
me, post = "did:plc:me", "at://did:plc:me/app.bsky.feed.post/abc"
def api(request):
nsid = request.url.path.removeprefix("/xrpc/")
query = parse_qs(request.url.query.decode())
profile = lambda d, **kw: {"did": d, "handle": f"{d[8:]}.example", "displayName": d[8:].title(), **kw}
view = lambda d, rkey, text: {"uri": f"at://{d}/app.bsky.feed.post/{rkey}", "author": profile(d),
"record": {"text": text, "createdAt": "2026-11-10T00:00:00Z"}}
if nsid == "com.atproto.server.createSession":
return httpx.Response(200, json={"did": me, "handle": "me.example", "accessJwt": "a", "refreshJwt": "r"})
if nsid == "app.bsky.feed.getLikes":
if query.get("cursor"):
return httpx.Response(200, json={"likes": [{"actor": profile("did:plc:carol")}]})
return httpx.Response(200, json={"cursor": "c1", "likes": [
{"actor": profile("did:plc:alice"), "createdAt": "2026-11-10T01:00:00Z"},
{"actor": profile("did:plc:shy", labels=[{"val": "!no-unauthenticated"}])},
{"actor": profile(me)}]})
if nsid == "app.bsky.feed.getRepostedBy":
return httpx.Response(200, json={"repostedBy": [profile("did:plc:bob")]})
if nsid == "app.bsky.feed.getQuotes":
return httpx.Response(200, json={"posts": [view("did:plc:dan", "q1", "Quoting this")]})
if nsid == "app.bsky.feed.getPostThread":
return httpx.Response(200, json={"thread": {"post": {}, "replies": [
{"post": view("did:plc:eve", "r1", "A reply")},
{"$type": "app.bsky.feed.defs#blockedPost", "uri": "at://x"}]}})
if nsid == "app.bsky.actor.getProfiles":
return httpx.Response(200, json={"profiles": [
profile("did:plc:dan", description="no thanks #nobot"), profile("did:plc:eve")]})
return httpx.Response(400, json={"error": "Unknown"})
bsky = BlueskyClient("https://bsky.example", "me.example", "pw",
http=httpx.Client(transport=httpx.MockTransport(api)))
harvest = bsky.responses(post, "https://bsky.app/profile/me.example/post/abc")
by = {(r.kind, r.author_id): r for r in harvest.responses}
assert set(by) == {("like", "did:plc:alice"), ("like", "did:plc:shy"), ("like", "did:plc:carol"),
("repost", "did:plc:bob"), ("quote", "did:plc:dan"), ("reply", "did:plc:eve")}
assert by[("like", "did:plc:shy")].opted_out
assert by[("quote", "did:plc:dan")].author_bio == "no thanks #nobot"
assert by[("reply", "did:plc:eve")].source_url == "https://bsky.app/profile/eve.example/post/r1"
assert by[("like", "did:plc:alice")].source_url.endswith("#liked-by-did:plc:alice")
assert harvest.complete
def test_threads_client_reads_replies_and_counts():
def api(request):
path = request.url.path
params = parse_qs(request.url.query.decode())
if path.endswith("/me"):
return httpx.Response(200, json={"id": "42", "username": "me"})
if path.endswith("/111/replies"):
assert "profile_picture_url" in params["fields"][0]
return httpx.Response(200, json={"data": [
{"id": "1", "text": "Hi", "username": "ann", "permalink": "https://www.threads.com/@ann/post/A",
"timestamp": "2026-11-10T00:00:00+0000", "hide_status": "NOT_HUSHED"},
{"id": "2", "text": "private"},
{"id": "3", "text": "hidden", "username": "x", "permalink": "p", "hide_status": "HIDDEN"},
{"id": "4", "text": "mine", "username": "me", "permalink": "p"}]})
if path.endswith("/111/insights"):
assert params["metric"] == ["likes,reposts,quotes"]
return httpx.Response(200, json={"data": [
{"name": "likes", "values": [{"value": 12}]}, {"name": "reposts", "values": [{"value": 3}]},
{"name": "quotes", "values": [{"value": 0}]}]})
return httpx.Response(404, json={"error": {"code": 100}})
threads = ThreadsClient("tok", http=httpx.Client(transport=httpx.MockTransport(api)))
harvest = threads.responses("111", "https://www.threads.com/@me/post/X")
assert [(r.kind, r.author_name, r.content_html) for r in harvest.responses] == [("reply", "ann", "<p>Hi</p>")]
assert harvest.counts == {"likes": 12, "reposts": 3, "quotes": 0}
# --- Migration ---------------------------------------------------------------------------
def test_migration_020_keeps_existing_webmentions(tmp_path):
import sqlite3
from pathlib import Path
from starpunk.database.schema import INITIAL_SCHEMA_SQL
from starpunk.migrations import execute_in_transaction
migrations = Path(__file__).parent.parent / "migrations"
conn = sqlite3.connect(tmp_path / "old.db")
conn.execute("PRAGMA foreign_keys=ON")
conn.executescript(INITIAL_SCHEMA_SQL)
for name in ("010_add_webmentions.sql", "012_add_syndications.sql", "014_webmention_source_details.sql"):
matches = list(migrations.glob(name.split("_")[0] + "_*.sql"))
conn.execute("BEGIN")
execute_in_transaction(conn, matches[0].read_text())
conn.commit()
conn.execute("INSERT INTO notes (slug, file_path, published) VALUES ('n', 'n.md', 1)")
conn.execute("INSERT INTO webmentions (source_url, target_url, target_note_id, mention_type, "
"validation_status, moderation_status, source_published) "
"VALUES ('https://a.example/r', 'https://b.example/note/n', 1, 'reply', 'valid', 'hidden', "
"'2026-01-01')")
conn.commit()
conn.execute("BEGIN")
execute_in_transaction(conn, (migrations / "020_backfeed.sql").read_text())
conn.commit()
row = conn.execute("SELECT mention_type, moderation_status, source_published, origin, target_note_id "
"FROM webmentions").fetchone()
assert row == ("reply", "hidden", "2026-01-01", "webmention", 1)
conn.execute("INSERT INTO webmentions (source_url, target_url, mention_type, origin) "
"VALUES ('https://q.example', 'https://b.example/note/n', 'quote', 'bluesky')")
columns = {r[1] for r in conn.execute("PRAGMA table_info(syndications)")}
assert {"backfeed_next_at", "backfeed_backfill", "likes_count"} <= columns
assert conn.execute("PRAGMA foreign_key_check").fetchall() == []
# --- Admin ----------------------------------------------------------------------------------
def test_admin_backfill_and_check_now(app, client):
from starpunk.auth import create_session
app.config.update(ADMIN_ME="https://example.com", CSRF_ENABLED=False)
note, sid = crossposted(app, age_days=200)
with app.test_request_context():
token = create_session("https://example.com")
browser = app.test_client()
browser.set_cookie("starpunk_session", token)
page = browser.get("/admin/webmentions").get_data(as_text=True)
assert "No backfill has been run" in page and "Backfill responses" in page
response = browser.post("/admin/backfeed/backfill", follow_redirects=True)
assert "Backfill started: 1 crosspost(s) to check" in response.get_data(as_text=True)
assert "1 crosspost(s) left to check" in browser.get("/admin/webmentions").get_data(as_text=True)
edit = browser.get(f"/admin/edit/{note.id}").get_data(as_text=True)
assert "Responses not checked yet" in edit and "Check now" in edit
browser.post(f"/admin/syndication/{sid}/check")
with app.app_context():
s = get_db(app).execute("SELECT backfeed_next_at FROM syndications WHERE id = ?", (sid,)).fetchone()
assert s["backfeed_next_at"]
backfeed.denied["gotosocial"] = "Needs a GoToSocial token with read:statuses"
assert "Needs a GoToSocial token" in browser.get("/admin/webmentions?origin=gotosocial").get_data(as_text=True)
def test_threads_avatar_kept_while_its_copy_exists(app, client, web):
note, _ = crossposted(app)
with app.app_context():
get_db(app).execute("UPDATE syndications SET target = 'threads'")
get_db(app).commit()
syndication._client_cache["threads"] = client
app.config["THREADS_ACCESS_TOKEN"] = "token"
web.files["https://cdn.example/p.jpg?sig=1"] = png()
first = reply(1)
first.author_photo = "https://cdn.example/p.jpg?sig=1"
client.harvest = Harvest([first])
run(app)
with app.app_context():
get_db(app).execute("UPDATE syndications SET backfeed_next_at = datetime('now')")
get_db(app).commit()
again = reply(1)
again.author_photo = "https://cdn.example/p.jpg?sig=2" # not fetchable: would be a refetch
client.harvest = Harvest([again])
run(app)
assert rows(app, note.id)[0]["author_photo"] == "https://cdn.example/p.jpg?sig=1"
def test_avatars_from_own_gotosocial_and_bluesky_pds(app, web, monkeypatch):
"""v2.13.1: own GoToSocial may resolve to a private address; Bluesky CDN avatars come from the PDS"""
did, cid = "did:plc:abc123", "bafkreiabc"
web.files[f"{GTS}/fileserver/a/avatar.png"] = png()
web.files[f"https://plc.directory/{did}"] = json.dumps(
{"service": [{"id": "#atproto_pds", "serviceEndpoint": "https://pds.example"}]}).encode()
web.files[f"https://pds.example/xrpc/com.atproto.sync.getBlob?did={did}&cid={cid}"] = png("blue")
real_resolve = resolve
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
lambda host, *a, **k: [(2, 1, 6, "", ("172.20.0.5", 0))]
if host == "social.example" else real_resolve(host))
data = app.config["DATA_PATH"]
with app.app_context():
assert avatars.fetch(data, f"{GTS}/fileserver/a/avatar.png", SITE)
assert avatars.fetch(data, f"https://cdn.bsky.app/img/avatar/plain/{did}/{cid}@jpeg", SITE)
# Other private addresses stay blocked
web.files["https://internal.example/a.png"] = png()
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
lambda host, *a, **k: [(2, 1, 6, "", ("10.0.0.5", 0))])
assert not avatars.fetch(data, "https://internal.example/a.png", SITE)
def test_own_gotosocial_redirect_to_remote_avatar_is_followed(app, web, monkeypatch):
"""v2.13.2: GoToSocial redirects uncached remote media to the original server"""
class Redirecting(FakeWeb):
def __call__(self, request):
url = f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
if url == f"{GTS}/fileserver/x/avatar.jpeg":
return httpx.Response(302, headers={"location": "https://remote.example/avatar.jpeg"})
return super().__call__(request)
fake = Redirecting()
fake.files["https://remote.example/avatar.jpeg"] = png()
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(fake))
with app.app_context():
assert avatars.fetch(app.config["DATA_PATH"], f"{GTS}/fileserver/x/avatar.jpeg", SITE)
def test_people_who_opt_out_are_counted_not_named(app, client):
"""v2.13.3: likes and reposts by people who opted out add to the totals anonymously"""
note, sid = crossposted(app)
shy = like(5, opted_out=True)
client.harvest = Harvest([like(1), shy, like(6, bio="#nobot"),
Response(kind="repost", source_url=f"{COPY}#reposted-by-7", remote_id="7",
author_name="Quiet", author_bio="#nobridge")])
run(app)
with app.app_context():
s = get_db(app).execute("SELECT likes_count, reposts_count FROM syndications WHERE id = ?",
(sid,)).fetchone()
assert (s["likes_count"], s["reposts_count"]) == (2, 1)
assert [r["source_url"] for r in rows(app, note.id)] == [like(1).source_url]
page = app.test_client().get(f"/note/{note.slug}").get_data(as_text=True)
assert '<span class="response-count-n">3</span>' in page and "+2 on GoToSocial" in page
assert "Liker 5" not in page and "Quiet" not in page
def test_home_page_shows_counts(app, client):
"""v2.13.4: lists of notes show like, repost and reply counts, linking to the responses"""
note, sid = crossposted(app)
with app.app_context():
quiet = create_note(content="Nobody saw this", published=True)
client.harvest = Harvest([like(1), like(2), reply(3)])
run(app)
with app.app_context():
get_db(app).execute("UPDATE syndications SET likes_count = 1, reposts_count = 2 WHERE id = ?", (sid,))
get_db(app).commit()
page = app.test_client().get("/").get_data(as_text=True)
assert f'href="{SITE}/note/{note.slug}#responses"' in page
assert 'title="3 likes"' in page and 'title="2 reposts"' in page and 'title="1 reply"' in page
assert f"/note/{quiet.slug}#responses" not in page
assert 'id="responses"' in app.test_client().get(f"/note/{note.slug}").get_data(as_text=True)
+12
View File
@@ -198,6 +198,18 @@ class TestQueries:
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["IndieWeb", "python"]
assert client.get("/micropub?q=category&filter=py", headers=h).get_json()["categories"] == ["python"]
def test_category_skips_unused_tags(self, client, h):
"""Tags dropped by an edit or on deleted posts aren't offered (v2.12.1)"""
create(client, h, category=["kept"])
retagged = create(client, h, category=["dropped"])
deleted = create(client, h, category=["trashed"])
body = {"action": "update", "url": retagged, "replace": {"category": ["kept"]}}
assert client.post("/micropub", json=body, headers=h).status_code in (200, 204)
assert client.post("/micropub", json={"action": "delete", "url": deleted}, headers=h).status_code == 204
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["kept"]
assert client.post("/micropub", json={"action": "undelete", "url": deleted}, headers=h).status_code < 300
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["kept", "trashed"]
def test_config_lists_queries(self, client, h):
config = client.get("/micropub?q=config", headers=h).get_json()
assert "category" in config["q"] and "media-endpoint" in config
+251
View File
@@ -0,0 +1,251 @@
"""
Tests for notifications through Apprise (v2.14.0, ADR-074)
Most tests replace the Apprise send with a recorder. One sends for real
through Apprise to a local HTTP server (json://), to check the wiring.
"""
import json
import sqlite3
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
import pytest
from starpunk import backfeed, create_app, notifications, syndication
from starpunk.backfeed import Harvest, Response
from starpunk.database import get_db
from starpunk.notes import create_note
SITE = "https://blog.example.com"
@pytest.fixture
def sent(monkeypatch):
messages = []
def deliver(targets, title, body):
messages.append({"targets": targets, "title": title, "body": body})
return True, ""
monkeypatch.setattr(notifications, "_deliver", deliver)
return messages
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
app = create_app({
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"NOTIFY_URLS": "gotifys://gotify.example.com/TOKEN, ntfys://topic?priority=max",
"NOTIFY_EVENTS": "reply,mention,like,problems",
"NOTIFY_PRIORITY": "high",
"CSRF_ENABLED": False,
})
yield app
syndication._client_cache.clear()
backfeed.denied.clear()
def note(app, content="Hello world", age_days=0):
with app.app_context():
n = create_note(content=content, published=True)
if age_days:
get_db(app).execute("UPDATE notes SET created_at = datetime('now', ?) WHERE id = ?",
(f"-{age_days} days", n.id))
get_db(app).commit()
return n
def response(app, n, kind, name, source, moderation="approved", origin="webmention",
snippet=None, hours_ago=0):
with app.app_context():
get_db(app).execute(
"INSERT INTO webmentions (source_url, target_url, target_note_id, mention_type, author_name, "
"validation_status, moderation_status, origin, content_snippet, received_at) "
"VALUES (?, ?, ?, ?, ?, 'valid', ?, ?, ?, datetime('now', ?))",
(source, f"{SITE}/note/{n.slug}", n.id, kind, name, moderation, origin, snippet,
f"-{hours_ago} hours"))
get_db(app).commit()
def sweep(app, sweeper=None):
sweeper = sweeper or notifications.Sweeper()
with app.app_context():
return sweeper.run(app.config, app.config["DATABASE_PATH"], SITE), sweeper
def unsent(app):
with app.app_context():
return [r[0] for r in get_db(app).execute(
"SELECT source_url FROM webmentions WHERE notified_at IS NULL ORDER BY id")]
# --- Settings --------------------------------------------------------------------------
def test_settings():
config = {"NOTIFY_URLS": " a://x,b://y c://z ", "NOTIFY_EVENTS": "Reply, likes, problems"}
assert notifications.urls(config) == ["a://x", "b://y", "c://z"]
assert notifications.events(config) == {"reply", "problems"}, "unknown names are ignored"
assert notifications.events({}) == {"reply", "mention", "quote", "problems"}
assert notifications.with_priority("gotifys://h/T", "high") == "gotifys://h/T?priority=high"
assert notifications.with_priority("ntfys://t?tags=x", "5") == "ntfys://t?tags=x&priority=5"
assert notifications.with_priority("ntfys://t?priority=max", "low") == "ntfys://t?priority=max"
# --- Sweep -------------------------------------------------------------------------------------
def test_new_responses_are_sent_once_and_likes_grouped(app, sent):
n = note(app, "Starpunk now supports backfeeds")
response(app, n, "reply", "Ana", "https://a.example/r", origin="bluesky",
snippet="<p>Great &amp; useful!</p>")
response(app, n, "like", "Bob", "https://b.example/l")
response(app, n, "like", "Cy", "https://c.example/l")
response(app, n, "repost", "Dee", "https://d.example/rp") # not selected
response(app, n, "mention", "Eve", "https://e.example/m", moderation="pending")
count, _ = sweep(app)
assert count == 4
titles = [m["title"] for m in sent]
assert "↩ Reply from Ana via Bluesky" in titles
assert "💬 Mention from Eve (awaiting moderation)" in titles
assert "♥ 2 new likes on “Starpunk now supports backfeeds”" in titles
reply = next(m for m in sent if m["title"].startswith("↩"))
assert reply["body"].startswith("Great & useful!")
assert f"({SITE}/note/{n.slug}#responses)" in reply["body"]
mention = next(m for m in sent if m["title"].startswith("💬"))
assert f"{SITE}/admin/webmentions/" in mention["body"], "pending ones link to moderation"
assert all(t.endswith("priority=high") or t.endswith("priority=max") for t in sent[0]["targets"])
assert unsent(app) == []
sent.clear()
assert sweep(app)[0] == 0 and sent == [], "nothing is sent twice"
def test_failed_send_is_retried_with_backoff(app, monkeypatch):
calls = []
monkeypatch.setattr(notifications, "_deliver", lambda t, ti, b: (calls.append(ti), (False, "boom"))[1])
n = note(app)
response(app, n, "reply", "Ana", "https://a.example/r")
count, sweeper = sweep(app)
assert count == 0 and unsent(app) == ["https://a.example/r"]
assert sweep(app, sweeper)[0] == 0 and len(calls) == 1, "waits before retrying"
with app.app_context():
assert "boom" in notifications.last_error(get_db(app))
sweeper._next_try = 0
monkeypatch.setattr(notifications, "_deliver", lambda t, ti, b: (True, ""))
assert sweep(app, sweeper)[0] == 1
with app.app_context():
assert notifications.last_error(get_db(app)) == ""
def test_old_unsent_responses_are_dropped(app, sent):
n = note(app)
response(app, n, "reply", "Ana", "https://a.example/r", hours_ago=30)
assert sweep(app)[0] == 0 and sent == [] and unsent(app) == []
def test_off_without_urls(app, sent):
app.config["NOTIFY_URLS"] = ""
n = note(app)
response(app, n, "reply", "Ana", "https://a.example/r")
assert sweep(app)[0] == 0 and unsent(app) == ["https://a.example/r"]
def test_backfill_and_late_first_checks_are_quiet(app, sent):
class Client:
config_token = "t"
def responses(self, remote_id, remote_url, cap=1000):
return Harvest([Response(kind="reply", source_url=f"{remote_url}/reply", remote_id="r",
author_name="Ana", content_html="<p>hi</p>")])
def maintain(self):
pass
app.config.update(GOTOSOCIAL_URL="https://social.example", GOTOSOCIAL_ACCESS_TOKEN="t",
BACKFEED_ENABLED=True)
syndication._client_cache["https://social.example"] = Client()
fresh, old = note(app, "New"), note(app, "Old", age_days=10)
with app.app_context():
for n, url in ((fresh, "https://social.example/1"), (old, "https://social.example/2")):
get_db(app).execute("INSERT INTO syndications (note_id, target, enabled, remote_id, remote_url, "
"state) VALUES (?, 'gotosocial', 1, 'x', ?, 'synced')", (n.id, url))
get_db(app).commit()
backfeed.process_due()
assert unsent(app) == ["https://social.example/1/reply"], "only the new note's reply is news"
# --- Problems and admin ----------------------------------------------------------------------------
def test_problems(app, sent):
with app.app_context():
notifications.problem(app.config, app.config["DATABASE_PATH"], "Crosspost to Bluesky failed",
"HTTP 400", f"{SITE}/admin/edit/1")
assert sent[0]["title"] == "⚠ Crosspost to Bluesky failed"
assert "HTTP 400" in sent[0]["body"] and "/admin/edit/1" in sent[0]["body"]
app.config["NOTIFY_EVENTS"] = "reply"
with app.app_context():
notifications.problem(app.config, app.config["DATABASE_PATH"], "x", "y", "z")
assert len(sent) == 1, "not selected: not sent"
def test_final_crosspost_failure_notifies(app, sent):
n = note(app)
app.config.update(GOTOSOCIAL_URL="https://social.example", GOTOSOCIAL_ACCESS_TOKEN="t")
with app.app_context():
cur = get_db(app).execute("INSERT INTO syndications (note_id, target, enabled, state, attempts) "
"VALUES (?, 'gotosocial', 1, 'pending', 0)", (n.id,))
get_db(app).commit()
row = get_db(app).execute("SELECT * FROM syndications WHERE id = ?", (cur.lastrowid,)).fetchone()
syndication._fail(get_db(app), row, syndication.SyndicationError("temporary", permanent=False))
assert sent == [], "a retry is coming: no message"
syndication._fail(get_db(app), row, syndication.SyndicationError("HTTP 422 bad", permanent=True))
assert sent and sent[0]["title"] == "⚠ Crosspost to GoToSocial failed"
def test_admin_status_and_test_button(app, sent):
from starpunk.auth import create_session
with app.test_request_context():
token = create_session("https://example.com")
browser = app.test_client()
browser.set_cookie("starpunk_session", token)
page = browser.get("/admin/webmentions").get_data(as_text=True)
assert "Sending like, mention, problems, reply to gotifys, ntfys at high priority" in page
page = browser.post("/admin/notifications/test", follow_redirects=True).get_data(as_text=True)
assert "Test notification sent" in page and sent[-1]["title"] == "StarPunk test notification"
# --- Real Apprise -------------------------------------------------------------------------------------
def test_apprise_delivers_markdown_to_a_service(app):
received = []
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
received.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
self.send_response(200)
self.end_headers()
def log_message(self, *args):
pass
server = HTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.handle_request, daemon=True).start()
app.config["NOTIFY_URLS"] = f"json://127.0.0.1:{server.server_port}/"
ok, error = notifications.send(app.config, "Title", "Body with [a link](https://x.example)")
server.server_close()
assert ok, error
assert received[0]["title"] == "Title" and "a link" in received[0]["message"]
app.config["NOTIFY_URLS"] = "notaservice://nope"
ok, error = notifications.send(app.config, "t", "b")
assert not ok and "Not a usable Apprise URL" in error
def test_migration_021_marks_history(tmp_path):
from pathlib import Path
conn = sqlite3.connect(tmp_path / "db")
conn.execute("CREATE TABLE webmentions (id INTEGER PRIMARY KEY, source_url TEXT)")
conn.execute("INSERT INTO webmentions (source_url) VALUES ('https://old.example')")
conn.executescript((Path(__file__).parent.parent / "migrations" / "021_notifications.sql").read_text())
assert conn.execute("SELECT notified_at IS NOT NULL FROM webmentions").fetchone()[0] == 1