Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
acd6bf819d | ||
|
|
bf6e0fdc0b | ||
|
|
952ca300c0 | ||
|
|
03a1c8fcea | ||
|
|
67c0411fd8 | ||
|
|
f7a936562c | ||
|
|
62e257e0e0 | ||
|
|
22e588377f | ||
|
|
b9e5321df2 | ||
|
|
52ecf0f713 | ||
|
|
5fb521e89e | ||
|
|
bba2178496 | ||
|
|
41c373f46c | ||
|
|
7158713a2a | ||
|
|
73a68157cc | ||
|
|
bda7adc449 | ||
|
|
aa88db50b2 |
No files matched your search
@@ -130,6 +130,9 @@ CSRF_ENABLED=true
|
||||
# Background worker that verifies received webmentions, sends outgoing ones,
|
||||
# and sends response notifications (v2.14.0) (default: true)
|
||||
# WEBMENTION_WORKER_ENABLED=true
|
||||
# Hosts never sent webmentions, comma-separated, e.g. self-hosted apps that
|
||||
# don't accept them. Links to this site itself are always skipped (v2.18.3)
|
||||
# WEBMENTION_SKIP_HOSTS=git.example.com,photos.example.com
|
||||
# Seconds between queue checks (default: 10) and after an error (default: 60)
|
||||
# WEBMENTION_POLL_INTERVAL=10
|
||||
# WEBMENTION_ERROR_BACKOFF=60
|
||||
|
||||
+105
@@ -7,6 +7,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [2.20.2] - 2026-11-13
|
||||
|
||||
### Fixed
|
||||
|
||||
- Replies to replies on crossposts were never captured. Backfeed read only the first level of replies (Threads `/replies`, GoToSocial's direct replies, Bluesky at depth 1). It now reads the whole conversation (Threads `/conversation`, every GoToSocial descendant, Bluesky ten levels deep)
|
||||
- Your own replies in those conversations were skipped, so the other side of a back-and-forth was missing. They are now kept when they answer someone's reply (a reply to your own crosspost is the thread continuing, and is still left out). They don't send you notifications
|
||||
|
||||
### Changed
|
||||
|
||||
- Replies are shown nested under the reply they answer, as comments of comments (`u-comment` inside the parent `h-cite`). Migration 026 adds `webmentions.parent_remote_id`. Existing crossposts pick up their threads at their next backfeed check
|
||||
|
||||
## [2.20.1] - 2026-11-13
|
||||
|
||||
### Fixed
|
||||
|
||||
- A reply to a Mastodon post with a GIF showed a still image. Mastodon turns GIFs into short silent looping MP4s ("gifv"), and only their first frame was kept. The MP4 is now saved on this site and plays muted and looping, as on Mastodon, with the still image as its poster
|
||||
- Animated GIFs (and animated WebP/PNG) from other sites stay animated in reply contexts instead of being cut to their first frame
|
||||
|
||||
### Changed
|
||||
|
||||
- Migration 025 adds `video` to `reply_contexts`, and has fediverse contexts with an image fetched again once, so existing replies pick up their gifv
|
||||
|
||||
## [2.20.0] - 2026-11-13
|
||||
|
||||
### Added
|
||||
|
||||
- Crossposted replies look like replies. A reply to a post on the same network is posted as a native reply: on GoToSocial it threads under the fediverse post and starts with the author's @mention (so they're notified and their server gets it); on Bluesky it threads under the Bluesky post
|
||||
- A reply to one of your own posts (its page here, or any of its crossposts) threads under that post's crosspost on every network, Threads included
|
||||
- A reply that can't thread on a network (a fediverse post crossposted to Bluesky or Threads, a blog post anywhere) starts with "↩ Replying to <author>" and the post's link. On Bluesky the author's name is the link, to save characters
|
||||
|
||||
Crossposts made before this release are unchanged: edits can't move a post into a thread.
|
||||
|
||||
## [2.19.0] - 2026-11-13
|
||||
|
||||
### Added
|
||||
|
||||
- Replies show what they reply to. A note's page shows a card with the post's author (avatar, name, handle), its text, its first image and when it was posted, linking to the original. The home, tag and likes pages show "↩ In reply to <author>" with a short quote of the post
|
||||
- Context for fediverse and Bluesky posts. These pages have no microformats, so StarPunk asks the post's Mastodon server's API, then this site's GoToSocial instance (which also reaches GoToSocial and locked-down Mastodon servers), or the public Bluesky API. Other pages without microformats use their OpenGraph tags
|
||||
- The avatars and images in reply contexts are saved on this site, like response avatars (the CSP doesn't allow images from other sites). Images are kept at most 640 px wide in `DATA_PATH/previews/`
|
||||
|
||||
### Fixed
|
||||
|
||||
- A reply to a Mastodon post showed only the server's name ("defcon.social"): the page had no h-entry, so the fetch found nothing and the domain stood in for the author. Contexts fetched before this release are fetched once more by the webmention worker
|
||||
- The quoted text of a replied-to post was shown with its HTML tags escaped
|
||||
|
||||
### Changed
|
||||
|
||||
- Migration 024 adds `author_handle`, `photo`, `photo_alt` and `source` to `reply_contexts`
|
||||
|
||||
## [2.18.3] - 2026-11-12
|
||||
|
||||
### Added
|
||||
|
||||
- `WEBMENTION_SKIP_HOSTS`: comma-separated hosts that are never sent webmentions, for self-hosted apps that don't accept them (Gitea, say). The Settings page lists them
|
||||
|
||||
### Fixed
|
||||
|
||||
- StarPunk sent webmentions to links pointing at itself. It only takes webmentions for its posts, and its own public address can time out from inside the container, so these failed and sat in the Log. Links to this site are now skipped
|
||||
- Sends to this site or a skipped host that were already queued or failed are dropped when the worker starts, so they leave the Log
|
||||
|
||||
## [2.18.2] - 2026-11-12
|
||||
|
||||
### Fixed
|
||||
- Tables in notes and articles: words in cells were split mid-word (a narrow
|
||||
first column read "Colo/r", "Blac/k") and cells had no padding or left
|
||||
alignment. Tables now keep words whole and render as a card that fills the
|
||||
column: raised background, rounded border, shaded header row, roomy cells,
|
||||
sans-serif text at reading size, bold first column. Rendered tables are
|
||||
wrapped in `<div class="table-wrap">`, which scrolls sideways on narrow
|
||||
screens.
|
||||
|
||||
### Changed
|
||||
- Public pages are wider: 48rem instead of 40rem (about 85 characters per
|
||||
line at the reading font). Admin pages stay at 60rem.
|
||||
- The site header (title, menu, search) and footer are 60rem wide on every
|
||||
page, so they sit in the same place on public and admin pages; only the
|
||||
content column differs.
|
||||
|
||||
## [2.18.1] - 2026-11-12
|
||||
|
||||
### Changed
|
||||
|
||||
- The MCP `create_note` tool says that articles live at `/article/<slug>` with a slug made from the title, so AI clients leave `slug` out unless they want a specific one
|
||||
|
||||
## [2.18.0] - 2026-11-12
|
||||
|
||||
A tidier admin area.
|
||||
|
||||
### Added
|
||||
- **Log page** (`/admin/log`): everything that is failing right now, newest
|
||||
first. Each entry shows when it happened, what failed (crosspost, response
|
||||
check, outgoing webmention, reply-context fetch, WebSub delivery,
|
||||
notifications, refused silo credentials), a link to the affected post and
|
||||
the error message, with Retry / Check now where that applies. It reads the
|
||||
error columns the workers already keep, so an entry goes away once the
|
||||
next attempt succeeds. The admin nav shows the error count.
|
||||
- **Settings page** (`/admin/settings`) for site-wide settings. Notification
|
||||
status and the test button live here now.
|
||||
- Webmentions, Log and Settings links in the admin nav.
|
||||
|
||||
### Changed
|
||||
- The Webmentions page no longer has the Back to Dashboard button or the
|
||||
Notifications section, and links to the Log instead of counting response
|
||||
check errors.
|
||||
|
||||
## [2.17.1] - 2026-11-12
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -139,8 +139,8 @@ again with the new content. `_fail()` does the same.
|
||||
a Threads container in `ERROR`/`EXPIRED`, and Graph API auth/permission
|
||||
errors (codes 10, 190, 200).
|
||||
- Other errors retry after `min(30 s × 2^attempts, 6 h)`. After 8 attempts
|
||||
the row becomes `failed`. The error is shown on the edit page with a
|
||||
Retry button. When a row finally fails, the owner also gets a `problems`
|
||||
the row becomes `failed`. The error is shown on the edit page and on the
|
||||
admin Log page (v2.18.0), both with a Retry button. When a row finally fails, the owner also gets a `problems`
|
||||
notification (ADR-074).
|
||||
- `RemoteGone` (404, or Graph error 100/33) during an update means the copy
|
||||
was deleted remotely, so a new one is created. During a delete it counts as
|
||||
@@ -273,7 +273,7 @@ reposts) that are due, each under a lease (`backfeed_claimed_until`):
|
||||
(`site_state.backfeed_backfill_last`). Start and finish times are in
|
||||
`site_state`.
|
||||
- **Refused credentials.** A 401/403 (or Graph auth error) puts the target
|
||||
in `backfeed.denied` until restart; the admin Webmentions page says which
|
||||
in `backfeed.denied` until restart; the admin Log page says which
|
||||
scopes are needed, and a `problems` notification is sent. Crossposting
|
||||
is unaffected.
|
||||
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# v2.18.1: MCP create_note describes article URLs and slugs
|
||||
|
||||
Documentation-only follow-up to ADR-077 (v2.17.x). The MCP `create_note`
|
||||
tool description now says that a title makes the post an article at
|
||||
`/article/<slug>`, that articles get their slug from the title and other
|
||||
notes from the time, and that `slug` is only needed for a specific slug.
|
||||
AI clients read tool descriptions to decide which arguments to send, so
|
||||
this keeps them from making up a slug the title would already give.
|
||||
No behaviour change.
|
||||
@@ -0,0 +1,34 @@
|
||||
# v2.18.3: Don't send webmentions to this site or skipped hosts
|
||||
|
||||
## Why
|
||||
|
||||
The admin Log showed two sends that could never succeed, both "timed out":
|
||||
|
||||
- a note linking to `https://starpunk.thesatelliteoflove.com/` (this site),
|
||||
- a note linking to a Gitea repository on `git.thesatelliteoflove.com`.
|
||||
|
||||
From inside the container both names resolve to the server's public address,
|
||||
which isn't reachable from there (no NAT hairpinning). Routing them to the
|
||||
internal proxy with `extra_hosts` wouldn't help: the SSRF guard blocks
|
||||
private addresses. And neither target wants the webmention: StarPunk only
|
||||
takes webmentions for its posts, and Gitea advertises no endpoint.
|
||||
|
||||
## What changed
|
||||
|
||||
| Area | Change |
|
||||
|---|---|
|
||||
| Config | `WEBMENTION_SKIP_HOSTS` (comma-separated, lowercased) |
|
||||
| Sender | `webmention_sender.skipped_hosts(config)` = `SITE_URL`'s host + the setting; `is_skipped(url, hosts)` matches the exact host, any port. `queue_webmentions_for_note` drops those links before the 10-link cap, and from re-sends on update |
|
||||
| Worker | `_send_webmention` marks a skipped target `no_endpoint` without fetching it (rows queued before the setting changed). At startup `_drop_skipped_targets` does the same for `pending` and `failed` rows, so they leave the Log |
|
||||
| Settings | A Webmentions section lists the skipped hosts |
|
||||
|
||||
`no_endpoint` with `last_error` = "Not sent: …" is used rather than a new
|
||||
status because `webmention_queue.status` has a CHECK constraint; changing it
|
||||
would mean rebuilding the table. The Log only lists `pending` and `failed`.
|
||||
|
||||
Exact host matching is deliberate: skipping `thesatelliteoflove.com` should
|
||||
not skip its subdomains, or the reverse.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_webmention_skip_hosts.py`.
|
||||
@@ -0,0 +1,108 @@
|
||||
# v2.19.0: Reply contexts that show the post
|
||||
|
||||
## Why
|
||||
|
||||
A reply to `https://defcon.social/@deviantollam/117611186265622833` showed
|
||||
"In reply to: defcon.social" and nothing else. Reply contexts were read only
|
||||
from microformats. Mastodon's status page is a JavaScript app shell with no
|
||||
h-entry, so `fetch_reply_context` stored `fetch_status='fetched'` with no
|
||||
content, and `extract_author` fell back to the URL's host for the author name.
|
||||
GoToSocial pages have an h-card but no h-entry; Bluesky pages have neither.
|
||||
The home page didn't use the context at all: it printed the raw URL.
|
||||
|
||||
The data was available elsewhere:
|
||||
|
||||
- Mastodon's public API (`/api/v1/statuses/<id>`) returned the author, text,
|
||||
attachments (with alt text) and date without authentication.
|
||||
- GoToSocial's API needs a token, but this site's own GoToSocial instance can
|
||||
resolve any fediverse URL with signed requests
|
||||
(`/api/v2/search?resolve=true`). The crossposter already does this for
|
||||
native likes and boosts (`MastodonClient.find_status`).
|
||||
- `public.api.bsky.app` answers `app.bsky.feed.getPostThread` for
|
||||
`at://<handle>/app.bsky.feed.post/<rkey>` without logging in. (`getPosts`
|
||||
rejects handles in the URI; `getPostThread` resolves them.)
|
||||
- Most pages carry OpenGraph tags: `og:title`, `og:description`, `og:image`
|
||||
(with `og:image:alt`), `og:site_name`, a published time. Fediverse servers
|
||||
put "Name (@user@host)" in `og:title` and the handle in
|
||||
`profile:username`.
|
||||
|
||||
## Sources, in order
|
||||
|
||||
| # | When | Source | `source` |
|
||||
|---|---|---|---|
|
||||
| 1 | `bsky.app/profile/<actor>/post/<rkey>` | Bluesky AppView API | `bluesky` |
|
||||
| 2 | The page has an h-entry | Microformats (as before), image from `u-photo` or `og:image` | `mf2` |
|
||||
| 3 | Fediverse path (`/@user/<id>`, `/@user/statuses/<id>`, `/users/…/statuses/…`, `/notes/…`, `/notice/…`, `/objects/…`) or a `profile:username` tag | The origin's Mastodon API (numeric Mastodon ids only), else this site's GoToSocial instance | `fediverse` |
|
||||
| 4 | Any other HTML page | OpenGraph tags; author from the page's h-card or `<meta name="author">` | `opengraph` |
|
||||
|
||||
All requests other than the instance lookup go through `fetch_source`, so
|
||||
they get the SSRF checks, size limit and timeouts. The instance lookup uses
|
||||
the crossposting client (`MastodonClient.resolve_status`, which
|
||||
`find_status` now calls too) and is only tried when GoToSocial crossposting
|
||||
is configured.
|
||||
|
||||
Mastodon statuses keep their content warnings: when `spoiler_text` is set,
|
||||
the snippet is "CW: …" and a `sensitive` post's image is left out. Content
|
||||
HTML is sanitized with the webmention sanitizer (nh3; `p br strong em b i a
|
||||
code pre blockquote`, `a[href]`) and cut to about 500 characters. Plain text
|
||||
(Bluesky, OpenGraph) is escaped and split into paragraphs.
|
||||
|
||||
An `mf2` or `opengraph` title that is only the start of the content
|
||||
(an mf2 implied name, a description that repeats the title) is dropped.
|
||||
The URL's host is no longer used as the author's name.
|
||||
|
||||
## Storage (migration 024)
|
||||
|
||||
`reply_contexts` gains `author_handle`, `photo`, `photo_alt` and `source`.
|
||||
Error rows get `source='error'`.
|
||||
|
||||
A row with no `source` was stored before v2.19.0. `fetch_reply_context`
|
||||
fetches such rows again instead of returning them, and the webmention
|
||||
worker calls `refresh_stale()` on every pass, which fetches up to five rows
|
||||
that a note still uses. Each row is fetched once: success stores a source,
|
||||
and a failure keeps the old data and marks it `legacy`.
|
||||
|
||||
## Images
|
||||
|
||||
The CSP only allows images from this site (`img-src 'self' data:`), so
|
||||
remote avatars and images can't be shown directly. As with response avatars
|
||||
(ADR-073 Decision 6), they are saved locally:
|
||||
|
||||
- Author avatars go into the existing `DATA_PATH/avatars/` (96 px squares).
|
||||
- Post images go into `DATA_PATH/previews/`, at most 640 px wide, as WebP,
|
||||
served from `/previews/<sha256>.webp`. Downloads are capped at 8 MB.
|
||||
|
||||
The images are saved right after the fetch. `avatars.fill_missing` also
|
||||
covers reply contexts that notes use, so the worker retries what failed, and
|
||||
`remove_unused` keeps those files (before this release it would have deleted
|
||||
any avatar that wasn't a webmention's). Until there is a copy, the card shows
|
||||
the coloured initial and leaves the image out.
|
||||
|
||||
## Display
|
||||
|
||||
`templates/partials/reply_context.html`:
|
||||
|
||||
- `reply_card(url, ctx)` on the note page: "↩ In reply to", then a card with
|
||||
the avatar, name (linked to the author) and handle, the title if there is
|
||||
one, the content (sanitized HTML), the image with its alt text, and a footer
|
||||
linking to the original with its date and site. Without a fetched context
|
||||
it is the bare link, as before.
|
||||
- `reply_line(url, ctx)` on the home, tag and likes pages: "↩ In reply to
|
||||
<author>" (or the title, "by <author>"), then up to 140 characters of the
|
||||
post as a quote. Without a context it is the raw URL, as before.
|
||||
|
||||
`attach_contexts` now gives replies their context as well as likes and
|
||||
reposts, and the note page uses that when webmentions are disabled.
|
||||
|
||||
Microformats are unchanged for consumers: the card is still the note's
|
||||
`u-in-reply-to h-cite`. Its author h-card and `e-content` are nested inside
|
||||
the h-cite, so they don't become the note's properties. Post Type Discovery
|
||||
still reads the note as a reply (`tests/test_reply_context_sources.py`).
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_reply_context_sources.py`, through the mocked network: the
|
||||
Mastodon API, content warnings, resolving through the instance, the
|
||||
OpenGraph fallback for fediverse and other pages, Bluesky, the once-only
|
||||
refresh of old rows (including keeping data when the refresh fails), and the
|
||||
note and home pages (content, local image and avatar, mf2 reply).
|
||||
@@ -0,0 +1,78 @@
|
||||
# v2.20.0: Crossposted replies look like replies
|
||||
|
||||
## Why
|
||||
|
||||
Crossposting ignored `in_reply_to`. A reply to
|
||||
`https://defcon.social/@deviantollam/117611186265622833` went to GoToSocial
|
||||
and Bluesky as an ordinary post (text, permalink, hashtags), with nothing to
|
||||
say it answered anyone.
|
||||
|
||||
## Behaviour
|
||||
|
||||
For each target, in order:
|
||||
|
||||
| Replying to | GoToSocial | Bluesky | Threads |
|
||||
|---|---|---|---|
|
||||
| One of our notes (its page, or a crosspost of it on any network) | Native reply to that note's GoToSocial status | Native reply to its Bluesky post | Native reply (`reply_to_id`) to its Threads post |
|
||||
| A fediverse post | Native reply (`in_reply_to_id`, resolved by our instance) starting with `@acct` | Reply line | Reply line |
|
||||
| A Bluesky post | Reply line | Native reply (`reply.root`/`reply.parent`) | Reply line |
|
||||
| Anything else | Reply line | Reply line | Reply line |
|
||||
|
||||
The reply line is `↩ Replying to <who>`, where *who* is the reply
|
||||
context's author name, else its title, else the URL's host:
|
||||
|
||||
- GoToSocial and Threads: the line, then the URL, then a blank line and the note.
|
||||
On Threads the URL counts towards Meta's limit of five links, so the
|
||||
body gets one link fewer.
|
||||
- Bluesky: the line without the URL. *Who* is a link facet to the post,
|
||||
which saves up to ~100 of the 300 characters.
|
||||
|
||||
`compose_text(..., lead=...)` keeps the line whole and shortens the body
|
||||
instead when the post is too long.
|
||||
|
||||
### The @mention
|
||||
|
||||
The user chose to start a native fediverse reply with the author's
|
||||
mention, as Mastodon and GoToSocial clients do. Without it the author isn't
|
||||
notified, and their server may never receive the reply, so it wouldn't show
|
||||
in their thread. There's no mention when replying to our own account (the
|
||||
search result's account id equals `verify_credentials`), or when the note
|
||||
already contains the mention.
|
||||
|
||||
### Our own posts
|
||||
|
||||
`_own_note_id(url)` maps a URL to a note: a synced `syndications.remote_url`
|
||||
on any target, or `/note/<slug>` and `/article/<slug>` on this site.
|
||||
`_reply_fields` then takes that note's `remote_id` on the target being
|
||||
posted to, as `Post.reply_remote_id`. GoToSocial skips the ids of
|
||||
favourites and boosts (`favourite:…`, `reblog:…`). Bluesky only takes
|
||||
`app.bsky.feed.post` URIs. Threads only takes numeric media ids. The
|
||||
Threads API can't look up other people's posts by URL, so only our own
|
||||
posts thread there.
|
||||
|
||||
### Bluesky threads
|
||||
|
||||
A reply's `root` must be the top of the thread. The parent is fetched with
|
||||
`app.bsky.feed.getPosts`. If the parent is itself a reply, its
|
||||
`record.reply.root` is used; otherwise the parent is the root.
|
||||
|
||||
## Not changed
|
||||
|
||||
- Crossposts made before this release. Edits can't move a status into a
|
||||
thread (GoToSocial, Mastodon), and Bluesky and Threads posts can't be edited at
|
||||
all. Re-threading would mean deleting and reposting, which loses likes and
|
||||
boosts. The user chose to leave them.
|
||||
- An edit of a reply on GoToSocial keeps the status where it is, and its
|
||||
text keeps the mention (`_prepare` resolves the parent again).
|
||||
- Fediverse authors bridged into Bluesky by Bridgy Fed have a copy of
|
||||
their post there. There is no reliable way to find its URI from the
|
||||
original URL, so these get the reply line.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_reply_crossposts.py`: native fediverse reply with mention, no
|
||||
mention for our own account, reply line with name and host fallback,
|
||||
Bluesky root/parent, Bluesky reply line with its link facet, Threads native
|
||||
reply only to our own posts, the line surviving shortening, and a reply to
|
||||
our own note threading on GoToSocial and Bluesky. The reply is addressed both
|
||||
by the note's permalink and by its GoToSocial crosspost URL.
|
||||
@@ -0,0 +1,27 @@
|
||||
# v2.20.1: Moving pictures in reply contexts
|
||||
|
||||
## Why
|
||||
|
||||
The reply https://starpunk.thesatelliteoflove.com/note/20261114032356 answers
|
||||
an infosec.exchange post with a GIF, but the card showed a still. Mastodon
|
||||
converts uploaded GIFs to a silent looping MP4 (`media_attachments[].type ==
|
||||
"gifv"`, here 320×240, 5.3 s, ~140 KB). Its `preview_url` is a PNG of the
|
||||
first frame, which is all v2.19.0 kept.
|
||||
|
||||
## Changes
|
||||
|
||||
| Area | Change |
|
||||
|---|---|
|
||||
| `reply_context._from_status` | A `gifv` attachment's `url` goes into the new `video` field (the preview stays in `photo`, used as the poster) |
|
||||
| Migration 025 | `reply_contexts.video`; fediverse rows with a photo get `source = NULL`, so the worker's `refresh_stale` fetches them once more |
|
||||
| `avatars.fetch_video` | Saves the MP4 as `DATA_PATH/previews/<sha256>.mp4` (8 MB cap, must have an `ftyp` box); `fill_missing`/`remove_unused` cover it |
|
||||
| `avatars._save` | Animated GIF/WebP/PNG previews are saved as animated WebP (frames narrowed to 640 px, at most 300 frames) instead of their first frame |
|
||||
| `/previews/<name>` | Also serves `<sha256>.mp4` as `video/mp4` |
|
||||
| Template | `<video autoplay loop muted playsinline poster=…>` with the alt text as `aria-label`, when there is a saved MP4 |
|
||||
|
||||
The MP4 is stored as it is: re-encoding would need ffmpeg in the image, and
|
||||
gifvs are small. Like the images, it has to be served from this site
|
||||
(`default-src 'self'` covers media).
|
||||
|
||||
Real videos (`type == "video"`) and Bluesky videos still show their
|
||||
thumbnail: they can be long and have sound.
|
||||
@@ -0,0 +1,38 @@
|
||||
# v2.20.2: Backfeed whole reply threads
|
||||
|
||||
## Why
|
||||
|
||||
https://starpunk.thesatelliteoflove.com/note/20261110191424 showed one reply
|
||||
(curiouskatej, via Threads). The Threads conversation had three:
|
||||
|
||||
```
|
||||
curiouskatej: "It keeps coming up on music compilations…" (kept)
|
||||
└ philskents: "That is terrifying..." (missed)
|
||||
└ curiouskatej: "To be fair it's just as bad as…" (missed)
|
||||
```
|
||||
|
||||
Each client read only the first level of replies: Threads `/{id}/replies`,
|
||||
GoToSocial descendants with `in_reply_to_id == our id`, and Bluesky
|
||||
`getPostThread` at depth 1. All of them also dropped the owner's own replies,
|
||||
which lost one side of the conversation.
|
||||
|
||||
## Changes
|
||||
|
||||
| Area | Change |
|
||||
|---|---|
|
||||
| `backfeed.thread_members(posts, root)` | Decides which replies to keep, and what each answers. Kept: others' replies to our copy, plus any reply (ours too) to a kept reply. Our own replies to our copy are the thread continuing, so they and their replies are left out, as before |
|
||||
| `Response.parent_id`, `.own` | The silo id of the reply this one answers (None for our copy itself), and whether we wrote it |
|
||||
| Threads | `/{id}/conversation` (every depth, newest first) with `replied_to` in the fields |
|
||||
| GoToSocial | All public descendants from `/context` |
|
||||
| Bluesky | `getPostThread` depth `BLUESKY_THREAD_DEPTH` (10), walked recursively; blocked or deleted posts drop their subtree |
|
||||
| Migration 026 | `webmentions.parent_remote_id` |
|
||||
| `backfeed.store` | Stores the parent; own replies are stored as already notified |
|
||||
| `webmention_queries.reply_threads` | Nests replies under their parent (same origin and remote id). A reply whose parent isn't shown goes to the top level |
|
||||
| Templates/CSS | `responses.reply` renders children in `.webmention-thread`, indented with a side rule, as `u-comment h-cite` inside the parent's h-cite |
|
||||
|
||||
The heading still counts every reply ("3 Replies"), and so does the count
|
||||
line under the note.
|
||||
|
||||
Existing crossposts get their threads at their next scheduled check. The
|
||||
check schedule slows with the note's age (ADR-073), and notes past the
|
||||
cutoff aren't checked.
|
||||
@@ -0,0 +1,9 @@
|
||||
-- Migration 024: Richer reply contexts (v2.19.0)
|
||||
-- Fediverse and Bluesky posts have no microformats, so their context now
|
||||
-- comes from the Mastodon or Bluesky API, or the page's OpenGraph tags.
|
||||
-- source records which one was used; rows from before have none, and the
|
||||
-- webmention worker fetches those again once.
|
||||
ALTER TABLE reply_contexts ADD COLUMN author_handle TEXT;
|
||||
ALTER TABLE reply_contexts ADD COLUMN photo TEXT;
|
||||
ALTER TABLE reply_contexts ADD COLUMN photo_alt TEXT;
|
||||
ALTER TABLE reply_contexts ADD COLUMN source TEXT;
|
||||
@@ -0,0 +1,6 @@
|
||||
-- Migration 025: Looping videos in reply contexts (v2.20.1)
|
||||
-- Mastodon turns GIFs into silent looping MP4s ("gifv"); video holds its URL,
|
||||
-- shown instead of the still preview. Fediverse contexts with an image are
|
||||
-- fetched again once (no source) so their gifv is picked up.
|
||||
ALTER TABLE reply_contexts ADD COLUMN video TEXT;
|
||||
UPDATE reply_contexts SET source = NULL WHERE source = 'fediverse' AND photo IS NOT NULL;
|
||||
@@ -0,0 +1,4 @@
|
||||
-- Migration 026: Threads of backfed replies (v2.20.2)
|
||||
-- parent_remote_id: the silo id of the response a reply answers; NULL when it
|
||||
-- answers our crosspost itself. Replies are shown nested under their parent.
|
||||
ALTER TABLE webmentions ADD COLUMN parent_remote_id TEXT;
|
||||
+18
-2
@@ -225,6 +225,22 @@ def create_app(config=None):
|
||||
name = cached_name(app.config.get('DATA_PATH', 'data'), url)
|
||||
return url_for('public.avatar', name=name) if name else None
|
||||
|
||||
# Template filter - local copy of a replied-to post's image, or None (v2.19.0)
|
||||
@app.template_filter('local_preview')
|
||||
def local_preview_filter(url):
|
||||
from flask import url_for
|
||||
from starpunk.avatars import cached_preview
|
||||
name = cached_preview(app.config.get('DATA_PATH', 'data'), url)
|
||||
return url_for('public.preview', name=name) if name else None
|
||||
|
||||
# Template filter - local copy of a replied-to post's gifv, or None (v2.20.1)
|
||||
@app.template_filter('local_video')
|
||||
def local_video_filter(url):
|
||||
from flask import url_for
|
||||
from starpunk.avatars import cached_video
|
||||
name = cached_video(app.config.get('DATA_PATH', 'data'), url)
|
||||
return url_for('public.preview', name=name) if name else None
|
||||
|
||||
# Template filter - URL to color for facepile (v1.6.0)
|
||||
@app.template_filter('url_to_color')
|
||||
def url_to_color_filter(url):
|
||||
@@ -468,5 +484,5 @@ def create_app(config=None):
|
||||
|
||||
# Package version (Semantic Versioning 2.0.0)
|
||||
# See docs/standards/versioning-strategy.md for details
|
||||
__version__ = "2.17.1"
|
||||
__version_info__ = (2, 17, 1)
|
||||
__version__ = "2.20.2"
|
||||
__version_info__ = (2, 20, 2)
|
||||
+164
-41
@@ -7,6 +7,9 @@ Each avatar is fetched once (SSRF-safe), shrunk to a 96 px WebP and saved
|
||||
as DATA_PATH/avatars/<sha256 of its URL>.webp. Pages show the saved copy,
|
||||
or the coloured initial until there is one.
|
||||
|
||||
The image of a post being replied to is kept the same way (v2.19.0): at most
|
||||
PREVIEW_WIDTH wide, in DATA_PATH/previews/.
|
||||
|
||||
Paths and URLs are passed in, so the webmention worker can use it. Fetching
|
||||
needs an app context (the SSRF-safe fetcher reads DEV_MODE).
|
||||
"""
|
||||
@@ -24,6 +27,12 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
SIZE = 96
|
||||
MAX_BYTES = 1024 * 1024
|
||||
PREVIEW_WIDTH = 640
|
||||
PREVIEW_MAX_BYTES = 8 * 1024 * 1024
|
||||
MAX_FRAMES = 300
|
||||
# Mastodon's "GIFs" are silent looping MP4s (gifv); kept as they are (v2.20.1)
|
||||
VIDEO_MAX_BYTES = 8 * 1024 * 1024
|
||||
VIDEO_NAME_RE = re.compile(r"^[0-9a-f]{64}\.mp4$")
|
||||
# Bluesky avatar on its CDN: .../img/<kind>/plain/<did>/<cid>[@jpeg]
|
||||
BSKY_CDN_RE = re.compile(
|
||||
r"^https://cdn\.bsky\.app/img/[a-z_]+/plain/(did:(?:plc|web):[A-Za-z0-9._:%-]+)/([a-z2-7]+)(?:@\w+)?$")
|
||||
@@ -39,12 +48,113 @@ def folder(data_path) -> Path:
|
||||
return Path(data_path) / "avatars"
|
||||
|
||||
|
||||
def cached_name(data_path, url: Optional[str]) -> Optional[str]:
|
||||
"""File name of the saved copy of this avatar, if there is one"""
|
||||
def preview_folder(data_path) -> Path:
|
||||
return Path(data_path) / "previews"
|
||||
|
||||
|
||||
def _cached(directory: Path, url: Optional[str]) -> Optional[str]:
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return None
|
||||
name = name_for(url)
|
||||
return name if (folder(data_path) / name).is_file() else None
|
||||
return name if (directory / name).is_file() else None
|
||||
|
||||
|
||||
def cached_name(data_path, url: Optional[str]) -> Optional[str]:
|
||||
"""File name of the saved copy of this avatar, if there is one"""
|
||||
return _cached(folder(data_path), url)
|
||||
|
||||
|
||||
def cached_preview(data_path, url: Optional[str]) -> Optional[str]:
|
||||
"""File name of the saved copy of a replied-to post's image, if there is one"""
|
||||
return _cached(preview_folder(data_path), url)
|
||||
|
||||
|
||||
def video_name_for(url: str) -> str:
|
||||
return hashlib.sha256(url.encode()).hexdigest() + ".mp4"
|
||||
|
||||
|
||||
def cached_video(data_path, url: Optional[str]) -> Optional[str]:
|
||||
"""File name of the saved copy of a replied-to post's looping video, if there is one"""
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return None
|
||||
name = video_name_for(url)
|
||||
return name if (preview_folder(data_path) / name).is_file() else None
|
||||
|
||||
|
||||
def fetch_video(data_path, url: Optional[str], site_url: str) -> bool:
|
||||
"""Save a copy of a replied-to post's gifv (a short silent MP4) unless there is one"""
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return False
|
||||
path = preview_folder(data_path) / video_name_for(url)
|
||||
if path.is_file():
|
||||
return True
|
||||
try:
|
||||
data = _download(url, site_url, VIDEO_MAX_BYTES)
|
||||
if data[4:8] != b"ftyp":
|
||||
raise ValueError("not an MP4")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_bytes(data)
|
||||
tmp.replace(path)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.info(f"Could not save video {url}: {type(e).__name__}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def _square(image):
|
||||
from PIL import Image
|
||||
side = min(image.size)
|
||||
left, top = (image.width - side) // 2, (image.height - side) // 2
|
||||
image = image.crop((left, top, left + side, top + side))
|
||||
return image.resize((SIZE, SIZE), Image.LANCZOS)
|
||||
|
||||
|
||||
def _narrow(image):
|
||||
from PIL import Image
|
||||
if image.width <= PREVIEW_WIDTH:
|
||||
return image
|
||||
height = max(1, round(image.height * PREVIEW_WIDTH / image.width))
|
||||
return image.resize((PREVIEW_WIDTH, height), Image.LANCZOS)
|
||||
|
||||
|
||||
def _save_animated(image, tmp: Path) -> None:
|
||||
"""An animated GIF/WebP/PNG as an animated WebP, frames narrowed (v2.20.1)"""
|
||||
from PIL import ImageSequence
|
||||
frames, durations = [], []
|
||||
for frame in ImageSequence.Iterator(image):
|
||||
if len(frames) >= MAX_FRAMES:
|
||||
break
|
||||
frames.append(_narrow(frame.convert("RGBA")))
|
||||
durations.append(frame.info.get("duration") or image.info.get("duration") or 100)
|
||||
frames[0].save(tmp, "WEBP", save_all=True, append_images=frames[1:], duration=durations,
|
||||
loop=0, quality=80)
|
||||
|
||||
|
||||
def _save(path: Path, url: Optional[str], site_url: str, shape, max_bytes: int,
|
||||
what: str) -> bool:
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return False
|
||||
if path.is_file():
|
||||
return True
|
||||
from PIL import Image
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(_download(url, site_url, max_bytes))) as image:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
if shape is _narrow and getattr(image, "n_frames", 1) > 1:
|
||||
_save_animated(image, tmp)
|
||||
else:
|
||||
image.load()
|
||||
image = image.convert("RGBA" if "A" in image.getbands() else "RGB")
|
||||
image = shape(image)
|
||||
image.save(tmp, "WEBP", quality=80)
|
||||
tmp.replace(path)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.info(f"Could not save {what} {url}: {type(e).__name__}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def fetch(data_path, url: Optional[str], site_url: str) -> bool:
|
||||
@@ -52,29 +162,18 @@ def fetch(data_path, url: Optional[str], site_url: str) -> bool:
|
||||
Save a copy of an avatar unless there is one. Returns True if one exists
|
||||
afterwards. Failures are logged and the page keeps showing the initial.
|
||||
"""
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
if not url:
|
||||
return False
|
||||
path = folder(data_path) / name_for(url)
|
||||
if path.is_file():
|
||||
return True
|
||||
from PIL import Image
|
||||
return _save(folder(data_path) / name_for(url), url, site_url, _square,
|
||||
MAX_BYTES, "avatar")
|
||||
|
||||
try:
|
||||
with Image.open(io.BytesIO(_download(url, site_url))) as image:
|
||||
image.load()
|
||||
image = image.convert("RGBA" if "A" in image.getbands() else "RGB")
|
||||
side = min(image.size)
|
||||
left, top = (image.width - side) // 2, (image.height - side) // 2
|
||||
image = image.crop((left, top, left + side, top + side))
|
||||
image = image.resize((SIZE, SIZE), Image.LANCZOS)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_suffix(".tmp")
|
||||
image.save(tmp, "WEBP", quality=80)
|
||||
tmp.replace(path)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.info(f"Could not save avatar {url}: {type(e).__name__}: {e}")
|
||||
|
||||
def fetch_preview(data_path, url: Optional[str], site_url: str) -> bool:
|
||||
"""Save a copy of a replied-to post's image unless there is one (v2.19.0)"""
|
||||
if not url:
|
||||
return False
|
||||
return _save(preview_folder(data_path) / name_for(url), url, site_url, _narrow,
|
||||
PREVIEW_MAX_BYTES, "preview")
|
||||
|
||||
|
||||
def _own_instance_host() -> Optional[str]:
|
||||
@@ -85,11 +184,11 @@ def _own_instance_host() -> Optional[str]:
|
||||
return urlparse(current_app.config.get("GOTOSOCIAL_URL") or "").hostname
|
||||
|
||||
|
||||
def _get(url: str, headers: dict) -> bytes:
|
||||
def _get(url: str, headers: dict, max_bytes: int = MAX_BYTES) -> bytes:
|
||||
"""GET a public URL through the SSRF-safe fetcher"""
|
||||
from starpunk.webmention_ssrf import safe_get
|
||||
|
||||
result = safe_get(url, headers=headers, max_bytes=MAX_BYTES, max_redirects=5,
|
||||
result = safe_get(url, headers=headers, max_bytes=max_bytes, max_redirects=5,
|
||||
timeout=10.0, truncate=False)
|
||||
if result.status_code >= 400:
|
||||
raise ValueError(f"HTTP {result.status_code}")
|
||||
@@ -110,7 +209,7 @@ def _pds(did: str, headers: dict) -> str:
|
||||
raise ValueError(f"no PDS for {did}")
|
||||
|
||||
|
||||
def _download(url: str, site_url: str) -> bytes:
|
||||
def _download(url: str, site_url: str, max_bytes: int = MAX_BYTES) -> bytes:
|
||||
"""
|
||||
The avatar's bytes (v2.13.1):
|
||||
- from StarPunk's own GoToSocial directly, as the crossposting client
|
||||
@@ -136,13 +235,13 @@ def _download(url: str, site_url: str) -> bytes:
|
||||
# GoToSocial sends media it hasn't cached to the original
|
||||
# server: that one is fetched like any outside URL (v2.13.2)
|
||||
from urllib.parse import urljoin
|
||||
return _get(urljoin(url, response.headers["location"]), headers)
|
||||
return _get(urljoin(url, response.headers["location"]), headers, max_bytes)
|
||||
if response.status_code >= 400:
|
||||
raise ValueError(f"HTTP {response.status_code}")
|
||||
data = b""
|
||||
for chunk in response.iter_bytes():
|
||||
data += chunk
|
||||
if len(data) > MAX_BYTES:
|
||||
if len(data) > max_bytes:
|
||||
raise ValueError("too large")
|
||||
return data
|
||||
match = BSKY_CDN_RE.match(url)
|
||||
@@ -150,11 +249,27 @@ def _download(url: str, site_url: str) -> bytes:
|
||||
did, cid = match.groups()
|
||||
url = (f"{_pds(did, headers)}/xrpc/com.atproto.sync.getBlob"
|
||||
f"?did={quote(did, safe=':')}&cid={cid}")
|
||||
return _get(url, headers)
|
||||
return _get(url, headers, max_bytes)
|
||||
|
||||
|
||||
def _context_images(conn) -> tuple[list, list, list]:
|
||||
"""Avatars, post images and gifvs of reply contexts that notes use (v2.19.0)"""
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT rc.author_photo, rc.photo, rc.video FROM reply_contexts rc
|
||||
WHERE rc.fetch_status = 'fetched' AND EXISTS (
|
||||
SELECT 1 FROM notes n WHERE n.deleted_at IS NULL AND rc.url IN
|
||||
(n.in_reply_to, n.like_of, n.repost_of, n.bookmark_of))
|
||||
ORDER BY rc.id DESC
|
||||
"""
|
||||
).fetchall()
|
||||
return ([r[0] for r in rows if r[0]], [r[1] for r in rows if r[1]],
|
||||
[r[2] for r in rows if r[2]])
|
||||
|
||||
|
||||
def fill_missing(db_path, data_path, site_url: str, limit: int = 20) -> int:
|
||||
"""Save avatars of shown responses that have no copy yet. Returns how many were tried."""
|
||||
"""Save avatars of shown responses and reply contexts, and the images of
|
||||
replied-to posts, that have no copy yet. Returns how many were tried."""
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
urls = [row[0] for row in conn.execute(
|
||||
@@ -165,35 +280,43 @@ def fill_missing(db_path, data_path, site_url: str, limit: int = 20) -> int:
|
||||
ORDER BY id DESC
|
||||
"""
|
||||
)]
|
||||
context_avatars, context_photos, context_videos = _context_images(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
wanted = [(url, cached_name, fetch) for url in dict.fromkeys(urls + context_avatars)]
|
||||
wanted += [(url, cached_preview, fetch_preview) for url in dict.fromkeys(context_photos)]
|
||||
wanted += [(url, cached_video, fetch_video) for url in dict.fromkeys(context_videos)]
|
||||
tried = 0
|
||||
for url in urls:
|
||||
for url, cached, save in wanted:
|
||||
if tried >= limit:
|
||||
break
|
||||
if cached_name(data_path, url) or not url.startswith(("http://", "https://")):
|
||||
if cached(data_path, url) or not url.startswith(("http://", "https://")):
|
||||
continue
|
||||
fetch(data_path, url, site_url)
|
||||
save(data_path, url, site_url)
|
||||
tried += 1
|
||||
return tried
|
||||
|
||||
|
||||
def remove_unused(db_path, data_path) -> int:
|
||||
"""Delete saved avatars no stored response uses. Returns how many were removed."""
|
||||
directory = folder(data_path)
|
||||
if not directory.is_dir():
|
||||
return 0
|
||||
"""Delete saved avatars and previews nothing uses. Returns how many were removed."""
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
used = {name_for(row[0]) for row in conn.execute(
|
||||
"SELECT DISTINCT author_photo FROM webmentions WHERE author_photo IS NOT NULL")}
|
||||
context_avatars, context_photos, context_videos = _context_images(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
used |= {name_for(url) for url in context_avatars}
|
||||
used_previews = {name_for(url) for url in context_photos}
|
||||
used_previews |= {video_name_for(url) for url in context_videos}
|
||||
removed = 0
|
||||
for path in directory.iterdir():
|
||||
if NAME_RE.match(path.name) and path.name not in used:
|
||||
path.unlink(missing_ok=True)
|
||||
removed += 1
|
||||
for directory, keep in ((folder(data_path), used), (preview_folder(data_path), used_previews)):
|
||||
if not directory.is_dir():
|
||||
continue
|
||||
for path in directory.iterdir():
|
||||
if (NAME_RE.match(path.name) or VIDEO_NAME_RE.match(path.name)) and path.name not in keep:
|
||||
path.unlink(missing_ok=True)
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
|
||||
|
||||
+33
-3
@@ -58,6 +58,34 @@ class Response:
|
||||
opted_out: bool = False # set by the client from silo-specific signals
|
||||
content_html: Optional[str] = None
|
||||
published: Optional[str] = None
|
||||
# Replies in a thread (v2.20.2): the silo id of the response this one
|
||||
# answers (None: our copy itself), and whether we wrote it
|
||||
parent_id: Optional[str] = None
|
||||
own: bool = False
|
||||
|
||||
|
||||
def thread_members(posts: List[dict], root: str) -> Dict[str, Optional[str]]:
|
||||
"""
|
||||
Which replies under one of our copies to keep, and what each answers
|
||||
(v2.20.2). `posts` are {'id', 'parent', 'own'} for every reply in the
|
||||
conversation, in any order. Kept: others' replies to our copy, and any
|
||||
reply (ours too) to a kept reply. Our own replies to our copy are the
|
||||
thread continuing, not responses. Returns {id: parent id or None}.
|
||||
"""
|
||||
kept: Dict[str, Optional[str]] = {}
|
||||
pending = list(posts)
|
||||
while True:
|
||||
rest = []
|
||||
for post in pending:
|
||||
if post["parent"] == root and not post["own"]:
|
||||
kept[post["id"]] = None
|
||||
elif post["parent"] in kept and post["parent"] != post["id"]:
|
||||
kept[post["id"]] = post["parent"]
|
||||
else:
|
||||
rest.append(post)
|
||||
if len(rest) == len(pending):
|
||||
return kept
|
||||
pending = rest
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -292,9 +320,9 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
|
||||
INSERT INTO webmentions (
|
||||
source_url, target_url, target_note_id, mention_type, author_name, author_url,
|
||||
author_photo, content_snippet, validation_status, moderation_status,
|
||||
validated_at, source_published, origin, remote_id, notified_at)
|
||||
validated_at, source_published, origin, remote_id, notified_at, parent_remote_id)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'valid', ?, CURRENT_TIMESTAMP, ?, ?, ?,
|
||||
CASE WHEN ? THEN CURRENT_TIMESTAMP END)
|
||||
CASE WHEN ? THEN CURRENT_TIMESTAMP END, ?)
|
||||
ON CONFLICT(source_url, target_url) DO UPDATE SET
|
||||
target_note_id = excluded.target_note_id,
|
||||
mention_type = excluded.mention_type,
|
||||
@@ -305,6 +333,7 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
|
||||
source_published = COALESCE(excluded.source_published, webmentions.source_published),
|
||||
origin = excluded.origin,
|
||||
remote_id = excluded.remote_id,
|
||||
parent_remote_id = excluded.parent_remote_id,
|
||||
validation_status = 'valid',
|
||||
validation_error = NULL,
|
||||
validated_at = CURRENT_TIMESTAMP,
|
||||
@@ -312,7 +341,8 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
|
||||
""",
|
||||
(r.source_url, target_url, note_id, r.kind, r.author_name or None, r.author_url or None,
|
||||
r.author_photo or None, snippet, moderation, r.published, origin, r.remote_id,
|
||||
1 if quiet else 0),
|
||||
# Our own replies in a thread are no news to us
|
||||
1 if quiet or r.own else 0, r.parent_id),
|
||||
)
|
||||
if dropped:
|
||||
marks = ",".join("?" * len(dropped))
|
||||
|
||||
@@ -106,6 +106,10 @@ def load_config(app, config_override=None):
|
||||
|
||||
# Webmention configuration (v1.6.0)
|
||||
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
|
||||
# Hosts never sent webmentions, besides this site itself (v2.18.3)
|
||||
app.config["WEBMENTION_SKIP_HOSTS"] = [
|
||||
h.strip().lower() for h in os.getenv("WEBMENTION_SKIP_HOSTS", "").split(",") if h.strip()
|
||||
]
|
||||
|
||||
# WebSub (v2.10.0, ADR-071): built-in hub unless WEBSUB_HUB_URL names another
|
||||
app.config["WEBSUB_ENABLED"] = os.getenv("WEBSUB_ENABLED", "true").lower() == "true"
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
"""
|
||||
Error log for the admin Log page (v2.18.0)
|
||||
|
||||
Collects what is failing right now from the tables that already record it:
|
||||
crossposts, response checks (backfeed), outgoing webmentions, reply-context
|
||||
fetches, WebSub deliveries and notifications. Nothing new is stored. An entry
|
||||
disappears once the next attempt succeeds, so the page lists only problems
|
||||
that still need attention.
|
||||
|
||||
Each entry is a dict:
|
||||
when -- timestamp text (UTC) or "" when unknown
|
||||
area -- short label, e.g. "Crosspost to Bluesky"
|
||||
note -- the affected Note, or None for site-wide problems
|
||||
detail -- extra context (a URL or "retrying"), may be ""
|
||||
error -- the recorded error message
|
||||
action -- optional (endpoint, id, label) for a retry button
|
||||
"""
|
||||
|
||||
from flask import current_app
|
||||
|
||||
from starpunk.database import get_db
|
||||
|
||||
SILOS = {"gotosocial": "GoToSocial", "bluesky": "Bluesky", "threads": "Threads"}
|
||||
|
||||
|
||||
def _silo(target: str) -> str:
|
||||
return SILOS.get(target, target)
|
||||
|
||||
|
||||
def _notes(ids) -> dict:
|
||||
"""Load the affected notes once each (deleted notes come back as None)"""
|
||||
from starpunk.notes import get_note
|
||||
|
||||
return {i: get_note(id=i) for i in set(ids) if i is not None}
|
||||
|
||||
|
||||
def entries() -> list:
|
||||
"""Every current error, newest first; site-wide problems come first"""
|
||||
from starpunk import backfeed, notifications
|
||||
|
||||
db = get_db(current_app)
|
||||
rows = []
|
||||
|
||||
for r in db.execute(
|
||||
"SELECT id, note_id, target, state, last_error, updated_at FROM syndications "
|
||||
"WHERE last_error IS NOT NULL AND state != 'synced'"
|
||||
):
|
||||
rows.append({
|
||||
"when": r["updated_at"], "area": f"Crosspost to {_silo(r['target'])}",
|
||||
"note_id": r["note_id"], "detail": "retrying" if r["state"] == "pending" else "",
|
||||
"error": r["last_error"],
|
||||
"action": ("admin.retry_syndication", r["id"], "Retry"),
|
||||
})
|
||||
|
||||
for r in db.execute(
|
||||
"SELECT id, note_id, target, backfeed_error, backfeed_checked_at, remote_url FROM syndications "
|
||||
"WHERE backfeed_error IS NOT NULL"
|
||||
):
|
||||
rows.append({
|
||||
"when": r["backfeed_checked_at"], "area": f"Response check on {_silo(r['target'])}",
|
||||
"note_id": r["note_id"], "detail": r["remote_url"] or "", "error": r["backfeed_error"],
|
||||
"action": ("admin.check_syndication_responses", r["id"], "Check now"),
|
||||
})
|
||||
|
||||
for r in db.execute(
|
||||
"SELECT source_note_id, target_url, status, last_error, COALESCE(started_at, created_at) AS at "
|
||||
"FROM webmention_queue WHERE last_error IS NOT NULL AND status IN ('pending', 'failed')"
|
||||
):
|
||||
rows.append({
|
||||
"when": r["at"], "area": "Webmention sent", "note_id": r["source_note_id"],
|
||||
"detail": r["target_url"] + (" (retrying)" if r["status"] == "pending" else ""),
|
||||
"error": r["last_error"], "action": None,
|
||||
})
|
||||
|
||||
for r in db.execute(
|
||||
"SELECT n.id AS note_id, rc.url, rc.fetch_error, rc.fetched_at, rc.created_at "
|
||||
"FROM reply_contexts rc JOIN notes n ON n.in_reply_to = rc.url "
|
||||
"WHERE rc.fetch_status = 'error' AND n.deleted_at IS NULL"
|
||||
):
|
||||
rows.append({
|
||||
"when": r["fetched_at"] or r["created_at"], "area": "Reply context fetch",
|
||||
"note_id": r["note_id"], "detail": r["url"], "error": r["fetch_error"] or "fetch failed",
|
||||
"action": None,
|
||||
})
|
||||
|
||||
for r in db.execute(
|
||||
"SELECT topic, callback, last_error, COALESCE(last_success_at, verified_at, created_at) AS at "
|
||||
"FROM websub_subscriptions WHERE last_error IS NOT NULL"
|
||||
):
|
||||
rows.append({
|
||||
"when": r["at"], "area": "WebSub delivery", "note_id": None,
|
||||
"detail": f"{r['topic']} → {r['callback']}", "error": r["last_error"], "action": None,
|
||||
})
|
||||
|
||||
rows.sort(key=lambda e: str(e["when"] or ""), reverse=True)
|
||||
|
||||
site = []
|
||||
for silo, message in backfeed.denied.items():
|
||||
site.append({"when": "", "area": f"Response checks on {_silo(silo)}", "note_id": None,
|
||||
"detail": "paused", "error": message, "action": None})
|
||||
notify_error = notifications.last_error(db)
|
||||
if notify_error:
|
||||
when, _, message = notify_error.partition(" UTC ")
|
||||
site.append({"when": when if message else "", "area": "Notifications", "note_id": None,
|
||||
"detail": "", "error": message or notify_error, "action": None})
|
||||
|
||||
entries = site + rows
|
||||
notes = _notes(e["note_id"] for e in entries)
|
||||
for e in entries:
|
||||
e["note"] = notes.get(e.pop("note_id"))
|
||||
return entries
|
||||
|
||||
|
||||
def count() -> int:
|
||||
"""Cheap total for the admin nav badge"""
|
||||
from starpunk import backfeed, notifications
|
||||
|
||||
db = get_db(current_app)
|
||||
total = db.execute(
|
||||
"SELECT (SELECT COUNT(*) FROM syndications WHERE last_error IS NOT NULL AND state != 'synced')"
|
||||
" + (SELECT COUNT(*) FROM syndications WHERE backfeed_error IS NOT NULL)"
|
||||
" + (SELECT COUNT(*) FROM webmention_queue WHERE last_error IS NOT NULL AND status IN ('pending', 'failed'))"
|
||||
" + (SELECT COUNT(*) FROM reply_contexts rc JOIN notes n ON n.in_reply_to = rc.url"
|
||||
" WHERE rc.fetch_status = 'error' AND n.deleted_at IS NULL)"
|
||||
" + (SELECT COUNT(*) FROM websub_subscriptions WHERE last_error IS NOT NULL)"
|
||||
).fetchone()[0]
|
||||
return total + len(backfeed.denied) + (1 if notifications.last_error(db) else 0)
|
||||
@@ -232,9 +232,11 @@ def register_tools(server: MCPServer, app: Flask) -> None:
|
||||
) -> dict:
|
||||
"""Create a note (Markdown). It is a draft unless draft is false;
|
||||
publishing makes it public and crossposts it to the syndicate_to
|
||||
targets. A title makes it an article; summary is its short summary,
|
||||
shown on the homepage and in feeds and crossposts. At most one of like_of,
|
||||
repost_of and bookmark_of. published_at is an ISO 8601 date."""
|
||||
targets. A title makes it an article, at /article/<slug>; summary is its
|
||||
short summary, shown on the homepage and in feeds and crossposts. The
|
||||
slug is made from the title for articles and from the time for other
|
||||
notes, so leave slug out unless a specific one is wanted. At most one of
|
||||
like_of, repost_of and bookmark_of. published_at is an ISO 8601 date."""
|
||||
info = _token_info()
|
||||
_require(info, "create", "draft")
|
||||
|
||||
|
||||
+3
-1
@@ -62,7 +62,9 @@ ALLOWED_ATTRIBUTES = {
|
||||
def render_markdown(text: str) -> str:
|
||||
"""Note Markdown (or HTML) to sanitized HTML"""
|
||||
html = markdown.markdown(text, extensions=MARKDOWN_EXTENSIONS)
|
||||
return nh3.clean(html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
|
||||
html = nh3.clean(html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
|
||||
# A wrapper lets a table fill the column yet scroll sideways when too wide (v2.18.2)
|
||||
return re.sub(r"<table\b", '<div class="table-wrap"><table', html).replace("</table>", "</table></div>")
|
||||
|
||||
|
||||
# Likes, reposts and bookmarks (v2.5.0, v2.9.0)
|
||||
|
||||
+457
-212
@@ -7,41 +7,79 @@ Per database-schema.md Story 2.4:
|
||||
- Store in reply_contexts table
|
||||
- Graceful degradation on fetch failure
|
||||
|
||||
Fediverse and Bluesky pages carry no h-entry, so from v2.19.0 the context
|
||||
comes from the first source that knows the post:
|
||||
|
||||
1. Bluesky post URLs: the public Bluesky AppView API
|
||||
2. Pages with an h-entry: microformats (IndieWeb sites)
|
||||
3. Fediverse posts: the origin's public Mastodon API, else this site's
|
||||
GoToSocial instance resolving the URL (as likes and boosts already do;
|
||||
it fetches with signed requests, so it works for GoToSocial and for
|
||||
Mastodon servers that refuse anonymous API calls)
|
||||
4. Anything else: the page's OpenGraph tags, with the page's h-card as author
|
||||
|
||||
The source used is stored in reply_contexts.source. Rows from before v2.19.0
|
||||
have none; refresh_stale() fetches those again once.
|
||||
|
||||
References:
|
||||
- Database Schema: /docs/design/v1.6.0/database-schema.md
|
||||
- Background Worker: /docs/design/v1.6.0/background-worker.md
|
||||
- Design: /docs/design/v2.19.0/2026-11-13-reply-context-sources.md
|
||||
"""
|
||||
|
||||
import html
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Dict, Any
|
||||
from urllib.parse import quote, urljoin, urlparse
|
||||
|
||||
import mf2py
|
||||
|
||||
from starpunk import mf2_utils
|
||||
from starpunk.webmention_processor import (
|
||||
fetch_source,
|
||||
extract_author,
|
||||
extract_content,
|
||||
ValidationError
|
||||
sanitize_content,
|
||||
ValidationError,
|
||||
_is_http_url,
|
||||
)
|
||||
import mf2py
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SNIPPET_LIMIT = 500
|
||||
|
||||
BLUESKY_POST_RE = re.compile(r"^https://bsky\.app/profile/([^/]+)/post/([^/?#]+)")
|
||||
BLUESKY_API = "https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread"
|
||||
# Mastodon: https://host/@user/123456 (numeric ids; the public API may answer)
|
||||
MASTODON_STATUS_RE = re.compile(r"^/@[^/]+/(\d+)/?$")
|
||||
# Fediverse post paths: Mastodon, GoToSocial, ActivityPub ids, Misskey,
|
||||
# Pleroma/Akkoma
|
||||
FEDIVERSE_PATH_RE = re.compile(
|
||||
r"^/(?:@[^/]+/(?:statuses/)?[A-Za-z0-9]+|users/[^/]+/statuses/[A-Za-z0-9]+|"
|
||||
r"notes/[A-Za-z0-9]+|notice/[A-Za-z0-9]+|objects/[A-Za-z0-9-]+)/?$")
|
||||
# og:title of fediverse posts: "Display Name (@user@host)"
|
||||
FEDI_TITLE_RE = re.compile(r"^(.*?)\s*\((@[^)\s]+)\)\s*$")
|
||||
|
||||
FIELDS = ('title', 'content_snippet', 'published_at', 'author_name', 'author_handle',
|
||||
'author_url', 'author_photo', 'site_name', 'photo', 'photo_alt', 'video')
|
||||
|
||||
|
||||
def fetch_reply_context(
|
||||
db_path: Path,
|
||||
reply_url: str,
|
||||
site_url: str
|
||||
site_url: str,
|
||||
refresh: bool = False
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch and parse reply context from a URL.
|
||||
|
||||
Per database-schema.md:
|
||||
- Fetch URL content with SSRF protection
|
||||
- Parse h-entry for title, content, published date
|
||||
- Parse h-card for author information
|
||||
- Parse h-entry, a Mastodon status, a Bluesky post or OpenGraph tags
|
||||
- Store in reply_contexts table
|
||||
- Graceful degradation: errors don't prevent reply creation
|
||||
|
||||
@@ -49,208 +87,439 @@ def fetch_reply_context(
|
||||
db_path: Path to SQLite database
|
||||
reply_url: URL being replied to
|
||||
site_url: Site URL for User-Agent header
|
||||
refresh: Fetch again even if a context is stored
|
||||
|
||||
Returns:
|
||||
Dict with fetch status and context data
|
||||
"""
|
||||
# Check if context already exists
|
||||
existing = get_reply_context(db_path, reply_url)
|
||||
# Contexts stored before v2.19.0 (no source) are fetched again
|
||||
if existing and existing.get('source') and not refresh:
|
||||
logger.debug(f"Reply context already cached for {reply_url}")
|
||||
return existing
|
||||
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
found = _find_context(reply_url, site_url)
|
||||
except Exception as e:
|
||||
if isinstance(e, ValidationError):
|
||||
logger.warning(f"Failed to fetch reply context for {reply_url}: {e}")
|
||||
error = str(e)
|
||||
else:
|
||||
logger.error(f"Error fetching reply context for {reply_url}: {e}", exc_info=True)
|
||||
error = f"Unexpected error: {e}"
|
||||
if existing and existing.get('fetch_status') == 'fetched':
|
||||
# Keep what an earlier fetch found; don't try again
|
||||
_mark_source(db_path, reply_url, existing.get('source') or 'legacy')
|
||||
return existing
|
||||
return _store_minimal_context(db_path, reply_url, error)
|
||||
|
||||
cursor.execute("""
|
||||
SELECT * FROM reply_contexts WHERE url = ?
|
||||
""", (reply_url,))
|
||||
context = _store_context(db_path, reply_url, found)
|
||||
_save_images(context, site_url)
|
||||
logger.info(f"Fetched reply context for {reply_url} ({context['source']})")
|
||||
return context
|
||||
|
||||
existing = cursor.fetchone()
|
||||
|
||||
def refresh_stale(db_path: Path, site_url: str, limit: int = 5) -> int:
|
||||
"""
|
||||
Fetch again the contexts stored before v2.19.0 that notes still use.
|
||||
The webmention worker calls this; each row is fetched once, since the
|
||||
new fetch records its source. Returns how many were fetched.
|
||||
"""
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
urls = [row[0] for row in conn.execute(
|
||||
"""
|
||||
SELECT rc.url FROM reply_contexts rc
|
||||
WHERE rc.source IS NULL AND EXISTS (
|
||||
SELECT 1 FROM notes n WHERE n.deleted_at IS NULL AND rc.url IN
|
||||
(n.in_reply_to, n.like_of, n.repost_of, n.bookmark_of))
|
||||
ORDER BY rc.id DESC LIMIT ?
|
||||
""", (limit,))]
|
||||
finally:
|
||||
conn.close()
|
||||
for url in urls:
|
||||
fetch_reply_context(db_path, url, site_url, refresh=True)
|
||||
return len(urls)
|
||||
|
||||
if existing:
|
||||
logger.debug(f"Reply context already cached for {reply_url}")
|
||||
return dict(existing)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to check existing context: {e}")
|
||||
# --- Sources ----------------------------------------------------------------
|
||||
|
||||
# Fetch and parse context
|
||||
def _find_context(url: str, site_url: str) -> Dict[str, Any]:
|
||||
"""Context fields and 'source' for a URL; raises ValidationError if nothing could be fetched"""
|
||||
if BLUESKY_POST_RE.match(url):
|
||||
try:
|
||||
found = _from_bluesky(url, site_url)
|
||||
except ValidationError as e:
|
||||
logger.info(f"Bluesky API had no post for {url}: {e}")
|
||||
found = None
|
||||
if found:
|
||||
return found
|
||||
|
||||
page_error = None
|
||||
content = content_type = final_url = None
|
||||
try:
|
||||
# Fetch URL content
|
||||
# fetch_source also returns the URL after redirects (v2.7.0); relative
|
||||
# links in the page resolve against it (v2.10.2)
|
||||
content, content_type, final_url = fetch_source(reply_url, site_url)
|
||||
|
||||
# Parse microformats (only for HTML)
|
||||
if 'html' not in content_type.lower():
|
||||
# Not HTML - store minimal context
|
||||
return _store_minimal_context(db_path, reply_url, "Not HTML content")
|
||||
|
||||
parsed_mf2 = mf2py.parse(doc=content, url=final_url)
|
||||
|
||||
# Extract h-entry data
|
||||
entry_data = _extract_entry_data(parsed_mf2)
|
||||
|
||||
# Extract author data
|
||||
author_data = extract_author(parsed_mf2, reply_url)
|
||||
|
||||
# Store in database
|
||||
context = _store_context(
|
||||
db_path,
|
||||
reply_url,
|
||||
entry_data.get('title'),
|
||||
entry_data.get('content_snippet'),
|
||||
entry_data.get('published_at'),
|
||||
author_data.get('author_name'),
|
||||
author_data.get('author_url'),
|
||||
author_data.get('author_photo'),
|
||||
entry_data.get('site_name')
|
||||
)
|
||||
|
||||
logger.info(f"Fetched reply context for {reply_url}")
|
||||
return context
|
||||
|
||||
content, content_type, final_url = fetch_source(url, site_url)
|
||||
except ValidationError as e:
|
||||
# Fetch/validation error - store error status
|
||||
logger.warning(f"Failed to fetch reply context for {reply_url}: {e}")
|
||||
return _store_minimal_context(db_path, reply_url, str(e))
|
||||
page_error = e
|
||||
|
||||
except Exception as e:
|
||||
# Unexpected error - store error status
|
||||
logger.error(f"Error fetching reply context for {reply_url}: {e}", exc_info=True)
|
||||
return _store_minimal_context(db_path, reply_url, f"Unexpected error: {e}")
|
||||
og: Dict[str, Optional[str]] = {}
|
||||
card: Dict[str, Optional[str]] = {}
|
||||
is_html = content is not None and 'html' in (content_type or '').lower()
|
||||
if is_html:
|
||||
parsed = mf2py.parse(doc=content, url=final_url)
|
||||
entry = mf2_utils.responding_entry(parsed, '')
|
||||
og = _opengraph(content, final_url)
|
||||
if entry:
|
||||
return _from_mf2(parsed, entry, final_url, og)
|
||||
card = _page_card(parsed, final_url)
|
||||
|
||||
if _looks_fediverse(url, og):
|
||||
status = _mastodon_api(url, site_url) or _resolve_with_instance(url)
|
||||
if status:
|
||||
return _from_status(status, url)
|
||||
|
||||
if page_error:
|
||||
raise page_error
|
||||
if not is_html:
|
||||
raise ValidationError("Not HTML content")
|
||||
return _from_opengraph(og, card)
|
||||
|
||||
|
||||
def _extract_entry_data(parsed_mf2: Dict) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract entry data from parsed microformats.
|
||||
|
||||
Args:
|
||||
parsed_mf2: Parsed microformats data from mf2py
|
||||
|
||||
Returns:
|
||||
Dict with title, content_snippet, published_at, site_name
|
||||
"""
|
||||
title = None
|
||||
content_snippet = None
|
||||
published_at = None
|
||||
site_name = None
|
||||
|
||||
# Find h-entry
|
||||
for item in parsed_mf2.get('items', []):
|
||||
if 'h-entry' not in item.get('type', []):
|
||||
continue
|
||||
|
||||
props = item.get('properties', {})
|
||||
|
||||
# Extract title (p-name)
|
||||
name = props.get('name', [])
|
||||
if name:
|
||||
title = name[0]
|
||||
if isinstance(title, dict):
|
||||
title = title.get('value', '')
|
||||
# Truncate if too long
|
||||
if title and len(title) > 200:
|
||||
title = title[:197] + '...'
|
||||
|
||||
# Extract content (reuse processor function)
|
||||
content_snippet = extract_content(parsed_mf2)
|
||||
|
||||
# Extract published date
|
||||
published = props.get('published', [])
|
||||
if published:
|
||||
pub_str = published[0]
|
||||
if isinstance(pub_str, dict):
|
||||
pub_str = pub_str.get('value', '')
|
||||
# Try to parse ISO 8601 datetime
|
||||
try:
|
||||
from dateutil import parser as dateutil_parser
|
||||
published_at = dateutil_parser.parse(pub_str)
|
||||
except Exception:
|
||||
# Parsing failed, leave as None
|
||||
pass
|
||||
|
||||
break # Use first h-entry
|
||||
|
||||
# Try to extract site name from h-card
|
||||
for item in parsed_mf2.get('items', []):
|
||||
if 'h-card' in item.get('type', []):
|
||||
props = item.get('properties', {})
|
||||
name = props.get('name', [])
|
||||
if name:
|
||||
site_name = name[0]
|
||||
if isinstance(site_name, dict):
|
||||
site_name = site_name.get('value', '')
|
||||
break
|
||||
|
||||
def _from_mf2(parsed: dict, entry: dict, page_url: str, og: dict) -> Dict[str, Any]:
|
||||
"""An IndieWeb post's h-entry"""
|
||||
author = mf2_utils.entry_author(parsed, entry, page_url)
|
||||
snippet = extract_content(parsed, entry)
|
||||
title = mf2_utils.first(entry, 'name')
|
||||
if title and _repeats(title, snippet):
|
||||
title = None # implied name: the content again
|
||||
photo, photo_alt = mf2_utils.first(entry, 'photo'), None
|
||||
for value in entry.get('properties', {}).get('photo', [])[:1]:
|
||||
if isinstance(value, dict):
|
||||
photo_alt = value.get('alt') or None
|
||||
if not _is_http_url(photo):
|
||||
photo, photo_alt = og.get('image'), og.get('image_alt')
|
||||
return {
|
||||
'title': title,
|
||||
'content_snippet': content_snippet,
|
||||
'published_at': published_at,
|
||||
'site_name': site_name
|
||||
'source': 'mf2',
|
||||
'title': _clip(title, 200),
|
||||
'content_snippet': snippet,
|
||||
'published_at': _iso(mf2_utils.first(entry, 'published')),
|
||||
'author_name': author['name'],
|
||||
'author_url': author['url'] if _is_http_url(author['url']) else None,
|
||||
'author_photo': author['photo'] if _is_http_url(author['photo']) else None,
|
||||
'site_name': og.get('site_name'),
|
||||
'photo': photo if _is_http_url(photo) else None,
|
||||
'photo_alt': photo_alt,
|
||||
}
|
||||
|
||||
|
||||
def _store_context(
|
||||
db_path: Path,
|
||||
url: str,
|
||||
title: Optional[str],
|
||||
content_snippet: Optional[str],
|
||||
published_at: Optional[datetime],
|
||||
author_name: Optional[str],
|
||||
author_url: Optional[str],
|
||||
author_photo: Optional[str],
|
||||
site_name: Optional[str]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Store reply context in database.
|
||||
def _from_bluesky(url: str, site_url: str) -> Optional[Dict[str, Any]]:
|
||||
"""A Bluesky post from the public AppView API (no login needed)"""
|
||||
actor, rkey = BLUESKY_POST_RE.match(url).groups()
|
||||
uri = quote(f"at://{actor}/app.bsky.feed.post/{rkey}", safe='')
|
||||
data = _get_json(f"{BLUESKY_API}?uri={uri}&depth=0&parentHeight=0", site_url)
|
||||
post = ((data or {}).get('thread') or {}).get('post') if isinstance(data, dict) else None
|
||||
if not post:
|
||||
return None
|
||||
author = post.get('author') or {}
|
||||
record = post.get('record') or {}
|
||||
handle = author.get('handle')
|
||||
photo, photo_alt = _bluesky_image(post.get('embed') or {})
|
||||
return {
|
||||
'source': 'bluesky',
|
||||
'title': None,
|
||||
'content_snippet': _text_snippet(record.get('text')),
|
||||
'published_at': _iso(record.get('createdAt')),
|
||||
'author_name': author.get('displayName') or handle,
|
||||
'author_handle': f"@{handle}" if handle else None,
|
||||
'author_url': f"https://bsky.app/profile/{handle}" if handle else None,
|
||||
'author_photo': author.get('avatar') if _is_http_url(author.get('avatar')) else None,
|
||||
'site_name': 'Bluesky',
|
||||
'photo': photo,
|
||||
'photo_alt': photo_alt,
|
||||
}
|
||||
|
||||
Args:
|
||||
db_path: Database path
|
||||
url: Reply-to URL
|
||||
title: Entry title
|
||||
content_snippet: Content snippet
|
||||
published_at: Published datetime
|
||||
author_name: Author name
|
||||
author_url: Author URL
|
||||
author_photo: Author photo URL
|
||||
site_name: Site name
|
||||
|
||||
Returns:
|
||||
Stored context dict
|
||||
"""
|
||||
def _bluesky_image(embed: dict) -> tuple:
|
||||
"""First image of a post's embed view: (url, alt)"""
|
||||
kind = embed.get('$type', '')
|
||||
if kind.startswith('app.bsky.embed.recordWithMedia'):
|
||||
return _bluesky_image(embed.get('media') or {})
|
||||
if kind.startswith('app.bsky.embed.images'):
|
||||
for image in embed.get('images') or []:
|
||||
if _is_http_url(image.get('thumb')):
|
||||
return image['thumb'], image.get('alt') or None
|
||||
if kind.startswith('app.bsky.embed.video') and _is_http_url(embed.get('thumbnail')):
|
||||
return embed['thumbnail'], embed.get('alt') or None
|
||||
if kind.startswith('app.bsky.embed.external'):
|
||||
external = embed.get('external') or {}
|
||||
if _is_http_url(external.get('thumb')):
|
||||
return external['thumb'], None
|
||||
return None, None
|
||||
|
||||
|
||||
def _looks_fediverse(url: str, og: dict) -> bool:
|
||||
return bool(FEDIVERSE_PATH_RE.match(urlparse(url).path) or og.get('username'))
|
||||
|
||||
|
||||
def _mastodon_api(url: str, site_url: str) -> Optional[dict]:
|
||||
"""The status from its own server's public API (Mastodon post URLs only)"""
|
||||
parts = urlparse(url)
|
||||
m = MASTODON_STATUS_RE.match(parts.path)
|
||||
if not m:
|
||||
return None
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
status = _get_json(f"{parts.scheme}://{parts.netloc}/api/v1/statuses/{m.group(1)}", site_url)
|
||||
except ValidationError:
|
||||
return None
|
||||
return status if isinstance(status, dict) and status.get('account') else None
|
||||
|
||||
cursor.execute("""
|
||||
|
||||
def _resolve_with_instance(url: str) -> Optional[dict]:
|
||||
"""The status as this site's GoToSocial instance sees it, if one is configured"""
|
||||
try:
|
||||
from starpunk.syndication import get_targets
|
||||
target = get_targets().get('gotosocial')
|
||||
if target is None:
|
||||
return None
|
||||
return target.client.resolve_status(url)
|
||||
except Exception as e:
|
||||
logger.info(f"Instance could not resolve {url}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def _from_status(status: dict, url: str) -> Dict[str, Any]:
|
||||
"""A Mastodon API status entity"""
|
||||
account = status.get('account') or {}
|
||||
acct = account.get('acct') or account.get('username')
|
||||
host = urlparse(account.get('url') or url).hostname
|
||||
if acct and '@' not in acct and host:
|
||||
acct = f"{acct}@{host}"
|
||||
spoiler = (status.get('spoiler_text') or '').strip()
|
||||
if spoiler:
|
||||
# Keep the content warning: show it, not what it covers
|
||||
snippet = _text_snippet(f"CW: {spoiler}")
|
||||
else:
|
||||
snippet = _snippet(status.get('content') or '') or None
|
||||
photo = photo_alt = video = None
|
||||
if not status.get('sensitive'):
|
||||
for media in status.get('media_attachments') or []:
|
||||
if media.get('type') in ('image', 'gifv', 'video') and _is_http_url(media.get('preview_url')):
|
||||
photo, photo_alt = media['preview_url'], media.get('description') or None
|
||||
# A "GIF" on Mastodon is a silent looping MP4; the preview is its first frame
|
||||
if media['type'] == 'gifv' and _is_http_url(media.get('url')):
|
||||
video = media['url']
|
||||
break
|
||||
avatar = account.get('avatar_static') or account.get('avatar')
|
||||
return {
|
||||
'source': 'fediverse',
|
||||
'title': None,
|
||||
'content_snippet': snippet,
|
||||
'published_at': _iso(status.get('created_at')),
|
||||
'author_name': account.get('display_name') or account.get('username'),
|
||||
'author_handle': f"@{acct}" if acct else None,
|
||||
'author_url': account.get('url') if _is_http_url(account.get('url')) else None,
|
||||
'author_photo': avatar if _is_http_url(avatar) else None,
|
||||
'site_name': urlparse(url).hostname,
|
||||
'photo': photo,
|
||||
'photo_alt': photo_alt,
|
||||
'video': video,
|
||||
}
|
||||
|
||||
|
||||
def _from_opengraph(og: dict, card: dict) -> Dict[str, Any]:
|
||||
"""Any other page: OpenGraph tags, the page's h-card as author"""
|
||||
title = og.get('title')
|
||||
author_name, handle = card.get('name') or og.get('author'), None
|
||||
m = FEDI_TITLE_RE.match(title or '')
|
||||
if m and (og.get('username') or not author_name):
|
||||
# A fediverse post's og:title is its author, not a title
|
||||
author_name, handle, title = m.group(1) or author_name, m.group(2), None
|
||||
elif og.get('username') and title:
|
||||
author_name, handle, title = title, f"@{og['username'].lstrip('@')}", None
|
||||
snippet = _text_snippet(og.get('description'))
|
||||
if title and snippet and _repeats(title, snippet):
|
||||
snippet = None
|
||||
return {
|
||||
'source': 'opengraph',
|
||||
'title': _clip(title, 200),
|
||||
'content_snippet': snippet,
|
||||
'published_at': _iso(og.get('published')),
|
||||
'author_name': author_name,
|
||||
'author_handle': handle,
|
||||
'author_url': card.get('url') if _is_http_url(card.get('url')) else None,
|
||||
'author_photo': card.get('photo') if _is_http_url(card.get('photo')) else None,
|
||||
'site_name': og.get('site_name'),
|
||||
'photo': og.get('image') if _is_http_url(og.get('image')) else None,
|
||||
'photo_alt': og.get('image_alt'),
|
||||
}
|
||||
|
||||
|
||||
OG_FIELDS = {
|
||||
'og:title': 'title', 'twitter:title': 'title',
|
||||
'og:description': 'description', 'twitter:description': 'description',
|
||||
'description': 'description',
|
||||
'og:image': 'image', 'og:image:url': 'image', 'twitter:image': 'image',
|
||||
'og:image:alt': 'image_alt', 'twitter:image:alt': 'image_alt',
|
||||
'og:site_name': 'site_name',
|
||||
'article:published_time': 'published', 'og:article:published_time': 'published',
|
||||
'og:published_time': 'published',
|
||||
'profile:username': 'username', 'og:profile:username': 'username',
|
||||
'author': 'author', 'article:author': 'author',
|
||||
}
|
||||
|
||||
|
||||
def _opengraph(content: str, page_url: str) -> Dict[str, Optional[str]]:
|
||||
"""OpenGraph (and plain) meta tags of a page; the first of each wins"""
|
||||
from lxml import html as lxml_html
|
||||
try:
|
||||
doc = lxml_html.document_fromstring(content)
|
||||
except Exception:
|
||||
return {}
|
||||
found: Dict[str, Optional[str]] = {}
|
||||
for meta in doc.iter('meta'):
|
||||
key = (meta.get('property') or meta.get('name') or '').strip().lower()
|
||||
value = (meta.get('content') or '').strip()
|
||||
field = OG_FIELDS.get(key)
|
||||
if not field or not value:
|
||||
continue
|
||||
if field == 'image_alt':
|
||||
# Only the alt text of the image we keep: the one right after it
|
||||
if 'image' in found and 'image_alt' not in found and not found.get('_alt_done'):
|
||||
found['image_alt'] = value
|
||||
continue
|
||||
if field == 'image':
|
||||
if 'image' in found:
|
||||
found['_alt_done'] = '1'
|
||||
continue
|
||||
value = urljoin(page_url, value)
|
||||
if field == 'author' and _is_http_url(value):
|
||||
continue # article:author is often a profile URL
|
||||
found.setdefault(field, value)
|
||||
found.pop('_alt_done', None)
|
||||
if 'title' not in found:
|
||||
titles = doc.findall('.//title')
|
||||
if titles and (titles[0].text_content() or '').strip():
|
||||
found['title'] = titles[0].text_content().strip()
|
||||
return found
|
||||
|
||||
|
||||
def _page_card(parsed: dict, page_url: str) -> Dict[str, Optional[str]]:
|
||||
"""The page's representative h-card, else its first h-card (GoToSocial pages)"""
|
||||
card = mf2_utils.representative_hcard(parsed, page_url)
|
||||
if card is None:
|
||||
card = next((i for i in mf2_utils.walk(parsed.get('items', []))
|
||||
if mf2_utils.is_type(i, 'h-card')), None)
|
||||
if not card:
|
||||
return {}
|
||||
return {'name': mf2_utils.first(card, 'name'), 'url': mf2_utils.first(card, 'url'),
|
||||
'photo': mf2_utils.first(card, 'photo')}
|
||||
|
||||
|
||||
# --- Helpers ----------------------------------------------------------------
|
||||
|
||||
def _get_json(url: str, site_url: str):
|
||||
"""GET JSON through the SSRF-safe fetcher; None if it isn't JSON"""
|
||||
content, _, _ = fetch_source(url, site_url)
|
||||
try:
|
||||
return json.loads(content)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _snippet(markup: str) -> str:
|
||||
"""Sanitized HTML of at most ~SNIPPET_LIMIT characters, with tags closed"""
|
||||
sanitized = sanitize_content(markup).strip()
|
||||
if len(sanitized) > SNIPPET_LIMIT:
|
||||
# Cutting can leave a tag or attribute open; sanitizing again closes it
|
||||
sanitized = sanitize_content(sanitized[:SNIPPET_LIMIT - 3] + '...')
|
||||
return sanitized
|
||||
|
||||
|
||||
def _text_snippet(text: Optional[str]) -> Optional[str]:
|
||||
"""Plain text as a sanitized HTML snippet"""
|
||||
text = (text or '').strip()
|
||||
if not text:
|
||||
return None
|
||||
if len(text) > SNIPPET_LIMIT:
|
||||
text = text[:SNIPPET_LIMIT - 1].rstrip() + '…'
|
||||
paragraphs = [p.strip() for p in re.split(r'\n\s*\n', text) if p.strip()]
|
||||
return ''.join(f"<p>{html.escape(p).replace(chr(10), '<br>')}</p>" for p in paragraphs)
|
||||
|
||||
|
||||
def _plain(markup: Optional[str]) -> str:
|
||||
return ' '.join(html.unescape(re.sub(r'<[^>]+>', ' ', markup or '')).split())
|
||||
|
||||
|
||||
def _repeats(title: str, snippet: Optional[str]) -> bool:
|
||||
"""True if a title is just the start of the content (mf2 implied names, og:description)"""
|
||||
a, b = ' '.join(title.split()).rstrip('.…'), _plain(snippet).rstrip('.…')
|
||||
return bool(a and b) and (b.startswith(a) or a.startswith(b))
|
||||
|
||||
|
||||
def _clip(text: Optional[str], limit: int) -> Optional[str]:
|
||||
if not text:
|
||||
return None
|
||||
text = ' '.join(text.split())
|
||||
return text if len(text) <= limit else text[:limit - 3] + '...'
|
||||
|
||||
|
||||
def _iso(value: Optional[str]) -> Optional[str]:
|
||||
"""An ISO 8601 date string, normalized; None if it doesn't parse"""
|
||||
if not value:
|
||||
return None
|
||||
from dateutil import parser as dateutil_parser
|
||||
try:
|
||||
return dateutil_parser.isoparse(value).isoformat()
|
||||
except (ValueError, OverflowError):
|
||||
try:
|
||||
return dateutil_parser.parse(value).isoformat()
|
||||
except (ValueError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _save_images(context: Dict[str, Any], site_url: str) -> None:
|
||||
"""Local copies of the author's avatar and the post's image (best effort;
|
||||
the webmention worker retries what's missing)"""
|
||||
try:
|
||||
from flask import current_app
|
||||
from starpunk import avatars
|
||||
data_path = current_app.config['DATA_PATH']
|
||||
avatars.fetch(data_path, context.get('author_photo'), site_url)
|
||||
avatars.fetch_preview(data_path, context.get('photo'), site_url)
|
||||
avatars.fetch_video(data_path, context.get('video'), site_url)
|
||||
except Exception as e:
|
||||
logger.info(f"Could not save reply context images: {e}")
|
||||
|
||||
|
||||
# --- Storage ----------------------------------------------------------------
|
||||
|
||||
def _store_context(db_path: Path, url: str, found: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Store a fetched reply context; returns it as stored"""
|
||||
values = [found.get(field) for field in FIELDS]
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
conn.execute(f"""
|
||||
INSERT OR REPLACE INTO reply_contexts
|
||||
(url, title, content_snippet, published_at,
|
||||
author_name, author_url, author_photo, site_name,
|
||||
fetch_status, fetched_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'fetched', CURRENT_TIMESTAMP)
|
||||
""", (
|
||||
url, title, content_snippet, published_at,
|
||||
author_name, author_url, author_photo, site_name
|
||||
))
|
||||
|
||||
(url, {', '.join(FIELDS)}, source, fetch_status, fetch_error, fetched_at)
|
||||
VALUES (?, {', '.join('?' for _ in FIELDS)}, ?, 'fetched', NULL, CURRENT_TIMESTAMP)
|
||||
""", (url, *values, found['source']))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
return {
|
||||
'url': url,
|
||||
'title': title,
|
||||
'content_snippet': content_snippet,
|
||||
'published_at': published_at,
|
||||
'author_name': author_name,
|
||||
'author_url': author_url,
|
||||
'author_photo': author_photo,
|
||||
'site_name': site_name,
|
||||
'fetch_status': 'fetched'
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to store reply context: {e}")
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
return {'url': url, **dict(zip(FIELDS, values)), 'source': found['source'],
|
||||
'fetch_status': 'fetched'}
|
||||
|
||||
|
||||
def _mark_source(db_path: Path, url: str, source: str) -> None:
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
conn.execute("UPDATE reply_contexts SET source = ? WHERE url = ?", (source, url))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _store_minimal_context(
|
||||
@@ -269,33 +538,20 @@ def _store_minimal_context(
|
||||
Returns:
|
||||
Minimal context dict with error
|
||||
"""
|
||||
context = {'url': url, 'fetch_status': 'error', 'fetch_error': error, 'source': 'error'}
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("""
|
||||
conn.execute("""
|
||||
INSERT OR REPLACE INTO reply_contexts
|
||||
(url, fetch_status, fetch_error, created_at)
|
||||
VALUES (?, 'error', ?, CURRENT_TIMESTAMP)
|
||||
(url, fetch_status, fetch_error, source, created_at)
|
||||
VALUES (?, 'error', ?, 'error', CURRENT_TIMESTAMP)
|
||||
""", (url, error))
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
return {
|
||||
'url': url,
|
||||
'fetch_status': 'error',
|
||||
'fetch_error': error
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
# Return the in-memory dict even if storage fails
|
||||
logger.error(f"Failed to store minimal context: {e}")
|
||||
# Return in-memory dict even if storage fails
|
||||
return {
|
||||
'url': url,
|
||||
'fetch_status': 'error',
|
||||
'fetch_error': error
|
||||
}
|
||||
return context
|
||||
|
||||
|
||||
def get_reply_context(db_path: Path, reply_url: str) -> Optional[Dict[str, Any]]:
|
||||
@@ -312,20 +568,9 @@ def get_reply_context(db_path: Path, reply_url: str) -> Optional[Dict[str, Any]]
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("""
|
||||
SELECT * FROM reply_contexts WHERE url = ?
|
||||
""", (reply_url,))
|
||||
|
||||
row = cursor.fetchone()
|
||||
row = conn.execute("SELECT * FROM reply_contexts WHERE url = ?", (reply_url,)).fetchone()
|
||||
conn.close()
|
||||
|
||||
if row:
|
||||
return dict(row)
|
||||
|
||||
return None
|
||||
|
||||
return dict(row) if row else None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get reply context: {e}")
|
||||
return None
|
||||
@@ -25,6 +25,20 @@ from starpunk.notes import (
|
||||
bp = Blueprint("admin", __name__, url_prefix="/admin")
|
||||
|
||||
|
||||
@bp.context_processor
|
||||
def _admin_nav():
|
||||
"""Error count for the Log link in the admin nav (v2.18.0)"""
|
||||
if not getattr(g, "me", None):
|
||||
return {}
|
||||
from starpunk import error_log
|
||||
|
||||
try:
|
||||
return {"error_count": error_log.count()}
|
||||
except Exception as e: # never break an admin page over the badge
|
||||
logger.warning(f"Could not count errors for the admin nav: {e}")
|
||||
return {"error_count": 0}
|
||||
|
||||
|
||||
@bp.route("/")
|
||||
@require_auth
|
||||
def dashboard():
|
||||
@@ -530,7 +544,6 @@ def webmentions():
|
||||
status_filter=status_filter,
|
||||
origin_filter=origin_filter,
|
||||
backfeed=backfeed.status(),
|
||||
notify=_notify_status(),
|
||||
page=page,
|
||||
total_pages=total_pages,
|
||||
user_me=g.me
|
||||
@@ -698,7 +711,7 @@ def revoke_token(token_id):
|
||||
|
||||
|
||||
def _notify_status() -> dict:
|
||||
"""Notification settings and last error for the Webmentions page (v2.14.0)"""
|
||||
"""Notification settings and last error for the Settings page (v2.14.0)"""
|
||||
from starpunk import notifications
|
||||
from starpunk.database import get_db
|
||||
|
||||
@@ -711,6 +724,25 @@ def _notify_status() -> dict:
|
||||
}
|
||||
|
||||
|
||||
@bp.route("/settings")
|
||||
@require_auth
|
||||
def settings():
|
||||
"""Site settings: notifications (v2.18.0). Template: templates/admin/settings.html"""
|
||||
from starpunk.webmention_sender import skipped_hosts
|
||||
|
||||
return render_template("admin/settings.html", notify=_notify_status(), user_me=g.me,
|
||||
skip_hosts=sorted(skipped_hosts(current_app.config)))
|
||||
|
||||
|
||||
@bp.route("/log")
|
||||
@require_auth
|
||||
def error_log():
|
||||
"""Current errors with the affected note (v2.18.0). Template: templates/admin/log.html"""
|
||||
from starpunk import error_log as log
|
||||
|
||||
return render_template("admin/log.html", entries=log.entries(), user_me=g.me)
|
||||
|
||||
|
||||
@bp.route("/notifications/test", methods=["POST"])
|
||||
@require_auth
|
||||
def notifications_test():
|
||||
@@ -722,7 +754,7 @@ def notifications_test():
|
||||
notifications._record(current_app.config["DATABASE_PATH"], ok, error)
|
||||
flash("Test notification sent" if ok else f"Test notification failed: {error or 'unknown error'}",
|
||||
"success" if ok else "error")
|
||||
return redirect(url_for("admin.webmentions"))
|
||||
return redirect(url_for("admin.settings"))
|
||||
|
||||
|
||||
@bp.route("/backfeed/backfill", methods=["POST"])
|
||||
|
||||
@@ -264,6 +264,21 @@ def avatar(name: str):
|
||||
return response
|
||||
|
||||
|
||||
@bp.route("/previews/<name>")
|
||||
def preview(name: str):
|
||||
"""A saved copy of a replied-to post's image (v2.19.0)"""
|
||||
from pathlib import Path
|
||||
from starpunk.avatars import NAME_RE, VIDEO_NAME_RE, preview_folder
|
||||
|
||||
if not (NAME_RE.match(name) or VIDEO_NAME_RE.match(name)):
|
||||
abort(404)
|
||||
response = send_from_directory(
|
||||
preview_folder(Path(current_app.config.get('DATA_PATH', 'data'))), name,
|
||||
mimetype='video/mp4' if name.endswith('.mp4') else None)
|
||||
response.headers['Cache-Control'] = 'public, max-age=604800'
|
||||
return response
|
||||
|
||||
|
||||
@bp.route("/")
|
||||
def index():
|
||||
"""
|
||||
@@ -310,14 +325,15 @@ def attach_response_counts(notes) -> None:
|
||||
|
||||
|
||||
def attach_contexts(notes) -> None:
|
||||
"""Give likes and reposts the fetched context of the post they point at"""
|
||||
"""Give replies, likes and reposts the fetched context of the post they point at"""
|
||||
from starpunk.webmention_queries import get_reply_context
|
||||
|
||||
for note in notes:
|
||||
context = None
|
||||
if note.reacts_to:
|
||||
url = note.in_reply_to or note.reacts_to
|
||||
if url:
|
||||
try:
|
||||
context = get_reply_context(note.reacts_to)
|
||||
context = get_reply_context(url)
|
||||
except Exception:
|
||||
context = None
|
||||
object.__setattr__(note, 'context', context)
|
||||
|
||||
+222
-45
@@ -25,6 +25,7 @@ import logging
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from urllib.parse import urlparse
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
@@ -102,10 +103,14 @@ class MastodonClient:
|
||||
"content_type": "text/markdown",
|
||||
}
|
||||
|
||||
def create_status(self, text: str, media_ids: List[str], idempotency_key: str) -> dict:
|
||||
def create_status(self, text: str, media_ids: List[str], idempotency_key: str,
|
||||
in_reply_to_id: Optional[str] = None) -> dict:
|
||||
body = self._status_body(text, media_ids)
|
||||
if in_reply_to_id:
|
||||
body["in_reply_to_id"] = in_reply_to_id
|
||||
response = self._call(
|
||||
"POST", "/api/v1/statuses",
|
||||
json=self._status_body(text, media_ids),
|
||||
json=body,
|
||||
headers={"Idempotency-Key": idempotency_key},
|
||||
)
|
||||
data = response.json()
|
||||
@@ -127,8 +132,8 @@ class MastodonClient:
|
||||
can_edit = True
|
||||
NATIVE = ("favourite", "reblog") # remote_id prefixes for likes/boosts (v2.5.0)
|
||||
|
||||
def find_status(self, url: str) -> Optional[str]:
|
||||
"""Local id of a fediverse post, fetching it if needed; None if the URL isn't one"""
|
||||
def resolve_status(self, url: str) -> Optional[dict]:
|
||||
"""A fediverse post as a Mastodon status, fetching it if needed; None if the URL isn't one"""
|
||||
try:
|
||||
response = self._call("GET", "/api/v2/search", params={
|
||||
"q": url, "resolve": "true", "type": "statuses", "limit": 1,
|
||||
@@ -136,16 +141,52 @@ class MastodonClient:
|
||||
except RemoteGone:
|
||||
return None
|
||||
statuses = response.json().get("statuses") or []
|
||||
return statuses[0]["id"] if statuses else None
|
||||
return statuses[0] if statuses else None
|
||||
|
||||
def find_status(self, url: str) -> Optional[str]:
|
||||
"""Local id of a fediverse post, fetching it if needed; None if the URL isn't one"""
|
||||
status = self.resolve_status(url)
|
||||
return status["id"] if status else None
|
||||
|
||||
def _reply_parent(self, post: "Post") -> tuple:
|
||||
"""(status id, '@acct ' mention) a reply threads under; (None, '') if it can't (v2.20.0)
|
||||
|
||||
Our own crosspost of the replied-to note, else the replied-to post as
|
||||
this instance resolves it (any fediverse post). Replies to others
|
||||
start with their @mention, as Mastodon and GoToSocial clients do, so
|
||||
they are notified and their server gets the reply."""
|
||||
if not post.reply_to:
|
||||
return None, ""
|
||||
if post.reply_remote_id and post.reply_remote_id.split(":", 1)[0] not in self.NATIVE:
|
||||
return post.reply_remote_id, ""
|
||||
try:
|
||||
status = self.resolve_status(post.reply_to)
|
||||
except SyndicationError as e:
|
||||
current_app.logger.info(f"Could not resolve {post.reply_to}: {e}")
|
||||
return None, ""
|
||||
if not status:
|
||||
return None, ""
|
||||
account = status.get("account") or {}
|
||||
try:
|
||||
mine = account.get("id") == self._account_id()
|
||||
except SyndicationError:
|
||||
mine = False
|
||||
mention = "" if mine or not account.get("acct") else f"@{account['acct']} "
|
||||
return status["id"], mention
|
||||
|
||||
def _prepare(self, post: "Post") -> tuple:
|
||||
"""(text, media ids, id of the status it replies to)"""
|
||||
if post.kind == "repost":
|
||||
return compose_repost_text(post, post.note.content, post.max_chars), []
|
||||
return compose_repost_text(post, post.note.content, post.max_chars), [], None
|
||||
parent, mention = self._reply_parent(post)
|
||||
# Images in the text are sent as attachments instead (v2.6.6)
|
||||
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
|
||||
body=strip_images(post.note.content))
|
||||
body = strip_images(post.note.content)
|
||||
if mention.strip() and mention.strip() not in body:
|
||||
body = mention + body.lstrip()
|
||||
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body,
|
||||
lead="" if parent else reply_line(post))
|
||||
media_ids = [self.upload_media(f["path"], f["mime_type"], f["description"]) for f in post.media]
|
||||
return text, media_ids
|
||||
return text, media_ids, parent
|
||||
|
||||
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
|
||||
if post.kind in ("like", "repost") and post.target_url:
|
||||
@@ -160,13 +201,13 @@ class MastodonClient:
|
||||
return {"id": f"{action}:{status_id}", "url": None}
|
||||
if post.kind == "like":
|
||||
return {"id": None, "url": None} # nothing to like here
|
||||
text, media_ids = self._prepare(post)
|
||||
return self.create_status(text, media_ids, idempotency_key=key)
|
||||
text, media_ids, parent = self._prepare(post)
|
||||
return self.create_status(text, media_ids, idempotency_key=key, in_reply_to_id=parent)
|
||||
|
||||
def update(self, remote_id: str, post: "Post") -> dict:
|
||||
if remote_id.split(":", 1)[0] in self.NATIVE:
|
||||
return {"id": remote_id, "url": None} # a favourite/boost has nothing to edit
|
||||
text, media_ids = self._prepare(post)
|
||||
text, media_ids, _ = self._prepare(post) # edits can't move a status in a thread
|
||||
return self.update_status(remote_id, text, media_ids)
|
||||
|
||||
# --- Backfeed (v2.13.0, ADR-073) ---
|
||||
@@ -199,8 +240,9 @@ class MastodonClient:
|
||||
}
|
||||
|
||||
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
|
||||
"""Likes, boosts and public direct replies of one of our statuses"""
|
||||
from starpunk.backfeed import Harvest, Response
|
||||
"""Likes, boosts and public replies of one of our statuses, with the
|
||||
replies to those replies (v2.20.2)"""
|
||||
from starpunk.backfeed import Harvest, Response, thread_members
|
||||
|
||||
me = self._account_id()
|
||||
found, complete = [], True
|
||||
@@ -214,14 +256,18 @@ class MastodonClient:
|
||||
found.append(Response(kind=kind, source_url=f"{remote_url}#{fragment}-{account['id']}",
|
||||
remote_id=account["id"], **self._author(account)))
|
||||
context = self._call("GET", f"/api/v1/statuses/{remote_id}/context").json() or {}
|
||||
for status in context.get("descendants") or []:
|
||||
account = status.get("account") or {}
|
||||
if (status.get("in_reply_to_id") != remote_id or status.get("visibility") != "public"
|
||||
or account.get("id") == me):
|
||||
public = [st for st in context.get("descendants") or [] if st.get("visibility") == "public"]
|
||||
kept = thread_members([{"id": st["id"], "parent": st.get("in_reply_to_id"),
|
||||
"own": (st.get("account") or {}).get("id") == me} for st in public],
|
||||
remote_id)
|
||||
for status in public:
|
||||
if status["id"] not in kept:
|
||||
continue
|
||||
account = status.get("account") or {}
|
||||
found.append(Response(kind="reply", source_url=status.get("url") or status["uri"],
|
||||
remote_id=status["id"], content_html=status.get("content") or "",
|
||||
published=status.get("created_at"), **self._author(account)))
|
||||
published=status.get("created_at"), parent_id=kept[status["id"]],
|
||||
own=account.get("id") == me, **self._author(account)))
|
||||
return Harvest(found, complete)
|
||||
|
||||
def delete(self, remote_id: str) -> None:
|
||||
@@ -236,6 +282,7 @@ class MastodonClient:
|
||||
|
||||
|
||||
BLUESKY_BLOB_LIMIT = 1_000_000 # app.bsky.embed.images maxSize
|
||||
BLUESKY_THREAD_DEPTH = 10 # how deep backfeed follows replies to replies
|
||||
_TID_CHARS = "234567abcdefghijklmnopqrstuvwxyz"
|
||||
|
||||
|
||||
@@ -354,14 +401,44 @@ class BlueskyClient:
|
||||
current_app.logger.info(f"No link card image for {url}: {e}")
|
||||
return {"$type": "app.bsky.embed.external", "external": external}
|
||||
|
||||
def _reply_ref(self, post: "Post") -> Optional[dict]:
|
||||
"""{'root', 'parent'} for a reply to a Bluesky post (ours or anyone's); None otherwise (v2.20.0)"""
|
||||
if not post.reply_to:
|
||||
return None
|
||||
uri = post.reply_remote_id if (post.reply_remote_id or "").startswith("at://") and \
|
||||
f"/{self.POST}/" in post.reply_remote_id else None
|
||||
try:
|
||||
if uri is None:
|
||||
found = self.resolve_post(post.reply_to)
|
||||
if not found:
|
||||
return None
|
||||
uri = found["uri"]
|
||||
views = self._call("GET", "app.bsky.feed.getPosts", params={"uris": uri}).json().get("posts") or []
|
||||
except SyndicationError as e:
|
||||
current_app.logger.info(f"Could not resolve {post.reply_to} on Bluesky: {e}")
|
||||
return None
|
||||
if not views:
|
||||
return None
|
||||
parent = {"uri": views[0]["uri"], "cid": views[0]["cid"]}
|
||||
root = ((views[0].get("record") or {}).get("reply") or {}).get("root") or parent
|
||||
return {"root": {"uri": root["uri"], "cid": root["cid"]}, "parent": parent}
|
||||
|
||||
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None,
|
||||
anchors: List[tuple] = ()) -> dict:
|
||||
anchors: List[tuple] = (), reply: Optional[dict] = None) -> dict:
|
||||
if text is None:
|
||||
# An article is posted as a teaser (compose_text), so its links aren't in the text
|
||||
body, anchors = (None, []) if post.note.title else \
|
||||
plain_text_links(post.note.content, links="anchor")
|
||||
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body)
|
||||
lead = ""
|
||||
if post.reply_to and not reply:
|
||||
# "↩ Replying to <name>", the name linking to the post (v2.20.0)
|
||||
lead = reply_line(post, with_url=False)
|
||||
anchors = [(reply_label(post), post.reply_to), *anchors]
|
||||
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body,
|
||||
lead=lead)
|
||||
record = {"$type": self.POST, "text": text, "createdAt": self._now()}
|
||||
if reply:
|
||||
record["reply"] = reply
|
||||
facets = bluesky_facets(text, anchors)
|
||||
if facets:
|
||||
record["facets"] = facets
|
||||
@@ -459,7 +536,7 @@ class BlueskyClient:
|
||||
existing = self._find_existing(post.permalink)
|
||||
if existing:
|
||||
return existing
|
||||
record = self._record(post)
|
||||
record = self._record(post, reply=self._reply_ref(post))
|
||||
response = self._call("POST", "com.atproto.repo.createRecord",
|
||||
json={"repo": self.did or self.identifier, "collection": self.POST,
|
||||
"record": record})
|
||||
@@ -517,8 +594,9 @@ class BlueskyClient:
|
||||
)
|
||||
|
||||
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
|
||||
"""Likes, reposts, quotes and direct replies of one of our posts"""
|
||||
from starpunk.backfeed import Harvest, Response
|
||||
"""Likes, reposts, quotes and replies of one of our posts, with the
|
||||
replies to those replies (v2.20.2)"""
|
||||
from starpunk.backfeed import Harvest, Response, thread_members
|
||||
|
||||
if not self.did:
|
||||
with self._lock:
|
||||
@@ -541,10 +619,26 @@ class BlueskyClient:
|
||||
complete = complete and done
|
||||
posts = [self._post_response("quote", view) for view in quotes]
|
||||
thread = self._call("GET", "app.bsky.feed.getPostThread",
|
||||
params={"uri": remote_id, "depth": 1, "parentHeight": 0}).json()
|
||||
for item in (thread.get("thread") or {}).get("replies") or []:
|
||||
if item.get("post"): # skips blocked and deleted replies
|
||||
posts.append(self._post_response("reply", item["post"]))
|
||||
params={"uri": remote_id, "depth": BLUESKY_THREAD_DEPTH,
|
||||
"parentHeight": 0}).json()
|
||||
# The whole conversation under our post (v2.20.2)
|
||||
views, stack = {}, [(remote_id, item) for item in (thread.get("thread") or {}).get("replies") or []]
|
||||
while stack:
|
||||
parent, item = stack.pop()
|
||||
post = item.get("post")
|
||||
if not post: # blocked or deleted: its replies go with it
|
||||
continue
|
||||
views[post["uri"]] = (parent, post)
|
||||
stack.extend((post["uri"], child) for child in item.get("replies") or [])
|
||||
kept = thread_members([{"id": uri, "parent": parent,
|
||||
"own": (post.get("author") or {}).get("did") == self.did}
|
||||
for uri, (parent, post) in views.items()], remote_id)
|
||||
for uri, (_, post) in sorted(views.items(), key=lambda v: v[1][1].get("indexedAt") or ""):
|
||||
if uri in kept:
|
||||
response = self._post_response("reply", post)
|
||||
response.parent_id = kept[uri]
|
||||
response.own = response.author_id == self.did
|
||||
posts.append(response)
|
||||
# Post views carry no bio; fetch them to honour #nobridge
|
||||
dids = list({p.author_id for p in posts if p.author_id})
|
||||
bios = {}
|
||||
@@ -555,7 +649,7 @@ class BlueskyClient:
|
||||
for post in posts:
|
||||
post.author_bio = bios.get(post.author_id, "")
|
||||
found.extend(posts)
|
||||
return Harvest([r for r in found if r.author_id != self.did], complete)
|
||||
return Harvest([r for r in found if r.author_id != self.did or r.own], complete)
|
||||
|
||||
def delete(self, remote_id: str) -> None:
|
||||
# at://<did>/app.bsky.feed.post/<rkey>; deleting a missing record succeeds
|
||||
@@ -838,15 +932,26 @@ class ThreadsClient:
|
||||
def _text(self, post: "Post") -> str:
|
||||
# Links written "text (url)"; the first one gets Threads' preview card.
|
||||
# Meta allows 5 different links per post and the permalink is one.
|
||||
body = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
|
||||
# A reply to someone else's post says so, with its link (v2.20.0);
|
||||
# one to our own Threads post is a native reply instead
|
||||
lead = "" if self._reply_to_id(post) else reply_line(post)
|
||||
links = THREADS_MAX_LINKS - 1 - (1 if lead else 0)
|
||||
body = plain_text(post.note.content, links="inline", max_urls=links)
|
||||
limit = post.max_chars
|
||||
while True:
|
||||
text = compose_text(post.note, post.permalink, [], limit, body=body)
|
||||
text = compose_text(post.note, post.permalink, [], limit, body=body, lead=lead)
|
||||
over = threads_length(text) - post.max_chars
|
||||
if over <= 0:
|
||||
return text
|
||||
limit -= over
|
||||
|
||||
@staticmethod
|
||||
def _reply_to_id(post: "Post") -> Optional[str]:
|
||||
"""Media id of our own Threads post this replies to. The API can't look
|
||||
up other people's posts by URL, so only those thread natively."""
|
||||
remote = post.reply_remote_id or ""
|
||||
return remote if post.reply_to and remote.isdigit() else None
|
||||
|
||||
def _image_container(self, uid: str, media: dict, **extra) -> str:
|
||||
params = {"media_type": "IMAGE", "image_url": media["url"], **extra}
|
||||
if media["description"]:
|
||||
@@ -869,6 +974,8 @@ class ThreadsClient:
|
||||
return self._call("POST", f"{uid}/threads", media_type="TEXT",
|
||||
text=self._repost_text(post))["id"]
|
||||
common = {"text": self._text(post)}
|
||||
if self._reply_to_id(post):
|
||||
common["reply_to_id"] = self._reply_to_id(post)
|
||||
topic = threads_topic(post.tags)
|
||||
if topic:
|
||||
common["topic_tag"] = topic
|
||||
@@ -929,15 +1036,17 @@ class ThreadsClient:
|
||||
|
||||
# --- Backfeed (v2.13.0, ADR-073) ---
|
||||
|
||||
REPLY_FIELDS = "id,text,username,permalink,timestamp,profile_picture_url,hide_status"
|
||||
REPLY_FIELDS = "id,text,username,permalink,timestamp,profile_picture_url,hide_status,replied_to"
|
||||
SHOWN = (None, "NOT_HUSHED", "UNHUSHED") # hide_status values of replies nobody hid
|
||||
|
||||
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
|
||||
"""
|
||||
Direct replies to one of our posts, and its like/repost/quote counts.
|
||||
The API doesn't say who liked, reposted or quoted (ADR-073 Decision 7).
|
||||
Replies to one of our posts, with the replies to those replies
|
||||
(/conversation, v2.20.2; /replies only has the first level), and its
|
||||
like/repost/quote counts. The API doesn't say who liked, reposted or
|
||||
quoted (ADR-073 Decision 7).
|
||||
"""
|
||||
from starpunk.backfeed import Harvest, Response, text_html
|
||||
from starpunk.backfeed import Harvest, Response, text_html, thread_members
|
||||
|
||||
self._me()
|
||||
replies, after, complete = [], None, True
|
||||
@@ -945,7 +1054,7 @@ class ThreadsClient:
|
||||
params = {"fields": self.REPLY_FIELDS}
|
||||
if after:
|
||||
params["after"] = after
|
||||
data = self._call("GET", f"{remote_id}/replies", **params)
|
||||
data = self._call("GET", f"{remote_id}/conversation", **params)
|
||||
page = data.get("data") or []
|
||||
replies.extend(page)
|
||||
if len(replies) >= cap:
|
||||
@@ -955,14 +1064,18 @@ class ThreadsClient:
|
||||
after = (paging.get("cursors") or {}).get("after")
|
||||
if not page or not after or not paging.get("next"):
|
||||
break
|
||||
# Private accounts have no username or permalink: nothing public to show
|
||||
shown = [r for r in replies if r.get("username") and r.get("permalink")
|
||||
and r.get("hide_status") in self.SHOWN]
|
||||
kept = thread_members([{"id": r["id"], "parent": (r.get("replied_to") or {}).get("id"),
|
||||
"own": r["username"] == self.username} for r in shown], remote_id)
|
||||
found = []
|
||||
for reply in replies:
|
||||
username = reply.get("username")
|
||||
# Private accounts have no username or permalink: nothing public to show
|
||||
if (not username or not reply.get("permalink") or username == self.username
|
||||
or reply.get("hide_status") not in self.SHOWN):
|
||||
for reply in sorted(shown, key=lambda r: r.get("timestamp") or ""):
|
||||
if reply["id"] not in kept:
|
||||
continue
|
||||
username = reply["username"]
|
||||
found.append(Response(
|
||||
parent_id=kept[reply["id"]], own=username == self.username,
|
||||
kind="reply", source_url=reply["permalink"], remote_id=reply["id"],
|
||||
author_id=username, author_name=username,
|
||||
author_url=f"https://www.threads.com/@{username}",
|
||||
@@ -1006,6 +1119,22 @@ class Post:
|
||||
kind: str = "note" # note, like or repost (v2.5.0)
|
||||
target_url: Optional[str] = None # what a like/repost points at
|
||||
target_title: Optional[str] = None # its fetched title, if any
|
||||
reply_to: Optional[str] = None # what a reply answers (v2.20.0)
|
||||
reply_name: Optional[str] = None # its author (or title), from the reply context
|
||||
reply_remote_id: Optional[str] = None # its crosspost on this target, if it's one of ours
|
||||
|
||||
|
||||
def reply_line(post: "Post", with_url: bool = True) -> str:
|
||||
"""'↩ Replying to <who>' (and the URL) for replies that can't thread natively (v2.20.0)"""
|
||||
if not post.reply_to:
|
||||
return ""
|
||||
who = post.reply_name or urlparse(post.reply_to).hostname or post.reply_to
|
||||
return f"↩ Replying to {who}" + (f"\n{post.reply_to}" if with_url else "")
|
||||
|
||||
|
||||
def reply_label(post: "Post") -> str:
|
||||
"""The linked part of a Bluesky reply line"""
|
||||
return post.reply_name or urlparse(post.reply_to).hostname or post.reply_to
|
||||
|
||||
|
||||
def compose_repost_text(post: "Post", comment: str, max_chars: int) -> str:
|
||||
@@ -1195,7 +1324,7 @@ def article_teaser(note) -> str:
|
||||
|
||||
|
||||
def compose_text(note, permalink: str, tags: List[str], max_chars: int,
|
||||
body: Optional[str] = None) -> str:
|
||||
body: Optional[str] = None, lead: str = "") -> str:
|
||||
"""
|
||||
Full note (Markdown unless `body` is given), then the permalink and
|
||||
hashtags; shortened at a word boundary if too long. On short targets the
|
||||
@@ -1216,9 +1345,14 @@ def compose_text(note, permalink: str, tags: List[str], max_chars: int,
|
||||
if getattr(note, "bookmark_of", None):
|
||||
# A bookmark leads with what it bookmarks (v2.9.0)
|
||||
body = f"🔖 {note.bookmark_of}\n\n{body}".strip()
|
||||
if len(body) + len(tail) > max_chars:
|
||||
room = max_chars - len(tail) - 1
|
||||
body = body[:room].rsplit(" ", 1)[0].rstrip() + "…"
|
||||
# A reply that can't thread says what it replies to (v2.20.0); the lead
|
||||
# is kept whole and the body shortened instead
|
||||
lead = lead.strip()
|
||||
head = f"{lead}\n\n" if lead and body else lead
|
||||
if len(head) + len(body) + len(tail) > max_chars:
|
||||
room = max_chars - len(tail) - len(head) - 1
|
||||
body = body[:max(room, 0)].rsplit(" ", 1)[0].rstrip() + "…"
|
||||
body = head + body
|
||||
# A photo post may have no text: start with the link (v2.6.5)
|
||||
return (body + tail).lstrip()
|
||||
|
||||
@@ -1505,6 +1639,48 @@ def _target_title(url: Optional[str]) -> Optional[str]:
|
||||
return " ".join(title.split())[:200] or None
|
||||
|
||||
|
||||
def _own_note_id(url: str) -> Optional[int]:
|
||||
"""The note a URL is, if it's one of ours: its page on this site, or one
|
||||
of its crossposts on any network (v2.20.0)"""
|
||||
db = get_db(current_app)
|
||||
row = db.execute("SELECT note_id FROM syndications WHERE remote_url = ? AND state = 'synced'",
|
||||
(url,)).fetchone()
|
||||
if row:
|
||||
return row["note_id"]
|
||||
site = urlparse(current_app.config.get("SITE_URL", ""))
|
||||
parsed = urlparse(url)
|
||||
parts = [p for p in parsed.path.split("/") if p]
|
||||
if parsed.netloc != site.netloc or len(parts) != 2 or \
|
||||
parts[0] not in ("note", "notes", "article", "articles"):
|
||||
return None
|
||||
row = db.execute("SELECT id FROM notes WHERE slug = ? AND deleted_at IS NULL",
|
||||
(parts[1],)).fetchone()
|
||||
return row["id"] if row else None
|
||||
|
||||
|
||||
def _reply_fields(url: Optional[str], target_id: str) -> dict:
|
||||
"""Post fields for a reply: who it answers, and its crosspost on this target if it's ours"""
|
||||
if not url:
|
||||
return {}
|
||||
fields = {"reply_to": url}
|
||||
try:
|
||||
from starpunk.webmention_queries import get_reply_context
|
||||
context = get_reply_context(url) or {}
|
||||
if context.get("fetch_status") == "fetched":
|
||||
name = context.get("author_name") or context.get("title")
|
||||
fields["reply_name"] = " ".join(name.split())[:100] if name else None
|
||||
note_id = _own_note_id(url)
|
||||
if note_id is not None:
|
||||
row = get_db(current_app).execute(
|
||||
"SELECT remote_id FROM syndications WHERE note_id = ? AND target = ? "
|
||||
"AND state = 'synced' AND remote_id IS NOT NULL", (note_id, target_id)).fetchone()
|
||||
if row:
|
||||
fields["reply_remote_id"] = row["remote_id"]
|
||||
except Exception as e:
|
||||
current_app.logger.info(f"Reply details for {url}: {e}")
|
||||
return fields
|
||||
|
||||
|
||||
def _reconcile(row, target: Target) -> tuple:
|
||||
"""Bring the remote post in line with the note. Returns (remote_id, remote_url)."""
|
||||
from starpunk.notes import get_note
|
||||
@@ -1533,6 +1709,7 @@ def _reconcile(row, target: Target) -> tuple:
|
||||
kind=note.kind,
|
||||
target_url=note.reacts_to,
|
||||
target_title=_target_title(note.reacts_to),
|
||||
**(_reply_fields(note.in_reply_to, target.id) if note.kind == "note" else {}),
|
||||
)
|
||||
|
||||
if row["remote_id"]:
|
||||
|
||||
@@ -107,9 +107,32 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
|
||||
else:
|
||||
grouped['mentions'].append(wm)
|
||||
|
||||
grouped['reply_threads'] = reply_threads(grouped['replies'])
|
||||
return grouped
|
||||
|
||||
|
||||
def reply_threads(replies: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Replies nested under the replies they answer (v2.20.2): each gets
|
||||
'children'; the top-level ones are returned. A reply whose parent isn't
|
||||
shown (hidden, removed) is shown at the top level.
|
||||
"""
|
||||
by_id = {}
|
||||
for wm in replies:
|
||||
wm['children'] = []
|
||||
if wm.get('remote_id'):
|
||||
by_id[(wm.get('origin'), wm['remote_id'])] = wm
|
||||
top = []
|
||||
for wm in replies:
|
||||
parent = by_id.get((wm.get('origin'), wm.get('parent_remote_id'))) \
|
||||
if wm.get('parent_remote_id') else None
|
||||
if parent is not None and parent is not wm:
|
||||
parent['children'].append(wm)
|
||||
else:
|
||||
top.append(wm)
|
||||
return top
|
||||
|
||||
|
||||
def response_counts(note_ids: List[int]) -> Dict[int, Dict[str, int]]:
|
||||
"""
|
||||
Likes, reposts, replies and quotes per note, for lists of notes
|
||||
|
||||
@@ -22,6 +22,22 @@ from urllib.parse import urlparse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SKIPPED_NOTE = "Not sent: links to this site and WEBMENTION_SKIP_HOSTS are skipped"
|
||||
|
||||
|
||||
def skipped_hosts(config) -> set[str]:
|
||||
"""Hosts that never get webmentions: this site and WEBMENTION_SKIP_HOSTS (v2.18.3)"""
|
||||
hosts = set(config.get("WEBMENTION_SKIP_HOSTS") or [])
|
||||
site = urlparse(config.get("SITE_URL") or "").hostname
|
||||
if site:
|
||||
hosts.add(site.lower())
|
||||
return hosts
|
||||
|
||||
|
||||
def is_skipped(url: str, hosts: set[str]) -> bool:
|
||||
"""True if url is on one of the skipped hosts (exact host, any port)"""
|
||||
return (urlparse(url).hostname or "").lower() in hosts
|
||||
|
||||
|
||||
def extract_links_from_html(html_content: str, exclude_code_blocks: bool = True) -> list[str]:
|
||||
"""
|
||||
@@ -189,17 +205,26 @@ def queue_webmentions_for_note(
|
||||
Returns:
|
||||
Number of webmentions queued
|
||||
"""
|
||||
# Links to this site or a skipped host are dropped before the cap (v2.18.3):
|
||||
# this site only takes webmentions for its posts, and its own public
|
||||
# address may not be reachable from inside the container
|
||||
from flask import current_app, has_app_context
|
||||
hosts = skipped_hosts(current_app.config) if has_app_context() else set()
|
||||
|
||||
def wanted(urls):
|
||||
return [u for u in urls if not is_skipped(u, hosts)]
|
||||
|
||||
# Targets for the current content: reply target first, then links in
|
||||
# document order, at most 10
|
||||
links = extract_links_from_html(content_html) if content_html else []
|
||||
current = _dedupe(([in_reply_to] if in_reply_to else []) + links)[:10]
|
||||
current = wanted(_dedupe(([in_reply_to] if in_reply_to else []) + links))[:10]
|
||||
|
||||
if old_content_html is not None:
|
||||
# Update (spec 3.1.4): also re-send to every target notified before,
|
||||
# and to links the edit removed, so they can update or remove the
|
||||
# mention. These are not capped.
|
||||
removed = [u for u in extract_links_from_html(old_content_html) if u not in links]
|
||||
target_urls = _dedupe(current + previous_targets(db_path, note_id) + removed)
|
||||
target_urls = _dedupe(current + wanted(previous_targets(db_path, note_id) + removed))
|
||||
else:
|
||||
target_urls = current
|
||||
|
||||
|
||||
@@ -117,6 +117,12 @@ class WebmentionWorker(threading.Thread):
|
||||
|
||||
# Phase 3: Process outgoing webmentions
|
||||
processed += self._process_pending_sends()
|
||||
# Reply contexts stored before v2.19.0 are fetched once more
|
||||
try:
|
||||
from starpunk.reply_context import refresh_stale
|
||||
refresh_stale(self.db_path, self.site_url)
|
||||
except Exception as e:
|
||||
logger.error(f"Reply context refresh failed: {e}", exc_info=True)
|
||||
# Local copies of response avatars (v2.13.0)
|
||||
self._avatar_upkeep.run(self.db_path, self.app.config['DATA_PATH'],
|
||||
self.site_url)
|
||||
@@ -205,6 +211,40 @@ class WebmentionWorker(threading.Thread):
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to cleanup orphaned items: {e}")
|
||||
|
||||
self._drop_skipped_targets()
|
||||
|
||||
def _skipped_hosts(self) -> set:
|
||||
from starpunk.webmention_sender import skipped_hosts
|
||||
return skipped_hosts(self.app.config) if self.app is not None else set()
|
||||
|
||||
def _drop_skipped_targets(self):
|
||||
"""
|
||||
Stop sending to this site and WEBMENTION_SKIP_HOSTS (v2.18.3)
|
||||
|
||||
Queued or failed sends to those hosts are set to 'no_endpoint' (nothing
|
||||
to send to), so they leave the retry queue and the admin Log.
|
||||
"""
|
||||
from starpunk.webmention_sender import SKIPPED_NOTE, is_skipped
|
||||
|
||||
hosts = self._skipped_hosts()
|
||||
if not hosts:
|
||||
return
|
||||
try:
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
rows = conn.execute(
|
||||
"SELECT id, target_url FROM webmention_queue WHERE status IN ('pending', 'failed')"
|
||||
).fetchall()
|
||||
ids = [(SKIPPED_NOTE, i) for i, url in rows if is_skipped(url, hosts)]
|
||||
if ids:
|
||||
conn.executemany(
|
||||
"UPDATE webmention_queue SET status = 'no_endpoint', last_error = ? WHERE id = ?",
|
||||
ids)
|
||||
conn.commit()
|
||||
logger.info(f"Dropped {len(ids)} queued webmention(s) to skipped hosts")
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to drop skipped webmention targets: {e}")
|
||||
|
||||
def _process_pending_validations(self) -> int:
|
||||
"""
|
||||
Process pending incoming webmentions
|
||||
@@ -583,6 +623,15 @@ class WebmentionWorker(threading.Thread):
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
from starpunk.webmention_sender import SKIPPED_NOTE, is_skipped
|
||||
if is_skipped(target_url, self._skipped_hosts()):
|
||||
conn = sqlite3.connect(self.db_path)
|
||||
conn.execute("UPDATE webmention_queue SET status = 'no_endpoint', last_error = ? WHERE id = ?",
|
||||
(SKIPPED_NOTE, queue_id))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return
|
||||
|
||||
try:
|
||||
# Discover endpoint if not cached
|
||||
endpoint = cached_endpoint
|
||||
|
||||
+45
-25
@@ -8,7 +8,7 @@
|
||||
--font-reading: Charter, 'Bitstream Charter', 'Sitka Text', Cambria, 'Noto Serif', Georgia, serif;
|
||||
--font-mono: ui-monospace, 'SF Mono', Monaco, Consolas, 'Liberation Mono', monospace;
|
||||
--spacing-xs: 0.25rem; --spacing-sm: 0.5rem; --spacing-md: 1rem; --spacing-lg: 2rem; --spacing-xl: 4rem;
|
||||
--max-width: 40rem; --border-radius: 6px;
|
||||
--max-width: 48rem; --frame-width: 60rem; --border-radius: 6px;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
@@ -40,6 +40,9 @@ pre code { padding: 0; background: none; }
|
||||
|
||||
/* Page frame */
|
||||
.site-header, main, .site-footer { width: 100%; max-width: var(--max-width); margin-left: auto; margin-right: auto; }
|
||||
/* Header and footer share one width on every page, public and admin (v2.18.2);
|
||||
only the content column (main) differs: 48rem to read, 60rem for admin tables. */
|
||||
.site-header, .site-footer { max-width: var(--frame-width); }
|
||||
main { flex: 1; margin-bottom: var(--spacing-xl); }
|
||||
|
||||
.site-header {
|
||||
@@ -91,6 +94,15 @@ main { flex: 1; margin-bottom: var(--spacing-xl); }
|
||||
.e-content blockquote { margin: 0 0 var(--spacing-md); padding-left: var(--spacing-md); border-left: 3px solid var(--color-border); color: var(--color-text-light); }
|
||||
.e-content ul, .e-content ol { margin: 0 0 var(--spacing-md) 1.25rem; }
|
||||
.e-content img { max-width: 100%; height: auto; }
|
||||
/* Tables (v2.18.2): a card that fills the column, in the page's sans font at
|
||||
the reading size; scrolls sideways when wider than the column. Words in
|
||||
cells are never split (overflow-wrap: anywhere above is for long URLs in
|
||||
prose; in a cell it squeezed columns to a few letters). */
|
||||
.e-content .table-wrap { overflow-x: auto; margin: 0 0 var(--spacing-md); background: var(--color-bg-alt); border: 1px solid var(--color-border); border-radius: var(--border-radius); }
|
||||
.e-content table { width: 100%; border-collapse: collapse; font-family: var(--font-body); font-size: 1.0625rem; line-height: 1.6; }
|
||||
.e-content th, .e-content td { padding: 0.9rem 1.25rem; text-align: left; vertical-align: top; overflow-wrap: normal; }
|
||||
.e-content th { font-weight: 600; white-space: nowrap; background: color-mix(in srgb, var(--color-text) 6%, var(--color-bg-alt)); }
|
||||
.e-content td:first-child { font-weight: 600; }
|
||||
.e-content h1, .e-content h2, .e-content h3 { font-family: var(--font-body); }
|
||||
.reaction { margin-bottom: var(--spacing-sm); font-size: 0.9375rem; color: var(--color-text-light); overflow-wrap: anywhere; }
|
||||
.reaction-verb { font-weight: 600; margin-right: 0.25rem; }
|
||||
@@ -266,30 +278,27 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
|
||||
|
||||
/* Webmention Styles (v1.6.0 Phase 5) */
|
||||
|
||||
/* Reply Context */
|
||||
.reply-context {
|
||||
margin-bottom: var(--spacing-md);
|
||||
padding: var(--spacing-sm);
|
||||
border-left: 3px solid var(--color-link);
|
||||
background: var(--color-bg-alt);
|
||||
}
|
||||
|
||||
.reply-context .reply-label {
|
||||
font-size: 0.875rem;
|
||||
color: var(--color-text-light);
|
||||
display: block;
|
||||
margin-bottom: var(--spacing-xs);
|
||||
}
|
||||
|
||||
.reply-context blockquote {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.reply-context .p-content {
|
||||
font-style: italic;
|
||||
margin-bottom: var(--spacing-xs);
|
||||
}
|
||||
/* Reply Context (card v2.19.0) */
|
||||
.reply-context { margin-bottom: var(--spacing-md); }
|
||||
.reply-context .reply-label { display: block; margin-bottom: var(--spacing-xs); font-size: 0.875rem; font-weight: 600; color: var(--color-text-light); }
|
||||
.reply-card { margin: 0; padding: var(--spacing-sm) var(--spacing-md); border-left: 3px solid var(--color-link); background: var(--color-bg-alt); border-radius: 0 var(--border-radius) var(--border-radius) 0; overflow-wrap: anywhere; }
|
||||
.reply-author { margin-bottom: var(--spacing-xs); }
|
||||
.reply-author .h-card { display: flex; align-items: center; gap: var(--spacing-sm); }
|
||||
.reply-author img, .reply-author .face-initial { width: 40px; height: 40px; border-radius: 50%; flex: none; }
|
||||
.reply-author .face-initial { display: inline-flex; align-items: center; justify-content: center; color: #fff; font-weight: 700; }
|
||||
.reply-who { display: flex; flex-wrap: wrap; column-gap: 0.4rem; line-height: 1.3; min-width: 0; }
|
||||
.reply-who .p-name { font-weight: 600; color: var(--color-text); }
|
||||
.reply-who a.p-name:hover { color: var(--color-link); }
|
||||
.reply-handle { font-size: 0.8125rem; color: var(--color-text-light); }
|
||||
.reply-title { margin: 0 0 var(--spacing-xs); font-weight: 600; }
|
||||
.reply-content { font-size: 0.9375rem; }
|
||||
.reply-content p { margin-bottom: var(--spacing-xs); }
|
||||
.reply-content > :last-child { margin-bottom: 0; }
|
||||
.reply-photo { display: block; max-width: 100%; max-height: 20rem; width: auto; height: auto; margin-top: var(--spacing-sm); border-radius: var(--border-radius); }
|
||||
.reply-source { margin-top: var(--spacing-xs); font-size: 0.8125rem; }
|
||||
.reply-source a { color: var(--color-text-light); }
|
||||
.reply-source a:hover { color: var(--color-link); }
|
||||
.reply-quote { display: block; margin-top: 0.15rem; padding-left: var(--spacing-sm); border-left: 2px solid var(--color-border); color: var(--color-text-light); font-style: italic; quotes: none; }
|
||||
|
||||
/* Webmentions Section */
|
||||
.webmentions {
|
||||
@@ -382,6 +391,10 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
|
||||
.webmention-reply .webmention-date::after { content: ""; position: absolute; inset: 0; }
|
||||
.webmention-reply .webmention-author a, .webmention-reply .e-content a { position: relative; z-index: 1; }
|
||||
.webmention-reply:hover { cursor: pointer; }
|
||||
/* Replies to replies (v2.20.2): nested, with a line down the side */
|
||||
.webmention-thread { position: relative; z-index: 1; margin-top: var(--spacing-sm); padding-left: var(--spacing-md); border-left: 2px solid var(--color-border); }
|
||||
.webmention-thread .webmention-reply { margin-bottom: var(--spacing-sm); padding: var(--spacing-xs) 0 var(--spacing-xs) var(--spacing-xs); background: none; }
|
||||
.webmention-thread .webmention-reply:last-child { margin-bottom: 0; }
|
||||
|
||||
/* Counts in lists of notes (v2.13.4) */
|
||||
.list-counts { display: inline-flex; gap: 0.6rem; color: var(--color-text-light); }
|
||||
@@ -465,6 +478,13 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
|
||||
.badge-like { background: #FECACA; color: #991B1B; }
|
||||
.badge-repost { background: #D1FAE5; color: #065F46; }
|
||||
.badge-bookmark { background: #FEF3C7; color: #92400E; }
|
||||
.badge-error { background: var(--color-error); color: #fff; }
|
||||
|
||||
/* Admin Log (v2.18.0) */
|
||||
.log-table td { vertical-align: top; }
|
||||
.log-table .button { white-space: nowrap; }
|
||||
.log-error code { white-space: pre-wrap; overflow-wrap: anywhere; font-size: 0.85rem; }
|
||||
.backfeed-status { margin-bottom: var(--spacing-lg); }
|
||||
.badge-mention { background: #E5E7EB; color: #374151; }
|
||||
|
||||
.status-badge {
|
||||
|
||||
@@ -8,7 +8,10 @@
|
||||
<nav class="admin-nav">
|
||||
<a href="{{ url_for('admin.dashboard') }}">Dashboard</a>
|
||||
<a href="{{ url_for('admin.new_note_form') }}">New Note</a>
|
||||
{% if config.WEBMENTION_ENABLED %}<a href="{{ url_for('admin.webmentions') }}">Webmentions</a>{% endif %}
|
||||
<a href="{{ url_for('admin.error_log') }}">Log{% if error_count %} <span class="badge badge-error">{{ error_count }}</span>{% endif %}</a>
|
||||
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
|
||||
<a href="{{ url_for('admin.settings') }}">Settings</a>
|
||||
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||
<button type="submit" class="button button-secondary">Logout</button>
|
||||
</form>
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
{% extends "admin/base.html" %}
|
||||
|
||||
{% block title %}Log - StarPunk Admin{% endblock %}
|
||||
|
||||
{% block admin_content %}
|
||||
<div class="log-admin">
|
||||
<h2>Log</h2>
|
||||
<p class="text-muted">What is failing right now. An entry goes away once the next attempt succeeds.</p>
|
||||
|
||||
{% if entries %}
|
||||
<div class="table-scroll">
|
||||
<table class="note-table log-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>When (UTC)</th>
|
||||
<th>What</th>
|
||||
<th>Post</th>
|
||||
<th>Error</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for e in entries %}
|
||||
<tr>
|
||||
<td class="note-date">{{ e.when or '' }}</td>
|
||||
<td>
|
||||
{{ e.area }}
|
||||
{% if e.detail %}<br><small class="note-slug">{% if e.detail.startswith('http') %}<a href="{{ e.detail.split(' ')[0] }}" target="_blank" rel="noopener">{{ e.detail|truncate(60) }}</a>{% else %}{{ e.detail|truncate(80) }}{% endif %}</small>{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
{% if e.note %}
|
||||
<a href="{{ url_for('admin.edit_note_form', note_id=e.note.id) }}">{{ (e.note.title or e.note.content or e.note.slug)|truncate(60) }}</a>
|
||||
{% if e.note.published %}<br><small><a href="{{ e.note.url }}" target="_blank">View</a></small>{% endif %}
|
||||
{% else %}
|
||||
<span class="text-muted">Site-wide</span>
|
||||
{% endif %}
|
||||
</td>
|
||||
<td class="log-error"><code>{{ e.error }}</code></td>
|
||||
<td>
|
||||
{% if e.action %}
|
||||
<form method="POST" action="{{ url_for(e.action[0], syndication_id=e.action[1]) }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||
<button type="submit" class="button button-small">{{ e.action[2] }}</button>
|
||||
</form>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="empty-state">
|
||||
<p>No errors. Crossposts, response checks, webmentions, WebSub and notifications are all working.</p>
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,30 @@
|
||||
{% extends "admin/base.html" %}
|
||||
|
||||
{% block title %}Settings - StarPunk Admin{% endblock %}
|
||||
|
||||
{% block admin_content %}
|
||||
<div class="settings-admin">
|
||||
<h2>Settings</h2>
|
||||
|
||||
{# Notifications (v2.14.0, ADR-074; moved here in v2.18.0) #}
|
||||
<section class="notify-status">
|
||||
<h3>Notifications</h3>
|
||||
{% if notify.enabled %}
|
||||
<p>Sending {{ notify.events|join(', ') }} to {{ notify.services|join(', ') }} at {{ notify.priority }} priority.</p>
|
||||
{% if notify.last_error %}<p class="flash flash-error">Last send failed: {{ notify.last_error }}</p>{% endif %}
|
||||
<form method="POST" action="{{ url_for('admin.notifications_test') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||
<button type="submit" class="button">Send test notification</button>
|
||||
</form>
|
||||
{% else %}
|
||||
<p>Off. Set <code>NOTIFY_URLS</code> to one or more Apprise URLs to be notified of new responses.</p>
|
||||
{% endif %}
|
||||
</section>
|
||||
|
||||
{# Outgoing webmentions (v2.18.3) #}
|
||||
<section>
|
||||
<h3>Webmentions</h3>
|
||||
<p>Never sent to: {% for h in skip_hosts %}<code>{{ h }}</code>{{ ", " if not loop.last }}{% endfor %}.
|
||||
This site is always skipped; add others with <code>WEBMENTION_SKIP_HOSTS</code>.</p>
|
||||
</section>
|
||||
</div>
|
||||
{% endblock %}
|
||||
@@ -6,35 +6,10 @@
|
||||
<div class="webmentions-admin">
|
||||
<h2>Webmentions</h2>
|
||||
|
||||
<div class="dashboard-actions">
|
||||
<a href="{{ url_for('admin.dashboard') }}" class="button">
|
||||
Back to Dashboard
|
||||
</a>
|
||||
</div>
|
||||
|
||||
{# Notifications (v2.14.0, ADR-074) #}
|
||||
{% if notify %}
|
||||
<section class="notify-status">
|
||||
<h3>Notifications</h3>
|
||||
{% if notify.enabled %}
|
||||
<p>Sending {{ notify.events|join(', ') }} to {{ notify.services|join(', ') }} at {{ notify.priority }} priority.</p>
|
||||
{% if notify.last_error %}<p class="flash flash-error">Last send failed: {{ notify.last_error }}</p>{% endif %}
|
||||
<form method="POST" action="{{ url_for('admin.notifications_test') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||
<button type="submit" class="button">Send test notification</button>
|
||||
</form>
|
||||
{% else %}
|
||||
<p>Off. Set <code>NOTIFY_URLS</code> to one or more Apprise URLs to be notified of new responses.</p>
|
||||
{% endif %}
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
{# Backfeed (v2.13.0, ADR-073) #}
|
||||
{% if backfeed and backfeed.enabled %}
|
||||
<section class="backfeed-status">
|
||||
<h3>Backfeed</h3>
|
||||
{% for silo, message in backfeed.denied.items() %}
|
||||
<p class="flash flash-error">{{ silo }}: {{ message }}</p>
|
||||
{% endfor %}
|
||||
{% if backfeed.remaining %}
|
||||
<p>Backfill started {{ backfeed.started }} UTC: {{ backfeed.remaining }} crosspost(s) left to check.</p>
|
||||
{% elif backfeed.finished %}
|
||||
@@ -42,7 +17,7 @@
|
||||
{% else %}
|
||||
<p>No backfill has been run. It checks every crosspost once for responses, however old the note.</p>
|
||||
{% endif %}
|
||||
{% if backfeed.errors %}<p>{{ backfeed.errors }} crosspost(s) had an error at their last check (see each note's edit page).</p>{% endif %}
|
||||
{% if backfeed.errors or backfeed.denied %}<p><a href="{{ url_for('admin.error_log') }}">Response checks have errors: see the Log.</a></p>{% endif %}
|
||||
<form method="POST" action="{{ url_for('admin.backfeed_backfill') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||
<button type="submit" class="button"{% if backfeed.remaining %} disabled{% endif %}>Backfill responses</button>
|
||||
</form>
|
||||
|
||||
+4
-24
@@ -1,6 +1,7 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "partials/media.html" import display_media %}
|
||||
{% from "partials/reaction.html" import reaction %}
|
||||
{% from "partials/reply_context.html" import reply_card %}
|
||||
{% import "partials/responses.html" as responses with context %}
|
||||
|
||||
{% block title %}{% if note.title %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.bookmark_of %}Bookmark{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
|
||||
@@ -15,30 +16,9 @@
|
||||
<p class="p-summary">{{ note.summary }}</p>
|
||||
{% endif %}
|
||||
|
||||
{# Reply context (v1.6.0 Phase 4) - displayed if this is a reply post #}
|
||||
{# Reply context (v1.6.0 Phase 4; card v2.19.0) - displayed if this is a reply post #}
|
||||
{% if note.in_reply_to %}
|
||||
<div class="reply-context">
|
||||
<span class="reply-label">In reply to:</span>
|
||||
{# u-in-reply-to on the h-cite itself makes it the entry's property (Post Type Discovery) #}
|
||||
<blockquote class="u-in-reply-to h-cite">
|
||||
{% if reply_context and reply_context.fetch_status == 'fetched' %}
|
||||
{% if reply_context.content_snippet %}
|
||||
<p class="p-content">{{ reply_context.content_snippet }}</p>
|
||||
{% endif %}
|
||||
<cite>
|
||||
<a class="u-url" href="{{ note.in_reply_to }}">
|
||||
{% if reply_context.author_name %}{{ reply_context.author_name }}{% endif %}
|
||||
{% if reply_context.site_name %} on {{ reply_context.site_name }}{% endif %}
|
||||
{% if not reply_context.author_name and not reply_context.site_name %}{{ note.in_reply_to }}{% endif %}
|
||||
</a>
|
||||
</cite>
|
||||
{% else %}
|
||||
<cite>
|
||||
<a class="u-url" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
|
||||
</cite>
|
||||
{% endif %}
|
||||
</blockquote>
|
||||
</div>
|
||||
{{ reply_card(note.in_reply_to, reply_context or (note.context if note.context is defined else None)) }}
|
||||
{% endif %}
|
||||
|
||||
{{ reaction(note) }}
|
||||
@@ -94,7 +74,7 @@
|
||||
{% if replies %}
|
||||
<div class="webmention-group webmention-replies">
|
||||
<h3>{{ replies|length }} {{ 'Reply' if replies|length == 1 else 'Replies' }}</h3>
|
||||
{% for wm in replies %}{{ responses.reply(wm) }}{% endfor %}
|
||||
{% for wm in reply_threads or replies %}{{ responses.reply(wm) }}{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
{# One note in a list (home, tag pages). v2.3.0 #}
|
||||
{% from "partials/media.html" import display_media %}
|
||||
{% from "partials/reaction.html" import reaction %}
|
||||
{% from "partials/reply_context.html" import reply_line %}
|
||||
{% from "partials/responses.html" import list_counts %}
|
||||
{% macro note_preview(note, author) %}
|
||||
<article class="h-entry note-preview">
|
||||
@@ -10,7 +11,7 @@
|
||||
{% endif %}
|
||||
|
||||
{% if note.in_reply_to %}
|
||||
<p class="reaction reply-to"><span class="reaction-verb">↩ In reply to</span> <a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a></p>
|
||||
{{ reply_line(note.in_reply_to, note.context if note.context is defined else None) }}
|
||||
{% endif %}
|
||||
|
||||
{{ reaction(note) }}
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
{# The post a reply answers (v2.19.0).
|
||||
The card is the note's u-in-reply-to h-cite (Post Type Discovery reads it
|
||||
as a reply). Avatars and images are this site's saved copies; until there
|
||||
is one, the initial is shown and the image left out. #}
|
||||
|
||||
{% macro _who(ctx, url) %}{{ ctx.author_name or ctx.site_name or url.split('/')[2] }}{% endmacro %}
|
||||
|
||||
{% macro _when(ctx, fmt) %}
|
||||
{%- set when = ctx.published_at -%}
|
||||
{%- if when and when.strftime is defined -%}
|
||||
<time class="dt-published" datetime="{{ when|iso_utc }}">{{ when.strftime(fmt) }}</time>
|
||||
{%- endif -%}
|
||||
{% endmacro %}
|
||||
|
||||
{# Full card, on the note's page #}
|
||||
{% macro reply_card(url, ctx) %}
|
||||
{% set fetched = ctx and ctx.fetch_status == 'fetched' %}
|
||||
<div class="reply-context">
|
||||
<span class="reply-label">↩ In reply to</span>
|
||||
<blockquote class="u-in-reply-to h-cite reply-card">
|
||||
{% if fetched and (ctx.author_name or ctx.content_snippet or ctx.title) %}
|
||||
<header class="reply-author">
|
||||
{% set avatar = ctx.author_photo|local_avatar %}
|
||||
<span class="p-author h-card">
|
||||
{%- if avatar %}<img class="u-photo" src="{{ avatar }}" alt="" width="40" height="40" loading="lazy">
|
||||
{%- else %}<span class="face-initial" style="background-color: {{ (ctx.author_url or url)|url_to_color }}" aria-hidden="true">{{ _who(ctx, url)[0]|upper }}</span>{% endif %}
|
||||
<span class="reply-who">
|
||||
{%- if ctx.author_url %}<a class="p-name u-url" href="{{ ctx.author_url }}">{{ _who(ctx, url) }}</a>
|
||||
{%- else %}<span class="p-name">{{ _who(ctx, url) }}</span>{% endif %}
|
||||
{%- if ctx.author_handle %} <span class="reply-handle">{{ ctx.author_handle }}</span>{% endif -%}
|
||||
</span>
|
||||
</span>
|
||||
</header>
|
||||
{% if ctx.title %}<p class="p-name reply-title">{{ ctx.title }}</p>{% endif %}
|
||||
{% if ctx.content_snippet %}<div class="e-content reply-content">{{ ctx.content_snippet|safe }}</div>{% endif %}
|
||||
{% set image = ctx.photo|local_preview %}
|
||||
{% set video = ctx.video|local_video if ctx.video is defined else None %}
|
||||
{% if video %}
|
||||
{# Mastodon "GIFs" are silent looping MP4s: shown as Mastodon shows them (v2.20.1) #}
|
||||
<video class="reply-photo" src="{{ video }}"{% if image %} poster="{{ image }}"{% endif %} autoplay loop muted playsinline{% if ctx.photo_alt %} aria-label="{{ ctx.photo_alt }}" title="{{ ctx.photo_alt }}"{% endif %}></video>
|
||||
{% elif image %}<img class="reply-photo" src="{{ image }}" alt="{{ ctx.photo_alt or '' }}" loading="lazy">{% endif %}
|
||||
<footer class="reply-source">
|
||||
<a class="u-url" href="{{ url }}">{{ _when(ctx, '%B %-d, %Y · %-I:%M %p') }}{% if ctx.published_at and ctx.published_at.strftime is defined %} · {% endif %}{{ ctx.site_name or url.split('/')[2] }} ↗</a>
|
||||
</footer>
|
||||
{% else %}
|
||||
<cite><a class="u-url" href="{{ url }}">{{ url }}</a></cite>
|
||||
{% endif %}
|
||||
</blockquote>
|
||||
</div>
|
||||
{% endmacro %}
|
||||
|
||||
{# One line and a short quote, in lists of notes #}
|
||||
{% macro reply_line(url, ctx) %}
|
||||
{% set fetched = ctx and ctx.fetch_status == 'fetched' %}
|
||||
<div class="reaction reply-to">
|
||||
<span class="reaction-verb">↩ In reply to</span>
|
||||
{% if fetched and (ctx.author_name or ctx.title) %}
|
||||
<a class="u-in-reply-to" href="{{ url }}">{{ ctx.title or ctx.author_name }}</a>
|
||||
{%- if ctx.title and ctx.author_name %} <span class="reaction-by">by {{ ctx.author_name }}</span>{% endif %}
|
||||
{% set quote = ctx.content_snippet|striptags|replace('\n', ' ')|trim if ctx.content_snippet else '' %}
|
||||
{% if quote and not ctx.title %}<q class="reply-quote">{{ quote|truncate(140) }}</q>{% endif %}
|
||||
{% else %}
|
||||
<a class="u-in-reply-to" href="{{ url }}">{{ url }}</a>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endmacro %}
|
||||
@@ -30,7 +30,8 @@
|
||||
</a></span>
|
||||
{%- endmacro %}
|
||||
|
||||
{# A reply, shown as a comment #}
|
||||
{# A reply, shown as a comment; replies to it are nested inside, as
|
||||
comments of the comment (v2.20.2) #}
|
||||
{% macro reply(wm) %}
|
||||
<article class="webmention webmention-reply u-comment h-cite">
|
||||
<div class="webmention-author">{{ author_card(wm) }}</div>
|
||||
@@ -38,6 +39,11 @@
|
||||
<div class="e-content">{{ wm.content_snippet|safe }}</div>
|
||||
{% endif %}
|
||||
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a> {{ via(wm) }}
|
||||
{% if wm.children %}
|
||||
<div class="webmention-thread">
|
||||
{% for child in wm.children %}{{ reply(child) }}{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
</article>
|
||||
{%- endmacro %}
|
||||
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
"""Admin Log and Settings pages (v2.18.0)"""
|
||||
|
||||
import pytest
|
||||
|
||||
from starpunk import backfeed, create_app, error_log
|
||||
from starpunk.auth import create_session
|
||||
from starpunk.database import get_db
|
||||
from starpunk.notes import create_note
|
||||
|
||||
SITE = "https://blog.example.com"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app(tmp_path):
|
||||
app = create_app({
|
||||
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com",
|
||||
"DATABASE_PATH": tmp_path / "starpunk.db", "DATA_PATH": tmp_path,
|
||||
"NOTES_PATH": tmp_path / "notes", "SESSION_SECRET": "test", "CSRF_ENABLED": False,
|
||||
"BACKGROUND_WORKERS_ENABLED": False, "WEBMENTION_ENABLED": True,
|
||||
})
|
||||
backfeed.denied.clear()
|
||||
yield app
|
||||
backfeed.denied.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def browser(app):
|
||||
with app.test_request_context():
|
||||
token = create_session("https://example.com")
|
||||
client = app.test_client()
|
||||
client.set_cookie("starpunk_session", token)
|
||||
return client
|
||||
|
||||
|
||||
def _failures(app):
|
||||
"""One of each kind of error, on two notes"""
|
||||
with app.app_context():
|
||||
broken = create_note(content="A crosspost that broke", published=True)
|
||||
reply = create_note(content="Replying", published=True)
|
||||
db = get_db(app)
|
||||
db.execute("UPDATE notes SET in_reply_to = 'https://gone.example/post' WHERE id = ?", (reply.id,))
|
||||
db.execute("INSERT INTO syndications (note_id, target, state, last_error, updated_at) "
|
||||
"VALUES (?, 'bluesky', 'failed', 'HTTP 400: post too long', '2026-11-12 10:00:00')",
|
||||
(broken.id,))
|
||||
db.execute("INSERT INTO syndications (note_id, target, state, remote_url, backfeed_error, "
|
||||
"backfeed_checked_at) VALUES (?, 'gotosocial', 'synced', 'https://social.example/1', "
|
||||
"'HTTP 404 from GoToSocial', '2026-11-12 11:00:00')", (broken.id,))
|
||||
db.execute("INSERT INTO webmention_queue (source_url, target_url, source_note_id, status, last_error) "
|
||||
"VALUES (?, 'https://other.example/a', ?, 'failed', 'Endpoint returned 500')",
|
||||
(f"{SITE}/note/{reply.slug}", reply.id))
|
||||
db.execute("INSERT INTO reply_contexts (url, fetch_status, fetch_error) "
|
||||
"VALUES ('https://gone.example/post', 'error', 'HTTP 410 Gone')")
|
||||
db.execute("INSERT INTO syndications (note_id, target, state, last_error) "
|
||||
"VALUES (?, 'threads', 'synced', 'old error, since fixed')", (reply.id,))
|
||||
db.commit()
|
||||
return broken, reply
|
||||
|
||||
|
||||
def test_log_lists_each_error_with_its_post(app, browser):
|
||||
broken, reply = _failures(app)
|
||||
backfeed.denied["threads"] = "Needs a Threads token with threads_manage_replies"
|
||||
|
||||
page = browser.get("/admin/log").get_data(as_text=True)
|
||||
for text in ("Crosspost to Bluesky", "HTTP 400: post too long", "Retry",
|
||||
"Response check on GoToSocial", "HTTP 404 from GoToSocial", "Check now",
|
||||
"Webmention sent", "https://other.example/a", "Endpoint returned 500",
|
||||
"Reply context fetch", "HTTP 410 Gone",
|
||||
"Response checks on Threads", "Needs a Threads token", "Site-wide"):
|
||||
assert text in page, text
|
||||
assert f"/admin/edit/{broken.id}" in page and f"/admin/edit/{reply.id}" in page
|
||||
assert "old error, since fixed" not in page, "synced crossposts aren't errors"
|
||||
assert 'class="badge badge-error">5<' in page
|
||||
|
||||
with app.app_context():
|
||||
assert error_log.count() == 5
|
||||
areas = [e["area"] for e in error_log.entries()]
|
||||
assert areas[0] == "Response checks on Threads", "site-wide problems first"
|
||||
assert areas.index("Response check on GoToSocial") < areas.index("Crosspost to Bluesky"), "newest first"
|
||||
|
||||
|
||||
def test_log_retry_clears_the_crosspost_error(app, browser):
|
||||
broken, _ = _failures(app)
|
||||
with app.app_context():
|
||||
sid = get_db(app).execute("SELECT id FROM syndications WHERE target = 'bluesky'").fetchone()[0]
|
||||
browser.post(f"/admin/syndication/{sid}/retry")
|
||||
assert "HTTP 400: post too long" not in browser.get("/admin/log").get_data(as_text=True)
|
||||
|
||||
|
||||
def test_empty_log_and_no_badge(browser):
|
||||
page = browser.get("/admin/log").get_data(as_text=True)
|
||||
assert "No errors." in page and "badge-error" not in page
|
||||
|
||||
|
||||
def test_notification_error_is_logged(app, browser):
|
||||
with app.app_context():
|
||||
get_db(app).execute("INSERT INTO site_state (key, value) VALUES "
|
||||
"('notify_last_error', '2026-11-12 09:00:00 UTC gotify: 401 Unauthorized')")
|
||||
get_db(app).commit()
|
||||
page = browser.get("/admin/log").get_data(as_text=True)
|
||||
assert "Notifications" in page and "gotify: 401 Unauthorized" in page and "2026-11-12 09:00:00" in page
|
||||
|
||||
|
||||
def test_webmentions_page_is_decluttered(browser):
|
||||
page = browser.get("/admin/webmentions").get_data(as_text=True)
|
||||
assert "Back to Dashboard" not in page
|
||||
assert "Send test notification" not in page and "<h3>Notifications</h3>" not in page
|
||||
|
||||
|
||||
def test_nav_and_settings_page(browser):
|
||||
page = browser.get("/admin/settings").get_data(as_text=True)
|
||||
assert "<h3>Notifications</h3>" in page and "NOTIFY_URLS" in page
|
||||
for link in ("/admin/webmentions", "/admin/log", "/admin/settings", "/admin/tokens"):
|
||||
assert f'href="{link}"' in page, link
|
||||
|
||||
|
||||
def test_pages_require_login(app):
|
||||
client = app.test_client()
|
||||
for path in ("/admin/log", "/admin/settings"):
|
||||
assert client.get(path).status_code == 302
|
||||
+109
-12
@@ -392,18 +392,25 @@ def mastodon_api(request):
|
||||
"account": who or account("r" + n)}
|
||||
return httpx.Response(200, json={"ancestors": [], "descendants": [
|
||||
status("10"), status("11", vis="unlisted"), status("12", parent="10"),
|
||||
status("13", who=account("me"))]})
|
||||
status("13", who=account("me")), status("14", parent="12", who=account("me")),
|
||||
status("15", parent="13")]})
|
||||
return httpx.Response(404)
|
||||
|
||||
|
||||
def test_mastodon_client_reads_likes_boosts_and_public_direct_replies():
|
||||
def test_mastodon_client_reads_likes_boosts_and_public_reply_threads():
|
||||
gts = MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(mastodon_api)))
|
||||
harvest = gts.responses("01COPY", COPY)
|
||||
assert harvest.complete
|
||||
got = [(r.kind, r.source_url) for r in harvest.responses]
|
||||
# 11 isn't public; 13 is our own thread continuing, so 15 under it is left out too.
|
||||
# 12 answers 10, and our 14 answers 12 (v2.20.2)
|
||||
assert got == [("like", f"{COPY}#liked-by-1"), ("like", f"{COPY}#liked-by-3"),
|
||||
("repost", f"{COPY}#reposted-by-2"), ("reply", f"{GTS}/@u/statuses/10")]
|
||||
assert harvest.responses[-1].content_html == "<p>reply 10</p>"
|
||||
("repost", f"{COPY}#reposted-by-2"), ("reply", f"{GTS}/@u/statuses/10"),
|
||||
("reply", f"{GTS}/@u/statuses/12"), ("reply", f"{GTS}/@u/statuses/14")]
|
||||
replies = {r.remote_id: r for r in harvest.responses if r.kind == "reply"}
|
||||
assert replies["10"].content_html == "<p>reply 10</p>"
|
||||
assert [(replies[i].parent_id, replies[i].own) for i in ("10", "12", "14")] == \
|
||||
[(None, False), ("10", False), ("12", True)]
|
||||
assert gts.responses("01COPY", COPY, cap=1).complete is False
|
||||
|
||||
|
||||
@@ -457,14 +464,22 @@ def test_threads_client_reads_replies_and_counts():
|
||||
params = parse_qs(request.url.query.decode())
|
||||
if path.endswith("/me"):
|
||||
return httpx.Response(200, json={"id": "42", "username": "me"})
|
||||
if path.endswith("/111/replies"):
|
||||
assert "profile_picture_url" in params["fields"][0]
|
||||
if path.endswith("/111/conversation"):
|
||||
# The whole conversation, newest first, as Meta returns it (v2.20.2)
|
||||
assert "profile_picture_url" in params["fields"][0] and "replied_to" in params["fields"][0]
|
||||
to = lambda i: {"replied_to": {"id": i}}
|
||||
return httpx.Response(200, json={"data": [
|
||||
{"id": "6", "text": "Ha", "username": "ann", "permalink": "https://www.threads.com/@ann/post/C",
|
||||
"timestamp": "2026-11-10T02:00:00+0000", "hide_status": "NOT_HUSHED", **to("5")},
|
||||
{"id": "5", "text": "Yes", "username": "me", "permalink": "https://www.threads.com/@me/post/B",
|
||||
"timestamp": "2026-11-10T01:00:00+0000", **to("1")},
|
||||
{"id": "7", "text": "under mine", "username": "bo", "permalink": "p", "hide_status": "NOT_HUSHED",
|
||||
**to("4")},
|
||||
{"id": "1", "text": "Hi", "username": "ann", "permalink": "https://www.threads.com/@ann/post/A",
|
||||
"timestamp": "2026-11-10T00:00:00+0000", "hide_status": "NOT_HUSHED"},
|
||||
{"id": "2", "text": "private"},
|
||||
{"id": "3", "text": "hidden", "username": "x", "permalink": "p", "hide_status": "HIDDEN"},
|
||||
{"id": "4", "text": "mine", "username": "me", "permalink": "p"}]})
|
||||
"timestamp": "2026-11-10T00:00:00+0000", "hide_status": "NOT_HUSHED", **to("111")},
|
||||
{"id": "2", "text": "private", **to("111")},
|
||||
{"id": "3", "text": "hidden", "username": "x", "permalink": "p", "hide_status": "HIDDEN", **to("111")},
|
||||
{"id": "4", "text": "mine", "username": "me", "permalink": "p", **to("111")}]})
|
||||
if path.endswith("/111/insights"):
|
||||
assert params["metric"] == ["likes,reposts,quotes"]
|
||||
return httpx.Response(200, json={"data": [
|
||||
@@ -474,7 +489,10 @@ def test_threads_client_reads_replies_and_counts():
|
||||
|
||||
threads = ThreadsClient("tok", http=httpx.Client(transport=httpx.MockTransport(api)))
|
||||
harvest = threads.responses("111", "https://www.threads.com/@me/post/X")
|
||||
assert [(r.kind, r.author_name, r.content_html) for r in harvest.responses] == [("reply", "ann", "<p>Hi</p>")]
|
||||
# Our own reply to our post continues the thread (4, and 7 under it, are left out);
|
||||
# our reply to ann (5) and her answer (6) are kept, oldest first
|
||||
assert [(r.author_name, r.content_html, r.parent_id, r.own) for r in harvest.responses] == [
|
||||
("ann", "<p>Hi</p>", None, False), ("me", "<p>Yes</p>", "1", True), ("ann", "<p>Ha</p>", "5", False)]
|
||||
assert harvest.counts == {"likes": 12, "reposts": 3, "quotes": 0}
|
||||
|
||||
|
||||
@@ -543,7 +561,8 @@ def test_admin_backfill_and_check_now(app, client):
|
||||
assert s["backfeed_next_at"]
|
||||
|
||||
backfeed.denied["gotosocial"] = "Needs a GoToSocial token with read:statuses"
|
||||
assert "Needs a GoToSocial token" in browser.get("/admin/webmentions?origin=gotosocial").get_data(as_text=True)
|
||||
assert "Needs a GoToSocial token" in browser.get("/admin/log").get_data(as_text=True)
|
||||
assert "Response checks have errors" in browser.get("/admin/webmentions").get_data(as_text=True)
|
||||
|
||||
|
||||
def test_threads_avatar_kept_while_its_copy_exists(app, client, web):
|
||||
@@ -657,3 +676,81 @@ def test_quote_counts_include_anonymous_ones(app, client):
|
||||
assert "Quoter" in page, "the named quote is listed under Responses"
|
||||
home = app.test_client().get("/").get_data(as_text=True)
|
||||
assert 'title="3 quotes"' in home
|
||||
|
||||
|
||||
# --- Reply threads (v2.20.2) ------------------------------------------------------
|
||||
|
||||
def test_thread_members_follow_replies_to_replies():
|
||||
from starpunk.backfeed import thread_members
|
||||
posts = [ # newest first, as Threads lists them
|
||||
{"id": "c", "parent": "b", "own": False},
|
||||
{"id": "b", "parent": "a", "own": True},
|
||||
{"id": "x", "parent": "self", "own": False},
|
||||
{"id": "self", "parent": "COPY", "own": True},
|
||||
{"id": "a", "parent": "COPY", "own": False},
|
||||
{"id": "loop", "parent": "loop", "own": False},
|
||||
]
|
||||
assert thread_members(posts, "COPY") == {"a": None, "b": "a", "c": "b"}
|
||||
|
||||
|
||||
def test_bluesky_client_reads_nested_replies():
|
||||
me, post = "did:plc:me", "at://did:plc:me/app.bsky.feed.post/abc"
|
||||
|
||||
def view(did, rkey, text, at):
|
||||
return {"uri": f"at://{did}/app.bsky.feed.post/{rkey}", "indexedAt": at,
|
||||
"author": {"did": did, "handle": did.rsplit(":", 1)[1] + ".example"},
|
||||
"record": {"text": text, "createdAt": at}}
|
||||
|
||||
def api(request):
|
||||
nsid = request.url.path.removeprefix("/xrpc/")
|
||||
if nsid == "com.atproto.server.createSession":
|
||||
return httpx.Response(200, json={"did": me, "handle": "me.example", "accessJwt": "a",
|
||||
"refreshJwt": "r"})
|
||||
if nsid in ("app.bsky.feed.getLikes", "app.bsky.feed.getRepostedBy", "app.bsky.feed.getQuotes"):
|
||||
return httpx.Response(200, json={"likes": [], "repostedBy": [], "posts": []})
|
||||
if nsid == "app.bsky.feed.getPostThread":
|
||||
assert int(dict(request.url.params)["depth"]) > 1
|
||||
return httpx.Response(200, json={"thread": {"post": {}, "replies": [
|
||||
{"post": view("did:plc:eve", "r1", "First", "2026-11-10T01:00:00Z"), "replies": [
|
||||
{"post": view(me, "r2", "Thanks", "2026-11-10T02:00:00Z"), "replies": [
|
||||
{"post": view("did:plc:eve", "r3", "Anytime", "2026-11-10T03:00:00Z")}]}]},
|
||||
{"post": view(me, "self", "More from me", "2026-11-10T00:30:00Z")}]}})
|
||||
if nsid == "app.bsky.actor.getProfiles":
|
||||
return httpx.Response(200, json={"profiles": []})
|
||||
return httpx.Response(400, json={"error": "Unknown"})
|
||||
|
||||
bsky = BlueskyClient("https://bsky.example", "me.example", "pw",
|
||||
http=httpx.Client(transport=httpx.MockTransport(api)))
|
||||
replies = [(r.remote_id.rsplit("/", 1)[1], r.parent_id and r.parent_id.rsplit("/", 1)[1], r.own)
|
||||
for r in bsky.responses(post, "https://bsky.app/profile/me.example/post/abc").responses]
|
||||
assert replies == [("r1", None, False), ("r2", "r1", True), ("r3", "r2", False)]
|
||||
|
||||
|
||||
def test_reply_threads_are_stored_and_shown_nested(app, client):
|
||||
app.config["WEBMENTION_ENABLED"] = True
|
||||
note, _ = crossposted(app)
|
||||
first, mine, answer = reply(1, "It keeps coming up"), reply(2, "That is terrifying"), reply(3, "To be fair")
|
||||
mine.parent_id, mine.own, mine.author_name = "1", True, "philskents"
|
||||
answer.parent_id = "2"
|
||||
client.harvest = Harvest([first, mine, answer])
|
||||
run(app)
|
||||
|
||||
stored = {r["remote_id"]: r for r in rows(app, note.id)}
|
||||
assert [stored[i]["parent_remote_id"] for i in "123"] == [None, "1", "2"]
|
||||
assert stored["2"]["notified_at"], "our own reply is no news to us"
|
||||
assert not stored["1"]["notified_at"]
|
||||
|
||||
page = app.test_client().get(f"/note/{note.slug}").data.decode()
|
||||
assert "3 Replies" in page
|
||||
# Each answer sits inside the reply it answers
|
||||
outer = page.index("It keeps coming up")
|
||||
assert outer < page.index('class="webmention-thread"') < page.index("That is terrifying") \
|
||||
< page.index("To be fair")
|
||||
assert page.count('class="webmention-thread"') == 2
|
||||
|
||||
import mf2py
|
||||
items = mf2py.parse(doc=page, url=f"{SITE}/note/{note.slug}")["items"]
|
||||
entry = next(i for i in items if "h-entry" in i["type"])
|
||||
(comment,) = entry["properties"]["comment"]
|
||||
(nested,) = comment["properties"]["comment"]
|
||||
assert "terrifying" in json.dumps(nested) and "To be fair" in json.dumps(nested["properties"]["comment"])
|
||||
@@ -0,0 +1,9 @@
|
||||
|
||||
|
||||
def test_tables_get_a_scroll_wrapper():
|
||||
"""v2.18.2: tables are wrapped so they can fill the column and scroll sideways"""
|
||||
from starpunk.models import render_markdown
|
||||
|
||||
html = render_markdown("| Color | Why |\n|---|---|\n| Black | Removal |\n\nAfter")
|
||||
assert html.startswith('<div class="table-wrap"><table>') and "</table></div>" in html
|
||||
assert "<p>After</p>" in html
|
||||
@@ -209,10 +209,10 @@ def test_admin_status_and_test_button(app, sent):
|
||||
token = create_session("https://example.com")
|
||||
browser = app.test_client()
|
||||
browser.set_cookie("starpunk_session", token)
|
||||
page = browser.get("/admin/webmentions").get_data(as_text=True)
|
||||
page = browser.get("/admin/settings").get_data(as_text=True)
|
||||
assert "Sending like, mention, problems, reply to gotifys, ntfys at high priority" in page
|
||||
page = browser.post("/admin/notifications/test", follow_redirects=True).get_data(as_text=True)
|
||||
assert "Test notification sent" in page and sent[-1]["title"] == "StarPunk test notification"
|
||||
assert "Test notification sent" in page and "<h2>Settings</h2>" in page and sent[-1]["title"] == "StarPunk test notification"
|
||||
|
||||
|
||||
# --- Real Apprise -------------------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
"""
|
||||
Reply contexts from fediverse, Bluesky and OpenGraph sources (v2.19.0)
|
||||
|
||||
Mastodon and GoToSocial pages have no h-entry, so a reply to one used to show
|
||||
only the server's domain. The context now comes from the Mastodon API, this
|
||||
site's GoToSocial instance, the Bluesky AppView API or OpenGraph tags, and the
|
||||
note page shows it as a card; lists show the author and a short quote.
|
||||
"""
|
||||
|
||||
import io
|
||||
import json
|
||||
import sqlite3
|
||||
|
||||
import httpx
|
||||
import mf2py
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from starpunk import avatars, syndication
|
||||
from starpunk.reply_context import fetch_reply_context, refresh_stale
|
||||
from starpunk.syndication import MastodonClient
|
||||
|
||||
PUBLIC = [(2, 1, 6, "", ("93.184.216.34", 0))]
|
||||
|
||||
TOOT = "https://masto.example/@ada/117611186265622833"
|
||||
GTS_POST = "https://gts.example/@bob/statuses/01M7BE51ABF9HP2G0CPH5DKKGG"
|
||||
SKEET = "https://bsky.app/profile/carol.example/post/3l6oveex3ii2l"
|
||||
ARTICLE = "https://news.example/2026/11/13/ground-stop"
|
||||
|
||||
# Mastodon's status page: an app shell with OpenGraph tags, no microformats
|
||||
TOOT_PAGE = """<html><head>
|
||||
<meta property="og:site_name" content="Masto Example">
|
||||
<meta property="og:title" content="Ada (@ada@masto.example)">
|
||||
<meta property="profile:username" content="ada@masto.example">
|
||||
</head><body><div id="mastodon"></div></body></html>"""
|
||||
|
||||
STATUS = {
|
||||
"id": "117611186265622833",
|
||||
"created_at": "2026-11-13T21:11:22.624Z",
|
||||
"spoiler_text": "",
|
||||
"sensitive": False,
|
||||
"content": '<p>All aircraft are <a href="https://x.example/" class="mention">ground</a> stopped.</p>'
|
||||
'<script>alert(1)</script>',
|
||||
"account": {"username": "ada", "acct": "ada", "display_name": "Ada Lovelace",
|
||||
"url": "https://masto.example/@ada",
|
||||
"avatar_static": "https://masto.example/avatars/ada.png"},
|
||||
"media_attachments": [{"type": "image", "preview_url": "https://masto.example/media/small.png",
|
||||
"description": "An advisory"}],
|
||||
}
|
||||
|
||||
GTS_PAGE = """<html><head>
|
||||
<meta property="og:title" content="Bob (@bob@gts.example)">
|
||||
<meta property="og:profile:username" content="bob@gts.example">
|
||||
</head><body><main><article class="status expanded">
|
||||
<a class="p-author h-card" href="https://gts.example/@bob"><span class="p-name">Bob</span></a>
|
||||
<div class="content e-content"><p>Posted on GoToSocial</p></div>
|
||||
</article></main></body></html>"""
|
||||
|
||||
ARTICLE_PAGE = """<html><head><title>Ignored</title>
|
||||
<meta property="og:site_name" content="News Example">
|
||||
<meta property="og:title" content="The FAA stops all departures">
|
||||
<meta property="og:description" content="Flights across the US were held on the ground on Friday.">
|
||||
<meta property="og:image" content="/img/lead.png">
|
||||
<meta property="og:image:alt" content="A departures board">
|
||||
<meta property="article:published_time" content="2026-11-13T22:00:00Z">
|
||||
<meta name="author" content="Dee Reporter">
|
||||
</head><body><p>Story</p></body></html>"""
|
||||
|
||||
THREAD = {"thread": {"post": {
|
||||
"author": {"handle": "carol.example", "displayName": "Carol",
|
||||
"avatar": "https://cdn.example/avatar/carol.png"},
|
||||
"record": {"text": "Hello <world>\n\nSecond paragraph", "createdAt": "2026-11-12T08:00:00.000Z"},
|
||||
"embed": {"$type": "app.bsky.embed.images#view",
|
||||
"images": [{"thumb": "https://cdn.example/thumb/1.png", "alt": "A cat"}]},
|
||||
}}}
|
||||
|
||||
|
||||
def png(color="red", size=(800, 400)):
|
||||
buffer = io.BytesIO()
|
||||
Image.new("RGB", size, color).save(buffer, "PNG")
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def web(monkeypatch):
|
||||
"""The mocked internet; tests change `routes` and read `seen`"""
|
||||
seen = []
|
||||
routes = {
|
||||
TOOT: lambda r: httpx.Response(200, html=TOOT_PAGE),
|
||||
"https://masto.example/api/v1/statuses/117611186265622833": lambda r: httpx.Response(200, json=STATUS),
|
||||
GTS_POST: lambda r: httpx.Response(200, html=GTS_PAGE),
|
||||
ARTICLE: lambda r: httpx.Response(200, html=ARTICLE_PAGE),
|
||||
}
|
||||
|
||||
def handler(request):
|
||||
# The SSRF-safe fetcher connects to the checked IP; the name is in Host
|
||||
url = f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
|
||||
seen.append(url)
|
||||
if url.startswith("https://public.api.bsky.app/"):
|
||||
return httpx.Response(200, json=THREAD)
|
||||
if url.endswith(".png"):
|
||||
return httpx.Response(200, content=png(), headers={"Content-Type": "image/png"})
|
||||
route = routes.get(url)
|
||||
return route(request) if route else httpx.Response(404)
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
|
||||
return type("Web", (), {"routes": routes, "seen": seen})
|
||||
|
||||
|
||||
def fetch(app, url, **kwargs):
|
||||
with app.app_context():
|
||||
return fetch_reply_context(app.config["DATABASE_PATH"], url, app.config["SITE_URL"], **kwargs)
|
||||
|
||||
|
||||
def test_mastodon_post_from_its_public_api(app, web):
|
||||
context = fetch(app, TOOT)
|
||||
|
||||
assert context["source"] == "fediverse"
|
||||
assert context["author_name"] == "Ada Lovelace"
|
||||
assert context["author_handle"] == "@ada@masto.example"
|
||||
assert context["author_url"] == "https://masto.example/@ada"
|
||||
assert "ground</a> stopped" in context["content_snippet"]
|
||||
assert "script" not in context["content_snippet"]
|
||||
assert context["photo"] == "https://masto.example/media/small.png"
|
||||
assert context["photo_alt"] == "An advisory"
|
||||
assert context["published_at"].startswith("2026-11-13T21:11:22")
|
||||
# Avatar and image are saved locally (the CSP allows only this site's images)
|
||||
data = app.config["DATA_PATH"]
|
||||
assert avatars.cached_name(data, context["author_photo"])
|
||||
preview = avatars.preview_folder(data) / avatars.cached_preview(data, context["photo"])
|
||||
with Image.open(preview) as image:
|
||||
assert image.width == avatars.PREVIEW_WIDTH
|
||||
|
||||
# Stored with its source, so it isn't fetched again
|
||||
web.seen.clear()
|
||||
assert fetch(app, TOOT)["author_name"] == "Ada Lovelace"
|
||||
assert web.seen == []
|
||||
|
||||
|
||||
def test_content_warning_is_shown_instead_of_the_post(app, web):
|
||||
web.routes["https://masto.example/api/v1/statuses/117611186265622833"] = lambda r: httpx.Response(
|
||||
200, json={**STATUS, "spoiler_text": "flight news", "sensitive": True})
|
||||
context = fetch(app, TOOT)
|
||||
assert context["content_snippet"] == "<p>CW: flight news</p>"
|
||||
assert context["photo"] is None
|
||||
|
||||
|
||||
def test_fediverse_post_through_own_instance(app, web):
|
||||
"""GoToSocial has no anonymous API; this site's instance resolves the post"""
|
||||
asked = []
|
||||
|
||||
def gts(request):
|
||||
asked.append(request.url.params.get("q"))
|
||||
status = {**STATUS, "content": "<p>Posted on GoToSocial</p>", "media_attachments": [],
|
||||
"account": {**STATUS["account"], "acct": "bob@gts.example", "display_name": "Bob",
|
||||
"url": "https://gts.example/@bob"}}
|
||||
return httpx.Response(200, json={"statuses": [status]})
|
||||
|
||||
own = "https://own.example"
|
||||
app.config.update(GOTOSOCIAL_URL=own, GOTOSOCIAL_ACCESS_TOKEN="t")
|
||||
syndication._client_cache[own] = MastodonClient(
|
||||
own, "t", http=httpx.Client(transport=httpx.MockTransport(gts)))
|
||||
try:
|
||||
with app.test_request_context():
|
||||
context = fetch_reply_context(app.config["DATABASE_PATH"], GTS_POST, app.config["SITE_URL"])
|
||||
finally:
|
||||
syndication._client_cache.clear()
|
||||
|
||||
assert asked == [GTS_POST]
|
||||
assert context["source"] == "fediverse"
|
||||
assert context["author_name"] == "Bob"
|
||||
assert context["author_handle"] == "@bob@gts.example"
|
||||
assert "Posted on GoToSocial" in context["content_snippet"]
|
||||
|
||||
|
||||
def test_fediverse_post_without_an_api_uses_opengraph(app, web):
|
||||
context = fetch(app, GTS_POST)
|
||||
assert context["source"] == "opengraph"
|
||||
assert context["author_name"] == "Bob"
|
||||
assert context["author_handle"] == "@bob@gts.example"
|
||||
assert context["author_url"] == "https://gts.example/@bob"
|
||||
assert context["title"] is None
|
||||
|
||||
|
||||
def test_bluesky_post_from_the_public_api(app, web):
|
||||
context = fetch(app, SKEET)
|
||||
assert context["source"] == "bluesky"
|
||||
assert context["author_name"] == "Carol"
|
||||
assert context["author_handle"] == "@carol.example"
|
||||
assert context["author_url"] == "https://bsky.app/profile/carol.example"
|
||||
assert context["content_snippet"] == "<p>Hello <world></p><p>Second paragraph</p>"
|
||||
assert (context["photo"], context["photo_alt"]) == ("https://cdn.example/thumb/1.png", "A cat")
|
||||
assert any("uri=at%3A%2F%2Fcarol.example%2Fapp.bsky.feed.post%2F3l6oveex3ii2l" in u for u in web.seen)
|
||||
|
||||
|
||||
def test_other_pages_use_opengraph(app, web):
|
||||
context = fetch(app, ARTICLE)
|
||||
assert context["source"] == "opengraph"
|
||||
assert context["title"] == "The FAA stops all departures"
|
||||
assert context["author_name"] == "Dee Reporter"
|
||||
assert context["site_name"] == "News Example"
|
||||
assert "held on the ground" in context["content_snippet"]
|
||||
assert context["photo"] == "https://news.example/img/lead.png"
|
||||
assert context["photo_alt"] == "A departures board"
|
||||
|
||||
|
||||
def _old_row(app, url, **values):
|
||||
"""A context stored before v2.19.0: no source, the domain as author"""
|
||||
conn = sqlite3.connect(app.config["DATABASE_PATH"])
|
||||
conn.execute("DELETE FROM reply_contexts WHERE url = ?", (url,))
|
||||
conn.execute("INSERT INTO reply_contexts (url, author_name, fetch_status) VALUES (?, ?, 'fetched')",
|
||||
(url, values.get("author_name", "masto.example")))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
|
||||
def _reply(app, url, content="Agreed"):
|
||||
from starpunk.notes import create_note
|
||||
with app.test_request_context():
|
||||
return create_note(content, published=True, in_reply_to=url)
|
||||
|
||||
|
||||
def test_contexts_from_before_are_fetched_again_once(app, web):
|
||||
_reply(app, TOOT)
|
||||
_old_row(app, TOOT)
|
||||
|
||||
with app.app_context():
|
||||
assert refresh_stale(app.config["DATABASE_PATH"], app.config["SITE_URL"]) == 1
|
||||
assert refresh_stale(app.config["DATABASE_PATH"], app.config["SITE_URL"]) == 0
|
||||
assert fetch(app, TOOT)["author_name"] == "Ada Lovelace"
|
||||
|
||||
|
||||
def test_failed_refetch_keeps_what_was_there(app, web):
|
||||
_reply(app, TOOT)
|
||||
_old_row(app, TOOT, author_name="Ada (old)")
|
||||
web.routes.clear() # everything 404s now
|
||||
|
||||
with app.app_context():
|
||||
refresh_stale(app.config["DATABASE_PATH"], app.config["SITE_URL"])
|
||||
assert refresh_stale(app.config["DATABASE_PATH"], app.config["SITE_URL"]) == 0
|
||||
context = fetch(app, TOOT)
|
||||
assert (context["fetch_status"], context["author_name"]) == ("fetched", "Ada (old)")
|
||||
|
||||
|
||||
def test_reply_page_and_home_page_show_the_context(app, web):
|
||||
note = _reply(app, TOOT, "This cannot be good")
|
||||
client = app.test_client()
|
||||
|
||||
page = client.get(f"/note/{note.slug}").data.decode()
|
||||
assert "Ada Lovelace" in page and "@ada@masto.example" in page
|
||||
assert "ground</a> stopped" in page
|
||||
assert "/previews/" in page and 'alt="An advisory"' in page
|
||||
assert "/avatars/" in page
|
||||
|
||||
home = client.get("/").data.decode()
|
||||
assert "↩ In reply to" in home
|
||||
assert ">Ada Lovelace</a>" in home
|
||||
assert "All aircraft are ground stopped." in home
|
||||
|
||||
# Still a reply to the toot (Post Type Discovery), on both pages, and the
|
||||
# card's author and content stay inside the h-cite
|
||||
for html in (page, home):
|
||||
items = mf2py.parse(doc=html, url="http://localhost:5000/")["items"]
|
||||
entries = [i for i in items if "h-entry" in i["type"]] or [
|
||||
c for i in items for c in i.get("children", []) if "h-entry" in c["type"]]
|
||||
(entry,) = [e for e in entries if "This cannot be good" in json.dumps(e)]
|
||||
reply_to = entry["properties"]["in-reply-to"][0]
|
||||
assert (reply_to if isinstance(reply_to, str) else reply_to["properties"]["url"][0]) == TOOT
|
||||
assert "Ada Lovelace" not in json.dumps(entry["properties"].get("author", ""))
|
||||
|
||||
|
||||
# --- Moving pictures (v2.20.1) --------------------------------------------------
|
||||
|
||||
MP4 = b"\x00\x00\x00\x18ftypmp42" + b"\x00" * 64
|
||||
|
||||
|
||||
def animated_gif():
|
||||
buffer = io.BytesIO()
|
||||
frames = [Image.new("RGB", (800, 400), color) for color in ("red", "green", "blue")]
|
||||
frames[0].save(buffer, "GIF", save_all=True, append_images=frames[1:], duration=80, loop=0)
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def test_mastodon_gifv_is_kept_as_a_looping_video(app, web, monkeypatch):
|
||||
status = {**STATUS, "media_attachments": [{
|
||||
"type": "gifv", "url": "https://masto.example/media/original.mp4",
|
||||
"preview_url": "https://masto.example/media/small.png", "description": "Engineering report"}]}
|
||||
web.routes["https://masto.example/api/v1/statuses/117611186265622833"] = \
|
||||
lambda r: httpx.Response(200, json=status)
|
||||
web.routes["https://masto.example/media/original.mp4"] = \
|
||||
lambda r: httpx.Response(200, content=MP4, headers={"Content-Type": "video/mp4"})
|
||||
|
||||
note = _reply(app, TOOT, "Same here")
|
||||
context = fetch(app, TOOT)
|
||||
assert context["video"] == "https://masto.example/media/original.mp4"
|
||||
assert avatars.cached_video(app.config["DATA_PATH"], context["video"])
|
||||
|
||||
client = app.test_client()
|
||||
page = client.get(f"/note/{note.slug}").data.decode()
|
||||
assert "<video" in page and "autoplay loop muted playsinline" in page
|
||||
assert 'aria-label="Engineering report"' in page
|
||||
src = page.split('<video class="reply-photo" src="', 1)[1].split('"', 1)[0]
|
||||
response = client.get(src)
|
||||
assert response.status_code == 200 and response.mimetype == "video/mp4"
|
||||
|
||||
|
||||
def test_animated_gif_stays_animated(app, web):
|
||||
web.routes[ARTICLE] = lambda r: httpx.Response(200, html=ARTICLE_PAGE.replace("/img/lead.png", "/img/lead.gif"))
|
||||
web.routes["https://news.example/img/lead.gif"] = \
|
||||
lambda r: httpx.Response(200, content=animated_gif(), headers={"Content-Type": "image/gif"})
|
||||
context = fetch(app, ARTICLE)
|
||||
data = app.config["DATA_PATH"]
|
||||
with Image.open(avatars.preview_folder(data) / avatars.cached_preview(data, context["photo"])) as image:
|
||||
assert image.n_frames == 3
|
||||
assert image.width == avatars.PREVIEW_WIDTH
|
||||
|
||||
|
||||
def test_unused_videos_are_removed(app, web):
|
||||
data = app.config["DATA_PATH"]
|
||||
folder = avatars.preview_folder(data)
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
stray = folder / avatars.video_name_for("https://gone.example/x.mp4")
|
||||
stray.write_bytes(MP4)
|
||||
avatars.remove_unused(app.config["DATABASE_PATH"], data)
|
||||
assert not stray.exists()
|
||||
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
Crossposted replies look like replies (v2.20.0)
|
||||
|
||||
A reply to a post on the same network threads under it natively (with an
|
||||
@mention on the fediverse); a reply to one of our own notes threads under
|
||||
that note's crosspost on every network; anything else starts with
|
||||
"↩ Replying to <author>" and the link.
|
||||
"""
|
||||
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from starpunk import create_app, syndication
|
||||
from starpunk.notes import create_note, get_note
|
||||
from starpunk.syndication import (BlueskyClient, MastodonClient, Post, ThreadsClient,
|
||||
compose_text, get_note_syndications, process_due,
|
||||
select_targets)
|
||||
|
||||
SITE = "https://blog.example.com"
|
||||
GTS = "https://social.example.com"
|
||||
FEDI_POST = "https://mastodon.example/@alice/111"
|
||||
OWN_FEDI_POST = "https://social.example.com/@phil/statuses/01OWN"
|
||||
BSKY_POST = "https://bsky.app/profile/alice.example/post/3abc"
|
||||
WEB_PAGE = "https://someblog.example/2026/a-post"
|
||||
|
||||
|
||||
def note(content="Agreed, this cannot be good"):
|
||||
return SimpleNamespace(content=content, title=None, bookmark_of=None, label=content[:30],
|
||||
summary=None, excerpt=content)
|
||||
|
||||
|
||||
def post(reply_to=None, **kwargs):
|
||||
return Post(note=note(), permalink=f"{SITE}/note/1", tags=[], max_chars=500,
|
||||
reply_to=reply_to, **kwargs)
|
||||
|
||||
|
||||
class FakeGTS:
|
||||
def __init__(self):
|
||||
self.created = []
|
||||
|
||||
def __call__(self, request):
|
||||
path = request.url.path
|
||||
if path == "/api/v1/accounts/verify_credentials":
|
||||
return httpx.Response(200, json={"id": "me"})
|
||||
if path == "/api/v2/search":
|
||||
q = request.url.params["q"]
|
||||
accounts = {FEDI_POST: {"id": "a2", "acct": "alice@mastodon.example"},
|
||||
OWN_FEDI_POST: {"id": "me", "acct": "phil"}}
|
||||
if q in accounts:
|
||||
return httpx.Response(200, json={"statuses": [{"id": "s1", "account": accounts[q]}]})
|
||||
return httpx.Response(200, json={"statuses": []})
|
||||
if path == "/api/v1/statuses" and request.method == "POST":
|
||||
self.created.append(json.loads(request.content))
|
||||
sid = f"p{len(self.created)}"
|
||||
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
|
||||
return httpx.Response(500)
|
||||
|
||||
|
||||
class FakePDS:
|
||||
def __init__(self):
|
||||
self.records = []
|
||||
|
||||
def __call__(self, request):
|
||||
nsid = request.url.path.removeprefix("/xrpc/")
|
||||
if nsid == "com.atproto.server.createSession":
|
||||
return httpx.Response(200, json={"did": "did:plc:me", "handle": "me.example",
|
||||
"accessJwt": "a", "refreshJwt": "r"})
|
||||
if nsid == "com.atproto.identity.resolveHandle":
|
||||
return httpx.Response(200, json={"did": "did:plc:alice"})
|
||||
if nsid == "app.bsky.feed.getPosts":
|
||||
uri = request.url.params["uris"]
|
||||
record = {"text": "parent"}
|
||||
if uri.endswith("/3abc"): # itself a reply in a thread
|
||||
record["reply"] = {"root": {"uri": "at://did:plc:alice/app.bsky.feed.post/root",
|
||||
"cid": "rootcid"}, "parent": {}}
|
||||
return httpx.Response(200, json={"posts": [{"uri": uri, "cid": "cid1", "record": record}]})
|
||||
if nsid == "com.atproto.repo.createRecord":
|
||||
body = json.loads(request.content)
|
||||
self.records.append(body["record"])
|
||||
uri = f"at://did:plc:me/{body['collection']}/r{len(self.records)}"
|
||||
return httpx.Response(200, json={"uri": uri, "cid": "c"})
|
||||
return httpx.Response(500, json={"error": nsid})
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app(tmp_path):
|
||||
data = tmp_path / "data"
|
||||
app = create_app({
|
||||
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com", "SESSION_SECRET": "x",
|
||||
"DATA_PATH": data, "NOTES_PATH": data / "notes", "DATABASE_PATH": data / "starpunk.db",
|
||||
"GOTOSOCIAL_URL": GTS, "GOTOSOCIAL_ACCESS_TOKEN": "t",
|
||||
"BLUESKY_HANDLE": "me.example", "BLUESKY_APP_PASSWORD": "p", "BLUESKY_SERVICE": "https://pds.example",
|
||||
})
|
||||
yield app
|
||||
syndication._client_cache.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gts():
|
||||
fake = FakeGTS()
|
||||
return fake, MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(fake)))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pds():
|
||||
fake = FakePDS()
|
||||
return fake, BlueskyClient("https://pds.example", "me.example", "p",
|
||||
http=httpx.Client(transport=httpx.MockTransport(fake)))
|
||||
|
||||
|
||||
# --- GoToSocial ---------------------------------------------------------------
|
||||
|
||||
def test_fediverse_reply_threads_and_mentions(app, gts):
|
||||
fake, client = gts
|
||||
with app.app_context():
|
||||
client.create(post(FEDI_POST, reply_name="Alice"), key="k")
|
||||
(body,) = fake.created
|
||||
assert body["in_reply_to_id"] == "s1"
|
||||
assert body["status"].startswith("@alice@mastodon.example Agreed")
|
||||
assert "Replying to" not in body["status"]
|
||||
|
||||
|
||||
def test_reply_to_own_fediverse_post_has_no_mention(app, gts):
|
||||
fake, client = gts
|
||||
with app.app_context():
|
||||
client.create(post(OWN_FEDI_POST), key="k")
|
||||
assert fake.created[0]["in_reply_to_id"] == "s1"
|
||||
assert fake.created[0]["status"].startswith("Agreed")
|
||||
|
||||
|
||||
def test_reply_to_a_web_page_says_so_on_the_fediverse(app, gts):
|
||||
fake, client = gts
|
||||
with app.app_context():
|
||||
client.create(post(WEB_PAGE, reply_name="Ada"), key="k1")
|
||||
client.create(post(WEB_PAGE), key="k2")
|
||||
named, unnamed = fake.created
|
||||
assert "in_reply_to_id" not in named
|
||||
assert named["status"].startswith(f"↩ Replying to Ada\n{WEB_PAGE}\n\nAgreed")
|
||||
assert unnamed["status"].startswith(f"↩ Replying to someblog.example\n{WEB_PAGE}")
|
||||
|
||||
|
||||
# --- Bluesky ------------------------------------------------------------------
|
||||
|
||||
def test_bluesky_reply_threads_under_the_post_and_its_root(app, pds):
|
||||
fake, client = pds
|
||||
with app.app_context():
|
||||
client.create(post(BSKY_POST, reply_name="Alice"), key="k")
|
||||
(record,) = fake.records
|
||||
assert record["reply"]["parent"] == {"uri": "at://did:plc:alice/app.bsky.feed.post/3abc", "cid": "cid1"}
|
||||
assert record["reply"]["root"] == {"uri": "at://did:plc:alice/app.bsky.feed.post/root", "cid": "rootcid"}
|
||||
assert record["text"].startswith("Agreed")
|
||||
|
||||
|
||||
def test_reply_to_a_fediverse_post_says_so_on_bluesky(app, pds):
|
||||
fake, client = pds
|
||||
with app.app_context():
|
||||
client.create(post(FEDI_POST, reply_name="Alice"), key="k")
|
||||
(record,) = fake.records
|
||||
assert "reply" not in record
|
||||
assert record["text"].startswith("↩ Replying to Alice\n\nAgreed")
|
||||
links = [f for f in record["facets"] if f["features"][0]["uri"] == FEDI_POST]
|
||||
(link,) = links
|
||||
start = record["text"].encode().index(b"Alice")
|
||||
assert link["index"] == {"byteStart": start, "byteEnd": start + len(b"Alice")}
|
||||
|
||||
|
||||
# --- Threads ------------------------------------------------------------------
|
||||
|
||||
def test_threads_replies_natively_only_to_our_own_posts(app):
|
||||
client = ThreadsClient("tok")
|
||||
calls = []
|
||||
client._call = lambda method, path, **params: calls.append(params) or {"id": "c1"}
|
||||
with app.app_context():
|
||||
client._container("me", post(f"{SITE}/note/0", reply_remote_id="1789"))
|
||||
client._container("me", post(FEDI_POST, reply_name="Alice"))
|
||||
own, other = calls
|
||||
assert own["reply_to_id"] == "1789" and not own["text"].startswith("↩")
|
||||
assert "reply_to_id" not in other
|
||||
assert other["text"].startswith(f"↩ Replying to Alice\n{FEDI_POST}\n\nAgreed")
|
||||
|
||||
|
||||
# --- Composing ----------------------------------------------------------------
|
||||
|
||||
def test_reply_line_survives_shortening():
|
||||
text = compose_text(note("word " * 200), f"{SITE}/note/1", [], 300,
|
||||
lead=f"↩ Replying to Alice\n{FEDI_POST}")
|
||||
assert text.startswith(f"↩ Replying to Alice\n{FEDI_POST}\n\nword")
|
||||
assert len(text) <= 300 and text.endswith(f"{SITE}/note/1")
|
||||
|
||||
|
||||
# --- Replies to our own notes thread under their crossposts ----------------------
|
||||
|
||||
@pytest.mark.parametrize("via", ["permalink", "crosspost"])
|
||||
def test_reply_to_own_note_threads_on_every_network(app, gts, pds, monkeypatch, via):
|
||||
monkeypatch.setattr("starpunk.reply_context.fetch_reply_context", lambda *a, **k: None)
|
||||
(gts_fake, gts_client), (pds_fake, pds_client) = gts, pds
|
||||
syndication._client_cache[GTS] = gts_client
|
||||
syndication._client_cache["bluesky:https://pds.example:me.example"] = pds_client
|
||||
|
||||
def crosspost(**kwargs):
|
||||
with app.test_request_context():
|
||||
n = create_note("Plain words without links", published=True, **kwargs)
|
||||
select_targets(n.id, ["gotosocial", "bluesky"])
|
||||
process_due(limit=10)
|
||||
return get_note(id=n.id).url, {r["target"]: r for r in get_note_syndications(n.id)}
|
||||
|
||||
first, rows = crosspost()
|
||||
target = first if via == "permalink" else rows["gotosocial"]["remote_url"]
|
||||
second, _ = crosspost(in_reply_to=target)
|
||||
|
||||
assert gts_fake.created[-1]["in_reply_to_id"] == rows["gotosocial"]["remote_id"]
|
||||
assert not gts_fake.created[-1]["status"].startswith(("@", "↩"))
|
||||
assert pds_fake.records[-1]["reply"]["parent"]["uri"] == rows["bluesky"]["remote_id"]
|
||||
@@ -0,0 +1,88 @@
|
||||
"""
|
||||
Webmentions are never sent to this site or WEBMENTION_SKIP_HOSTS (v2.18.3)
|
||||
|
||||
From inside the container the site's own public address can time out, and
|
||||
self-hosted apps like Gitea don't take webmentions, so these sends only
|
||||
failed and stayed in the admin Log.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
from starpunk.webmention_sender import (
|
||||
SKIPPED_NOTE, is_skipped, queue_webmentions_for_note, skipped_hosts)
|
||||
from starpunk.webmention_worker import WebmentionWorker
|
||||
|
||||
SKIP = "git.example.com"
|
||||
|
||||
|
||||
def queued(db_path):
|
||||
conn = sqlite3.connect(db_path)
|
||||
rows = conn.execute("SELECT target_url, status, last_error FROM webmention_queue ORDER BY id").fetchall()
|
||||
conn.close()
|
||||
return rows
|
||||
|
||||
|
||||
def test_hosts_include_this_site(app):
|
||||
app.config["WEBMENTION_SKIP_HOSTS"] = [SKIP]
|
||||
hosts = skipped_hosts(app.config)
|
||||
assert hosts == {"localhost", SKIP}
|
||||
assert is_skipped("https://GIT.example.com:443/phil/x", hosts)
|
||||
assert not is_skipped("https://example.com/", hosts) # exact host only
|
||||
|
||||
|
||||
def test_links_to_skipped_hosts_are_not_queued(app):
|
||||
app.config["WEBMENTION_SKIP_HOSTS"] = [SKIP]
|
||||
html = "".join(f'<p><a href="{u}">x</a></p>' for u in [
|
||||
"http://localhost:5000/note/other", f"https://{SKIP}/phil/StarPunk"]
|
||||
+ [f"https://site{i}.example/" for i in range(10)])
|
||||
with app.app_context():
|
||||
n = queue_webmentions_for_note(app.config["DATABASE_PATH"], 1, "http://localhost:5000/note/a",
|
||||
html, in_reply_to="http://localhost:5000/note/b")
|
||||
targets = [t for t, _, _ in queued(app.config["DATABASE_PATH"])]
|
||||
assert n == 10 and targets == [f"https://site{i}.example/" for i in range(10)] # skipped ones don't use the cap
|
||||
|
||||
|
||||
def test_worker_drops_queued_and_failed_sends(app):
|
||||
app.config["WEBMENTION_SKIP_HOSTS"] = [SKIP]
|
||||
db_path = app.config["DATABASE_PATH"]
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.executemany(
|
||||
"INSERT INTO webmention_queue (source_url, target_url, status, last_error) VALUES (?, ?, ?, ?)",
|
||||
[("http://localhost:5000/note/1", f"https://{SKIP}/phil/StarPunk", "failed", "timed out"),
|
||||
("http://localhost:5000/note/1", "http://localhost:5000/", "pending", "timed out"),
|
||||
("http://localhost:5000/note/1", "https://example.org/post", "failed", "timed out"),
|
||||
("http://localhost:5000/note/2", "http://localhost:5000/x", "sent", None)])
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"])._drop_skipped_targets()
|
||||
|
||||
assert [(s, e) for _, s, e in queued(db_path)] == [
|
||||
("no_endpoint", SKIPPED_NOTE), ("no_endpoint", SKIPPED_NOTE),
|
||||
("failed", "timed out"), # other sites are left alone
|
||||
("sent", None)]
|
||||
|
||||
|
||||
def test_worker_skips_without_fetching(app):
|
||||
db_path = app.config["DATABASE_PATH"]
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.execute("INSERT INTO webmention_queue (source_url, target_url, status) VALUES (?, ?, 'pending')",
|
||||
("http://localhost:5000/note/1", "http://localhost:5000/"))
|
||||
conn.commit()
|
||||
(queue_id,) = conn.execute("SELECT id FROM webmention_queue").fetchone()
|
||||
conn.close()
|
||||
|
||||
worker = WebmentionWorker(app, db_path=db_path, site_url=app.config["SITE_URL"])
|
||||
worker._discover_endpoint = lambda url: (_ for _ in ()).throw(AssertionError("fetched"))
|
||||
worker._send_webmention(queue_id, "http://localhost:5000/note/1", "http://localhost:5000/", None, 0)
|
||||
assert queued(db_path)[0][1:] == ("no_endpoint", SKIPPED_NOTE)
|
||||
|
||||
|
||||
def test_settings_page_lists_skipped_hosts(app, client):
|
||||
from starpunk.auth import create_session
|
||||
|
||||
app.config["WEBMENTION_SKIP_HOSTS"] = [SKIP]
|
||||
with app.test_request_context():
|
||||
client.set_cookie("starpunk_session", create_session(app.config["ADMIN_ME"]))
|
||||
html = client.get("/admin/settings").data.decode()
|
||||
assert f"<code>{SKIP}</code>" in html and "<code>localhost</code>" in html
|
||||
Reference in new issue
Block a user