Compare commits

..
5 Commits
Author SHA1 Message Date
Phil acd6bf819d Merge branch 'fix/v2.20.2-reply-threads' for v2.20.2 release (includes v2.20.1)
Build Container / build (push) Successful in 9m49s
2026-11-13 21:09:06 -07:00
PhilandClaude Opus 5.5 bf6e0fdc0b fix: backfeed whole reply threads and nest them (v2.20.2)
Backfeed only read first-level replies (Threads /replies, GoToSocial
direct replies, Bluesky depth 1) and skipped the owner's own replies, so
conversations under crossposts lost everything after the first reply.

- Threads /conversation, all GoToSocial descendants, Bluesky depth 10
- backfeed.thread_members keeps others' replies to our copy and any reply
  (ours included) to a kept reply; own replies are stored as notified
- Migration 026: webmentions.parent_remote_id; replies render nested as
  comments of comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:43:30 -07:00
PhilandClaude Opus 5.5 952ca300c0 fix: keep GIFs moving in reply contexts (v2.20.1)
Mastodon turns GIFs into silent looping MP4s (gifv); only the first-frame
preview was kept. The MP4 is now saved locally and played muted and
looping with the still as poster. Animated GIFs from other sites are saved
as animated WebP. Migration 025 adds reply_contexts.video and has
fediverse contexts with an image fetched again once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:28:20 -07:00
Phil 03a1c8fcea Merge branch 'feat/v2.20.0-reply-crossposts' for v2.20.0 release
Build Container / build (push) Successful in 9m41s
2026-11-13 20:07:56 -07:00
PhilandClaude Opus 5.5 67c0411fd8 feat: crossposted replies look like replies (v2.20.0)
- Native replies on the same network: GoToSocial in_reply_to_id (resolved
  by our instance) with the author's @mention; Bluesky reply root/parent
- Replies to our own notes (permalink or any crosspost URL) thread under
  that note's crosspost on every network, Threads included
- Otherwise "↩ Replying to <author>" and the link; on Bluesky the name is
  the link. compose_text keeps the line whole when shortening

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:32:14 -07:00
20 changed files with 939 additions and 77 deletions

No files matched your search

+32
View File
@@ -7,6 +7,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.20.2] - 2026-11-13
### Fixed
- Replies to replies on crossposts were never captured. Backfeed read only the first level of replies (Threads `/replies`, GoToSocial's direct replies, Bluesky at depth 1). It now reads the whole conversation (Threads `/conversation`, every GoToSocial descendant, Bluesky ten levels deep)
- Your own replies in those conversations were skipped, so the other side of a back-and-forth was missing. They are now kept when they answer someone's reply (a reply to your own crosspost is the thread continuing, and is still left out). They don't send you notifications
### Changed
- Replies are shown nested under the reply they answer, as comments of comments (`u-comment` inside the parent `h-cite`). Migration 026 adds `webmentions.parent_remote_id`. Existing crossposts pick up their threads at their next backfeed check
## [2.20.1] - 2026-11-13
### Fixed
- A reply to a Mastodon post with a GIF showed a still image. Mastodon turns GIFs into short silent looping MP4s ("gifv"), and only their first frame was kept. The MP4 is now saved on this site and plays muted and looping, as on Mastodon, with the still image as its poster
- Animated GIFs (and animated WebP/PNG) from other sites stay animated in reply contexts instead of being cut to their first frame
### Changed
- Migration 025 adds `video` to `reply_contexts`, and has fediverse contexts with an image fetched again once, so existing replies pick up their gifv
## [2.20.0] - 2026-11-13
### Added
- Crossposted replies look like replies. A reply to a post on the same network is posted as a native reply: on GoToSocial it threads under the fediverse post and starts with the author's @mention (so they're notified and their server gets it); on Bluesky it threads under the Bluesky post
- A reply to one of your own posts (its page here, or any of its crossposts) threads under that post's crosspost on every network, Threads included
- A reply that can't thread on a network (a fediverse post crossposted to Bluesky or Threads, a blog post anywhere) starts with "↩ Replying to <author>" and the post's link. On Bluesky the author's name is the link, to save characters
Crossposts made before this release are unchanged: edits can't move a post into a thread.
## [2.19.0] - 2026-11-13
### Added
@@ -0,0 +1,78 @@
# v2.20.0: Crossposted replies look like replies
## Why
Crossposting ignored `in_reply_to`. A reply to
`https://defcon.social/@deviantollam/117611186265622833` went to GoToSocial
and Bluesky as an ordinary post (text, permalink, hashtags), with nothing to
say it answered anyone.
## Behaviour
For each target, in order:
| Replying to | GoToSocial | Bluesky | Threads |
|---|---|---|---|
| One of our notes (its page, or a crosspost of it on any network) | Native reply to that note's GoToSocial status | Native reply to its Bluesky post | Native reply (`reply_to_id`) to its Threads post |
| A fediverse post | Native reply (`in_reply_to_id`, resolved by our instance) starting with `@acct` | Reply line | Reply line |
| A Bluesky post | Reply line | Native reply (`reply.root`/`reply.parent`) | Reply line |
| Anything else | Reply line | Reply line | Reply line |
The reply line is `↩ Replying to <who>`, where *who* is the reply
context's author name, else its title, else the URL's host:
- GoToSocial and Threads: the line, then the URL, then a blank line and the note.
On Threads the URL counts towards Meta's limit of five links, so the
body gets one link fewer.
- Bluesky: the line without the URL. *Who* is a link facet to the post,
which saves up to ~100 of the 300 characters.
`compose_text(..., lead=...)` keeps the line whole and shortens the body
instead when the post is too long.
### The @mention
The user chose to start a native fediverse reply with the author's
mention, as Mastodon and GoToSocial clients do. Without it the author isn't
notified, and their server may never receive the reply, so it wouldn't show
in their thread. There's no mention when replying to our own account (the
search result's account id equals `verify_credentials`), or when the note
already contains the mention.
### Our own posts
`_own_note_id(url)` maps a URL to a note: a synced `syndications.remote_url`
on any target, or `/note/<slug>` and `/article/<slug>` on this site.
`_reply_fields` then takes that note's `remote_id` on the target being
posted to, as `Post.reply_remote_id`. GoToSocial skips the ids of
favourites and boosts (`favourite:…`, `reblog:…`). Bluesky only takes
`app.bsky.feed.post` URIs. Threads only takes numeric media ids. The
Threads API can't look up other people's posts by URL, so only our own
posts thread there.
### Bluesky threads
A reply's `root` must be the top of the thread. The parent is fetched with
`app.bsky.feed.getPosts`. If the parent is itself a reply, its
`record.reply.root` is used; otherwise the parent is the root.
## Not changed
- Crossposts made before this release. Edits can't move a status into a
thread (GoToSocial, Mastodon), and Bluesky and Threads posts can't be edited at
all. Re-threading would mean deleting and reposting, which loses likes and
boosts. The user chose to leave them.
- An edit of a reply on GoToSocial keeps the status where it is, and its
text keeps the mention (`_prepare` resolves the parent again).
- Fediverse authors bridged into Bluesky by Bridgy Fed have a copy of
their post there. There is no reliable way to find its URI from the
original URL, so these get the reply line.
## Tests
`tests/test_reply_crossposts.py`: native fediverse reply with mention, no
mention for our own account, reply line with name and host fallback,
Bluesky root/parent, Bluesky reply line with its link facet, Threads native
reply only to our own posts, the line surviving shortening, and a reply to
our own note threading on GoToSocial and Bluesky. The reply is addressed both
by the note's permalink and by its GoToSocial crosspost URL.
@@ -0,0 +1,27 @@
# v2.20.1: Moving pictures in reply contexts
## Why
The reply https://starpunk.thesatelliteoflove.com/note/20261114032356 answers
an infosec.exchange post with a GIF, but the card showed a still. Mastodon
converts uploaded GIFs to a silent looping MP4 (`media_attachments[].type ==
"gifv"`, here 320×240, 5.3 s, ~140 KB). Its `preview_url` is a PNG of the
first frame, which is all v2.19.0 kept.
## Changes
| Area | Change |
|---|---|
| `reply_context._from_status` | A `gifv` attachment's `url` goes into the new `video` field (the preview stays in `photo`, used as the poster) |
| Migration 025 | `reply_contexts.video`; fediverse rows with a photo get `source = NULL`, so the worker's `refresh_stale` fetches them once more |
| `avatars.fetch_video` | Saves the MP4 as `DATA_PATH/previews/<sha256>.mp4` (8 MB cap, must have an `ftyp` box); `fill_missing`/`remove_unused` cover it |
| `avatars._save` | Animated GIF/WebP/PNG previews are saved as animated WebP (frames narrowed to 640 px, at most 300 frames) instead of their first frame |
| `/previews/<name>` | Also serves `<sha256>.mp4` as `video/mp4` |
| Template | `<video autoplay loop muted playsinline poster=…>` with the alt text as `aria-label`, when there is a saved MP4 |
The MP4 is stored as it is: re-encoding would need ffmpeg in the image, and
gifvs are small. Like the images, it has to be served from this site
(`default-src 'self'` covers media).
Real videos (`type == "video"`) and Bluesky videos still show their
thumbnail: they can be long and have sound.
@@ -0,0 +1,38 @@
# v2.20.2: Backfeed whole reply threads
## Why
https://starpunk.thesatelliteoflove.com/note/20261110191424 showed one reply
(curiouskatej, via Threads). The Threads conversation had three:
```
curiouskatej: "It keeps coming up on music compilations…" (kept)
└ philskents: "That is terrifying..." (missed)
└ curiouskatej: "To be fair it's just as bad as…" (missed)
```
Each client read only the first level of replies: Threads `/{id}/replies`,
GoToSocial descendants with `in_reply_to_id == our id`, and Bluesky
`getPostThread` at depth 1. All of them also dropped the owner's own replies,
which lost one side of the conversation.
## Changes
| Area | Change |
|---|---|
| `backfeed.thread_members(posts, root)` | Decides which replies to keep, and what each answers. Kept: others' replies to our copy, plus any reply (ours too) to a kept reply. Our own replies to our copy are the thread continuing, so they and their replies are left out, as before |
| `Response.parent_id`, `.own` | The silo id of the reply this one answers (None for our copy itself), and whether we wrote it |
| Threads | `/{id}/conversation` (every depth, newest first) with `replied_to` in the fields |
| GoToSocial | All public descendants from `/context` |
| Bluesky | `getPostThread` depth `BLUESKY_THREAD_DEPTH` (10), walked recursively; blocked or deleted posts drop their subtree |
| Migration 026 | `webmentions.parent_remote_id` |
| `backfeed.store` | Stores the parent; own replies are stored as already notified |
| `webmention_queries.reply_threads` | Nests replies under their parent (same origin and remote id). A reply whose parent isn't shown goes to the top level |
| Templates/CSS | `responses.reply` renders children in `.webmention-thread`, indented with a side rule, as `u-comment h-cite` inside the parent's h-cite |
The heading still counts every reply ("3 Replies"), and so does the count
line under the note.
Existing crossposts get their threads at their next scheduled check. The
check schedule slows with the note's age (ADR-073), and notes past the
cutoff aren't checked.
+6
View File
@@ -0,0 +1,6 @@
-- Migration 025: Looping videos in reply contexts (v2.20.1)
-- Mastodon turns GIFs into silent looping MP4s ("gifv"); video holds its URL,
-- shown instead of the still preview. Fediverse contexts with an image are
-- fetched again once (no source) so their gifv is picked up.
ALTER TABLE reply_contexts ADD COLUMN video TEXT;
UPDATE reply_contexts SET source = NULL WHERE source = 'fediverse' AND photo IS NOT NULL;
@@ -0,0 +1,4 @@
-- Migration 026: Threads of backfed replies (v2.20.2)
-- parent_remote_id: the silo id of the response a reply answers; NULL when it
-- answers our crosspost itself. Replies are shown nested under their parent.
ALTER TABLE webmentions ADD COLUMN parent_remote_id TEXT;
+10 -2
View File
@@ -233,6 +233,14 @@ def create_app(config=None):
name = cached_preview(app.config.get('DATA_PATH', 'data'), url)
return url_for('public.preview', name=name) if name else None
# Template filter - local copy of a replied-to post's gifv, or None (v2.20.1)
@app.template_filter('local_video')
def local_video_filter(url):
from flask import url_for
from starpunk.avatars import cached_video
name = cached_video(app.config.get('DATA_PATH', 'data'), url)
return url_for('public.preview', name=name) if name else None
# Template filter - URL to color for facepile (v1.6.0)
@app.template_filter('url_to_color')
def url_to_color_filter(url):
@@ -476,5 +484,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.19.0"
__version_info__ = (2, 19, 0)
__version__ = "2.20.2"
__version_info__ = (2, 20, 2)
+67 -11
View File
@@ -29,6 +29,10 @@ SIZE = 96
MAX_BYTES = 1024 * 1024
PREVIEW_WIDTH = 640
PREVIEW_MAX_BYTES = 8 * 1024 * 1024
MAX_FRAMES = 300
# Mastodon's "GIFs" are silent looping MP4s (gifv); kept as they are (v2.20.1)
VIDEO_MAX_BYTES = 8 * 1024 * 1024
VIDEO_NAME_RE = re.compile(r"^[0-9a-f]{64}\.mp4$")
# Bluesky avatar on its CDN: .../img/<kind>/plain/<did>/<cid>[@jpeg]
BSKY_CDN_RE = re.compile(
r"^https://cdn\.bsky\.app/img/[a-z_]+/plain/(did:(?:plc|web):[A-Za-z0-9._:%-]+)/([a-z2-7]+)(?:@\w+)?$")
@@ -65,6 +69,39 @@ def cached_preview(data_path, url: Optional[str]) -> Optional[str]:
return _cached(preview_folder(data_path), url)
def video_name_for(url: str) -> str:
return hashlib.sha256(url.encode()).hexdigest() + ".mp4"
def cached_video(data_path, url: Optional[str]) -> Optional[str]:
"""File name of the saved copy of a replied-to post's looping video, if there is one"""
if not url or not url.startswith(("http://", "https://")):
return None
name = video_name_for(url)
return name if (preview_folder(data_path) / name).is_file() else None
def fetch_video(data_path, url: Optional[str], site_url: str) -> bool:
"""Save a copy of a replied-to post's gifv (a short silent MP4) unless there is one"""
if not url or not url.startswith(("http://", "https://")):
return False
path = preview_folder(data_path) / video_name_for(url)
if path.is_file():
return True
try:
data = _download(url, site_url, VIDEO_MAX_BYTES)
if data[4:8] != b"ftyp":
raise ValueError("not an MP4")
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
tmp.write_bytes(data)
tmp.replace(path)
return True
except Exception as e:
logger.info(f"Could not save video {url}: {type(e).__name__}: {e}")
return False
def _square(image):
from PIL import Image
side = min(image.size)
@@ -81,6 +118,19 @@ def _narrow(image):
return image.resize((PREVIEW_WIDTH, height), Image.LANCZOS)
def _save_animated(image, tmp: Path) -> None:
"""An animated GIF/WebP/PNG as an animated WebP, frames narrowed (v2.20.1)"""
from PIL import ImageSequence
frames, durations = [], []
for frame in ImageSequence.Iterator(image):
if len(frames) >= MAX_FRAMES:
break
frames.append(_narrow(frame.convert("RGBA")))
durations.append(frame.info.get("duration") or image.info.get("duration") or 100)
frames[0].save(tmp, "WEBP", save_all=True, append_images=frames[1:], duration=durations,
loop=0, quality=80)
def _save(path: Path, url: Optional[str], site_url: str, shape, max_bytes: int,
what: str) -> bool:
if not url or not url.startswith(("http://", "https://")):
@@ -91,12 +141,15 @@ def _save(path: Path, url: Optional[str], site_url: str, shape, max_bytes: int,
try:
with Image.open(io.BytesIO(_download(url, site_url, max_bytes))) as image:
image.load()
image = image.convert("RGBA" if "A" in image.getbands() else "RGB")
image = shape(image)
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(".tmp")
image.save(tmp, "WEBP", quality=80)
if shape is _narrow and getattr(image, "n_frames", 1) > 1:
_save_animated(image, tmp)
else:
image.load()
image = image.convert("RGBA" if "A" in image.getbands() else "RGB")
image = shape(image)
image.save(tmp, "WEBP", quality=80)
tmp.replace(path)
return True
except Exception as e:
@@ -199,18 +252,19 @@ def _download(url: str, site_url: str, max_bytes: int = MAX_BYTES) -> bytes:
return _get(url, headers, max_bytes)
def _context_images(conn) -> tuple[list, list]:
"""Avatars and post images of reply contexts that notes use (v2.19.0)"""
def _context_images(conn) -> tuple[list, list, list]:
"""Avatars, post images and gifvs of reply contexts that notes use (v2.19.0)"""
rows = conn.execute(
"""
SELECT rc.author_photo, rc.photo FROM reply_contexts rc
SELECT rc.author_photo, rc.photo, rc.video FROM reply_contexts rc
WHERE rc.fetch_status = 'fetched' AND EXISTS (
SELECT 1 FROM notes n WHERE n.deleted_at IS NULL AND rc.url IN
(n.in_reply_to, n.like_of, n.repost_of, n.bookmark_of))
ORDER BY rc.id DESC
"""
).fetchall()
return [r[0] for r in rows if r[0]], [r[1] for r in rows if r[1]]
return ([r[0] for r in rows if r[0]], [r[1] for r in rows if r[1]],
[r[2] for r in rows if r[2]])
def fill_missing(db_path, data_path, site_url: str, limit: int = 20) -> int:
@@ -226,11 +280,12 @@ def fill_missing(db_path, data_path, site_url: str, limit: int = 20) -> int:
ORDER BY id DESC
"""
)]
context_avatars, context_photos = _context_images(conn)
context_avatars, context_photos, context_videos = _context_images(conn)
finally:
conn.close()
wanted = [(url, cached_name, fetch) for url in dict.fromkeys(urls + context_avatars)]
wanted += [(url, cached_preview, fetch_preview) for url in dict.fromkeys(context_photos)]
wanted += [(url, cached_video, fetch_video) for url in dict.fromkeys(context_videos)]
tried = 0
for url, cached, save in wanted:
if tried >= limit:
@@ -248,17 +303,18 @@ def remove_unused(db_path, data_path) -> int:
try:
used = {name_for(row[0]) for row in conn.execute(
"SELECT DISTINCT author_photo FROM webmentions WHERE author_photo IS NOT NULL")}
context_avatars, context_photos = _context_images(conn)
context_avatars, context_photos, context_videos = _context_images(conn)
finally:
conn.close()
used |= {name_for(url) for url in context_avatars}
used_previews = {name_for(url) for url in context_photos}
used_previews |= {video_name_for(url) for url in context_videos}
removed = 0
for directory, keep in ((folder(data_path), used), (preview_folder(data_path), used_previews)):
if not directory.is_dir():
continue
for path in directory.iterdir():
if NAME_RE.match(path.name) and path.name not in keep:
if (NAME_RE.match(path.name) or VIDEO_NAME_RE.match(path.name)) and path.name not in keep:
path.unlink(missing_ok=True)
removed += 1
return removed
+33 -3
View File
@@ -58,6 +58,34 @@ class Response:
opted_out: bool = False # set by the client from silo-specific signals
content_html: Optional[str] = None
published: Optional[str] = None
# Replies in a thread (v2.20.2): the silo id of the response this one
# answers (None: our copy itself), and whether we wrote it
parent_id: Optional[str] = None
own: bool = False
def thread_members(posts: List[dict], root: str) -> Dict[str, Optional[str]]:
"""
Which replies under one of our copies to keep, and what each answers
(v2.20.2). `posts` are {'id', 'parent', 'own'} for every reply in the
conversation, in any order. Kept: others' replies to our copy, and any
reply (ours too) to a kept reply. Our own replies to our copy are the
thread continuing, not responses. Returns {id: parent id or None}.
"""
kept: Dict[str, Optional[str]] = {}
pending = list(posts)
while True:
rest = []
for post in pending:
if post["parent"] == root and not post["own"]:
kept[post["id"]] = None
elif post["parent"] in kept and post["parent"] != post["id"]:
kept[post["id"]] = post["parent"]
else:
rest.append(post)
if len(rest) == len(pending):
return kept
pending = rest
@dataclass
@@ -292,9 +320,9 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
INSERT INTO webmentions (
source_url, target_url, target_note_id, mention_type, author_name, author_url,
author_photo, content_snippet, validation_status, moderation_status,
validated_at, source_published, origin, remote_id, notified_at)
validated_at, source_published, origin, remote_id, notified_at, parent_remote_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'valid', ?, CURRENT_TIMESTAMP, ?, ?, ?,
CASE WHEN ? THEN CURRENT_TIMESTAMP END)
CASE WHEN ? THEN CURRENT_TIMESTAMP END, ?)
ON CONFLICT(source_url, target_url) DO UPDATE SET
target_note_id = excluded.target_note_id,
mention_type = excluded.mention_type,
@@ -305,6 +333,7 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
source_published = COALESCE(excluded.source_published, webmentions.source_published),
origin = excluded.origin,
remote_id = excluded.remote_id,
parent_remote_id = excluded.parent_remote_id,
validation_status = 'valid',
validation_error = NULL,
validated_at = CURRENT_TIMESTAMP,
@@ -312,7 +341,8 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
""",
(r.source_url, target_url, note_id, r.kind, r.author_name or None, r.author_url or None,
r.author_photo or None, snippet, moderation, r.published, origin, r.remote_id,
1 if quiet else 0),
# Our own replies in a thread are no news to us
1 if quiet or r.own else 0, r.parent_id),
)
if dropped:
marks = ",".join("?" * len(dropped))
+7 -2
View File
@@ -65,7 +65,7 @@ FEDIVERSE_PATH_RE = re.compile(
FEDI_TITLE_RE = re.compile(r"^(.*?)\s*\((@[^)\s]+)\)\s*$")
FIELDS = ('title', 'content_snippet', 'published_at', 'author_name', 'author_handle',
'author_url', 'author_photo', 'site_name', 'photo', 'photo_alt')
'author_url', 'author_photo', 'site_name', 'photo', 'photo_alt', 'video')
def fetch_reply_context(
@@ -300,11 +300,14 @@ def _from_status(status: dict, url: str) -> Dict[str, Any]:
snippet = _text_snippet(f"CW: {spoiler}")
else:
snippet = _snippet(status.get('content') or '') or None
photo = photo_alt = None
photo = photo_alt = video = None
if not status.get('sensitive'):
for media in status.get('media_attachments') or []:
if media.get('type') in ('image', 'gifv', 'video') and _is_http_url(media.get('preview_url')):
photo, photo_alt = media['preview_url'], media.get('description') or None
# A "GIF" on Mastodon is a silent looping MP4; the preview is its first frame
if media['type'] == 'gifv' and _is_http_url(media.get('url')):
video = media['url']
break
avatar = account.get('avatar_static') or account.get('avatar')
return {
@@ -319,6 +322,7 @@ def _from_status(status: dict, url: str) -> Dict[str, Any]:
'site_name': urlparse(url).hostname,
'photo': photo,
'photo_alt': photo_alt,
'video': video,
}
@@ -482,6 +486,7 @@ def _save_images(context: Dict[str, Any], site_url: str) -> None:
data_path = current_app.config['DATA_PATH']
avatars.fetch(data_path, context.get('author_photo'), site_url)
avatars.fetch_preview(data_path, context.get('photo'), site_url)
avatars.fetch_video(data_path, context.get('video'), site_url)
except Exception as e:
logger.info(f"Could not save reply context images: {e}")
+4 -3
View File
@@ -268,12 +268,13 @@ def avatar(name: str):
def preview(name: str):
"""A saved copy of a replied-to post's image (v2.19.0)"""
from pathlib import Path
from starpunk.avatars import NAME_RE, preview_folder
from starpunk.avatars import NAME_RE, VIDEO_NAME_RE, preview_folder
if not NAME_RE.match(name):
if not (NAME_RE.match(name) or VIDEO_NAME_RE.match(name)):
abort(404)
response = send_from_directory(
preview_folder(Path(current_app.config.get('DATA_PATH', 'data'))), name)
preview_folder(Path(current_app.config.get('DATA_PATH', 'data'))), name,
mimetype='video/mp4' if name.endswith('.mp4') else None)
response.headers['Cache-Control'] = 'public, max-age=604800'
return response
+214 -42
View File
@@ -25,6 +25,7 @@ import logging
import os
import re
import threading
from urllib.parse import urlparse
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
@@ -102,10 +103,14 @@ class MastodonClient:
"content_type": "text/markdown",
}
def create_status(self, text: str, media_ids: List[str], idempotency_key: str) -> dict:
def create_status(self, text: str, media_ids: List[str], idempotency_key: str,
in_reply_to_id: Optional[str] = None) -> dict:
body = self._status_body(text, media_ids)
if in_reply_to_id:
body["in_reply_to_id"] = in_reply_to_id
response = self._call(
"POST", "/api/v1/statuses",
json=self._status_body(text, media_ids),
json=body,
headers={"Idempotency-Key": idempotency_key},
)
data = response.json()
@@ -143,14 +148,45 @@ class MastodonClient:
status = self.resolve_status(url)
return status["id"] if status else None
def _reply_parent(self, post: "Post") -> tuple:
"""(status id, '@acct ' mention) a reply threads under; (None, '') if it can't (v2.20.0)
Our own crosspost of the replied-to note, else the replied-to post as
this instance resolves it (any fediverse post). Replies to others
start with their @mention, as Mastodon and GoToSocial clients do, so
they are notified and their server gets the reply."""
if not post.reply_to:
return None, ""
if post.reply_remote_id and post.reply_remote_id.split(":", 1)[0] not in self.NATIVE:
return post.reply_remote_id, ""
try:
status = self.resolve_status(post.reply_to)
except SyndicationError as e:
current_app.logger.info(f"Could not resolve {post.reply_to}: {e}")
return None, ""
if not status:
return None, ""
account = status.get("account") or {}
try:
mine = account.get("id") == self._account_id()
except SyndicationError:
mine = False
mention = "" if mine or not account.get("acct") else f"@{account['acct']} "
return status["id"], mention
def _prepare(self, post: "Post") -> tuple:
"""(text, media ids, id of the status it replies to)"""
if post.kind == "repost":
return compose_repost_text(post, post.note.content, post.max_chars), []
return compose_repost_text(post, post.note.content, post.max_chars), [], None
parent, mention = self._reply_parent(post)
# Images in the text are sent as attachments instead (v2.6.6)
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=strip_images(post.note.content))
body = strip_images(post.note.content)
if mention.strip() and mention.strip() not in body:
body = mention + body.lstrip()
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body,
lead="" if parent else reply_line(post))
media_ids = [self.upload_media(f["path"], f["mime_type"], f["description"]) for f in post.media]
return text, media_ids
return text, media_ids, parent
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
if post.kind in ("like", "repost") and post.target_url:
@@ -165,13 +201,13 @@ class MastodonClient:
return {"id": f"{action}:{status_id}", "url": None}
if post.kind == "like":
return {"id": None, "url": None} # nothing to like here
text, media_ids = self._prepare(post)
return self.create_status(text, media_ids, idempotency_key=key)
text, media_ids, parent = self._prepare(post)
return self.create_status(text, media_ids, idempotency_key=key, in_reply_to_id=parent)
def update(self, remote_id: str, post: "Post") -> dict:
if remote_id.split(":", 1)[0] in self.NATIVE:
return {"id": remote_id, "url": None} # a favourite/boost has nothing to edit
text, media_ids = self._prepare(post)
text, media_ids, _ = self._prepare(post) # edits can't move a status in a thread
return self.update_status(remote_id, text, media_ids)
# --- Backfeed (v2.13.0, ADR-073) ---
@@ -204,8 +240,9 @@ class MastodonClient:
}
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
"""Likes, boosts and public direct replies of one of our statuses"""
from starpunk.backfeed import Harvest, Response
"""Likes, boosts and public replies of one of our statuses, with the
replies to those replies (v2.20.2)"""
from starpunk.backfeed import Harvest, Response, thread_members
me = self._account_id()
found, complete = [], True
@@ -219,14 +256,18 @@ class MastodonClient:
found.append(Response(kind=kind, source_url=f"{remote_url}#{fragment}-{account['id']}",
remote_id=account["id"], **self._author(account)))
context = self._call("GET", f"/api/v1/statuses/{remote_id}/context").json() or {}
for status in context.get("descendants") or []:
account = status.get("account") or {}
if (status.get("in_reply_to_id") != remote_id or status.get("visibility") != "public"
or account.get("id") == me):
public = [st for st in context.get("descendants") or [] if st.get("visibility") == "public"]
kept = thread_members([{"id": st["id"], "parent": st.get("in_reply_to_id"),
"own": (st.get("account") or {}).get("id") == me} for st in public],
remote_id)
for status in public:
if status["id"] not in kept:
continue
account = status.get("account") or {}
found.append(Response(kind="reply", source_url=status.get("url") or status["uri"],
remote_id=status["id"], content_html=status.get("content") or "",
published=status.get("created_at"), **self._author(account)))
published=status.get("created_at"), parent_id=kept[status["id"]],
own=account.get("id") == me, **self._author(account)))
return Harvest(found, complete)
def delete(self, remote_id: str) -> None:
@@ -241,6 +282,7 @@ class MastodonClient:
BLUESKY_BLOB_LIMIT = 1_000_000 # app.bsky.embed.images maxSize
BLUESKY_THREAD_DEPTH = 10 # how deep backfeed follows replies to replies
_TID_CHARS = "234567abcdefghijklmnopqrstuvwxyz"
@@ -359,14 +401,44 @@ class BlueskyClient:
current_app.logger.info(f"No link card image for {url}: {e}")
return {"$type": "app.bsky.embed.external", "external": external}
def _reply_ref(self, post: "Post") -> Optional[dict]:
"""{'root', 'parent'} for a reply to a Bluesky post (ours or anyone's); None otherwise (v2.20.0)"""
if not post.reply_to:
return None
uri = post.reply_remote_id if (post.reply_remote_id or "").startswith("at://") and \
f"/{self.POST}/" in post.reply_remote_id else None
try:
if uri is None:
found = self.resolve_post(post.reply_to)
if not found:
return None
uri = found["uri"]
views = self._call("GET", "app.bsky.feed.getPosts", params={"uris": uri}).json().get("posts") or []
except SyndicationError as e:
current_app.logger.info(f"Could not resolve {post.reply_to} on Bluesky: {e}")
return None
if not views:
return None
parent = {"uri": views[0]["uri"], "cid": views[0]["cid"]}
root = ((views[0].get("record") or {}).get("reply") or {}).get("root") or parent
return {"root": {"uri": root["uri"], "cid": root["cid"]}, "parent": parent}
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None,
anchors: List[tuple] = ()) -> dict:
anchors: List[tuple] = (), reply: Optional[dict] = None) -> dict:
if text is None:
# An article is posted as a teaser (compose_text), so its links aren't in the text
body, anchors = (None, []) if post.note.title else \
plain_text_links(post.note.content, links="anchor")
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body)
lead = ""
if post.reply_to and not reply:
# "↩ Replying to <name>", the name linking to the post (v2.20.0)
lead = reply_line(post, with_url=False)
anchors = [(reply_label(post), post.reply_to), *anchors]
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body,
lead=lead)
record = {"$type": self.POST, "text": text, "createdAt": self._now()}
if reply:
record["reply"] = reply
facets = bluesky_facets(text, anchors)
if facets:
record["facets"] = facets
@@ -464,7 +536,7 @@ class BlueskyClient:
existing = self._find_existing(post.permalink)
if existing:
return existing
record = self._record(post)
record = self._record(post, reply=self._reply_ref(post))
response = self._call("POST", "com.atproto.repo.createRecord",
json={"repo": self.did or self.identifier, "collection": self.POST,
"record": record})
@@ -522,8 +594,9 @@ class BlueskyClient:
)
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
"""Likes, reposts, quotes and direct replies of one of our posts"""
from starpunk.backfeed import Harvest, Response
"""Likes, reposts, quotes and replies of one of our posts, with the
replies to those replies (v2.20.2)"""
from starpunk.backfeed import Harvest, Response, thread_members
if not self.did:
with self._lock:
@@ -546,10 +619,26 @@ class BlueskyClient:
complete = complete and done
posts = [self._post_response("quote", view) for view in quotes]
thread = self._call("GET", "app.bsky.feed.getPostThread",
params={"uri": remote_id, "depth": 1, "parentHeight": 0}).json()
for item in (thread.get("thread") or {}).get("replies") or []:
if item.get("post"): # skips blocked and deleted replies
posts.append(self._post_response("reply", item["post"]))
params={"uri": remote_id, "depth": BLUESKY_THREAD_DEPTH,
"parentHeight": 0}).json()
# The whole conversation under our post (v2.20.2)
views, stack = {}, [(remote_id, item) for item in (thread.get("thread") or {}).get("replies") or []]
while stack:
parent, item = stack.pop()
post = item.get("post")
if not post: # blocked or deleted: its replies go with it
continue
views[post["uri"]] = (parent, post)
stack.extend((post["uri"], child) for child in item.get("replies") or [])
kept = thread_members([{"id": uri, "parent": parent,
"own": (post.get("author") or {}).get("did") == self.did}
for uri, (parent, post) in views.items()], remote_id)
for uri, (_, post) in sorted(views.items(), key=lambda v: v[1][1].get("indexedAt") or ""):
if uri in kept:
response = self._post_response("reply", post)
response.parent_id = kept[uri]
response.own = response.author_id == self.did
posts.append(response)
# Post views carry no bio; fetch them to honour #nobridge
dids = list({p.author_id for p in posts if p.author_id})
bios = {}
@@ -560,7 +649,7 @@ class BlueskyClient:
for post in posts:
post.author_bio = bios.get(post.author_id, "")
found.extend(posts)
return Harvest([r for r in found if r.author_id != self.did], complete)
return Harvest([r for r in found if r.author_id != self.did or r.own], complete)
def delete(self, remote_id: str) -> None:
# at://<did>/app.bsky.feed.post/<rkey>; deleting a missing record succeeds
@@ -843,15 +932,26 @@ class ThreadsClient:
def _text(self, post: "Post") -> str:
# Links written "text (url)"; the first one gets Threads' preview card.
# Meta allows 5 different links per post and the permalink is one.
body = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
# A reply to someone else's post says so, with its link (v2.20.0);
# one to our own Threads post is a native reply instead
lead = "" if self._reply_to_id(post) else reply_line(post)
links = THREADS_MAX_LINKS - 1 - (1 if lead else 0)
body = plain_text(post.note.content, links="inline", max_urls=links)
limit = post.max_chars
while True:
text = compose_text(post.note, post.permalink, [], limit, body=body)
text = compose_text(post.note, post.permalink, [], limit, body=body, lead=lead)
over = threads_length(text) - post.max_chars
if over <= 0:
return text
limit -= over
@staticmethod
def _reply_to_id(post: "Post") -> Optional[str]:
"""Media id of our own Threads post this replies to. The API can't look
up other people's posts by URL, so only those thread natively."""
remote = post.reply_remote_id or ""
return remote if post.reply_to and remote.isdigit() else None
def _image_container(self, uid: str, media: dict, **extra) -> str:
params = {"media_type": "IMAGE", "image_url": media["url"], **extra}
if media["description"]:
@@ -874,6 +974,8 @@ class ThreadsClient:
return self._call("POST", f"{uid}/threads", media_type="TEXT",
text=self._repost_text(post))["id"]
common = {"text": self._text(post)}
if self._reply_to_id(post):
common["reply_to_id"] = self._reply_to_id(post)
topic = threads_topic(post.tags)
if topic:
common["topic_tag"] = topic
@@ -934,15 +1036,17 @@ class ThreadsClient:
# --- Backfeed (v2.13.0, ADR-073) ---
REPLY_FIELDS = "id,text,username,permalink,timestamp,profile_picture_url,hide_status"
REPLY_FIELDS = "id,text,username,permalink,timestamp,profile_picture_url,hide_status,replied_to"
SHOWN = (None, "NOT_HUSHED", "UNHUSHED") # hide_status values of replies nobody hid
def responses(self, remote_id: str, remote_url: str, cap: int = 1000):
"""
Direct replies to one of our posts, and its like/repost/quote counts.
The API doesn't say who liked, reposted or quoted (ADR-073 Decision 7).
Replies to one of our posts, with the replies to those replies
(/conversation, v2.20.2; /replies only has the first level), and its
like/repost/quote counts. The API doesn't say who liked, reposted or
quoted (ADR-073 Decision 7).
"""
from starpunk.backfeed import Harvest, Response, text_html
from starpunk.backfeed import Harvest, Response, text_html, thread_members
self._me()
replies, after, complete = [], None, True
@@ -950,7 +1054,7 @@ class ThreadsClient:
params = {"fields": self.REPLY_FIELDS}
if after:
params["after"] = after
data = self._call("GET", f"{remote_id}/replies", **params)
data = self._call("GET", f"{remote_id}/conversation", **params)
page = data.get("data") or []
replies.extend(page)
if len(replies) >= cap:
@@ -960,14 +1064,18 @@ class ThreadsClient:
after = (paging.get("cursors") or {}).get("after")
if not page or not after or not paging.get("next"):
break
# Private accounts have no username or permalink: nothing public to show
shown = [r for r in replies if r.get("username") and r.get("permalink")
and r.get("hide_status") in self.SHOWN]
kept = thread_members([{"id": r["id"], "parent": (r.get("replied_to") or {}).get("id"),
"own": r["username"] == self.username} for r in shown], remote_id)
found = []
for reply in replies:
username = reply.get("username")
# Private accounts have no username or permalink: nothing public to show
if (not username or not reply.get("permalink") or username == self.username
or reply.get("hide_status") not in self.SHOWN):
for reply in sorted(shown, key=lambda r: r.get("timestamp") or ""):
if reply["id"] not in kept:
continue
username = reply["username"]
found.append(Response(
parent_id=kept[reply["id"]], own=username == self.username,
kind="reply", source_url=reply["permalink"], remote_id=reply["id"],
author_id=username, author_name=username,
author_url=f"https://www.threads.com/@{username}",
@@ -1011,6 +1119,22 @@ class Post:
kind: str = "note" # note, like or repost (v2.5.0)
target_url: Optional[str] = None # what a like/repost points at
target_title: Optional[str] = None # its fetched title, if any
reply_to: Optional[str] = None # what a reply answers (v2.20.0)
reply_name: Optional[str] = None # its author (or title), from the reply context
reply_remote_id: Optional[str] = None # its crosspost on this target, if it's one of ours
def reply_line(post: "Post", with_url: bool = True) -> str:
"""'↩ Replying to <who>' (and the URL) for replies that can't thread natively (v2.20.0)"""
if not post.reply_to:
return ""
who = post.reply_name or urlparse(post.reply_to).hostname or post.reply_to
return f"↩ Replying to {who}" + (f"\n{post.reply_to}" if with_url else "")
def reply_label(post: "Post") -> str:
"""The linked part of a Bluesky reply line"""
return post.reply_name or urlparse(post.reply_to).hostname or post.reply_to
def compose_repost_text(post: "Post", comment: str, max_chars: int) -> str:
@@ -1200,7 +1324,7 @@ def article_teaser(note) -> str:
def compose_text(note, permalink: str, tags: List[str], max_chars: int,
body: Optional[str] = None) -> str:
body: Optional[str] = None, lead: str = "") -> str:
"""
Full note (Markdown unless `body` is given), then the permalink and
hashtags; shortened at a word boundary if too long. On short targets the
@@ -1221,9 +1345,14 @@ def compose_text(note, permalink: str, tags: List[str], max_chars: int,
if getattr(note, "bookmark_of", None):
# A bookmark leads with what it bookmarks (v2.9.0)
body = f"🔖 {note.bookmark_of}\n\n{body}".strip()
if len(body) + len(tail) > max_chars:
room = max_chars - len(tail) - 1
body = body[:room].rsplit(" ", 1)[0].rstrip() + "…"
# A reply that can't thread says what it replies to (v2.20.0); the lead
# is kept whole and the body shortened instead
lead = lead.strip()
head = f"{lead}\n\n" if lead and body else lead
if len(head) + len(body) + len(tail) > max_chars:
room = max_chars - len(tail) - len(head) - 1
body = body[:max(room, 0)].rsplit(" ", 1)[0].rstrip() + "…"
body = head + body
# A photo post may have no text: start with the link (v2.6.5)
return (body + tail).lstrip()
@@ -1510,6 +1639,48 @@ def _target_title(url: Optional[str]) -> Optional[str]:
return " ".join(title.split())[:200] or None
def _own_note_id(url: str) -> Optional[int]:
"""The note a URL is, if it's one of ours: its page on this site, or one
of its crossposts on any network (v2.20.0)"""
db = get_db(current_app)
row = db.execute("SELECT note_id FROM syndications WHERE remote_url = ? AND state = 'synced'",
(url,)).fetchone()
if row:
return row["note_id"]
site = urlparse(current_app.config.get("SITE_URL", ""))
parsed = urlparse(url)
parts = [p for p in parsed.path.split("/") if p]
if parsed.netloc != site.netloc or len(parts) != 2 or \
parts[0] not in ("note", "notes", "article", "articles"):
return None
row = db.execute("SELECT id FROM notes WHERE slug = ? AND deleted_at IS NULL",
(parts[1],)).fetchone()
return row["id"] if row else None
def _reply_fields(url: Optional[str], target_id: str) -> dict:
"""Post fields for a reply: who it answers, and its crosspost on this target if it's ours"""
if not url:
return {}
fields = {"reply_to": url}
try:
from starpunk.webmention_queries import get_reply_context
context = get_reply_context(url) or {}
if context.get("fetch_status") == "fetched":
name = context.get("author_name") or context.get("title")
fields["reply_name"] = " ".join(name.split())[:100] if name else None
note_id = _own_note_id(url)
if note_id is not None:
row = get_db(current_app).execute(
"SELECT remote_id FROM syndications WHERE note_id = ? AND target = ? "
"AND state = 'synced' AND remote_id IS NOT NULL", (note_id, target_id)).fetchone()
if row:
fields["reply_remote_id"] = row["remote_id"]
except Exception as e:
current_app.logger.info(f"Reply details for {url}: {e}")
return fields
def _reconcile(row, target: Target) -> tuple:
"""Bring the remote post in line with the note. Returns (remote_id, remote_url)."""
from starpunk.notes import get_note
@@ -1538,6 +1709,7 @@ def _reconcile(row, target: Target) -> tuple:
kind=note.kind,
target_url=note.reacts_to,
target_title=_target_title(note.reacts_to),
**(_reply_fields(note.in_reply_to, target.id) if note.kind == "note" else {}),
)
if row["remote_id"]:
+23
View File
@@ -107,9 +107,32 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
else:
grouped['mentions'].append(wm)
grouped['reply_threads'] = reply_threads(grouped['replies'])
return grouped
def reply_threads(replies: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""
Replies nested under the replies they answer (v2.20.2): each gets
'children'; the top-level ones are returned. A reply whose parent isn't
shown (hidden, removed) is shown at the top level.
"""
by_id = {}
for wm in replies:
wm['children'] = []
if wm.get('remote_id'):
by_id[(wm.get('origin'), wm['remote_id'])] = wm
top = []
for wm in replies:
parent = by_id.get((wm.get('origin'), wm.get('parent_remote_id'))) \
if wm.get('parent_remote_id') else None
if parent is not None and parent is not wm:
parent['children'].append(wm)
else:
top.append(wm)
return top
def response_counts(note_ids: List[int]) -> Dict[int, Dict[str, int]]:
"""
Likes, reposts, replies and quotes per note, for lists of notes
+4
View File
@@ -391,6 +391,10 @@ small { display: block; margin-top: var(--spacing-xs); color: var(--color-text-l
.webmention-reply .webmention-date::after { content: ""; position: absolute; inset: 0; }
.webmention-reply .webmention-author a, .webmention-reply .e-content a { position: relative; z-index: 1; }
.webmention-reply:hover { cursor: pointer; }
/* Replies to replies (v2.20.2): nested, with a line down the side */
.webmention-thread { position: relative; z-index: 1; margin-top: var(--spacing-sm); padding-left: var(--spacing-md); border-left: 2px solid var(--color-border); }
.webmention-thread .webmention-reply { margin-bottom: var(--spacing-sm); padding: var(--spacing-xs) 0 var(--spacing-xs) var(--spacing-xs); background: none; }
.webmention-thread .webmention-reply:last-child { margin-bottom: 0; }
/* Counts in lists of notes (v2.13.4) */
.list-counts { display: inline-flex; gap: 0.6rem; color: var(--color-text-light); }
+1 -1
View File
@@ -74,7 +74,7 @@
{% if replies %}
<div class="webmention-group webmention-replies">
<h3>{{ replies|length }} {{ 'Reply' if replies|length == 1 else 'Replies' }}</h3>
{% for wm in replies %}{{ responses.reply(wm) }}{% endfor %}
{% for wm in reply_threads or replies %}{{ responses.reply(wm) }}{% endfor %}
</div>
{% endif %}
+5 -1
View File
@@ -34,7 +34,11 @@
{% if ctx.title %}<p class="p-name reply-title">{{ ctx.title }}</p>{% endif %}
{% if ctx.content_snippet %}<div class="e-content reply-content">{{ ctx.content_snippet|safe }}</div>{% endif %}
{% set image = ctx.photo|local_preview %}
{% if image %}<img class="reply-photo" src="{{ image }}" alt="{{ ctx.photo_alt or '' }}" loading="lazy">{% endif %}
{% set video = ctx.video|local_video if ctx.video is defined else None %}
{% if video %}
{# Mastodon "GIFs" are silent looping MP4s: shown as Mastodon shows them (v2.20.1) #}
<video class="reply-photo" src="{{ video }}"{% if image %} poster="{{ image }}"{% endif %} autoplay loop muted playsinline{% if ctx.photo_alt %} aria-label="{{ ctx.photo_alt }}" title="{{ ctx.photo_alt }}"{% endif %}></video>
{% elif image %}<img class="reply-photo" src="{{ image }}" alt="{{ ctx.photo_alt or '' }}" loading="lazy">{% endif %}
<footer class="reply-source">
<a class="u-url" href="{{ url }}">{{ _when(ctx, '%B %-d, %Y · %-I:%M %p') }}{% if ctx.published_at and ctx.published_at.strftime is defined %} · {% endif %}{{ ctx.site_name or url.split('/')[2] }} ↗</a>
</footer>
+7 -1
View File
@@ -30,7 +30,8 @@
</a></span>
{%- endmacro %}
{# A reply, shown as a comment #}
{# A reply, shown as a comment; replies to it are nested inside, as
comments of the comment (v2.20.2) #}
{% macro reply(wm) %}
<article class="webmention webmention-reply u-comment h-cite">
<div class="webmention-author">{{ author_card(wm) }}</div>
@@ -38,6 +39,11 @@
<div class="e-content">{{ wm.content_snippet|safe }}</div>
{% endif %}
<a class="u-url webmention-date" href="{{ response_url(wm) }}">{{ response_time(wm) }}</a> {{ via(wm) }}
{% if wm.children %}
<div class="webmention-thread">
{% for child in wm.children %}{{ reply(child) }}{% endfor %}
</div>
{% endif %}
</article>
{%- endmacro %}
+107 -11
View File
@@ -392,18 +392,25 @@ def mastodon_api(request):
"account": who or account("r" + n)}
return httpx.Response(200, json={"ancestors": [], "descendants": [
status("10"), status("11", vis="unlisted"), status("12", parent="10"),
status("13", who=account("me"))]})
status("13", who=account("me")), status("14", parent="12", who=account("me")),
status("15", parent="13")]})
return httpx.Response(404)
def test_mastodon_client_reads_likes_boosts_and_public_direct_replies():
def test_mastodon_client_reads_likes_boosts_and_public_reply_threads():
gts = MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(mastodon_api)))
harvest = gts.responses("01COPY", COPY)
assert harvest.complete
got = [(r.kind, r.source_url) for r in harvest.responses]
# 11 isn't public; 13 is our own thread continuing, so 15 under it is left out too.
# 12 answers 10, and our 14 answers 12 (v2.20.2)
assert got == [("like", f"{COPY}#liked-by-1"), ("like", f"{COPY}#liked-by-3"),
("repost", f"{COPY}#reposted-by-2"), ("reply", f"{GTS}/@u/statuses/10")]
assert harvest.responses[-1].content_html == "<p>reply 10</p>"
("repost", f"{COPY}#reposted-by-2"), ("reply", f"{GTS}/@u/statuses/10"),
("reply", f"{GTS}/@u/statuses/12"), ("reply", f"{GTS}/@u/statuses/14")]
replies = {r.remote_id: r for r in harvest.responses if r.kind == "reply"}
assert replies["10"].content_html == "<p>reply 10</p>"
assert [(replies[i].parent_id, replies[i].own) for i in ("10", "12", "14")] == \
[(None, False), ("10", False), ("12", True)]
assert gts.responses("01COPY", COPY, cap=1).complete is False
@@ -457,14 +464,22 @@ def test_threads_client_reads_replies_and_counts():
params = parse_qs(request.url.query.decode())
if path.endswith("/me"):
return httpx.Response(200, json={"id": "42", "username": "me"})
if path.endswith("/111/replies"):
assert "profile_picture_url" in params["fields"][0]
if path.endswith("/111/conversation"):
# The whole conversation, newest first, as Meta returns it (v2.20.2)
assert "profile_picture_url" in params["fields"][0] and "replied_to" in params["fields"][0]
to = lambda i: {"replied_to": {"id": i}}
return httpx.Response(200, json={"data": [
{"id": "6", "text": "Ha", "username": "ann", "permalink": "https://www.threads.com/@ann/post/C",
"timestamp": "2026-11-10T02:00:00+0000", "hide_status": "NOT_HUSHED", **to("5")},
{"id": "5", "text": "Yes", "username": "me", "permalink": "https://www.threads.com/@me/post/B",
"timestamp": "2026-11-10T01:00:00+0000", **to("1")},
{"id": "7", "text": "under mine", "username": "bo", "permalink": "p", "hide_status": "NOT_HUSHED",
**to("4")},
{"id": "1", "text": "Hi", "username": "ann", "permalink": "https://www.threads.com/@ann/post/A",
"timestamp": "2026-11-10T00:00:00+0000", "hide_status": "NOT_HUSHED"},
{"id": "2", "text": "private"},
{"id": "3", "text": "hidden", "username": "x", "permalink": "p", "hide_status": "HIDDEN"},
{"id": "4", "text": "mine", "username": "me", "permalink": "p"}]})
"timestamp": "2026-11-10T00:00:00+0000", "hide_status": "NOT_HUSHED", **to("111")},
{"id": "2", "text": "private", **to("111")},
{"id": "3", "text": "hidden", "username": "x", "permalink": "p", "hide_status": "HIDDEN", **to("111")},
{"id": "4", "text": "mine", "username": "me", "permalink": "p", **to("111")}]})
if path.endswith("/111/insights"):
assert params["metric"] == ["likes,reposts,quotes"]
return httpx.Response(200, json={"data": [
@@ -474,7 +489,10 @@ def test_threads_client_reads_replies_and_counts():
threads = ThreadsClient("tok", http=httpx.Client(transport=httpx.MockTransport(api)))
harvest = threads.responses("111", "https://www.threads.com/@me/post/X")
assert [(r.kind, r.author_name, r.content_html) for r in harvest.responses] == [("reply", "ann", "<p>Hi</p>")]
# Our own reply to our post continues the thread (4, and 7 under it, are left out);
# our reply to ann (5) and her answer (6) are kept, oldest first
assert [(r.author_name, r.content_html, r.parent_id, r.own) for r in harvest.responses] == [
("ann", "<p>Hi</p>", None, False), ("me", "<p>Yes</p>", "1", True), ("ann", "<p>Ha</p>", "5", False)]
assert harvest.counts == {"likes": 12, "reposts": 3, "quotes": 0}
@@ -658,3 +676,81 @@ def test_quote_counts_include_anonymous_ones(app, client):
assert "Quoter" in page, "the named quote is listed under Responses"
home = app.test_client().get("/").get_data(as_text=True)
assert 'title="3 quotes"' in home
# --- Reply threads (v2.20.2) ------------------------------------------------------
def test_thread_members_follow_replies_to_replies():
from starpunk.backfeed import thread_members
posts = [ # newest first, as Threads lists them
{"id": "c", "parent": "b", "own": False},
{"id": "b", "parent": "a", "own": True},
{"id": "x", "parent": "self", "own": False},
{"id": "self", "parent": "COPY", "own": True},
{"id": "a", "parent": "COPY", "own": False},
{"id": "loop", "parent": "loop", "own": False},
]
assert thread_members(posts, "COPY") == {"a": None, "b": "a", "c": "b"}
def test_bluesky_client_reads_nested_replies():
me, post = "did:plc:me", "at://did:plc:me/app.bsky.feed.post/abc"
def view(did, rkey, text, at):
return {"uri": f"at://{did}/app.bsky.feed.post/{rkey}", "indexedAt": at,
"author": {"did": did, "handle": did.rsplit(":", 1)[1] + ".example"},
"record": {"text": text, "createdAt": at}}
def api(request):
nsid = request.url.path.removeprefix("/xrpc/")
if nsid == "com.atproto.server.createSession":
return httpx.Response(200, json={"did": me, "handle": "me.example", "accessJwt": "a",
"refreshJwt": "r"})
if nsid in ("app.bsky.feed.getLikes", "app.bsky.feed.getRepostedBy", "app.bsky.feed.getQuotes"):
return httpx.Response(200, json={"likes": [], "repostedBy": [], "posts": []})
if nsid == "app.bsky.feed.getPostThread":
assert int(dict(request.url.params)["depth"]) > 1
return httpx.Response(200, json={"thread": {"post": {}, "replies": [
{"post": view("did:plc:eve", "r1", "First", "2026-11-10T01:00:00Z"), "replies": [
{"post": view(me, "r2", "Thanks", "2026-11-10T02:00:00Z"), "replies": [
{"post": view("did:plc:eve", "r3", "Anytime", "2026-11-10T03:00:00Z")}]}]},
{"post": view(me, "self", "More from me", "2026-11-10T00:30:00Z")}]}})
if nsid == "app.bsky.actor.getProfiles":
return httpx.Response(200, json={"profiles": []})
return httpx.Response(400, json={"error": "Unknown"})
bsky = BlueskyClient("https://bsky.example", "me.example", "pw",
http=httpx.Client(transport=httpx.MockTransport(api)))
replies = [(r.remote_id.rsplit("/", 1)[1], r.parent_id and r.parent_id.rsplit("/", 1)[1], r.own)
for r in bsky.responses(post, "https://bsky.app/profile/me.example/post/abc").responses]
assert replies == [("r1", None, False), ("r2", "r1", True), ("r3", "r2", False)]
def test_reply_threads_are_stored_and_shown_nested(app, client):
app.config["WEBMENTION_ENABLED"] = True
note, _ = crossposted(app)
first, mine, answer = reply(1, "It keeps coming up"), reply(2, "That is terrifying"), reply(3, "To be fair")
mine.parent_id, mine.own, mine.author_name = "1", True, "philskents"
answer.parent_id = "2"
client.harvest = Harvest([first, mine, answer])
run(app)
stored = {r["remote_id"]: r for r in rows(app, note.id)}
assert [stored[i]["parent_remote_id"] for i in "123"] == [None, "1", "2"]
assert stored["2"]["notified_at"], "our own reply is no news to us"
assert not stored["1"]["notified_at"]
page = app.test_client().get(f"/note/{note.slug}").data.decode()
assert "3 Replies" in page
# Each answer sits inside the reply it answers
outer = page.index("It keeps coming up")
assert outer < page.index('class="webmention-thread"') < page.index("That is terrifying") \
< page.index("To be fair")
assert page.count('class="webmention-thread"') == 2
import mf2py
items = mf2py.parse(doc=page, url=f"{SITE}/note/{note.slug}")["items"]
entry = next(i for i in items if "h-entry" in i["type"])
(comment,) = entry["properties"]["comment"]
(nested,) = comment["properties"]["comment"]
assert "terrifying" in json.dumps(nested) and "To be fair" in json.dumps(nested["properties"]["comment"])
+56
View File
@@ -268,3 +268,59 @@ def test_reply_page_and_home_page_show_the_context(app, web):
reply_to = entry["properties"]["in-reply-to"][0]
assert (reply_to if isinstance(reply_to, str) else reply_to["properties"]["url"][0]) == TOOT
assert "Ada Lovelace" not in json.dumps(entry["properties"].get("author", ""))
# --- Moving pictures (v2.20.1) --------------------------------------------------
MP4 = b"\x00\x00\x00\x18ftypmp42" + b"\x00" * 64
def animated_gif():
buffer = io.BytesIO()
frames = [Image.new("RGB", (800, 400), color) for color in ("red", "green", "blue")]
frames[0].save(buffer, "GIF", save_all=True, append_images=frames[1:], duration=80, loop=0)
return buffer.getvalue()
def test_mastodon_gifv_is_kept_as_a_looping_video(app, web, monkeypatch):
status = {**STATUS, "media_attachments": [{
"type": "gifv", "url": "https://masto.example/media/original.mp4",
"preview_url": "https://masto.example/media/small.png", "description": "Engineering report"}]}
web.routes["https://masto.example/api/v1/statuses/117611186265622833"] = \
lambda r: httpx.Response(200, json=status)
web.routes["https://masto.example/media/original.mp4"] = \
lambda r: httpx.Response(200, content=MP4, headers={"Content-Type": "video/mp4"})
note = _reply(app, TOOT, "Same here")
context = fetch(app, TOOT)
assert context["video"] == "https://masto.example/media/original.mp4"
assert avatars.cached_video(app.config["DATA_PATH"], context["video"])
client = app.test_client()
page = client.get(f"/note/{note.slug}").data.decode()
assert "<video" in page and "autoplay loop muted playsinline" in page
assert 'aria-label="Engineering report"' in page
src = page.split('<video class="reply-photo" src="', 1)[1].split('"', 1)[0]
response = client.get(src)
assert response.status_code == 200 and response.mimetype == "video/mp4"
def test_animated_gif_stays_animated(app, web):
web.routes[ARTICLE] = lambda r: httpx.Response(200, html=ARTICLE_PAGE.replace("/img/lead.png", "/img/lead.gif"))
web.routes["https://news.example/img/lead.gif"] = \
lambda r: httpx.Response(200, content=animated_gif(), headers={"Content-Type": "image/gif"})
context = fetch(app, ARTICLE)
data = app.config["DATA_PATH"]
with Image.open(avatars.preview_folder(data) / avatars.cached_preview(data, context["photo"])) as image:
assert image.n_frames == 3
assert image.width == avatars.PREVIEW_WIDTH
def test_unused_videos_are_removed(app, web):
data = app.config["DATA_PATH"]
folder = avatars.preview_folder(data)
folder.mkdir(parents=True, exist_ok=True)
stray = folder / avatars.video_name_for("https://gone.example/x.mp4")
stray.write_bytes(MP4)
avatars.remove_unused(app.config["DATABASE_PATH"], data)
assert not stray.exists()
+216
View File
@@ -0,0 +1,216 @@
"""
Crossposted replies look like replies (v2.20.0)
A reply to a post on the same network threads under it natively (with an
@mention on the fediverse); a reply to one of our own notes threads under
that note's crosspost on every network; anything else starts with
"↩ Replying to <author>" and the link.
"""
import json
from types import SimpleNamespace
import httpx
import pytest
from starpunk import create_app, syndication
from starpunk.notes import create_note, get_note
from starpunk.syndication import (BlueskyClient, MastodonClient, Post, ThreadsClient,
compose_text, get_note_syndications, process_due,
select_targets)
SITE = "https://blog.example.com"
GTS = "https://social.example.com"
FEDI_POST = "https://mastodon.example/@alice/111"
OWN_FEDI_POST = "https://social.example.com/@phil/statuses/01OWN"
BSKY_POST = "https://bsky.app/profile/alice.example/post/3abc"
WEB_PAGE = "https://someblog.example/2026/a-post"
def note(content="Agreed, this cannot be good"):
return SimpleNamespace(content=content, title=None, bookmark_of=None, label=content[:30],
summary=None, excerpt=content)
def post(reply_to=None, **kwargs):
return Post(note=note(), permalink=f"{SITE}/note/1", tags=[], max_chars=500,
reply_to=reply_to, **kwargs)
class FakeGTS:
def __init__(self):
self.created = []
def __call__(self, request):
path = request.url.path
if path == "/api/v1/accounts/verify_credentials":
return httpx.Response(200, json={"id": "me"})
if path == "/api/v2/search":
q = request.url.params["q"]
accounts = {FEDI_POST: {"id": "a2", "acct": "alice@mastodon.example"},
OWN_FEDI_POST: {"id": "me", "acct": "phil"}}
if q in accounts:
return httpx.Response(200, json={"statuses": [{"id": "s1", "account": accounts[q]}]})
return httpx.Response(200, json={"statuses": []})
if path == "/api/v1/statuses" and request.method == "POST":
self.created.append(json.loads(request.content))
sid = f"p{len(self.created)}"
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
return httpx.Response(500)
class FakePDS:
def __init__(self):
self.records = []
def __call__(self, request):
nsid = request.url.path.removeprefix("/xrpc/")
if nsid == "com.atproto.server.createSession":
return httpx.Response(200, json={"did": "did:plc:me", "handle": "me.example",
"accessJwt": "a", "refreshJwt": "r"})
if nsid == "com.atproto.identity.resolveHandle":
return httpx.Response(200, json={"did": "did:plc:alice"})
if nsid == "app.bsky.feed.getPosts":
uri = request.url.params["uris"]
record = {"text": "parent"}
if uri.endswith("/3abc"): # itself a reply in a thread
record["reply"] = {"root": {"uri": "at://did:plc:alice/app.bsky.feed.post/root",
"cid": "rootcid"}, "parent": {}}
return httpx.Response(200, json={"posts": [{"uri": uri, "cid": "cid1", "record": record}]})
if nsid == "com.atproto.repo.createRecord":
body = json.loads(request.content)
self.records.append(body["record"])
uri = f"at://did:plc:me/{body['collection']}/r{len(self.records)}"
return httpx.Response(200, json={"uri": uri, "cid": "c"})
return httpx.Response(500, json={"error": nsid})
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
app = create_app({
"TESTING": True, "SITE_URL": SITE, "ADMIN_ME": "https://example.com", "SESSION_SECRET": "x",
"DATA_PATH": data, "NOTES_PATH": data / "notes", "DATABASE_PATH": data / "starpunk.db",
"GOTOSOCIAL_URL": GTS, "GOTOSOCIAL_ACCESS_TOKEN": "t",
"BLUESKY_HANDLE": "me.example", "BLUESKY_APP_PASSWORD": "p", "BLUESKY_SERVICE": "https://pds.example",
})
yield app
syndication._client_cache.clear()
@pytest.fixture
def gts():
fake = FakeGTS()
return fake, MastodonClient(GTS, "t", http=httpx.Client(transport=httpx.MockTransport(fake)))
@pytest.fixture
def pds():
fake = FakePDS()
return fake, BlueskyClient("https://pds.example", "me.example", "p",
http=httpx.Client(transport=httpx.MockTransport(fake)))
# --- GoToSocial ---------------------------------------------------------------
def test_fediverse_reply_threads_and_mentions(app, gts):
fake, client = gts
with app.app_context():
client.create(post(FEDI_POST, reply_name="Alice"), key="k")
(body,) = fake.created
assert body["in_reply_to_id"] == "s1"
assert body["status"].startswith("@alice@mastodon.example Agreed")
assert "Replying to" not in body["status"]
def test_reply_to_own_fediverse_post_has_no_mention(app, gts):
fake, client = gts
with app.app_context():
client.create(post(OWN_FEDI_POST), key="k")
assert fake.created[0]["in_reply_to_id"] == "s1"
assert fake.created[0]["status"].startswith("Agreed")
def test_reply_to_a_web_page_says_so_on_the_fediverse(app, gts):
fake, client = gts
with app.app_context():
client.create(post(WEB_PAGE, reply_name="Ada"), key="k1")
client.create(post(WEB_PAGE), key="k2")
named, unnamed = fake.created
assert "in_reply_to_id" not in named
assert named["status"].startswith(f"↩ Replying to Ada\n{WEB_PAGE}\n\nAgreed")
assert unnamed["status"].startswith(f"↩ Replying to someblog.example\n{WEB_PAGE}")
# --- Bluesky ------------------------------------------------------------------
def test_bluesky_reply_threads_under_the_post_and_its_root(app, pds):
fake, client = pds
with app.app_context():
client.create(post(BSKY_POST, reply_name="Alice"), key="k")
(record,) = fake.records
assert record["reply"]["parent"] == {"uri": "at://did:plc:alice/app.bsky.feed.post/3abc", "cid": "cid1"}
assert record["reply"]["root"] == {"uri": "at://did:plc:alice/app.bsky.feed.post/root", "cid": "rootcid"}
assert record["text"].startswith("Agreed")
def test_reply_to_a_fediverse_post_says_so_on_bluesky(app, pds):
fake, client = pds
with app.app_context():
client.create(post(FEDI_POST, reply_name="Alice"), key="k")
(record,) = fake.records
assert "reply" not in record
assert record["text"].startswith("↩ Replying to Alice\n\nAgreed")
links = [f for f in record["facets"] if f["features"][0]["uri"] == FEDI_POST]
(link,) = links
start = record["text"].encode().index(b"Alice")
assert link["index"] == {"byteStart": start, "byteEnd": start + len(b"Alice")}
# --- Threads ------------------------------------------------------------------
def test_threads_replies_natively_only_to_our_own_posts(app):
client = ThreadsClient("tok")
calls = []
client._call = lambda method, path, **params: calls.append(params) or {"id": "c1"}
with app.app_context():
client._container("me", post(f"{SITE}/note/0", reply_remote_id="1789"))
client._container("me", post(FEDI_POST, reply_name="Alice"))
own, other = calls
assert own["reply_to_id"] == "1789" and not own["text"].startswith("↩")
assert "reply_to_id" not in other
assert other["text"].startswith(f"↩ Replying to Alice\n{FEDI_POST}\n\nAgreed")
# --- Composing ----------------------------------------------------------------
def test_reply_line_survives_shortening():
text = compose_text(note("word " * 200), f"{SITE}/note/1", [], 300,
lead=f"↩ Replying to Alice\n{FEDI_POST}")
assert text.startswith(f"↩ Replying to Alice\n{FEDI_POST}\n\nword")
assert len(text) <= 300 and text.endswith(f"{SITE}/note/1")
# --- Replies to our own notes thread under their crossposts ----------------------
@pytest.mark.parametrize("via", ["permalink", "crosspost"])
def test_reply_to_own_note_threads_on_every_network(app, gts, pds, monkeypatch, via):
monkeypatch.setattr("starpunk.reply_context.fetch_reply_context", lambda *a, **k: None)
(gts_fake, gts_client), (pds_fake, pds_client) = gts, pds
syndication._client_cache[GTS] = gts_client
syndication._client_cache["bluesky:https://pds.example:me.example"] = pds_client
def crosspost(**kwargs):
with app.test_request_context():
n = create_note("Plain words without links", published=True, **kwargs)
select_targets(n.id, ["gotosocial", "bluesky"])
process_due(limit=10)
return get_note(id=n.id).url, {r["target"]: r for r in get_note_syndications(n.id)}
first, rows = crosspost()
target = first if via == "permalink" else rows["gotosocial"]["remote_url"]
second, _ = crosspost(in_reply_to=target)
assert gts_fake.created[-1]["in_reply_to_id"] == rows["gotosocial"]["remote_id"]
assert not gts_fake.created[-1]["status"].startswith(("@", "↩"))
assert pds_fake.records[-1]["reply"]["parent"]["uri"] == rows["bluesky"]["remote_id"]