Compare commits

...
6 Commits
Author SHA1 Message Date
PhilandClaude Opus 5.5 2474628646 Merge branch 'fix/v2.6.3-link-cards' for v2.6.3 release
Build Container / build (push) Successful in 8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:25 -07:00
PhilandClaude Opus 5.5 0fe341161e feat: Crosspost cards show the note's own link (v2.6.3)
When a note links somewhere (an article, say), the first link gets the
preview card instead of the note (ADR-070). Bluesky cards are built from
the page's OG/Twitter/HTML title, description and image (thumb shrunk
under 1 MB); Threads previews the first link in the text. Markdown links
no longer lose their URL on Bluesky (link text becomes a facet) or Threads
("text (url)"); Threads keeps to Meta's 5-link limit. Page fetches are
public-address only, checked at every redirect, size-capped, and never
fail the post.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:24 -07:00
PhilandClaude Opus 5.5 a62077ab30 Merge branch 'fix/v2.6.2-threads-link-preview' for v2.6.2 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 fade75fd2d fix: Let Threads build the link preview itself (v2.6.2)
An explicit link_attachment for a URL Meta hasn't fetched yet fails at
threads_publish with "Invalid Link Attachment" (subcode 4279047), which
is every new note; it only went out on the retry. Tested live with fresh
URLs: with link_attachment it fails, without it it publishes first time
and Threads still sets link_attachment_url from the first link in the
text. Reposts likewise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 0dd59fbc18 Merge branch 'fix/v2.6.1-threads-publish-retry' for v2.6.1 release
Build Container / build (push) Successful in 6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:18:11 -06:00
PhilandClaude Opus 5.5 11e09a0d02 fix: Retry Threads publish errors, show full error detail (v2.6.1)
Meta's threads_publish returned a one-off HTTP 400 "Fatal" for a plain
text post; the same post published fine on retry. Publish-step errors
are now retryable unless they are auth/permission errors (codes 10, 190,
200); the retry's existing-post check prevents doubles. Errors carry the
Graph code, subcode, user message and fbtrace id. A short pause before
the first container status poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:18:11 -06:00
7 changed files with 460 additions and 35 deletions

No files matched your search

+25
View File
@@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.6.3] - 2026-11-01
### Changed
- **A note's own link gets the preview card** (ADR-070). When a note links somewhere (say, an article), the crosspost's card shows that page instead of the note. Bluesky cards are built from the page's title, description and image; Threads previews it from the text. Notes without links keep their own card
### Fixed
- Markdown links (`[text](url)`) lost their URL on Bluesky and Threads. On Bluesky the link text is now clickable; on Threads it's written "text (url)"
- Notes with many links failed on Threads (Meta allows 5 per post); extra links now keep just their text
## [2.6.2] - 2026-11-01
### Fixed
- New notes failed their first Threads crosspost with "Invalid Link Attachment" (Meta rejects an explicit link preview for a page it hasn't fetched yet) and only went out on the retry a minute later. StarPunk no longer sends the explicit link preview; Threads still shows one, built from the note's link in the text
## [2.6.1] - 2026-11-01
### Fixed
- Threads crossposts gave up after Meta's publish step returned a one-off `HTTP 400 Fatal`; publish errors are now retried (a retry checks for an existing post first, so nothing is posted twice). Only a bad token or missing permission stops retries
- Threads errors now include Meta's error code, subcode, user message and trace id, so the editor shows why a post failed
- Wait a moment before the first readiness check on a Threads post, since Meta can report it ready too early
## [2.6.0] - 2026-11-01
### Added
@@ -22,8 +22,11 @@ ADR-065 and ADR-066 crosspost notes to GoToSocial and Bluesky. Threads
token replaces the saved one. A file lock stops gunicorn workers from
refreshing at the same time. The file avoids a migration for one value.
3. **Posting.** Create a container, poll its status until `FINISHED`, then
`threads_publish`. Text-only notes are `TEXT` posts with the permalink as
`link_attachment`. One image is an `IMAGE` post and 2–4 images are a
`threads_publish`. Text-only notes are `TEXT` posts. Threads previews
the first link in the text (the permalink) by itself. An explicit
`link_attachment` is not sent, because Meta rejects one for a page it
hasn't fetched yet ("Invalid Link Attachment", subcode 4279047), which
is always the case for a new note (v2.6.2). One image is an `IMAGE` post and 2–4 images are a
`CAROUSEL`. Meta fetches images from their public `/media/` URL. Only
JPEG and PNG up to 8 MB are allowed, so GIF and WebP images are left off.
Alt text goes in `alt_text`.
@@ -0,0 +1,44 @@
# ADR-070: Links and link cards in crossposts
## Status
Accepted (v2.6.3, 2026-11-01)
## Context
Every crosspost carries the note's permalink. Notes often link elsewhere
too, usually to the article being discussed. Before this change:
- Bluesky and Threads get plain text, and a Markdown link `[text](url)`
lost its URL, so readers there couldn't follow it.
- The Bluesky card always showed the note, and the Threads preview showed
whichever link came first.
- Threads rejects posts with more than 5 different links.
## Decision
1. **The card shows what the note links to.** If the note has its own link
(a Markdown link, `<url>` or a bare URL), the first one gets the
preview card. A note without links gets a card for itself, as before.
Images still take the place of a card on Bluesky.
2. **Bluesky.** The card is built by StarPunk: it fetches the page's Open
Graph, Twitter or plain HTML title, description and image, and uploads
the image (shrunk to under 1 MB) as the thumbnail. If the page can't
be fetched, the card has just the URL. Markdown link text becomes a
clickable link facet, so no characters are spent on the URL.
3. **Threads.** It has no clickable link text, so Markdown links are
written "text (url)". Threads previews the first link in the text, which
is now the linked page. To stay within Meta's 5-link limit, after 4
different links in the body a Markdown link keeps only its text.
4. **GoToSocial** already gets Markdown, so its links are unchanged.
Receiving servers build their own cards.
5. **Fetching is careful.** Only http(s) URLs on public addresses are
fetched, and this is checked again at every redirect. At most 512 KB of
a page and 5 MB of an image are read. A failed fetch never fails the
post.
## Consequences
- One or two extra requests to the linked site when a note is crossposted
to Bluesky.
- Reposts (ADR-068) are unchanged: their card is the reposted page.
+2 -2
View File
@@ -464,5 +464,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.6.0"
__version_info__ = (2, 6, 0)
__version__ = "2.6.3"
__version_info__ = (2, 6, 3)
+229 -27
View File
@@ -1,5 +1,5 @@
"""
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066, v2.6.0 ADR-069)
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066, v2.6.0 ADR-069, v2.6.3 ADR-070)
Published notes are copied to syndication targets, and later edits and
deletes follow them. Three target kinds exist: a GoToSocial (or any
@@ -277,12 +277,28 @@ class BlueskyClient:
def _now() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None) -> dict:
def _link_card(self, url: str) -> dict:
"""External embed for a page, with its title, description and image when we can get them"""
card = page_card(self.http, url)
external = {"uri": url, "title": card["title"][:300], "description": card["description"][:1000]}
if card["image"]:
try:
data, mime_type, _ = _fetch_public(self.http, card["image"], CARD_IMAGE_LIMIT,
truncate=False)
data, mime_type, _, _ = _fit_image(data, mime_type.split(";")[0] or "image/jpeg",
BLUESKY_BLOB_LIMIT)
external["thumb"] = self.upload_blob(data, mime_type)
except Exception as e: # a card without a picture is fine
current_app.logger.info(f"No link card image for {url}: {e}")
return {"$type": "app.bsky.embed.external", "external": external}
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None,
anchors: List[tuple] = ()) -> dict:
if text is None:
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=plain_text(post.note.content))
body, anchors = plain_text_links(post.note.content, links="anchor")
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body)
record = {"$type": self.POST, "text": text, "createdAt": self._now()}
facets = bluesky_facets(text)
facets = bluesky_facets(text, anchors)
if facets:
record["facets"] = facets
if embed is not None:
@@ -290,6 +306,9 @@ class BlueskyClient:
elif post.media:
record["embed"] = {"$type": "app.bsky.embed.images",
"images": [self._image(m) for m in post.media]}
elif note_links(post.note.content):
# The note links somewhere: the card shows that page, not the note
record["embed"] = self._link_card(note_links(post.note.content)[0])
else:
record["embed"] = {"$type": "app.bsky.embed.external", "external": {
"uri": post.permalink,
@@ -321,7 +340,8 @@ class BlueskyClient:
def _react(self, post: "Post") -> dict:
"""Like, repost or quote a Bluesky post; fall back to a link post for reposts"""
comment = plain_text(post.note.content) if post.note.content.strip() else ""
comment, anchors = (plain_text_links(post.note.content, links="anchor")
if post.note.content.strip() else ("", []))
subject = self.resolve_post(post.target_url)
if subject and post.kind == "like":
uri = self._create("app.bsky.feed.like", {
@@ -335,7 +355,8 @@ class BlueskyClient:
text = comment if len(comment) <= post.max_chars else \
comment[:post.max_chars - 1].rsplit(" ", 1)[0].rstrip() + "…"
uri = self._create(self.POST, self._record(
post, text=text, embed={"$type": "app.bsky.embed.record", "record": subject}))
post, text=text, embed={"$type": "app.bsky.embed.record", "record": subject},
anchors=anchors))
return {"id": uri, "url": self._web_url(uri)}
if post.kind == "like":
return {"id": None, "url": None} # not a Bluesky post: nothing to like
@@ -389,11 +410,15 @@ class BlueskyClient:
json={"repo": repo, "collection": collection, "rkey": rkey})
def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
"""Image bytes no bigger than `limit`, re-encoded as JPEG if needed. Returns (data, mime, w, h)."""
def _fit_image(path, mime_type: str, limit: int) -> tuple:
"""
Image (a path, or bytes) no bigger than `limit`, re-encoded as JPEG if
needed. Returns (data, mime, w, h).
"""
from PIL import Image
data = Path(path).read_bytes()
data = path if isinstance(path, bytes) else Path(path).read_bytes()
name = "image" if isinstance(path, bytes) else Path(path).name
with Image.open(io.BytesIO(data)) as img:
width, height = img.size
if len(data) <= limit:
@@ -406,15 +431,96 @@ def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
copy.save(buf, format="JPEG", quality=quality, optimize=True)
if buf.tell() <= limit:
return buf.getvalue(), "image/jpeg", copy.size[0], copy.size[1]
raise SyndicationError(f"{Path(path).name}: image too large for Bluesky", permanent=True)
raise SyndicationError(f"{name}: image too large for Bluesky", permanent=True)
# --- Link cards ---------------------------------------------------------------
CARD_PAGE_LIMIT = 512 * 1024 # the <head> is near the start; read no more
CARD_IMAGE_LIMIT = 5 * 1024 * 1024
def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple:
"""
GET a public web URL, checking every redirect hop (no requests into the
LAN). Reads at most `limit` bytes: stops there if `truncate`, else fails.
Returns (data, content_type, final_url).
"""
from urllib.parse import urljoin, urlparse
from starpunk.micropub import _public_address
site = current_app.config.get("SITE_URL", "")
for _ in range(5):
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise SyndicationError(f"not a web URL: {url}", permanent=True)
if not current_app.config.get("TESTING") and not _public_address(parsed.hostname):
raise SyndicationError(f"not a public address: {url}", permanent=True)
try:
with http.stream("GET", url, follow_redirects=False, timeout=10.0,
headers={"User-Agent": f"StarPunk (+{site})"}) as response:
if response.is_redirect and "location" in response.headers:
url = urljoin(url, response.headers["location"])
continue
if response.status_code >= 400:
raise SyndicationError(f"{url}: HTTP {response.status_code}")
data = bytearray()
for chunk in response.iter_bytes():
data += chunk
if len(data) > limit:
if truncate:
break
raise SyndicationError(f"{url}: too large")
return bytes(data), response.headers.get("content-type", ""), url
except httpx.HTTPError as e:
raise SyndicationError(f"{url}: {type(e).__name__}")
raise SyndicationError(f"{url}: too many redirects")
def page_card(http: httpx.Client, url: str) -> dict:
"""
A page's title, description and image URL (Open Graph, Twitter or plain
HTML tags), for a link card. Missing parts come back empty; a page that
can't be fetched gives just the URL as the title.
"""
from urllib.parse import urljoin
from bs4 import BeautifulSoup
card = {"title": url, "description": "", "image": ""}
try:
data, content_type, final = _fetch_public(http, url, CARD_PAGE_LIMIT, truncate=True)
except SyndicationError as e:
current_app.logger.info(f"No link card details for {url}: {e}")
return card
if "html" not in content_type.lower():
return card
soup = BeautifulSoup(data, "html.parser")
def meta(*names):
for name in names:
tag = soup.find("meta", attrs={"property": name}) or soup.find("meta", attrs={"name": name})
if tag and (tag.get("content") or "").strip():
return " ".join(tag["content"].split())
return ""
title = meta("og:title", "twitter:title")
if not title and soup.title:
title = " ".join(soup.title.get_text(" ").split())
image = meta("og:image:secure_url", "og:image", "og:image:url", "twitter:image")
card.update(title=title or url,
description=meta("og:description", "twitter:description", "description"),
image=urljoin(final, image) if image else "")
return card
THREADS_API = "https://graph.threads.net/v1.0"
THREADS_IMAGE_TYPES = {"image/jpeg", "image/png"}
THREADS_IMAGE_LIMIT = 8_000_000
THREADS_MAX_LINKS = 5
THREADS_REFRESH_EVERY = 24 * 3600 # tokens last 60 days; Meta allows a refresh once one is a day old
# Graph API error codes that are temporary (rate limits, service hiccups)
_GRAPH_TRANSIENT = {1, 2, 4, 17, 32, 341, 613}
_GRAPH_AUTH = {10, 190, 200} # bad token or missing permission: retrying won't help
def threads_length(text: str) -> int:
@@ -536,13 +642,19 @@ class ThreadsClient:
if response.status_code >= 400 or error:
error = error if isinstance(error, dict) else {}
code = error.get("code")
detail = " / ".join(str(error[k]) for k in ("error_user_title", "error_user_msg")
if error.get(k))
message = (f"{method} {path}: HTTP {response.status_code} "
f"{error.get('message') or response.text[:200]}")
f"{error.get('message') or response.text[:200]}"
f" (code {code}, subcode {error.get('error_subcode')}"
f"{', ' + detail if detail else ''}, fbtrace {error.get('fbtrace_id')})")
if response.status_code == 404 or (code == 100 and error.get("error_subcode") == 33):
raise RemoteGone(message, permanent=True)
permanent = (400 <= response.status_code < 500 and response.status_code != 429
and code not in _GRAPH_TRANSIENT)
raise SyndicationError(message, permanent=permanent)
exc = SyndicationError(message, permanent=permanent)
exc.code = code
raise exc
return body
def _me(self) -> str:
@@ -555,6 +667,7 @@ class ThreadsClient:
def _wait(self, container_id: str) -> None:
"""Wait until Meta has processed a container (images take a few seconds)"""
delay = 1
self.sleep(2) # Meta can report a container ready a moment too early
for _ in range(8):
status = self._call("GET", container_id, fields="status,error_message")
state = status.get("status")
@@ -571,7 +684,9 @@ class ThreadsClient:
# --- Posts ---
def _text(self, post: "Post") -> str:
body = plain_text(post.note.content)
# Links written "text (url)"; the first one gets Threads' preview card.
# Meta allows 5 different links per post and the permalink is one.
body = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
limit = post.max_chars
while True:
text = compose_text(post.note, post.permalink, [], limit, body=body)
@@ -587,7 +702,7 @@ class ThreadsClient:
return self._call("POST", f"{uid}/threads", **params)["id"]
def _repost_text(self, post: "Post") -> str:
comment = plain_text(post.note.content)
comment = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
limit = post.max_chars
while True:
text = compose_repost_text(post, comment, limit)
@@ -598,9 +713,9 @@ class ThreadsClient:
def _container(self, uid: str, post: "Post") -> str:
if post.kind == "repost":
# No native repost of outside content: post "🔁 title url" with a link preview
# No native repost of outside content: post "🔁 title url"
return self._call("POST", f"{uid}/threads", media_type="TEXT",
text=self._repost_text(post), link_attachment=post.target_url)["id"]
text=self._repost_text(post))["id"]
common = {"text": self._text(post)}
topic = threads_topic(post.tags)
if topic:
@@ -609,8 +724,10 @@ class ThreadsClient:
images = [m for m in post.media if m["mime_type"] in THREADS_IMAGE_TYPES
and Path(m["path"]).stat().st_size <= THREADS_IMAGE_LIMIT]
if not images:
return self._call("POST", f"{uid}/threads", media_type="TEXT",
link_attachment=post.permalink, **common)["id"]
# No explicit link_attachment: Meta rejects one for a page it hasn't
# seen yet ("Invalid Link Attachment"), but previews the text's
# first link by itself without failing
return self._call("POST", f"{uid}/threads", media_type="TEXT", **common)["id"]
if len(images) == 1:
return self._image_container(uid, images[0], **common)
children = [self._image_container(uid, m, is_carousel_item="true") for m in images]
@@ -641,7 +758,14 @@ class ThreadsClient:
return existing
container = self._container(uid, post)
self._wait(container)
media_id = self._call("POST", f"{uid}/threads_publish", creation_id=container)["id"]
try:
media_id = self._call("POST", f"{uid}/threads_publish", creation_id=container)["id"]
except SyndicationError as e:
if getattr(e, "code", None) in _GRAPH_AUTH:
raise
# Meta's publish step fails now and then ("Fatal") and works on a
# later try; the retry checks for an existing post first
raise SyndicationError(str(e))
try:
url = self._call("GET", media_id, fields="permalink").get("permalink")
except SyndicationError:
@@ -885,14 +1009,75 @@ def compose_text(note, permalink: str, tags: List[str], max_chars: int,
_BLOCKS = ["p", "h1", "h2", "h3", "h4", "h5", "h6", "pre", "blockquote", "ul", "ol", "hr", "table"]
def plain_text(markdown_text: str) -> str:
"""Markdown as readable plain text (for targets without Markdown)"""
def _note_soup(markdown_text: str):
import markdown
from bs4 import BeautifulSoup
soup = BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
return BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
def _clean_url(url: str) -> str:
return url.rstrip(".,;:!?)\"'…")
def _is_web(url: str) -> bool:
return url.startswith(("http://", "https://"))
def note_links(markdown_text: str) -> List[str]:
"""Web links in a note, in reading order: Markdown links, <autolinks> and bare URLs"""
from bs4 import NavigableString
links = []
for node in _note_soup(markdown_text).descendants:
if getattr(node, "name", None) == "a" and _is_web(node.get("href", "")):
links.append(node["href"])
elif isinstance(node, NavigableString) and not node.find_parent("a"):
links.extend(_clean_url(m.group(0)) for m in _URL_RE.finditer(str(node)))
return list(dict.fromkeys(links))
def plain_text(markdown_text: str, links: str = "text", max_urls: Optional[int] = None) -> str:
"""Markdown as readable plain text (for targets without Markdown). See `plain_text_links`."""
return plain_text_links(markdown_text, links, max_urls)[0]
def plain_text_links(markdown_text: str, links: str = "text",
max_urls: Optional[int] = None) -> tuple:
"""
Markdown as plain text, and the links whose text stands in for a URL.
A Markdown link `[text](url)` becomes:
links="text" just its text
links="inline" "text (url)", for targets with no clickable link text;
after `max_urls` different URLs, just the text
links="anchor" its text, listed in the returned anchors as (text, url)
so the target can make that text clickable
A link whose text is the URL itself stays a bare URL.
"""
from bs4 import NavigableString
soup = _note_soup(markdown_text)
for img in soup.find_all("img"):
img.decompose()
anchors = []
seen = set()
for node in list(soup.descendants): # bare URLs count towards max_urls first
if isinstance(node, NavigableString) and not node.find_parent("a"):
seen.update(_clean_url(m.group(0)) for m in _URL_RE.finditer(str(node)))
for a in soup.find_all("a"):
href, label = a.get("href", ""), a.get_text()
if not _is_web(href):
continue
bare = re.sub(r"^https?://", "", href).rstrip("/")
if label.strip().rstrip("/") in (href.rstrip("/"), bare):
a.replace_with(href)
seen.add(href)
elif links == "inline" and (href in seen or max_urls is None or len(seen) < max_urls):
a.replace_with(f"{label} ({href})")
seen.add(href)
elif links == "anchor":
anchors.append((label, href))
for node in soup.find_all(string=True):
if not node.strip() and "\n" in node and node.parent.name != "pre":
node.extract() # layout newlines between tags; we add our own
@@ -905,24 +1090,41 @@ def plain_text(markdown_text: str) -> str:
block.append("\n\n")
text = soup.get_text()
text = re.sub(r"[ \t]+\n", "\n", text)
return re.sub(r"\n{3,}", "\n\n", text).strip()
return re.sub(r"\n{3,}", "\n\n", text).strip(), anchors
_URL_RE = re.compile(r"https?://[^\s<>\"]+")
_TAG_RE = re.compile(r"(?<![\w#])#(\w*[^\W\d]\w*)")
def bluesky_facets(text: str) -> List[dict]:
"""Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post"""
def bluesky_facets(text: str, anchors: List[tuple] = ()) -> List[dict]:
"""
Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post.
`anchors` are (text, url) pairs: that text, where it appears, links to url.
"""
def span(start: int, end: int) -> dict:
return {"byteStart": len(text[:start].encode()), "byteEnd": len(text[:end].encode())}
facets = []
for m in _URL_RE.finditer(text):
url = m.group(0).rstrip(".,;:!?)\"'…")
url = _clean_url(m.group(0))
facets.append({"index": span(m.start(), m.start() + len(url)),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
taken = [(m.start(), m.end()) for m in _URL_RE.finditer(text)]
pos = 0
for label, url in anchors:
start = text.find(label, pos) if label.strip() else -1
if start < 0:
continue # shortened away
end = start + len(label)
if any(a < end and start < b for a, b in taken):
continue
taken.append((start, end))
pos = end
facets.append({"index": span(start, end),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
urls = [(f["index"]["byteStart"], f["index"]["byteEnd"]) for f in facets]
facets.sort(key=lambda f: f["index"]["byteStart"])
for m in _TAG_RE.finditer(text):
index = span(m.start(), m.end())
if any(a <= index["byteStart"] < b for a, b in urls):
+120 -2
View File
@@ -92,8 +92,31 @@ def pds():
return FakePDS()
class FakeWeb:
"""Other people's pages, for link cards: url -> (status, headers, body)"""
def __init__(self):
self.pages = {}
self.fetched = []
def __call__(self, request):
url = str(request.url)
self.fetched.append(url)
status, headers, body = self.pages.get(url, (404, {}, b"nope"))
return httpx.Response(status, headers=headers, content=body)
@pytest.fixture
def app(tmp_path, pds):
def web():
return FakeWeb()
def routed(pds, web):
return httpx.MockTransport(
lambda r: pds(r) if r.url.host in ("bsky.example", "pds.example") else web(r))
@pytest.fixture
def app(tmp_path, pds, web):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
@@ -108,7 +131,7 @@ def app(tmp_path, pds):
"BLUESKY_SERVICE": SERVICE,
})
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"] = BlueskyClient(
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=httpx.MockTransport(pds))
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=routed(pds, web))
)
yield app
syndication._client_cache.clear()
@@ -258,6 +281,101 @@ class TestMedia:
assert image["aspectRatio"]["width"] > 0
ARTICLE = "https://news.example/story"
ARTICLE_HTML = b"""<html><head><title>Fallback title</title>
<meta property="og:title" content="A Great Story">
<meta property="og:description" content="What happened, in short.">
<meta property="og:image" content="/img/story.png">
</head><body>...</body></html>"""
def png_bytes():
buf = io.BytesIO()
Image.new("RGB", (60, 40), (10, 120, 200)).save(buf, format="PNG")
return buf.getvalue()
class TestLinkCards:
def test_note_link_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
web.pages["https://news.example/img/story.png"] = (200, {"content-type": "image/png"}, png_bytes())
note = publish(app, content=f"Read [this story]({ARTICLE}) now")
record = only_record(pds)
external = record["embed"]["external"]
assert external["uri"] == ARTICLE
assert external["title"] == "A Great Story"
assert external["description"] == "What happened, in short."
assert external["thumb"]["$type"] == "blob"
# The link text is clickable and the note's permalink is still in the text
permalink = f"https://blog.example.com/note/{note.slug}"
assert record["text"].startswith("Read this story now") and permalink in record["text"]
raw = record["text"].encode()
links = {raw[f["index"]["byteStart"]:f["index"]["byteEnd"]].decode(): f["features"][0]["uri"]
for f in record["facets"]}
assert links["this story"] == ARTICLE
assert links[permalink] == permalink
def test_bare_url_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
publish(app, content=f"See {ARTICLE}.")
assert only_record(pds)["embed"]["external"]["uri"] == ARTICLE
def test_unreachable_page_still_gets_a_card(self, app, pds, web):
note = publish(app, content=f"See {ARTICLE}")
external = only_record(pds)["embed"]["external"]
assert external == {"uri": ARTICLE, "title": ARTICLE, "description": ""}
assert row(app, note)["state"] == "synced"
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
monkeypatch.setattr("starpunk.micropub._public_address", lambda host: host != "internal.example")
app.config["TESTING"] = False
try:
publish(app, content=f"See {ARTICLE}")
finally:
app.config["TESTING"] = True
assert "http://internal.example/admin" not in web.fetched
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE
def test_images_win_over_link_card(self, app, pds, web):
from starpunk.media import attach_media_to_note, save_media
with app.test_request_context():
note = create_note(content=f"Look {ARTICLE}", published=True)
media = save_media(png_bytes(), "a.png")
attach_media_to_note(note.id, [media["id"]], ["pic"])
select_targets(note.id, ["bluesky"])
process_due()
assert only_record(pds)["embed"]["$type"] == "app.bsky.embed.images"
assert web.fetched == []
class TestLinkText:
def test_note_links_in_reading_order(self):
from starpunk.syndication import note_links
md = "a [x](https://one.example) b https://two.example/p. c <https://three.example> [rel](/local)"
assert note_links(md) == ["https://one.example", "https://two.example/p", "https://three.example"]
def test_link_modes(self):
from starpunk.syndication import plain_text_links
md = "[a story](https://s.example) and [https://u.example](https://u.example)"
assert plain_text_links(md)[0] == "a story and https://u.example"
assert plain_text_links(md, "inline")[0] == "a story (https://s.example) and https://u.example"
assert plain_text_links(md, "anchor") == ("a story and https://u.example",
[("a story", "https://s.example")])
def test_inline_url_budget(self):
from starpunk.syndication import plain_text_links
md = "https://bare.example then [one](https://1.example) [two](https://2.example) [one again](https://1.example)"
text = plain_text_links(md, "inline", max_urls=2)[0]
assert text == "https://bare.example then one (https://1.example) two one again (https://1.example)"
def test_anchor_facet_byte_offsets(self):
text = "Café ☕ this story and more"
(facet,) = bluesky_facets(text, [("this story", "https://s.example")])
raw = text.encode()
assert raw[facet["index"]["byteStart"]:facet["index"]["byteEnd"]] == b"this story"
def test_both_targets_listed_for_micropub(tmp_path):
data = tmp_path / "d"
app = create_app({
+35 -2
View File
@@ -40,6 +40,7 @@ class FakeThreads:
self.pending_polls = 0
self.lose_next_publish = False
self.fail_next = None
self.fail_next_publish = False
def _json(self, code, body):
return httpx.Response(code, json=body)
@@ -83,6 +84,9 @@ class FakeThreads:
return self._json(200, {"data": [
{"id": pid, "text": p["text"], "permalink": f"https://www.threads.com/@phil/post/{pid}"}
for pid, p in self.posts.items()]})
if path == f"{UID}/threads_publish" and self.fail_next_publish:
self.fail_next_publish = False
return self._error(400, "Fatal", 24, 4279009)
if path == f"{UID}/threads_publish":
container = self.containers[params["creation_id"]]
pid = f"p{len(self.posts) + 1}"
@@ -176,7 +180,7 @@ class TestLifecycle:
permalink = f"https://blog.example.com/note/{note.slug}"
assert post["media_type"] == "TEXT"
assert post["text"] == f"Hello Threads\n\n{permalink}"
assert post["link_attachment"] == permalink
assert "link_attachment" not in post # Threads previews the first link itself
assert post["topic_tag"] == "IndieWeb"
r = row(app, note)
assert r["state"] == "synced"
@@ -237,6 +241,19 @@ class TestErrors:
assert len(api.posts) == 1
assert row(app, note)["state"] == "synced"
def test_publish_fatal_is_retried(self, app, api):
api.fail_next_publish = True
note = publish(app)
r = row(app, note)
assert r["state"] == "pending" and "Fatal" in r["last_error"] and "subcode 4279009" in r["last_error"]
with app.test_request_context():
from starpunk.database import get_db
get_db(app).execute("UPDATE syndications SET next_attempt_at = datetime('now')")
get_db(app).commit()
process_due()
assert len(api.posts) == 1
assert row(app, note)["state"] == "synced"
def test_bad_token_fails_permanently(self, app, api):
api.token = "other"
note = publish(app)
@@ -251,6 +268,22 @@ class TestErrors:
assert row(app, note)["state"] == "pending"
class TestLinks:
def test_markdown_link_comes_first_for_the_preview(self, app, api):
note = publish(app, content="Read [this story](https://news.example/story) now")
text = only_post(api)["text"]
assert text.startswith("Read this story (https://news.example/story) now")
assert text.index("https://news.example/story") < text.index(f"/note/{note.slug}")
def test_stays_within_five_links(self, app, api):
import re
links = " ".join(f"[l{i}](https://l{i}.example)" for i in range(7))
publish(app, content=f"Lots: {links}")
text = only_post(api)["text"]
assert len(set(re.findall(r"https?://[^\s)]+", text))) == 5
assert "l6" in text # the extra links keep their text
class TestMedia:
def _note_with(self, app, images):
from starpunk.media import attach_media_to_note, save_media
@@ -374,7 +407,7 @@ class TestLikesReposts:
self._post(app, monkeypatch, repost_of=self.PAGE)
post = only_post(api)
assert post["text"].startswith("🔁") and self.PAGE in post["text"]
assert post["link_attachment"] == self.PAGE
assert "link_attachment" not in post
def test_repost_with_comment_keeps_comment(self, app, api, monkeypatch):
note = self._post(app, monkeypatch, content="So **true**", repost_of=self.PAGE)