Compare commits

...
10 Commits
Author SHA1 Message Date
Phil 1ce88b38a5 Merge branch 'fix/v2.6.5-micropub-media' for v2.6.5 release
Build Container / build (push) Successful in 6s
2026-11-01 10:53:37 -07:00
PhilandClaude Opus 5.5 4ffe36f9d3 fix: Photos without alt text use the post's text (v2.6.5)
Uncaptioned photos rendered alt="Image", which mf2 parsers report as
the photo's alt text. They now take the post's text (excerpt, up to 150
characters), falling back to "Image" only for text-free posts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 10:44:54 -07:00
PhilandClaude Opus 5.5 a551770c76 fix: Replies and photo posts read as the right post type (v2.6.5)
Checked rendered pages with mf2py against W3C Post Type Discovery:

- Note pages put u-in-reply-to inside a bare h-cite, making it the
  h-cite's property, so replies parsed as notes (or photo posts). The
  reply context is now the entry's u-in-reply-to h-cite.
- Home page previews had no reply markup; they now show "In reply to"
  with u-in-reply-to.
- Previews always emit e-content, so a text-free photo post can't get
  an implied name.
- Text-free page titles follow PTD order (Repost, Like, Reply, Photo).

tests/test_post_types.py runs PTD on parsed note pages and the home page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 10:06:44 -07:00
PhilandClaude Opus 5.5 d0bfc80c2e fix: Micropub photo posts and failed creates (v2.6.5)
- A create that failed after the note was written (e.g. the same photo
  twice) answered 500 but left the post published without photos.
  Photos are now resolved first, and a failed note is removed.
- Pooled connections roll back unfinished writes, so one failure no
  longer locks the database for the next ~10 s.
- The same photo twice in one post is shown once.
- Photo posts no longer need text (Micropub doesn't require content,
  and q=config already advertised a photo-only post type). Updates may
  clear the text while photos remain. Crossposts start with the link;
  the page is titled "Photo".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 09:50:30 -07:00
PhilandClaude Opus 5.5 d16463ef8a Merge branch 'fix/v2.6.4-tag-order' for v2.6.4 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:59:19 -07:00
PhilandClaude Opus 5.5 4b56f7cd4b fix: Tags keep the order they were entered in (v2.6.4)
add_tags_to_note() rewrites note_tags in entry order on every save, so
note_tags.id already records it; get_note_tags(entered_order=True) reads
it back. Crossposts (Threads topic, which hashtags fit on Bluesky), the
editor and Micropub q=source/update use it, so saving no longer
reorders tags alphabetically. Public pages stay alphabetical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:59:18 -07:00
PhilandClaude Opus 5.5 2474628646 Merge branch 'fix/v2.6.3-link-cards' for v2.6.3 release
Build Container / build (push) Successful in 8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:25 -07:00
PhilandClaude Opus 5.5 0fe341161e feat: Crosspost cards show the note's own link (v2.6.3)
When a note links somewhere (an article, say), the first link gets the
preview card instead of the note (ADR-070). Bluesky cards are built from
the page's OG/Twitter/HTML title, description and image (thumb shrunk
under 1 MB); Threads previews the first link in the text. Markdown links
no longer lose their URL on Bluesky (link text becomes a facet) or Threads
("text (url)"); Threads keeps to Meta's 5-link limit. Page fetches are
public-address only, checked at every redirect, size-capped, and never
fail the post.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:12:24 -07:00
PhilandClaude Opus 5.5 a62077ab30 Merge branch 'fix/v2.6.2-threads-link-preview' for v2.6.2 release
Build Container / build (push) Successful in 7s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
PhilandClaude Opus 5.5 fade75fd2d fix: Let Threads build the link preview itself (v2.6.2)
An explicit link_attachment for a URL Meta hasn't fetched yet fails at
threads_publish with "Invalid Link Attachment" (subcode 4279047), which
is every new note; it only went out on the retry. Tested live with fresh
URLs: with link_attachment it fails, without it it publishes first time
and Threads still sets link_attachment_url from the first link in the
text. Reposts likewise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 21:38:00 -06:00
21 changed files with 932 additions and 83 deletions

No files matched your search

+39
View File
@@ -7,6 +7,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.6.5] - 2026-11-01
### Fixed
- A Micropub post that failed partway (for example, the same photo listed twice) answered "Failed to create post" but stayed published without its photos, so a client retrying would post it twice. Photos are now looked up before the post is made, and a post that fails after that is removed again
- After such a failure, every request failed with "database is locked" for about 10 seconds: database connections now roll back unfinished writes when they're returned to the pool
- The same photo listed twice in one post is shown once instead of failing the post
- Replies didn't read as replies to IndieWeb readers and webmention receivers: on note pages `u-in-reply-to` sat inside an unrelated `h-cite`, and the home page didn't mark replies at all, so [Post Type Discovery](https://www.w3.org/TR/post-type-discovery/) saw notes (or photo posts). The reply context is now the entry's `u-in-reply-to h-cite`, and home page entries show "↩ In reply to"
- Photos without alt text used "Image" as their description, which IndieWeb readers passed on as if you'd written it. They now use the post's text (up to 150 characters), and fall back to "Image" only when the post has no text
- Home page entries always have an `e-content`, so a photo post without text never gets an implied name (which readers would show as its caption)
### Changed
- Photo posts no longer need text. Micropub doesn't require `content`, and `q=config` already offered a Photo post type with only `photo`. A Micropub update may also clear the text of a post that keeps its photos. Crossposts of a photo post without text start with its link, and its page is titled "Photo"
## [2.6.4] - 2026-11-01
### Fixed
- Tags keep the order you entered them in. The first tag is the note's main tag: it's the Threads topic, and Bluesky keeps the first tags when space runs out. Before, both used the alphabetically first tag. Public pages still list tags alphabetically
- Opening a note in the editor or updating it via Micropub reordered its tags alphabetically on save; the editor and Micropub `q=source` now show, and keep, the entered order
## [2.6.3] - 2026-11-01
### Changed
- **A note's own link gets the preview card** (ADR-070). When a note links somewhere (say, an article), the crosspost's card shows that page instead of the note. Bluesky cards are built from the page's title, description and image; Threads previews it from the text. Notes without links keep their own card
### Fixed
- Markdown links (`[text](url)`) lost their URL on Bluesky and Threads. On Bluesky the link text is now clickable; on Threads it's written "text (url)"
- Notes with many links failed on Threads (Meta allows 5 per post); extra links now keep just their text
## [2.6.2] - 2026-11-01
### Fixed
- New notes failed their first Threads crosspost with "Invalid Link Attachment" (Meta rejects an explicit link preview for a page it hasn't fetched yet) and only went out on the retry a minute later. StarPunk no longer sends the explicit link preview; Threads still shows one, built from the note's link in the text
## [2.6.1] - 2026-11-01
### Fixed
@@ -22,8 +22,11 @@ ADR-065 and ADR-066 crosspost notes to GoToSocial and Bluesky. Threads
token replaces the saved one. A file lock stops gunicorn workers from
refreshing at the same time. The file avoids a migration for one value.
3. **Posting.** Create a container, poll its status until `FINISHED`, then
`threads_publish`. Text-only notes are `TEXT` posts with the permalink as
`link_attachment`. One image is an `IMAGE` post and 2–4 images are a
`threads_publish`. Text-only notes are `TEXT` posts. Threads previews
the first link in the text (the permalink) by itself. An explicit
`link_attachment` is not sent, because Meta rejects one for a page it
hasn't fetched yet ("Invalid Link Attachment", subcode 4279047), which
is always the case for a new note (v2.6.2). One image is an `IMAGE` post and 2–4 images are a
`CAROUSEL`. Meta fetches images from their public `/media/` URL. Only
JPEG and PNG up to 8 MB are allowed, so GIF and WebP images are left off.
Alt text goes in `alt_text`.
@@ -0,0 +1,44 @@
# ADR-070: Links and link cards in crossposts
## Status
Accepted (v2.6.3, 2026-11-01)
## Context
Every crosspost carries the note's permalink. Notes often link elsewhere
too, usually to the article being discussed. Before this change:
- Bluesky and Threads get plain text, and a Markdown link `[text](url)`
lost its URL, so readers there couldn't follow it.
- The Bluesky card always showed the note, and the Threads preview showed
whichever link came first.
- Threads rejects posts with more than 5 different links.
## Decision
1. **The card shows what the note links to.** If the note has its own link
(a Markdown link, `<url>` or a bare URL), the first one gets the
preview card. A note without links gets a card for itself, as before.
Images still take the place of a card on Bluesky.
2. **Bluesky.** The card is built by StarPunk: it fetches the page's Open
Graph, Twitter or plain HTML title, description and image, and uploads
the image (shrunk to under 1 MB) as the thumbnail. If the page can't
be fetched, the card has just the URL. Markdown link text becomes a
clickable link facet, so no characters are spent on the URL.
3. **Threads.** It has no clickable link text, so Markdown links are
written "text (url)". Threads previews the first link in the text, which
is now the linked page. To stay within Meta's 5-link limit, after 4
different links in the body a Markdown link keeps only its text.
4. **GoToSocial** already gets Markdown, so its links are unchanged.
Receiving servers build their own cards.
5. **Fetching is careful.** Only http(s) URLs on public addresses are
fetched, and this is checked again at every redirect. At most 512 KB of
a page and 5 MB of an image are read. A failed fetch never fails the
post.
## Consequences
- One or two extra requests to the linked site when a note is crossposted
to Bluesky.
- Reposts (ADR-068) are unchanged: their card is the reposted page.
@@ -0,0 +1,89 @@
# v2.6.5: Micropub media fixes
Found while adding media uploads to tuipub, a Micropub client written to the
spec rather than to StarPunk.
## Problems
1. **Failed creates left a post behind.** `handle_create` made the note, then
attached photos. Listing one photo twice broke the `note_media` unique key,
so the client got a 500 while the note stayed published without photos.
A retry would post it twice.
2. **Then the database stayed locked.** `attach_media_to_note` didn't roll
back, and `ConnectionPool.return_connection` pooled the connection with
its write transaction still open. Other requests got "database is locked"
until the 10 s timeout.
3. **Photo posts needed text.** The Micropub spec doesn't require `content`,
and `q=config` advertised a `photo` post type with only `photo`, yet
creates without content got 400 "Content is required".
## Changes
- `database/pool.py`: `return_connection` rolls back an open transaction
(or closes the connection if that fails).
- `media.py`: `attach_media_to_note` rolls back on error.
- `micropub.py`:
- `handle_create` resolves photos (by media URL, downloading remote ones)
before `create_note`. If attaching or crosspost selection fails, it
hard-deletes the note before answering 500.
- `_resolve_photos` skips repeated media.
- Content is optional when photos resolve. Content-less posts whose photos
all fail still get a 400 from `create_note`.
- `handle_update` allows `replace: {content: [""]}` and `delete: ["content"]`
while the note keeps photos (or is a like or repost). Otherwise it answers
400 "A post needs text or a photo".
- `notes.py`: `create_note` and `update_note` take `has_media`, so
empty content is allowed for photo posts.
- `syndication.py`: `compose_text` drops the leading blank line when there's
no body.
## Post types (IndieWeb standards check)
Post Type Discovery (W3C Note) has no separate "photo" type to store. A post
with a valid `photo` is a photo post, unless it's first a repost, like or
reply (step order: repost, like, reply, video, photo). Otherwise it's a note,
or an article when its `name` isn't a prefix of its content. h-entry treats
`u-photo` as the primary content, with `name` as a caption. mf2 implies a
`name` only when an entry has no other `p-*`/`e-*` properties and no nested
microformats.
StarPunk already stores photo posts as notes with media and renders
`u-photo` outside `e-content`, which is what the standards ask for. Checking
the rendered pages with mf2py found:
- **Replies weren't replies.** In `note.html`, `u-in-reply-to` was inside a
bare `h-cite`, so it was the h-cite's property, not the entry's.
`note_preview.html` had no reply markup at all. Fixed: the blockquote is
`u-in-reply-to h-cite` (with `u-url` inside), and previews show
"↩ In reply to" with `u-in-reply-to`.
- **Implied names were possible.** Previews left out `e-content` for posts
without text. They worked only because the author h-card blocked an
implied name. Previews now always emit `e-content`.
- Page title for text-free posts follows PTD order: Repost, Like, Reply,
Photo.
Photos without a caption take the post's text as alt (`partials/media.html`
`fallback_alt`) instead of the made-up "Image". The Micropub `q=source`
still reports the stored caption (`""`).
`tests/test_post_types.py` parses note pages and the home page with mf2py and
runs PTD for notes, articles, photo posts (no text, caption, title), replies
(with and without a photo) and reposts.
## Not changed (noted for later)
- Text-free photos without alt text still render `alt="Image"`. Captioned
photos use their caption, and others now use the post's text (`display_media`
`fallback_alt`, the excerpt cut to 150 characters).
- More than 4 photos are still cut to 4 without an error (ADR-057).
- `video` and `audio` are ignored. This is allowed by the spec, and they
aren't in `q=config` post-types.
- Media endpoint errors for non-images expose Python text
("cannot identify image file <_io.BytesIO …>").
## Tests
`tests/test_micropub_v265.py`. The full suite runs with
`SESSION_SECRET=test`. 8 tests in `test_media_upload.py` and
`test_migration_race_condition.py` already fail on main, and still fail here.
+2 -2
View File
@@ -464,5 +464,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.6.1"
__version_info__ = (2, 6, 1)
__version__ = "2.6.5"
__version_info__ = (2, 6, 4)
+10
View File
@@ -107,6 +107,16 @@ class ConnectionPool:
if not conn:
return
# A request that failed mid-write must not hand an open transaction
# (and its write lock) to the next request (v2.6.5)
if conn.in_transaction:
try:
conn.rollback()
except Exception:
conn.close()
self._stats['connections_closed'] += 1
return
with self._lock:
if len(self._pool) < self.pool_size:
# Return to pool
+16 -12
View File
@@ -868,20 +868,24 @@ def attach_media_to_note(note_id: int, media_ids: List[int], captions: List[str]
db = get_db(current_app)
# Delete existing associations (for edit case)
db.execute("DELETE FROM note_media WHERE note_id = ?", (note_id,))
try:
# Delete existing associations (for edit case)
db.execute("DELETE FROM note_media WHERE note_id = ?", (note_id,))
# Insert new associations
for i, (media_id, caption) in enumerate(zip(media_ids, captions)):
db.execute(
"""
INSERT INTO note_media (note_id, media_id, display_order, caption)
VALUES (?, ?, ?, ?)
""",
(note_id, media_id, i, caption or None)
)
# Insert new associations
for i, (media_id, caption) in enumerate(zip(media_ids, captions)):
db.execute(
"""
INSERT INTO note_media (note_id, media_id, display_order, caption)
VALUES (?, ?, ?, ?)
""",
(note_id, media_id, i, caption or None)
)
db.commit()
db.commit()
except Exception:
db.rollback()
raise
def get_note_media(note_id: int) -> List[Dict]:
+50 -15
View File
@@ -195,6 +195,16 @@ def extract_content(properties: dict) -> str:
return content.strip()
def _has_text(content) -> bool:
"""Whether a content property has any text. Likes, reposts (v2.5.0) and
photo posts (v2.6.5) may come without."""
values = content if isinstance(content, list) else [content]
return any(
(c.strip() if isinstance(c, str) else (c.get("text") or c.get("html") or "").strip())
for c in values if c
)
def extract_title(properties: dict) -> Optional[str]:
"""
Extract the explicit title (Micropub 'name' property)
@@ -420,7 +430,8 @@ def _resolve_photos(photos: list[dict]) -> tuple[list[int], list[str]]:
if len(media_ids) == MAX_PHOTOS:
break
media_id = photo.get("media_id") or _media_id_for_url(photo["url"])
if media_id:
# The same photo twice is shown once (v2.6.5)
if media_id and media_id not in media_ids:
media_ids.append(media_id)
captions.append(photo.get("alt", ""))
return media_ids, captions
@@ -525,24 +536,29 @@ def handle_create(data: dict, token_info: dict, files=None):
repost_of = extract_url_property(properties, "repost-of")
if like_of and repost_of:
raise MicropubValidationError("A post can't be both a like and a repost")
# Likes and reposts may come without a comment (v2.5.0)
has_content = any(
(c.strip() if isinstance(c, str) else (c.get("text") or c.get("html") or "").strip())
for c in properties.get("content", []) if c
has_content = _has_text(properties.get("content"))
photos = extract_photos(properties) + _save_uploads(files)
# Photo posts may have no text (v2.6.5): Micropub doesn't require content
content = (
extract_content(properties)
if has_content or not (like_of or repost_of or photos)
else ""
)
content = extract_content(properties) if has_content or not (like_of or repost_of) else ""
title = extract_title(properties)
tags = extract_tags(properties)
published_date = extract_published_date(properties)
published = extract_post_status(properties)
in_reply_to = extract_in_reply_to(properties)
photos = extract_photos(properties) + _save_uploads(files)
# Look up and download photos before the note exists, so a bad photo
# can't leave a half-made post behind (v2.6.5)
media_ids, captions = _resolve_photos(photos)
except MicropubError:
raise
except Exception as e:
current_app.logger.error(f"Property extraction failed: {e}")
raise MicropubValidationError(f"Failed to parse request: {str(e)}")
note = None
try:
note = create_note(
content=add_title_heading(content, title),
@@ -553,10 +569,12 @@ def handle_create(data: dict, token_info: dict, files=None):
in_reply_to=in_reply_to,
like_of=like_of,
repost_of=repost_of,
has_media=bool(media_ids),
)
if photos:
_attach_photos_to_note(note.id, photos)
if media_ids:
from starpunk.media import attach_media_to_note
attach_media_to_note(note.id, media_ids, captions)
# Crosspost after photos are attached (v2.1.0); drafts wait until published
from starpunk.syndication import get_targets, resolve_targets, select_targets
@@ -573,6 +591,13 @@ def handle_create(data: dict, token_info: dict, files=None):
raise
except Exception as e:
current_app.logger.error(f"Failed to create note via Micropub: {e}")
# The client is told it failed, so don't leave the post up (v2.6.5)
if note is not None:
try:
from starpunk.notes import delete_note
delete_note(id=note.id, soft=False)
except Exception as cleanup:
current_app.logger.error(f"Could not remove failed note {note.slug}: {cleanup}")
raise MicropubError("server_error", "Failed to create post", status_code=500)
@@ -603,8 +628,10 @@ def handle_update(data: dict, token_info: dict):
if set(add) - {"category", "photo"}:
raise MicropubValidationError("Only category and photo can be added to")
from starpunk.tags import get_note_tags
title, body = split_title(note.content)
tags = [t["display_name"] for t in note.tags]
tags = [t["display_name"] for t in get_note_tags(note.id, entered_order=True)]
photos = _note_photos(note)
changes = {}
@@ -614,7 +641,7 @@ def handle_update(data: dict, token_info: dict):
if prop == "name":
title = None
elif prop == "content":
raise MicropubValidationError("content can't be deleted")
body = ""
elif prop == "category":
tags = []
elif prop == "photo":
@@ -635,7 +662,8 @@ def handle_update(data: dict, token_info: dict):
raise MicropubValidationError(f"Can't delete individual values of {prop}")
if "content" in replace:
body = extract_content(replace)
# Empty text is fine if the note keeps its photos; update_note checks (v2.6.5)
body = extract_content(replace) if _has_text(replace["content"]) else ""
if "name" in replace:
title = extract_title(replace)
if "category" in replace:
@@ -653,6 +681,10 @@ def handle_update(data: dict, token_info: dict):
if "photo" in add:
photos += extract_photos(add)
# Text may go only if photos stay (or it's a like or repost) (v2.6.5)
if not (body or "").strip() and not (photos or note.like_of or note.repost_of):
raise MicropubValidationError("A post needs text or a photo")
new_content = add_title_heading(body, title)
if new_content != note.content:
changes["content"] = new_content
@@ -661,7 +693,7 @@ def handle_update(data: dict, token_info: dict):
try:
if changes:
update_note(id=note.id, **changes)
update_note(id=note.id, has_media=bool(photos), **changes)
if "photo" in touched:
_attach_photos_to_note(note.id, photos)
note_changed(note.id)
@@ -717,8 +749,11 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
}
if title:
props["name"] = [title]
if note.tags:
props["category"] = [tag["display_name"] for tag in note.tags]
from starpunk.tags import get_note_tags
tags = get_note_tags(note.id, entered_order=True)
if tags:
props["category"] = [tag["display_name"] for tag in tags]
if getattr(note, "in_reply_to", None):
props["in-reply-to"] = [note.in_reply_to]
if getattr(note, "like_of", None):
+6 -4
View File
@@ -142,6 +142,7 @@ def create_note(
in_reply_to: Optional[str] = None,
like_of: Optional[str] = None,
repost_of: Optional[str] = None,
has_media: bool = False,
) -> Note:
"""
Create a new note
@@ -205,11 +206,11 @@ def create_note(
- created_at and updated_at set to same value initially
"""
# 1. VALIDATION (before any changes)
# Likes and reposts may have no comment (v2.5.0)
# Likes and reposts may have no comment (v2.5.0), photo posts no text (v2.6.5)
if like_of and repost_of:
raise InvalidNoteDataError("like_of", like_of, "A note can't be both a like and a repost")
content = content or ""
if not content.strip() and not (like_of or repost_of):
if not content.strip() and not (like_of or repost_of or has_media):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
@@ -604,7 +605,8 @@ def update_note(
content: Optional[str] = None,
published: Optional[bool] = None,
tags: Optional[list[str]] = None,
in_reply_to: Optional[str] = None
in_reply_to: Optional[str] = None,
has_media: bool = False,
) -> Note:
"""
Update a note's content and/or published status
@@ -690,7 +692,7 @@ def update_note(
identifier = slug if slug is not None else id
raise NoteNotFoundError(identifier)
if empty_content and not (existing_note.like_of or existing_note.repost_of):
if empty_content and not (existing_note.like_of or existing_note.repost_of or has_media):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
+2 -2
View File
@@ -226,8 +226,8 @@ def edit_note_form(note_id: int):
flash("Note not found", "error")
return redirect(url_for("admin.dashboard")), 404
# Pre-load tags for the edit form (v1.3.0 Phase 3)
tags = get_note_tags(note.id)
# Pre-load tags for the edit form (v1.3.0 Phase 3), as entered so saving keeps the order
tags = get_note_tags(note.id, entered_order=True)
object.__setattr__(note, '_cached_tags', tags)
from starpunk.syndication import get_note_syndications, get_targets
+214 -26
View File
@@ -1,5 +1,5 @@
"""
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066, v2.6.0 ADR-069)
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066, v2.6.0 ADR-069, v2.6.3 ADR-070)
Published notes are copied to syndication targets, and later edits and
deletes follow them. Three target kinds exist: a GoToSocial (or any
@@ -277,12 +277,28 @@ class BlueskyClient:
def _now() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None) -> dict:
def _link_card(self, url: str) -> dict:
"""External embed for a page, with its title, description and image when we can get them"""
card = page_card(self.http, url)
external = {"uri": url, "title": card["title"][:300], "description": card["description"][:1000]}
if card["image"]:
try:
data, mime_type, _ = _fetch_public(self.http, card["image"], CARD_IMAGE_LIMIT,
truncate=False)
data, mime_type, _, _ = _fit_image(data, mime_type.split(";")[0] or "image/jpeg",
BLUESKY_BLOB_LIMIT)
external["thumb"] = self.upload_blob(data, mime_type)
except Exception as e: # a card without a picture is fine
current_app.logger.info(f"No link card image for {url}: {e}")
return {"$type": "app.bsky.embed.external", "external": external}
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None,
anchors: List[tuple] = ()) -> dict:
if text is None:
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=plain_text(post.note.content))
body, anchors = plain_text_links(post.note.content, links="anchor")
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body)
record = {"$type": self.POST, "text": text, "createdAt": self._now()}
facets = bluesky_facets(text)
facets = bluesky_facets(text, anchors)
if facets:
record["facets"] = facets
if embed is not None:
@@ -290,6 +306,9 @@ class BlueskyClient:
elif post.media:
record["embed"] = {"$type": "app.bsky.embed.images",
"images": [self._image(m) for m in post.media]}
elif note_links(post.note.content):
# The note links somewhere: the card shows that page, not the note
record["embed"] = self._link_card(note_links(post.note.content)[0])
else:
record["embed"] = {"$type": "app.bsky.embed.external", "external": {
"uri": post.permalink,
@@ -321,7 +340,8 @@ class BlueskyClient:
def _react(self, post: "Post") -> dict:
"""Like, repost or quote a Bluesky post; fall back to a link post for reposts"""
comment = plain_text(post.note.content) if post.note.content.strip() else ""
comment, anchors = (plain_text_links(post.note.content, links="anchor")
if post.note.content.strip() else ("", []))
subject = self.resolve_post(post.target_url)
if subject and post.kind == "like":
uri = self._create("app.bsky.feed.like", {
@@ -335,7 +355,8 @@ class BlueskyClient:
text = comment if len(comment) <= post.max_chars else \
comment[:post.max_chars - 1].rsplit(" ", 1)[0].rstrip() + "…"
uri = self._create(self.POST, self._record(
post, text=text, embed={"$type": "app.bsky.embed.record", "record": subject}))
post, text=text, embed={"$type": "app.bsky.embed.record", "record": subject},
anchors=anchors))
return {"id": uri, "url": self._web_url(uri)}
if post.kind == "like":
return {"id": None, "url": None} # not a Bluesky post: nothing to like
@@ -389,11 +410,15 @@ class BlueskyClient:
json={"repo": repo, "collection": collection, "rkey": rkey})
def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
"""Image bytes no bigger than `limit`, re-encoded as JPEG if needed. Returns (data, mime, w, h)."""
def _fit_image(path, mime_type: str, limit: int) -> tuple:
"""
Image (a path, or bytes) no bigger than `limit`, re-encoded as JPEG if
needed. Returns (data, mime, w, h).
"""
from PIL import Image
data = Path(path).read_bytes()
data = path if isinstance(path, bytes) else Path(path).read_bytes()
name = "image" if isinstance(path, bytes) else Path(path).name
with Image.open(io.BytesIO(data)) as img:
width, height = img.size
if len(data) <= limit:
@@ -406,12 +431,92 @@ def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
copy.save(buf, format="JPEG", quality=quality, optimize=True)
if buf.tell() <= limit:
return buf.getvalue(), "image/jpeg", copy.size[0], copy.size[1]
raise SyndicationError(f"{Path(path).name}: image too large for Bluesky", permanent=True)
raise SyndicationError(f"{name}: image too large for Bluesky", permanent=True)
# --- Link cards ---------------------------------------------------------------
CARD_PAGE_LIMIT = 512 * 1024 # the <head> is near the start; read no more
CARD_IMAGE_LIMIT = 5 * 1024 * 1024
def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple:
"""
GET a public web URL, checking every redirect hop (no requests into the
LAN). Reads at most `limit` bytes: stops there if `truncate`, else fails.
Returns (data, content_type, final_url).
"""
from urllib.parse import urljoin, urlparse
from starpunk.micropub import _public_address
site = current_app.config.get("SITE_URL", "")
for _ in range(5):
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise SyndicationError(f"not a web URL: {url}", permanent=True)
if not current_app.config.get("TESTING") and not _public_address(parsed.hostname):
raise SyndicationError(f"not a public address: {url}", permanent=True)
try:
with http.stream("GET", url, follow_redirects=False, timeout=10.0,
headers={"User-Agent": f"StarPunk (+{site})"}) as response:
if response.is_redirect and "location" in response.headers:
url = urljoin(url, response.headers["location"])
continue
if response.status_code >= 400:
raise SyndicationError(f"{url}: HTTP {response.status_code}")
data = bytearray()
for chunk in response.iter_bytes():
data += chunk
if len(data) > limit:
if truncate:
break
raise SyndicationError(f"{url}: too large")
return bytes(data), response.headers.get("content-type", ""), url
except httpx.HTTPError as e:
raise SyndicationError(f"{url}: {type(e).__name__}")
raise SyndicationError(f"{url}: too many redirects")
def page_card(http: httpx.Client, url: str) -> dict:
"""
A page's title, description and image URL (Open Graph, Twitter or plain
HTML tags), for a link card. Missing parts come back empty; a page that
can't be fetched gives just the URL as the title.
"""
from urllib.parse import urljoin
from bs4 import BeautifulSoup
card = {"title": url, "description": "", "image": ""}
try:
data, content_type, final = _fetch_public(http, url, CARD_PAGE_LIMIT, truncate=True)
except SyndicationError as e:
current_app.logger.info(f"No link card details for {url}: {e}")
return card
if "html" not in content_type.lower():
return card
soup = BeautifulSoup(data, "html.parser")
def meta(*names):
for name in names:
tag = soup.find("meta", attrs={"property": name}) or soup.find("meta", attrs={"name": name})
if tag and (tag.get("content") or "").strip():
return " ".join(tag["content"].split())
return ""
title = meta("og:title", "twitter:title")
if not title and soup.title:
title = " ".join(soup.title.get_text(" ").split())
image = meta("og:image:secure_url", "og:image", "og:image:url", "twitter:image")
card.update(title=title or url,
description=meta("og:description", "twitter:description", "description"),
image=urljoin(final, image) if image else "")
return card
THREADS_API = "https://graph.threads.net/v1.0"
THREADS_IMAGE_TYPES = {"image/jpeg", "image/png"}
THREADS_IMAGE_LIMIT = 8_000_000
THREADS_MAX_LINKS = 5
THREADS_REFRESH_EVERY = 24 * 3600 # tokens last 60 days; Meta allows a refresh once one is a day old
# Graph API error codes that are temporary (rate limits, service hiccups)
_GRAPH_TRANSIENT = {1, 2, 4, 17, 32, 341, 613}
@@ -579,7 +684,9 @@ class ThreadsClient:
# --- Posts ---
def _text(self, post: "Post") -> str:
body = plain_text(post.note.content)
# Links written "text (url)"; the first one gets Threads' preview card.
# Meta allows 5 different links per post and the permalink is one.
body = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
limit = post.max_chars
while True:
text = compose_text(post.note, post.permalink, [], limit, body=body)
@@ -595,7 +702,7 @@ class ThreadsClient:
return self._call("POST", f"{uid}/threads", **params)["id"]
def _repost_text(self, post: "Post") -> str:
comment = plain_text(post.note.content)
comment = plain_text(post.note.content, links="inline", max_urls=THREADS_MAX_LINKS - 1)
limit = post.max_chars
while True:
text = compose_repost_text(post, comment, limit)
@@ -606,9 +713,9 @@ class ThreadsClient:
def _container(self, uid: str, post: "Post") -> str:
if post.kind == "repost":
# No native repost of outside content: post "🔁 title url" with a link preview
# No native repost of outside content: post "🔁 title url"
return self._call("POST", f"{uid}/threads", media_type="TEXT",
text=self._repost_text(post), link_attachment=post.target_url)["id"]
text=self._repost_text(post))["id"]
common = {"text": self._text(post)}
topic = threads_topic(post.tags)
if topic:
@@ -617,8 +724,10 @@ class ThreadsClient:
images = [m for m in post.media if m["mime_type"] in THREADS_IMAGE_TYPES
and Path(m["path"]).stat().st_size <= THREADS_IMAGE_LIMIT]
if not images:
return self._call("POST", f"{uid}/threads", media_type="TEXT",
link_attachment=post.permalink, **common)["id"]
# No explicit link_attachment: Meta rejects one for a page it hasn't
# seen yet ("Invalid Link Attachment"), but previews the text's
# first link by itself without failing
return self._call("POST", f"{uid}/threads", media_type="TEXT", **common)["id"]
if len(images) == 1:
return self._image_container(uid, images[0], **common)
children = [self._image_container(uid, m, is_carousel_item="true") for m in images]
@@ -894,20 +1003,82 @@ def compose_text(note, permalink: str, tags: List[str], max_chars: int,
if len(body) + len(tail) > max_chars:
room = max_chars - len(tail) - 1
body = body[:room].rsplit(" ", 1)[0].rstrip() + "…"
return body + tail
# A photo post may have no text: start with the link (v2.6.5)
return (body + tail).lstrip()
_BLOCKS = ["p", "h1", "h2", "h3", "h4", "h5", "h6", "pre", "blockquote", "ul", "ol", "hr", "table"]
def plain_text(markdown_text: str) -> str:
"""Markdown as readable plain text (for targets without Markdown)"""
def _note_soup(markdown_text: str):
import markdown
from bs4 import BeautifulSoup
soup = BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
return BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
def _clean_url(url: str) -> str:
return url.rstrip(".,;:!?)\"'…")
def _is_web(url: str) -> bool:
return url.startswith(("http://", "https://"))
def note_links(markdown_text: str) -> List[str]:
"""Web links in a note, in reading order: Markdown links, <autolinks> and bare URLs"""
from bs4 import NavigableString
links = []
for node in _note_soup(markdown_text).descendants:
if getattr(node, "name", None) == "a" and _is_web(node.get("href", "")):
links.append(node["href"])
elif isinstance(node, NavigableString) and not node.find_parent("a"):
links.extend(_clean_url(m.group(0)) for m in _URL_RE.finditer(str(node)))
return list(dict.fromkeys(links))
def plain_text(markdown_text: str, links: str = "text", max_urls: Optional[int] = None) -> str:
"""Markdown as readable plain text (for targets without Markdown). See `plain_text_links`."""
return plain_text_links(markdown_text, links, max_urls)[0]
def plain_text_links(markdown_text: str, links: str = "text",
max_urls: Optional[int] = None) -> tuple:
"""
Markdown as plain text, and the links whose text stands in for a URL.
A Markdown link `[text](url)` becomes:
links="text" just its text
links="inline" "text (url)", for targets with no clickable link text;
after `max_urls` different URLs, just the text
links="anchor" its text, listed in the returned anchors as (text, url)
so the target can make that text clickable
A link whose text is the URL itself stays a bare URL.
"""
from bs4 import NavigableString
soup = _note_soup(markdown_text)
for img in soup.find_all("img"):
img.decompose()
anchors = []
seen = set()
for node in list(soup.descendants): # bare URLs count towards max_urls first
if isinstance(node, NavigableString) and not node.find_parent("a"):
seen.update(_clean_url(m.group(0)) for m in _URL_RE.finditer(str(node)))
for a in soup.find_all("a"):
href, label = a.get("href", ""), a.get_text()
if not _is_web(href):
continue
bare = re.sub(r"^https?://", "", href).rstrip("/")
if label.strip().rstrip("/") in (href.rstrip("/"), bare):
a.replace_with(href)
seen.add(href)
elif links == "inline" and (href in seen or max_urls is None or len(seen) < max_urls):
a.replace_with(f"{label} ({href})")
seen.add(href)
elif links == "anchor":
anchors.append((label, href))
for node in soup.find_all(string=True):
if not node.strip() and "\n" in node and node.parent.name != "pre":
node.extract() # layout newlines between tags; we add our own
@@ -920,24 +1091,41 @@ def plain_text(markdown_text: str) -> str:
block.append("\n\n")
text = soup.get_text()
text = re.sub(r"[ \t]+\n", "\n", text)
return re.sub(r"\n{3,}", "\n\n", text).strip()
return re.sub(r"\n{3,}", "\n\n", text).strip(), anchors
_URL_RE = re.compile(r"https?://[^\s<>\"]+")
_TAG_RE = re.compile(r"(?<![\w#])#(\w*[^\W\d]\w*)")
def bluesky_facets(text: str) -> List[dict]:
"""Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post"""
def bluesky_facets(text: str, anchors: List[tuple] = ()) -> List[dict]:
"""
Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post.
`anchors` are (text, url) pairs: that text, where it appears, links to url.
"""
def span(start: int, end: int) -> dict:
return {"byteStart": len(text[:start].encode()), "byteEnd": len(text[:end].encode())}
facets = []
for m in _URL_RE.finditer(text):
url = m.group(0).rstrip(".,;:!?)\"'…")
url = _clean_url(m.group(0))
facets.append({"index": span(m.start(), m.start() + len(url)),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
taken = [(m.start(), m.end()) for m in _URL_RE.finditer(text)]
pos = 0
for label, url in anchors:
start = text.find(label, pos) if label.strip() else -1
if start < 0:
continue # shortened away
end = start + len(label)
if any(a < end and start < b for a, b in taken):
continue
taken.append((start, end))
pos = end
facets.append({"index": span(start, end),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
urls = [(f["index"]["byteStart"], f["index"]["byteEnd"]) for f in facets]
facets.sort(key=lambda f: f["index"]["byteStart"])
for m in _TAG_RE.finditer(text):
index = span(m.start(), m.end())
if any(a <= index["byteStart"] < b for a, b in urls):
@@ -1052,7 +1240,7 @@ def _reconcile(row, target: Target) -> tuple:
post = Post(
note=note,
permalink=f"{site}/note/{note.slug}",
tags=[t["display_name"] for t in get_note_tags(note.id)],
tags=[t["display_name"] for t in get_note_tags(note.id, entered_order=True)],
max_chars=target.max_chars,
media=_media_files(note.id, target.max_media) if note.kind == "note" else [],
kind=note.kind,
+11 -6
View File
@@ -115,27 +115,32 @@ def add_tags_to_note(note_id: int, tags: list[str]) -> None:
db.commit()
def get_note_tags(note_id: int) -> list[dict]:
def get_note_tags(note_id: int, entered_order: bool = False) -> list[dict]:
"""
Get all tags for a note
Args:
note_id: Note database ID
entered_order: In the order they were entered (the first is the
note's main tag, e.g. its Threads topic) instead of alphabetical.
add_tags_to_note() rewrites the links in entry order on every
save, so note_tags.id records it. Use this wherever the tags are
saved back (editor, Micropub update) so the order survives.
Returns:
List of tag dicts with 'name' and 'display_name'
Ordered alphabetically by display_name (case-insensitive)
Query should use: ORDER BY LOWER(tags.display_name) ASC
Ordered alphabetically by display_name (case-insensitive), unless
entered_order
"""
db = get_db(current_app)
order = "note_tags.id" if entered_order else "LOWER(tags.display_name)"
tags = db.execute(
"""
f"""
SELECT tags.name, tags.display_name
FROM tags
JOIN note_tags ON tags.id = note_tags.tag_id
WHERE note_tags.note_id = ?
ORDER BY LOWER(tags.display_name) ASC
ORDER BY {order} ASC
""",
(note_id,)
).fetchall()
+6 -5
View File
@@ -2,7 +2,7 @@
{% from "partials/media.html" import display_media %}
{% from "partials/reaction.html" import reaction %}
{% block title %}{% if note.content.strip().startswith('#') %}{{ note.title }}{% else %}{{ note.excerpt|truncate(60) }}{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block title %}{% if note.content.strip().startswith('#') %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
{% block content %}
<article class="h-entry">
@@ -18,13 +18,14 @@
{% if note.in_reply_to %}
<div class="reply-context">
<span class="reply-label">In reply to:</span>
<blockquote class="h-cite">
{# u-in-reply-to on the h-cite itself makes it the entry's property (Post Type Discovery) #}
<blockquote class="u-in-reply-to h-cite">
{% if reply_context and reply_context.fetch_status == 'fetched' %}
{% if reply_context.content_snippet %}
<p class="p-content">{{ reply_context.content_snippet }}</p>
{% endif %}
<cite>
<a class="u-in-reply-to" href="{{ note.in_reply_to }}">
<a class="u-url" href="{{ note.in_reply_to }}">
{% if reply_context.author_name %}{{ reply_context.author_name }}{% endif %}
{% if reply_context.site_name %} on {{ reply_context.site_name }}{% endif %}
{% if not reply_context.author_name and not reply_context.site_name %}{{ note.in_reply_to }}{% endif %}
@@ -32,7 +33,7 @@
</cite>
{% else %}
<cite>
<a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
<a class="u-url" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a>
</cite>
{% endif %}
</blockquote>
@@ -43,7 +44,7 @@
{# u-photo placement: Per draft spec, u-photo must be direct child of h-entry, #}
{# NOT inside e-content. Media is rendered ABOVE e-content to meet this requirement. #}
{{ display_media(note.media) }}
{{ display_media(note.media, note.excerpt|truncate(150)) }}
{# e-content: note content BELOW media (per ADR-057) #}
<div class="e-content">
+3 -2
View File
@@ -1,11 +1,12 @@
{# Reusable media display macro #}
{% macro display_media(media_items) %}
{# fallback_alt: for photos without a caption, usually the post's text (v2.6.5) #}
{% macro display_media(media_items, fallback_alt='') %}
{% if media_items %}
<div class="note-media">
{% for item in media_items %}
<figure class="media-item">
<img src="{{ url_for('public.media_file', path=item.path) }}"
alt="{{ item.caption or 'Image' }}"
alt="{{ item.caption or fallback_alt or 'Image' }}"
class="u-photo"
loading="lazy">
{# No figcaption - caption is for alt text only #}
+7 -2
View File
@@ -9,9 +9,13 @@
<h2 class="p-name"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">{{ note.title }}</a></h2>
{% endif %}
{% if note.in_reply_to %}
<p class="reaction reply-to"><span class="reaction-verb">↩ In reply to</span> <a class="u-in-reply-to" href="{{ note.in_reply_to }}">{{ note.in_reply_to }}</a></p>
{% endif %}
{{ reaction(note) }}
{{ display_media(note.media) }}
{{ display_media(note.media, note.excerpt|truncate(150)) }}
{% if has_explicit_title %}
{# Titled notes are articles: show a summary and link through #}
@@ -19,7 +23,8 @@
<p>{{ note.excerpt }}</p>
</div>
<p class="read-more"><a href="{{ absolute_url_for('public.note', slug=note.slug) }}">Read more →</a></p>
{% elif note.html %}
{% else %}
{# Always present, even empty: without it a text-free photo post gets an implied name (mf2) #}
<div class="e-content">
{{ note.html|safe }}
</div>
+1 -1
View File
@@ -118,7 +118,7 @@ class TestUpdate:
def test_categories(self, app, client, h):
url = create(client, h, category=["one", "two"])
self.update(client, h, url, add={"category": ["three", "One"]})
assert source(client, h, url)["properties"]["category"] == ["one", "three", "two"]
assert source(client, h, url)["properties"]["category"] == ["one", "two", "three"] # entered order (v2.6.4)
self.update(client, h, url, delete={"category": ["two"]})
assert source(client, h, url)["properties"]["category"] == ["one", "three"]
self.update(client, h, url, delete=["category"])
+105
View File
@@ -0,0 +1,105 @@
"""
Micropub media fixes (v2.6.5): photo posts without text, the same photo
twice, and failed creates that left a post (and a database lock) behind.
"""
import io
from starpunk.database.pool import ConnectionPool
from starpunk.notes import list_notes
from starpunk.syndication import compose_text
from tests.test_micropub_v240 import SITE, app, client, h, jpeg, note_at, source # noqa: F401
def upload(client, h, color="red"):
response = client.post("/micropub/media", data={"file": (io.BytesIO(jpeg(color)), "a.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 201
return response.headers["Location"]
def post(client, h, props):
return client.post("/micropub", json={"type": ["h-entry"], "properties": props}, headers=h)
def update(client, h, url, **ops):
return client.post("/micropub", json={"action": "update", "url": url, **ops}, headers=h)
class TestPhotoWithoutText:
def test_json_photo_only(self, app, client, h):
photo = upload(client, h)
response = post(client, h, {"photo": [{"value": photo, "alt": "Red"}]})
assert response.status_code == 201, response.get_json()
url = response.headers["Location"]
assert note_at(app, url).content == ""
assert source(client, h, url, "photo")["properties"]["photo"] == [{"value": photo, "alt": "Red"}]
page = client.get(url.replace(SITE, ""))
assert page.status_code == 200 and b"<title>Photo \xc2\xb7 " in page.data
def test_multipart_photo_only(self, client, h):
response = client.post("/micropub", data={"h": "entry", "photo": (io.BytesIO(jpeg()), "p.jpg")},
headers=h, content_type="multipart/form-data")
assert response.status_code == 201, response.get_json()
def test_nothing_at_all_is_still_rejected(self, client, h):
assert post(client, h, {}).status_code == 400
# A photo that can't be found doesn't count
assert post(client, h, {"photo": [f"{SITE}/media/nope.jpg"]}).status_code == 400
def test_text_can_be_cleared_while_photos_remain(self, app, client, h):
photo = upload(client, h)
url = post(client, h, {"content": ["Hi"], "photo": [photo]}).headers["Location"]
assert update(client, h, url, replace={"content": [""]}).status_code == 204
assert note_at(app, url).content == ""
assert update(client, h, url, replace={"content": ["Back"]}).status_code == 204
assert update(client, h, url, delete=["content"]).status_code == 204
assert note_at(app, url).content == ""
# Without photos the text has to stay
response = update(client, h, url, delete=["content", "photo"])
assert response.status_code == 400
def test_text_is_required_without_photos(self, client, h):
url = post(client, h, {"content": ["Hi"]}).headers["Location"]
assert update(client, h, url, replace={"content": [""]}).status_code == 400
assert update(client, h, url, delete=["content"]).status_code == 400
def test_crosspost_text_starts_with_the_link(self, app, client, h):
url = post(client, h, {"photo": [upload(client, h)]}).headers["Location"]
with app.app_context():
note = note_at(app, url)
assert compose_text(note, url, ["cats"], 500) == f"{url}\n\n#cats"
class TestFailures:
def test_same_photo_twice_is_shown_once(self, client, h):
photo = upload(client, h)
response = post(client, h, {"content": ["Twice"], "photo": [photo, photo]})
assert response.status_code == 201, response.get_json()
assert source(client, h, response.headers["Location"], "photo")["properties"]["photo"] == [{"value": photo, "alt": ""}]
def test_failed_create_leaves_no_post_and_no_lock(self, app, client, h, monkeypatch):
import starpunk.syndication as syndication
def boom(*args, **kwargs):
raise RuntimeError("crosspost store down")
monkeypatch.setattr(syndication, "select_targets", boom)
response = post(client, h, {"content": ["Doomed"], "photo": [upload(client, h)]})
assert response.status_code == 500
with app.app_context():
assert list_notes(published_only=False) == []
monkeypatch.undo()
# The database is usable straight away
assert post(client, h, {"content": ["Next"]}).status_code == 201
def test_pool_rolls_back_open_transactions(self, tmp_path):
pool = ConnectionPool(str(tmp_path / "x.db"), pool_size=1, timeout=1)
conn = pool.get_connection()
conn.execute("CREATE TABLE t (x)")
conn.commit()
conn.execute("INSERT INTO t VALUES (1)")
assert conn.in_transaction
pool.return_connection(conn)
assert not conn.in_transaction
assert pool.get_connection().execute("SELECT count(*) FROM t").fetchone()[0] == 0
+98
View File
@@ -0,0 +1,98 @@
"""
Rendered posts read as the right post type (v2.6.5).
Parses note pages and the home page with mf2py, then runs W3C Post Type
Discovery (https://www.w3.org/TR/post-type-discovery/) on what a consumer
sees. A photo post is any post with a photo, unless it's first a repost,
like or reply.
"""
import pytest
import mf2py
import starpunk.reply_context as reply_context
from tests.test_micropub_v240 import SITE, app, client, h, jpeg # noqa: F401
from tests.test_micropub_v265 import upload
def post_type(props):
"""W3C Post Type Discovery, section 5 (no events or RSVPs here)."""
for prop, kind in [("repost-of", "repost"), ("like-of", "like"), ("in-reply-to", "reply"),
("video", "video"), ("photo", "photo")]:
if props.get(prop):
return kind
def first(prop):
for v in props.get(prop, []):
v = v.get("value", "") if isinstance(v, dict) else v
if v.strip():
return " ".join(v.split())
return ""
content = first("content") or first("summary")
name = first("name")
if not content or not name or content.startswith(name):
return "note"
return "article"
def entries(items):
for item in items:
if "h-entry" in item.get("type", []):
yield item
yield from entries(item.get("children", []))
def value(v):
"""A u-* value, or the url of a nested microformat (h-cite)."""
return v.get("value") or v.get("properties", {}).get("url", [None])[0] if isinstance(v, dict) else v
CASES = {
"note": ({"content": ["Just a note"]}, "note"),
"article": ({"name": ["A Title"], "content": ["Body text here"]}, "article"),
"photo without text": ({"photo": ["PHOTO"]}, "photo"),
"photo with caption": ({"content": ["Sunset"], "photo": [{"value": "PHOTO", "alt": "Orange sky"}]}, "photo"),
"photo with title": ({"name": ["Trip"], "content": ["Day one"], "photo": ["PHOTO"]}, "photo"),
"reply": ({"content": ["Agreed"], "in-reply-to": ["https://other.example/post/1"]}, "reply"),
"reply with photo": ({"content": ["Mine"], "in-reply-to": ["https://other.example/post/2"],
"photo": ["PHOTO"]}, "reply"),
"repost": ({"repost-of": ["https://other.example/post/4"]}, "repost"),
}
@pytest.mark.parametrize("case", CASES)
def test_post_type(app, client, h, monkeypatch, case):
monkeypatch.setattr(reply_context, "fetch_reply_context", lambda *a, **k: None, raising=False)
props, want = CASES[case]
photo = upload(client, h) if "PHOTO" in str(props) else None
props = {k: [photo if v == "PHOTO" else ({**v, "value": photo} if isinstance(v, dict) else v)
for v in vs] for k, vs in props.items()}
response = client.post("/micropub", json={"type": ["h-entry"], "properties": props}, headers=h)
assert response.status_code == 201, response.get_json()
url = response.headers["Location"]
(entry,) = entries(mf2py.parse(doc=client.get(url.replace(SITE, "")).data.decode(), url=url)["items"])
home = mf2py.parse(doc=client.get("/").data.decode(), url=f"{SITE}/")
(listed,) = [e for e in entries(home["items"]) if e["properties"].get("url") == [url]]
for e in (entry, listed):
assert post_type(e["properties"]) == want
if want == "reply":
assert value(e["properties"]["in-reply-to"][0]) == props["in-reply-to"][0]
if want == "photo" and "name" not in props:
assert "name" not in e["properties"], "no implied name to show as a caption"
def test_photo_alt_text(app, client, h):
"""Alt text is the caption, else the post's text, never made up (v2.6.5)."""
def alts(props):
url = client.post("/micropub", json={"type": ["h-entry"], "properties": props},
headers=h).headers["Location"]
(entry,) = entries(mf2py.parse(doc=client.get(url.replace(SITE, "")).data.decode(), url=url)["items"])
return [p["alt"] for p in entry["properties"]["photo"]]
assert alts({"content": ["Beach day"], "photo": [{"value": upload(client, h), "alt": "Waves"}]}) == ["Waves"]
assert alts({"content": ["Beach day"], "photo": [upload(client, h, "blue")]}) == ["Beach day"]
assert alts({"name": ["Trip"], "content": ["Day one"], "photo": [upload(client, h, "green")]}) == ["Day one"]
assert alts({"photo": [upload(client, h, "gray")]}) == ["Image"]
+120 -2
View File
@@ -92,8 +92,31 @@ def pds():
return FakePDS()
class FakeWeb:
"""Other people's pages, for link cards: url -> (status, headers, body)"""
def __init__(self):
self.pages = {}
self.fetched = []
def __call__(self, request):
url = str(request.url)
self.fetched.append(url)
status, headers, body = self.pages.get(url, (404, {}, b"nope"))
return httpx.Response(status, headers=headers, content=body)
@pytest.fixture
def app(tmp_path, pds):
def web():
return FakeWeb()
def routed(pds, web):
return httpx.MockTransport(
lambda r: pds(r) if r.url.host in ("bsky.example", "pds.example") else web(r))
@pytest.fixture
def app(tmp_path, pds, web):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
@@ -108,7 +131,7 @@ def app(tmp_path, pds):
"BLUESKY_SERVICE": SERVICE,
})
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"] = BlueskyClient(
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=httpx.MockTransport(pds))
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=routed(pds, web))
)
yield app
syndication._client_cache.clear()
@@ -258,6 +281,101 @@ class TestMedia:
assert image["aspectRatio"]["width"] > 0
ARTICLE = "https://news.example/story"
ARTICLE_HTML = b"""<html><head><title>Fallback title</title>
<meta property="og:title" content="A Great Story">
<meta property="og:description" content="What happened, in short.">
<meta property="og:image" content="/img/story.png">
</head><body>...</body></html>"""
def png_bytes():
buf = io.BytesIO()
Image.new("RGB", (60, 40), (10, 120, 200)).save(buf, format="PNG")
return buf.getvalue()
class TestLinkCards:
def test_note_link_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
web.pages["https://news.example/img/story.png"] = (200, {"content-type": "image/png"}, png_bytes())
note = publish(app, content=f"Read [this story]({ARTICLE}) now")
record = only_record(pds)
external = record["embed"]["external"]
assert external["uri"] == ARTICLE
assert external["title"] == "A Great Story"
assert external["description"] == "What happened, in short."
assert external["thumb"]["$type"] == "blob"
# The link text is clickable and the note's permalink is still in the text
permalink = f"https://blog.example.com/note/{note.slug}"
assert record["text"].startswith("Read this story now") and permalink in record["text"]
raw = record["text"].encode()
links = {raw[f["index"]["byteStart"]:f["index"]["byteEnd"]].decode(): f["features"][0]["uri"]
for f in record["facets"]}
assert links["this story"] == ARTICLE
assert links[permalink] == permalink
def test_bare_url_gets_the_card(self, app, pds, web):
web.pages[ARTICLE] = (200, {"content-type": "text/html"}, ARTICLE_HTML)
publish(app, content=f"See {ARTICLE}.")
assert only_record(pds)["embed"]["external"]["uri"] == ARTICLE
def test_unreachable_page_still_gets_a_card(self, app, pds, web):
note = publish(app, content=f"See {ARTICLE}")
external = only_record(pds)["embed"]["external"]
assert external == {"uri": ARTICLE, "title": ARTICLE, "description": ""}
assert row(app, note)["state"] == "synced"
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
monkeypatch.setattr("starpunk.micropub._public_address", lambda host: host != "internal.example")
app.config["TESTING"] = False
try:
publish(app, content=f"See {ARTICLE}")
finally:
app.config["TESTING"] = True
assert "http://internal.example/admin" not in web.fetched
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE
def test_images_win_over_link_card(self, app, pds, web):
from starpunk.media import attach_media_to_note, save_media
with app.test_request_context():
note = create_note(content=f"Look {ARTICLE}", published=True)
media = save_media(png_bytes(), "a.png")
attach_media_to_note(note.id, [media["id"]], ["pic"])
select_targets(note.id, ["bluesky"])
process_due()
assert only_record(pds)["embed"]["$type"] == "app.bsky.embed.images"
assert web.fetched == []
class TestLinkText:
def test_note_links_in_reading_order(self):
from starpunk.syndication import note_links
md = "a [x](https://one.example) b https://two.example/p. c <https://three.example> [rel](/local)"
assert note_links(md) == ["https://one.example", "https://two.example/p", "https://three.example"]
def test_link_modes(self):
from starpunk.syndication import plain_text_links
md = "[a story](https://s.example) and [https://u.example](https://u.example)"
assert plain_text_links(md)[0] == "a story and https://u.example"
assert plain_text_links(md, "inline")[0] == "a story (https://s.example) and https://u.example"
assert plain_text_links(md, "anchor") == ("a story and https://u.example",
[("a story", "https://s.example")])
def test_inline_url_budget(self):
from starpunk.syndication import plain_text_links
md = "https://bare.example then [one](https://1.example) [two](https://2.example) [one again](https://1.example)"
text = plain_text_links(md, "inline", max_urls=2)[0]
assert text == "https://bare.example then one (https://1.example) two one again (https://1.example)"
def test_anchor_facet_byte_offsets(self):
text = "Café ☕ this story and more"
(facet,) = bluesky_facets(text, [("this story", "https://s.example")])
raw = text.encode()
assert raw[facet["index"]["byteStart"]:facet["index"]["byteEnd"]] == b"this story"
def test_both_targets_listed_for_micropub(tmp_path):
data = tmp_path / "d"
app = create_app({
+18 -2
View File
@@ -180,7 +180,7 @@ class TestLifecycle:
permalink = f"https://blog.example.com/note/{note.slug}"
assert post["media_type"] == "TEXT"
assert post["text"] == f"Hello Threads\n\n{permalink}"
assert post["link_attachment"] == permalink
assert "link_attachment" not in post # Threads previews the first link itself
assert post["topic_tag"] == "IndieWeb"
r = row(app, note)
assert r["state"] == "synced"
@@ -268,6 +268,22 @@ class TestErrors:
assert row(app, note)["state"] == "pending"
class TestLinks:
def test_markdown_link_comes_first_for_the_preview(self, app, api):
note = publish(app, content="Read [this story](https://news.example/story) now")
text = only_post(api)["text"]
assert text.startswith("Read this story (https://news.example/story) now")
assert text.index("https://news.example/story") < text.index(f"/note/{note.slug}")
def test_stays_within_five_links(self, app, api):
import re
links = " ".join(f"[l{i}](https://l{i}.example)" for i in range(7))
publish(app, content=f"Lots: {links}")
text = only_post(api)["text"]
assert len(set(re.findall(r"https?://[^\s)]+", text))) == 5
assert "l6" in text # the extra links keep their text
class TestMedia:
def _note_with(self, app, images):
from starpunk.media import attach_media_to_note, save_media
@@ -391,7 +407,7 @@ class TestLikesReposts:
self._post(app, monkeypatch, repost_of=self.PAGE)
post = only_post(api)
assert post["text"].startswith("🔁") and self.PAGE in post["text"]
assert post["link_attachment"] == self.PAGE
assert "link_attachment" not in post
def test_repost_with_comment_keeps_comment(self, app, api, monkeypatch):
note = self._post(app, monkeypatch, content="So **true**", repost_of=self.PAGE)
+86
View File
@@ -0,0 +1,86 @@
"""
Tags keep the order they were entered in (v2.6.4)
Public pages list tags alphabetically; crossposts (the Threads topic, which
tags fit on Bluesky), the editor and Micropub use the entered order, and
saving from the editor or Micropub keeps it.
"""
import pytest
from starpunk import create_app
from starpunk.auth import create_session
from starpunk.indieauth import create_access_token
from starpunk.notes import create_note, get_note
from starpunk.tags import get_note_tags
ENTERED = ["Zebra", "apple", "Mango"]
@pytest.fixture
def app(tmp_path):
data = tmp_path / "data"
data.mkdir()
yield create_app(config={
"TESTING": True, "DATABASE_PATH": data / "starpunk.db", "DATA_PATH": data,
"NOTES_PATH": data / "notes", "SESSION_SECRET": "s", "ADMIN_ME": "https://test.example.com",
"SITE_URL": "http://localhost:5000", "DEV_MODE": False,
})
@pytest.fixture
def note(app):
with app.test_request_context():
return create_note(content="Tagged", published=True, tags=ENTERED)
def names(app, note_id, **kw):
with app.app_context():
return [t["display_name"] for t in get_note_tags(note_id, **kw)]
def test_alphabetical_by_default_entered_on_request(app, note):
assert names(app, note.id) == ["apple", "Mango", "Zebra"]
assert names(app, note.id, entered_order=True) == ENTERED
def test_public_page_stays_alphabetical(app, note):
html = app.test_client().get(f"/note/{note.slug}").data.decode()
assert html.index(">apple<") < html.index(">Mango<") < html.index(">Zebra<")
def test_editor_shows_and_keeps_entered_order(app, note):
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://test.example.com"))
html = client.get(f"/admin/edit/{note.id}").data.decode()
assert 'value="Zebra, apple, Mango"' in html
# Saving the form unchanged keeps the order
response = client.post(f"/admin/edit/{note.id}", data={
"content": "Tagged, edited", "published": "on", "tags": "Zebra, apple, Mango"})
assert response.status_code in (200, 302)
assert names(app, note.id, entered_order=True) == ENTERED
def test_micropub_source_and_update_keep_entered_order(app, note):
with app.test_request_context():
token = create_access_token("https://quill.example/", "create update")["access_token"]
client = app.test_client()
auth = {"Authorization": f"Bearer {token}"}
url = f"http://localhost:5000/note/{note.slug}"
source = client.get(f"/micropub?q=source&url={url}", headers=auth).get_json()
assert source["properties"]["category"] == ENTERED
response = client.post("/micropub", headers=auth, json={
"action": "update", "url": url, "add": {"category": ["Kiwi"]}})
assert response.status_code in (200, 204), response.get_data(as_text=True)
assert names(app, note.id, entered_order=True) == ENTERED + ["Kiwi"]
def test_crossposts_use_entered_order(app, note):
from starpunk.syndication import compose_text, threads_topic
tags = names(app, note.id, entered_order=True)
assert threads_topic(tags) == "Zebra"
with app.app_context():
text = compose_text(get_note(id=note.id, load_content=True),
"https://x.example/note/1", tags, 25)
assert "#Zebra" in text and "#Mango" not in text # room for the first tags only