fix: keep inline images in Micropub articles (v2.16.0)

- Content sent as both html and text is stored as the HTML; the text is
  usually a plain fallback, so images, links and headings were lost.
- Images embedded from other sites are copied into the media library
  (SSRF-checked fetcher, best effort, max 20) and the text points at the
  copies, on create and on content replace.
- HTML content is sanitized as is instead of going through Markdown, which
  added stray <br>s and empty paragraphs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
PhilandClaude Opus 5.5 committed 2026-11-11 17:11:46 -07:00
1 parent 978c13358b
commit ccb9f6dce2
5 files changed
+117 -14

No files matched your search

+3
View File
@@ -24,12 +24,15 @@ Articles as the IndieWeb defines them: titles and summaries stored apart from th
- A `# Heading` at the top of a note is now just a heading, not a title. Micropub `name` is stored as the title instead of being added to the text; a heading that repeats it is dropped
- Feed items have a title only for articles. RSS items and JSON Feed items for notes have none, and Atom entries an empty one, so feed readers no longer show notes as headlines
- On the homepage and tag pages an article shows its title, its summary (or the start of its text) and "Read more", without its images
- Micropub content sent as both `html` and `text` is stored as the HTML. The text is usually a plain fallback, so articles lost their images, links and headings
- Images embedded in Micropub content from other sites are copied into the media library and the text points at the copies, so an article doesn't break when the client's or a CDN's host goes away. Best effort: an image that can't be fetched keeps its URL (up to 20 per post)
- Articles are crossposted to GoToSocial, Bluesky and Threads as title, summary (or the start of the text), link and hashtags, without images. On Bluesky the link card is the article. Notes are crossposted in full as before
### Fixed
- Feeds showed an article's title twice: as the item title and again at the top of the content
- A line starting with a hashtag (`#indieweb is fun`) was shown as a heading on pages and in feeds, and crossposted without its `#`. Only `#` followed by a space starts a heading now, as in CommonMark
- HTML content from Micropub is sanitized as is instead of going through Markdown, which added stray line breaks and empty paragraphs
- Previews, image alt text and crosspost teasers showed raw Markdown (`**bold**`, `[link](url)`); the excerpt is now plain text
## [2.15.0] - 2026-11-11
@@ -25,6 +25,8 @@ article.
- **Hashtag lines.** Python-Markdown made `#indieweb is fun` an `<h1>`; CommonMark needs a space after `#`. `markdown_ext.HashtagLineExtension` escapes the first `#` of a line where `#`s run straight into text, after fenced code and raw HTML are set aside. It is used for pages and feeds (`MARKDOWN_EXTENSIONS`) and crosspost plain text (`syndication._note_soup`), so the hashtag also reaches Bluesky as a tag. Search's first-line label keeps hashtags too.
- **Plain-text excerpt.** `Note.excerpt` only stripped `#` marks, so previews showed `**bold**` and `[link](url)`. It is now `syndication.plain_text` of the content, cut to 200 characters with `…`; `article_teaser` is the summary or this excerpt.
- **Inline images in Micropub articles.** (1) `extract_content` prefers `html` over `text` and marks it `content_format = 'html'`: clients send `text` as a plain fallback, so images, links and headings were lost. (2) `localize_inline_images` copies images from other sites (Markdown `![]()` or `<img>`) into the media library through the SSRF-checked fetcher used for `photo`, on create and on content replace, and rewrites their URLs; failures keep the original URL, at most 20 per post. Images stay inline (not `u-photo`), so the post remains an article. (3) `Note.html` sanitizes HTML content with nh3 directly instead of passing it through Markdown.
## Choices made with the owner
- Title column rather than an editor-only field, as the most standards-compliant option.
+41 -11
View File
@@ -205,10 +205,11 @@ def extract_content(properties: dict) -> str:
content = content_list[0]
# Handle structured content ({"html": "...", "text": "..."})
# Handle structured content ({"html": "...", "text": "..."}). The HTML is
# what was written; clients send text as a plain fallback, which loses
# images, links and headings (v2.16.0; before, text was preferred)
if isinstance(content, dict):
# Prefer text over html for markdown storage
content = content.get("text") or content.get("html", "")
content = content.get("html") or content.get("text", "")
if not isinstance(content, str) or not content.strip():
raise MicropubValidationError("Content cannot be empty")
@@ -217,9 +218,9 @@ def extract_content(properties: dict) -> str:
def content_format(properties: dict) -> Optional[str]:
"""'html' if the content was sent only as HTML (returned as such by q=source)"""
"""'html' if the content was sent as HTML (returned as such by q=source)"""
content = _first(properties.get("content"))
if isinstance(content, dict) and content.get("html") and not content.get("text"):
if isinstance(content, dict) and content.get("html"):
return "html"
return None
@@ -413,9 +414,10 @@ def _public_address(host: str) -> bool:
return bool(infos)
def _fetch_remote_photo(url: str) -> Optional[int]:
def _fetch_remote_photo(url: str) -> Optional[dict]:
"""
Download a photo hosted elsewhere into the media library. Returns media id.
Download a photo hosted elsewhere into the media library. Returns the
saved media ({'id', 'path', ...}), or None if it couldn't be fetched.
Uses the webmention fetcher (v2.9.0): every redirect is checked against
private addresses and the connection is pinned to the checked address.
@@ -432,7 +434,7 @@ def _fetch_remote_photo(url: str) -> Optional[int]:
if result.status_code != 200:
raise ValueError(f"HTTP {result.status_code}")
name = urlparse(result.url).path.rsplit("/", 1)[-1] or "photo.jpg"
return save_media(result.content, name)["id"]
return save_media(result.content, name)
except Exception as e:
current_app.logger.warning(f"Could not fetch photo {url}: {e}")
return None
@@ -452,7 +454,35 @@ def _media_id_for_url(url: str) -> Optional[int]:
return row[0]
current_app.logger.warning(f"Photo URL not found in media: {url}")
return None
return _fetch_remote_photo(url)
media = _fetch_remote_photo(url)
return media["id"] if media else None
MAX_INLINE_IMAGES = 20
def localize_inline_images(content: str) -> str:
"""
Copy images embedded in the text from other sites into the media library
and point the text at the copies (v2.16.0), so an article doesn't depend
on the client's or a CDN's host staying up. Best effort: an image that
can't be fetched keeps its URL.
"""
from html import escape
from starpunk.syndication import inline_images
if not content or "<img" not in content and "![" not in content:
return content
site_url = current_app.config.get("SITE_URL", "").rstrip("/")
remote = [url for url, _ in inline_images(content)
if url.startswith(("http://", "https://")) and not url.startswith(site_url + "/")]
for url in list(dict.fromkeys(remote))[:MAX_INLINE_IMAGES]:
media = _fetch_remote_photo(url)
if media:
local = f"{site_url}/media/{media['path']}"
# As written in Markdown, and as it may be escaped in HTML attributes
content = content.replace(url, local).replace(escape(url), local)
return content
def _save_uploads(files) -> list[dict]:
@@ -602,7 +632,7 @@ def handle_create(data: dict, token_info: dict, files=None):
if has_content or not (like_of or repost_of or bookmark_of or photos or title)
else ""
)
content = drop_title_heading(content, title)
content = localize_inline_images(drop_title_heading(content, title))
# A summary with content is the author's summary; alone it is the
# content (Post Type Discovery), as extract_content does (v2.16.0)
summary = extract_summary(properties) if _has_text(properties.get("content")) else None
@@ -733,7 +763,7 @@ def handle_update(data: dict, token_info: dict):
if "content" in replace:
# Empty text is fine if the note keeps its photos; update_note checks (v2.6.5)
body = extract_content(replace) if _has_text(replace["content"]) else ""
body = localize_inline_images(extract_content(replace)) if _has_text(replace["content"]) else ""
changes["content_format"] = content_format(replace) if body else None
if "name" in replace:
title = extract_title(replace)
+7 -2
View File
@@ -298,8 +298,13 @@ class Note:
<p>This is my note content...</p>
"""
if self._cached_html is None:
# Render markdown to sanitized HTML (v2.9.0)
html = render_markdown(self.content) # triggers lazy load of content
if self.content_format == "html":
# Sent as HTML via Micropub: sanitize it as is. Through Markdown
# it picked up stray <br>s and empty paragraphs (v2.16.0)
html = nh3.clean(self.content, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
else:
# Render markdown to sanitized HTML (v2.9.0)
html = render_markdown(self.content) # triggers lazy load of content
# Cache it
object.__setattr__(self, "_cached_html", html)
+64 -1
View File
@@ -9,11 +9,12 @@ without its images; the admin editor has Title and Summary fields.
import json
import xml.etree.ElementTree as ET
import httpx
import mf2py
from starpunk.auth import create_session
from starpunk.notes import create_note, get_note
from tests.test_micropub_v240 import SITE, app, client, h # noqa: F401
from tests.test_micropub_v240 import SITE, app, client, h, jpeg, source # noqa: F401
from tests.test_micropub_v265 import upload
ATOM = {"a": "http://www.w3.org/2005/Atom"}
@@ -130,3 +131,65 @@ class TestAdmin:
from starpunk.notes import list_notes
(note,) = list_notes()
assert note.title == "Only a title" and note.content == ""
class TestInlineImages:
"""Articles posted with images in their text (v2.16.0)"""
def remote_host(self, monkeypatch, ok=("/a.jpg",)):
fetched = []
def serve(request): # the fetcher connects to the pinned address, so match on path
fetched.append(request.url.path)
if request.url.path in ok:
return httpx.Response(200, content=jpeg("green"))
return httpx.Response(404)
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(serve))
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
lambda *a, **k: [(2, 1, 6, "", ("93.184.216.34", 0))])
return fetched
def test_html_is_kept_over_text_fallback(self, app, client, h, monkeypatch):
self.remote_host(monkeypatch)
html = '<p>Intro <a href="https://x.example">link</a></p><img src="https://cdn.example/a.jpg" alt="A">'
url = micropub(client, h, name=["T"], content=[{"html": html, "text": "Intro link"}])
(content,) = source(client, h, url)["properties"]["content"]
assert "html" in content and 'href="https://x.example"' in content["html"]
assert "<img" in content["html"]
def test_remote_images_are_copied(self, app, client, h, monkeypatch):
fetched = self.remote_host(monkeypatch)
url = micropub(client, h, name=["T"], content=[
"Intro\n\n![A](https://cdn.example/a.jpg)\n\n![B](https://cdn.example/gone.jpg)\n\n"
"Again ![A](https://cdn.example/a.jpg)"])
with app.app_context():
note = get_note(slug=url.rsplit("/", 1)[1])
assert fetched.count("/a.jpg") == 1 # each image once
assert "https://cdn.example/a.jpg" not in note.content
assert note.content.count(f"]({SITE}/media/") == 2
assert "https://cdn.example/gone.jpg" in note.content # couldn't fetch: left as is
assert client.get(note.content.split(f"]({SITE}")[1].split(")")[0]).status_code == 200
def test_html_images_copied_and_rendered_cleanly(self, app, client, h, monkeypatch):
self.remote_host(monkeypatch)
url = micropub(client, h, name=["T"], content=[{"html":
'<p>Intro</p><img src="https://cdn.example/a.jpg" alt="A"><p>More</p><script>x()</script>'}])
with app.app_context():
note = get_note(slug=url.rsplit("/", 1)[1])
assert note.content_format == "html"
assert note.html.startswith("<p>Intro</p><img") and f'src="{SITE}/media/' in note.html
assert "<br" not in note.html and "<p></p>" not in note.html and "script" not in note.html
def test_own_images_not_fetched(self, app, client, h, monkeypatch):
fetched = self.remote_host(monkeypatch)
micropub(client, h, content=[f"![A]({SITE}/media/2026/11/x.jpg)"])
assert fetched == []
def test_update_copies_new_images(self, app, client, h, monkeypatch):
self.remote_host(monkeypatch)
url = micropub(client, h, name=["T"], content=["Before"])
client.post("/micropub", json={"action": "update", "url": url,
"replace": {"content": ["![A](https://cdn.example/a.jpg)"]}}, headers=h)
with app.app_context():
assert f"]({SITE}/media/" in get_note(slug=url.rsplit("/", 1)[1]).content