fix: keep inline images in Micropub articles (v2.16.0)
- Content sent as both html and text is stored as the HTML; the text is usually a plain fallback, so images, links and headings were lost. - Images embedded from other sites are copied into the media library (SSRF-checked fetcher, best effort, max 20) and the text points at the copies, on create and on content replace. - HTML content is sanitized as is instead of going through Markdown, which added stray <br>s and empty paragraphs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
978c13358b
commit
ccb9f6dce2
5 files changed
+117
-14
No files matched your search
@@ -24,12 +24,15 @@ Articles as the IndieWeb defines them: titles and summaries stored apart from th
|
||||
- A `# Heading` at the top of a note is now just a heading, not a title. Micropub `name` is stored as the title instead of being added to the text; a heading that repeats it is dropped
|
||||
- Feed items have a title only for articles. RSS items and JSON Feed items for notes have none, and Atom entries an empty one, so feed readers no longer show notes as headlines
|
||||
- On the homepage and tag pages an article shows its title, its summary (or the start of its text) and "Read more", without its images
|
||||
- Micropub content sent as both `html` and `text` is stored as the HTML. The text is usually a plain fallback, so articles lost their images, links and headings
|
||||
- Images embedded in Micropub content from other sites are copied into the media library and the text points at the copies, so an article doesn't break when the client's or a CDN's host goes away. Best effort: an image that can't be fetched keeps its URL (up to 20 per post)
|
||||
- Articles are crossposted to GoToSocial, Bluesky and Threads as title, summary (or the start of the text), link and hashtags, without images. On Bluesky the link card is the article. Notes are crossposted in full as before
|
||||
|
||||
### Fixed
|
||||
|
||||
- Feeds showed an article's title twice: as the item title and again at the top of the content
|
||||
- A line starting with a hashtag (`#indieweb is fun`) was shown as a heading on pages and in feeds, and crossposted without its `#`. Only `#` followed by a space starts a heading now, as in CommonMark
|
||||
- HTML content from Micropub is sanitized as is instead of going through Markdown, which added stray line breaks and empty paragraphs
|
||||
- Previews, image alt text and crosspost teasers showed raw Markdown (`**bold**`, `[link](url)`); the excerpt is now plain text
|
||||
## [2.15.0] - 2026-11-11
|
||||
|
||||
|
||||
@@ -25,6 +25,8 @@ article.
|
||||
- **Hashtag lines.** Python-Markdown made `#indieweb is fun` an `<h1>`; CommonMark needs a space after `#`. `markdown_ext.HashtagLineExtension` escapes the first `#` of a line where `#`s run straight into text, after fenced code and raw HTML are set aside. It is used for pages and feeds (`MARKDOWN_EXTENSIONS`) and crosspost plain text (`syndication._note_soup`), so the hashtag also reaches Bluesky as a tag. Search's first-line label keeps hashtags too.
|
||||
- **Plain-text excerpt.** `Note.excerpt` only stripped `#` marks, so previews showed `**bold**` and `[link](url)`. It is now `syndication.plain_text` of the content, cut to 200 characters with `…`; `article_teaser` is the summary or this excerpt.
|
||||
|
||||
- **Inline images in Micropub articles.** (1) `extract_content` prefers `html` over `text` and marks it `content_format = 'html'`: clients send `text` as a plain fallback, so images, links and headings were lost. (2) `localize_inline_images` copies images from other sites (Markdown `![]()` or `<img>`) into the media library through the SSRF-checked fetcher used for `photo`, on create and on content replace, and rewrites their URLs; failures keep the original URL, at most 20 per post. Images stay inline (not `u-photo`), so the post remains an article. (3) `Note.html` sanitizes HTML content with nh3 directly instead of passing it through Markdown.
|
||||
|
||||
## Choices made with the owner
|
||||
|
||||
- Title column rather than an editor-only field, as the most standards-compliant option.
|
||||
|
||||
+41
-11
@@ -205,10 +205,11 @@ def extract_content(properties: dict) -> str:
|
||||
|
||||
content = content_list[0]
|
||||
|
||||
# Handle structured content ({"html": "...", "text": "..."})
|
||||
# Handle structured content ({"html": "...", "text": "..."}). The HTML is
|
||||
# what was written; clients send text as a plain fallback, which loses
|
||||
# images, links and headings (v2.16.0; before, text was preferred)
|
||||
if isinstance(content, dict):
|
||||
# Prefer text over html for markdown storage
|
||||
content = content.get("text") or content.get("html", "")
|
||||
content = content.get("html") or content.get("text", "")
|
||||
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
raise MicropubValidationError("Content cannot be empty")
|
||||
@@ -217,9 +218,9 @@ def extract_content(properties: dict) -> str:
|
||||
|
||||
|
||||
def content_format(properties: dict) -> Optional[str]:
|
||||
"""'html' if the content was sent only as HTML (returned as such by q=source)"""
|
||||
"""'html' if the content was sent as HTML (returned as such by q=source)"""
|
||||
content = _first(properties.get("content"))
|
||||
if isinstance(content, dict) and content.get("html") and not content.get("text"):
|
||||
if isinstance(content, dict) and content.get("html"):
|
||||
return "html"
|
||||
return None
|
||||
|
||||
@@ -413,9 +414,10 @@ def _public_address(host: str) -> bool:
|
||||
return bool(infos)
|
||||
|
||||
|
||||
def _fetch_remote_photo(url: str) -> Optional[int]:
|
||||
def _fetch_remote_photo(url: str) -> Optional[dict]:
|
||||
"""
|
||||
Download a photo hosted elsewhere into the media library. Returns media id.
|
||||
Download a photo hosted elsewhere into the media library. Returns the
|
||||
saved media ({'id', 'path', ...}), or None if it couldn't be fetched.
|
||||
|
||||
Uses the webmention fetcher (v2.9.0): every redirect is checked against
|
||||
private addresses and the connection is pinned to the checked address.
|
||||
@@ -432,7 +434,7 @@ def _fetch_remote_photo(url: str) -> Optional[int]:
|
||||
if result.status_code != 200:
|
||||
raise ValueError(f"HTTP {result.status_code}")
|
||||
name = urlparse(result.url).path.rsplit("/", 1)[-1] or "photo.jpg"
|
||||
return save_media(result.content, name)["id"]
|
||||
return save_media(result.content, name)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Could not fetch photo {url}: {e}")
|
||||
return None
|
||||
@@ -452,7 +454,35 @@ def _media_id_for_url(url: str) -> Optional[int]:
|
||||
return row[0]
|
||||
current_app.logger.warning(f"Photo URL not found in media: {url}")
|
||||
return None
|
||||
return _fetch_remote_photo(url)
|
||||
media = _fetch_remote_photo(url)
|
||||
return media["id"] if media else None
|
||||
|
||||
|
||||
MAX_INLINE_IMAGES = 20
|
||||
|
||||
|
||||
def localize_inline_images(content: str) -> str:
|
||||
"""
|
||||
Copy images embedded in the text from other sites into the media library
|
||||
and point the text at the copies (v2.16.0), so an article doesn't depend
|
||||
on the client's or a CDN's host staying up. Best effort: an image that
|
||||
can't be fetched keeps its URL.
|
||||
"""
|
||||
from html import escape
|
||||
from starpunk.syndication import inline_images
|
||||
|
||||
if not content or "<img" not in content and "![" not in content:
|
||||
return content
|
||||
site_url = current_app.config.get("SITE_URL", "").rstrip("/")
|
||||
remote = [url for url, _ in inline_images(content)
|
||||
if url.startswith(("http://", "https://")) and not url.startswith(site_url + "/")]
|
||||
for url in list(dict.fromkeys(remote))[:MAX_INLINE_IMAGES]:
|
||||
media = _fetch_remote_photo(url)
|
||||
if media:
|
||||
local = f"{site_url}/media/{media['path']}"
|
||||
# As written in Markdown, and as it may be escaped in HTML attributes
|
||||
content = content.replace(url, local).replace(escape(url), local)
|
||||
return content
|
||||
|
||||
|
||||
def _save_uploads(files) -> list[dict]:
|
||||
@@ -602,7 +632,7 @@ def handle_create(data: dict, token_info: dict, files=None):
|
||||
if has_content or not (like_of or repost_of or bookmark_of or photos or title)
|
||||
else ""
|
||||
)
|
||||
content = drop_title_heading(content, title)
|
||||
content = localize_inline_images(drop_title_heading(content, title))
|
||||
# A summary with content is the author's summary; alone it is the
|
||||
# content (Post Type Discovery), as extract_content does (v2.16.0)
|
||||
summary = extract_summary(properties) if _has_text(properties.get("content")) else None
|
||||
@@ -733,7 +763,7 @@ def handle_update(data: dict, token_info: dict):
|
||||
|
||||
if "content" in replace:
|
||||
# Empty text is fine if the note keeps its photos; update_note checks (v2.6.5)
|
||||
body = extract_content(replace) if _has_text(replace["content"]) else ""
|
||||
body = localize_inline_images(extract_content(replace)) if _has_text(replace["content"]) else ""
|
||||
changes["content_format"] = content_format(replace) if body else None
|
||||
if "name" in replace:
|
||||
title = extract_title(replace)
|
||||
|
||||
+7
-2
@@ -298,8 +298,13 @@ class Note:
|
||||
<p>This is my note content...</p>
|
||||
"""
|
||||
if self._cached_html is None:
|
||||
# Render markdown to sanitized HTML (v2.9.0)
|
||||
html = render_markdown(self.content) # triggers lazy load of content
|
||||
if self.content_format == "html":
|
||||
# Sent as HTML via Micropub: sanitize it as is. Through Markdown
|
||||
# it picked up stray <br>s and empty paragraphs (v2.16.0)
|
||||
html = nh3.clean(self.content, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
|
||||
else:
|
||||
# Render markdown to sanitized HTML (v2.9.0)
|
||||
html = render_markdown(self.content) # triggers lazy load of content
|
||||
|
||||
# Cache it
|
||||
object.__setattr__(self, "_cached_html", html)
|
||||
|
||||
+64
-1
@@ -9,11 +9,12 @@ without its images; the admin editor has Title and Summary fields.
|
||||
import json
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import httpx
|
||||
import mf2py
|
||||
|
||||
from starpunk.auth import create_session
|
||||
from starpunk.notes import create_note, get_note
|
||||
from tests.test_micropub_v240 import SITE, app, client, h # noqa: F401
|
||||
from tests.test_micropub_v240 import SITE, app, client, h, jpeg, source # noqa: F401
|
||||
from tests.test_micropub_v265 import upload
|
||||
|
||||
ATOM = {"a": "http://www.w3.org/2005/Atom"}
|
||||
@@ -130,3 +131,65 @@ class TestAdmin:
|
||||
from starpunk.notes import list_notes
|
||||
(note,) = list_notes()
|
||||
assert note.title == "Only a title" and note.content == ""
|
||||
|
||||
|
||||
class TestInlineImages:
|
||||
"""Articles posted with images in their text (v2.16.0)"""
|
||||
|
||||
def remote_host(self, monkeypatch, ok=("/a.jpg",)):
|
||||
fetched = []
|
||||
|
||||
def serve(request): # the fetcher connects to the pinned address, so match on path
|
||||
fetched.append(request.url.path)
|
||||
if request.url.path in ok:
|
||||
return httpx.Response(200, content=jpeg("green"))
|
||||
return httpx.Response(404)
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(serve))
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||
lambda *a, **k: [(2, 1, 6, "", ("93.184.216.34", 0))])
|
||||
return fetched
|
||||
|
||||
def test_html_is_kept_over_text_fallback(self, app, client, h, monkeypatch):
|
||||
self.remote_host(monkeypatch)
|
||||
html = '<p>Intro <a href="https://x.example">link</a></p><img src="https://cdn.example/a.jpg" alt="A">'
|
||||
url = micropub(client, h, name=["T"], content=[{"html": html, "text": "Intro link"}])
|
||||
(content,) = source(client, h, url)["properties"]["content"]
|
||||
assert "html" in content and 'href="https://x.example"' in content["html"]
|
||||
assert "<img" in content["html"]
|
||||
|
||||
def test_remote_images_are_copied(self, app, client, h, monkeypatch):
|
||||
fetched = self.remote_host(monkeypatch)
|
||||
url = micropub(client, h, name=["T"], content=[
|
||||
"Intro\n\n\n\n\n\n"
|
||||
"Again "])
|
||||
with app.app_context():
|
||||
note = get_note(slug=url.rsplit("/", 1)[1])
|
||||
assert fetched.count("/a.jpg") == 1 # each image once
|
||||
assert "https://cdn.example/a.jpg" not in note.content
|
||||
assert note.content.count(f"]({SITE}/media/") == 2
|
||||
assert "https://cdn.example/gone.jpg" in note.content # couldn't fetch: left as is
|
||||
assert client.get(note.content.split(f"]({SITE}")[1].split(")")[0]).status_code == 200
|
||||
|
||||
def test_html_images_copied_and_rendered_cleanly(self, app, client, h, monkeypatch):
|
||||
self.remote_host(monkeypatch)
|
||||
url = micropub(client, h, name=["T"], content=[{"html":
|
||||
'<p>Intro</p><img src="https://cdn.example/a.jpg" alt="A"><p>More</p><script>x()</script>'}])
|
||||
with app.app_context():
|
||||
note = get_note(slug=url.rsplit("/", 1)[1])
|
||||
assert note.content_format == "html"
|
||||
assert note.html.startswith("<p>Intro</p><img") and f'src="{SITE}/media/' in note.html
|
||||
assert "<br" not in note.html and "<p></p>" not in note.html and "script" not in note.html
|
||||
|
||||
def test_own_images_not_fetched(self, app, client, h, monkeypatch):
|
||||
fetched = self.remote_host(monkeypatch)
|
||||
micropub(client, h, content=[f""])
|
||||
assert fetched == []
|
||||
|
||||
def test_update_copies_new_images(self, app, client, h, monkeypatch):
|
||||
self.remote_host(monkeypatch)
|
||||
url = micropub(client, h, name=["T"], content=["Before"])
|
||||
client.post("/micropub", json={"action": "update", "url": url,
|
||||
"replace": {"content": [""]}}, headers=h)
|
||||
with app.app_context():
|
||||
assert f"]({SITE}/media/" in get_note(slug=url.rsplit("/", 1)[1]).content
|
||||
Reference in new issue
Block a user