Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
409b8b31b4 | ||
|
|
b520b8b4d3 | ||
|
|
720a1e6d6f | ||
|
|
c89c2cd5ca | ||
|
|
ab6ff60f5c | ||
|
|
57682165a4 | ||
|
|
8c51b52ed6 |
No files matched your search
@@ -7,6 +7,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [2.10.0] - 2026-11-03
|
||||
|
||||
### Added
|
||||
|
||||
- **WebSub** ([ADR-071](docs/decisions/ADR-071-websub.md)): readers that support it get new posts, edits and deletions within seconds instead of polling. StarPunk runs its own hub at `/websub` for its own feeds: the home page, `/feed.atom`, `/feed.rss`, `/feed.json`, `/likes` and each tag page. Each advertises the hub and itself (`Link` headers, and links inside the page or feed). Subscriptions are verified with a challenge, leases expire, deliveries carry the full feed, signed with `X-Hub-Signature` when the subscriber gave a secret, and failures are retried for about nine hours. Requests to subscribers go through the same private-address protections as webmentions, and there are limits per subscriber host and in total. Set `WEBSUB_HUB_URL` to use another hub instead; `WEBSUB_ENABLED=false` turns it off
|
||||
- The admin dashboard shows WebSub subscribers per feed
|
||||
|
||||
### Fixed
|
||||
|
||||
- Feeds could be up to five minutes out of date after posting, editing or deleting, and each server process cached its own copy. Any change to a public note now refreshes the feeds everywhere at once
|
||||
- The memory monitor stopped after the first request (the same bug the webmention worker had before v2.6.8)
|
||||
|
||||
### Changed
|
||||
|
||||
- The RSS feed calls itself `/feed.rss` instead of the older `/feed.xml`. Both addresses still work
|
||||
|
||||
## [2.9.0] - 2026-11-03
|
||||
|
||||
Micropub review against the [W3C Recommendation](https://www.w3.org/TR/micropub/).
|
||||
|
||||
### Changed
|
||||
|
||||
- **Micropub posts are only crossposted to the targets the client asks for.** If a client doesn't send `mp-syndicate-to`, nothing is crossposted (before, everything was). Posts made in the admin editor are unchanged
|
||||
- Posts sent with `visibility` private or unlisted are saved as drafts instead of being published (StarPunk has no private posts)
|
||||
- A token with only the `draft` scope can create drafts, and nothing else
|
||||
- `q=source` and `q=category` need a token that can post (`create`, `update`, `delete` or `draft`); before, any token could list every post, drafts included
|
||||
- Sending the access token in both the header and the body is refused (400)
|
||||
|
||||
### Added
|
||||
|
||||
- Bookmarks: Micropub `bookmark-of` is stored and shown ("🔖 Bookmarked …", `u-bookmark-of`), returned by `q=source`, listed in `q=config`, and crossposts lead with the bookmarked link (migration 015)
|
||||
- A post with only a `summary` uses it as the text; a post with only a `name` is a titled post with no body
|
||||
|
||||
### Fixed
|
||||
|
||||
- Quill and other JSON clients put `mp-syndicate-to` and `mp-slug` inside `properties`; they were ignored, so posts crossposted everywhere and slugs were not used
|
||||
- HTML in posts is sanitized: scripts, event handlers, iframes and `javascript:` links are removed when notes are shown. Formatting, images, code highlighting, footnotes and microformats are kept
|
||||
- A photo that can't be downloaded or found made the post without it while reporting success. Create and update now fail with 400 and change nothing
|
||||
- Remote photos are fetched the same way as webmentions: every redirect is checked for private addresses and the connection goes to the checked address
|
||||
- JSON values that weren't lists were read a character at a time (`"category": "solo"` became tags s, o, l). Single values are now treated as a one-item list; `delete` values in updates must be lists
|
||||
- Posts written as HTML are returned by `q=source` as `{"html": …}`, as the spec requires; empty `content` is left out
|
||||
- A client-supplied `published` time is stored as UTC like every other post; a date that can't be read is refused instead of silently replaced with now
|
||||
- Malformed JSON gets 400 instead of 500, and error messages no longer include Python internals
|
||||
- `video` and `audio` (not supported) are refused with 400 instead of being dropped; person tags in `category` are ignored instead of failing the post
|
||||
- An update with no `replace`, `add` or `delete` is refused
|
||||
- `Authorization: bearer …` is accepted (the scheme is case-insensitive), and 401 responses include `WWW-Authenticate`
|
||||
|
||||
## [2.8.1] - 2026-11-03
|
||||
|
||||
### Fixed
|
||||
|
||||
- Starting the server after an upgrade with a database migration could fail: several server processes applied the same migration at once and one stopped with "duplicate column name", taking the server down until it restarted (this happened deploying v2.8.0, for about two seconds). The migration lock was released right after it was taken, and again before each migration was recorded. Migrations now run entirely inside the lock
|
||||
|
||||
## [2.8.0] - 2026-11-03
|
||||
|
||||
Microformats2 review: the markup now matches the [microformats2](https://microformats.org/wiki/microformats2) vocabularies, so IndieWeb readers and tools see what the page shows.
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
# ADR-071: WebSub publishing with a built-in hub
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (v2.10.0, 2026-11-03)
|
||||
|
||||
## Context
|
||||
|
||||
Readers poll StarPunk's feeds. WebSub (W3C Recommendation,
|
||||
https://www.w3.org/TR/websub/) lets readers that support it (Monocle,
|
||||
Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and
|
||||
have new content pushed to them within seconds.
|
||||
|
||||
WebSub has three roles. StarPunk is the **publisher**. Someone has to run
|
||||
the **hub** that keeps subscriptions and delivers content. Public hubs
|
||||
exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a
|
||||
third-party dependency and some have shut down. The rest of this site is
|
||||
self-hosted.
|
||||
|
||||
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
|
||||
|
||||
Two existing problems matter here:
|
||||
|
||||
- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears
|
||||
the cache when a note changes. A hub fetching right after a change would
|
||||
get stale content. Regular readers are also up to 5 minutes behind.
|
||||
- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and
|
||||
calls itself `/feed.xml`.
|
||||
|
||||
## Decision
|
||||
|
||||
StarPunk runs its own hub, **for its own topics only**, at `/websub`.
|
||||
`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk
|
||||
advertises that hub, pings it on changes, and its own `/websub` is off.
|
||||
|
||||
### 1. Topics
|
||||
|
||||
Absolute URLs under `SITE_URL`:
|
||||
|
||||
| Topic | Content type |
|
||||
|---|---|
|
||||
| `/` (h-feed) | text/html |
|
||||
| `/feed.atom` | application/atom+xml |
|
||||
| `/feed.rss` | application/rss+xml |
|
||||
| `/feed.json` | application/feed+json |
|
||||
| `/likes` | text/html |
|
||||
| `/tag/<name>` | text/html |
|
||||
|
||||
`/feed.xml` and the negotiated `/feed` advertise the topic of the format
|
||||
they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The
|
||||
RSS feed's own self link changes from `/feed.xml` to `/feed.rss`.
|
||||
|
||||
### 2. Discovery (publisher, spec 4)
|
||||
|
||||
Every topic response carries `Link: <hub>; rel="hub"` and exactly one
|
||||
`Link: <topic>; rel="self"`. Embedded links as well: `<link rel="hub">` and
|
||||
`<link rel="self">` in the HTML head of topic pages, `<link rel="hub">`
|
||||
next to the existing self link in Atom, `<atom:link rel="hub">` in RSS, and
|
||||
`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed.
|
||||
|
||||
### 3. What changed, and when
|
||||
|
||||
`websub.note_changed(note, old_tags=())` is called wherever
|
||||
`syndication.note_changed` is, plus on create. It:
|
||||
|
||||
1. clears the feed cache (all formats), and
|
||||
2. if the note is or was public, queues delivery for `/`, the three
|
||||
feeds, `/likes` if it is a like, and each `/tag/<name>` for its tags
|
||||
before and after the change.
|
||||
|
||||
Drafts that stay drafts change nothing.
|
||||
|
||||
### 4. Hub endpoint (spec 5)
|
||||
|
||||
`POST /websub`, `application/x-www-form-urlencoded`:
|
||||
|
||||
- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the
|
||||
topics above (after normalising), else 400 with a plain-text reason;
|
||||
`hub.callback` must be an absolute http(s) URL that passes the SSRF
|
||||
check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to
|
||||
1 day to 30 days (default 10 days). Unknown parameters are ignored.
|
||||
- Response: 202 Accepted, independent of what verification later finds.
|
||||
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions
|
||||
per callback host and 1,000 in total; beyond that, 429.
|
||||
|
||||
### 5. Verification of intent (spec 5.3)
|
||||
|
||||
A background worker sends `GET <callback>` with the callback's query
|
||||
string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32
|
||||
bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body
|
||||
equal to the challenge; anything else, including redirects, fails and
|
||||
leaves the subscription state unchanged. On success a subscribe becomes
|
||||
active with `expires_at = now + lease`, and an unsubscribe deletes it.
|
||||
A re-subscribe replaces the secret and lease only once verified.
|
||||
|
||||
### 6. Content distribution (spec 6, 7)
|
||||
|
||||
For each change, one delivery per active, unexpired subscription to the
|
||||
topic. The worker renders the topic inside the app (no HTTP round trip
|
||||
through Caddy), so subscribers get exactly what the public URL serves,
|
||||
then `POST`s it to the callback with:
|
||||
|
||||
- `Content-Type` of the topic,
|
||||
- `Link` headers with `rel=hub` and `rel=self`,
|
||||
- `X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>` if a secret was
|
||||
given.
|
||||
|
||||
A 2xx is success. 410 Gone removes the subscription. Anything else is
|
||||
retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a
|
||||
subscription whose deliveries fail for 3 days is removed. Only the latest
|
||||
content is sent: a queued delivery for a topic is replaced by a newer one.
|
||||
|
||||
### 7. Outbound safety
|
||||
|
||||
Callbacks are arbitrary URLs, so verification and delivery go through
|
||||
`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the
|
||||
connection pinned to the checked address, no redirects, 10 s time limit,
|
||||
1 MB response cap.
|
||||
|
||||
### 8. Data (migration 016)
|
||||
|
||||
- `websub_subscriptions(id, topic, callback, secret, lease_seconds,
|
||||
state ['pending','active'], expires_at, created_at, verified_at,
|
||||
last_success_at, last_error)`, unique on `(topic, callback)`.
|
||||
- `websub_intents(id, topic, callback, mode, secret, lease_seconds,
|
||||
attempts, next_attempt_at, created_at)`: verifications to do.
|
||||
- `websub_deliveries(id, subscription_id, attempts, next_attempt_at,
|
||||
last_error, created_at)`, unique on `subscription_id`.
|
||||
|
||||
### 9. Worker
|
||||
|
||||
The same pattern as the Webmention worker: one thread per gunicorn
|
||||
process, a file lock so only one process works the queues, app context
|
||||
around each poll. Expired subscriptions are pruned once a day.
|
||||
|
||||
### 10. Admin
|
||||
|
||||
The admin dashboard shows active subscribers per topic and failing
|
||||
deliveries.
|
||||
|
||||
### 11. Configuration
|
||||
|
||||
`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in),
|
||||
`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Readers that support WebSub get posts and edits within seconds.
|
||||
- Feeds are fresh right after a change for every reader, because the cache
|
||||
is cleared.
|
||||
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF
|
||||
protections, per-host and total limits, and rate limiting bound that.
|
||||
- websub.rocks' hub tests use websub.rocks' own topics, which this hub
|
||||
refuses by design. Testing uses unit tests of each spec section and a
|
||||
real subscription from a reader after deploy.
|
||||
- The RSS self link changes to `/feed.rss`. Readers keep whichever URL
|
||||
they subscribed to; both keep working.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Public hub only.** Least code, but a third-party dependency for a
|
||||
self-hosted site. Still available through `WEBSUB_HUB_URL`.
|
||||
- **A separate hub service** (another container). More to deploy and
|
||||
keep updated, and few maintained hubs exist.
|
||||
- **Accepting any topic** (a general-purpose hub). Not needed, and it
|
||||
would let the hub be used to fetch and push arbitrary URLs.
|
||||
@@ -0,0 +1,33 @@
|
||||
# v2.8.1: Migration lock held for the whole run
|
||||
|
||||
## Incident
|
||||
|
||||
Deploying v2.8.0 (migration 014 adds two columns): one gunicorn worker
|
||||
applied 014, another failed with `duplicate column name: source_post_url`,
|
||||
gunicorn shut down ("Worker failed to boot") and Docker restarted the
|
||||
container. About two seconds of downtime; schema correct afterwards.
|
||||
|
||||
## Causes
|
||||
|
||||
`run_migrations` takes `BEGIN IMMEDIATE` so one worker migrates at a time,
|
||||
but the lock was released early twice:
|
||||
|
||||
1. `create_migrations_table()`, called right after `BEGIN IMMEDIATE`,
|
||||
ended with `conn.commit()`. Everything after it ran without the lock.
|
||||
2. `conn.executescript()` COMMITs any open transaction before running the
|
||||
script, so the lock was dropped before the migration was recorded in
|
||||
`schema_migrations`.
|
||||
|
||||
## Fix
|
||||
|
||||
- `create_migrations_table(conn, commit=True)`; `run_migrations` passes
|
||||
`commit=False`.
|
||||
- `execute_in_transaction()` runs each statement from
|
||||
`split_sql_statements()` (built on `sqlite3.complete_statement`, so
|
||||
trigger bodies stay whole) with `conn.execute`, inside the transaction.
|
||||
|
||||
## Verification
|
||||
|
||||
Four processes migrating a database at 013 simultaneously: 6/30 trials
|
||||
failed before, 0/30 after. `tests/test_migration_concurrency.py` runs this
|
||||
five times, plus a rollback test showing a migration commits nothing early.
|
||||
@@ -0,0 +1,51 @@
|
||||
# v2.9.0: Micropub review
|
||||
|
||||
Review against the W3C Micropub Recommendation (https://www.w3.org/TR/micropub/),
|
||||
run through the Flask test client and mapped to the micropub.rocks server
|
||||
tests. Everyday create/update/delete/query/media/auth already passed; the
|
||||
gaps were privacy, security and silent data loss.
|
||||
|
||||
## Decisions (user)
|
||||
|
||||
- No `mp-syndicate-to` from a Micropub client means no crossposting
|
||||
(was: all targets, v2.1.0). The admin editor's default is unchanged.
|
||||
- `visibility` private/unlisted are saved as drafts.
|
||||
- Rendered HTML is sanitized with an allowlist (nh3 defaults plus class,
|
||||
title, id, lang, dir on all tags; video/audio/source/track/picture).
|
||||
|
||||
## Changes
|
||||
|
||||
| Area | Before | After |
|
||||
|---|---|---|
|
||||
| `mp-*` in JSON `properties` | ignored (all targets, timestamp slug) | read as commands, never stored |
|
||||
| absent `mp-syndicate-to` | all targets | none |
|
||||
| `q=source` / `q=category` | any token | create, update, delete or draft scope |
|
||||
| HTML content | rendered raw (stored XSS) | `render_markdown()` = Markdown + nh3 |
|
||||
| `visibility` | ignored (published) | private/unlisted -> draft; unknown -> 400 |
|
||||
| `draft` scope | 403 | creates drafts only |
|
||||
| photo fetch/lookup failure | dropped, 201 | 400, nothing created or changed |
|
||||
| remote photo fetch | first host checked, redirects followed | `webmention_ssrf.safe_get` (per hop, pinned) |
|
||||
| JSON scalars | indexed per character | wrapped on create; update `delete` values must be arrays |
|
||||
| `bookmark-of` | dropped | `notes.bookmark_of`, kind `bookmark` (migration 015) |
|
||||
| HTML-authored content in `q=source` | string | `{"html": ...}` via `notes.content_format` |
|
||||
| `summary` / `name` only | 400 | used as text / titled post |
|
||||
| `video` / `audio` | dropped | 400 |
|
||||
| `published` | stored with offset; garbage -> now | naive UTC; garbage -> 400 |
|
||||
| malformed JSON | 500 | 400 |
|
||||
| update without operations | 204 | 400 |
|
||||
| auth | `Bearer` only; header+body accepted; no challenge | case-insensitive; both -> 400; `WWW-Authenticate` |
|
||||
|
||||
## Kept as is
|
||||
|
||||
- 403 (RFC 6750) for insufficient_scope; the Micropub table says 401.
|
||||
- `add` only for category and photo; `published`, `syndication`,
|
||||
`location` not updatable.
|
||||
- `location`/`checkin` ignored (3.3.5 allows).
|
||||
- GET `access_token` query parameter still accepted.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/test_micropub_v290.py` (31 of 34 fail on v2.8.1). Updated:
|
||||
likes/reposts helper sends `mp-syndicate-to` in properties;
|
||||
`test_micropub_default_is_no_targets`; remote photo test uses the pinned
|
||||
fetcher; corrupted-image message.
|
||||
@@ -0,0 +1,7 @@
|
||||
-- Migration 015: Bookmarks and HTML-authored content (v2.9.0)
|
||||
-- bookmark_of: URL bookmarked (Micropub bookmark-of); the note is a bookmark.
|
||||
-- content_format: 'html' when a Micropub client sent the content as HTML,
|
||||
-- so q=source returns it as {"html": ...} (Micropub 3.7.2). NULL = Markdown.
|
||||
|
||||
ALTER TABLE notes ADD COLUMN bookmark_of TEXT;
|
||||
ALTER TABLE notes ADD COLUMN content_format TEXT;
|
||||
@@ -0,0 +1,55 @@
|
||||
-- Migration 016: WebSub hub (v2.10.0, ADR-071)
|
||||
|
||||
-- Active and pending subscriptions to this site's topics
|
||||
CREATE TABLE IF NOT EXISTS websub_subscriptions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
topic TEXT NOT NULL,
|
||||
callback TEXT NOT NULL,
|
||||
secret TEXT,
|
||||
lease_seconds INTEGER NOT NULL,
|
||||
expires_at TIMESTAMP NOT NULL,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
verified_at TIMESTAMP,
|
||||
last_success_at TIMESTAMP,
|
||||
last_error TEXT,
|
||||
UNIQUE(topic, callback)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_topic ON websub_subscriptions(topic);
|
||||
|
||||
-- Subscribe/unsubscribe requests waiting for verification of intent
|
||||
CREATE TABLE IF NOT EXISTS websub_intents (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
topic TEXT NOT NULL,
|
||||
callback TEXT NOT NULL,
|
||||
mode TEXT NOT NULL CHECK (mode IN ('subscribe', 'unsubscribe')),
|
||||
secret TEXT,
|
||||
lease_seconds INTEGER,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(topic, callback)
|
||||
);
|
||||
|
||||
-- Content to push: one row per subscription (built-in hub) or per topic
|
||||
-- (ping to an external hub, subscription_id NULL)
|
||||
CREATE TABLE IF NOT EXISTS websub_deliveries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
subscription_id INTEGER REFERENCES websub_subscriptions(id) ON DELETE CASCADE,
|
||||
topic TEXT NOT NULL,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
last_error TEXT,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_subscription
|
||||
ON websub_deliveries(subscription_id) WHERE subscription_id IS NOT NULL;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_ping
|
||||
ON websub_deliveries(topic) WHERE subscription_id IS NULL;
|
||||
|
||||
-- Site-wide values. content_version changes whenever a public note does, so
|
||||
-- every gunicorn process knows its cached feed note list is stale.
|
||||
CREATE TABLE IF NOT EXISTS site_state (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
INSERT OR IGNORE INTO site_state (key, value) VALUES ('content_version', '0');
|
||||
+15
-8
@@ -309,6 +309,11 @@ def create_app(config=None):
|
||||
response.headers.add('Link', '</webmention>; rel="webmention"')
|
||||
return response
|
||||
|
||||
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
|
||||
from starpunk.websub import add_discovery_headers, template_links
|
||||
app.after_request(add_discovery_headers)
|
||||
app.add_template_global(template_links, 'websub_links')
|
||||
|
||||
# Register centralized error handlers
|
||||
from starpunk.errors import register_error_handlers
|
||||
|
||||
@@ -322,12 +327,8 @@ def create_app(config=None):
|
||||
memory_monitor.start()
|
||||
app.memory_monitor = memory_monitor
|
||||
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
|
||||
|
||||
# Register cleanup handler
|
||||
@app.teardown_appcontext
|
||||
def cleanup_memory_monitor(error=None):
|
||||
if hasattr(app, 'memory_monitor') and app.memory_monitor.is_alive():
|
||||
app.memory_monitor.stop()
|
||||
# No teardown_appcontext stop: that ran after every request and stopped
|
||||
# the monitor straight away (v2.10.0). It is a daemon thread.
|
||||
|
||||
# Start webmention worker thread (v1.6.0 Phase 2)
|
||||
# Per background-worker.md: Skip in test mode
|
||||
@@ -350,6 +351,12 @@ def create_app(config=None):
|
||||
f"batch_size={webmention_worker.batch_size})"
|
||||
)
|
||||
|
||||
# Start WebSub worker (v2.10.0)
|
||||
from starpunk.websub_worker import start_worker
|
||||
app.websub_worker = start_worker(app)
|
||||
if app.websub_worker:
|
||||
app.logger.info("WebSub worker started")
|
||||
|
||||
# Start syndication worker when a target is configured (v2.1.0)
|
||||
from starpunk.syndication import SyndicationWorker, targets_configured
|
||||
if targets_configured(app.config) and not app.config.get("TESTING", False):
|
||||
@@ -463,5 +470,5 @@ def create_app(config=None):
|
||||
|
||||
# Package version (Semantic Versioning 2.0.0)
|
||||
# See docs/standards/versioning-strategy.md for details
|
||||
__version__ = "2.8.0"
|
||||
__version_info__ = (2, 8, 0)
|
||||
__version__ = "2.10.0"
|
||||
__version_info__ = (2, 10, 0)
|
||||
@@ -111,6 +111,13 @@ def load_config(app, config_override=None):
|
||||
|
||||
# Webmention configuration (v1.6.0)
|
||||
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
|
||||
|
||||
# WebSub (v2.10.0, ADR-071): built-in hub unless WEBSUB_HUB_URL names another
|
||||
app.config["WEBSUB_ENABLED"] = os.getenv("WEBSUB_ENABLED", "true").lower() == "true"
|
||||
app.config["WEBSUB_HUB_URL"] = os.getenv("WEBSUB_HUB_URL", "").strip()
|
||||
app.config["WEBSUB_MAX_SUBSCRIPTIONS"] = int(os.getenv("WEBSUB_MAX_SUBSCRIPTIONS", "1000"))
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = int(os.getenv("WEBSUB_MAX_PER_HOST", "25"))
|
||||
app.config["WEBSUB_POLL_INTERVAL"] = int(os.getenv("WEBSUB_POLL_INTERVAL", "5"))
|
||||
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
|
||||
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
|
||||
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
|
||||
|
||||
@@ -93,7 +93,7 @@ def generate_rss(
|
||||
fg.language("en")
|
||||
|
||||
# Add self-link for feed discovery (Atom namespace)
|
||||
fg.link(href=f"{site_url}/feed.xml", rel="self", type="application/rss+xml")
|
||||
fg.link(href=f"{site_url}/feed.rss", rel="self", type="application/rss+xml")
|
||||
|
||||
# Set last build date to now
|
||||
fg.lastBuildDate(datetime.now(timezone.utc))
|
||||
@@ -246,7 +246,7 @@ def generate_rss_streaming(
|
||||
yield f" <description>{_escape_xml(site_description or site_name)}</description>\n"
|
||||
yield " <language>en</language>\n"
|
||||
yield f" <lastBuildDate>{last_build}</lastBuildDate>\n"
|
||||
yield f' <atom:link href="{_escape_xml(site_url)}/feed.xml" rel="self" type="application/rss+xml"/>\n'
|
||||
yield f' <atom:link href="{_escape_xml(site_url)}/feed.rss" rel="self" type="application/rss+xml"/>\n'
|
||||
|
||||
# Yield items (newest first)
|
||||
# Notes from database are already in DESC order (newest first)
|
||||
|
||||
+6
-2
@@ -93,6 +93,10 @@ def get_optimization_params(file_size: int) -> Tuple[int, int]:
|
||||
return (1280, 85)
|
||||
|
||||
|
||||
# Shown to clients; the parser's own message names internal objects (v2.9.0)
|
||||
UNSUPPORTED_IMAGE = "Not a supported image: use JPEG, PNG, GIF, WebP or HEIC"
|
||||
|
||||
|
||||
def validate_image(file_data: bytes, filename: str) -> Tuple[Image.Image, str, int, int]:
|
||||
"""
|
||||
Validate image file and return PIL Image for memory-efficient processing
|
||||
@@ -158,9 +162,9 @@ def validate_image(file_data: bytes, filename: str) -> Tuple[Image.Image, str, i
|
||||
current_app.logger.info(f'Saved failed upload for analysis: {debug_file}')
|
||||
except RuntimeError:
|
||||
pass # Outside app context (e.g., tests)
|
||||
raise ValueError(f"Invalid or corrupted image: {e}")
|
||||
raise ValueError(UNSUPPORTED_IMAGE) from e
|
||||
else:
|
||||
raise ValueError(f"Invalid or corrupted image: {e}")
|
||||
raise ValueError(UNSUPPORTED_IMAGE) from e
|
||||
|
||||
# Store original format before any conversions
|
||||
original_format = img.format
|
||||
|
||||
+158
-80
@@ -25,7 +25,7 @@ References:
|
||||
- IndieAuth Specification: https://www.w3.org/TR/indieauth/
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from flask import Request, current_app, jsonify
|
||||
@@ -81,9 +81,17 @@ def error_response(error: str, error_description: str, status_code: int = 400):
|
||||
Returns:
|
||||
Tuple of (response, status_code)
|
||||
"""
|
||||
headers = {}
|
||||
if status_code == 401:
|
||||
# RFC 6750 3: tell the client how to authenticate (v2.9.0)
|
||||
challenge = 'Bearer realm="micropub"'
|
||||
if error == "unauthorized" and "Invalid" in error_description:
|
||||
challenge += ', error="invalid_token"'
|
||||
headers["WWW-Authenticate"] = challenge
|
||||
return (
|
||||
jsonify({"error": error, "error_description": error_description}),
|
||||
status_code,
|
||||
headers,
|
||||
)
|
||||
|
||||
|
||||
@@ -104,18 +112,21 @@ def extract_bearer_token(request: Request) -> Optional[str]:
|
||||
Returns:
|
||||
Token string if found, None otherwise
|
||||
"""
|
||||
# Try Authorization header first
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
if auth_header.startswith("Bearer "):
|
||||
return auth_header[7:] # Remove "Bearer " prefix
|
||||
# The scheme is case-insensitive (RFC 7235)
|
||||
scheme, _, credentials = request.headers.get("Authorization", "").partition(" ")
|
||||
header_token = credentials.strip() if scheme.lower() == "bearer" else None
|
||||
|
||||
# Try form parameter
|
||||
body_token = None
|
||||
if request.method == "POST":
|
||||
return request.form.get("access_token")
|
||||
body_token = request.form.get("access_token")
|
||||
elif request.method == "GET":
|
||||
return request.args.get("access_token")
|
||||
body_token = request.args.get("access_token")
|
||||
|
||||
return None
|
||||
# Only one method may be used (RFC 6750 2) (v2.9.0)
|
||||
if header_token and body_token:
|
||||
raise MicropubValidationError("Send the access token in the header or the body, not both")
|
||||
|
||||
return header_token or body_token
|
||||
|
||||
|
||||
# Property Normalization
|
||||
@@ -135,9 +146,18 @@ def normalize_properties(data: dict) -> dict:
|
||||
Returns:
|
||||
Normalized properties dict with all values as lists
|
||||
"""
|
||||
# JSON format has properties nested
|
||||
# JSON format has properties nested. Values MUST be arrays (3.3.2); a
|
||||
# bare value is wrapped rather than read character by character.
|
||||
# mp-* commands and access_token are not properties (3.2).
|
||||
if "properties" in data:
|
||||
return data["properties"]
|
||||
properties = data["properties"]
|
||||
if not isinstance(properties, dict):
|
||||
raise MicropubValidationError("properties must be an object")
|
||||
return {
|
||||
key: value if isinstance(value, list) else [value]
|
||||
for key, value in properties.items()
|
||||
if not key.startswith("mp-") and key != "access_token"
|
||||
}
|
||||
|
||||
# Form format - convert to properties dict
|
||||
properties = {}
|
||||
@@ -176,7 +196,7 @@ def extract_content(properties: dict) -> str:
|
||||
Raises:
|
||||
MicropubValidationError: If content is missing or empty
|
||||
"""
|
||||
content_list = properties.get("content", [])
|
||||
content_list = properties.get("content") or properties.get("summary") or []
|
||||
|
||||
# Handle both plain text and HTML/text objects
|
||||
if not content_list:
|
||||
@@ -189,12 +209,20 @@ def extract_content(properties: dict) -> str:
|
||||
# Prefer text over html for markdown storage
|
||||
content = content.get("text") or content.get("html", "")
|
||||
|
||||
if not content or not content.strip():
|
||||
if not isinstance(content, str) or not content.strip():
|
||||
raise MicropubValidationError("Content cannot be empty")
|
||||
|
||||
return content.strip()
|
||||
|
||||
|
||||
def content_format(properties: dict) -> Optional[str]:
|
||||
"""'html' if the content was sent only as HTML (returned as such by q=source)"""
|
||||
content = _first(properties.get("content"))
|
||||
if isinstance(content, dict) and content.get("html") and not content.get("text"):
|
||||
return "html"
|
||||
return None
|
||||
|
||||
|
||||
def _has_text(content) -> bool:
|
||||
"""Whether a content property has any text. Likes, reposts (v2.5.0) and
|
||||
photo posts (v2.6.5) may come without."""
|
||||
@@ -245,8 +273,9 @@ def extract_tags(properties: dict) -> list[str]:
|
||||
List of tag strings
|
||||
"""
|
||||
categories = properties.get("category", [])
|
||||
# Filter out empty strings and strip whitespace
|
||||
return [tag.strip() for tag in categories if tag and tag.strip()]
|
||||
# Strings only: person tags (h-card objects) aren't supported and are
|
||||
# ignored, as Micropub says to ignore what isn't recognised (3.3.5)
|
||||
return [tag.strip() for tag in categories if isinstance(tag, str) and tag.strip()]
|
||||
|
||||
|
||||
def extract_published_date(properties: dict) -> Optional[datetime]:
|
||||
@@ -259,18 +288,18 @@ def extract_published_date(properties: dict) -> Optional[datetime]:
|
||||
Returns:
|
||||
Datetime object if published date provided, None otherwise
|
||||
"""
|
||||
published = properties.get("published", [""])[0]
|
||||
published = _first(properties.get("published"))
|
||||
if not published:
|
||||
return None
|
||||
|
||||
try:
|
||||
# Parse ISO 8601 datetime
|
||||
# datetime.fromisoformat handles most ISO formats
|
||||
return datetime.fromisoformat(published.replace("Z", "+00:00"))
|
||||
except (ValueError, AttributeError):
|
||||
# If parsing fails, log and return None (will use current time)
|
||||
current_app.logger.warning(f"Failed to parse published date: {published}")
|
||||
return None
|
||||
value = datetime.fromisoformat(str(published).strip().replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
raise MicropubValidationError(f"published is not an ISO 8601 date: {published}")
|
||||
# Stored times are naive UTC, like every other note (v2.9.0)
|
||||
if value.tzinfo is not None:
|
||||
value = value.astimezone(timezone.utc).replace(tzinfo=None)
|
||||
return value
|
||||
|
||||
|
||||
# Action Handlers
|
||||
@@ -306,6 +335,24 @@ def extract_post_status(properties: dict) -> bool:
|
||||
raise MicropubValidationError(f"Unsupported post-status: {status}")
|
||||
|
||||
|
||||
def extract_visibility_draft(properties: dict) -> bool:
|
||||
"""
|
||||
True if visibility asks for a non-public post (v2.9.0)
|
||||
|
||||
StarPunk only has public posts and drafts, so private and unlisted posts
|
||||
are kept as drafts rather than published.
|
||||
"""
|
||||
visibility = _first(properties.get("visibility"))
|
||||
if visibility in (None, "", "public"):
|
||||
return False
|
||||
if visibility in ("private", "unlisted"):
|
||||
return True
|
||||
raise MicropubValidationError(f"Unsupported visibility: {visibility}")
|
||||
|
||||
|
||||
UNSUPPORTED_MEDIA = ("video", "audio")
|
||||
|
||||
|
||||
def extract_url_property(properties: dict, name: str) -> Optional[str]:
|
||||
"""A URL property (in-reply-to, like-of, repost-of): string or h-cite"""
|
||||
value = _first(properties.get(name))
|
||||
@@ -361,27 +408,25 @@ def _public_address(host: str) -> bool:
|
||||
|
||||
|
||||
def _fetch_remote_photo(url: str) -> Optional[int]:
|
||||
"""Download a photo hosted elsewhere into the media library. Returns media id."""
|
||||
"""
|
||||
Download a photo hosted elsewhere into the media library. Returns media id.
|
||||
|
||||
Uses the webmention fetcher (v2.9.0): every redirect is checked against
|
||||
private addresses and the connection is pinned to the checked address.
|
||||
"""
|
||||
from urllib.parse import urlparse
|
||||
import httpx
|
||||
from starpunk.media import save_media
|
||||
from starpunk.webmention_ssrf import safe_get, user_agent
|
||||
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||
return None
|
||||
if not current_app.config.get("TESTING") and not _public_address(parsed.hostname):
|
||||
current_app.logger.warning(f"Refusing to fetch photo from non-public address: {url}")
|
||||
return None
|
||||
try:
|
||||
with httpx.stream("GET", url, timeout=20.0, follow_redirects=True) as response:
|
||||
response.raise_for_status()
|
||||
data = bytearray()
|
||||
for chunk in response.iter_bytes():
|
||||
data += chunk
|
||||
if len(data) > MAX_REMOTE_PHOTO_BYTES:
|
||||
raise ValueError("photo too large")
|
||||
name = parsed.path.rsplit("/", 1)[-1] or "photo.jpg"
|
||||
return save_media(bytes(data), name)["id"]
|
||||
result = safe_get(
|
||||
url, headers={"User-Agent": user_agent(current_app.config.get("SITE_URL", ""))},
|
||||
max_bytes=MAX_REMOTE_PHOTO_BYTES, timeout=20,
|
||||
)
|
||||
if result.status_code != 200:
|
||||
raise ValueError(f"HTTP {result.status_code}")
|
||||
name = urlparse(result.url).path.rsplit("/", 1)[-1] or "photo.jpg"
|
||||
return save_media(result.content, name)["id"]
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Could not fetch photo {url}: {e}")
|
||||
return None
|
||||
@@ -430,8 +475,11 @@ def _resolve_photos(photos: list[dict]) -> tuple[list[int], list[str]]:
|
||||
if len(media_ids) == MAX_PHOTOS:
|
||||
break
|
||||
media_id = photo.get("media_id") or _media_id_for_url(photo["url"])
|
||||
if not media_id:
|
||||
# Don't create or change the post without it (v2.9.0)
|
||||
raise MicropubValidationError(f"Could not get photo {photo.get('url')}")
|
||||
# The same photo twice is shown once (v2.6.5)
|
||||
if media_id and media_id not in media_ids:
|
||||
if media_id not in media_ids:
|
||||
media_ids.append(media_id)
|
||||
captions.append(photo.get("alt", ""))
|
||||
return media_ids, captions
|
||||
@@ -442,8 +490,7 @@ def _attach_photos_to_note(note_id: int, photos: list[dict]) -> None:
|
||||
from starpunk.media import attach_media_to_note
|
||||
|
||||
media_ids, captions = _resolve_photos(photos)
|
||||
if media_ids or not photos:
|
||||
attach_media_to_note(note_id, media_ids, captions)
|
||||
attach_media_to_note(note_id, media_ids, captions)
|
||||
|
||||
|
||||
def _note_photos(note) -> list[dict]:
|
||||
@@ -508,47 +555,54 @@ def handle_create(data: dict, token_info: dict, files=None):
|
||||
Returns:
|
||||
Tuple of (response_body, status_code, headers)
|
||||
"""
|
||||
_require_scope(token_info, "create")
|
||||
_require_scope(token_info, "create", "draft")
|
||||
# A token with only the draft scope may create drafts (v2.9.0)
|
||||
drafts_only = not check_scope("create", token_info.get("scope", ""))
|
||||
_check_type(data)
|
||||
|
||||
# mp-slug and mp-syndicate-to are server commands, not properties
|
||||
custom_slug = None
|
||||
if isinstance(data, dict) and 'mp-slug' in data:
|
||||
slug_value = data.get('mp-slug')
|
||||
if isinstance(slug_value, list) and slug_value:
|
||||
custom_slug = slug_value[0]
|
||||
elif isinstance(slug_value, str):
|
||||
custom_slug = slug_value
|
||||
# mp-slug and mp-syndicate-to are server commands, not properties. Form
|
||||
# posts send them at the top level; JSON clients inside properties too.
|
||||
commands = dict(data) if isinstance(data, dict) else {}
|
||||
if isinstance(commands.get("properties"), dict):
|
||||
commands.update({k: v for k, v in commands["properties"].items() if k.startswith("mp-")})
|
||||
|
||||
# Absent means "all targets"; present means exactly those (v2.1.0)
|
||||
requested_targets = None
|
||||
if isinstance(data, dict):
|
||||
for key in ("mp-syndicate-to", "mp-syndicate-to[]"):
|
||||
if key in data:
|
||||
value = data[key]
|
||||
requested_targets = (requested_targets or []) + (
|
||||
value if isinstance(value, list) else [value]
|
||||
)
|
||||
custom_slug = _first(commands.get("mp-slug"))
|
||||
if custom_slug is not None and not isinstance(custom_slug, str):
|
||||
raise MicropubValidationError("mp-slug must be a string")
|
||||
|
||||
# Crossposting only to the targets asked for: absent means none (v2.9.0)
|
||||
requested_targets = []
|
||||
for key in ("mp-syndicate-to", "mp-syndicate-to[]"):
|
||||
if key in commands:
|
||||
value = commands[key]
|
||||
requested_targets += value if isinstance(value, list) else [value]
|
||||
|
||||
try:
|
||||
properties = normalize_properties(data)
|
||||
like_of = extract_url_property(properties, "like-of")
|
||||
repost_of = extract_url_property(properties, "repost-of")
|
||||
if like_of and repost_of:
|
||||
raise MicropubValidationError("A post can't be both a like and a repost")
|
||||
has_content = _has_text(properties.get("content"))
|
||||
bookmark_of = extract_url_property(properties, "bookmark-of")
|
||||
if sum(bool(u) for u in (like_of, repost_of, bookmark_of)) > 1:
|
||||
raise MicropubValidationError("A post can only be one of a like, repost or bookmark")
|
||||
for prop in UNSUPPORTED_MEDIA:
|
||||
if properties.get(prop) or (files and (files.getlist(prop) or files.getlist(f"{prop}[]"))):
|
||||
raise MicropubValidationError(f"{prop} posts are not supported")
|
||||
has_content = _has_text(properties.get("content")) or _has_text(properties.get("summary"))
|
||||
photos = extract_photos(properties) + _save_uploads(files)
|
||||
# Photo posts may have no text (v2.6.5): Micropub doesn't require content
|
||||
title = extract_title(properties)
|
||||
# Photo posts may have no text (v2.6.5), nor titled posts (v2.9.0)
|
||||
content = (
|
||||
extract_content(properties)
|
||||
if has_content or not (like_of or repost_of or photos)
|
||||
if has_content or not (like_of or repost_of or bookmark_of or photos or title)
|
||||
else ""
|
||||
)
|
||||
title = extract_title(properties)
|
||||
tags = extract_tags(properties)
|
||||
published_date = extract_published_date(properties)
|
||||
published = extract_post_status(properties)
|
||||
published = extract_post_status(properties) and not extract_visibility_draft(properties)
|
||||
if drafts_only:
|
||||
published = False
|
||||
in_reply_to = extract_in_reply_to(properties)
|
||||
fmt = content_format(properties) if has_content else None
|
||||
# Look up and download photos before the note exists, so a bad photo
|
||||
# can't leave a half-made post behind (v2.6.5)
|
||||
media_ids, captions = _resolve_photos(photos)
|
||||
@@ -556,7 +610,7 @@ def handle_create(data: dict, token_info: dict, files=None):
|
||||
raise
|
||||
except Exception as e:
|
||||
current_app.logger.error(f"Property extraction failed: {e}")
|
||||
raise MicropubValidationError(f"Failed to parse request: {str(e)}")
|
||||
raise MicropubValidationError("Could not read the post's properties")
|
||||
|
||||
note = None
|
||||
try:
|
||||
@@ -569,7 +623,9 @@ def handle_create(data: dict, token_info: dict, files=None):
|
||||
in_reply_to=in_reply_to,
|
||||
like_of=like_of,
|
||||
repost_of=repost_of,
|
||||
bookmark_of=bookmark_of,
|
||||
has_media=bool(media_ids),
|
||||
content_format=fmt,
|
||||
)
|
||||
|
||||
if media_ids:
|
||||
@@ -577,11 +633,8 @@ def handle_create(data: dict, token_info: dict, files=None):
|
||||
attach_media_to_note(note.id, media_ids, captions)
|
||||
|
||||
# Crosspost after photos are attached (v2.1.0); drafts wait until published
|
||||
from starpunk.syndication import get_targets, resolve_targets, select_targets
|
||||
if requested_targets is None:
|
||||
select_targets(note.id, list(get_targets()))
|
||||
else:
|
||||
select_targets(note.id, resolve_targets(requested_targets))
|
||||
from starpunk.syndication import resolve_targets, select_targets
|
||||
select_targets(note.id, resolve_targets(requested_targets))
|
||||
|
||||
return "", 201, {"Location": absolute_url_for("public.note", slug=note.slug)}
|
||||
|
||||
@@ -618,9 +671,12 @@ def handle_update(data: dict, token_info: dict):
|
||||
delete = data.get("delete") or {}
|
||||
if not isinstance(replace, dict) or not isinstance(add, dict) or not isinstance(delete, (dict, list)):
|
||||
raise MicropubValidationError("replace and add must be objects; delete an object or list")
|
||||
for prop, values in list(replace.items()) + list(add.items()):
|
||||
for prop, values in list(replace.items()) + list(add.items()) + (
|
||||
list(delete.items()) if isinstance(delete, dict) else []):
|
||||
if not isinstance(values, list):
|
||||
raise MicropubValidationError(f"Values for {prop} must be an array")
|
||||
if not (replace or add or delete):
|
||||
raise MicropubValidationError("An update needs replace, add or delete")
|
||||
touched = set(replace) | set(add) | set(delete)
|
||||
unsupported = sorted(touched - UPDATABLE)
|
||||
if unsupported:
|
||||
@@ -664,6 +720,7 @@ def handle_update(data: dict, token_info: dict):
|
||||
if "content" in replace:
|
||||
# Empty text is fine if the note keeps its photos; update_note checks (v2.6.5)
|
||||
body = extract_content(replace) if _has_text(replace["content"]) else ""
|
||||
changes["content_format"] = content_format(replace) if body else None
|
||||
if "name" in replace:
|
||||
title = extract_title(replace)
|
||||
if "category" in replace:
|
||||
@@ -681,13 +738,19 @@ def handle_update(data: dict, token_info: dict):
|
||||
if "photo" in add:
|
||||
photos += extract_photos(add)
|
||||
|
||||
# Text may go only if photos stay (or it's a like or repost) (v2.6.5)
|
||||
if not (body or "").strip() and not (photos or note.like_of or note.repost_of):
|
||||
# Text may go only if photos stay (or it's a like, repost or bookmark) (v2.6.5)
|
||||
if not (body or "").strip() and not (photos or note.reacts_to):
|
||||
raise MicropubValidationError("A post needs text or a photo")
|
||||
|
||||
# Photos must all resolve before anything changes (v2.9.0)
|
||||
if "photo" in touched:
|
||||
media_ids, captions = _resolve_photos(photos)
|
||||
|
||||
new_content = add_title_heading(body, title)
|
||||
if new_content != note.content:
|
||||
changes["content"] = new_content
|
||||
elif "content_format" in changes and changes["content_format"] == note.content_format:
|
||||
del changes["content_format"]
|
||||
if {"category"} & touched:
|
||||
changes["tags"] = tags
|
||||
|
||||
@@ -695,8 +758,12 @@ def handle_update(data: dict, token_info: dict):
|
||||
if changes:
|
||||
update_note(id=note.id, has_media=bool(photos), **changes)
|
||||
if "photo" in touched:
|
||||
_attach_photos_to_note(note.id, photos)
|
||||
from starpunk.media import attach_media_to_note
|
||||
attach_media_to_note(note.id, media_ids, captions)
|
||||
note_changed(note.id)
|
||||
if not changes: # update_note already told WebSub otherwise
|
||||
from starpunk import websub
|
||||
websub.note_changed(note, was_public=note.published)
|
||||
except InvalidNoteDataError as e:
|
||||
raise MicropubValidationError(str(e))
|
||||
except MicropubError:
|
||||
@@ -742,7 +809,8 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
|
||||
|
||||
title, content = split_title(note.content)
|
||||
props = {
|
||||
"content": [content],
|
||||
# HTML-authored content comes back as HTML (3.7.2, v2.9.0)
|
||||
"content": [{"html": content} if note.content_format == "html" else content],
|
||||
"published": [iso_utc(note.created_at)],
|
||||
"url": [absolute_url_for("public.note", slug=note.slug)],
|
||||
"post-status": ["published" if note.published else "draft"],
|
||||
@@ -760,6 +828,10 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
|
||||
props["like-of"] = [note.like_of]
|
||||
if getattr(note, "repost_of", None):
|
||||
props["repost-of"] = [note.repost_of]
|
||||
if getattr(note, "bookmark_of", None):
|
||||
props["bookmark-of"] = [note.bookmark_of]
|
||||
if not content.strip():
|
||||
del props["content"] # photo posts, likes, reposts and bookmarks without text
|
||||
photos = _note_photos(note)
|
||||
if photos:
|
||||
props["photo"] = [{"value": p["url"], "alt": p["alt"]} for p in photos]
|
||||
@@ -812,6 +884,11 @@ def handle_query(args, token_info: dict):
|
||||
|
||||
q = args.get("q")
|
||||
|
||||
# Posts (drafts included) and tags are for apps that may post (v2.9.0);
|
||||
# config and syndicate-to are fine for any token
|
||||
if q in ("source", "category"):
|
||||
_require_scope(token_info, "create", "update", "delete", "draft")
|
||||
|
||||
if q == "config":
|
||||
site_url = current_app.config.get("SITE_URL", "http://localhost:5000").rstrip('/')
|
||||
return jsonify({
|
||||
@@ -825,6 +902,7 @@ def handle_query(args, token_info: dict):
|
||||
{"type": "reply", "name": "Reply", "properties": ["in-reply-to", "content"]},
|
||||
{"type": "like", "name": "Like", "properties": ["like-of"]},
|
||||
{"type": "repost", "name": "Repost", "properties": ["repost-of", "content"]},
|
||||
{"type": "bookmark", "name": "Bookmark", "properties": ["bookmark-of", "content"]},
|
||||
],
|
||||
}), 200
|
||||
|
||||
@@ -853,4 +931,4 @@ def handle_query(args, token_info: dict):
|
||||
if q == "category":
|
||||
return jsonify({"categories": _all_categories(args.get("filter") or "")}), 200
|
||||
|
||||
return error_response("invalid_request", f"Unknown query: {q}")
|
||||
return error_response("invalid_request", f"Unknown query: {q}" if q else "Missing q parameter")
|
||||
+42
-6
@@ -31,12 +31,47 @@ class MigrationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def create_migrations_table(conn):
|
||||
def split_sql_statements(script: str) -> list[str]:
|
||||
"""
|
||||
Split a SQL script into complete statements (v2.8.1)
|
||||
|
||||
Uses sqlite3.complete_statement, so semicolons inside strings and
|
||||
trigger bodies (BEGIN ... END) don't split a statement.
|
||||
"""
|
||||
statements = []
|
||||
current = ""
|
||||
for line in script.splitlines(keepends=True):
|
||||
current += line
|
||||
if sqlite3.complete_statement(current):
|
||||
if current.strip().strip(";").strip():
|
||||
statements.append(current.strip())
|
||||
current = ""
|
||||
if current.strip():
|
||||
statements.append(current.strip())
|
||||
return statements
|
||||
|
||||
|
||||
def execute_in_transaction(conn, script: str) -> None:
|
||||
"""
|
||||
Run a SQL script inside the connection's open transaction
|
||||
|
||||
Not conn.executescript(): that COMMITs first, which released the
|
||||
migration lock mid-migration and let another gunicorn worker apply the
|
||||
same migration again ("duplicate column name", v2.8.0 deploy).
|
||||
"""
|
||||
for statement in split_sql_statements(script):
|
||||
conn.execute(statement)
|
||||
|
||||
|
||||
def create_migrations_table(conn, commit=True):
|
||||
"""
|
||||
Create schema_migrations tracking table if it doesn't exist
|
||||
|
||||
Args:
|
||||
conn: SQLite connection
|
||||
commit: Commit afterwards. run_migrations passes False: committing
|
||||
there released its BEGIN IMMEDIATE lock straight away, so
|
||||
gunicorn workers applied migrations concurrently (v2.8.1)
|
||||
"""
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
@@ -51,7 +86,8 @@ def create_migrations_table(conn):
|
||||
ON schema_migrations(migration_name)
|
||||
""")
|
||||
|
||||
conn.commit()
|
||||
if commit:
|
||||
conn.commit()
|
||||
|
||||
|
||||
def is_schema_current(conn):
|
||||
@@ -287,7 +323,7 @@ def apply_migration(conn, migration_name, migration_path, logger=None):
|
||||
|
||||
# Execute migration in transaction
|
||||
conn.execute("BEGIN")
|
||||
conn.executescript(migration_sql)
|
||||
execute_in_transaction(conn, migration_sql)
|
||||
|
||||
# Record migration as applied
|
||||
conn.execute(
|
||||
@@ -374,7 +410,7 @@ def run_migrations(db_path, logger=None):
|
||||
|
||||
try:
|
||||
# Ensure migrations tracking table exists
|
||||
create_migrations_table(conn)
|
||||
create_migrations_table(conn, commit=False)
|
||||
|
||||
# Quick check: have migrations already been applied by another worker?
|
||||
cursor = conn.execute("SELECT COUNT(*) FROM schema_migrations")
|
||||
@@ -472,8 +508,8 @@ def run_migrations(db_path, logger=None):
|
||||
|
||||
logger.debug(f"Applying migration: {migration_name}")
|
||||
|
||||
# Execute migration (already in transaction)
|
||||
conn.executescript(migration_sql)
|
||||
# Execute migration inside the locked transaction
|
||||
execute_in_transaction(conn, migration_sql)
|
||||
|
||||
# Record migration as applied
|
||||
conn.execute(
|
||||
|
||||
+42
-14
@@ -16,6 +16,7 @@ import sqlite3
|
||||
|
||||
# Third-party imports
|
||||
import markdown
|
||||
import nh3
|
||||
|
||||
# Local imports
|
||||
from starpunk.utils import (
|
||||
@@ -42,6 +43,32 @@ MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br", BareUrlExtension()]
|
||||
# Constants - Content limits
|
||||
MAX_TITLE_LENGTH = 200
|
||||
|
||||
# HTML allowed in rendered notes (v2.9.0). Markdown passes raw HTML through,
|
||||
# and Micropub clients may send HTML, so scripts, event handlers, iframes and
|
||||
# javascript: links are stripped. class keeps code highlighting and
|
||||
# microformats; id keeps footnote links.
|
||||
ALLOWED_TAGS = set(nh3.ALLOWED_TAGS) | {"video", "audio", "source", "track", "picture"}
|
||||
ALLOWED_ATTRIBUTES = {
|
||||
**{tag: set(attrs) for tag, attrs in nh3.ALLOWED_ATTRIBUTES.items()},
|
||||
"*": {"class", "title", "id", "lang", "dir"},
|
||||
"img": set(nh3.ALLOWED_ATTRIBUTES.get("img", ())) | {"loading", "srcset", "sizes"},
|
||||
"video": {"src", "controls", "poster", "width", "height", "loop", "muted", "playsinline", "preload"},
|
||||
"audio": {"src", "controls", "loop", "muted", "preload"},
|
||||
"source": {"src", "type", "srcset", "media", "sizes"},
|
||||
"track": {"src", "kind", "label", "srclang", "default"},
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(text: str) -> str:
|
||||
"""Note Markdown (or HTML) to sanitized HTML"""
|
||||
html = markdown.markdown(text, extensions=MARKDOWN_EXTENSIONS)
|
||||
return nh3.clean(html, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
|
||||
|
||||
|
||||
# Likes, reposts and bookmarks (v2.5.0, v2.9.0)
|
||||
REACTION_VERBS = {"like": "Liked", "repost": "Reposted", "bookmark": "Bookmarked"}
|
||||
REACTION_ICONS = {"like": "★", "repost": "🔁", "bookmark": "🔖"}
|
||||
|
||||
# A Markdown ATX heading: up to 6 '#', whitespace, then text
|
||||
TITLE_RE = re.compile(r"#{1,6}[ \t]+(\S.*)$")
|
||||
EXCERPT_LENGTH = 200
|
||||
@@ -142,6 +169,8 @@ class Note:
|
||||
in_reply_to: Optional[str] = None # v1.6.0: URL being replied to
|
||||
like_of: Optional[str] = None # v2.5.0: URL liked (this note is a like)
|
||||
repost_of: Optional[str] = None # v2.5.0: URL reposted (this note is a repost)
|
||||
bookmark_of: Optional[str] = None # v2.9.0: URL bookmarked (this note is a bookmark)
|
||||
content_format: Optional[str] = None # v2.9.0: 'html' if sent as HTML via Micropub
|
||||
_cached_content: Optional[str] = field(
|
||||
default=None, repr=False, compare=False, init=False
|
||||
)
|
||||
@@ -202,6 +231,8 @@ class Note:
|
||||
in_reply_to=data.get("in_reply_to"),
|
||||
like_of=data.get("like_of"),
|
||||
repost_of=data.get("repost_of"),
|
||||
bookmark_of=data.get("bookmark_of"),
|
||||
content_format=data.get("content_format"),
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -260,11 +291,8 @@ class Note:
|
||||
<p>This is my note content...</p>
|
||||
"""
|
||||
if self._cached_html is None:
|
||||
# Render markdown to HTML
|
||||
html = markdown.markdown(
|
||||
self.content, # This triggers lazy load of content
|
||||
extensions=MARKDOWN_EXTENSIONS,
|
||||
)
|
||||
# Render markdown to sanitized HTML (v2.9.0)
|
||||
html = render_markdown(self.content) # triggers lazy load of content
|
||||
|
||||
# Cache it
|
||||
object.__setattr__(self, "_cached_html", html)
|
||||
@@ -282,7 +310,7 @@ class Note:
|
||||
title, body = split_title(self.content)
|
||||
if title is None:
|
||||
return self.html
|
||||
return markdown.markdown(body, extensions=MARKDOWN_EXTENSIONS)
|
||||
return render_markdown(body)
|
||||
|
||||
@property
|
||||
def has_title(self) -> bool:
|
||||
@@ -386,9 +414,8 @@ class Note:
|
||||
excerpt += "..."
|
||||
|
||||
if not excerpt and self.reacts_to:
|
||||
# A like or repost with no comment
|
||||
verb = "Liked" if self.kind == "like" else "Reposted"
|
||||
return f"{verb} {self.reacts_to}"
|
||||
# A like, repost or bookmark with no comment
|
||||
return f"{REACTION_VERBS[self.kind]} {self.reacts_to}"
|
||||
|
||||
return excerpt
|
||||
|
||||
@@ -398,17 +425,19 @@ class Note:
|
||||
|
||||
@property
|
||||
def kind(self) -> str:
|
||||
"""'like', 'repost' or 'note' (v2.5.0)"""
|
||||
"""'like', 'repost', 'bookmark' (v2.9.0) or 'note' (v2.5.0)"""
|
||||
if self.like_of:
|
||||
return "like"
|
||||
if self.repost_of:
|
||||
return "repost"
|
||||
if self.bookmark_of:
|
||||
return "bookmark"
|
||||
return "note"
|
||||
|
||||
@property
|
||||
def reacts_to(self) -> Optional[str]:
|
||||
"""The URL a like or repost points at"""
|
||||
return self.like_of or self.repost_of
|
||||
"""The URL a like, repost or bookmark points at"""
|
||||
return self.like_of or self.repost_of or self.bookmark_of
|
||||
|
||||
@property
|
||||
def feed_html(self) -> str:
|
||||
@@ -417,9 +446,8 @@ class Note:
|
||||
|
||||
if not self.reacts_to:
|
||||
return self.html
|
||||
verb = "Liked" if self.kind == "like" else "Reposted"
|
||||
url = escape(self.reacts_to, quote=True)
|
||||
line = f'<p>{"★" if self.kind == "like" else "🔁"} {verb} <a href="{url}">{url}</a></p>'
|
||||
line = f'<p>{REACTION_ICONS[self.kind]} {REACTION_VERBS[self.kind]} <a href="{url}">{url}</a></p>'
|
||||
return line + (self.html or "")
|
||||
|
||||
@property
|
||||
|
||||
+61
-11
@@ -133,6 +133,10 @@ def _get_existing_slugs(db) -> set[str]:
|
||||
# Core CRUD Functions
|
||||
|
||||
|
||||
# update_note: leave a field as it is
|
||||
UNCHANGED = object()
|
||||
|
||||
|
||||
def create_note(
|
||||
content: str,
|
||||
published: bool = False,
|
||||
@@ -143,6 +147,8 @@ def create_note(
|
||||
like_of: Optional[str] = None,
|
||||
repost_of: Optional[str] = None,
|
||||
has_media: bool = False,
|
||||
bookmark_of: Optional[str] = None,
|
||||
content_format: Optional[str] = None,
|
||||
) -> Note:
|
||||
"""
|
||||
Create a new note
|
||||
@@ -207,10 +213,13 @@ def create_note(
|
||||
"""
|
||||
# 1. VALIDATION (before any changes)
|
||||
# Likes and reposts may have no comment (v2.5.0), photo posts no text (v2.6.5)
|
||||
if like_of and repost_of:
|
||||
raise InvalidNoteDataError("like_of", like_of, "A note can't be both a like and a repost")
|
||||
# Bookmarks too (v2.9.0)
|
||||
if sum(bool(u) for u in (like_of, repost_of, bookmark_of)) > 1:
|
||||
raise InvalidNoteDataError(
|
||||
"like_of", like_of, "A note can only be one of a like, repost or bookmark"
|
||||
)
|
||||
content = content or ""
|
||||
if not content.strip() and not (like_of or repost_of or has_media):
|
||||
if not content.strip() and not (like_of or repost_of or bookmark_of or has_media):
|
||||
raise InvalidNoteDataError(
|
||||
"content", content, "Content cannot be empty or whitespace-only"
|
||||
)
|
||||
@@ -271,10 +280,12 @@ def create_note(
|
||||
try:
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash, in_reply_to, like_of, repost_of)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash,
|
||||
in_reply_to, like_of, repost_of, bookmark_of, content_format)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(slug, file_path_rel, published, created_at, updated_at, content_hash, in_reply_to, like_of, repost_of),
|
||||
(slug, file_path_rel, published, created_at, updated_at, content_hash,
|
||||
in_reply_to, like_of, repost_of, bookmark_of, content_format),
|
||||
)
|
||||
db.commit()
|
||||
except Exception as e:
|
||||
@@ -342,7 +353,7 @@ def create_note(
|
||||
note_id=note_id,
|
||||
note_url=note_url,
|
||||
content_html=html_content,
|
||||
in_reply_to=in_reply_to or like_of or repost_of
|
||||
in_reply_to=in_reply_to or like_of or repost_of or bookmark_of
|
||||
)
|
||||
except Exception as e:
|
||||
# Webmention queueing failure should not prevent note creation
|
||||
@@ -350,7 +361,7 @@ def create_note(
|
||||
|
||||
# 12. FETCH REPLY CONTEXT (v1.6.0 Phase 3)
|
||||
# Fetch context for the post being replied to, liked or reposted
|
||||
context_url = in_reply_to or like_of or repost_of
|
||||
context_url = in_reply_to or like_of or repost_of or bookmark_of
|
||||
if context_url:
|
||||
try:
|
||||
from starpunk.reply_context import fetch_reply_context
|
||||
@@ -363,9 +374,28 @@ def create_note(
|
||||
# Reply context fetch failure should not prevent note creation
|
||||
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
|
||||
|
||||
_websub_changed(note)
|
||||
return note
|
||||
|
||||
|
||||
def _tag_names(note_id: int) -> list[str]:
|
||||
"""A note's tag names, for WebSub tag topics (best effort)"""
|
||||
try:
|
||||
from starpunk.tags import get_note_tags
|
||||
return [t["name"] for t in get_note_tags(note_id)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def _websub_changed(note, was_public: bool = False, old_tags=()) -> None:
|
||||
"""Clear cached feeds and queue WebSub deliveries (v2.10.0, best effort)"""
|
||||
try:
|
||||
from starpunk import websub
|
||||
websub.note_changed(note, was_public=was_public, old_tags=old_tags)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue WebSub update for note {note.slug}: {e}")
|
||||
|
||||
|
||||
def get_note(
|
||||
slug: Optional[str] = None, id: Optional[int] = None, load_content: bool = True
|
||||
) -> Optional[Note]:
|
||||
@@ -607,10 +637,13 @@ def update_note(
|
||||
tags: Optional[list[str]] = None,
|
||||
in_reply_to: Optional[str] = None,
|
||||
has_media: bool = False,
|
||||
content_format=UNCHANGED,
|
||||
) -> Note:
|
||||
"""
|
||||
Update a note's content and/or published status
|
||||
|
||||
content_format (v2.9.0): 'html' or None (Markdown); left alone unless given
|
||||
|
||||
Updates note content and/or metadata, maintaining atomic synchronization
|
||||
between file and database. At least one of content or published must
|
||||
be provided.
|
||||
@@ -691,8 +724,9 @@ def update_note(
|
||||
if existing_note is None:
|
||||
identifier = slug if slug is not None else id
|
||||
raise NoteNotFoundError(identifier)
|
||||
old_tag_names = _tag_names(existing_note.id) # for WebSub tag topics (v2.10.0)
|
||||
|
||||
if empty_content and not (existing_note.like_of or existing_note.repost_of or has_media):
|
||||
if empty_content and not (existing_note.reacts_to or has_media):
|
||||
raise InvalidNoteDataError(
|
||||
"content", content, "Content cannot be empty or whitespace-only"
|
||||
)
|
||||
@@ -756,6 +790,10 @@ def update_note(
|
||||
update_fields.append("in_reply_to = ?")
|
||||
params.append(in_reply_to)
|
||||
|
||||
if content_format is not UNCHANGED:
|
||||
update_fields.append("content_format = ?")
|
||||
params.append(content_format)
|
||||
|
||||
# Add WHERE clause parameter
|
||||
if slug is not None:
|
||||
where_clause = "slug = ?"
|
||||
@@ -841,7 +879,7 @@ def update_note(
|
||||
html_content = updated_note.html if updated_note.published else None
|
||||
|
||||
# Determine in_reply_to (use new value if provided, otherwise existing)
|
||||
reply_to = (in_reply_to if in_reply_to is not None else updated_note.in_reply_to) or updated_note.like_of or updated_note.repost_of
|
||||
reply_to = (in_reply_to if in_reply_to is not None else updated_note.in_reply_to) or updated_note.reacts_to
|
||||
|
||||
# Queue webmentions (best-effort)
|
||||
# Pass old_html for URL diffing (Phase 6)
|
||||
@@ -878,6 +916,8 @@ def update_note(
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue syndication update: {e}")
|
||||
|
||||
_websub_changed(updated_note, was_public=existing_note.published, old_tags=old_tag_names)
|
||||
|
||||
# 10. RETURN UPDATED NOTE
|
||||
return updated_note
|
||||
|
||||
@@ -977,6 +1017,10 @@ def delete_note(
|
||||
# For idempotency, don't raise error - just return
|
||||
return
|
||||
|
||||
# What feeds showed before, for WebSub (v2.10.0)
|
||||
was_public = bool(existing_note.published) and not getattr(existing_note, "deleted_at", None)
|
||||
deleted_tags = _tag_names(existing_note.id)
|
||||
|
||||
# 4. SETUP
|
||||
data_dir = Path(current_app.config["DATA_PATH"])
|
||||
note_path = data_dir / existing_note.file_path
|
||||
@@ -1063,6 +1107,10 @@ def delete_note(
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
|
||||
|
||||
# Feeds and pages no longer show it (v2.10.0)
|
||||
object.__setattr__(existing_note, "published", False)
|
||||
_websub_changed(existing_note, was_public=was_public, old_tags=deleted_tags)
|
||||
|
||||
# 6. RETURN (no value)
|
||||
return None
|
||||
|
||||
@@ -1119,4 +1167,6 @@ def undelete_note(slug: str) -> Note:
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
|
||||
|
||||
return get_note(slug=slug)
|
||||
restored = get_note(slug=slug)
|
||||
_websub_changed(restored)
|
||||
return restored
|
||||
@@ -7,7 +7,7 @@ admin, auth, and (conditionally) dev auth routes.
|
||||
|
||||
from flask import Flask
|
||||
|
||||
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention
|
||||
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention, websub
|
||||
|
||||
|
||||
def register_routes(app: Flask) -> None:
|
||||
@@ -44,6 +44,9 @@ def register_routes(app: Flask) -> None:
|
||||
if app.config.get("WEBMENTION_ENABLED", True):
|
||||
app.register_blueprint(webmention.bp)
|
||||
|
||||
# WebSub hub (v2.10.0); refuses requests when disabled or using another hub
|
||||
app.register_blueprint(websub.bp)
|
||||
|
||||
# Conditionally register dev auth routes
|
||||
if app.config.get("DEV_MODE"):
|
||||
app.logger.warning(
|
||||
|
||||
@@ -72,12 +72,23 @@ def dashboard():
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# WebSub subscribers (v2.10.0)
|
||||
websub_stats = None
|
||||
if current_app.config.get('WEBSUB_ENABLED', True):
|
||||
try:
|
||||
from starpunk import websub
|
||||
from starpunk.database import get_db
|
||||
websub_stats = websub.stats(get_db(current_app))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return render_template(
|
||||
"admin/dashboard.html",
|
||||
notes=notes,
|
||||
page=page,
|
||||
total_pages=total_pages,
|
||||
webmention_pending_count=webmention_pending_count,
|
||||
websub_stats=websub_stats,
|
||||
user_me=g.me
|
||||
)
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ from starpunk.indieauth import (
|
||||
validate_authorization_request,
|
||||
verify_access_token,
|
||||
)
|
||||
from starpunk.micropub import extract_bearer_token
|
||||
from starpunk.micropub import MicropubError, extract_bearer_token
|
||||
from starpunk.utils import absolute_url_for
|
||||
|
||||
bp = Blueprint("indieauth", __name__)
|
||||
@@ -215,7 +215,11 @@ def token():
|
||||
@bp.route("/auth/token", methods=["GET"])
|
||||
def token_verify():
|
||||
"""Legacy token verification for clients that still call it"""
|
||||
info = verify_access_token(extract_bearer_token(request))
|
||||
try:
|
||||
token = extract_bearer_token(request)
|
||||
except MicropubError as e:
|
||||
return _oauth_error("invalid_request", e.error_description)
|
||||
info = verify_access_token(token)
|
||||
if not info:
|
||||
return _oauth_error("unauthorized", "Invalid or expired access token", 401)
|
||||
return _json(info)
|
||||
|
||||
@@ -70,7 +70,10 @@ def micropub_endpoint():
|
||||
}
|
||||
"""
|
||||
# Extract and verify token
|
||||
token = extract_bearer_token(request)
|
||||
try:
|
||||
token = extract_bearer_token(request)
|
||||
except MicropubError as e:
|
||||
return error_response(e.error, e.error_description, e.status_code)
|
||||
if not token:
|
||||
return error_response("unauthorized", "No access token provided", 401)
|
||||
|
||||
@@ -94,7 +97,10 @@ def micropub_endpoint():
|
||||
try:
|
||||
# Parse request based on content type
|
||||
if "application/json" in content_type:
|
||||
data = request.get_json() or {}
|
||||
# Malformed JSON is the client's error, not ours (v2.9.0)
|
||||
data = request.get_json(silent=True)
|
||||
if not isinstance(data, dict):
|
||||
return error_response("invalid_request", "Request body must be a JSON object", 400)
|
||||
action = data.get("action", "create")
|
||||
else:
|
||||
# Form-encoded or multipart (files handled separately)
|
||||
@@ -136,7 +142,10 @@ def media_endpoint():
|
||||
from starpunk.media import save_media
|
||||
|
||||
# Extract and verify token
|
||||
token = extract_bearer_token(request)
|
||||
try:
|
||||
token = extract_bearer_token(request)
|
||||
except MicropubError as e:
|
||||
return error_response(e.error, e.error_description, e.status_code)
|
||||
if not token:
|
||||
return error_response("unauthorized", "No access token provided", 401)
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ from datetime import datetime, timedelta
|
||||
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
|
||||
|
||||
from starpunk.notes import list_notes, get_note
|
||||
from starpunk.websub import embed_hub
|
||||
from starpunk.feed import generate_feed_streaming # Legacy RSS
|
||||
from starpunk.feeds import (
|
||||
generate_rss,
|
||||
@@ -55,8 +56,14 @@ def _get_cached_notes():
|
||||
cache_duration = timedelta(seconds=cache_seconds)
|
||||
now = datetime.utcnow()
|
||||
|
||||
# The cached list is stale once any public note changed, in any process
|
||||
# (v2.10.0): every change bumps content_version in the database
|
||||
from starpunk.database import get_db
|
||||
from starpunk.websub import get_content_version
|
||||
version = (str(current_app.config["DATABASE_PATH"]), get_content_version(get_db(current_app)))
|
||||
|
||||
# Check if note list cache is valid
|
||||
if _feed_cache["notes"] and _feed_cache["timestamp"]:
|
||||
if _feed_cache["notes"] and _feed_cache["timestamp"] and _feed_cache.get("version") == version:
|
||||
cache_age = now - _feed_cache["timestamp"]
|
||||
if cache_age < cache_duration:
|
||||
# Use cached note list
|
||||
@@ -82,6 +89,7 @@ def _get_cached_notes():
|
||||
|
||||
_feed_cache["notes"] = notes
|
||||
_feed_cache["timestamp"] = now
|
||||
_feed_cache["version"] = version
|
||||
|
||||
return notes
|
||||
|
||||
@@ -122,6 +130,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
||||
notes=notes,
|
||||
limit=max_items,
|
||||
)
|
||||
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
|
||||
|
||||
# Create response with proper Content-Type including charset
|
||||
mime_type = get_mime_type(format_name)
|
||||
@@ -170,6 +179,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
||||
notes=notes,
|
||||
limit=max_items,
|
||||
)
|
||||
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
|
||||
|
||||
# Store in cache and get ETag
|
||||
etag = feed_cache.set(format_name, content, notes_checksum)
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
"""
|
||||
WebSub hub endpoint (v2.10.0, ADR-071)
|
||||
|
||||
POST /websub with hub.mode subscribe/unsubscribe, hub.topic, hub.callback,
|
||||
optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
|
||||
(spec 5.1); errors are 4xx with a plain-text reason.
|
||||
"""
|
||||
|
||||
from flask import Blueprint, current_app, request
|
||||
|
||||
from starpunk.database import get_db
|
||||
from starpunk.rate_limiter import RateLimiter
|
||||
from starpunk import websub
|
||||
|
||||
bp = Blueprint("websub", __name__)
|
||||
|
||||
# Subscription requests per client IP per minute
|
||||
_limiter = RateLimiter(limit=30, window_seconds=60)
|
||||
|
||||
|
||||
def _text(message: str, status: int):
|
||||
return message + "\n", status, {"Content-Type": "text/plain; charset=utf-8"}
|
||||
|
||||
|
||||
@bp.route("/websub", methods=["POST"])
|
||||
def hub():
|
||||
if not websub.enabled():
|
||||
return _text("WebSub is not enabled on this site", 404)
|
||||
|
||||
from starpunk.routes.webmention import get_client_ip
|
||||
if not _limiter.is_allowed(get_client_ip()):
|
||||
return _text("Too many requests", 429)
|
||||
|
||||
if "application/x-www-form-urlencoded" not in request.headers.get("Content-Type", ""):
|
||||
return _text("Send an application/x-www-form-urlencoded body", 400)
|
||||
|
||||
try:
|
||||
websub.accept_subscription_request(request.form, get_db(current_app))
|
||||
except websub.SubscriptionError as e:
|
||||
return _text(str(e), e.status)
|
||||
|
||||
return _text("Accepted; the subscription will be verified", 202)
|
||||
|
||||
|
||||
@bp.route("/websub", methods=["GET"])
|
||||
def hub_info():
|
||||
return _text(
|
||||
"WebSub hub for this site's own feeds. Subscribe with a POST "
|
||||
"(https://www.w3.org/TR/websub/).", 200)
|
||||
@@ -1002,6 +1002,9 @@ def compose_text(note, permalink: str, tags: List[str], max_chars: int,
|
||||
hashtags = f"{hashtags} {h}".strip()
|
||||
tail = f"\n\n{permalink}" + (f"\n\n{hashtags}" if hashtags else "")
|
||||
body = (note.content if body is None else body).strip()
|
||||
if getattr(note, "bookmark_of", None):
|
||||
# A bookmark leads with what it bookmarks (v2.9.0)
|
||||
body = f"🔖 {note.bookmark_of}\n\n{body}".strip()
|
||||
if len(body) + len(tail) > max_chars:
|
||||
room = max_chars - len(tail) - 1
|
||||
body = body[:room].rsplit(" ", 1)[0].rstrip() + "…"
|
||||
|
||||
+31
-14
@@ -18,6 +18,7 @@ import ipaddress
|
||||
import socket
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import httpx
|
||||
@@ -157,20 +158,6 @@ def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
||||
)
|
||||
|
||||
|
||||
def safe_post(url: str, data: dict, headers: dict, timeout: float = TIMEOUT) -> httpx.Response:
|
||||
"""
|
||||
POST form data to a checked, pinned address. Redirects are not followed.
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL is not allowed
|
||||
httpx.HTTPError: Network errors
|
||||
"""
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
||||
response = client.send(_pinned_request(client, 'POST', url, headers, data=data))
|
||||
response.read()
|
||||
return response
|
||||
|
||||
|
||||
@dataclass
|
||||
class FetchResult:
|
||||
"""A fetched response, after redirects"""
|
||||
@@ -188,6 +175,36 @@ class FetchResult:
|
||||
return self.content.decode('utf-8', errors='replace')
|
||||
|
||||
|
||||
def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = None,
|
||||
timeout: float = TIMEOUT, content: Optional[bytes] = None,
|
||||
max_bytes: int = 64 * 1024) -> FetchResult:
|
||||
"""
|
||||
POST form data (or a raw body) to a checked, pinned address.
|
||||
|
||||
Redirects are not followed. Only the first max_bytes of the response are
|
||||
read (callers need the status and maybe an error message).
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL is not allowed
|
||||
httpx.HTTPError: Network errors
|
||||
"""
|
||||
kwargs = {'content': content} if content is not None else {'data': data}
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
||||
request = _pinned_request(client, 'POST', url, headers or {}, **kwargs)
|
||||
response = client.send(request, stream=True)
|
||||
try:
|
||||
body = b''
|
||||
for chunk in response.iter_bytes(chunk_size=8192):
|
||||
body += chunk
|
||||
if len(body) >= max_bytes:
|
||||
body = body[:max_bytes]
|
||||
break
|
||||
return FetchResult(url=url, status_code=response.status_code, headers=response.headers,
|
||||
content=body, encoding=response.charset_encoding or 'utf-8')
|
||||
finally:
|
||||
response.close()
|
||||
|
||||
|
||||
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
|
||||
"""
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
"""
|
||||
WebSub publishing and the built-in hub (v2.10.0, ADR-071)
|
||||
|
||||
Publisher: topic responses advertise the hub and their own URL; a change to a
|
||||
public note clears cached feeds and queues delivery of every affected topic.
|
||||
|
||||
Hub: subscriptions to this site's own topics only. Requests are accepted
|
||||
here (202) and verified, delivered and retried by the WebSub worker.
|
||||
|
||||
References:
|
||||
- W3C WebSub: https://www.w3.org/TR/websub/
|
||||
- ADR-071: WebSub publishing with a built-in hub
|
||||
"""
|
||||
|
||||
import json
|
||||
import secrets
|
||||
from typing import Iterable, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from flask import current_app, request
|
||||
|
||||
|
||||
# Topic endpoints and their fixed paths (tag pages are added per tag)
|
||||
FEED_TOPICS = {
|
||||
"public.feed_atom": "/feed.atom",
|
||||
"public.feed_rss": "/feed.rss",
|
||||
"public.feed_xml_legacy": "/feed.rss", # legacy alias of the RSS topic
|
||||
"public.feed_json": "/feed.json",
|
||||
}
|
||||
PAGE_TOPICS = {"public.index": "/", "public.likes": "/likes"}
|
||||
NEGOTIATED = { # /feed: the topic of whichever format was served
|
||||
"application/atom+xml": "/feed.atom",
|
||||
"application/rss+xml": "/feed.rss",
|
||||
"application/feed+json": "/feed.json",
|
||||
"application/json": "/feed.json",
|
||||
}
|
||||
|
||||
LEASE_DEFAULT = 10 * 24 * 3600
|
||||
LEASE_MIN = 24 * 3600
|
||||
LEASE_MAX = 30 * 24 * 3600
|
||||
MAX_SECRET_BYTES = 200
|
||||
|
||||
|
||||
class SubscriptionError(Exception):
|
||||
"""A subscription request the hub refuses (4xx with a plain-text reason)"""
|
||||
|
||||
def __init__(self, message: str, status: int = 400):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
|
||||
|
||||
# --- Configuration -----------------------------------------------------------
|
||||
|
||||
def enabled() -> bool:
|
||||
return bool(current_app.config.get("WEBSUB_ENABLED", True))
|
||||
|
||||
|
||||
def external_hub() -> Optional[str]:
|
||||
"""URL of an external hub, if one is configured instead of the built-in one"""
|
||||
return current_app.config.get("WEBSUB_HUB_URL") or None
|
||||
|
||||
|
||||
def hub_url() -> str:
|
||||
# Built from SITE_URL so the worker can use it outside a request
|
||||
return external_hub() or site_url() + "/websub"
|
||||
|
||||
|
||||
def site_url() -> str:
|
||||
return current_app.config["SITE_URL"].rstrip("/")
|
||||
|
||||
|
||||
# --- Topics ------------------------------------------------------------------
|
||||
|
||||
def topic_url(path: str) -> str:
|
||||
return site_url() + path
|
||||
|
||||
|
||||
def tag_topic(tag_name: str) -> str:
|
||||
from urllib.parse import quote
|
||||
return f"{site_url()}/tag/{quote(tag_name, safe='')}"
|
||||
|
||||
|
||||
def topic_for_request(response=None) -> Optional[str]:
|
||||
"""The topic URL the current request serves, or None if it isn't a topic"""
|
||||
endpoint = request.endpoint
|
||||
if endpoint in PAGE_TOPICS:
|
||||
return topic_url(PAGE_TOPICS[endpoint])
|
||||
if endpoint in FEED_TOPICS:
|
||||
return topic_url(FEED_TOPICS[endpoint])
|
||||
if endpoint == "public.feed" and response is not None:
|
||||
mime = (response.mimetype or "").lower()
|
||||
return topic_url(NEGOTIATED[mime]) if mime in NEGOTIATED else None
|
||||
if endpoint == "public.tag":
|
||||
from starpunk.tags import normalize_tag
|
||||
tag = (request.view_args or {}).get("tag", "")
|
||||
return tag_topic(normalize_tag(tag)[0]) if tag else None
|
||||
return None
|
||||
|
||||
|
||||
def canonical_topic(url: str) -> Optional[str]:
|
||||
"""
|
||||
A subscription's hub.topic as one of this site's topics, or None
|
||||
|
||||
Tag topics must name an existing tag.
|
||||
"""
|
||||
if not url:
|
||||
return None
|
||||
site = urlparse(site_url())
|
||||
parsed = urlparse(url.strip())
|
||||
if (parsed.scheme, parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
|
||||
return None
|
||||
path = parsed.path or "/"
|
||||
if parsed.query or parsed.fragment:
|
||||
return None
|
||||
if path in set(PAGE_TOPICS.values()) | set(FEED_TOPICS.values()):
|
||||
return topic_url(path)
|
||||
if path.startswith("/tag/"):
|
||||
from starpunk.tags import get_tag_by_name, normalize_tag
|
||||
name = normalize_tag(path[len("/tag/"):])[0]
|
||||
if name and get_tag_by_name(name):
|
||||
return tag_topic(name)
|
||||
return None
|
||||
|
||||
|
||||
def link_header_values(topic: str) -> list[str]:
|
||||
return [f'<{hub_url()}>; rel="hub"', f'<{topic}>; rel="self"']
|
||||
|
||||
|
||||
def add_discovery_headers(response):
|
||||
"""after_request: Link rel=hub and rel=self on topic responses (spec 4)"""
|
||||
if not enabled() or response.status_code not in (200, 304):
|
||||
return response
|
||||
topic = topic_for_request(response)
|
||||
if topic:
|
||||
for value in link_header_values(topic):
|
||||
response.headers.add("Link", value)
|
||||
return response
|
||||
|
||||
|
||||
def template_links() -> Optional[dict]:
|
||||
"""For base.html: {'hub', 'self'} on topic pages"""
|
||||
if not enabled() or not request:
|
||||
return None
|
||||
topic = topic_for_request()
|
||||
return {"hub": hub_url(), "self": topic} if topic else None
|
||||
|
||||
|
||||
def embed_hub(format_name: str, content: str) -> str:
|
||||
"""Add the hub link inside a generated feed, next to its self link"""
|
||||
if not enabled():
|
||||
return content
|
||||
hub = hub_url()
|
||||
if format_name == "atom":
|
||||
marker = '<link rel="self"'
|
||||
return content.replace(marker, f'<link rel="hub" href="{_xml(hub)}"/>\n {marker}', 1)
|
||||
if format_name == "rss":
|
||||
marker = "<atom:link "
|
||||
return content.replace(marker, f'<atom:link href="{_xml(hub)}" rel="hub"/>\n {marker}', 1)
|
||||
if format_name == "json":
|
||||
feed = json.loads(content)
|
||||
feed["hubs"] = [{"type": "WebSub", "url": hub}]
|
||||
return json.dumps(feed, ensure_ascii=False, indent=2)
|
||||
return content
|
||||
|
||||
|
||||
def _xml(value: str) -> str:
|
||||
return (value.replace("&", "&").replace("<", "<")
|
||||
.replace(">", ">").replace('"', """))
|
||||
|
||||
|
||||
# --- Changes -------------------------------------------------------------------
|
||||
|
||||
def get_content_version(db) -> str:
|
||||
row = db.execute("SELECT value FROM site_state WHERE key = 'content_version'").fetchone()
|
||||
return row[0] if row else "0"
|
||||
|
||||
|
||||
def bump_content_version(db) -> None:
|
||||
db.execute(
|
||||
"INSERT INTO site_state (key, value) VALUES ('content_version', '1') "
|
||||
"ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1"
|
||||
)
|
||||
|
||||
|
||||
def note_changed(note, was_public: bool = False, old_tags: Iterable[str] = ()) -> None:
|
||||
"""
|
||||
A note was created, edited, deleted, restored or (un)published
|
||||
|
||||
Clears cached feeds in every process and queues delivery of the topics
|
||||
that show it. Changes that are never public (a draft edited) do nothing.
|
||||
|
||||
Args:
|
||||
note: The note (after the change; for a delete, before it)
|
||||
was_public: Whether it was public before the change
|
||||
old_tags: Its tag names before the change
|
||||
"""
|
||||
from starpunk.database import get_db
|
||||
|
||||
is_public = bool(note.published) and not getattr(note, "deleted_at", None)
|
||||
if not (is_public or was_public):
|
||||
return
|
||||
|
||||
db = get_db(current_app)
|
||||
bump_content_version(db)
|
||||
db.commit()
|
||||
|
||||
if not enabled():
|
||||
return
|
||||
|
||||
from starpunk.tags import get_note_tags
|
||||
try:
|
||||
tags = {t["name"] for t in get_note_tags(note.id)}
|
||||
except Exception:
|
||||
tags = set()
|
||||
tags |= set(old_tags)
|
||||
|
||||
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
|
||||
if getattr(note, "like_of", None):
|
||||
topics.append(topic_url("/likes"))
|
||||
topics += [tag_topic(name) for name in sorted(tags)]
|
||||
queue_topics(db, topics)
|
||||
|
||||
|
||||
def queue_topics(db, topics: Iterable[str]) -> int:
|
||||
"""Queue delivery of topics: to their subscribers, or a ping to an external hub"""
|
||||
topics = list(dict.fromkeys(topics))
|
||||
if not topics:
|
||||
return 0
|
||||
if external_hub():
|
||||
for topic in topics:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NULL AND topic = ?", (topic,))
|
||||
db.execute("INSERT INTO websub_deliveries (topic) VALUES (?)", (topic,))
|
||||
db.commit()
|
||||
return len(topics)
|
||||
|
||||
placeholders = ",".join("?" * len(topics))
|
||||
rows = db.execute(
|
||||
f"SELECT id, topic FROM websub_subscriptions WHERE topic IN ({placeholders}) "
|
||||
"AND verified_at IS NOT NULL AND expires_at > datetime('now')",
|
||||
topics,
|
||||
).fetchall()
|
||||
for sub_id, topic in rows:
|
||||
# The newest content replaces anything still waiting (ADR-071 6)
|
||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id = ?", (sub_id,))
|
||||
db.execute("INSERT INTO websub_deliveries (subscription_id, topic) VALUES (?, ?)", (sub_id, topic))
|
||||
db.commit()
|
||||
return len(rows)
|
||||
|
||||
|
||||
# --- Hub: subscription requests (spec 5.1) -------------------------------------
|
||||
|
||||
def accept_subscription_request(form, db) -> None:
|
||||
"""
|
||||
Validate a subscription request and queue it for verification
|
||||
|
||||
Raises:
|
||||
SubscriptionError: with the status and plain-text reason to return
|
||||
"""
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
|
||||
|
||||
if external_hub():
|
||||
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
|
||||
|
||||
mode = (form.get("hub.mode") or "").strip()
|
||||
if mode not in ("subscribe", "unsubscribe"):
|
||||
raise SubscriptionError("hub.mode must be subscribe or unsubscribe")
|
||||
|
||||
callback = (form.get("hub.callback") or "").strip()
|
||||
parsed = urlparse(callback)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
|
||||
try:
|
||||
validate_url_for_fetch(callback)
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
raise SubscriptionError(f"hub.callback is not reachable: {e}")
|
||||
|
||||
topic = canonical_topic(form.get("hub.topic") or "")
|
||||
if not topic:
|
||||
raise SubscriptionError("hub.topic is not a topic of this site; use the rel=self URL")
|
||||
|
||||
secret = form.get("hub.secret")
|
||||
if mode == "unsubscribe":
|
||||
secret = None # MUST be ignored for unsubscriptions
|
||||
if secret is not None and len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
|
||||
raise SubscriptionError("hub.secret must be less than 200 bytes")
|
||||
|
||||
lease = LEASE_DEFAULT
|
||||
if form.get("hub.lease_seconds"):
|
||||
try:
|
||||
lease = int(form["hub.lease_seconds"])
|
||||
except ValueError:
|
||||
raise SubscriptionError("hub.lease_seconds must be a number")
|
||||
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
|
||||
|
||||
if mode == "subscribe":
|
||||
existing = db.execute(
|
||||
"SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?", (topic, callback)
|
||||
).fetchone()
|
||||
if not existing:
|
||||
total = db.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0]
|
||||
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
|
||||
raise SubscriptionError("This hub has reached its subscription limit", 429)
|
||||
host = parsed.hostname.lower()
|
||||
per_host = sum(
|
||||
1 for (cb,) in db.execute("SELECT callback FROM websub_subscriptions")
|
||||
if (urlparse(cb).hostname or "").lower() == host
|
||||
)
|
||||
if per_host >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
|
||||
raise SubscriptionError(f"Too many subscriptions for {host}", 429)
|
||||
|
||||
# The latest request for a topic and callback replaces any pending one
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO websub_intents (topic, callback, mode, secret, lease_seconds)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||
mode = excluded.mode, secret = excluded.secret,
|
||||
lease_seconds = excluded.lease_seconds, attempts = 0,
|
||||
next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
|
||||
""",
|
||||
(topic, callback, mode, secret, lease),
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def stats(db) -> dict:
|
||||
"""Active subscribers per topic and deliveries being retried (admin)"""
|
||||
by_topic = db.execute(
|
||||
"SELECT topic, COUNT(*) FROM websub_subscriptions WHERE expires_at > datetime('now') "
|
||||
"GROUP BY topic ORDER BY COUNT(*) DESC, topic"
|
||||
).fetchall()
|
||||
retrying = db.execute(
|
||||
"SELECT COUNT(*) FROM websub_deliveries WHERE attempts > 0"
|
||||
).fetchone()[0]
|
||||
return {"by_topic": [(t, n) for t, n in by_topic], "retrying": retrying,
|
||||
"external": external_hub()}
|
||||
|
||||
|
||||
def new_challenge() -> str:
|
||||
return secrets.token_urlsafe(32)
|
||||
@@ -0,0 +1,287 @@
|
||||
"""
|
||||
WebSub worker: verification of intent and content distribution (v2.10.0)
|
||||
|
||||
Runs like the Webmention worker: a daemon thread in every gunicorn process,
|
||||
but only the process holding the file lock works the queues.
|
||||
|
||||
- Intents (spec 5.3): GET the callback with a challenge; 2xx echoing it
|
||||
verifies. Anything else (including redirects) leaves the subscription as
|
||||
it was. Network errors and 5xx are retried a few times.
|
||||
- Deliveries (spec 6, 7): render the topic in-app and POST it with
|
||||
Link rel=hub/self and, if there is a secret, X-Hub-Signature (sha256).
|
||||
- External hub: a delivery without a subscription is a publish ping.
|
||||
|
||||
References:
|
||||
- https://www.w3.org/TR/websub/
|
||||
- ADR-071
|
||||
"""
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import hmac
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DELIVERY_BACKOFF = [60, 300, 1800, 7200, 21600] # then give up on this delivery
|
||||
INTENT_BACKOFF = [60, 300, 900]
|
||||
DROP_AFTER = timedelta(days=3) # failing this long without a success: unsubscribe
|
||||
PRUNE_EVERY = timedelta(hours=1)
|
||||
BATCH = 10
|
||||
|
||||
|
||||
def callback_with_params(callback: str, params: dict) -> str:
|
||||
"""Append parameters to the callback, keeping its query string (spec 5.3)"""
|
||||
parts = urlsplit(callback)
|
||||
query = parts.query + ("&" if parts.query else "") + urlencode(params)
|
||||
return urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))
|
||||
|
||||
|
||||
def signature(secret: str, body: bytes) -> str:
|
||||
return "sha256=" + hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
def _later(seconds: int) -> str:
|
||||
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
class WebSubWorker(threading.Thread):
|
||||
"""Daemon thread that verifies subscriptions and delivers content"""
|
||||
|
||||
def __init__(self, app, poll_interval: int = 5):
|
||||
super().__init__(daemon=True, name="websub-worker")
|
||||
self.app = app
|
||||
self.poll_interval = poll_interval
|
||||
self._stop_event = threading.Event()
|
||||
self._lock_file = None
|
||||
self._last_prune = None
|
||||
|
||||
# --- thread plumbing ----------------------------------------------------
|
||||
|
||||
def run(self):
|
||||
logger.info("WebSub worker started")
|
||||
while not self._stop_event.is_set():
|
||||
if not self._acquire_lock():
|
||||
self._stop_event.wait(self.poll_interval)
|
||||
continue
|
||||
try:
|
||||
with self.app.app_context():
|
||||
done = self.process_once()
|
||||
except Exception as e:
|
||||
logger.error(f"WebSub worker error: {e}", exc_info=True)
|
||||
done = 0
|
||||
if not done:
|
||||
self._stop_event.wait(self.poll_interval)
|
||||
self._release_lock()
|
||||
|
||||
def stop(self):
|
||||
self._stop_event.set()
|
||||
|
||||
def _acquire_lock(self) -> bool:
|
||||
if self._lock_file:
|
||||
return True
|
||||
path = Path(self.app.config["DATABASE_PATH"]).parent / "websub-worker.lock"
|
||||
lock_file = open(path, "w")
|
||||
try:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
lock_file.close()
|
||||
return False
|
||||
self._lock_file = lock_file
|
||||
logger.info("WebSub worker holds the queue lock")
|
||||
return True
|
||||
|
||||
def _release_lock(self):
|
||||
if self._lock_file:
|
||||
self._lock_file.close()
|
||||
self._lock_file = None
|
||||
|
||||
# --- one pass (also called directly by tests) ---------------------------
|
||||
|
||||
def process_once(self) -> int:
|
||||
from starpunk.database import get_db
|
||||
|
||||
db = get_db(self.app)
|
||||
done = 0
|
||||
for row in db.execute(
|
||||
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY id LIMIT ?",
|
||||
(_now(), BATCH),
|
||||
).fetchall():
|
||||
self.verify(db, row)
|
||||
done += 1
|
||||
for row in db.execute(
|
||||
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY id LIMIT ?",
|
||||
(_now(), BATCH),
|
||||
).fetchall():
|
||||
self.deliver(db, row)
|
||||
done += 1
|
||||
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
|
||||
self.prune(db)
|
||||
return done
|
||||
|
||||
# --- verification of intent (spec 5.3) ----------------------------------
|
||||
|
||||
def verify(self, db, intent) -> None:
|
||||
from starpunk.websub import new_challenge
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, safe_get, user_agent
|
||||
|
||||
challenge = new_challenge()
|
||||
params = {"hub.mode": intent["mode"], "hub.topic": intent["topic"], "hub.challenge": challenge}
|
||||
if intent["mode"] == "subscribe":
|
||||
params["hub.lease_seconds"] = intent["lease_seconds"]
|
||||
url = callback_with_params(intent["callback"], params)
|
||||
|
||||
try:
|
||||
result = safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
|
||||
max_redirects=0, max_bytes=64 * 1024)
|
||||
verified = 200 <= result.status_code < 300 and result.text.strip() == challenge
|
||||
retry = result.status_code >= 500
|
||||
error = None if verified else f"HTTP {result.status_code}" + (
|
||||
"" if result.status_code >= 300 else ": challenge not echoed")
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
verified, retry, error = False, False, f"Blocked: {e}"
|
||||
except httpx.TooManyRedirects:
|
||||
verified, retry, error = False, False, "Redirected (not allowed)"
|
||||
except httpx.HTTPError as e:
|
||||
verified, retry, error = False, True, str(e)[:200]
|
||||
|
||||
if verified:
|
||||
if intent["mode"] == "subscribe":
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO websub_subscriptions
|
||||
(topic, callback, secret, lease_seconds, expires_at, verified_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||
secret = excluded.secret, lease_seconds = excluded.lease_seconds,
|
||||
expires_at = excluded.expires_at, verified_at = excluded.verified_at,
|
||||
last_error = NULL
|
||||
""",
|
||||
(intent["topic"], intent["callback"], intent["secret"], intent["lease_seconds"],
|
||||
_later(intent["lease_seconds"]), _now()),
|
||||
)
|
||||
else:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE topic = ? AND callback = ?",
|
||||
(intent["topic"], intent["callback"]))
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ?", (intent["id"],))
|
||||
logger.info(f"WebSub {intent['mode']} verified: {intent['callback']} -> {intent['topic']}")
|
||||
elif retry and intent["attempts"] < len(INTENT_BACKOFF):
|
||||
db.execute("UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? WHERE id = ?",
|
||||
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"]))
|
||||
else:
|
||||
# Verification failed: the subscription state stays as it was
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ?", (intent["id"],))
|
||||
logger.info(f"WebSub {intent['mode']} not verified for {intent['callback']}: {error}")
|
||||
db.commit()
|
||||
|
||||
# --- content distribution (spec 6, 7) ------------------------------------
|
||||
|
||||
def render_topic(self, topic: str):
|
||||
"""(status, content_type, body) of a topic, rendered in-app"""
|
||||
path = urlparse(topic).path or "/"
|
||||
response = self.app.test_client().get(path, headers={"Accept": "*/*"})
|
||||
return response.status_code, response.headers.get("Content-Type", ""), response.get_data()
|
||||
|
||||
def deliver(self, db, delivery) -> None:
|
||||
from starpunk.websub import hub_url
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, safe_post, user_agent
|
||||
|
||||
site = self.app.config["SITE_URL"]
|
||||
sub = None
|
||||
try:
|
||||
if delivery["subscription_id"] is None:
|
||||
# External hub: ping it (hub.url is the de facto parameter)
|
||||
result = safe_post(hub_url(), data={"hub.mode": "publish", "hub.url": delivery["topic"],
|
||||
"hub.topic": delivery["topic"]},
|
||||
headers={"User-Agent": user_agent(site)})
|
||||
else:
|
||||
sub = db.execute(
|
||||
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
|
||||
(delivery["subscription_id"], _now()),
|
||||
).fetchone()
|
||||
if sub is None: # unsubscribed or expired meanwhile
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
status, content_type, body = self.render_topic(sub["topic"])
|
||||
if status != 200: # e.g. a tag page whose last note went
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
headers = {
|
||||
"Content-Type": content_type,
|
||||
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
|
||||
"User-Agent": user_agent(site),
|
||||
}
|
||||
if sub["secret"]:
|
||||
headers["X-Hub-Signature"] = signature(sub["secret"], body)
|
||||
result = safe_post(sub["callback"], content=body, headers=headers)
|
||||
status, error = result.status_code, f"HTTP {result.status_code}"
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
status, error = None, f"Blocked: {e}"
|
||||
except httpx.HTTPError as e:
|
||||
status, error = None, str(e)[:200] or type(e).__name__
|
||||
|
||||
if status is not None and 200 <= status < 300:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_success_at = ?, last_error = NULL WHERE id = ?",
|
||||
(_now(), sub["id"]))
|
||||
elif sub is not None and status == 410:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
logger.info(f"WebSub subscriber gone (410), removed: {sub['callback']}")
|
||||
elif delivery["attempts"] < len(DELIVERY_BACKOFF) and not error.startswith("Blocked"):
|
||||
db.execute(
|
||||
"UPDATE websub_deliveries SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE id = ?",
|
||||
(_later(DELIVERY_BACKOFF[delivery["attempts"]]), error, delivery["id"]),
|
||||
)
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||
else:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.warning(f"WebSub delivery of {delivery['topic']} gave up: {error}")
|
||||
if sub is not None:
|
||||
self._drop_if_failing(db, sub, error)
|
||||
db.commit()
|
||||
|
||||
def _drop_if_failing(self, db, sub, error: str) -> None:
|
||||
last_ok = sub["last_success_at"] or sub["verified_at"] or sub["created_at"]
|
||||
try:
|
||||
since = datetime.fromisoformat(str(last_ok))
|
||||
except ValueError:
|
||||
return
|
||||
if datetime.utcnow() - since > DROP_AFTER:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
logger.info(f"WebSub subscription failing for {DROP_AFTER.days} days, removed: {sub['callback']}")
|
||||
else:
|
||||
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||
|
||||
# --- housekeeping -------------------------------------------------------
|
||||
|
||||
def prune(self, db) -> int:
|
||||
"""Remove subscriptions whose lease has run out (they weren't renewed)"""
|
||||
cursor = db.execute("DELETE FROM websub_subscriptions WHERE expires_at <= ?", (_now(),))
|
||||
db.commit()
|
||||
self._last_prune = datetime.utcnow()
|
||||
if cursor.rowcount:
|
||||
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
|
||||
return cursor.rowcount
|
||||
|
||||
|
||||
def start_worker(app) -> Optional[WebSubWorker]:
|
||||
if not app.config.get("WEBSUB_ENABLED", True) or app.config.get("TESTING", False):
|
||||
return None
|
||||
worker = WebSubWorker(app, poll_interval=app.config.get("WEBSUB_POLL_INTERVAL", 5))
|
||||
worker.start()
|
||||
return worker
|
||||
@@ -20,6 +20,21 @@
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
{# WebSub subscribers (v2.10.0) #}
|
||||
{% if websub_stats %}
|
||||
<p class="note-meta websub-stats">
|
||||
{% if websub_stats.external %}
|
||||
WebSub: using {{ websub_stats.external }}
|
||||
{% elif websub_stats.by_topic %}
|
||||
WebSub subscribers:
|
||||
{% for topic, count in websub_stats.by_topic %}{{ topic|replace(config.SITE_URL.rstrip('/'), '') or '/' }} {{ count }}{% if not loop.last %}, {% endif %}{% endfor %}
|
||||
{% if websub_stats.retrying %} · {{ websub_stats.retrying }} deliveries being retried{% endif %}
|
||||
{% else %}
|
||||
WebSub: no subscribers yet
|
||||
{% endif %}
|
||||
</p>
|
||||
{% endif %}
|
||||
|
||||
{% if notes %}
|
||||
<div class="table-scroll">
|
||||
<table class="note-table">
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
<form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form">
|
||||
<div class="form-group">
|
||||
{% if note.reacts_to %}
|
||||
<p class="note-meta">{{ '★ Like of' if note.kind == 'like' else '🔁 Repost of' }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
|
||||
<p class="note-meta">{{ {'like': '★ Like of', 'repost': '🔁 Repost of', 'bookmark': '🔖 Bookmark of'}[note.kind] }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
|
||||
{% endif %}
|
||||
<label for="content">{% if note.reacts_to %}Comment (optional, Markdown){% else %}Content (Markdown){% endif %}</label>
|
||||
<textarea
|
||||
|
||||
@@ -14,6 +14,13 @@
|
||||
<link rel="webmention" href="/webmention">
|
||||
{% endif %}
|
||||
|
||||
{# WebSub hub and topic on feed pages (v2.10.0) #}
|
||||
{% set websub = websub_links() %}
|
||||
{% if websub %}
|
||||
<link rel="hub" href="{{ websub.hub }}">
|
||||
<link rel="self" href="{{ websub.self }}">
|
||||
{% endif %}
|
||||
|
||||
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
|
||||
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
|
||||
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
{% from "partials/reaction.html" import reaction %}
|
||||
{% import "partials/responses.html" as responses with context %}
|
||||
|
||||
{% block title %}{% if note.has_title %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
|
||||
{% block title %}{% if note.has_title %}{{ note.title }}{% elif note.excerpt.strip() %}{{ note.excerpt|truncate(60) }}{% elif note.repost_of %}Repost{% elif note.like_of %}Like{% elif note.bookmark_of %}Bookmark{% elif note.in_reply_to %}Reply{% else %}Photo{% endif %} · {{ config.SITE_NAME }}{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<article class="h-entry">
|
||||
|
||||
@@ -4,8 +4,8 @@
|
||||
{% set ctx = note.context if note.context is defined else None %}
|
||||
{% if ctx and ctx.title %}{% set label = ctx.title %}{% elif ctx and ctx.content_snippet %}{% set label = ctx.content_snippet|striptags|truncate(90) %}{% elif ctx and (ctx.author_name or ctx.site_name) %}{% set label = 'a post' %}{% else %}{% set label = note.reacts_to %}{% endif %}
|
||||
<p class="reaction reaction-{{ note.kind }}">
|
||||
<span class="reaction-verb">{% if note.kind == 'like' %}★ Liked{% else %}🔁 Reposted{% endif %}</span>
|
||||
<a class="{{ 'u-like-of' if note.kind == 'like' else 'u-repost-of' }}" href="{{ note.reacts_to }}">{{ label }}</a>
|
||||
<span class="reaction-verb">{{ {'like': '★ Liked', 'repost': '🔁 Reposted', 'bookmark': '🔖 Bookmarked'}[note.kind] }}</span>
|
||||
<a class="u-{{ note.kind }}-of" href="{{ note.reacts_to }}">{{ label }}</a>
|
||||
{% if ctx and (ctx.author_name or ctx.site_name) %}<span class="reaction-by">by {{ ctx.author_name or ctx.site_name }}</span>{% endif %}
|
||||
</p>
|
||||
{% endif %}
|
||||
|
||||
+1
-1
@@ -230,7 +230,7 @@ class TestGenerateFeed:
|
||||
|
||||
# Check for Atom namespace and self-link
|
||||
assert "atom" in feed_xml
|
||||
assert "feed.xml" in feed_xml
|
||||
assert "feed.rss" in feed_xml # self link (v2.10.0: /feed.rss is the RSS topic)
|
||||
assert 'rel="self"' in feed_xml
|
||||
|
||||
def test_generate_feed_item_structure(self, app, sample_notes):
|
||||
|
||||
@@ -114,6 +114,11 @@ def post(app, monkeypatch):
|
||||
client = app.test_client()
|
||||
|
||||
def _post(**props):
|
||||
# Crosspost to every configured target; absent means none (v2.9.0).
|
||||
# Sent inside properties, as JSON clients like Quill do.
|
||||
with app.app_context():
|
||||
from starpunk.syndication import get_targets
|
||||
props["mp-syndicate-to"] = [t.uid for t in get_targets().values()]
|
||||
response = client.post("/micropub", json={"type": ["h-entry"], "properties": props},
|
||||
headers={"Authorization": f"Bearer {token}"})
|
||||
assert response.status_code == 201, response.get_json()
|
||||
|
||||
@@ -168,7 +168,7 @@ class TestImageValidation:
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
validate_image(corrupted_data, 'corrupt.jpg')
|
||||
|
||||
assert "Invalid or corrupted" in str(exc_info.value)
|
||||
assert "Not a supported image" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestHEICSupport:
|
||||
|
||||
@@ -87,8 +87,11 @@ class TestCreate:
|
||||
assert "photo" in source(client, h, response.headers["Location"])["properties"]
|
||||
|
||||
def test_remote_photo_is_downloaded(self, app, client, h, monkeypatch):
|
||||
transport = httpx.MockTransport(lambda req: httpx.Response(200, content=jpeg("blue")))
|
||||
monkeypatch.setattr(httpx, "stream", lambda method, url, **kw: httpx.Client(transport=transport).stream(method, url))
|
||||
# Fetched through the SSRF-checked, address-pinned fetcher (v2.9.0)
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT",
|
||||
httpx.MockTransport(lambda req: httpx.Response(200, content=jpeg("blue"))))
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||
lambda *a, **k: [(2, 1, 6, "", ("93.184.216.34", 0))])
|
||||
url = create(client, h, photo=[{"value": "https://photos.example/cat.jpg", "alt": "A cat"}])
|
||||
(photo,) = source(client, h, url)["properties"]["photo"]
|
||||
assert photo["value"].startswith(f"{SITE}/media/") and photo["alt"] == "A cat"
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
"""
|
||||
Micropub compliance fixes (v2.9.0)
|
||||
|
||||
Section numbers refer to the W3C Micropub Recommendation.
|
||||
"""
|
||||
|
||||
import io
|
||||
import sqlite3
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from tests.test_micropub_v240 import SITE, app, auth, client, create, h, jpeg, note_at, source # noqa: F401
|
||||
|
||||
|
||||
def syndications(app, url):
|
||||
note = note_at(app, url)
|
||||
db = sqlite3.connect(app.config["DATABASE_PATH"])
|
||||
return sorted(r[0] for r in db.execute("SELECT target FROM syndications WHERE note_id = ?", (note.id,)))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def targets(app):
|
||||
app.config.update(BLUESKY_HANDLE="me.example", BLUESKY_APP_PASSWORD="p")
|
||||
from starpunk import syndication
|
||||
syndication._client_cache.clear()
|
||||
with app.app_context():
|
||||
return {t.uid: name for name, t in syndication.get_targets().items()}
|
||||
|
||||
|
||||
# Crossposting: mp-* in JSON properties; absent means none
|
||||
|
||||
def test_mp_syndicate_to_inside_json_properties(app, client, h, targets):
|
||||
bsky_uid = next(uid for uid, name in targets.items() if name == "bluesky")
|
||||
url = create(client, h, **{"mp-syndicate-to": [bsky_uid]})
|
||||
assert syndications(app, url) == ["bluesky"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extra", [{}, {"mp-syndicate-to": []}])
|
||||
def test_no_choice_means_no_crossposting(app, client, h, targets, extra):
|
||||
url = create(client, h, **extra)
|
||||
assert syndications(app, url) == []
|
||||
|
||||
|
||||
def test_form_mp_syndicate_to(app, client, h, targets):
|
||||
bsky_uid = next(uid for uid, name in targets.items() if name == "bluesky")
|
||||
response = client.post("/micropub", data={"content": "x", "mp-syndicate-to[]": bsky_uid}, headers=h)
|
||||
assert syndications(app, response.headers["Location"]) == ["bluesky"]
|
||||
|
||||
|
||||
def test_mp_slug_inside_json_properties(app, client, h):
|
||||
url = create(client, h, **{"mp-slug": ["my-slug"]})
|
||||
assert url.endswith("/note/my-slug")
|
||||
assert "mp-slug" not in source(client, h, url)["properties"]
|
||||
|
||||
|
||||
# Scopes
|
||||
|
||||
def test_source_needs_a_posting_scope(app, client, h):
|
||||
create(client, h, **{"post-status": ["draft"]})
|
||||
profile_only = auth(app, "profile")
|
||||
assert client.get("/micropub?q=source", headers=profile_only).status_code == 403
|
||||
assert client.get("/micropub?q=category", headers=profile_only).status_code == 403
|
||||
assert client.get("/micropub?q=config", headers=profile_only).status_code == 200
|
||||
|
||||
|
||||
def test_draft_scope_only_creates_drafts(app, client):
|
||||
url = create(client, auth(app, "draft"))
|
||||
assert note_at(app, url).published is False
|
||||
|
||||
|
||||
# Visibility
|
||||
|
||||
@pytest.mark.parametrize("visibility,published", [("public", True), ("unlisted", False), ("private", False)])
|
||||
def test_visibility(app, client, h, visibility, published):
|
||||
url = create(client, h, visibility=[visibility])
|
||||
assert note_at(app, url).published is published
|
||||
|
||||
|
||||
def test_unknown_visibility_rejected(client, h):
|
||||
body = {"type": ["h-entry"], "properties": {"content": ["x"], "visibility": ["friends"]}}
|
||||
assert client.post("/micropub", json=body, headers=h).status_code == 400
|
||||
|
||||
|
||||
# Photos that can't be used fail the request
|
||||
|
||||
def test_unreachable_photo_is_an_error(app, client, h, monkeypatch):
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(lambda r: httpx.Response(404)))
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||
lambda *a, **k: [(2, 1, 6, "", ("93.184.216.34", 0))])
|
||||
body = {"type": ["h-entry"], "properties": {"content": ["x"], "photo": ["https://p.example/404.jpg"]}}
|
||||
assert client.post("/micropub", json=body, headers=h).status_code == 400
|
||||
assert client.get("/micropub?q=source", headers=h).get_json()["items"] == []
|
||||
|
||||
|
||||
def test_unknown_local_media_is_an_error_on_update(app, client, h):
|
||||
url = create(client, h)
|
||||
response = client.post("/micropub", json={"action": "update", "url": url,
|
||||
"replace": {"photo": [f"{SITE}/media/2026/11/nope.jpg"]}}, headers=h)
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
def test_redirect_to_private_address_is_not_fetched(app, client, h, monkeypatch):
|
||||
def resolve(host, *a, **k):
|
||||
return [(2, 1, 6, "", ("10.0.0.5" if host == "internal.example" else "93.184.216.34", 0))]
|
||||
|
||||
def site(request):
|
||||
return httpx.Response(302, headers={"Location": "http://internal.example/secret.jpg"})
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(site))
|
||||
body = {"type": ["h-entry"], "properties": {"content": ["x"], "photo": ["https://p.example/a.jpg"]}}
|
||||
assert client.post("/micropub", json=body, headers=h).status_code == 400
|
||||
|
||||
|
||||
# JSON values must be arrays (3.3.2)
|
||||
|
||||
def test_scalar_json_values_are_not_split(app, client, h):
|
||||
body = {"type": ["h-entry"], "properties": {"content": "whole text", "category": "solo", "name": "Title"}}
|
||||
response = client.post("/micropub", json=body, headers=h)
|
||||
props = source(client, h, response.headers["Location"])["properties"]
|
||||
assert props["content"] == ["whole text"]
|
||||
assert props["category"] == ["solo"]
|
||||
assert props["name"] == ["Title"]
|
||||
|
||||
|
||||
def test_update_delete_values_must_be_arrays(app, client, h):
|
||||
url = create(client, h, category=["a", "b"])
|
||||
response = client.post("/micropub", json={"action": "update", "url": url, "delete": {"category": "a"}}, headers=h)
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
def test_update_needs_an_operation(app, client, h):
|
||||
url = create(client, h)
|
||||
assert client.post("/micropub", json={"action": "update", "url": url}, headers=h).status_code == 400
|
||||
|
||||
|
||||
def test_person_tag_categories_are_ignored(app, client, h):
|
||||
url = create(client, h, category=["kept", {"type": ["h-card"], "properties": {"url": ["https://a.example/"]}}])
|
||||
assert source(client, h, url)["properties"]["category"] == ["kept"]
|
||||
|
||||
|
||||
# Properties that used to be dropped
|
||||
|
||||
def test_bookmark_of(app, client, h):
|
||||
body = {"type": ["h-entry"], "properties": {"bookmark-of": ["https://read.example/post"]}}
|
||||
url = client.post("/micropub", json=body, headers=h).headers["Location"]
|
||||
note = note_at(app, url)
|
||||
assert note.kind == "bookmark" and note.bookmark_of == "https://read.example/post"
|
||||
props = source(client, h, url)["properties"]
|
||||
assert props["bookmark-of"] == ["https://read.example/post"] and "content" not in props
|
||||
page = client.get(url.replace(SITE, "")).data.decode()
|
||||
assert 'class="u-bookmark-of" href="https://read.example/post"' in page
|
||||
|
||||
|
||||
def test_summary_or_name_without_content(app, client, h):
|
||||
response = client.post("/micropub", json={"type": ["h-entry"], "properties": {"summary": ["Just a summary"]}},
|
||||
headers=h)
|
||||
assert source(client, h, response.headers["Location"])["properties"]["content"] == ["Just a summary"]
|
||||
|
||||
response = client.post("/micropub", json={"type": ["h-entry"], "properties": {"name": ["Only a title"]}},
|
||||
headers=h)
|
||||
assert source(client, h, response.headers["Location"])["properties"]["name"] == ["Only a title"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("prop", ["video", "audio"])
|
||||
def test_unsupported_media_is_refused(client, h, prop):
|
||||
body = {"type": ["h-entry"], "properties": {"content": ["x"], prop: ["https://m.example/clip"]}}
|
||||
assert client.post("/micropub", json=body, headers=h).status_code == 400
|
||||
upload = {"content": "x", prop: (io.BytesIO(b"data"), "clip.mp4")}
|
||||
assert client.post("/micropub", data=upload, headers=h, content_type="multipart/form-data").status_code == 400
|
||||
|
||||
|
||||
# HTML content (3.7.2) and sanitizing
|
||||
|
||||
def test_html_content_comes_back_as_html(app, client, h):
|
||||
url = create(client, h, content=[{"html": "<p>Some <b>bold</b> text</p>"}])
|
||||
assert source(client, h, url)["properties"]["content"] == [{"html": "<p>Some <b>bold</b> text</p>"}]
|
||||
url = create(client, h, content=["plain *markdown*"])
|
||||
assert source(client, h, url)["properties"]["content"] == ["plain *markdown*"]
|
||||
|
||||
|
||||
def test_update_with_html_content(app, client, h):
|
||||
url = create(client, h)
|
||||
client.post("/micropub", json={"action": "update", "url": url,
|
||||
"replace": {"content": [{"html": "<p>new</p>"}]}}, headers=h)
|
||||
assert source(client, h, url)["properties"]["content"] == [{"html": "<p>new</p>"}]
|
||||
|
||||
|
||||
def test_html_is_sanitized_on_the_page(app, client, h):
|
||||
url = create(client, h, content=[{"html": '<p>hi<script>alert(1)</script><img src="/x.jpg" onerror="evil()">'
|
||||
'<a href="javascript:alert(2)">j</a><iframe src="https://e.example"></iframe></p>'}])
|
||||
page = client.get(url.replace(SITE, "")).data.decode()
|
||||
for bad in ("<script>alert(1)", "onerror", "javascript:alert", "<iframe"):
|
||||
assert bad not in page
|
||||
assert '<img src="/x.jpg">' in page
|
||||
|
||||
|
||||
# published (3.3)
|
||||
|
||||
def test_published_is_stored_as_utc(app, client, h):
|
||||
url = create(client, h, published=["2020-02-03T04:05:06-08:00"])
|
||||
assert source(client, h, url)["properties"]["published"] == ["2020-02-03T12:05:06Z"]
|
||||
created = note_at(app, url).created_at
|
||||
assert created.tzinfo is None and (created.hour, created.minute) == (12, 5)
|
||||
|
||||
|
||||
def test_unparseable_published_is_an_error(client, h):
|
||||
body = {"type": ["h-entry"], "properties": {"content": ["x"], "published": ["yesterday"]}}
|
||||
assert client.post("/micropub", json=body, headers=h).status_code == 400
|
||||
|
||||
|
||||
# Errors (3.8) and authentication (5)
|
||||
|
||||
@pytest.mark.parametrize("payload", ["{not json", "[1, 2]"])
|
||||
def test_malformed_json_is_400(client, h, payload):
|
||||
response = client.post("/micropub", data=payload, headers={**h, "Content-Type": "application/json"})
|
||||
assert response.status_code == 400
|
||||
assert response.get_json()["error"] == "invalid_request"
|
||||
|
||||
|
||||
def test_bearer_scheme_is_case_insensitive(client, h):
|
||||
lower = {"Authorization": h["Authorization"].replace("Bearer", "bearer")}
|
||||
assert client.get("/micropub?q=config", headers=lower).status_code == 200
|
||||
|
||||
|
||||
def test_token_in_header_and_body_is_400(client, h):
|
||||
response = client.post("/micropub", data={"content": "x", "access_token": "other"}, headers=h)
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
def test_401_has_www_authenticate(client):
|
||||
response = client.get("/micropub?q=config")
|
||||
assert response.status_code == 401
|
||||
assert response.headers["WWW-Authenticate"].startswith("Bearer")
|
||||
response = client.get("/micropub?q=config", headers={"Authorization": "Bearer nope"})
|
||||
assert 'error="invalid_token"' in response.headers["WWW-Authenticate"]
|
||||
|
||||
|
||||
def test_media_error_does_not_leak_internals(client, h):
|
||||
response = client.post("/micropub/media", data={"file": (io.BytesIO(b"not an image"), "a.jpg")},
|
||||
headers=h, content_type="multipart/form-data")
|
||||
assert response.status_code == 400
|
||||
assert "BytesIO" not in response.get_json()["error_description"]
|
||||
|
||||
|
||||
def test_config_lists_bookmarks(client, h):
|
||||
types = {t["type"] for t in client.get("/micropub?q=config", headers=h).get_json()["post-types"]}
|
||||
assert "bookmark" in types
|
||||
@@ -0,0 +1,86 @@
|
||||
"""
|
||||
Migrations under concurrent gunicorn workers (v2.8.1)
|
||||
|
||||
The v2.8.0 deploy failed to boot: one worker applied migration 014 while
|
||||
another applied it again ("duplicate column name"), because
|
||||
conn.executescript() committed and released the migration lock early.
|
||||
"""
|
||||
|
||||
import multiprocessing
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from starpunk.database.schema import INITIAL_SCHEMA_SQL
|
||||
from starpunk.migrations import execute_in_transaction, run_migrations, split_sql_statements
|
||||
|
||||
MIGRATIONS = Path(__file__).parent.parent / "migrations"
|
||||
|
||||
|
||||
def test_script_runs_inside_the_transaction(tmp_path):
|
||||
conn = sqlite3.connect(tmp_path / "t.db")
|
||||
conn.execute("CREATE TABLE t (a INTEGER)")
|
||||
conn.commit()
|
||||
|
||||
conn.execute("BEGIN IMMEDIATE")
|
||||
execute_in_transaction(conn, "ALTER TABLE t ADD COLUMN b TEXT;\nALTER TABLE t ADD COLUMN c TEXT;")
|
||||
conn.rollback()
|
||||
|
||||
assert [row[1] for row in conn.execute("PRAGMA table_info(t)")] == ["a"]
|
||||
|
||||
|
||||
def test_split_keeps_trigger_bodies_together():
|
||||
script = """
|
||||
-- comment; with a semicolon
|
||||
CREATE TABLE t (a TEXT DEFAULT 'x;y');
|
||||
CREATE TRIGGER tr AFTER DELETE ON t BEGIN
|
||||
DELETE FROM t WHERE a = old.a;
|
||||
SELECT 1;
|
||||
END;
|
||||
"""
|
||||
statements = split_sql_statements(script)
|
||||
assert len(statements) == 2
|
||||
assert statements[1].startswith("CREATE TRIGGER") and statements[1].endswith("END;")
|
||||
|
||||
|
||||
def _migrate(db_path, barrier, errors):
|
||||
try:
|
||||
barrier.wait()
|
||||
run_migrations(db_path)
|
||||
except Exception as e: # report to the parent
|
||||
errors.put(repr(e))
|
||||
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.parametrize("trial", range(5)) # the race hit ~1 in 5 runs
|
||||
def test_concurrent_workers_apply_each_migration_once(tmp_path, trial):
|
||||
# A database at migration 013, like production before v2.8.0
|
||||
db_path = tmp_path / "starpunk.db"
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.executescript(INITIAL_SCHEMA_SQL) # as init_db does
|
||||
conn.close()
|
||||
run_migrations(db_path)
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.execute("DELETE FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'")
|
||||
conn.commit()
|
||||
columns = [row[1] for row in conn.execute("PRAGMA table_info(webmentions)")]
|
||||
if "source_post_url" in columns:
|
||||
conn.execute("ALTER TABLE webmentions DROP COLUMN source_post_url")
|
||||
conn.execute("ALTER TABLE webmentions DROP COLUMN source_published")
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
ctx = multiprocessing.get_context("fork")
|
||||
barrier, errors = ctx.Barrier(4), ctx.Queue()
|
||||
workers = [ctx.Process(target=_migrate, args=(db_path, barrier, errors)) for _ in range(4)]
|
||||
for w in workers:
|
||||
w.start()
|
||||
for w in workers:
|
||||
w.join(timeout=60)
|
||||
|
||||
assert errors.empty(), errors.get()
|
||||
conn = sqlite3.connect(db_path)
|
||||
assert conn.execute(
|
||||
"SELECT COUNT(*) FROM schema_migrations WHERE migration_name = '014_webmention_source_details.sql'"
|
||||
).fetchone()[0] == 1
|
||||
@@ -302,12 +302,13 @@ class TestErrorHandling:
|
||||
|
||||
def test_rollback_failure_causes_system_exit(self, temp_db):
|
||||
"""Test that rollback failure raises SystemExit"""
|
||||
with patch('sqlite3.connect') as mock_connect:
|
||||
with patch('sqlite3.connect') as mock_connect, \
|
||||
patch('starpunk.migrations.execute_in_transaction',
|
||||
side_effect=Exception("Migration failed")):
|
||||
mock_conn = MagicMock()
|
||||
mock_connect.return_value = mock_conn
|
||||
|
||||
# Make both migration and rollback fail
|
||||
mock_conn.executescript.side_effect = Exception("Migration failed")
|
||||
mock_conn.rollback.side_effect = Exception("Rollback failed")
|
||||
mock_conn.execute.return_value.fetchone.return_value = (0,)
|
||||
|
||||
|
||||
@@ -356,7 +356,8 @@ class TestEntryPoints:
|
||||
).get_json()
|
||||
assert data["syndicate-to"] == [{"uid": f"{GTS}/", "name": "GoToSocial"}]
|
||||
|
||||
def test_micropub_default_is_all_targets(self, app):
|
||||
def test_micropub_default_is_no_targets(self, app):
|
||||
"""Absent mp-syndicate-to means no crossposting (v2.9.0)"""
|
||||
token = self._token(app)
|
||||
response = app.test_client().post(
|
||||
"/micropub", data={"h": "entry", "content": "via quill"},
|
||||
@@ -367,7 +368,7 @@ class TestEntryPoints:
|
||||
assert response.headers["Location"].startswith("https://blog.example.com/note/")
|
||||
with app.app_context():
|
||||
from starpunk.database import get_db
|
||||
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 1
|
||||
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 0
|
||||
|
||||
def test_micropub_explicit_empty_choice(self, app):
|
||||
token = self._token(app)
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
"""
|
||||
WebSub publisher and built-in hub (v2.10.0, ADR-071)
|
||||
|
||||
Section numbers refer to the W3C WebSub Recommendation. Subscribers are
|
||||
simulated with an httpx MockTransport behind the SSRF-safe fetcher.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import sqlite3
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
import httpx
|
||||
import mf2py
|
||||
import pytest
|
||||
|
||||
from starpunk.websub_worker import WebSubWorker
|
||||
|
||||
SITE = "http://localhost:5000"
|
||||
HUB = f"{SITE}/websub"
|
||||
PUBLIC = [(2, 1, 6, "", ("93.184.216.34", 0))]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def web(monkeypatch):
|
||||
"""Subscriber callbacks: handler(request) -> response, by host"""
|
||||
hosts, seen = {}, []
|
||||
|
||||
def handler(request):
|
||||
host = request.headers["host"]
|
||||
seen.append(request)
|
||||
return hosts[host](request)
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
|
||||
return hosts, seen
|
||||
|
||||
|
||||
def echo(request):
|
||||
"""A subscriber that confirms every verification and accepts content"""
|
||||
if request.method == "GET":
|
||||
return httpx.Response(200, text=parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0])
|
||||
return httpx.Response(202)
|
||||
|
||||
|
||||
def db(app):
|
||||
conn = sqlite3.connect(app.config["DATABASE_PATH"])
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def run_worker(app, times=1):
|
||||
worker = WebSubWorker(app)
|
||||
with app.app_context():
|
||||
for _ in range(times):
|
||||
worker.process_once()
|
||||
return worker
|
||||
|
||||
|
||||
def subscribe(client, topic, callback="https://sub.example/cb?id=1", **extra):
|
||||
form = {"hub.mode": "subscribe", "hub.topic": topic, "hub.callback": callback, **extra}
|
||||
return client.post("/websub", data=form)
|
||||
|
||||
|
||||
def make_note(app, content="Hello", **kw):
|
||||
from starpunk.notes import create_note
|
||||
with app.test_request_context():
|
||||
return create_note(content, published=kw.pop("published", True), **kw)
|
||||
|
||||
|
||||
def links(response):
|
||||
return response.headers.getlist("Link")
|
||||
|
||||
|
||||
# 4 Discovery -------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("path,self_url", [
|
||||
("/", f"{SITE}/"),
|
||||
("/feed.atom", f"{SITE}/feed.atom"),
|
||||
("/feed.rss", f"{SITE}/feed.rss"),
|
||||
("/feed.xml", f"{SITE}/feed.rss"),
|
||||
("/feed.json", f"{SITE}/feed.json"),
|
||||
("/likes", f"{SITE}/likes"),
|
||||
])
|
||||
def test_topics_advertise_hub_and_self(client, path, self_url):
|
||||
response = client.get(path)
|
||||
values = links(response)
|
||||
assert f'<{HUB}>; rel="hub"' in values
|
||||
assert [v for v in values if 'rel="self"' in v] == [f'<{self_url}>; rel="self"']
|
||||
|
||||
|
||||
def test_negotiated_feed_self_is_the_format_served(client):
|
||||
response = client.get("/feed", headers={"Accept": "application/atom+xml"})
|
||||
assert f'<{SITE}/feed.atom>; rel="self"' in links(response)
|
||||
|
||||
|
||||
def test_tag_page_is_a_topic(app, client):
|
||||
make_note(app, tags=["Python"])
|
||||
assert f'<{SITE}/tag/python>; rel="self"' in links(client.get("/tag/Python"))
|
||||
|
||||
|
||||
def test_note_page_is_not_a_topic(app, client):
|
||||
note = make_note(app)
|
||||
assert not [v for v in links(client.get(f"/note/{note.slug}")) if 'rel="hub"' in v]
|
||||
|
||||
|
||||
def test_embedded_links(client):
|
||||
rels = mf2py.parse(doc=client.get("/").data.decode(), url=SITE + "/")["rels"]
|
||||
assert rels["hub"] == [HUB] and rels["self"] == [f"{SITE}/"]
|
||||
assert f'<link rel="hub" href="{HUB}"/>' in client.get("/feed.atom").data.decode()
|
||||
assert f'<atom:link href="{HUB}" rel="hub"/>' in client.get("/feed.rss").data.decode()
|
||||
assert json.loads(client.get("/feed.json").data)["hubs"] == [{"type": "WebSub", "url": HUB}]
|
||||
|
||||
|
||||
def test_external_hub(app, client, web):
|
||||
app.config["WEBSUB_HUB_URL"] = "https://hub.example/"
|
||||
hosts, seen = web
|
||||
hosts["hub.example"] = lambda r: httpx.Response(204)
|
||||
assert '<https://hub.example/>; rel="hub"' in links(client.get("/"))
|
||||
assert subscribe(client, f"{SITE}/").status_code == 400 # our hub is off
|
||||
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
pings = {parse_qs(r.content.decode())["hub.url"][0] for r in seen}
|
||||
assert pings == {f"{SITE}/", f"{SITE}/feed.atom", f"{SITE}/feed.rss", f"{SITE}/feed.json"}
|
||||
|
||||
|
||||
# 5.1 Subscription requests ----------------------------------------------------------
|
||||
|
||||
def test_subscription_request_is_accepted(app, client, web):
|
||||
response = subscribe(client, f"{SITE}/feed.atom", **{"hub.secret": "s3cret", "extra": "ignored"})
|
||||
assert response.status_code == 202
|
||||
row = db(app).execute("SELECT * FROM websub_intents").fetchone()
|
||||
assert (row["mode"], row["topic"], row["secret"]) == ("subscribe", f"{SITE}/feed.atom", "s3cret")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("form,reason", [
|
||||
({"hub.mode": "watch"}, "hub.mode"),
|
||||
({"hub.topic": "https://elsewhere.example/feed"}, "hub.topic"),
|
||||
({"hub.topic": f"{SITE}/note/x"}, "hub.topic"),
|
||||
({"hub.topic": f"{SITE}/tag/missing"}, "hub.topic"),
|
||||
({"hub.callback": "ftp://sub.example/cb"}, "hub.callback"),
|
||||
({"hub.secret": "x" * 200}, "hub.secret"),
|
||||
])
|
||||
def test_bad_requests(client, web, form, reason):
|
||||
base = {"hub.mode": "subscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb"}
|
||||
response = client.post("/websub", data={**base, **form})
|
||||
assert response.status_code == 400 and reason in response.data.decode()
|
||||
|
||||
|
||||
def test_private_callback_refused(client, monkeypatch):
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||
lambda *a, **k: [(2, 1, 6, "", ("10.0.0.2", 0))])
|
||||
assert subscribe(client, f"{SITE}/").status_code == 400
|
||||
|
||||
|
||||
def test_per_host_limit(app, client, web):
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = 1
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
assert subscribe(client, f"{SITE}/feed.atom").status_code == 429
|
||||
|
||||
|
||||
# 5.3 Verification of intent --------------------------------------------------------
|
||||
|
||||
def test_verification_activates_subscription(app, client, web):
|
||||
hosts, seen = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/", **{"hub.lease_seconds": "86400"})
|
||||
run_worker(app)
|
||||
|
||||
query = parse_qs(urlparse(str(seen[0].url)).query)
|
||||
assert query["id"] == ["1"] # the callback's own query string is kept
|
||||
assert query["hub.mode"] == ["subscribe"] and query["hub.topic"] == [f"{SITE}/"]
|
||||
assert query["hub.lease_seconds"] == ["86400"]
|
||||
sub = db(app).execute("SELECT * FROM websub_subscriptions").fetchone()
|
||||
assert sub["topic"] == f"{SITE}/" and sub["verified_at"]
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("response", [
|
||||
httpx.Response(200, text="wrong"),
|
||||
httpx.Response(404),
|
||||
httpx.Response(302, headers={"Location": "https://sub.example/other"}),
|
||||
])
|
||||
def test_failed_verification_changes_nothing(app, client, web, response):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = lambda r: response
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_unsubscribe(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
client.post("/websub", data={"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/",
|
||||
"hub.callback": "https://sub.example/cb?id=1"})
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_resubscribe_changes_secret_only_once_verified(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "one"})
|
||||
run_worker(app)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(404)
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "two"})
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT secret FROM websub_subscriptions").fetchone()[0] == "one"
|
||||
|
||||
|
||||
# 6, 7 Content distribution ---------------------------------------------------------
|
||||
|
||||
def _subscribed(app, client, web, topic=f"{SITE}/feed.atom", secret=None):
|
||||
hosts, seen = web
|
||||
hosts["sub.example"] = echo
|
||||
extra = {"hub.secret": secret} if secret else {}
|
||||
subscribe(client, topic, **extra)
|
||||
run_worker(app)
|
||||
seen.clear()
|
||||
return hosts, seen
|
||||
|
||||
|
||||
def test_new_note_is_delivered_signed(app, client, web):
|
||||
hosts, seen = _subscribed(app, client, web, secret="s3cret")
|
||||
make_note(app, "Fresh post")
|
||||
run_worker(app)
|
||||
|
||||
(post,) = [r for r in seen if r.method == "POST"]
|
||||
assert post.headers["content-type"].startswith("application/atom+xml")
|
||||
assert b"Fresh post" in post.content
|
||||
assert f'<{HUB}>; rel="hub"' in post.headers["link"] and f'<{SITE}/feed.atom>; rel="self"' in post.headers["link"]
|
||||
expected = "sha256=" + hmac.new(b"s3cret", post.content, hashlib.sha256).hexdigest()
|
||||
assert post.headers["x-hub-signature"] == expected
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_no_signature_without_secret(app, client, web):
|
||||
hosts, seen = _subscribed(app, client, web)
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
assert "x-hub-signature" not in [r for r in seen if r.method == "POST"][0].headers
|
||||
|
||||
|
||||
def test_drafts_are_not_delivered(app, client, web):
|
||||
_subscribed(app, client, web)
|
||||
make_note(app, published=False)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_tag_topics_include_removed_tags(app, client, web):
|
||||
from starpunk.notes import update_note
|
||||
note = make_note(app, tags=["old"])
|
||||
_subscribed(app, client, web, topic=f"{SITE}/tag/old")
|
||||
with app.test_request_context():
|
||||
update_note(id=note.id, content="Hello", tags=["new"])
|
||||
assert db(app).execute("SELECT topic FROM websub_deliveries").fetchone()[0] == f"{SITE}/tag/old"
|
||||
|
||||
|
||||
def test_delete_is_delivered(app, client, web):
|
||||
from starpunk.notes import delete_note
|
||||
note = make_note(app, "Going away")
|
||||
hosts, seen = _subscribed(app, client, web)
|
||||
with app.test_request_context():
|
||||
delete_note(id=note.id, soft=False)
|
||||
run_worker(app)
|
||||
(post,) = [r for r in seen if r.method == "POST"]
|
||||
assert b"Going away" not in post.content
|
||||
|
||||
|
||||
def test_410_removes_subscription(app, client, web):
|
||||
hosts, _ = _subscribed(app, client, web)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(410)
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_failures_are_retried(app, client, web):
|
||||
hosts, _ = _subscribed(app, client, web)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(500)
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
row = db(app).execute("SELECT attempts, next_attempt_at, last_error FROM websub_deliveries").fetchone()
|
||||
assert row["attempts"] == 1 and row["last_error"] == "HTTP 500"
|
||||
|
||||
|
||||
def test_expired_subscriptions_get_nothing(app, client, web):
|
||||
_subscribed(app, client, web)
|
||||
conn = db(app)
|
||||
conn.execute("UPDATE websub_subscriptions SET expires_at = datetime('now', '-1 minute')")
|
||||
conn.commit()
|
||||
make_note(app)
|
||||
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
run_worker(app) # prunes
|
||||
assert conn.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
|
||||
|
||||
# Feeds are fresh right after a change (ADR-071 3) ------------------------------------
|
||||
|
||||
def test_feed_shows_new_note_immediately(app, client):
|
||||
from starpunk.routes import public
|
||||
make_note(app, "First")
|
||||
assert b"First" in client.get("/feed.atom").data
|
||||
make_note(app, "Second") # cached note list is stale now
|
||||
assert b"Second" in client.get("/feed.atom").data
|
||||
assert public._feed_cache["notes"] is not None
|
||||
Reference in new issue
Block a user