Merge branch 'feat/v2.10.0-websub' for v2.10.0 release
Build Container / build (push) Successful in 7m29s
Build Container / build (push) Successful in 7m29s
This commit is contained in:
commit
409b8b31b4
19 files changed
+1368
-28
No files matched your search
@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [2.10.0] - 2026-11-03
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **WebSub** ([ADR-071](docs/decisions/ADR-071-websub.md)): readers that support it get new posts, edits and deletions within seconds instead of polling. StarPunk runs its own hub at `/websub` for its own feeds: the home page, `/feed.atom`, `/feed.rss`, `/feed.json`, `/likes` and each tag page. Each advertises the hub and itself (`Link` headers, and links inside the page or feed). Subscriptions are verified with a challenge, leases expire, deliveries carry the full feed, signed with `X-Hub-Signature` when the subscriber gave a secret, and failures are retried for about nine hours. Requests to subscribers go through the same private-address protections as webmentions, and there are limits per subscriber host and in total. Set `WEBSUB_HUB_URL` to use another hub instead; `WEBSUB_ENABLED=false` turns it off
|
||||||
|
- The admin dashboard shows WebSub subscribers per feed
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Feeds could be up to five minutes out of date after posting, editing or deleting, and each server process cached its own copy. Any change to a public note now refreshes the feeds everywhere at once
|
||||||
|
- The memory monitor stopped after the first request (the same bug the webmention worker had before v2.6.8)
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- The RSS feed calls itself `/feed.rss` instead of the older `/feed.xml`. Both addresses still work
|
||||||
|
|
||||||
## [2.9.0] - 2026-11-03
|
## [2.9.0] - 2026-11-03
|
||||||
|
|
||||||
Micropub review against the [W3C Recommendation](https://www.w3.org/TR/micropub/).
|
Micropub review against the [W3C Recommendation](https://www.w3.org/TR/micropub/).
|
||||||
|
|||||||
@@ -0,0 +1,166 @@
|
|||||||
|
# ADR-071: WebSub publishing with a built-in hub
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted (v2.10.0, 2026-11-03)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Readers poll StarPunk's feeds. WebSub (W3C Recommendation,
|
||||||
|
https://www.w3.org/TR/websub/) lets readers that support it (Monocle,
|
||||||
|
Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and
|
||||||
|
have new content pushed to them within seconds.
|
||||||
|
|
||||||
|
WebSub has three roles. StarPunk is the **publisher**. Someone has to run
|
||||||
|
the **hub** that keeps subscriptions and delivers content. Public hubs
|
||||||
|
exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a
|
||||||
|
third-party dependency and some have shut down. The rest of this site is
|
||||||
|
self-hosted.
|
||||||
|
|
||||||
|
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
|
||||||
|
|
||||||
|
Two existing problems matter here:
|
||||||
|
|
||||||
|
- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears
|
||||||
|
the cache when a note changes. A hub fetching right after a change would
|
||||||
|
get stale content. Regular readers are also up to 5 minutes behind.
|
||||||
|
- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and
|
||||||
|
calls itself `/feed.xml`.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
StarPunk runs its own hub, **for its own topics only**, at `/websub`.
|
||||||
|
`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk
|
||||||
|
advertises that hub, pings it on changes, and its own `/websub` is off.
|
||||||
|
|
||||||
|
### 1. Topics
|
||||||
|
|
||||||
|
Absolute URLs under `SITE_URL`:
|
||||||
|
|
||||||
|
| Topic | Content type |
|
||||||
|
|---|---|
|
||||||
|
| `/` (h-feed) | text/html |
|
||||||
|
| `/feed.atom` | application/atom+xml |
|
||||||
|
| `/feed.rss` | application/rss+xml |
|
||||||
|
| `/feed.json` | application/feed+json |
|
||||||
|
| `/likes` | text/html |
|
||||||
|
| `/tag/<name>` | text/html |
|
||||||
|
|
||||||
|
`/feed.xml` and the negotiated `/feed` advertise the topic of the format
|
||||||
|
they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The
|
||||||
|
RSS feed's own self link changes from `/feed.xml` to `/feed.rss`.
|
||||||
|
|
||||||
|
### 2. Discovery (publisher, spec 4)
|
||||||
|
|
||||||
|
Every topic response carries `Link: <hub>; rel="hub"` and exactly one
|
||||||
|
`Link: <topic>; rel="self"`. Embedded links as well: `<link rel="hub">` and
|
||||||
|
`<link rel="self">` in the HTML head of topic pages, `<link rel="hub">`
|
||||||
|
next to the existing self link in Atom, `<atom:link rel="hub">` in RSS, and
|
||||||
|
`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed.
|
||||||
|
|
||||||
|
### 3. What changed, and when
|
||||||
|
|
||||||
|
`websub.note_changed(note, old_tags=())` is called wherever
|
||||||
|
`syndication.note_changed` is, plus on create. It:
|
||||||
|
|
||||||
|
1. clears the feed cache (all formats), and
|
||||||
|
2. if the note is or was public, queues delivery for `/`, the three
|
||||||
|
feeds, `/likes` if it is a like, and each `/tag/<name>` for its tags
|
||||||
|
before and after the change.
|
||||||
|
|
||||||
|
Drafts that stay drafts change nothing.
|
||||||
|
|
||||||
|
### 4. Hub endpoint (spec 5)
|
||||||
|
|
||||||
|
`POST /websub`, `application/x-www-form-urlencoded`:
|
||||||
|
|
||||||
|
- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the
|
||||||
|
topics above (after normalising), else 400 with a plain-text reason;
|
||||||
|
`hub.callback` must be an absolute http(s) URL that passes the SSRF
|
||||||
|
check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to
|
||||||
|
1 day to 30 days (default 10 days). Unknown parameters are ignored.
|
||||||
|
- Response: 202 Accepted, independent of what verification later finds.
|
||||||
|
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions
|
||||||
|
per callback host and 1,000 in total; beyond that, 429.
|
||||||
|
|
||||||
|
### 5. Verification of intent (spec 5.3)
|
||||||
|
|
||||||
|
A background worker sends `GET <callback>` with the callback's query
|
||||||
|
string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32
|
||||||
|
bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body
|
||||||
|
equal to the challenge; anything else, including redirects, fails and
|
||||||
|
leaves the subscription state unchanged. On success a subscribe becomes
|
||||||
|
active with `expires_at = now + lease`, and an unsubscribe deletes it.
|
||||||
|
A re-subscribe replaces the secret and lease only once verified.
|
||||||
|
|
||||||
|
### 6. Content distribution (spec 6, 7)
|
||||||
|
|
||||||
|
For each change, one delivery per active, unexpired subscription to the
|
||||||
|
topic. The worker renders the topic inside the app (no HTTP round trip
|
||||||
|
through Caddy), so subscribers get exactly what the public URL serves,
|
||||||
|
then `POST`s it to the callback with:
|
||||||
|
|
||||||
|
- `Content-Type` of the topic,
|
||||||
|
- `Link` headers with `rel=hub` and `rel=self`,
|
||||||
|
- `X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>` if a secret was
|
||||||
|
given.
|
||||||
|
|
||||||
|
A 2xx is success. 410 Gone removes the subscription. Anything else is
|
||||||
|
retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a
|
||||||
|
subscription whose deliveries fail for 3 days is removed. Only the latest
|
||||||
|
content is sent: a queued delivery for a topic is replaced by a newer one.
|
||||||
|
|
||||||
|
### 7. Outbound safety
|
||||||
|
|
||||||
|
Callbacks are arbitrary URLs, so verification and delivery go through
|
||||||
|
`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the
|
||||||
|
connection pinned to the checked address, no redirects, 10 s time limit,
|
||||||
|
1 MB response cap.
|
||||||
|
|
||||||
|
### 8. Data (migration 016)
|
||||||
|
|
||||||
|
- `websub_subscriptions(id, topic, callback, secret, lease_seconds,
|
||||||
|
state ['pending','active'], expires_at, created_at, verified_at,
|
||||||
|
last_success_at, last_error)`, unique on `(topic, callback)`.
|
||||||
|
- `websub_intents(id, topic, callback, mode, secret, lease_seconds,
|
||||||
|
attempts, next_attempt_at, created_at)`: verifications to do.
|
||||||
|
- `websub_deliveries(id, subscription_id, attempts, next_attempt_at,
|
||||||
|
last_error, created_at)`, unique on `subscription_id`.
|
||||||
|
|
||||||
|
### 9. Worker
|
||||||
|
|
||||||
|
The same pattern as the Webmention worker: one thread per gunicorn
|
||||||
|
process, a file lock so only one process works the queues, app context
|
||||||
|
around each poll. Expired subscriptions are pruned once a day.
|
||||||
|
|
||||||
|
### 10. Admin
|
||||||
|
|
||||||
|
The admin dashboard shows active subscribers per topic and failing
|
||||||
|
deliveries.
|
||||||
|
|
||||||
|
### 11. Configuration
|
||||||
|
|
||||||
|
`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in),
|
||||||
|
`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Readers that support WebSub get posts and edits within seconds.
|
||||||
|
- Feeds are fresh right after a change for every reader, because the cache
|
||||||
|
is cleared.
|
||||||
|
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF
|
||||||
|
protections, per-host and total limits, and rate limiting bound that.
|
||||||
|
- websub.rocks' hub tests use websub.rocks' own topics, which this hub
|
||||||
|
refuses by design. Testing uses unit tests of each spec section and a
|
||||||
|
real subscription from a reader after deploy.
|
||||||
|
- The RSS self link changes to `/feed.rss`. Readers keep whichever URL
|
||||||
|
they subscribed to; both keep working.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
- **Public hub only.** Least code, but a third-party dependency for a
|
||||||
|
self-hosted site. Still available through `WEBSUB_HUB_URL`.
|
||||||
|
- **A separate hub service** (another container). More to deploy and
|
||||||
|
keep updated, and few maintained hubs exist.
|
||||||
|
- **Accepting any topic** (a general-purpose hub). Not needed, and it
|
||||||
|
would let the hub be used to fetch and push arbitrary URLs.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
-- Migration 016: WebSub hub (v2.10.0, ADR-071)
|
||||||
|
|
||||||
|
-- Active and pending subscriptions to this site's topics
|
||||||
|
CREATE TABLE IF NOT EXISTS websub_subscriptions (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
topic TEXT NOT NULL,
|
||||||
|
callback TEXT NOT NULL,
|
||||||
|
secret TEXT,
|
||||||
|
lease_seconds INTEGER NOT NULL,
|
||||||
|
expires_at TIMESTAMP NOT NULL,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
verified_at TIMESTAMP,
|
||||||
|
last_success_at TIMESTAMP,
|
||||||
|
last_error TEXT,
|
||||||
|
UNIQUE(topic, callback)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_topic ON websub_subscriptions(topic);
|
||||||
|
|
||||||
|
-- Subscribe/unsubscribe requests waiting for verification of intent
|
||||||
|
CREATE TABLE IF NOT EXISTS websub_intents (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
topic TEXT NOT NULL,
|
||||||
|
callback TEXT NOT NULL,
|
||||||
|
mode TEXT NOT NULL CHECK (mode IN ('subscribe', 'unsubscribe')),
|
||||||
|
secret TEXT,
|
||||||
|
lease_seconds INTEGER,
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
UNIQUE(topic, callback)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Content to push: one row per subscription (built-in hub) or per topic
|
||||||
|
-- (ping to an external hub, subscription_id NULL)
|
||||||
|
CREATE TABLE IF NOT EXISTS websub_deliveries (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
subscription_id INTEGER REFERENCES websub_subscriptions(id) ON DELETE CASCADE,
|
||||||
|
topic TEXT NOT NULL,
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
last_error TEXT,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_subscription
|
||||||
|
ON websub_deliveries(subscription_id) WHERE subscription_id IS NOT NULL;
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_ping
|
||||||
|
ON websub_deliveries(topic) WHERE subscription_id IS NULL;
|
||||||
|
|
||||||
|
-- Site-wide values. content_version changes whenever a public note does, so
|
||||||
|
-- every gunicorn process knows its cached feed note list is stale.
|
||||||
|
CREATE TABLE IF NOT EXISTS site_state (
|
||||||
|
key TEXT PRIMARY KEY,
|
||||||
|
value TEXT NOT NULL
|
||||||
|
);
|
||||||
|
INSERT OR IGNORE INTO site_state (key, value) VALUES ('content_version', '0');
|
||||||
+15
-8
@@ -309,6 +309,11 @@ def create_app(config=None):
|
|||||||
response.headers.add('Link', '</webmention>; rel="webmention"')
|
response.headers.add('Link', '</webmention>; rel="webmention"')
|
||||||
return response
|
return response
|
||||||
|
|
||||||
|
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
|
||||||
|
from starpunk.websub import add_discovery_headers, template_links
|
||||||
|
app.after_request(add_discovery_headers)
|
||||||
|
app.add_template_global(template_links, 'websub_links')
|
||||||
|
|
||||||
# Register centralized error handlers
|
# Register centralized error handlers
|
||||||
from starpunk.errors import register_error_handlers
|
from starpunk.errors import register_error_handlers
|
||||||
|
|
||||||
@@ -322,12 +327,8 @@ def create_app(config=None):
|
|||||||
memory_monitor.start()
|
memory_monitor.start()
|
||||||
app.memory_monitor = memory_monitor
|
app.memory_monitor = memory_monitor
|
||||||
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
|
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
|
||||||
|
# No teardown_appcontext stop: that ran after every request and stopped
|
||||||
# Register cleanup handler
|
# the monitor straight away (v2.10.0). It is a daemon thread.
|
||||||
@app.teardown_appcontext
|
|
||||||
def cleanup_memory_monitor(error=None):
|
|
||||||
if hasattr(app, 'memory_monitor') and app.memory_monitor.is_alive():
|
|
||||||
app.memory_monitor.stop()
|
|
||||||
|
|
||||||
# Start webmention worker thread (v1.6.0 Phase 2)
|
# Start webmention worker thread (v1.6.0 Phase 2)
|
||||||
# Per background-worker.md: Skip in test mode
|
# Per background-worker.md: Skip in test mode
|
||||||
@@ -350,6 +351,12 @@ def create_app(config=None):
|
|||||||
f"batch_size={webmention_worker.batch_size})"
|
f"batch_size={webmention_worker.batch_size})"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Start WebSub worker (v2.10.0)
|
||||||
|
from starpunk.websub_worker import start_worker
|
||||||
|
app.websub_worker = start_worker(app)
|
||||||
|
if app.websub_worker:
|
||||||
|
app.logger.info("WebSub worker started")
|
||||||
|
|
||||||
# Start syndication worker when a target is configured (v2.1.0)
|
# Start syndication worker when a target is configured (v2.1.0)
|
||||||
from starpunk.syndication import SyndicationWorker, targets_configured
|
from starpunk.syndication import SyndicationWorker, targets_configured
|
||||||
if targets_configured(app.config) and not app.config.get("TESTING", False):
|
if targets_configured(app.config) and not app.config.get("TESTING", False):
|
||||||
@@ -463,5 +470,5 @@ def create_app(config=None):
|
|||||||
|
|
||||||
# Package version (Semantic Versioning 2.0.0)
|
# Package version (Semantic Versioning 2.0.0)
|
||||||
# See docs/standards/versioning-strategy.md for details
|
# See docs/standards/versioning-strategy.md for details
|
||||||
__version__ = "2.9.0"
|
__version__ = "2.10.0"
|
||||||
__version_info__ = (2, 9, 0)
|
__version_info__ = (2, 10, 0)
|
||||||
@@ -111,6 +111,13 @@ def load_config(app, config_override=None):
|
|||||||
|
|
||||||
# Webmention configuration (v1.6.0)
|
# Webmention configuration (v1.6.0)
|
||||||
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
|
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
|
||||||
|
|
||||||
|
# WebSub (v2.10.0, ADR-071): built-in hub unless WEBSUB_HUB_URL names another
|
||||||
|
app.config["WEBSUB_ENABLED"] = os.getenv("WEBSUB_ENABLED", "true").lower() == "true"
|
||||||
|
app.config["WEBSUB_HUB_URL"] = os.getenv("WEBSUB_HUB_URL", "").strip()
|
||||||
|
app.config["WEBSUB_MAX_SUBSCRIPTIONS"] = int(os.getenv("WEBSUB_MAX_SUBSCRIPTIONS", "1000"))
|
||||||
|
app.config["WEBSUB_MAX_PER_HOST"] = int(os.getenv("WEBSUB_MAX_PER_HOST", "25"))
|
||||||
|
app.config["WEBSUB_POLL_INTERVAL"] = int(os.getenv("WEBSUB_POLL_INTERVAL", "5"))
|
||||||
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
|
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
|
||||||
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
|
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
|
||||||
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
|
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
|
||||||
|
|||||||
@@ -93,7 +93,7 @@ def generate_rss(
|
|||||||
fg.language("en")
|
fg.language("en")
|
||||||
|
|
||||||
# Add self-link for feed discovery (Atom namespace)
|
# Add self-link for feed discovery (Atom namespace)
|
||||||
fg.link(href=f"{site_url}/feed.xml", rel="self", type="application/rss+xml")
|
fg.link(href=f"{site_url}/feed.rss", rel="self", type="application/rss+xml")
|
||||||
|
|
||||||
# Set last build date to now
|
# Set last build date to now
|
||||||
fg.lastBuildDate(datetime.now(timezone.utc))
|
fg.lastBuildDate(datetime.now(timezone.utc))
|
||||||
@@ -246,7 +246,7 @@ def generate_rss_streaming(
|
|||||||
yield f" <description>{_escape_xml(site_description or site_name)}</description>\n"
|
yield f" <description>{_escape_xml(site_description or site_name)}</description>\n"
|
||||||
yield " <language>en</language>\n"
|
yield " <language>en</language>\n"
|
||||||
yield f" <lastBuildDate>{last_build}</lastBuildDate>\n"
|
yield f" <lastBuildDate>{last_build}</lastBuildDate>\n"
|
||||||
yield f' <atom:link href="{_escape_xml(site_url)}/feed.xml" rel="self" type="application/rss+xml"/>\n'
|
yield f' <atom:link href="{_escape_xml(site_url)}/feed.rss" rel="self" type="application/rss+xml"/>\n'
|
||||||
|
|
||||||
# Yield items (newest first)
|
# Yield items (newest first)
|
||||||
# Notes from database are already in DESC order (newest first)
|
# Notes from database are already in DESC order (newest first)
|
||||||
|
|||||||
@@ -761,6 +761,9 @@ def handle_update(data: dict, token_info: dict):
|
|||||||
from starpunk.media import attach_media_to_note
|
from starpunk.media import attach_media_to_note
|
||||||
attach_media_to_note(note.id, media_ids, captions)
|
attach_media_to_note(note.id, media_ids, captions)
|
||||||
note_changed(note.id)
|
note_changed(note.id)
|
||||||
|
if not changes: # update_note already told WebSub otherwise
|
||||||
|
from starpunk import websub
|
||||||
|
websub.note_changed(note, was_public=note.published)
|
||||||
except InvalidNoteDataError as e:
|
except InvalidNoteDataError as e:
|
||||||
raise MicropubValidationError(str(e))
|
raise MicropubValidationError(str(e))
|
||||||
except MicropubError:
|
except MicropubError:
|
||||||
|
|||||||
+33
-1
@@ -374,9 +374,28 @@ def create_note(
|
|||||||
# Reply context fetch failure should not prevent note creation
|
# Reply context fetch failure should not prevent note creation
|
||||||
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
|
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
|
||||||
|
|
||||||
|
_websub_changed(note)
|
||||||
return note
|
return note
|
||||||
|
|
||||||
|
|
||||||
|
def _tag_names(note_id: int) -> list[str]:
|
||||||
|
"""A note's tag names, for WebSub tag topics (best effort)"""
|
||||||
|
try:
|
||||||
|
from starpunk.tags import get_note_tags
|
||||||
|
return [t["name"] for t in get_note_tags(note_id)]
|
||||||
|
except Exception:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _websub_changed(note, was_public: bool = False, old_tags=()) -> None:
|
||||||
|
"""Clear cached feeds and queue WebSub deliveries (v2.10.0, best effort)"""
|
||||||
|
try:
|
||||||
|
from starpunk import websub
|
||||||
|
websub.note_changed(note, was_public=was_public, old_tags=old_tags)
|
||||||
|
except Exception as e:
|
||||||
|
current_app.logger.warning(f"Failed to queue WebSub update for note {note.slug}: {e}")
|
||||||
|
|
||||||
|
|
||||||
def get_note(
|
def get_note(
|
||||||
slug: Optional[str] = None, id: Optional[int] = None, load_content: bool = True
|
slug: Optional[str] = None, id: Optional[int] = None, load_content: bool = True
|
||||||
) -> Optional[Note]:
|
) -> Optional[Note]:
|
||||||
@@ -705,6 +724,7 @@ def update_note(
|
|||||||
if existing_note is None:
|
if existing_note is None:
|
||||||
identifier = slug if slug is not None else id
|
identifier = slug if slug is not None else id
|
||||||
raise NoteNotFoundError(identifier)
|
raise NoteNotFoundError(identifier)
|
||||||
|
old_tag_names = _tag_names(existing_note.id) # for WebSub tag topics (v2.10.0)
|
||||||
|
|
||||||
if empty_content and not (existing_note.reacts_to or has_media):
|
if empty_content and not (existing_note.reacts_to or has_media):
|
||||||
raise InvalidNoteDataError(
|
raise InvalidNoteDataError(
|
||||||
@@ -896,6 +916,8 @@ def update_note(
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
current_app.logger.warning(f"Failed to queue syndication update: {e}")
|
current_app.logger.warning(f"Failed to queue syndication update: {e}")
|
||||||
|
|
||||||
|
_websub_changed(updated_note, was_public=existing_note.published, old_tags=old_tag_names)
|
||||||
|
|
||||||
# 10. RETURN UPDATED NOTE
|
# 10. RETURN UPDATED NOTE
|
||||||
return updated_note
|
return updated_note
|
||||||
|
|
||||||
@@ -995,6 +1017,10 @@ def delete_note(
|
|||||||
# For idempotency, don't raise error - just return
|
# For idempotency, don't raise error - just return
|
||||||
return
|
return
|
||||||
|
|
||||||
|
# What feeds showed before, for WebSub (v2.10.0)
|
||||||
|
was_public = bool(existing_note.published) and not getattr(existing_note, "deleted_at", None)
|
||||||
|
deleted_tags = _tag_names(existing_note.id)
|
||||||
|
|
||||||
# 4. SETUP
|
# 4. SETUP
|
||||||
data_dir = Path(current_app.config["DATA_PATH"])
|
data_dir = Path(current_app.config["DATA_PATH"])
|
||||||
note_path = data_dir / existing_note.file_path
|
note_path = data_dir / existing_note.file_path
|
||||||
@@ -1081,6 +1107,10 @@ def delete_note(
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
|
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
|
||||||
|
|
||||||
|
# Feeds and pages no longer show it (v2.10.0)
|
||||||
|
object.__setattr__(existing_note, "published", False)
|
||||||
|
_websub_changed(existing_note, was_public=was_public, old_tags=deleted_tags)
|
||||||
|
|
||||||
# 6. RETURN (no value)
|
# 6. RETURN (no value)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@@ -1137,4 +1167,6 @@ def undelete_note(slug: str) -> Note:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
|
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
|
||||||
|
|
||||||
return get_note(slug=slug)
|
restored = get_note(slug=slug)
|
||||||
|
_websub_changed(restored)
|
||||||
|
return restored
|
||||||
@@ -7,7 +7,7 @@ admin, auth, and (conditionally) dev auth routes.
|
|||||||
|
|
||||||
from flask import Flask
|
from flask import Flask
|
||||||
|
|
||||||
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention
|
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention, websub
|
||||||
|
|
||||||
|
|
||||||
def register_routes(app: Flask) -> None:
|
def register_routes(app: Flask) -> None:
|
||||||
@@ -44,6 +44,9 @@ def register_routes(app: Flask) -> None:
|
|||||||
if app.config.get("WEBMENTION_ENABLED", True):
|
if app.config.get("WEBMENTION_ENABLED", True):
|
||||||
app.register_blueprint(webmention.bp)
|
app.register_blueprint(webmention.bp)
|
||||||
|
|
||||||
|
# WebSub hub (v2.10.0); refuses requests when disabled or using another hub
|
||||||
|
app.register_blueprint(websub.bp)
|
||||||
|
|
||||||
# Conditionally register dev auth routes
|
# Conditionally register dev auth routes
|
||||||
if app.config.get("DEV_MODE"):
|
if app.config.get("DEV_MODE"):
|
||||||
app.logger.warning(
|
app.logger.warning(
|
||||||
|
|||||||
@@ -72,12 +72,23 @@ def dashboard():
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
# WebSub subscribers (v2.10.0)
|
||||||
|
websub_stats = None
|
||||||
|
if current_app.config.get('WEBSUB_ENABLED', True):
|
||||||
|
try:
|
||||||
|
from starpunk import websub
|
||||||
|
from starpunk.database import get_db
|
||||||
|
websub_stats = websub.stats(get_db(current_app))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
return render_template(
|
return render_template(
|
||||||
"admin/dashboard.html",
|
"admin/dashboard.html",
|
||||||
notes=notes,
|
notes=notes,
|
||||||
page=page,
|
page=page,
|
||||||
total_pages=total_pages,
|
total_pages=total_pages,
|
||||||
webmention_pending_count=webmention_pending_count,
|
webmention_pending_count=webmention_pending_count,
|
||||||
|
websub_stats=websub_stats,
|
||||||
user_me=g.me
|
user_me=g.me
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ from datetime import datetime, timedelta
|
|||||||
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
|
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
|
||||||
|
|
||||||
from starpunk.notes import list_notes, get_note
|
from starpunk.notes import list_notes, get_note
|
||||||
|
from starpunk.websub import embed_hub
|
||||||
from starpunk.feed import generate_feed_streaming # Legacy RSS
|
from starpunk.feed import generate_feed_streaming # Legacy RSS
|
||||||
from starpunk.feeds import (
|
from starpunk.feeds import (
|
||||||
generate_rss,
|
generate_rss,
|
||||||
@@ -55,8 +56,14 @@ def _get_cached_notes():
|
|||||||
cache_duration = timedelta(seconds=cache_seconds)
|
cache_duration = timedelta(seconds=cache_seconds)
|
||||||
now = datetime.utcnow()
|
now = datetime.utcnow()
|
||||||
|
|
||||||
|
# The cached list is stale once any public note changed, in any process
|
||||||
|
# (v2.10.0): every change bumps content_version in the database
|
||||||
|
from starpunk.database import get_db
|
||||||
|
from starpunk.websub import get_content_version
|
||||||
|
version = (str(current_app.config["DATABASE_PATH"]), get_content_version(get_db(current_app)))
|
||||||
|
|
||||||
# Check if note list cache is valid
|
# Check if note list cache is valid
|
||||||
if _feed_cache["notes"] and _feed_cache["timestamp"]:
|
if _feed_cache["notes"] and _feed_cache["timestamp"] and _feed_cache.get("version") == version:
|
||||||
cache_age = now - _feed_cache["timestamp"]
|
cache_age = now - _feed_cache["timestamp"]
|
||||||
if cache_age < cache_duration:
|
if cache_age < cache_duration:
|
||||||
# Use cached note list
|
# Use cached note list
|
||||||
@@ -82,6 +89,7 @@ def _get_cached_notes():
|
|||||||
|
|
||||||
_feed_cache["notes"] = notes
|
_feed_cache["notes"] = notes
|
||||||
_feed_cache["timestamp"] = now
|
_feed_cache["timestamp"] = now
|
||||||
|
_feed_cache["version"] = version
|
||||||
|
|
||||||
return notes
|
return notes
|
||||||
|
|
||||||
@@ -122,6 +130,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
|||||||
notes=notes,
|
notes=notes,
|
||||||
limit=max_items,
|
limit=max_items,
|
||||||
)
|
)
|
||||||
|
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
|
||||||
|
|
||||||
# Create response with proper Content-Type including charset
|
# Create response with proper Content-Type including charset
|
||||||
mime_type = get_mime_type(format_name)
|
mime_type = get_mime_type(format_name)
|
||||||
@@ -170,6 +179,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
|||||||
notes=notes,
|
notes=notes,
|
||||||
limit=max_items,
|
limit=max_items,
|
||||||
)
|
)
|
||||||
|
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
|
||||||
|
|
||||||
# Store in cache and get ETag
|
# Store in cache and get ETag
|
||||||
etag = feed_cache.set(format_name, content, notes_checksum)
|
etag = feed_cache.set(format_name, content, notes_checksum)
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
"""
|
||||||
|
WebSub hub endpoint (v2.10.0, ADR-071)
|
||||||
|
|
||||||
|
POST /websub with hub.mode subscribe/unsubscribe, hub.topic, hub.callback,
|
||||||
|
optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
|
||||||
|
(spec 5.1); errors are 4xx with a plain-text reason.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from flask import Blueprint, current_app, request
|
||||||
|
|
||||||
|
from starpunk.database import get_db
|
||||||
|
from starpunk.rate_limiter import RateLimiter
|
||||||
|
from starpunk import websub
|
||||||
|
|
||||||
|
bp = Blueprint("websub", __name__)
|
||||||
|
|
||||||
|
# Subscription requests per client IP per minute
|
||||||
|
_limiter = RateLimiter(limit=30, window_seconds=60)
|
||||||
|
|
||||||
|
|
||||||
|
def _text(message: str, status: int):
|
||||||
|
return message + "\n", status, {"Content-Type": "text/plain; charset=utf-8"}
|
||||||
|
|
||||||
|
|
||||||
|
@bp.route("/websub", methods=["POST"])
|
||||||
|
def hub():
|
||||||
|
if not websub.enabled():
|
||||||
|
return _text("WebSub is not enabled on this site", 404)
|
||||||
|
|
||||||
|
from starpunk.routes.webmention import get_client_ip
|
||||||
|
if not _limiter.is_allowed(get_client_ip()):
|
||||||
|
return _text("Too many requests", 429)
|
||||||
|
|
||||||
|
if "application/x-www-form-urlencoded" not in request.headers.get("Content-Type", ""):
|
||||||
|
return _text("Send an application/x-www-form-urlencoded body", 400)
|
||||||
|
|
||||||
|
try:
|
||||||
|
websub.accept_subscription_request(request.form, get_db(current_app))
|
||||||
|
except websub.SubscriptionError as e:
|
||||||
|
return _text(str(e), e.status)
|
||||||
|
|
||||||
|
return _text("Accepted; the subscription will be verified", 202)
|
||||||
|
|
||||||
|
|
||||||
|
@bp.route("/websub", methods=["GET"])
|
||||||
|
def hub_info():
|
||||||
|
return _text(
|
||||||
|
"WebSub hub for this site's own feeds. Subscribe with a POST "
|
||||||
|
"(https://www.w3.org/TR/websub/).", 200)
|
||||||
+31
-14
@@ -18,6 +18,7 @@ import ipaddress
|
|||||||
import socket
|
import socket
|
||||||
import time
|
import time
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
|
from typing import Optional
|
||||||
from urllib.parse import urljoin, urlparse
|
from urllib.parse import urljoin, urlparse
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
@@ -157,20 +158,6 @@ def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def safe_post(url: str, data: dict, headers: dict, timeout: float = TIMEOUT) -> httpx.Response:
|
|
||||||
"""
|
|
||||||
POST form data to a checked, pinned address. Redirects are not followed.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
SSRFBlockedError, ValueError: URL is not allowed
|
|
||||||
httpx.HTTPError: Network errors
|
|
||||||
"""
|
|
||||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
|
||||||
response = client.send(_pinned_request(client, 'POST', url, headers, data=data))
|
|
||||||
response.read()
|
|
||||||
return response
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class FetchResult:
|
class FetchResult:
|
||||||
"""A fetched response, after redirects"""
|
"""A fetched response, after redirects"""
|
||||||
@@ -188,6 +175,36 @@ class FetchResult:
|
|||||||
return self.content.decode('utf-8', errors='replace')
|
return self.content.decode('utf-8', errors='replace')
|
||||||
|
|
||||||
|
|
||||||
|
def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = None,
|
||||||
|
timeout: float = TIMEOUT, content: Optional[bytes] = None,
|
||||||
|
max_bytes: int = 64 * 1024) -> FetchResult:
|
||||||
|
"""
|
||||||
|
POST form data (or a raw body) to a checked, pinned address.
|
||||||
|
|
||||||
|
Redirects are not followed. Only the first max_bytes of the response are
|
||||||
|
read (callers need the status and maybe an error message).
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
SSRFBlockedError, ValueError: URL is not allowed
|
||||||
|
httpx.HTTPError: Network errors
|
||||||
|
"""
|
||||||
|
kwargs = {'content': content} if content is not None else {'data': data}
|
||||||
|
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
||||||
|
request = _pinned_request(client, 'POST', url, headers or {}, **kwargs)
|
||||||
|
response = client.send(request, stream=True)
|
||||||
|
try:
|
||||||
|
body = b''
|
||||||
|
for chunk in response.iter_bytes(chunk_size=8192):
|
||||||
|
body += chunk
|
||||||
|
if len(body) >= max_bytes:
|
||||||
|
body = body[:max_bytes]
|
||||||
|
break
|
||||||
|
return FetchResult(url=url, status_code=response.status_code, headers=response.headers,
|
||||||
|
content=body, encoding=response.charset_encoding or 'utf-8')
|
||||||
|
finally:
|
||||||
|
response.close()
|
||||||
|
|
||||||
|
|
||||||
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||||
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
|
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -0,0 +1,340 @@
|
|||||||
|
"""
|
||||||
|
WebSub publishing and the built-in hub (v2.10.0, ADR-071)
|
||||||
|
|
||||||
|
Publisher: topic responses advertise the hub and their own URL; a change to a
|
||||||
|
public note clears cached feeds and queues delivery of every affected topic.
|
||||||
|
|
||||||
|
Hub: subscriptions to this site's own topics only. Requests are accepted
|
||||||
|
here (202) and verified, delivered and retried by the WebSub worker.
|
||||||
|
|
||||||
|
References:
|
||||||
|
- W3C WebSub: https://www.w3.org/TR/websub/
|
||||||
|
- ADR-071: WebSub publishing with a built-in hub
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import secrets
|
||||||
|
from typing import Iterable, Optional
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from flask import current_app, request
|
||||||
|
|
||||||
|
|
||||||
|
# Topic endpoints and their fixed paths (tag pages are added per tag)
|
||||||
|
FEED_TOPICS = {
|
||||||
|
"public.feed_atom": "/feed.atom",
|
||||||
|
"public.feed_rss": "/feed.rss",
|
||||||
|
"public.feed_xml_legacy": "/feed.rss", # legacy alias of the RSS topic
|
||||||
|
"public.feed_json": "/feed.json",
|
||||||
|
}
|
||||||
|
PAGE_TOPICS = {"public.index": "/", "public.likes": "/likes"}
|
||||||
|
NEGOTIATED = { # /feed: the topic of whichever format was served
|
||||||
|
"application/atom+xml": "/feed.atom",
|
||||||
|
"application/rss+xml": "/feed.rss",
|
||||||
|
"application/feed+json": "/feed.json",
|
||||||
|
"application/json": "/feed.json",
|
||||||
|
}
|
||||||
|
|
||||||
|
LEASE_DEFAULT = 10 * 24 * 3600
|
||||||
|
LEASE_MIN = 24 * 3600
|
||||||
|
LEASE_MAX = 30 * 24 * 3600
|
||||||
|
MAX_SECRET_BYTES = 200
|
||||||
|
|
||||||
|
|
||||||
|
class SubscriptionError(Exception):
|
||||||
|
"""A subscription request the hub refuses (4xx with a plain-text reason)"""
|
||||||
|
|
||||||
|
def __init__(self, message: str, status: int = 400):
|
||||||
|
super().__init__(message)
|
||||||
|
self.status = status
|
||||||
|
|
||||||
|
|
||||||
|
# --- Configuration -----------------------------------------------------------
|
||||||
|
|
||||||
|
def enabled() -> bool:
|
||||||
|
return bool(current_app.config.get("WEBSUB_ENABLED", True))
|
||||||
|
|
||||||
|
|
||||||
|
def external_hub() -> Optional[str]:
|
||||||
|
"""URL of an external hub, if one is configured instead of the built-in one"""
|
||||||
|
return current_app.config.get("WEBSUB_HUB_URL") or None
|
||||||
|
|
||||||
|
|
||||||
|
def hub_url() -> str:
|
||||||
|
# Built from SITE_URL so the worker can use it outside a request
|
||||||
|
return external_hub() or site_url() + "/websub"
|
||||||
|
|
||||||
|
|
||||||
|
def site_url() -> str:
|
||||||
|
return current_app.config["SITE_URL"].rstrip("/")
|
||||||
|
|
||||||
|
|
||||||
|
# --- Topics ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def topic_url(path: str) -> str:
|
||||||
|
return site_url() + path
|
||||||
|
|
||||||
|
|
||||||
|
def tag_topic(tag_name: str) -> str:
|
||||||
|
from urllib.parse import quote
|
||||||
|
return f"{site_url()}/tag/{quote(tag_name, safe='')}"
|
||||||
|
|
||||||
|
|
||||||
|
def topic_for_request(response=None) -> Optional[str]:
|
||||||
|
"""The topic URL the current request serves, or None if it isn't a topic"""
|
||||||
|
endpoint = request.endpoint
|
||||||
|
if endpoint in PAGE_TOPICS:
|
||||||
|
return topic_url(PAGE_TOPICS[endpoint])
|
||||||
|
if endpoint in FEED_TOPICS:
|
||||||
|
return topic_url(FEED_TOPICS[endpoint])
|
||||||
|
if endpoint == "public.feed" and response is not None:
|
||||||
|
mime = (response.mimetype or "").lower()
|
||||||
|
return topic_url(NEGOTIATED[mime]) if mime in NEGOTIATED else None
|
||||||
|
if endpoint == "public.tag":
|
||||||
|
from starpunk.tags import normalize_tag
|
||||||
|
tag = (request.view_args or {}).get("tag", "")
|
||||||
|
return tag_topic(normalize_tag(tag)[0]) if tag else None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_topic(url: str) -> Optional[str]:
|
||||||
|
"""
|
||||||
|
A subscription's hub.topic as one of this site's topics, or None
|
||||||
|
|
||||||
|
Tag topics must name an existing tag.
|
||||||
|
"""
|
||||||
|
if not url:
|
||||||
|
return None
|
||||||
|
site = urlparse(site_url())
|
||||||
|
parsed = urlparse(url.strip())
|
||||||
|
if (parsed.scheme, parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
|
||||||
|
return None
|
||||||
|
path = parsed.path or "/"
|
||||||
|
if parsed.query or parsed.fragment:
|
||||||
|
return None
|
||||||
|
if path in set(PAGE_TOPICS.values()) | set(FEED_TOPICS.values()):
|
||||||
|
return topic_url(path)
|
||||||
|
if path.startswith("/tag/"):
|
||||||
|
from starpunk.tags import get_tag_by_name, normalize_tag
|
||||||
|
name = normalize_tag(path[len("/tag/"):])[0]
|
||||||
|
if name and get_tag_by_name(name):
|
||||||
|
return tag_topic(name)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def link_header_values(topic: str) -> list[str]:
|
||||||
|
return [f'<{hub_url()}>; rel="hub"', f'<{topic}>; rel="self"']
|
||||||
|
|
||||||
|
|
||||||
|
def add_discovery_headers(response):
|
||||||
|
"""after_request: Link rel=hub and rel=self on topic responses (spec 4)"""
|
||||||
|
if not enabled() or response.status_code not in (200, 304):
|
||||||
|
return response
|
||||||
|
topic = topic_for_request(response)
|
||||||
|
if topic:
|
||||||
|
for value in link_header_values(topic):
|
||||||
|
response.headers.add("Link", value)
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def template_links() -> Optional[dict]:
|
||||||
|
"""For base.html: {'hub', 'self'} on topic pages"""
|
||||||
|
if not enabled() or not request:
|
||||||
|
return None
|
||||||
|
topic = topic_for_request()
|
||||||
|
return {"hub": hub_url(), "self": topic} if topic else None
|
||||||
|
|
||||||
|
|
||||||
|
def embed_hub(format_name: str, content: str) -> str:
|
||||||
|
"""Add the hub link inside a generated feed, next to its self link"""
|
||||||
|
if not enabled():
|
||||||
|
return content
|
||||||
|
hub = hub_url()
|
||||||
|
if format_name == "atom":
|
||||||
|
marker = '<link rel="self"'
|
||||||
|
return content.replace(marker, f'<link rel="hub" href="{_xml(hub)}"/>\n {marker}', 1)
|
||||||
|
if format_name == "rss":
|
||||||
|
marker = "<atom:link "
|
||||||
|
return content.replace(marker, f'<atom:link href="{_xml(hub)}" rel="hub"/>\n {marker}', 1)
|
||||||
|
if format_name == "json":
|
||||||
|
feed = json.loads(content)
|
||||||
|
feed["hubs"] = [{"type": "WebSub", "url": hub}]
|
||||||
|
return json.dumps(feed, ensure_ascii=False, indent=2)
|
||||||
|
return content
|
||||||
|
|
||||||
|
|
||||||
|
def _xml(value: str) -> str:
|
||||||
|
return (value.replace("&", "&").replace("<", "<")
|
||||||
|
.replace(">", ">").replace('"', """))
|
||||||
|
|
||||||
|
|
||||||
|
# --- Changes -------------------------------------------------------------------
|
||||||
|
|
||||||
|
def get_content_version(db) -> str:
|
||||||
|
row = db.execute("SELECT value FROM site_state WHERE key = 'content_version'").fetchone()
|
||||||
|
return row[0] if row else "0"
|
||||||
|
|
||||||
|
|
||||||
|
def bump_content_version(db) -> None:
|
||||||
|
db.execute(
|
||||||
|
"INSERT INTO site_state (key, value) VALUES ('content_version', '1') "
|
||||||
|
"ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def note_changed(note, was_public: bool = False, old_tags: Iterable[str] = ()) -> None:
|
||||||
|
"""
|
||||||
|
A note was created, edited, deleted, restored or (un)published
|
||||||
|
|
||||||
|
Clears cached feeds in every process and queues delivery of the topics
|
||||||
|
that show it. Changes that are never public (a draft edited) do nothing.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
note: The note (after the change; for a delete, before it)
|
||||||
|
was_public: Whether it was public before the change
|
||||||
|
old_tags: Its tag names before the change
|
||||||
|
"""
|
||||||
|
from starpunk.database import get_db
|
||||||
|
|
||||||
|
is_public = bool(note.published) and not getattr(note, "deleted_at", None)
|
||||||
|
if not (is_public or was_public):
|
||||||
|
return
|
||||||
|
|
||||||
|
db = get_db(current_app)
|
||||||
|
bump_content_version(db)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
if not enabled():
|
||||||
|
return
|
||||||
|
|
||||||
|
from starpunk.tags import get_note_tags
|
||||||
|
try:
|
||||||
|
tags = {t["name"] for t in get_note_tags(note.id)}
|
||||||
|
except Exception:
|
||||||
|
tags = set()
|
||||||
|
tags |= set(old_tags)
|
||||||
|
|
||||||
|
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
|
||||||
|
if getattr(note, "like_of", None):
|
||||||
|
topics.append(topic_url("/likes"))
|
||||||
|
topics += [tag_topic(name) for name in sorted(tags)]
|
||||||
|
queue_topics(db, topics)
|
||||||
|
|
||||||
|
|
||||||
|
def queue_topics(db, topics: Iterable[str]) -> int:
|
||||||
|
"""Queue delivery of topics: to their subscribers, or a ping to an external hub"""
|
||||||
|
topics = list(dict.fromkeys(topics))
|
||||||
|
if not topics:
|
||||||
|
return 0
|
||||||
|
if external_hub():
|
||||||
|
for topic in topics:
|
||||||
|
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NULL AND topic = ?", (topic,))
|
||||||
|
db.execute("INSERT INTO websub_deliveries (topic) VALUES (?)", (topic,))
|
||||||
|
db.commit()
|
||||||
|
return len(topics)
|
||||||
|
|
||||||
|
placeholders = ",".join("?" * len(topics))
|
||||||
|
rows = db.execute(
|
||||||
|
f"SELECT id, topic FROM websub_subscriptions WHERE topic IN ({placeholders}) "
|
||||||
|
"AND verified_at IS NOT NULL AND expires_at > datetime('now')",
|
||||||
|
topics,
|
||||||
|
).fetchall()
|
||||||
|
for sub_id, topic in rows:
|
||||||
|
# The newest content replaces anything still waiting (ADR-071 6)
|
||||||
|
db.execute("DELETE FROM websub_deliveries WHERE subscription_id = ?", (sub_id,))
|
||||||
|
db.execute("INSERT INTO websub_deliveries (subscription_id, topic) VALUES (?, ?)", (sub_id, topic))
|
||||||
|
db.commit()
|
||||||
|
return len(rows)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Hub: subscription requests (spec 5.1) -------------------------------------
|
||||||
|
|
||||||
|
def accept_subscription_request(form, db) -> None:
|
||||||
|
"""
|
||||||
|
Validate a subscription request and queue it for verification
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
SubscriptionError: with the status and plain-text reason to return
|
||||||
|
"""
|
||||||
|
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
|
||||||
|
|
||||||
|
if external_hub():
|
||||||
|
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
|
||||||
|
|
||||||
|
mode = (form.get("hub.mode") or "").strip()
|
||||||
|
if mode not in ("subscribe", "unsubscribe"):
|
||||||
|
raise SubscriptionError("hub.mode must be subscribe or unsubscribe")
|
||||||
|
|
||||||
|
callback = (form.get("hub.callback") or "").strip()
|
||||||
|
parsed = urlparse(callback)
|
||||||
|
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||||
|
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
|
||||||
|
try:
|
||||||
|
validate_url_for_fetch(callback)
|
||||||
|
except (SSRFBlockedError, ValueError) as e:
|
||||||
|
raise SubscriptionError(f"hub.callback is not reachable: {e}")
|
||||||
|
|
||||||
|
topic = canonical_topic(form.get("hub.topic") or "")
|
||||||
|
if not topic:
|
||||||
|
raise SubscriptionError("hub.topic is not a topic of this site; use the rel=self URL")
|
||||||
|
|
||||||
|
secret = form.get("hub.secret")
|
||||||
|
if mode == "unsubscribe":
|
||||||
|
secret = None # MUST be ignored for unsubscriptions
|
||||||
|
if secret is not None and len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
|
||||||
|
raise SubscriptionError("hub.secret must be less than 200 bytes")
|
||||||
|
|
||||||
|
lease = LEASE_DEFAULT
|
||||||
|
if form.get("hub.lease_seconds"):
|
||||||
|
try:
|
||||||
|
lease = int(form["hub.lease_seconds"])
|
||||||
|
except ValueError:
|
||||||
|
raise SubscriptionError("hub.lease_seconds must be a number")
|
||||||
|
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
|
||||||
|
|
||||||
|
if mode == "subscribe":
|
||||||
|
existing = db.execute(
|
||||||
|
"SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?", (topic, callback)
|
||||||
|
).fetchone()
|
||||||
|
if not existing:
|
||||||
|
total = db.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0]
|
||||||
|
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
|
||||||
|
raise SubscriptionError("This hub has reached its subscription limit", 429)
|
||||||
|
host = parsed.hostname.lower()
|
||||||
|
per_host = sum(
|
||||||
|
1 for (cb,) in db.execute("SELECT callback FROM websub_subscriptions")
|
||||||
|
if (urlparse(cb).hostname or "").lower() == host
|
||||||
|
)
|
||||||
|
if per_host >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
|
||||||
|
raise SubscriptionError(f"Too many subscriptions for {host}", 429)
|
||||||
|
|
||||||
|
# The latest request for a topic and callback replaces any pending one
|
||||||
|
db.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO websub_intents (topic, callback, mode, secret, lease_seconds)
|
||||||
|
VALUES (?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||||
|
mode = excluded.mode, secret = excluded.secret,
|
||||||
|
lease_seconds = excluded.lease_seconds, attempts = 0,
|
||||||
|
next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
|
||||||
|
""",
|
||||||
|
(topic, callback, mode, secret, lease),
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def stats(db) -> dict:
|
||||||
|
"""Active subscribers per topic and deliveries being retried (admin)"""
|
||||||
|
by_topic = db.execute(
|
||||||
|
"SELECT topic, COUNT(*) FROM websub_subscriptions WHERE expires_at > datetime('now') "
|
||||||
|
"GROUP BY topic ORDER BY COUNT(*) DESC, topic"
|
||||||
|
).fetchall()
|
||||||
|
retrying = db.execute(
|
||||||
|
"SELECT COUNT(*) FROM websub_deliveries WHERE attempts > 0"
|
||||||
|
).fetchone()[0]
|
||||||
|
return {"by_topic": [(t, n) for t, n in by_topic], "retrying": retrying,
|
||||||
|
"external": external_hub()}
|
||||||
|
|
||||||
|
|
||||||
|
def new_challenge() -> str:
|
||||||
|
return secrets.token_urlsafe(32)
|
||||||
@@ -0,0 +1,287 @@
|
|||||||
|
"""
|
||||||
|
WebSub worker: verification of intent and content distribution (v2.10.0)
|
||||||
|
|
||||||
|
Runs like the Webmention worker: a daemon thread in every gunicorn process,
|
||||||
|
but only the process holding the file lock works the queues.
|
||||||
|
|
||||||
|
- Intents (spec 5.3): GET the callback with a challenge; 2xx echoing it
|
||||||
|
verifies. Anything else (including redirects) leaves the subscription as
|
||||||
|
it was. Network errors and 5xx are retried a few times.
|
||||||
|
- Deliveries (spec 6, 7): render the topic in-app and POST it with
|
||||||
|
Link rel=hub/self and, if there is a secret, X-Hub-Signature (sha256).
|
||||||
|
- External hub: a delivery without a subscription is a publish ping.
|
||||||
|
|
||||||
|
References:
|
||||||
|
- https://www.w3.org/TR/websub/
|
||||||
|
- ADR-071
|
||||||
|
"""
|
||||||
|
|
||||||
|
import fcntl
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
import logging
|
||||||
|
import threading
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DELIVERY_BACKOFF = [60, 300, 1800, 7200, 21600] # then give up on this delivery
|
||||||
|
INTENT_BACKOFF = [60, 300, 900]
|
||||||
|
DROP_AFTER = timedelta(days=3) # failing this long without a success: unsubscribe
|
||||||
|
PRUNE_EVERY = timedelta(hours=1)
|
||||||
|
BATCH = 10
|
||||||
|
|
||||||
|
|
||||||
|
def callback_with_params(callback: str, params: dict) -> str:
|
||||||
|
"""Append parameters to the callback, keeping its query string (spec 5.3)"""
|
||||||
|
parts = urlsplit(callback)
|
||||||
|
query = parts.query + ("&" if parts.query else "") + urlencode(params)
|
||||||
|
return urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))
|
||||||
|
|
||||||
|
|
||||||
|
def signature(secret: str, body: bytes) -> str:
|
||||||
|
return "sha256=" + hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _now() -> str:
|
||||||
|
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
|
def _later(seconds: int) -> str:
|
||||||
|
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
|
class WebSubWorker(threading.Thread):
|
||||||
|
"""Daemon thread that verifies subscriptions and delivers content"""
|
||||||
|
|
||||||
|
def __init__(self, app, poll_interval: int = 5):
|
||||||
|
super().__init__(daemon=True, name="websub-worker")
|
||||||
|
self.app = app
|
||||||
|
self.poll_interval = poll_interval
|
||||||
|
self._stop_event = threading.Event()
|
||||||
|
self._lock_file = None
|
||||||
|
self._last_prune = None
|
||||||
|
|
||||||
|
# --- thread plumbing ----------------------------------------------------
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
logger.info("WebSub worker started")
|
||||||
|
while not self._stop_event.is_set():
|
||||||
|
if not self._acquire_lock():
|
||||||
|
self._stop_event.wait(self.poll_interval)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
with self.app.app_context():
|
||||||
|
done = self.process_once()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"WebSub worker error: {e}", exc_info=True)
|
||||||
|
done = 0
|
||||||
|
if not done:
|
||||||
|
self._stop_event.wait(self.poll_interval)
|
||||||
|
self._release_lock()
|
||||||
|
|
||||||
|
def stop(self):
|
||||||
|
self._stop_event.set()
|
||||||
|
|
||||||
|
def _acquire_lock(self) -> bool:
|
||||||
|
if self._lock_file:
|
||||||
|
return True
|
||||||
|
path = Path(self.app.config["DATABASE_PATH"]).parent / "websub-worker.lock"
|
||||||
|
lock_file = open(path, "w")
|
||||||
|
try:
|
||||||
|
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
except OSError:
|
||||||
|
lock_file.close()
|
||||||
|
return False
|
||||||
|
self._lock_file = lock_file
|
||||||
|
logger.info("WebSub worker holds the queue lock")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def _release_lock(self):
|
||||||
|
if self._lock_file:
|
||||||
|
self._lock_file.close()
|
||||||
|
self._lock_file = None
|
||||||
|
|
||||||
|
# --- one pass (also called directly by tests) ---------------------------
|
||||||
|
|
||||||
|
def process_once(self) -> int:
|
||||||
|
from starpunk.database import get_db
|
||||||
|
|
||||||
|
db = get_db(self.app)
|
||||||
|
done = 0
|
||||||
|
for row in db.execute(
|
||||||
|
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY id LIMIT ?",
|
||||||
|
(_now(), BATCH),
|
||||||
|
).fetchall():
|
||||||
|
self.verify(db, row)
|
||||||
|
done += 1
|
||||||
|
for row in db.execute(
|
||||||
|
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY id LIMIT ?",
|
||||||
|
(_now(), BATCH),
|
||||||
|
).fetchall():
|
||||||
|
self.deliver(db, row)
|
||||||
|
done += 1
|
||||||
|
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
|
||||||
|
self.prune(db)
|
||||||
|
return done
|
||||||
|
|
||||||
|
# --- verification of intent (spec 5.3) ----------------------------------
|
||||||
|
|
||||||
|
def verify(self, db, intent) -> None:
|
||||||
|
from starpunk.websub import new_challenge
|
||||||
|
from starpunk.webmention_ssrf import SSRFBlockedError, safe_get, user_agent
|
||||||
|
|
||||||
|
challenge = new_challenge()
|
||||||
|
params = {"hub.mode": intent["mode"], "hub.topic": intent["topic"], "hub.challenge": challenge}
|
||||||
|
if intent["mode"] == "subscribe":
|
||||||
|
params["hub.lease_seconds"] = intent["lease_seconds"]
|
||||||
|
url = callback_with_params(intent["callback"], params)
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
|
||||||
|
max_redirects=0, max_bytes=64 * 1024)
|
||||||
|
verified = 200 <= result.status_code < 300 and result.text.strip() == challenge
|
||||||
|
retry = result.status_code >= 500
|
||||||
|
error = None if verified else f"HTTP {result.status_code}" + (
|
||||||
|
"" if result.status_code >= 300 else ": challenge not echoed")
|
||||||
|
except (SSRFBlockedError, ValueError) as e:
|
||||||
|
verified, retry, error = False, False, f"Blocked: {e}"
|
||||||
|
except httpx.TooManyRedirects:
|
||||||
|
verified, retry, error = False, False, "Redirected (not allowed)"
|
||||||
|
except httpx.HTTPError as e:
|
||||||
|
verified, retry, error = False, True, str(e)[:200]
|
||||||
|
|
||||||
|
if verified:
|
||||||
|
if intent["mode"] == "subscribe":
|
||||||
|
db.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO websub_subscriptions
|
||||||
|
(topic, callback, secret, lease_seconds, expires_at, verified_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||||
|
secret = excluded.secret, lease_seconds = excluded.lease_seconds,
|
||||||
|
expires_at = excluded.expires_at, verified_at = excluded.verified_at,
|
||||||
|
last_error = NULL
|
||||||
|
""",
|
||||||
|
(intent["topic"], intent["callback"], intent["secret"], intent["lease_seconds"],
|
||||||
|
_later(intent["lease_seconds"]), _now()),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
db.execute("DELETE FROM websub_subscriptions WHERE topic = ? AND callback = ?",
|
||||||
|
(intent["topic"], intent["callback"]))
|
||||||
|
db.execute("DELETE FROM websub_intents WHERE id = ?", (intent["id"],))
|
||||||
|
logger.info(f"WebSub {intent['mode']} verified: {intent['callback']} -> {intent['topic']}")
|
||||||
|
elif retry and intent["attempts"] < len(INTENT_BACKOFF):
|
||||||
|
db.execute("UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? WHERE id = ?",
|
||||||
|
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"]))
|
||||||
|
else:
|
||||||
|
# Verification failed: the subscription state stays as it was
|
||||||
|
db.execute("DELETE FROM websub_intents WHERE id = ?", (intent["id"],))
|
||||||
|
logger.info(f"WebSub {intent['mode']} not verified for {intent['callback']}: {error}")
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
# --- content distribution (spec 6, 7) ------------------------------------
|
||||||
|
|
||||||
|
def render_topic(self, topic: str):
|
||||||
|
"""(status, content_type, body) of a topic, rendered in-app"""
|
||||||
|
path = urlparse(topic).path or "/"
|
||||||
|
response = self.app.test_client().get(path, headers={"Accept": "*/*"})
|
||||||
|
return response.status_code, response.headers.get("Content-Type", ""), response.get_data()
|
||||||
|
|
||||||
|
def deliver(self, db, delivery) -> None:
|
||||||
|
from starpunk.websub import hub_url
|
||||||
|
from starpunk.webmention_ssrf import SSRFBlockedError, safe_post, user_agent
|
||||||
|
|
||||||
|
site = self.app.config["SITE_URL"]
|
||||||
|
sub = None
|
||||||
|
try:
|
||||||
|
if delivery["subscription_id"] is None:
|
||||||
|
# External hub: ping it (hub.url is the de facto parameter)
|
||||||
|
result = safe_post(hub_url(), data={"hub.mode": "publish", "hub.url": delivery["topic"],
|
||||||
|
"hub.topic": delivery["topic"]},
|
||||||
|
headers={"User-Agent": user_agent(site)})
|
||||||
|
else:
|
||||||
|
sub = db.execute(
|
||||||
|
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
|
||||||
|
(delivery["subscription_id"], _now()),
|
||||||
|
).fetchone()
|
||||||
|
if sub is None: # unsubscribed or expired meanwhile
|
||||||
|
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||||
|
db.commit()
|
||||||
|
return
|
||||||
|
status, content_type, body = self.render_topic(sub["topic"])
|
||||||
|
if status != 200: # e.g. a tag page whose last note went
|
||||||
|
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||||
|
db.commit()
|
||||||
|
return
|
||||||
|
headers = {
|
||||||
|
"Content-Type": content_type,
|
||||||
|
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
|
||||||
|
"User-Agent": user_agent(site),
|
||||||
|
}
|
||||||
|
if sub["secret"]:
|
||||||
|
headers["X-Hub-Signature"] = signature(sub["secret"], body)
|
||||||
|
result = safe_post(sub["callback"], content=body, headers=headers)
|
||||||
|
status, error = result.status_code, f"HTTP {result.status_code}"
|
||||||
|
except (SSRFBlockedError, ValueError) as e:
|
||||||
|
status, error = None, f"Blocked: {e}"
|
||||||
|
except httpx.HTTPError as e:
|
||||||
|
status, error = None, str(e)[:200] or type(e).__name__
|
||||||
|
|
||||||
|
if status is not None and 200 <= status < 300:
|
||||||
|
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||||
|
if sub is not None:
|
||||||
|
db.execute("UPDATE websub_subscriptions SET last_success_at = ?, last_error = NULL WHERE id = ?",
|
||||||
|
(_now(), sub["id"]))
|
||||||
|
elif sub is not None and status == 410:
|
||||||
|
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||||
|
logger.info(f"WebSub subscriber gone (410), removed: {sub['callback']}")
|
||||||
|
elif delivery["attempts"] < len(DELIVERY_BACKOFF) and not error.startswith("Blocked"):
|
||||||
|
db.execute(
|
||||||
|
"UPDATE websub_deliveries SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE id = ?",
|
||||||
|
(_later(DELIVERY_BACKOFF[delivery["attempts"]]), error, delivery["id"]),
|
||||||
|
)
|
||||||
|
if sub is not None:
|
||||||
|
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||||
|
else:
|
||||||
|
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||||
|
logger.warning(f"WebSub delivery of {delivery['topic']} gave up: {error}")
|
||||||
|
if sub is not None:
|
||||||
|
self._drop_if_failing(db, sub, error)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
def _drop_if_failing(self, db, sub, error: str) -> None:
|
||||||
|
last_ok = sub["last_success_at"] or sub["verified_at"] or sub["created_at"]
|
||||||
|
try:
|
||||||
|
since = datetime.fromisoformat(str(last_ok))
|
||||||
|
except ValueError:
|
||||||
|
return
|
||||||
|
if datetime.utcnow() - since > DROP_AFTER:
|
||||||
|
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||||
|
logger.info(f"WebSub subscription failing for {DROP_AFTER.days} days, removed: {sub['callback']}")
|
||||||
|
else:
|
||||||
|
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||||
|
|
||||||
|
# --- housekeeping -------------------------------------------------------
|
||||||
|
|
||||||
|
def prune(self, db) -> int:
|
||||||
|
"""Remove subscriptions whose lease has run out (they weren't renewed)"""
|
||||||
|
cursor = db.execute("DELETE FROM websub_subscriptions WHERE expires_at <= ?", (_now(),))
|
||||||
|
db.commit()
|
||||||
|
self._last_prune = datetime.utcnow()
|
||||||
|
if cursor.rowcount:
|
||||||
|
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
|
||||||
|
return cursor.rowcount
|
||||||
|
|
||||||
|
|
||||||
|
def start_worker(app) -> Optional[WebSubWorker]:
|
||||||
|
if not app.config.get("WEBSUB_ENABLED", True) or app.config.get("TESTING", False):
|
||||||
|
return None
|
||||||
|
worker = WebSubWorker(app, poll_interval=app.config.get("WEBSUB_POLL_INTERVAL", 5))
|
||||||
|
worker.start()
|
||||||
|
return worker
|
||||||
@@ -20,6 +20,21 @@
|
|||||||
{% endif %}
|
{% endif %}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{# WebSub subscribers (v2.10.0) #}
|
||||||
|
{% if websub_stats %}
|
||||||
|
<p class="note-meta websub-stats">
|
||||||
|
{% if websub_stats.external %}
|
||||||
|
WebSub: using {{ websub_stats.external }}
|
||||||
|
{% elif websub_stats.by_topic %}
|
||||||
|
WebSub subscribers:
|
||||||
|
{% for topic, count in websub_stats.by_topic %}{{ topic|replace(config.SITE_URL.rstrip('/'), '') or '/' }} {{ count }}{% if not loop.last %}, {% endif %}{% endfor %}
|
||||||
|
{% if websub_stats.retrying %} · {{ websub_stats.retrying }} deliveries being retried{% endif %}
|
||||||
|
{% else %}
|
||||||
|
WebSub: no subscribers yet
|
||||||
|
{% endif %}
|
||||||
|
</p>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
{% if notes %}
|
{% if notes %}
|
||||||
<div class="table-scroll">
|
<div class="table-scroll">
|
||||||
<table class="note-table">
|
<table class="note-table">
|
||||||
|
|||||||
@@ -14,6 +14,13 @@
|
|||||||
<link rel="webmention" href="/webmention">
|
<link rel="webmention" href="/webmention">
|
||||||
{% endif %}
|
{% endif %}
|
||||||
|
|
||||||
|
{# WebSub hub and topic on feed pages (v2.10.0) #}
|
||||||
|
{% set websub = websub_links() %}
|
||||||
|
{% if websub %}
|
||||||
|
<link rel="hub" href="{{ websub.hub }}">
|
||||||
|
<link rel="self" href="{{ websub.self }}">
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
|
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
|
||||||
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
|
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
|
||||||
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
|
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
|
||||||
|
|||||||
+1
-1
@@ -230,7 +230,7 @@ class TestGenerateFeed:
|
|||||||
|
|
||||||
# Check for Atom namespace and self-link
|
# Check for Atom namespace and self-link
|
||||||
assert "atom" in feed_xml
|
assert "atom" in feed_xml
|
||||||
assert "feed.xml" in feed_xml
|
assert "feed.rss" in feed_xml # self link (v2.10.0: /feed.rss is the RSS topic)
|
||||||
assert 'rel="self"' in feed_xml
|
assert 'rel="self"' in feed_xml
|
||||||
|
|
||||||
def test_generate_feed_item_structure(self, app, sample_notes):
|
def test_generate_feed_item_structure(self, app, sample_notes):
|
||||||
|
|||||||
@@ -0,0 +1,315 @@
|
|||||||
|
"""
|
||||||
|
WebSub publisher and built-in hub (v2.10.0, ADR-071)
|
||||||
|
|
||||||
|
Section numbers refer to the W3C WebSub Recommendation. Subscribers are
|
||||||
|
simulated with an httpx MockTransport behind the SSRF-safe fetcher.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import mf2py
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from starpunk.websub_worker import WebSubWorker
|
||||||
|
|
||||||
|
SITE = "http://localhost:5000"
|
||||||
|
HUB = f"{SITE}/websub"
|
||||||
|
PUBLIC = [(2, 1, 6, "", ("93.184.216.34", 0))]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def web(monkeypatch):
|
||||||
|
"""Subscriber callbacks: handler(request) -> response, by host"""
|
||||||
|
hosts, seen = {}, []
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
host = request.headers["host"]
|
||||||
|
seen.append(request)
|
||||||
|
return hosts[host](request)
|
||||||
|
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
|
||||||
|
return hosts, seen
|
||||||
|
|
||||||
|
|
||||||
|
def echo(request):
|
||||||
|
"""A subscriber that confirms every verification and accepts content"""
|
||||||
|
if request.method == "GET":
|
||||||
|
return httpx.Response(200, text=parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0])
|
||||||
|
return httpx.Response(202)
|
||||||
|
|
||||||
|
|
||||||
|
def db(app):
|
||||||
|
conn = sqlite3.connect(app.config["DATABASE_PATH"])
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
return conn
|
||||||
|
|
||||||
|
|
||||||
|
def run_worker(app, times=1):
|
||||||
|
worker = WebSubWorker(app)
|
||||||
|
with app.app_context():
|
||||||
|
for _ in range(times):
|
||||||
|
worker.process_once()
|
||||||
|
return worker
|
||||||
|
|
||||||
|
|
||||||
|
def subscribe(client, topic, callback="https://sub.example/cb?id=1", **extra):
|
||||||
|
form = {"hub.mode": "subscribe", "hub.topic": topic, "hub.callback": callback, **extra}
|
||||||
|
return client.post("/websub", data=form)
|
||||||
|
|
||||||
|
|
||||||
|
def make_note(app, content="Hello", **kw):
|
||||||
|
from starpunk.notes import create_note
|
||||||
|
with app.test_request_context():
|
||||||
|
return create_note(content, published=kw.pop("published", True), **kw)
|
||||||
|
|
||||||
|
|
||||||
|
def links(response):
|
||||||
|
return response.headers.getlist("Link")
|
||||||
|
|
||||||
|
|
||||||
|
# 4 Discovery -------------------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("path,self_url", [
|
||||||
|
("/", f"{SITE}/"),
|
||||||
|
("/feed.atom", f"{SITE}/feed.atom"),
|
||||||
|
("/feed.rss", f"{SITE}/feed.rss"),
|
||||||
|
("/feed.xml", f"{SITE}/feed.rss"),
|
||||||
|
("/feed.json", f"{SITE}/feed.json"),
|
||||||
|
("/likes", f"{SITE}/likes"),
|
||||||
|
])
|
||||||
|
def test_topics_advertise_hub_and_self(client, path, self_url):
|
||||||
|
response = client.get(path)
|
||||||
|
values = links(response)
|
||||||
|
assert f'<{HUB}>; rel="hub"' in values
|
||||||
|
assert [v for v in values if 'rel="self"' in v] == [f'<{self_url}>; rel="self"']
|
||||||
|
|
||||||
|
|
||||||
|
def test_negotiated_feed_self_is_the_format_served(client):
|
||||||
|
response = client.get("/feed", headers={"Accept": "application/atom+xml"})
|
||||||
|
assert f'<{SITE}/feed.atom>; rel="self"' in links(response)
|
||||||
|
|
||||||
|
|
||||||
|
def test_tag_page_is_a_topic(app, client):
|
||||||
|
make_note(app, tags=["Python"])
|
||||||
|
assert f'<{SITE}/tag/python>; rel="self"' in links(client.get("/tag/Python"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_note_page_is_not_a_topic(app, client):
|
||||||
|
note = make_note(app)
|
||||||
|
assert not [v for v in links(client.get(f"/note/{note.slug}")) if 'rel="hub"' in v]
|
||||||
|
|
||||||
|
|
||||||
|
def test_embedded_links(client):
|
||||||
|
rels = mf2py.parse(doc=client.get("/").data.decode(), url=SITE + "/")["rels"]
|
||||||
|
assert rels["hub"] == [HUB] and rels["self"] == [f"{SITE}/"]
|
||||||
|
assert f'<link rel="hub" href="{HUB}"/>' in client.get("/feed.atom").data.decode()
|
||||||
|
assert f'<atom:link href="{HUB}" rel="hub"/>' in client.get("/feed.rss").data.decode()
|
||||||
|
assert json.loads(client.get("/feed.json").data)["hubs"] == [{"type": "WebSub", "url": HUB}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_external_hub(app, client, web):
|
||||||
|
app.config["WEBSUB_HUB_URL"] = "https://hub.example/"
|
||||||
|
hosts, seen = web
|
||||||
|
hosts["hub.example"] = lambda r: httpx.Response(204)
|
||||||
|
assert '<https://hub.example/>; rel="hub"' in links(client.get("/"))
|
||||||
|
assert subscribe(client, f"{SITE}/").status_code == 400 # our hub is off
|
||||||
|
|
||||||
|
make_note(app)
|
||||||
|
run_worker(app)
|
||||||
|
pings = {parse_qs(r.content.decode())["hub.url"][0] for r in seen}
|
||||||
|
assert pings == {f"{SITE}/", f"{SITE}/feed.atom", f"{SITE}/feed.rss", f"{SITE}/feed.json"}
|
||||||
|
|
||||||
|
|
||||||
|
# 5.1 Subscription requests ----------------------------------------------------------
|
||||||
|
|
||||||
|
def test_subscription_request_is_accepted(app, client, web):
|
||||||
|
response = subscribe(client, f"{SITE}/feed.atom", **{"hub.secret": "s3cret", "extra": "ignored"})
|
||||||
|
assert response.status_code == 202
|
||||||
|
row = db(app).execute("SELECT * FROM websub_intents").fetchone()
|
||||||
|
assert (row["mode"], row["topic"], row["secret"]) == ("subscribe", f"{SITE}/feed.atom", "s3cret")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("form,reason", [
|
||||||
|
({"hub.mode": "watch"}, "hub.mode"),
|
||||||
|
({"hub.topic": "https://elsewhere.example/feed"}, "hub.topic"),
|
||||||
|
({"hub.topic": f"{SITE}/note/x"}, "hub.topic"),
|
||||||
|
({"hub.topic": f"{SITE}/tag/missing"}, "hub.topic"),
|
||||||
|
({"hub.callback": "ftp://sub.example/cb"}, "hub.callback"),
|
||||||
|
({"hub.secret": "x" * 200}, "hub.secret"),
|
||||||
|
])
|
||||||
|
def test_bad_requests(client, web, form, reason):
|
||||||
|
base = {"hub.mode": "subscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb"}
|
||||||
|
response = client.post("/websub", data={**base, **form})
|
||||||
|
assert response.status_code == 400 and reason in response.data.decode()
|
||||||
|
|
||||||
|
|
||||||
|
def test_private_callback_refused(client, monkeypatch):
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||||
|
lambda *a, **k: [(2, 1, 6, "", ("10.0.0.2", 0))])
|
||||||
|
assert subscribe(client, f"{SITE}/").status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_per_host_limit(app, client, web):
|
||||||
|
app.config["WEBSUB_MAX_PER_HOST"] = 1
|
||||||
|
hosts, _ = web
|
||||||
|
hosts["sub.example"] = echo
|
||||||
|
subscribe(client, f"{SITE}/")
|
||||||
|
run_worker(app)
|
||||||
|
assert subscribe(client, f"{SITE}/feed.atom").status_code == 429
|
||||||
|
|
||||||
|
|
||||||
|
# 5.3 Verification of intent --------------------------------------------------------
|
||||||
|
|
||||||
|
def test_verification_activates_subscription(app, client, web):
|
||||||
|
hosts, seen = web
|
||||||
|
hosts["sub.example"] = echo
|
||||||
|
subscribe(client, f"{SITE}/", **{"hub.lease_seconds": "86400"})
|
||||||
|
run_worker(app)
|
||||||
|
|
||||||
|
query = parse_qs(urlparse(str(seen[0].url)).query)
|
||||||
|
assert query["id"] == ["1"] # the callback's own query string is kept
|
||||||
|
assert query["hub.mode"] == ["subscribe"] and query["hub.topic"] == [f"{SITE}/"]
|
||||||
|
assert query["hub.lease_seconds"] == ["86400"]
|
||||||
|
sub = db(app).execute("SELECT * FROM websub_subscriptions").fetchone()
|
||||||
|
assert sub["topic"] == f"{SITE}/" and sub["verified_at"]
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("response", [
|
||||||
|
httpx.Response(200, text="wrong"),
|
||||||
|
httpx.Response(404),
|
||||||
|
httpx.Response(302, headers={"Location": "https://sub.example/other"}),
|
||||||
|
])
|
||||||
|
def test_failed_verification_changes_nothing(app, client, web, response):
|
||||||
|
hosts, _ = web
|
||||||
|
hosts["sub.example"] = lambda r: response
|
||||||
|
subscribe(client, f"{SITE}/")
|
||||||
|
run_worker(app)
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_unsubscribe(app, client, web):
|
||||||
|
hosts, _ = web
|
||||||
|
hosts["sub.example"] = echo
|
||||||
|
subscribe(client, f"{SITE}/")
|
||||||
|
run_worker(app)
|
||||||
|
client.post("/websub", data={"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/",
|
||||||
|
"hub.callback": "https://sub.example/cb?id=1"})
|
||||||
|
run_worker(app)
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_resubscribe_changes_secret_only_once_verified(app, client, web):
|
||||||
|
hosts, _ = web
|
||||||
|
hosts["sub.example"] = echo
|
||||||
|
subscribe(client, f"{SITE}/", **{"hub.secret": "one"})
|
||||||
|
run_worker(app)
|
||||||
|
hosts["sub.example"] = lambda r: httpx.Response(404)
|
||||||
|
subscribe(client, f"{SITE}/", **{"hub.secret": "two"})
|
||||||
|
run_worker(app)
|
||||||
|
assert db(app).execute("SELECT secret FROM websub_subscriptions").fetchone()[0] == "one"
|
||||||
|
|
||||||
|
|
||||||
|
# 6, 7 Content distribution ---------------------------------------------------------
|
||||||
|
|
||||||
|
def _subscribed(app, client, web, topic=f"{SITE}/feed.atom", secret=None):
|
||||||
|
hosts, seen = web
|
||||||
|
hosts["sub.example"] = echo
|
||||||
|
extra = {"hub.secret": secret} if secret else {}
|
||||||
|
subscribe(client, topic, **extra)
|
||||||
|
run_worker(app)
|
||||||
|
seen.clear()
|
||||||
|
return hosts, seen
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_note_is_delivered_signed(app, client, web):
|
||||||
|
hosts, seen = _subscribed(app, client, web, secret="s3cret")
|
||||||
|
make_note(app, "Fresh post")
|
||||||
|
run_worker(app)
|
||||||
|
|
||||||
|
(post,) = [r for r in seen if r.method == "POST"]
|
||||||
|
assert post.headers["content-type"].startswith("application/atom+xml")
|
||||||
|
assert b"Fresh post" in post.content
|
||||||
|
assert f'<{HUB}>; rel="hub"' in post.headers["link"] and f'<{SITE}/feed.atom>; rel="self"' in post.headers["link"]
|
||||||
|
expected = "sha256=" + hmac.new(b"s3cret", post.content, hashlib.sha256).hexdigest()
|
||||||
|
assert post.headers["x-hub-signature"] == expected
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_signature_without_secret(app, client, web):
|
||||||
|
hosts, seen = _subscribed(app, client, web)
|
||||||
|
make_note(app)
|
||||||
|
run_worker(app)
|
||||||
|
assert "x-hub-signature" not in [r for r in seen if r.method == "POST"][0].headers
|
||||||
|
|
||||||
|
|
||||||
|
def test_drafts_are_not_delivered(app, client, web):
|
||||||
|
_subscribed(app, client, web)
|
||||||
|
make_note(app, published=False)
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_tag_topics_include_removed_tags(app, client, web):
|
||||||
|
from starpunk.notes import update_note
|
||||||
|
note = make_note(app, tags=["old"])
|
||||||
|
_subscribed(app, client, web, topic=f"{SITE}/tag/old")
|
||||||
|
with app.test_request_context():
|
||||||
|
update_note(id=note.id, content="Hello", tags=["new"])
|
||||||
|
assert db(app).execute("SELECT topic FROM websub_deliveries").fetchone()[0] == f"{SITE}/tag/old"
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_is_delivered(app, client, web):
|
||||||
|
from starpunk.notes import delete_note
|
||||||
|
note = make_note(app, "Going away")
|
||||||
|
hosts, seen = _subscribed(app, client, web)
|
||||||
|
with app.test_request_context():
|
||||||
|
delete_note(id=note.id, soft=False)
|
||||||
|
run_worker(app)
|
||||||
|
(post,) = [r for r in seen if r.method == "POST"]
|
||||||
|
assert b"Going away" not in post.content
|
||||||
|
|
||||||
|
|
||||||
|
def test_410_removes_subscription(app, client, web):
|
||||||
|
hosts, _ = _subscribed(app, client, web)
|
||||||
|
hosts["sub.example"] = lambda r: httpx.Response(410)
|
||||||
|
make_note(app)
|
||||||
|
run_worker(app)
|
||||||
|
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_failures_are_retried(app, client, web):
|
||||||
|
hosts, _ = _subscribed(app, client, web)
|
||||||
|
hosts["sub.example"] = lambda r: httpx.Response(500)
|
||||||
|
make_note(app)
|
||||||
|
run_worker(app)
|
||||||
|
row = db(app).execute("SELECT attempts, next_attempt_at, last_error FROM websub_deliveries").fetchone()
|
||||||
|
assert row["attempts"] == 1 and row["last_error"] == "HTTP 500"
|
||||||
|
|
||||||
|
|
||||||
|
def test_expired_subscriptions_get_nothing(app, client, web):
|
||||||
|
_subscribed(app, client, web)
|
||||||
|
conn = db(app)
|
||||||
|
conn.execute("UPDATE websub_subscriptions SET expires_at = datetime('now', '-1 minute')")
|
||||||
|
conn.commit()
|
||||||
|
make_note(app)
|
||||||
|
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||||
|
run_worker(app) # prunes
|
||||||
|
assert conn.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
# Feeds are fresh right after a change (ADR-071 3) ------------------------------------
|
||||||
|
|
||||||
|
def test_feed_shows_new_note_immediately(app, client):
|
||||||
|
from starpunk.routes import public
|
||||||
|
make_note(app, "First")
|
||||||
|
assert b"First" in client.get("/feed.atom").data
|
||||||
|
make_note(app, "Second") # cached note list is stale now
|
||||||
|
assert b"Second" in client.get("/feed.atom").data
|
||||||
|
assert public._feed_cache["notes"] is not None
|
||||||
Reference in new issue
Block a user