550 lines
27 KiB
Markdown
550 lines
27 KiB
Markdown
# IndieWeb and W3C Protocols
|
|
|
|
How StarPunk implements each protocol it speaks, as of v2.14.0, with its
|
|
compliance status and the gaps that remain. Each section links to the
|
|
decision records and review documents behind it.
|
|
|
|
| Protocol | Role | Status | Main code |
|
|
|---|---|---|---|
|
|
| [Micropub](#micropub) | Server (posts and media) | Reviewed against the Recommendation in v2.9.0; deliberate deviations listed | `starpunk/micropub.py`, `starpunk/routes/micropub.py` |
|
|
| [IndieAuth](#indieauth) | Authorization server for Micropub clients | Authorization code + PKCE, metadata, revocation | `starpunk/indieauth.py`, `starpunk/routes/indieauth.py` |
|
|
| [Webmention](#webmention) | Sender and receiver | Every MUST and SHOULD checked in v2.7.0 and v2.7.2; known limits listed | `starpunk/webmention_*.py`, `starpunk/routes/webmention.py` |
|
|
| [Microformats2](#microformats2) | Publisher and consumer | Reviewed in v2.8.0 | `templates/`, `starpunk/mf2_utils.py`, `starpunk/author_discovery.py` |
|
|
| [WebSub](#websub) | Publisher and built-in hub (own topics only) | Adversarial review in v2.10.1 | `starpunk/websub.py`, `starpunk/websub_worker.py`, `starpunk/routes/websub.py` |
|
|
| [Feeds](#feeds) | RSS 2.0, Atom 1.0, JSON Feed 1.1, OPML 2.0 | Stable | `starpunk/feeds/`, `starpunk/routes/public.py` |
|
|
| Backfeed | Responses to crossposts read from the silo APIs and stored as webmentions (v2.13.0); see [syndication-architecture.md](syndication-architecture.md#backfeed-v2130-adr-073) | ADR-073 | `starpunk/backfeed.py` |
|
|
| [Web actions](#moderation-and-display) | `<indie-action>` on the like, repost and reply counts (v2.13.2) | Markup only | `templates/partials/responses.html` |
|
|
|
|
Crossposting to other networks (POSSE) is covered in
|
|
[syndication-architecture.md](syndication-architecture.md).
|
|
|
|
## Shared infrastructure: the SSRF-safe fetcher
|
|
|
|
Webmention (fetching sources, discovering endpoints, sending), WebSub
|
|
(verification and delivery), Micropub (downloading remote photos), IndieAuth
|
|
(client pages), reply contexts, crosspost link cards (v2.11.0) and response
|
|
avatars (v2.13.0; except those on StarPunk's own GoToSocial) all make
|
|
requests to URLs chosen by someone else. They use
|
|
`starpunk/webmention_ssrf.py`:
|
|
|
|
- `validate_url_for_fetch()`: http(s) only. Every address from
|
|
`getaddrinfo` must be public. Private, loopback, link-local, multicast,
|
|
reserved, unspecified, CGNAT (`100.64.0.0/10`), `0.0.0.0/8` and
|
|
IPv4-mapped forms are blocked. It returns the checked address, preferring
|
|
IPv4. `DEV_MODE` allows localhost.
|
|
- **DNS pinning** (v2.7.2): each request connects to the checked IP, with
|
|
the original host in the `Host` header and as the TLS SNI name. The
|
|
certificate is still verified against the real hostname, so a rebinding
|
|
DNS server can't send the second lookup elsewhere.
|
|
- `safe_get()` follows up to 10 redirects by hand and checks each hop. The
|
|
response is capped at 1 MB by default.
|
|
- `safe_post()` doesn't follow redirects.
|
|
- **Hard deadlines** (v2.10.1): the whole exchange, not each read, is
|
|
limited to 10 s. When the deadline passes, the client is closed, so a
|
|
server trickling bytes can't hold a worker.
|
|
- DNS resolution failures raise `ResolutionError` (a `ValueError`), which
|
|
callers can treat as temporary.
|
|
- User-Agent: `StarPunk/<version> Webmention (+<SITE_URL>)`.
|
|
|
|
Workers that drain queues (Webmention, WebSub) run as one daemon thread per
|
|
gunicorn process. Each takes a non-blocking `flock` on a lock file in the
|
|
data directory, so only one process works the queue at a time and another
|
|
takes over if it dies.
|
|
|
|
---
|
|
|
|
## Micropub
|
|
|
|
Spec: https://www.w3.org/TR/micropub/. Decisions:
|
|
[ADR-028](../decisions/ADR-028-micropub-implementation.md),
|
|
[ADR-029](../decisions/ADR-029-micropub-indieauth-integration.md),
|
|
[ADR-057](../decisions/ADR-057-media-attachment-model.md) (photos),
|
|
[ADR-067](../decisions/ADR-067-micropub-update-delete.md) (update/delete),
|
|
[ADR-068](../decisions/ADR-068-likes-and-reposts.md) (likes/reposts).
|
|
Review: [v2.9.0 Micropub review](../design/v2.9.0/2026-11-03-micropub-review.md).
|
|
|
|
### Endpoints and discovery
|
|
|
|
| Endpoint | Purpose |
|
|
|---|---|
|
|
| `GET /micropub?q=...` | Queries |
|
|
| `POST /micropub` | Create, update, delete, undelete |
|
|
| `POST /micropub/media` (and legacy `POST /media`) | Media endpoint |
|
|
|
|
Every page has `<link rel="micropub">` plus the IndieAuth discovery links
|
|
(see [IndieAuth](#indieauth)). Authentication, token placement, scopes and
|
|
error codes are described in
|
|
[simplified-auth-architecture.md §3](simplified-auth-architecture.md#3-token-use-at-the-micropub-and-media-endpoints).
|
|
|
|
### Create
|
|
|
|
Accepts form-encoded, multipart (with photo files) and JSON bodies. Only
|
|
`h=entry` / `type: ["h-entry"]` is accepted; other types get 400. Success is
|
|
**201** with `Location` set to the permalink `/note/<slug>`.
|
|
|
|
| Property / command | Handling |
|
|
|---|---|
|
|
| `content` | String, or `{"html": ...}` / `{"text": ...}`. Text is preferred. HTML-only content is stored with `content_format = 'html'` (migration 015) and rendered through `render_markdown()` (Markdown, then nh3 allowlist sanitization) |
|
|
| `summary` | Used as the text when there is no `content` |
|
|
| `name` | Stored as a `# name` heading on the first line (StarPunk has no title column) |
|
|
| `category` | Tags (strings only; person tags as h-card objects are ignored) |
|
|
| `in-reply-to`, `like-of`, `repost-of`, `bookmark-of` | URL or h-cite. At most one of like/repost/bookmark. Content is optional for these |
|
|
| `photo` | URLs, `{"value", "alt"}` objects, or uploaded files; at most 4 (ADR-057), duplicates collapsed. Own `/media/` URLs are looked up; other URLs are downloaded through `safe_get` (20 MB cap). Any photo that can't be resolved means 400 and nothing is created |
|
|
| `published` | ISO 8601; converted to naive UTC; invalid gets 400 |
|
|
| `post-status` | `published` (default) or `draft`; others get 400 |
|
|
| `visibility` | `public`; `private` and `unlisted` are saved as drafts (StarPunk has no private posts); others get 400 |
|
|
| `video`, `audio` | 400 (not supported) |
|
|
| `location`, `checkin`, others | Ignored (allowed by 3.3.5) |
|
|
| `mp-slug` | Custom slug |
|
|
| `mp-syndicate-to` | Crosspost targets. **Absent means none** (v2.9.0). See [syndication-architecture.md](syndication-architecture.md#choosing-targets) |
|
|
|
|
`mp-*` keys are read as commands from the top level or from inside JSON
|
|
`properties`, and are never stored. JSON property values must be arrays; a
|
|
bare scalar is wrapped on create. If anything fails after the note is
|
|
created, the note is removed and 500 is returned, so the client never
|
|
believes a half-made post failed.
|
|
|
|
Post types advertised in `q=config`: note, article, photo, reply, like,
|
|
repost, bookmark.
|
|
|
|
### Update, delete, undelete
|
|
|
|
- **Update** (JSON only, `update` scope) targets a `/note/<slug>` URL on
|
|
this site.
|
|
- `replace` works on `content`, `name`, `category`, `in-reply-to`,
|
|
`post-status` and `photo`.
|
|
- `add` works on `category` and `photo` only.
|
|
- `delete` takes a list of property names, or values to remove for
|
|
`category` and `photo`.
|
|
- Values must be arrays. An update without any operation gets 400.
|
|
Unsupported properties get 400.
|
|
- Photos are resolved before anything changes. Success is **204**.
|
|
- Edits follow crossposts and are re-announced to Webmention targets and
|
|
WebSub subscribers.
|
|
- **Delete** (`delete` scope) is soft: the note is moved to the trash, its
|
|
page returns 410, and `undelete` restores it. Both return **204**.
|
|
|
|
### Queries
|
|
|
|
| Query | Scope | Result |
|
|
|---|---|---|
|
|
| `q=config` | any token | `media-endpoint`, `syndicate-to`, `q` list, `post-types` |
|
|
| `q=syndicate-to` | any token | `{"syndicate-to": [{"uid", "name"}]}` |
|
|
| `q=source&url=...[&properties[]=...]` | create/update/delete/draft | One post as mf2 JSON. Content comes back as `{"html": ...}` if it was authored as HTML. Includes `syndication` URLs and `post-status` |
|
|
| `q=source` without `url` | same | `{"items": [...]}`, drafts included, `limit` (default 10, max 50) and `offset` |
|
|
| `q=category[&filter=...]` | same | Tags on notes that aren't deleted (drafts count), most used first (v2.12.1) |
|
|
|
|
Unknown or missing `q` gets 400.
|
|
|
|
### Media endpoint
|
|
|
|
`POST /micropub/media`, multipart with one part named `file`, `media` or
|
|
`create` scope. It returns **201** with `Location: <SITE_URL>/media/<path>`.
|
|
Images are JPEG, PNG, GIF, WebP or HEIC, up to 50 MB in. They are optimized
|
|
and get responsive variants (`starpunk/media.py`,
|
|
[ADR-058](../decisions/ADR-058-image-optimization-strategy.md)). Validation
|
|
failures get 400.
|
|
|
|
### Known deviations and gaps
|
|
|
|
- **403 for `insufficient_scope`** (RFC 6750) where the Micropub error table
|
|
says 401. Kept on purpose.
|
|
- `add` is only supported for `category` and `photo`. `published`,
|
|
`syndication` and `location` can't be updated. `mp-syndicate-to` is only
|
|
read on create.
|
|
- `location` / `checkin` are ignored, and person tags are dropped.
|
|
- `video` and `audio` are refused.
|
|
- The `access_token` query parameter is still accepted on GET.
|
|
- Only notes (`/note/<slug>`) can be the target of update/delete/source.
|
|
|
|
---
|
|
|
|
## IndieAuth
|
|
|
|
StarPunk is its own IndieAuth authorization server for Micropub clients,
|
|
issuing tokens for the single identity `ADMIN_ME`. The admin approves
|
|
clients on a consent screen protected by OIDC login. Supported: the
|
|
authorization code flow with mandatory PKCE S256, `iss` in responses (RFC
|
|
9207), the metadata endpoint, a profile-only flow, a token endpoint, RFC
|
|
7009 revocation, and legacy token verification (`GET /auth/token`). Client
|
|
`redirect_uri`s must be same-origin or published by the client (JSON
|
|
metadata or `rel=redirect_uri`).
|
|
|
|
Full description, scopes and gaps:
|
|
[simplified-auth-architecture.md](simplified-auth-architecture.md).
|
|
Decision: [ADR-064](../decisions/ADR-064-oidc-admin-login-and-builtin-indieauth-server.md).
|
|
|
|
Known gaps: `scopes_supported` doesn't list `draft`. There are no
|
|
introspection, userinfo or refresh tokens.
|
|
|
|
---
|
|
|
|
## Webmention
|
|
|
|
Spec: https://www.w3.org/TR/webmention/ (Recommendation, 12 January 2017).
|
|
Decision: [ADR-063](../decisions/ADR-063-webmention-implementation.md).
|
|
Reviews: [v2.6.9 fixes](../design/v2.6.9/2026-11-03-webmention-processing-fixes.md),
|
|
[v2.7.0 compliance](../design/v2.7.0/2026-11-03-webmention-spec-compliance.md),
|
|
[v2.7.2 remaining SHOULDs](../design/v2.7.2/2026-11-03-webmention-shoulds.md).
|
|
|
|
### Endpoint discovery (advertising)
|
|
|
|
With `WEBMENTION_ENABLED` (default true), every HTML response carries
|
|
`Link: </webmention>; rel="webmention"`, and `base.html` has
|
|
`<link rel="webmention" href="/webmention">`. When it is off, the receiving
|
|
route isn't registered at all.
|
|
|
|
### Receiving
|
|
|
|
`POST /webmention` (`routes/webmention.py`):
|
|
|
|
1. Rate limits: 60 requests per minute per client IP and 10 per minute per
|
|
source domain (`WEBMENTION_RATE_LIMIT_*`). Over the limit gets 429 with
|
|
`Retry-After`. Counts are shared by all server processes (`rate_limits`
|
|
table, v2.11.0). The client IP
|
|
comes from `X-Forwarded-For` only when `TRUSTED_PROXY=true`.
|
|
2. Form-encoded (JSON also accepted). `source` and `target` are required,
|
|
http(s), at most 2048 characters, and different from each other.
|
|
Otherwise 400.
|
|
3. The target's host and port must be this site's. The fragment is stripped
|
|
before storing. The target must be a **published note**: anything else,
|
|
including the home page, gets 400 `invalid_target`.
|
|
4. The row is upserted into `webmentions` (unique on source + target) as
|
|
`pending`, and **202** is returned. A repeat from the same source resets
|
|
it to `pending` (an update, 3.2.4).
|
|
|
|
Verification is asynchronous (`WebmentionWorker`, poll
|
|
`WEBMENTION_POLL_INTERVAL` 10 s, batch of 5):
|
|
|
|
- **Fetch** the source with `safe_get` (redirects followed and checked,
|
|
1 MB, 10 s), with an `Accept` header preferring HTML.
|
|
- **Exact-match verification** (3.2.2), per media type:
|
|
- HTML: every link-bearing attribute (`lxml iterlinks`: `a`/`link` href,
|
|
`img`/`video`/`audio` src, ...), resolved against the final URL.
|
|
- JSON: any string value.
|
|
- Other `text/*`: substring match.
|
|
- Any other type: no match.
|
|
- Fragments are ignored on both sides.
|
|
- **Parse mf2** (HTML only) with mf2py:
|
|
- `mf2_utils.responding_entry()` picks the h-entry that responds to the
|
|
target.
|
|
- `response_type()` classifies it as reply, like, repost, bookmark or
|
|
mention.
|
|
- `entry_author()` follows the authorship algorithm without extra
|
|
fetches: author property, then the parent h-feed's author, then
|
|
`rel=author`, then the representative h-card.
|
|
- Also stored: a content snippet (e-content or p-summary, sanitized with
|
|
nh3, at most 500 characters), the source's `u-url` and `dt-published`
|
|
(migration 014).
|
|
- The author URL and photo are kept only if they are absolute http(s)
|
|
URLs (XSS, 4.1).
|
|
- **Outcomes**:
|
|
- Valid: `validation_status = 'valid'`. A first-time mention gets
|
|
`moderation_status = 'approved'` if `WEBMENTION_AUTO_APPROVE` (default
|
|
true), otherwise `pending`. An update keeps the admin's earlier
|
|
moderation decision, including `hidden` and `deleted`.
|
|
- Source returns 4xx (including 410), or no longer links: `invalid`,
|
|
which hides it (3.2.4 deletion).
|
|
- Network error, timeout or 5xx: a mention that was valid before stays
|
|
valid with its old data. A new one becomes `error`.
|
|
- A source that is one of the note's own syndicated copies
|
|
(`syndications.remote_url`) is `invalid` (v2.8.0).
|
|
- A source that backfeed has already stored for the note is `invalid`
|
|
("Already backfed from the silo", v2.13.0).
|
|
- **Re-send on display change** (3.1.4, v2.7.2): if what the note shows
|
|
for this webmention changed, the note's own outgoing webmentions are
|
|
re-queued. That covers a new response, a changed author or snippet, or a
|
|
change of shown/hidden. Unchanged re-validations send nothing, so two
|
|
sites doing the same don't ping-pong.
|
|
|
|
### Moderation and display
|
|
|
|
`/admin/webmentions` lists, filters (by status, including Deleted, and by
|
|
origin), and approves, hides or deletes responses, one at a time or in bulk;
|
|
it also has the Backfill button. Deleting keeps the row as a `deleted`
|
|
tombstone, so a re-sent webmention or backfeed doesn't bring it back.
|
|
Changes that alter what is shown also re-send the note's webmentions.
|
|
|
|
Note pages show approved, valid responses. Under the note, a line of counts
|
|
(♥ likes, 🔁 reposts, 💬 replies, ❝ quotes (v2.14.2), bookmarks) opens a card on hover or focus
|
|
listing who responded and where from, including anonymous silo counts. Each
|
|
like, repost and reply count holds an `<indie-action do with>` web action
|
|
whose fallback links to the note's crossposts (v2.13.2). The Responses
|
|
section below holds replies (`u-comment h-cite`), quotes and mentions;
|
|
backfed ones say "via <silo>". See [Microformats2](#microformats2).
|
|
|
|
### Notifications (v2.14.0, ADR-074)
|
|
|
|
With `NOTIFY_URLS` set, new valid responses (received and backfed) are sent
|
|
to the owner once through Apprise by a sweep in the webmention worker, which
|
|
sets `webmentions.notified_at` (migration 021). Likes, reposts and bookmarks
|
|
are grouped per note; which kinds are sent is set by `NOTIFY_EVENTS`
|
|
(default replies, mentions, quotes and problems). Items still unsent after
|
|
24 hours are dropped. Responses found by a backfill, or by a copy's first
|
|
check on a note older than a day, are never notified.
|
|
|
|
### Sending
|
|
|
|
Queued by `webmention_sender.queue_webmentions_for_note()` from `notes.py`.
|
|
Sending isn't gated by `WEBMENTION_ENABLED`; it runs whenever the worker
|
|
does (`WEBMENTION_WORKER_ENABLED`).
|
|
|
|
| Event | Targets |
|
|
|---|---|
|
|
| Publish | Reply / like / repost / bookmark target first, then `<a href>` links from the rendered HTML in document order (not inside `<pre>`/`<code>`), absolute http(s) only, de-duplicated, at most 10 |
|
|
| Edit (published) | The same, plus every target notified before and links the edit removed (not capped) (3.1.4) |
|
|
| Delete | Every previously notified target is re-queued (3.1.5); for a hard delete the queue rows are detached first so the cascade doesn't remove them |
|
|
| Undelete | Re-queued again |
|
|
| Displayed responses change | Re-queued (see above) |
|
|
|
|
Re-queuing resets the cached endpoint, so discovery runs again (3.1.4).
|
|
|
|
**Endpoint discovery** (3.1.2): GET the target with `safe_get` (redirects
|
|
checked). The first `Link` header with a `rel` token `webmention` wins. The
|
|
header is parsed per RFC 8288: several links per header, several headers,
|
|
rel token lists, case-insensitive. Otherwise the first `<link>` or `<a>`
|
|
with `rel~=webmention` and an `href`, in document order. Relative URLs
|
|
resolve against the final URL, an empty `href` means the target itself, and
|
|
the query string is kept.
|
|
|
|
**Send** (3.1.3): a form-encoded POST through `safe_post`, pinned and with
|
|
no redirects followed (a 301/302 would turn the POST into a GET).
|
|
|
|
| Result | Outcome |
|
|
|---|---|
|
|
| Any 2xx | `sent` |
|
|
| 429, 5xx, network error | Retried after 1 h, 6 h, 24 h; `failed` after the 4th attempt |
|
|
| Other 3xx/4xx, SSRF-blocked endpoint | `failed` |
|
|
| No endpoint | `no_endpoint` |
|
|
|
|
**Deleted posts**: `/note/<slug>` returns **410 Gone** with a tombstone page
|
|
for soft-deleted notes, and for hard-deleted notes that had sent
|
|
webmentions (3.1.5).
|
|
|
|
### Security (section 4)
|
|
|
|
- XSS: author URL and photo must be http(s). Snippets are sanitized with
|
|
nh3 (and re-sanitized after truncation). The mentions list uses
|
|
`striptags`.
|
|
- SSRF: all fetches and POSTs go through the pinned fetcher described
|
|
above, so the endpoint URL is checked before POSTing.
|
|
- DoS: rate limits on receiving, a 1 MB / 10 s cap on fetches, and
|
|
asynchronous processing.
|
|
|
|
### Known limits
|
|
|
|
From the v2.7.0 review, still current:
|
|
|
|
- Home-page mentions (and mentions of anything that isn't a published note)
|
|
are refused rather than stored.
|
|
- Discovery ignores HTTP cache headers (5.3). Targets are fetched only when
|
|
a note is created, edited or deleted.
|
|
- The 410 page is a generic tombstone without microformats.
|
|
- `error` rows (transient failures of a new mention, or unexpected errors)
|
|
are not retried automatically. They are re-checked only when the sender
|
|
re-sends.
|
|
- The 202 response has no `Location` status URL.
|
|
|
|
---
|
|
|
|
## Microformats2
|
|
|
|
Specs: microformats2 parsing, h-entry, h-card, h-feed, h-cite,
|
|
representative h-card, authorship, Post Type Discovery. Decisions:
|
|
[ADR-040](../decisions/ADR-040-microformats2-compliance.md),
|
|
[ADR-061](../decisions/ADR-061-author-discovery.md). Review:
|
|
[v2.8.0 microformats2 review](../design/v2.8.0/2026-11-03-microformats2-review.md).
|
|
|
|
### What pages publish
|
|
|
|
| Page | Markup |
|
|
|---|---|
|
|
| `/` (home) | `h-feed` with `p-name`, hidden `u-url`, hidden `p-author h-card` (photo, name, URL, note), feed `u-photo`; each note an `h-entry` preview with `u-uid` |
|
|
| `/note/<slug>` | `h-entry`: `p-name` only when the note starts with a `# Heading` (so plain notes have no implied name); `u-photo` as direct children above `e-content`; `u-url u-uid` with `dt-published` (RFC 3339 UTC `Z`); `dt-updated`; `p-category` with `rel=tag`; `u-syndication` links; hidden `p-author h-card` |
|
|
| Replies | `u-in-reply-to h-cite` holding the cached reply context (`p-content`, `u-url`) |
|
|
| Likes / reposts / bookmarks | `u-like-of` / `u-repost-of` / `u-bookmark-of` link with "★ Liked" / "🔁 Reposted" / "🔖 Bookmarked" |
|
|
| Received responses | `u-comment h-cite` (replies: `p-author h-card` with `u-url` on an inner link, `u-photo`, `p-name`; `e-content`; `u-url`; `dt-published`), likes, reposts and bookmarks as `u-like`, `u-repost`, `u-bookmark` h-cites in the counts line card; mentions and quotes as `u-mention` h-cites |
|
|
| `/likes`, `/tag/<tag>` | `h-feed` with `p-name`, `u-url`, `p-author h-card` |
|
|
| Response counts | `/`, `/tag/<tag>` and `/likes` show each note's like, repost and reply counts, linking to its `#responses` (v2.13.4) |
|
|
|
|
Post Type Discovery gives note, article, reply, like, repost, bookmark and
|
|
photo correctly.
|
|
|
|
### Identity links
|
|
|
|
`base.html` always emits `rel=me` and `rel=author` pointing at the author's
|
|
home page (`author.url`, else `author.me`, else `ADMIN_ME`), even on a fresh
|
|
install (v2.8.0). Further `rel=me` links come from the discovered profile.
|
|
Because the `p-author h-card`'s `u-url` matches the `rel=me` URL, parsers
|
|
treat it as the page's representative h-card. Without a discovered profile
|
|
there is no h-card, only the `rel` links.
|
|
|
|
### Consuming
|
|
|
|
- **Author profile** (`author_discovery.py`, ADR-061): on login, StarPunk
|
|
fetches `ADMIN_ME`, picks its representative h-card with
|
|
`mf2_utils.representative_hcard()` (uid and url match the page, then
|
|
url matches a `rel=me`, then a single h-card whose url is the page),
|
|
parsed with the URL after redirects. It stores name, photo, URL, note and
|
|
rel-me links, cached for 24 hours. If discovery fails, the stale cache is
|
|
used.
|
|
- **Webmention sources**: `responding_entry`, `response_type` and
|
|
`entry_author` (see [Webmention](#webmention)).
|
|
- **Reply context** (`reply_context.py`): title, snippet, author and site
|
|
name of the post being replied to, liked, reposted or bookmarked, cached
|
|
in `reply_contexts`.
|
|
|
|
### Known gaps
|
|
|
|
From the v2.8.0 review, "Not done":
|
|
|
|
- The reply-context `h-cite` doesn't carry the cached author h-card, name or
|
|
published time.
|
|
- Photo alt text falls back to "Image" when a photo-only post has no
|
|
caption.
|
|
- Person tags (`p-category h-card`) aren't supported.
|
|
- Webmentions received before v2.8.0 keep NULL source details until they
|
|
are re-validated.
|
|
|
|
---
|
|
|
|
## WebSub
|
|
|
|
Spec: https://www.w3.org/TR/websub/. Decision:
|
|
[ADR-071](../decisions/ADR-071-websub.md), including its v2.10.1
|
|
amendments, which replace the original text where they differ.
|
|
|
|
### Topics
|
|
|
|
All absolute URLs under `SITE_URL`:
|
|
|
|
| Topic | Content type | Updated when |
|
|
|---|---|---|
|
|
| `/` | text/html | A public non-like note changes |
|
|
| `/feed.atom`, `/feed.rss`, `/feed.json` | feed types | A public non-like note changes |
|
|
| `/likes` | text/html | A public like changes |
|
|
| `/tag/<name>` | text/html | A note with that tag (before or after the change) changes |
|
|
|
|
`/feed.xml` (legacy) advertises `/feed.rss` as its `rel=self`. The
|
|
negotiated `/feed` advertises the topic of the format it served.
|
|
|
|
### Discovery
|
|
|
|
Topic responses (200/304) carry `Link: <hub>; rel="hub"` and
|
|
`Link: <topic>; rel="self"`. The same links are embedded: `<link rel="hub">`
|
|
and `<link rel="self">` in HTML heads, a hub `<link>` in Atom, an
|
|
`<atom:link rel="hub">` in RSS, and `"hubs"` in JSON Feed. All of this
|
|
happens only while `WEBSUB_ENABLED` (default true).
|
|
|
|
### On change
|
|
|
|
`websub.note_changed()` runs on create, edit, publish/unpublish, delete and
|
|
undelete, and once photos are attached. If the note is or was public, it:
|
|
|
|
1. bumps `content_version`, which is in the feed cache key and the ETag, so
|
|
every process serves fresh feeds at once;
|
|
2. queues one delivery per active subscription for each affected topic.
|
|
Only the latest content is sent: a newer delivery replaces a waiting one.
|
|
|
|
Drafts that stay drafts change nothing.
|
|
|
|
### Hub
|
|
|
|
`POST /websub`, form-encoded only (case-insensitive check):
|
|
|
|
- `hub.mode` is `subscribe` or `unsubscribe`.
|
|
- `hub.topic` must be one of the topics above, after decoding unreserved
|
|
percent-encodings and comparing scheme and host case-insensitively. Tag
|
|
topics must name an existing tag.
|
|
- `hub.callback` must:
|
|
- be printable, at most 2,048 characters, and absolute http(s);
|
|
- not be on this site;
|
|
- use port 80, 443 or 1024 and up;
|
|
- pass the SSRF check.
|
|
|
|
It is stored with an IDNA host.
|
|
- `hub.secret` must be non-empty and under 200 bytes. `hub.lease_seconds`
|
|
is clamped to 1 to 30 days (default 10). Both are ignored for
|
|
unsubscriptions.
|
|
- The hub answers **202**, then verifies later. Errors are 4xx with a
|
|
plain-text reason.
|
|
|
|
Limits (429 when hit):
|
|
|
|
- 30 requests per minute per client IP (shared by all processes; needs
|
|
`TRUSTED_PROXY=true` behind a proxy);
|
|
- at most 200 pending verifications;
|
|
- `WEBSUB_MAX_SUBSCRIPTIONS` (1000) in total;
|
|
- `WEBSUB_MAX_PER_HOST` (25) per callback IP address.
|
|
|
|
The limits count pending requests too. Renewals are always allowed.
|
|
|
|
`WebSubWorker` (file lock, poll `WEBSUB_POLL_INTERVAL` 5 s) handles each
|
|
queued item on its own, so one bad item can't stop it:
|
|
|
|
- **Verification of intent** (5.3): `GET` the callback (existing query
|
|
kept) with `hub.mode`, the topic exactly as requested, `hub.challenge`
|
|
and `hub.lease_seconds`. Success is a 2xx whose body equals the challenge.
|
|
Replies over 4 KB and redirects fail. Network/DNS errors and 5xx are
|
|
retried after 1, 5 and 15 minutes. A request id per intent (migration
|
|
017) stops an older verification from consuming a newer request. A
|
|
subscription row exists only once verified.
|
|
- **Distribution** (7): the topic is rendered inside the app, once per pass,
|
|
and POSTed with its `Content-Type`, `Link` rel=hub/self, and
|
|
`X-Hub-Signature: sha256=<HMAC>` when there is a secret.
|
|
- 2xx: success.
|
|
- 410: removes the subscription.
|
|
- Other failures: retried after 1 min, 5 min, 30 min, 2 h and 6 h, then
|
|
that delivery is dropped. The subscription stays until its lease ends.
|
|
- A topic that renders 404/410 ends its subscriptions with a
|
|
`hub.mode=denied` notice.
|
|
- Expired subscriptions are pruned hourly.
|
|
|
|
Outbound requests use `safe_get` / `safe_post`: pinned, no redirects,
|
|
10 s hard deadline.
|
|
|
|
**External hub**: with `WEBSUB_HUB_URL` set, StarPunk advertises that hub
|
|
and sends it a `hub.mode=publish` ping per changed topic. Its own `/websub`
|
|
refuses new subscriptions but still accepts unsubscriptions.
|
|
|
|
The admin dashboard shows subscribers per topic and deliveries being
|
|
retried.
|
|
|
|
### Known limits
|
|
|
|
- The hub serves only this site's topics, by design. websub.rocks' hub
|
|
tests use their own topics, so they can't be run against it.
|
|
- Deliveries always carry the full topic content (no diffs). Fat pings
|
|
aren't sent to external hubs.
|
|
|
|
---
|
|
|
|
## Feeds
|
|
|
|
Decisions: [ADR-014](../decisions/ADR-014-rss-feed-implementation.md),
|
|
[ADR-038](../decisions/ADR-038-syndication-formats.md),
|
|
[ADR-059](../decisions/ADR-059-full-feed-media-standardization.md).
|
|
|
|
| URL | Format |
|
|
|---|---|
|
|
| `/feed.rss` | RSS 2.0 (with `atom:link rel=self` and Media RSS) |
|
|
| `/feed.xml` | Legacy alias of `/feed.rss` |
|
|
| `/feed.atom` | Atom 1.0 |
|
|
| `/feed.json` | JSON Feed 1.1 |
|
|
| `/feed` | Content negotiation on `Accept` (RSS default; 406 with `X-Available-Formats` if nothing fits; `Vary: Accept`) |
|
|
| `/opml.xml` | OPML 2.0 listing the three feeds |
|
|
|
|
- Feeds contain the newest `FEED_MAX_ITEMS` (50) published notes, with full
|
|
HTML content and media. Likes are excluded, as on the home page.
|
|
Reposts and bookmarks start with a "🔁 Reposted" / "🔖 Bookmarked" line.
|
|
- Feeds are cached for `FEED_CACHE_SECONDS` (300). The cache key and ETag
|
|
include `content_version`, so any change to a public note invalidates
|
|
them in every process. `If-None-Match` gets 304.
|
|
- `base.html` advertises the negotiated `/feed` (as RSS) and the OPML file
|
|
with `rel=alternate`. Atom and JSON Feed are listed in the OPML but have
|
|
no `<link rel=alternate>` of their own.
|
|
- WebSub hub links are embedded in each feed (see above).
|
|
|
|
Gaps: there are no per-tag feeds (tag pages are h-feeds and WebSub topics,
|
|
but have no RSS/Atom/JSON equivalent). The `/likes` page has no feed.
|