Compare commits

..
23 Commits
Author SHA1 Message Date
Phil 191c99b59b Merge branch 'feat/v2.17.0-article-urls' for v2.17.0 release
Build Container / build (push) Failing after 9m31s
2026-11-12 12:27:55 -07:00
PhilandClaude Opus 5.5 b7f8e1612c feat: articles at /article/<slug> with slugs from their titles (v2.17.0, ADR-077)
Articles (notes with a title) move from /note/<slug> to /article/<slug>;
the other prefix 301-redirects. New articles take their slug from the
title (mp-slug still wins; notes keep timestamp slugs). A Micropub update
that adds or removes a title answers 201 with the new Location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:27:31 -07:00
Phil 66aecc2bcb Merge branch 'feat/v2.16.0-articles' for v2.16.0 release
Build Container / build (push) Successful in 9m35s
2026-11-11 17:25:39 -07:00
PhilandClaude Opus 5.5 ccb9f6dce2 fix: keep inline images in Micropub articles (v2.16.0)
- Content sent as both html and text is stored as the HTML; the text is
  usually a plain fallback, so images, links and headings were lost.
- Images embedded from other sites are copied into the media library
  (SSRF-checked fetcher, best effort, max 20) and the text points at the
  copies, on create and on content replace.
- HTML content is sanitized as is instead of going through Markdown, which
  added stray <br>s and empty paragraphs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:11:46 -07:00
PhilandClaude Opus 5.5 978c13358b feat: article titles and summaries stored apart from the text (v2.16.0, ADR-076)
Articles as the IndieWeb defines them: a name separate from the content.

- Migration 023 adds notes.title and notes.summary. A one-time Python data
  step (migrations.DATA_MIGRATIONS) moves each note's '# Heading' first line
  into title, rewrites the file and hash, empties FTS for a rebuild, and puts
  files back if the migration fails. Trashed notes included.
- Micropub: name -> title, summary stored when sent with content, summary
  updatable, q=source returns both. MCP tools take a summary.
- Feeds: titles only for articles (RSS/JSON none, Atom empty); Atom and
  JSON Feed summaries; the title is no longer repeated in the content.
- Pages: p-name/p-summary on the article page. Homepage and tag pages show
  articles as title, summary and "Read more" without images (hidden u-photo
  keeps the post type for parsers).
- Crossposts: articles go out as title, summary, link and hashtags, no
  images; the Bluesky card is the article. Notes unchanged.
- Editor: Title and Summary fields; an article may have no body.
- Fix: a line starting with a hashtag ('#indieweb ...') was rendered as a
  heading and crossposted without its '#' (HashtagLineExtension).
- Fix: excerpts showed raw Markdown; they are plain text now.

Back up the database and the data folder before deploying: the migration
rewrites note files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:05:38 -07:00
Phil 9a0d82bab2 Merge branch 'feat/v2.15.0-mcp' for v2.15.0 release
Build Container / build (push) Successful in 9m40s
2026-11-11 16:39:39 -07:00
PhilandClaude Opus 5.5 84411a16d9 feat: MCP server for AI clients (v2.15.0, ADR-075)
Add an MCP server at SITE_URL/mcp so AI assistants (claude.ai connectors,
Claude Code) can search, read and write notes.

- starpunk/mcp_server: official MCP SDK 2.3, stateless Streamable HTTP with
  JSON responses, served by uvicorn as a sidecar from the same image.
  Ten tools; writes go through the Micropub handlers.
- IndieAuth: RFC 8707 resource indicators bound to codes and tokens
  (migration 022), client ID metadata document + public client metadata,
  draft in scopes_supported, default MCP scopes, consent screen note.
- BACKGROUND_WORKERS_ENABLED flag so the sidecar runs no workers.
- Docs: ADR-075, architecture/mcp.md and updates; CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:36:58 -07:00
Phil 4ed2033cf9 Merge branch 'fix/v2.14.2-quotes-cleanup' for v2.14.2 release
Build Container / build (push) Successful in 9m37s
2026-11-10 15:54:55 -07:00
PhilandClaude Opus 5.5 d354dc9733 fix: quote counts in the counts line; remove dead editor scripts and files (v2.14.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:54:46 -07:00
Phil 88aea09e04 Merge branch 'fix/v2.14.1-tags-captions' for v2.14.1 release
Build Container / build (push) Successful in 9m6s
2026-11-10 15:37:43 -07:00
PhilandClaude Opus 5.5 7e2cdef150 fix: clear all tags from the editor; escape captions in feed HTML (v2.14.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:37:35 -07:00
Phil 058b1959f9 Merge branch 'docs/v2.14.0-doc-review': documentation review for v2.14.0 2026-11-10 15:35:00 -07:00
PhilandClaude Opus 5.5 ceef048003 docs: documentation review for v2.14.0
Architecture, README, standards, CLAUDE.md, settings reference, indexes,
backlog, release plans and CHANGELOG checked against the code at v2.14.0:
backfeed, notifications, counts line and web actions documented; test
policy (changed area locally, full suite in CI); stale versions, statuses
and links fixed; legacy and superseded docs moved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:51:57 -07:00
Phil caa66f7bb1 Merge branch 'feat/v2.14.0-notifications' for v2.14.0 release
Build Container / build (push) Successful in 9m15s
2026-11-10 14:30:10 -07:00
PhilandClaude Opus 5.5 0465ff924b feat: notifications through Apprise (v2.14.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:30:01 -07:00
Phil 815aff5443 Merge branch 'docs/v2.14.0-notifications-adr': ADR-074 notifications 2026-11-10 14:25:58 -07:00
PhilandClaude Opus 5.5 4306633a42 docs: ADR-074 accepted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:25:57 -07:00
PhilandClaude Opus 5.5 c5c2d9832e docs: ADR-074: one configurable priority for all messages
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:24:43 -07:00
PhilandClaude Opus 5.5 483c24b565 docs: ADR-074 notifications through Apprise (proposed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:17:30 -07:00
Phil d312fa0074 Merge branch 'fix/v2.13.4-list-counts' for v2.13.4 release
Build Container / build (push) Successful in 8m56s
2026-11-10 14:05:05 -07:00
PhilandClaude Opus 5.5 647f6c723b feat: counts on lists of notes, reply cards link to the reply, fix class clash (v2.13.4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:05:03 -07:00
Phil 81c730813c Merge branch 'fix/v2.13.3-count-opted-out' for v2.13.3 release
Build Container / build (push) Successful in 8m35s
2026-11-10 13:19:18 -07:00
PhilandClaude Opus 5.5 113f28aff1 fix: count opted-out likes and reposts without naming them (v2.13.3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:16:19 -07:00
132 changed files with 4353 additions and 743 deletions

No files matched your search

+24 -5
View File
@@ -1,4 +1,4 @@
# StarPunk configuration template (v2.10.1)
# StarPunk configuration template (v2.14.0)
#
# Copy to .env and fill in your values. DO NOT commit .env.
# Every variable here is read by starpunk/config.py. Uncommented lines are
@@ -124,10 +124,11 @@ CSRF_ENABLED=true
# WEBMENTION
# =============================================================================
# Receive webmentions at /webmention (default: true)
# Webmention endpoint at /webmention, its Link header, and showing responses
# (webmentions and backfeed) on notes; sending is not affected (default: true)
# WEBMENTION_ENABLED=true
# Background worker that verifies received webmentions and sends outgoing
# ones (default: true)
# Background worker that verifies received webmentions, sends outgoing ones,
# and sends response notifications (v2.14.0) (default: true)
# WEBMENTION_WORKER_ENABLED=true
# Seconds between queue checks (default: 10) and after an error (default: 60)
# WEBMENTION_POLL_INTERVAL=10
@@ -201,11 +202,29 @@ THREADS_USERNAME=
# SYNDICATION_POLL_INTERVAL=10
# Backfeed (v2.13.0, ADR-073): bring likes, reposts, quotes and replies to the
# crossposts back to the note as responses. Needs the read scopes above.
# crossposts back to the note as responses. Needs the read scopes above, and
# runs in the syndication worker, so at least one target must be configured.
# BACKFEED_ENABLED=false
# Stop checking a note's responses after this many days (default: 90)
# BACKFEED_MAX_AGE_DAYS=90
# =============================================================================
# NOTIFICATIONS (v2.14.0, ADR-074; optional)
# =============================================================================
# Apprise URLs to notify, space- or comma-separated
# (https://github.com/caronc/apprise/wiki). Empty: notifications off.
# Examples: gotifys://gotify.example.com/APPTOKEN ntfys://my-topic
# apprises://apprise.example.com/KEY (an apprise-api server)
# NOTIFY_URLS=
# What to notify: reply, mention, quote, like, repost, bookmark, problems
# (a crosspost that failed for good, a silo refusing backfeed). Names are
# case-insensitive; unknown ones are ignored.
# NOTIFY_EVENTS=reply,mention,quote,problems
# Priority of every message, added to URLs that don't set one:
# low, moderate, normal, high, emergency (Gotify also takes 0-10)
# NOTIFY_PRIORITY=normal
# =============================================================================
# DEBUGGING
# =============================================================================
+108 -5
View File
@@ -7,6 +7,112 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.17.0] - 2026-11-12
Articles get URLs of their own (ADR-077).
### Changed
- Articles live at `/article/<slug>` instead of `/note/<slug>`. The other address redirects (301), so links keep working when a note gains or loses a title. Feeds, pages, Micropub, crossposts, webmentions, notifications, search and the MCP tools all give the new URLs
- A new article's slug comes from its title (`Why I Left Twitter` → `why-i-left-twitter`; `-2` if taken). An `mp-slug` still wins, notes keep timestamp slugs, and slugs don't change when a title is edited later
- A Micropub update that adds or removes a title answers 201 with the post's new URL in `Location`, as the spec requires
## [2.16.0] - 2026-11-11
Articles as the IndieWeb defines them: titles and summaries stored apart from the text (ADR-076).
**Before upgrading, back up the database and the `data` folder (notes and `.trash`).** Migration 023 moves each note's `# Heading` title out of its Markdown file into the database.
### Added
- Notes have a title and a summary of their own. A note with a title is an article. The editor has Title and Summary fields, and an article may have no body text
- Micropub `summary` is stored when sent with `content`, can be replaced and deleted, and comes back from `q=source`. A summary on its own is still the content, as Post Type Discovery reads it. The MCP `create_note` and `update_note` tools take a `summary`
- Article pages mark up the summary as `p-summary`; Atom entries have `<summary>` and JSON Feed items `summary`
### Changed
- A `# Heading` at the top of a note is now just a heading, not a title. Micropub `name` is stored as the title instead of being added to the text; a heading that repeats it is dropped
- Feed items have a title only for articles. RSS items and JSON Feed items for notes have none, and Atom entries an empty one, so feed readers no longer show notes as headlines
- On the homepage and tag pages an article shows its title, its summary (or the start of its text) and "Read more", without its images
- Micropub content sent as both `html` and `text` is stored as the HTML. The text is usually a plain fallback, so articles lost their images, links and headings
- Images embedded in Micropub content from other sites are copied into the media library and the text points at the copies, so an article doesn't break when the client's or a CDN's host goes away. Best effort: an image that can't be fetched keeps its URL (up to 20 per post)
- Articles are crossposted to GoToSocial, Bluesky and Threads as title, summary (or the start of the text), link and hashtags, without images. On Bluesky the link card is the article. Notes are crossposted in full as before
### Fixed
- Feeds showed an article's title twice: as the item title and again at the top of the content
- A line starting with a hashtag (`#indieweb is fun`) was shown as a heading on pages and in feeds, and crossposted without its `#`. Only `#` followed by a space starts a heading now, as in CommonMark
- HTML content from Micropub is sanitized as is instead of going through Markdown, which added stray line breaks and empty paragraphs
- Previews, image alt text and crosspost teasers showed raw Markdown (`**bold**`, `[link](url)`); the excerpt is now plain text
## [2.15.0] - 2026-11-11
MCP server for AI clients (ADR-075).
### Added
- An [MCP](https://modelcontextprotocol.io/) server at `SITE_URL/mcp`. AI assistants such as claude.ai custom connectors and Claude Code can search, list and read notes and drafts, list tags, create notes (drafts by default), update, delete and undelete them, and upload photos. Writes go through the Micropub handlers, so scopes, crossposting, webmentions and WebSub behave the same as with any Micropub app
- The MCP server runs as a sidecar from the same image (`uvicorn starpunk.mcp_server.app:app --port 8001`) with `BACKGROUND_WORKERS_ENABLED=false`; the proxy sends `/mcp*` and `/.well-known/oauth-protected-resource*` to it. New dependencies: `mcp` 2.3 and `uvicorn` 0.54
- The IndieAuth server accepts RFC 8707 `resource` for the MCP endpoint and binds codes and tokens to it (migration 022). Its metadata advertises client ID metadata documents and public clients (`token_endpoint_auth_methods_supported: ["none"]`), so Claude connects without dynamic client registration. The consent screen says when an AI assistant is asking, and Connected Apps marks MCP tokens
- `BACKGROUND_WORKERS_ENABLED` (default `true`) turns off the webmention, WebSub and syndication workers
### Fixed
- `scopes_supported` in the IndieAuth metadata lists `draft`, which Micropub already honoured
## [2.14.2] - 2026-11-10
### Added
- Quotes have a count in the line under each note (❝) and in the counts on the home, tag and Likes pages. It includes quotes a silo reports without names (Threads, and people who opted out of backfeed), which were stored but shown nowhere
### Removed
- The editor pages no longer load `marked` from a CDN and an empty `static/js/preview.js`; nothing used them
- The empty `templates/feed.xml`, the v1 `static/indielogin-test.html`, and the `THREADS_API` setting that crossposting read but nothing could set
## [2.14.1] - 2026-11-10
### Fixed
- Clearing the Tags field in the note editor removes all of the note's tags. It used to leave them unchanged, so a note's last tags couldn't be removed from the editor
- Image captions are escaped in the HTML of RSS, Atom and JSON Feed items. A caption containing `"`, `<` or `&` broke the item's image markup (the Atom enclosure's title was already escaped)
## [2.14.0] - 2026-11-10
Notifications (ADR-074).
### Added
- StarPunk can notify you of new responses through [Apprise](https://github.com/caronc/apprise): Gotify, ntfy, Pushover, email, Matrix, Discord and about 100 more, or an apprise-api server. Set `NOTIFY_URLS` to one or more Apprise URLs (e.g. `gotifys://gotify.example.com/APPTOKEN`)
- `NOTIFY_EVENTS` chooses what you hear about: `reply`, `mention`, `quote`, `like`, `repost`, `bookmark`, and `problems` (a crosspost that failed for good, or a silo refusing backfeed). Default `reply,mention,quote,problems`. Replies, mentions and quotes come one at a time with their text; likes, reposts and bookmarks are grouped per note
- `NOTIFY_PRIORITY` (default `normal`) sets the priority of every message
- Received webmentions and backfed responses are notified alike, each once. Responses waiting for moderation say so and link to the admin
- Admin → Webmentions shows the notification settings and the last send error, with **Send test notification**
### Changed
- New dependency: `apprise` 2.2. Migration 021 marks every existing response as already notified, and responses found by a backfill (or on a crosspost's first check of an older note) are never notified
## [2.13.4] - 2026-11-10
### Added
- The home page, tag pages and the Likes page show each note's like, repost and reply counts next to its date (♥ 3 🔁 1 💬 2), linking to the note's responses. Notes with none show nothing
### Changed
- Clicking anywhere on a reply opens it where it was posted (the reply on Bluesky, GoToSocial or Threads, or the page that sent the webmention). The author link and links in the reply still go to their own pages
### Fixed
- The "Liked", "Reposted" and "In reply to" lines on notes picked up styles meant for v2.13.2's counts line (both used the class `reaction`). The counts line's classes are now `response-*`
## [2.13.3] - 2026-11-10
### Changed
- Likes and reposts by people who opted out of backfeed (`#nobridge` or `#nobot` in their profile, or Bluesky's "don't show me to logged-out users" label) are counted in a note's totals, without their name or avatar ("+1 on Bluesky"). Before, they were left out entirely, so totals were lower than on the silo
## [2.13.2] - 2026-11-10
### Changed
@@ -598,7 +704,7 @@ Webmention now follows the [W3C Recommendation](https://www.w3.org/TR/webmention
- Successful uploads logged at INFO level with file details
- Removed duplicate logging in Micropub media endpoint
## [1.4.0rc1] - 2025-12-16
## [1.4.0-rc.1] - 2025-12-16
### Added
@@ -2141,7 +2247,4 @@ This is a release candidate for testing. Stable 1.0.0 will be released after tes
- ADR-007: Slug generation algorithm
- ADR-008: Versioning strategy
[Unreleased]: https://github.com/YOUR_USERNAME/starpunk/compare/v0.4.0...HEAD
[0.4.0]: https://github.com/YOUR_USERNAME/starpunk/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/YOUR_USERNAME/starpunk/compare/v0.1.0...v0.3.0
[0.1.0]: https://github.com/YOUR_USERNAME/starpunk/releases/tag/v0.1.0
Releases are tagged `vX.Y.Z` in https://git.thesatelliteoflove.com/phil/StarPunk
+3 -3
View File
@@ -8,7 +8,7 @@ This file contains operational instructions for Claude agents working on this pr
- All Python commands must be run with `uv run` prefix
- Example: `uv run pytest`, `uv run flask --app app.py run --debug`
- There is no `pyproject.toml`: create the environment with `uv venv --python 3.14 && uv pip install -r requirements-dev.txt`
- Tests need `SESSION_SECRET` set: `SESSION_SECRET=test uv run pytest`. The full suite takes about 6 minutes, 5 of them in `tests/test_migration_race_condition.py`; add `--ignore=tests/test_migration_race_condition.py` while iterating, but run everything before a release
- Tests need `SESSION_SECRET` set. Locally, run only the tests for the area you changed (`SESSION_SECRET=test uv run pytest tests/test_<area>.py`); CI runs the full suite (about 6 minutes) on every `vX.Y.Z` tag, so don't run the whole suite yourself unless asked
- Local login without OIDC: `DEV_MODE=true` and `DEV_ADMIN_ME` in `.env`, then `/dev/login`
- Setup details: `docs/standards/development-setup.md`
@@ -50,12 +50,12 @@ When invoking the agent-developer, always remind it to:
5. **Follow git protocol**
- Adhere to git branching strategy in `docs/standards/git-branching-strategy.md`
- Create feature branches for non-trivial changes, in their own git worktree (`<prefix>/vX.Y.Z-description`, prefixes `feat/`, `fix/`, `ci/`, `docs/`)
- Create feature branches for non-trivial changes, in their own git worktree (`<prefix>/vX.Y.Z-description`, prefixes `feat/`, `fix/`, `ci/`, `chore/`, `docs/`)
- Write clear commit messages (`feat: ... (vX.Y.Z)`)
## Release Process
Follow "Release process" in `docs/standards/git-branching-strategy.md`. In short: work in a worktree on a `feat/`/`fix/` branch → bump `__version__` and `__version_info__` → `CHANGELOG.md` entry → design note in `docs/design/vX.Y.Z/` (or ADR amendment) → full test suite → commit → `git merge --no-ff` into `main` ("Merge branch '<branch>' for vX.Y.Z release") → annotated tag `vX.Y.Z` → push `main` and the tag. CI (`.gitea/workflows/build-container.yml`) runs the tests and publishes the image; production is then updated from the separate Ansible deployment repository. Back up the database before deploying a release with a migration.
Follow "Release process" in `docs/standards/git-branching-strategy.md`. In short: work in a worktree on a `feat/`/`fix/`/`chore/` branch → bump `__version__` and `__version_info__` → `CHANGELOG.md` entry → design note in `docs/design/vX.Y.Z/` (or ADR amendment) → tests for the changed area → commit → `git merge --no-ff` into `main` ("Merge branch '<branch>' for vX.Y.Z release") → annotated tag `vX.Y.Z` → push `main` and the tag. CI (`.gitea/workflows/build-container.yml`) runs the full test suite and publishes the image; production is then updated from the separate Ansible deployment repository. Back up the database before deploying a release with a migration.
Merging to `main`, tagging, pushing and deploying happen only when the project owner asks for a release.
+2
View File
@@ -78,6 +78,8 @@ USER starpunk
# Expose application port
EXPOSE 8000
# MCP sidecar (same image, run with: uvicorn starpunk.mcp_server.app:app --host 0.0.0.0 --port 8001)
EXPOSE 8001
# Health check
# Uses httpx (already in requirements) to verify app is responding
+18 -8
View File
@@ -23,9 +23,13 @@ what changed in each release is in [CHANGELOG.md](CHANGELOG.md).
- Built-in [IndieAuth](https://indieauth.spec.indieweb.org/) server for
Micropub apps (consent screen, tokens, revocation, Connected Apps page)
- Admin login through any OpenID Connect provider (e.g. Pocket ID)
- [MCP](https://modelcontextprotocol.io/) server at `/mcp` so AI assistants
(claude.ai connectors, Claude Code…) can search, read and write notes
after you approve them, run as an optional sidecar (ADR-075)
- [Webmention](https://www.w3.org/TR/webmention/): receiving (verified,
moderated, rate-limited) and sending; replies, likes and reposts shown on
each note
moderated, rate-limited) and sending; replies, quotes and mentions shown
on each note, with likes, reposts and replies as a line of counts with
hover cards and `<indie-action>` web actions
- [WebSub](https://www.w3.org/TR/websub/) with a built-in hub, so readers get
new posts within seconds
- [Microformats2](https://microformats.org/wiki/h-entry): h-entry, h-feed and
@@ -38,10 +42,15 @@ what changed in each release is in [CHANGELOG.md](CHANGELOG.md).
- POSSE crossposting to GoToSocial/Mastodon, Bluesky and Threads, with
`u-syndication` links back; edits and deletes follow, native likes and
boosts where the target supports them
- Backfeed: likes, reposts and replies to crossposts on GoToSocial and
Bluesky (and Bluesky quotes), and replies on Threads (with its like and
repost counts), come back as responses (`BACKFEED_ENABLED`, ADR-073)
**Operations**
- Migrations applied automatically at startup
- `/health` endpoint, `/admin/health` diagnostics, structured logs
- Notifications of new responses and crossposting problems through
Apprise (Gotify, ntfy, email…; `NOTIFY_URLS`, ADR-074)
- Fetches of user-supplied URLs are limited to public addresses, size and
time
@@ -54,7 +63,6 @@ git clone https://git.thesatelliteoflove.com/phil/StarPunk.git && cd StarPunk
cp .env.example .env # set SITE_URL, ADMIN_ME, SESSION_SECRET, OIDC_*
docker build -t starpunk -f Containerfile .
docker run -d --name starpunk -p 127.0.0.1:8000:8000 --env-file .env \
-e FLASK_ENV=production -e FLASK_DEBUG=0 \
-v ./starpunk-data:/data starpunk
```
@@ -72,8 +80,8 @@ redirect URI with your OIDC provider. Details:
uv venv --python 3.14
uv pip install -r requirements-dev.txt
cp .env.example .env # set SESSION_SECRET; add DEV_MODE=true and DEV_ADMIN_ME
uv run flask --app app.py run --debug # http://localhost:5000
SESSION_SECRET=test uv run pytest # tests (about 6 minutes)
uv run flask --app app.py run --debug # http://localhost:5000
SESSION_SECRET=test uv run pytest tests/test_<area>.py # tests for what you changed; CI runs the full suite
```
With `DEV_MODE=true`, `/dev/login` logs you in without OIDC. Never enable
@@ -86,13 +94,15 @@ All settings are environment variables (or a `.env` file), each listed with
its default in [`.env.example`](.env.example). Required in production:
`SESSION_SECRET`, `SITE_URL`, `ADMIN_ME`, `OIDC_ISSUER`, `OIDC_CLIENT_ID`,
`OIDC_CLIENT_SECRET` and `OIDC_ADMIN_SUB`. Crossposting is enabled by
setting a target's credentials.
setting a target's credentials. Backfeed is off unless
`BACKFEED_ENABLED=true`; notifications start when `NOTIFY_URLS` is set.
## Data and backups
Everything lives in the data directory (`/data` in the container): the
SQLite database, note files in `notes/`, uploaded media and logs. Back it
up regularly, and always before upgrading to a release with a migration.
SQLite database, note files in `notes/`, uploaded media, saved response
avatars in `avatars/`, the Threads token (`threads_token.json`) and logs.
Back it up regularly, and always before upgrading to a release with a migration.
## Documentation
+9 -7
View File
@@ -1,9 +1,11 @@
# Architecture Documentation Index
This directory describes the **current** design of StarPunk (v2.10.1). Per
This directory describes the **current** design of StarPunk (v2.14.0). Per
[../DOCUMENTATION.md](../DOCUMENTATION.md) it is updated at the end of each
release. Decisions and their history are in [../decisions/](../decisions/);
per-release designs and reports are in [../design/](../design/).
per-release designs and reports are in [../design/](../design/). The
original v0.x technology choices are kept in
[../design/v0.x/technology-stack-legacy.md](../design/v0.x/technology-stack-legacy.md).
## Documents
@@ -11,12 +13,12 @@ per-release designs and reports are in [../design/](../design/).
|----------|-------------|
| [overview.md](overview.md) | System overview: purpose, request flow, modules and routes, background workers, storage, auth model, URL surface, caching, security posture |
| [technology-stack.md](technology-stack.md) | Current runtime, Python dependencies, standards implemented, testing and CI |
| [database.md](database.md) | SQLite schema after migrations 001–019, migration runner and its lock, connection pool |
| [database.md](database.md) | SQLite schema after migrations 001–022, migration runner and its lock, connection pool |
| [deployment.md](deployment.md) | Container image stages, CI workflow, release process, `/data` layout, reverse proxy, all environment variables |
| [simplified-auth-architecture.md](simplified-auth-architecture.md) | Admin login via OpenID Connect and the built-in IndieAuth server that issues Micropub tokens (ADR-064) |
| [indieweb-protocols.md](indieweb-protocols.md) | How Micropub, IndieAuth, Webmention, microformats2, WebSub and the feeds are implemented, with compliance status and known gaps |
| [syndication-architecture.md](syndication-architecture.md) | Crossposting (POSSE) to GoToSocial, Bluesky and Threads: syndications table, worker, reconcile rules |
| [technology-stack-legacy.md](technology-stack-legacy.md) | Historical: the original v0.x technology choices (IndieLogin era); not current |
| [indieweb-protocols.md](indieweb-protocols.md) | How Micropub, IndieAuth, Webmention, microformats2, WebSub and the feeds are implemented (including backfeed responses, web actions and notifications), with compliance status and known gaps |
| [mcp.md](mcp.md) | The MCP server for AI clients: sidecar, authorization flow, tools (ADR-075) |
| [syndication-architecture.md](syndication-architecture.md) | Crossposting (POSSE) to GoToSocial, Bluesky and Threads: syndications table, worker, reconcile rules, backfeed of responses (ADR-073) |
## Where to start
@@ -37,4 +39,4 @@ per-release designs and reports are in [../design/](../design/).
---
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
+30 -15
View File
@@ -1,8 +1,8 @@
# Database Architecture
**Version**: 2.11.0 (2026-11-03)
**Version**: 2.14.0 (2026-11-10)
**Status**: Current. Verified against a database created by `create_app()`
with migrations 001–019.
with migrations 001–022.
StarPunk uses one SQLite file, `starpunk.db` under `DATA_PATH` (`/data` in
the container), in WAL mode. Note *content* is not in the database: it lives
@@ -73,6 +73,9 @@ Background: [ADR-031](../decisions/ADR-031-database-migration-system-redesign.md
| `017_websub_hardening.sql` | 2.10.1 | `websub_intents.request_id`, `requested_topic`, `callback_ip`; `websub_subscriptions.callback_ip` |
| `018_foreign_key_cleanup.sql` | 2.11.0 | Removes rows orphaned while foreign keys weren't enforced |
| `019_shared_rate_limits.sql` | 2.11.0 | `rate_limits` (counters shared by all processes) |
| `020_backfeed.sql` | 2.13.0 | Rebuilds `webmentions` adding `origin`, `remote_id`, the `quote` type and an index on (`target_note_id`, `origin`); `syndications.backfeed_next_at`, `backfeed_checked_at`, `backfeed_claimed_until`, `backfeed_attempts`, `backfeed_error`, `backfeed_backfill`, `likes_count`, `reposts_count`, `quotes_count` |
| `021_notifications.sql` | 2.14.0 | `webmentions.notified_at`; existing rows are marked as notified |
| `022_indieauth_resource.sql` | 2.15.0 | `indieauth_codes.resource` and `indieauth_tokens.resource` (RFC 8707, MCP tokens; ADR-075) |
New migrations are numbered `NNN_description.sql`, must be safe to run inside
one transaction, and should use `IF NOT EXISTS` where SQLite allows it.
@@ -126,12 +129,12 @@ value), `me`, `created_at`, `expires_at`, `last_used_at`, `user_agent`,
(primary key), `nonce`, `code_verifier` (PKCE), `next_url`, `expires_at`.
Consumed once on callback.
**`indieauth_codes`**: authorization codes issued to Micropub clients:
**`indieauth_codes`**: authorization codes issued to Micropub and MCP clients:
`code_hash`, `client_id`, `redirect_uri`, `scope`, `code_challenge`,
`expires_at`, `used_at` (single use).
`resource` (`SITE_URL/mcp` or NULL), `expires_at`, `used_at` (single use).
**`indieauth_tokens`**: access tokens: `token_hash`, `client_id`, `scope`,
`created_at`, `expires_at`, `last_used_at`, `revoked_at`. Listed and revoked
`resource` (`SITE_URL/mcp` for MCP tokens, NULL for Micropub), `created_at`, `expires_at`, `last_used_at`, `revoked_at`. Listed and revoked
in `/admin/tokens`.
**Legacy tables, unused**: `auth_state` (v1 IndieLogin login state), `tokens`
@@ -143,17 +146,20 @@ above.
### IndieWeb interactions
**`webmentions`**: received webmentions, one per (`source_url`,
`target_url`).
**`webmentions`**: received webmentions and backfed silo responses
(v2.13.0), one per (`source_url`, `target_url`).
| Column | Purpose |
|--------|---------|
| `target_note_id` | The note mentioned (SET NULL if the note is deleted) |
| `mention_type` | `reply`, `like`, `repost`, `bookmark`, `mention` |
| `mention_type` | `reply`, `like`, `repost`, `bookmark`, `mention`, `quote` |
| `author_name`, `author_url`, `author_photo`, `content_snippet` | From the source's mf2 (content sanitised) |
| `source_post_url`, `source_published` | The source entry's own `u-url` and `dt-published` |
| `validation_status` | `pending`, `valid`, `invalid`, `error` (set by the worker) |
| `moderation_status` | `pending`, `approved`, `hidden`, `deleted` (auto-approved when `WEBMENTION_AUTO_APPROVE`) |
| `moderation_status` | `pending`, `approved`, `hidden`, `deleted` (auto-approved when `WEBMENTION_AUTO_APPROVE`). `deleted` is a tombstone: deleting in the admin keeps the row, so a re-sent webmention or the next backfeed check doesn't bring it back; re-validation keeps `hidden` and `deleted` |
| `origin` | `webmention`, or the silo it was backfed from (`gotosocial`, `bluesky`, `threads`) |
| `remote_id` | Silo id of a backfed response (or of the person, for likes and reposts) |
| `notified_at` | When the owner was notified of it (v2.14.0); NULL means not yet |
| `validation_error`, `validated_at`, `received_at`, `updated_at` | Bookkeeping |
**`webmention_queue`**: outgoing webmentions, one per (`source_url`,
@@ -178,7 +184,12 @@ the note should be on that target; `remote_id` and `remote_url` identify the
remote post. `state` (`pending`, `synced`, `failed`), `version` (bumped on
every note change, so an edit made during a send is not lost), `attempts`,
`last_error`, `next_attempt_at` and `claimed_until` (a 5-minute lease that
lets any gunicorn process take the row exclusively) drive the worker. See
lets any gunicorn process take the row exclusively) drive the worker.
Backfeed (v2.13.0) adds `backfeed_next_at`, `backfeed_checked_at`,
`backfeed_claimed_until` (its own lease), `backfeed_attempts`,
`backfeed_error` and `backfeed_backfill` (set while a backfill is due), and
`likes_count`, `reposts_count`, `quotes_count`: silo-only counts (all of
Threads; elsewhere people who opted out). See
[syndication-architecture.md](syndication-architecture.md).
### WebSub
@@ -201,7 +212,10 @@ pending row of each kind. `attempts`, `next_attempt_at`, `last_error`.
**`site_state`**: key/value pairs. `content_version` is bumped whenever a
public note changes; every gunicorn process compares it to decide whether
its cached feed data is stale.
its cached feed data is stale. Backfeed keeps `backfeed_backfill_last` (spacing
backfill checks across processes), `backfeed_backfill_started` and
`backfeed_backfill_finished`; notifications keep `notify_last_error` (shown
in the admin, cleared by a successful send).
### Bookkeeping
@@ -223,7 +237,8 @@ server process enforces one shared limit. Old windows are pruned as it goes.
it goes back to the pool, rolled back first if a transaction was left
open.
- **Direct connections**: the migration runner, the FTS start-up check, the
webmention worker and sender, reply-context fetching and search helpers
webmention worker and sender, the notification sweep (`notifications.py`),
avatar upkeep (`avatars.py`), reply-context fetching and search helpers
open their own `sqlite3.connect(db_path)` connections.
- WAL mode lets readers carry on while one writer writes. The four gunicorn
processes and the worker threads share the one file; writes are short, so
@@ -233,9 +248,9 @@ server process enforces one shared limit. Old windows are pruned as it goes.
The application does not back itself up. A consistent copy needs the
database (use SQLite's online backup, e.g. `sqlite3 starpunk.db ".backup …"`,
because of WAL) together with `notes/` and `media/`; see
[deployment.md](deployment.md).
because of WAL) together with `notes/` and `media/`. `avatars/` is optional:
missing avatars are fetched again. See [deployment.md](deployment.md).
---
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
+40 -23
View File
@@ -1,6 +1,6 @@
# Deployment Architecture
**Version**: 2.10.1 (2026-11-03)
**Version**: 2.14.0 (2026-11-10)
**Status**: Current
How StarPunk is built, configured and run in production. The production site
@@ -70,21 +70,18 @@ authentication.
lower case.
4. Remove the builder and log out, even after a failure.
There is no CI on branches or pull requests; run the tests locally before
merging (see [technology-stack.md](technology-stack.md#testing)).
There is no CI on branches or pull requests. Locally, run the tests for the
area you changed; CI runs the full suite on the tag (see
[technology-stack.md](technology-stack.md#testing)).
## Release process
1. Finish the change on a branch, including the version bump in
`starpunk/__init__.py` (`__version__`, `__version_info__`) and a
`CHANGELOG.md` entry.
2. Merge to `main`.
3. Tag the merge commit `vX.Y.Z` and push the tag. CI tests, builds and
pushes the image.
4. In the Ansible-managed deployment, change the StarPunk image tag to the new
version and apply it. The container is recreated from the new image; new
migrations run automatically at start-up.
5. Check `/health` reports the new version.
Branching, version bump, changelog, merge and tagging are in
[git-branching-strategy.md](../standards/git-branching-strategy.md#release-process).
Pushing a `vX.Y.Z` tag makes CI test, build and push the image. Then, in the
Ansible-managed deployment, set `starpunk_version` to the new tag and deploy
the `starpunk` service: the container is recreated from the new image and new
migrations run at start-up. Check that `/health` reports the new version.
Rolling back means deploying the previous tag. Migrations are not reversed
automatically, so a release with a migration needs a database backup first
@@ -105,6 +102,7 @@ persistent is there; the image itself holds no state.
| `media/YYYY/MM/` | Uploaded images and their variants; `media/.tmp/` while processing |
| `logs/starpunk.log` | Application log, rotated at 10 MB, 10 files kept |
| `webmention-worker.lock`, `websub-worker.lock` | `flock` files choosing the process that runs each worker |
| `avatars/<sha256>.webp` | Saved copies of response authors' avatars (v2.13.0), served at `/avatars/`; unused ones removed weekly |
| `threads_token.json` (+ `.lock`) | Current Threads access token, refreshed daily (only with Threads configured) |
| `debug/` | Failed uploads, only with `DEBUG_SAVE_FAILED_UPLOADS=true`; cleaned up after `DEBUG_FILE_MAX_AGE_DAYS` |
@@ -127,6 +125,14 @@ WebSub rate limits need; without it, every client appears as the proxy's
address and shares one limit. Do not set it if the container port can be
reached without the proxy, because the header could then be forged.
The MCP sidecar (v2.15.0, ADR-075) is a second container from the same
image, with the same environment and `/data` volume plus
`BACKGROUND_WORKERS_ENABLED=false`, running
`uvicorn starpunk.mcp_server.app:app --host 0.0.0.0 --port 8001`. The proxy
sends `/mcp*` and `/.well-known/oauth-protected-resource*` to it, keeping the
`Host` header; everything else goes to port 8000. Without the sidecar the
site simply has no MCP endpoint. See [mcp.md](mcp.md).
`SITE_URL` must be the public `https://` URL: it is used for absolute links,
feed and WebSub topic URLs, the IndieAuth issuer and redirect URIs, and to
decide whether the session cookie is `Secure`.
@@ -161,6 +167,7 @@ unless noted.
| `INDIEAUTH_TOKEN_LIFETIME_DAYS` | `90` | Lifetime of Micropub access tokens |
| `DEV_MODE` | `false` | Enables `/dev/login`; never in production |
| `DEV_ADMIN_ME` | empty | Required when `DEV_MODE=true` |
| `CSRF_ENABLED` | `true` | CSRF tokens on admin forms, logout and IndieAuth consent; leave on |
`TOKEN_ENDPOINT` and `INDIELOGIN_URL` are v1 settings; if present they are
ignored with a warning.
@@ -176,6 +183,7 @@ ignored with a warning.
| `FLASK_DEBUG` | `0` | Debug off unless `1` (`flask run --debug` sets it locally); the image sets `0` |
| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| `VERSION` | package `__version__` | Overrides the reported version; normally unset |
| `BACKGROUND_WORKERS_ENABLED` | `true` | `false` stops the webmention, WebSub and syndication workers; set in the MCP sidecar (v2.15.0) |
### Feeds
@@ -190,8 +198,8 @@ ignored with a warning.
| Variable | Default | Notes |
|----------|---------|-------|
| `WEBMENTION_ENABLED` | `true` | Endpoint, `Link` header and sending |
| `WEBMENTION_WORKER_ENABLED` | `true` | |
| `WEBMENTION_ENABLED` | `true` | Endpoint, `Link` header, showing responses on notes (not sending) |
| `WEBMENTION_WORKER_ENABLED` | `true` | Verifies received and sends outgoing webmentions; saves avatars; sends response notifications |
| `WEBMENTION_POLL_INTERVAL` | `10` | Seconds |
| `WEBMENTION_ERROR_BACKOFF` | `60` | Seconds after a worker error |
| `WEBMENTION_BATCH_SIZE` | `5` | Items per pass |
@@ -219,7 +227,7 @@ starts only if at least one is.
| Variable | Default | Notes |
|----------|---------|-------|
| `GOTOSOCIAL_URL`, `GOTOSOCIAL_ACCESS_TOKEN` | empty | GoToSocial or any Mastodon-API server |
| `GOTOSOCIAL_URL`, `GOTOSOCIAL_ACCESS_TOKEN` | empty | GoToSocial or any Mastodon-API server. Token scopes: `write:statuses write:media read:search write:favourites`, plus `read:statuses read:accounts` (or `read`) for backfeed |
| `GOTOSOCIAL_NAME` | `GoToSocial` | Label in the UI and `syndicate-to` |
| `GOTOSOCIAL_VISIBILITY` | `public` | |
| `GOTOSOCIAL_MAX_CHARS` | `5000` | |
@@ -228,12 +236,22 @@ starts only if at least one is.
| `BLUESKY_SERVICE` | `https://bsky.social` | |
| `BLUESKY_NAME` | `Bluesky` | |
| `BLUESKY_MAX_CHARS` | `300` | |
| `THREADS_ACCESS_TOKEN` | empty | Long-lived token; refreshed automatically into `threads_token.json` |
| `THREADS_ACCESS_TOKEN` | empty | Long-lived token; refreshed automatically into `threads_token.json`. Backfeed also needs `threads_read_replies` and `threads_manage_insights` |
| `THREADS_USERNAME` | empty | |
| `THREADS_NAME` | `Threads` | |
| `THREADS_MAX_CHARS` | `500` | |
| `THREADS_MAX_MEDIA` | `4` | |
| `SYNDICATION_POLL_INTERVAL` | `10` | Seconds |
| `BACKFEED_ENABLED` | `false` | Bring responses to crossposts back to notes (v2.13.0); needs read scopes on the GoToSocial and Threads tokens |
| `BACKFEED_MAX_AGE_DAYS` | `90` | Days after posting to keep checking a crosspost's responses |
### Notifications
| Variable | Default | Notes |
|----------|---------|-------|
| `NOTIFY_URLS` | empty (off) | Apprise URLs, space- or comma-separated (v2.14.0) |
| `NOTIFY_EVENTS` | `reply,mention,quote,problems` | Also `like`, `repost`, `bookmark` |
| `NOTIFY_PRIORITY` | `normal` | Added to URLs that don't set a priority |
### Debugging
@@ -252,15 +270,14 @@ from the environment.
## Running locally
```bash
uv venv && uv pip install -r requirements-dev.txt
uv venv --python 3.14 && uv pip install -r requirements-dev.txt
cp .env.example .env # set SESSION_SECRET; DEV_MODE=true and DEV_ADMIN_ME for local login
flask --app app.py run # or: gunicorn app:app
uv run flask --app app.py run --debug # or: uv run gunicorn app:app
```
`compose.yaml` builds and runs the image with `./container-data` mounted at
`/data`; its image tag and `VERSION` values are out of date, so override them
or build with `podman build -t starpunk .`.
`compose.yaml` builds the image from this checkout (`starpunk:local`) and runs
it with `./container-data` mounted at `/data`.
---
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
+39 -14
View File
@@ -1,6 +1,6 @@
# IndieWeb and W3C Protocols
How StarPunk implements each protocol it speaks, as of v2.10.1, with its
How StarPunk implements each protocol it speaks, as of v2.14.0, with its
compliance status and the gaps that remain. Each section links to the
decision records and review documents behind it.
@@ -12,6 +12,8 @@ decision records and review documents behind it.
| [Microformats2](#microformats2) | Publisher and consumer | Reviewed in v2.8.0 | `templates/`, `starpunk/mf2_utils.py`, `starpunk/author_discovery.py` |
| [WebSub](#websub) | Publisher and built-in hub (own topics only) | Adversarial review in v2.10.1 | `starpunk/websub.py`, `starpunk/websub_worker.py`, `starpunk/routes/websub.py` |
| [Feeds](#feeds) | RSS 2.0, Atom 1.0, JSON Feed 1.1, OPML 2.0 | Stable | `starpunk/feeds/`, `starpunk/routes/public.py` |
| Backfeed | Responses to crossposts read from the silo APIs and stored as webmentions (v2.13.0); see [syndication-architecture.md](syndication-architecture.md#backfeed-v2130-adr-073) | ADR-073 | `starpunk/backfeed.py` |
| [Web actions](#moderation-and-display) | `<indie-action>` on the like, repost and reply counts (v2.13.2) | Markup only | `templates/partials/responses.html` |
Crossposting to other networks (POSSE) is covered in
[syndication-architecture.md](syndication-architecture.md).
@@ -19,8 +21,10 @@ Crossposting to other networks (POSSE) is covered in
## Shared infrastructure: the SSRF-safe fetcher
Webmention (fetching sources, discovering endpoints, sending), WebSub
(verification and delivery) and Micropub (downloading remote photos) all
make requests to URLs chosen by someone else. They use
(verification and delivery), Micropub (downloading remote photos), IndieAuth
(client pages), reply contexts, crosspost link cards (v2.11.0) and response
avatars (v2.13.0; except those on StarPunk's own GoToSocial) all make
requests to URLs chosen by someone else. They use
`starpunk/webmention_ssrf.py`:
- `validate_url_for_fetch()`: http(s) only. Every address from
@@ -128,7 +132,7 @@ repost, bookmark.
| `q=syndicate-to` | any token | `{"syndicate-to": [{"uid", "name"}]}` |
| `q=source&url=...[&properties[]=...]` | create/update/delete/draft | One post as mf2 JSON. Content comes back as `{"html": ...}` if it was authored as HTML. Includes `syndication` URLs and `post-status` |
| `q=source` without `url` | same | `{"items": [...]}`, drafts included, `limit` (default 10, max 50) and `offset` |
| `q=category[&filter=...]` | same | Tags in use, most used first |
| `q=category[&filter=...]` | same | Tags on notes that aren't deleted (drafts count), most used first (v2.12.1) |
Unknown or missing `q` gets 400.
@@ -171,8 +175,7 @@ Full description, scopes and gaps:
Decision: [ADR-064](../decisions/ADR-064-oidc-admin-login-and-builtin-indieauth-server.md).
Known gaps: `scopes_supported` doesn't list `draft`. There are no
introspection, userinfo or refresh tokens. The client metadata fetch
doesn't use the pinned SSRF fetcher.
introspection, userinfo or refresh tokens.
---
@@ -239,25 +242,46 @@ Verification is asynchronous (`WebmentionWorker`, poll
- Valid: `validation_status = 'valid'`. A first-time mention gets
`moderation_status = 'approved'` if `WEBMENTION_AUTO_APPROVE` (default
true), otherwise `pending`. An update keeps the admin's earlier
moderation decision.
moderation decision, including `hidden` and `deleted`.
- Source returns 4xx (including 410), or no longer links: `invalid`,
which hides it (3.2.4 deletion).
- Network error, timeout or 5xx: a mention that was valid before stays
valid with its old data. A new one becomes `error`.
- A source that is one of the note's own syndicated copies
(`syndications.remote_url`) is `invalid` (v2.8.0).
- A source that backfeed has already stored for the note is `invalid`
("Already backfed from the silo", v2.13.0).
- **Re-send on display change** (3.1.4, v2.7.2): if what the note shows
for this webmention changed, the note's own outgoing webmentions are
re-queued. That covers a new response, a changed author or snippet, or a
change of shown/hidden. Unchanged re-validations send nothing, so two
sites doing the same don't ping-pong.
**Moderation and display.** `/admin/webmentions` lists, filters, and
approves, hides or deletes mentions, one at a time or in bulk. Changes that
alter what is shown also re-send the note's webmentions. Note pages show
approved, valid mentions grouped as replies (`u-comment h-cite`), likes,
reposts and bookmarks (facepiles), and mentions. See
[Microformats2](#microformats2).
### Moderation and display
`/admin/webmentions` lists, filters (by status, including Deleted, and by
origin), and approves, hides or deletes responses, one at a time or in bulk;
it also has the Backfill button. Deleting keeps the row as a `deleted`
tombstone, so a re-sent webmention or backfeed doesn't bring it back.
Changes that alter what is shown also re-send the note's webmentions.
Note pages show approved, valid responses. Under the note, a line of counts
(♥ likes, 🔁 reposts, 💬 replies, ❝ quotes (v2.14.2), bookmarks) opens a card on hover or focus
listing who responded and where from, including anonymous silo counts. Each
like, repost and reply count holds an `<indie-action do with>` web action
whose fallback links to the note's crossposts (v2.13.2). The Responses
section below holds replies (`u-comment h-cite`), quotes and mentions;
backfed ones say "via <silo>". See [Microformats2](#microformats2).
### Notifications (v2.14.0, ADR-074)
With `NOTIFY_URLS` set, new valid responses (received and backfed) are sent
to the owner once through Apprise by a sweep in the webmention worker, which
sets `webmentions.notified_at` (migration 021). Likes, reposts and bookmarks
are grouped per note; which kinds are sent is set by `NOTIFY_EVENTS`
(default replies, mentions, quotes and problems). Items still unsent after
24 hours are dropped. Responses found by a backfill, or by a copy's first
check on a note older than a day, are never notified.
### Sending
@@ -339,8 +363,9 @@ representative h-card, authorship, Post Type Discovery. Decisions:
| `/note/<slug>` | `h-entry`: `p-name` only when the note starts with a `# Heading` (so plain notes have no implied name); `u-photo` as direct children above `e-content`; `u-url u-uid` with `dt-published` (RFC 3339 UTC `Z`); `dt-updated`; `p-category` with `rel=tag`; `u-syndication` links; hidden `p-author h-card` |
| Replies | `u-in-reply-to h-cite` holding the cached reply context (`p-content`, `u-url`) |
| Likes / reposts / bookmarks | `u-like-of` / `u-repost-of` / `u-bookmark-of` link with "★ Liked" / "🔁 Reposted" / "🔖 Bookmarked" |
| Received responses | `u-comment h-cite` (replies: `p-author h-card` with `u-url` on an inner link, `u-photo`, `p-name`; `e-content`; `u-url`; `dt-published`), and `u-like`, `u-repost`, `u-bookmark`, `u-mention` h-cites. Faces beyond 8 are `hidden`, not omitted |
| Received responses | `u-comment h-cite` (replies: `p-author h-card` with `u-url` on an inner link, `u-photo`, `p-name`; `e-content`; `u-url`; `dt-published`), likes, reposts and bookmarks as `u-like`, `u-repost`, `u-bookmark` h-cites in the counts line card; mentions and quotes as `u-mention` h-cites |
| `/likes`, `/tag/<tag>` | `h-feed` with `p-name`, `u-url`, `p-author h-card` |
| Response counts | `/`, `/tag/<tag>` and `/likes` show each note's like, repost and reply counts, linking to its `#responses` (v2.13.4) |
Post Type Discovery gives note, article, reply, like, repost, bookmark and
photo correctly.
+77
View File
@@ -0,0 +1,77 @@
# MCP server (v2.15.0)
How AI clients use StarPunk through the
[Model Context Protocol](https://modelcontextprotocol.io/). The decision and
alternatives are in [ADR-075](../decisions/ADR-075-mcp-server.md).
## Shape
```
client ──HTTPS──> reverse proxy
/mcp*, /.well-known/oauth-protected-resource* ──> starpunk-mcp:8001 (uvicorn, MCP SDK)
everything else (incl. /auth/*, /.well-known/oauth-authorization-server)
──> starpunk:8000 (gunicorn, Flask)
starpunk and starpunk-mcp: same image, same environment, same /data volume
```
| Module | Role |
|---|---|
| `starpunk/mcp_server/app.py` | ASGI entry: `uvicorn starpunk.mcp_server.app:app --host 0.0.0.0 --port 8001` |
| `starpunk/mcp_server/server.py` | `create_mcp_server(app)` and `create_asgi_app(app=None)`: stateless Streamable HTTP, JSON responses, 16 MB request limit, DNS rebinding protection |
| `starpunk/mcp_server/auth.py` | `StarPunkTokenVerifier`: StarPunk tokens to SDK `AccessToken`s |
| `starpunk/mcp_server/tools.py` | The tools; each runs in a thread inside `app.test_request_context(base_url=SITE_URL)` |
`create_asgi_app()` with no argument builds the Flask app from the
environment with `BACKGROUND_WORKERS_ENABLED=false`. The sidecar container
should also set that variable, for clarity.
## Authorization flow
1. The client POSTs to `/mcp` without a token and gets `401` with
`WWW-Authenticate: Bearer ... resource_metadata="SITE_URL/.well-known/oauth-protected-resource/mcp"`.
2. That document names `SITE_URL/mcp` as the resource and `SITE_URL/` as
the authorization server. The client reads
`/.well-known/oauth-authorization-server`, which says it accepts client ID
metadata documents and public clients.
3. The client sends the owner to `/auth/authorize` with its `client_id` URL,
PKCE and `resource=SITE_URL/mcp`. StarPunk fetches the client's metadata
document to check `redirect_uri` (unless it has the same origin as the
`client_id`), and the owner logs in with OIDC and approves. With no
`scope`, the default MCP scopes are offered.
4. The token endpoint exchanges the code. The token is bound to the
resource.
5. Each MCP request carries the token. The verifier looks it up and the SDK
refuses it unless its resource is `SITE_URL/mcp`.
Details of codes, tokens and scopes are in
[simplified-auth-architecture.md](simplified-auth-architecture.md).
## Tools
| Tool | Does | Scope |
|---|---|---|
| `search_notes(query, include_drafts, limit)` | FTS5 search, with snippets | any; drafts need a posting scope |
| `list_notes(include_drafts, limit, offset)` | Newest first, summaries | any; drafts need a posting scope |
| `get_note(url_or_slug)` | Microformats2 properties (as Micropub `q=source`) | any; drafts need a posting scope |
| `list_tags(filter)` | Tags in use, most used first | posting scope |
| `get_site_config()` | Micropub `q=config`, including `syndicate-to` targets | any |
| `create_note(content, title, tags, draft=True, photos, in_reply_to, like_of, repost_of, bookmark_of, syndicate_to, slug, published_at)` | Micropub create | `create` or `draft` |
| `update_note(url, content, title, tags, add_tags, remove_tags, draft, photos, in_reply_to)` | Micropub update | `update` |
| `delete_note(url)`, `undelete_note(url)` | Soft delete and restore | `delete` |
| `upload_media(filename, data_base64)` | Save an image, return its URL | `media` or `create` |
"Posting scope" means `create`, `update`, `delete` or `draft`. Limits are
capped at 100. Micropub errors come back as tool errors (`isError`) with the
Micropub error description.
## Operating it
- Environment: the same as the main container, plus
`BACKGROUND_WORKERS_ENABLED=false`.
- Proxy: route `/mcp*` and `/.well-known/oauth-protected-resource*` to port
8001, keeping the `Host` header (DNS rebinding protection checks it
against `SITE_URL`).
- Connecting: claude.ai Settings → Connectors → Add custom connector with
`SITE_URL/mcp`; Claude Code: `claude mcp add --transport http starpunk
SITE_URL/mcp`, then `/mcp` to authenticate.
- Revoking: Admin → Connected Apps (MCP tokens are marked "(MCP)").
+66 -28
View File
@@ -1,6 +1,6 @@
# StarPunk Architecture Overview
**Version**: 2.10.1 (2026-11-03)
**Version**: 2.14.0 (2026-11-10)
**Status**: Current. Describes the system as it is in the code today.
## Purpose
@@ -10,7 +10,9 @@ owner) writes short notes, which are published as microformats2-marked HTML
pages and as RSS, Atom and JSON feeds. The site takes part in the IndieWeb:
it accepts posts from Micropub clients, issues its own IndieAuth tokens for
them, sends and receives Webmentions, pushes feed updates with WebSub, and
crossposts (POSSE) to GoToSocial, Bluesky and Threads.
crossposts (POSSE) to GoToSocial, Bluesky and Threads. Responses to those
crossposts are pulled back and shown on the note (backfeed), and the owner
can be notified of new responses.
Guiding principle: "every line of code must justify its existence". It is a
single Flask application with server-side rendering, SQLite and plain files.
@@ -39,6 +41,9 @@ Key decisions:
[ADR-067](../decisions/ADR-067-micropub-update-delete.md) (Micropub update/delete),
[ADR-068](../decisions/ADR-068-likes-and-reposts.md) (likes and reposts),
[ADR-071](../decisions/ADR-071-websub.md) (WebSub),
[ADR-072](../decisions/ADR-072-remove-metrics-dashboard.md) (metrics dashboard removed),
[ADR-073](../decisions/ADR-073-backfeed.md) (backfeed),
[ADR-074](../decisions/ADR-074-notifications.md) (notifications),
[ADR-057](../decisions/ADR-057-media-attachment-model.md) and
[ADR-058](../decisions/ADR-058-image-optimization-strategy.md) (media),
[ADR-034](../decisions/ADR-034-full-text-search.md) (search),
@@ -70,8 +75,9 @@ Key decisions:
│ starpunk.db │ │ OIDC provider, IndieAuth │
│ notes/*.md │ │ client pages, webmention │
│ media/ │ │ sources/targets, WebSub │
│ logs/, locks │ │ callbacks, GoToSocial, │
└────────────────┘ │ Bluesky, Threads │
│ avatars/ │ │ callbacks, GoToSocial, │
│ logs/, locks │ │ Bluesky, Threads, avatars, │
└────────────────┘ │ notifications (Apprise) │
└─────────────────────────────┘
```
@@ -127,12 +133,16 @@ wait for outbound network calls; the workers make them later.
| `reply_context.py` | Fetches and caches the post a reply responds to |
| `webmention_sender.py` | Extracts links from rendered notes and queues outgoing webmentions |
| `webmention_processor.py` | Validates incoming webmentions: fetch source, parse mf2, classify, sanitise content |
| `webmention_worker.py` | Background thread working the incoming and outgoing webmention queues |
| `webmention_worker.py` | Background thread working the incoming and outgoing webmention queues; also runs avatar upkeep and the notification sweep |
| `webmention_queries.py` | Webmention queries for display and moderation |
| `webmention_ssrf.py` | SSRF-safe fetcher: public-address checks, pinned connections, manual redirects, size and time limits |
| `websub.py` | WebSub publisher (discovery headers and links, `content_version`, queueing) and hub request handling |
| `websub_worker.py` | Background thread: verification of intent, content delivery, publish pings to an external hub |
| `syndication.py` | POSSE clients (Mastodon API/GoToSocial, Bluesky, Threads), reconcile logic, syndication worker |
| `syndication.py` | POSSE clients (Mastodon API/GoToSocial, Bluesky, Threads), reconcile logic, syndication worker (its loop also calls `backfeed.process_due()`) |
| `backfeed.py` | Backfeed: reads responses to crossposts from the silos, stores them as webmentions, schedules checks and backfills |
| `avatars.py` | Saves response authors' avatars as 96 px WebP files in `avatars/`; fills missing ones and removes unused ones |
| `notifications.py` | Notifications through Apprise: the sweep of new responses and immediate problem messages |
| `csrf.py` | CSRF tokens (HMAC of the session cookie) for admin forms, logout and the IndieAuth consent |
| `rate_limiter.py` | In-memory sliding-window rate limiter |
| `errors.py` | Central error handlers |
| `feeds/` | `rss.py` (feedgen), `atom.py`, `json_feed.py`, `opml.py`, `negotiation.py` (Accept header), `cache.py` (LRU + TTL feed cache with ETags) |
@@ -144,7 +154,7 @@ wait for outbound network calls; the workers make them later.
| Blueprint | File | Paths |
|-----------|------|-------|
| `public` | `public.py` | `/`, `/note/<slug>`, `/tag/<tag>`, `/likes`, `/media/<path>`, feeds, `/opml.xml` |
| `public` | `public.py` | `/`, `/note/<slug>`, `/tag/<tag>`, `/likes`, `/media/<path>`, `/avatars/<name>`, feeds, `/opml.xml` |
| `auth` | `auth.py` | `/auth/login` (GET, POST), `/auth/callback`, `/auth/logout` (POST) |
| `indieauth` | `indieauth.py` | `/.well-known/oauth-authorization-server`, `/auth/authorize` (GET, POST), `/auth/authorize/approve`, `/auth/token` (GET, POST), `/auth/revoke` |
| `micropub` | `micropub.py` | `/micropub` (GET, POST), `/micropub/media` (POST), `/media` (POST, legacy alias) |
@@ -163,20 +173,20 @@ Jinja2 templates live in `templates/`: public pages, `admin/`, `partials/`
reposts and bookmarks) and one page per HTTP error. `templates/base.html`
carries the discovery links (`rel=micropub`, `indieauth-metadata`,
`authorization_endpoint`, `token_endpoint`, `webmention`, `hub`/`self`, `me`,
feeds). Static assets are one stylesheet (`static/css/style.css`) and one
script (`static/js/preview.js`, the Markdown preview in the admin editor).
Everything works without JavaScript.
feeds). Static assets are one stylesheet (`static/css/style.css`); admin
pages have small inline scripts. Everything works without JavaScript.
## Public URL surface
| Path | Purpose |
|------|---------|
| `/` | Home page: recent published notes (h-feed); likes are kept off it |
| `/note/<slug>` | Note permalink (h-entry), with approved webmentions and reply context |
| `/note/<slug>` | Note permalink (h-entry), with approved responses (webmentions and backfed silo responses), a counts line with web actions, and reply context |
| `/tag/<tag>` | Notes with a tag |
| `/likes` | Likes |
| `/search`, `/api/search` | Full-text search page and JSON API |
| `/media/<path>` | Uploaded images and their variants |
| `/avatars/<name>` | Saved copies of response authors' avatars |
| `/feed` | Feed chosen by the `Accept` header (RSS by default; `Vary: Accept`) |
| `/feed.rss`, `/feed.atom`, `/feed.json` | RSS 2.0, Atom 1.0, JSON Feed 1.1 |
| `/feed.xml` | Legacy alias of `/feed.rss` |
@@ -191,8 +201,15 @@ Everything works without JavaScript.
| `/admin/webmentions`, `/admin/webmentions/<id>`, `…/action`, `…/bulk-action` | Webmention moderation |
| `/admin/tokens`, `/admin/tokens/<id>/revoke` | Issued IndieAuth tokens |
| `/admin/syndication/<id>/retry` | Retry a failed crosspost |
| `/admin/syndication/<id>/check` (POST) | Check one crosspost for responses now |
| `/admin/backfeed/backfill` (POST) | Check every crosspost for responses once (backfill) |
| `/admin/notifications/test` (POST) | Send a test notification |
| `/admin/health` | Full diagnostics (JSON): database, pool statistics, filesystem, disk |
| `/health` | Public liveness check (`{"status": "ok", "version": …}`); `?detailed=true` requires login |
| `/dev/login` | Development login; only when `DEV_MODE=true` |
`/`, `/tag/<tag>` and `/likes` show each note's response counts, linking to
its responses (v2.13.4).
## Background workers
@@ -202,9 +219,12 @@ coordinated through the data volume:
| Thread | Started when | Coordination | Work |
|--------|--------------|--------------|------|
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries |
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries; saves missing response avatars every 10 minutes and removes unused ones weekly (v2.13.0); sweeps new responses into notifications and sets `webmentions.notified_at` (v2.14.0, `notifications.py`, ADR-074) |
| WebSub worker (`websub_worker.py`) | `WEBSUB_ENABLED` and not testing | Same pattern with `websub-worker.lock` | Verifies subscription intents, delivers topic content (signed with `X-Hub-Signature` when the subscriber gave a secret), retries, expires leases; pings an external hub instead if `WEBSUB_HUB_URL` is set |
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts |
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts. Then, only when `BACKFEED_ENABLED`, backfeed: checks copies for responses, each claimed with a `backfeed_claimed_until` lease (v2.13.0, `backfeed.py`, ADR-073) |
`BACKGROUND_WORKERS_ENABLED=false` starts none of them; the MCP sidecar
(below) sets it so the workers only run in the main container.
All workers run inside an app context, are daemon threads (they end with the
process) and sleep for their poll interval when there is nothing to do. Under
@@ -225,9 +245,12 @@ Content is kept in files and everything else in SQLite, both under
- **Media**: `media/YYYY/MM/<uuid>.<ext>` plus `<uuid>_<variant>.<ext>`
variants, staged in `media/.tmp/` while being processed.
- **Database**: `starpunk.db` (WAL mode): note metadata, tags, media records,
sessions, IndieAuth codes and tokens, webmentions and the outgoing queue,
syndications, WebSub subscriptions and queues, caches (author h-card, reply
sessions, IndieAuth codes and tokens, webmentions (including backfed silo
responses, marked by `origin` and `remote_id`) and the outgoing queue,
syndications (with backfeed state and silo-only counts), WebSub subscriptions and queues, caches (author h-card, reply
contexts), the FTS5 index and `site_state`. See [database.md](database.md).
- **Avatars**: `avatars/<sha256>.webp`, saved copies of response authors'
avatars (v2.13.0); they can be re-fetched, so losing them is harmless.
- **Other files**: `logs/starpunk.log` (rotating), worker lock files,
`threads_token.json`, and `debug/` for failed uploads when
`DEBUG_SAVE_FAILED_UPLOADS=true`.
@@ -253,7 +276,14 @@ are in [simplified-auth-architecture.md](simplified-auth-architecture.md).
OIDC) approves; the client exchanges the code at `/auth/token` (PKCE
required). Codes and tokens are stored hashed. Tokens last
`INDIEAUTH_TOKEN_LIFETIME_DAYS` and can be revoked at `/auth/revoke` or in
`/admin/tokens`. Scopes: `create`, `update`, `delete`, `media`, `profile`.
`/admin/tokens`. Scopes: `create`, `update`, `delete`, `media`, `draft`,
`profile`.
- **MCP (v2.15.0, ADR-075)**: AI clients use the same flow, identified by a
client ID metadata document and asking for `resource=SITE_URL/mcp`. Their
tokens are bound to that resource and checked by the MCP sidecar, a second
container (uvicorn, official MCP SDK) that the proxy sends `/mcp` to. Its
tools read, search and write notes through the Micropub handlers. See
[mcp.md](mcp.md).
- **Development**: `DEV_MODE=true` adds `/dev/login`, which logs in as
`DEV_ADMIN_ME` without any checks. Never enabled in production; start-up
logs a warning when it is on.
@@ -279,20 +309,28 @@ their credentials are set. The full list with defaults is in
- **Author h-card** (`author_profile`, 24 hours) and **reply contexts**
(`reply_contexts`) are cached in the database.
- The webmention worker caches endpoint discovery results in memory.
- **Response avatars** are saved once as files named by the SHA-256 of the
source URL and served from `/avatars/<name>` with
`Cache-Control: public, max-age=604800` (one week).
## Security posture
- **Outbound requests (SSRF)**: fetches of URLs that come from outside
(webmention sources and targets, endpoint discovery, reply contexts, WebSub
callbacks, remote Micropub photos, IndieAuth client pages, crosspost link
cards) are checked first: every address the host resolves to must be public
(private, loopback, link-local, CGNAT, multicast and reserved ranges are
refused). Every fetch of a user-supplied URL (webmentions, reply contexts,
WebSub, remote Micropub photos, IndieAuth client pages and, since v2.11.0,
crosspost link cards) uses the fetcher in `webmention_ssrf.py`: it follows
redirects by hand and re-checks each hop, connects to the address it
checked (no second DNS lookup) and limits response size (1 MB by default)
and total time. `DEV_MODE` relaxes the checks for local testing.
- **Outbound requests (SSRF)**: every fetch of a URL that comes from
outside (webmention sources and targets, endpoint discovery, reply
contexts, WebSub callbacks, remote Micropub photos, IndieAuth client pages,
crosspost link cards since v2.11.0, response avatars since v2.13.0) goes
through the fetcher in `webmention_ssrf.py`. Every address the host
resolves to must be public (private, loopback, link-local, CGNAT,
multicast and reserved ranges are refused); it follows redirects by hand
and re-checks each hop, connects to the address it checked (no second DNS
lookup) and limits response size (1 MB by default) and total time.
`DEV_MODE` relaxes the checks for local testing. Exceptions: avatars on
the site's own `GOTOSOCIAL_URL` host are fetched directly, as the
crossposting client does, since behind the same proxy it may resolve to an
internal address (v2.13.1); Bluesky CDN avatars are fetched from the
account's PDS with `com.atproto.sync.getBlob`, itself through the safe
fetcher. Apprise notifications go to the owner-configured `NOTIFY_URLS`,
outside the safe fetcher.
- **HTML sanitising**: rendered notes (Markdown or HTML sent by a Micropub
client) and webmention content are cleaned with nh3 allow-lists; search
highlighting escapes everything except `<mark>`.
@@ -327,4 +365,4 @@ their credentials are set. The full list with defaults is in
---
**Last Updated**: 2026-11-03 (v2.12.0)
**Last Updated**: 2026-11-10 (v2.14.0)
@@ -1,6 +1,6 @@
# Authentication and Authorization Architecture
Current as of v2.10.1. Decision record: [ADR-064](../decisions/ADR-064-oidc-admin-login-and-builtin-indieauth-server.md),
Current as of v2.14.0 (no auth changes since v2.11.0). Decision record: [ADR-064](../decisions/ADR-064-oidc-admin-login-and-builtin-indieauth-server.md),
which supersedes the "no self-hosted IndieAuth" decisions
([ADR-050](../decisions/ADR-050-remove-custom-indieauth-server.md),
[ADR-056](../decisions/ADR-056-no-selfhosted-indieauth.md)) and the external
@@ -137,8 +137,10 @@ form `{"error", "error_description"}`.
"token_endpoint": "<SITE_URL>/auth/token",
"revocation_endpoint": "<SITE_URL>/auth/revoke",
"revocation_endpoint_auth_methods_supported": ["none"],
"scopes_supported": ["create", "update", "delete", "media", "profile"],
"scopes_supported": ["create", "update", "delete", "media", "draft", "profile"],
"response_types_supported": ["code"],
"token_endpoint_auth_methods_supported": ["none"],
"client_id_metadata_document_supported": true,
"grant_types_supported": ["authorization_code"],
"code_challenge_methods_supported": ["S256"],
"authorization_response_iss_parameter_supported": true
@@ -192,7 +194,7 @@ requested and left checked, issues a code, and redirects with `code`,
| Value | 32 random bytes, URL-safe | 32 random bytes, URL-safe |
| Stored as | SHA-256 hash | SHA-256 hash |
| Lifetime | 10 minutes | `INDIEAUTH_TOKEN_LIFETIME_DAYS` (default 90) |
| Bound to | `client_id`, `redirect_uri`, `scope`, `code_challenge` | `client_id`, `scope` |
| Bound to | `client_id`, `redirect_uri`, `scope`, `code_challenge`, `resource` | `client_id`, `scope`, `resource` |
`redeem_authorization_code` handles redemption for both `POST /auth/token`
and `POST /auth/authorize`:
@@ -213,11 +215,22 @@ refresh tokens: a client re-authorizes when its token expires.
Expired codes are deleted whenever a new code is issued.
**Resource indicators (v2.15.0, ADR-075).** An authorization request may
include `resource` (RFC 8707). The only accepted value is `SITE_URL/mcp`
(a trailing slash is ignored); anything else is `invalid_target`. With it,
a request without `scope` is offered `create update delete media draft`, and
the consent screen says the client is an AI assistant using MCP. The resource
is stored on the code and then on the token. A token request that names a
different resource gets `invalid_target`. Micropub clients send no resource
and their tokens have none. The MCP endpoint only accepts tokens for its
resource; see [mcp.md](mcp.md).
### Verification and revocation
- `verify_access_token(token)` hashes the token and accepts it if it is
neither revoked nor expired, then updates `last_used_at`. It returns
`{me: ADMIN_ME, client_id, scope}`.
`{me: ADMIN_ME, client_id, scope, resource, expires_at}`. The legacy
`GET /auth/token` still answers with only `me`, `client_id` and `scope`.
- `check_scope(required, scope)` matches whole words in the space-separated
scope string.
- Revocation sets `revoked_at`. It can happen by token value, through
@@ -308,8 +321,8 @@ enough. If it is another page, that page must advertise StarPunk's
|---|---|---|
| `sessions` | Admin browser sessions | `session_token_hash`, `me`, `expires_at`, `last_used_at`, `user_agent`, `ip_address` |
| `oidc_login_state` | One pending OIDC login | `state` (PK), `nonce`, `code_verifier`, `next_url`, `expires_at` |
| `indieauth_codes` | Authorization codes | `code_hash`, `client_id`, `redirect_uri`, `scope`, `code_challenge`, `expires_at`, `used_at` |
| `indieauth_tokens` | Access tokens | `token_hash`, `client_id`, `scope`, `expires_at`, `last_used_at`, `revoked_at` |
| `indieauth_codes` | Authorization codes | `code_hash`, `client_id`, `redirect_uri`, `scope`, `code_challenge`, `resource` (migration 022), `expires_at`, `used_at` |
| `indieauth_tokens` | Access tokens | `token_hash`, `client_id`, `scope`, `resource` (migration 022), `expires_at`, `last_used_at`, `revoked_at` |
The legacy v1 tables `auth_state`, `tokens` and `authorization_codes` are
not used (see the migration's header comment).
@@ -338,7 +351,6 @@ connection pinned, a 5-second limit).
Known gaps:
- `scopes_supported` leaves out `draft`, although Micropub honours it and
the consent screen grants it when requested. Unknown scopes are also shown
and granted as requested; they just have no effect.
- Unknown scopes are shown and granted as requested; they just have no
effect. (`draft` has been listed in `scopes_supported` since v2.15.0.)
- No token introspection endpoint and no refresh tokens.
+26 -10
View File
@@ -1,6 +1,6 @@
# Syndication (POSSE) Architecture
Current as of v2.10.1. Code: `starpunk/syndication.py`.
Current as of v2.14.0. Code: `starpunk/syndication.py`, `starpunk/backfeed.py`.
StarPunk publishes on its own site first and then copies each note to other
networks (POSSE: Publish on your Own Site, Syndicate Elsewhere). Later
@@ -16,7 +16,9 @@ Decisions: [ADR-065](../decisions/ADR-065-direct-syndication-posse.md)
(Bluesky), [ADR-068](../decisions/ADR-068-likes-and-reposts.md) (likes and
reposts), [ADR-069](../decisions/ADR-069-threads-syndication.md) (Threads),
[ADR-070](../decisions/ADR-070-crosspost-links-and-cards.md) (links and link
cards), and the [v2.9.0 Micropub review](../design/v2.9.0/2026-11-03-micropub-review.md)
cards), [ADR-073](../decisions/ADR-073-backfeed.md) (backfeed),
[ADR-074](../decisions/ADR-074-notifications.md) (notifications, including
crossposting problems), and the [v2.9.0 Micropub review](../design/v2.9.0/2026-11-03-micropub-review.md)
(bookmarks, `mp-syndicate-to` default).
## Targets
@@ -34,7 +36,10 @@ in the process.
Display names come from `GOTOSOCIAL_NAME`, `BLUESKY_NAME` and
`THREADS_NAME`. GoToSocial statuses use `GOTOSOCIAL_VISIBILITY` (default
`public`). The GoToSocial token needs `write:statuses` and `write:media`,
plus `read:search` and `write:favourites` for likes and boosts.
plus `read:search` and `write:favourites` for likes and boosts, and
`read:statuses` and `read:accounts` (or `read`) for backfeed. Backfeed on
Threads needs `threads_read_replies` and `threads_manage_insights`. The
Threads client uses the API host `graph.threads.com` (v2.13.0).
Bluesky and Threads can't edit posts through their APIs. Once a copy exists
there, edits to the note don't change it, but unpublish and delete still
@@ -59,6 +64,9 @@ is sent until the note is published.
## Data: the `syndications` table (migration 012)
Migration 020 adds the backfeed columns; see [Backfeed](#backfeed-v2130-adr-073)
and [database.md](database.md#crossposting).
One row per (note, target), unique on `(note_id, target)`:
| Column | Meaning |
@@ -83,6 +91,8 @@ delete and the worker can still remove the copy.
| Micropub photo changes | `note_changed(id)` | Same |
| `delete_note()` (soft or hard), `undelete_note()` | `note_changed(id)` | Same |
| Retry button on the edit page | `retry(id)` | That row set to `pending` now, attempts reset |
| Check now (`POST /admin/syndication/<id>/check`) | `backfeed.check_now(id)` | That copy's responses checked at the next worker pass |
| Backfill responses (`POST /admin/backfeed/backfill`) | `backfeed.start_backfill()` | Every eligible copy flagged for one check |
## Worker
@@ -130,7 +140,8 @@ again with the new content. `_fail()` does the same.
errors (codes 10, 190, 200).
- Other errors retry after `min(30 s × 2^attempts, 6 h)`. After 8 attempts
the row becomes `failed`. The error is shown on the edit page with a
Retry button.
Retry button. When a row finally fails, the owner also gets a `problems`
notification (ADR-074).
- `RemoteGone` (404, or Graph error 100/33) during an update means the copy
was deleted remotely, so a new one is created. During a delete it counts as
success.
@@ -227,7 +238,7 @@ to itself (v2.8.0).
`starpunk/backfeed.py` brings responses to the copies back to the note.
After each crosspost pass, the worker loop calls `backfeed.process_due()`,
which checks up to three synced copies of published notes (not likes or
which does nothing unless `BACKFEED_ENABLED`; otherwise it checks up to three synced copies of published notes (not likes or
reposts) that are due, each under a lease (`backfeed_claimed_until`):
- **Reading.** Each client has `responses(remote_id, remote_url)`, which
@@ -244,10 +255,14 @@ reposts) that are due, each under a lease (`backfeed_claimed_until`):
`WEBMENTION_AUTO_APPROVE` default; an existing row keeps its moderation.
A complete harvest marks that silo's rows for the note that weren't
found `invalid` ("No longer on <silo>"). Authors with `#nobridge` or
`#nobot` in their bio, or Bluesky's `!no-unauthenticated` label, are
skipped and their rows deleted. If the note's shown responses changed,
its webmentions are re-sent (spec 3.1.4). Threads counts go in
`syndications.likes_count`, `reposts_count`, `quotes_count`.
`#nobot` in their bio, or Bluesky's `!no-unauthenticated` label, are not
stored and their rows are deleted; since v2.13.3 their likes, reposts and
quotes are counted anonymously instead (`hidden_counts()`). If the note's
shown responses changed, its webmentions are re-sent (spec 3.1.4).
`syndications.likes_count`, `reposts_count` and `quotes_count` hold the
silo-only counts: all of Threads; elsewhere people who opted out. New
responses found by a backfill, or by a copy's first check on a note older
than a day, are stored as already notified (v2.14.0).
- **Schedule.** `backfeed_next_at` follows the note's age: 15 minutes (day
1), 1 hour (week 1), 1 day (to 30 days), 1 week (to
`BACKFEED_MAX_AGE_DAYS`), with up to 10% jitter; none after. Failures back
@@ -259,7 +274,8 @@ reposts) that are due, each under a lease (`backfeed_claimed_until`):
`site_state`.
- **Refused credentials.** A 401/403 (or Graph auth error) puts the target
in `backfeed.denied` until restart; the admin Webmentions page says which
scopes are needed. Crossposting is unaffected.
scopes are needed, and a `problems` notification is sent. Crossposting
is unaffected.
Avatars of all responses (received and backfed) are saved by
`starpunk/avatars.py` and served from `/avatars/<sha256>.webp`; the
+16 -10
View File
@@ -1,8 +1,8 @@
# StarPunk Technology Stack
**Version**: 2.10.1 (2026-11-03)
**Version**: 2.14.0 (2026-11-10)
**Status**: Current. The original v0.x stack write-up is kept as
[technology-stack-legacy.md](technology-stack-legacy.md) for history.
[technology-stack-legacy.md](../design/v0.x/technology-stack-legacy.md) for history.
StarPunk deliberately uses few dependencies: a WSGI web framework, SQLite
from the standard library, plain files, server-rendered templates and a
@@ -21,7 +21,7 @@ the pieces fit together.
| Content storage | Markdown files on disk |
| Background work | `threading` daemon threads; `fcntl.flock` file locks to pick one process |
| Templates | Jinja2, server-side rendered |
| Front end | One hand-written stylesheet; one small vanilla JS file for the editor's Markdown preview |
| Front end | One hand-written stylesheet; small inline scripts on admin pages; no JavaScript on public pages |
| Packaging | `uv` creates the virtual environment and installs requirements (locally and in the image) |
| Container | Multi-stage `Containerfile` (Podman/Docker), non-root user |
| Reverse proxy | Caddy in production (examples for Caddy and nginx in the repo) |
@@ -43,10 +43,13 @@ vulnerabilities (`pip-audit`); check again before each release.
| python-dateutil | `4.0.*` | Dates in reply contexts |
| feedgen | `1.0.*` | RSS 2.0 feed (`starpunk/feeds/rss.py`); Atom and JSON Feed are generated by StarPunk's own code |
| httpx | `0.28.*` | All outbound HTTP; requests to user-supplied URLs go through `webmention_ssrf.safe_get`/`safe_post` |
| apprise | `2.2.*` | Notifications to the owner's services (`starpunk/notifications.py`, ADR-074); sends to the URLs the owner configures, outside the SSRF-safe fetcher |
| PyJWT[crypto] | `2.15.*` | Verifying OIDC ID tokens (JWKS, RS/ES signatures via `cryptography`) |
| python-dotenv | `1.2.*` | Loading `.env` |
| Pillow | `13.0.*` | Image validation, resizing and variants |
| Pillow | `13.0.*` | Image validation, resizing and variants; converting response avatars to 96 px WebP (`avatars.py`) |
| pillow-heif | `1.8.*` | HEIC/HEIF uploads (iPhone photos) |
| mcp | `2.3.*` | MCP server for AI clients (`starpunk/mcp_server/`, ADR-075); brings Starlette, pydantic and anyio. Only the sidecar imports it |
| uvicorn | `0.54.*` | ASGI server for the MCP sidecar |
`markupsafe` comes with Flask/Jinja2.
@@ -68,10 +71,11 @@ front-end framework or bundler, no separate search server.
| Area | Standards |
|------|-----------|
| IndieWeb | Micropub (create, update, delete, undelete, queries, media endpoint), IndieAuth (authorization server with PKCE, metadata endpoint, token revocation), Webmention (send and receive), WebSub (publisher and built-in hub), microformats2 (h-entry, h-feed, h-card markup and parsing), rel=me |
| IndieWeb | Micropub (create, update, delete, undelete, queries, media endpoint), IndieAuth (authorization server with PKCE, metadata endpoint, token revocation), Webmention (send and receive), WebSub (publisher and built-in hub), microformats2 (h-entry, h-feed, h-card markup and parsing), rel=me, web actions (`<indie-action>`), backfeed of responses from the silos |
| Login | OpenID Connect authorization code flow with PKCE (provider: Pocket ID) |
| Feeds | RSS 2.0, Atom 1.0, JSON Feed 1.1, OPML 2.0, HTTP content negotiation |
| Crossposting | Mastodon client API (GoToSocial), AT Protocol (Bluesky XRPC), Threads Graph API |
| AI clients | Model Context Protocol (Streamable HTTP, stateless), OAuth 2.0 Protected Resource Metadata (RFC 9728), Resource Indicators (RFC 8707), Client ID Metadata Documents |
| Crossposting | Mastodon client API (GoToSocial), AT Protocol (Bluesky XRPC), Threads Graph API; the same APIs are read for backfeed |
Protocol details: [indieweb-protocols.md](indieweb-protocols.md),
[simplified-auth-architecture.md](simplified-auth-architecture.md),
@@ -79,7 +83,7 @@ Protocol details: [indieweb-protocols.md](indieweb-protocols.md),
## Testing
- **Framework**: pytest with pytest-mock; about 1,250 tests in `tests/`
- **Framework**: pytest with pytest-mock; about 1,260 tests in `tests/`
(`test_*.py` per module or feature, shared fixtures in `tests/conftest.py`,
helpers in `tests/helpers/`).
- **Isolation**: the `app` fixture calls `create_app()` with `TESTING=True`
@@ -92,10 +96,12 @@ Protocol details: [indieweb-protocols.md](indieweb-protocols.md),
- **Running locally**:
```bash
uv venv && uv pip install -r requirements-dev.txt
.venv/bin/python -m pytest
uv venv --python 3.14 && uv pip install -r requirements-dev.txt
SESSION_SECRET=test uv run pytest tests/test_<area>.py
```
Locally, run the tests for the area you changed; CI runs the full suite.
- **In CI**: the `test` stage of the `Containerfile` installs the production
requirements plus `requirements-test.txt` and runs the whole suite. The
image is only built and pushed if it passes. Checklist:
@@ -119,4 +125,4 @@ standards.
---
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
@@ -6,6 +6,8 @@ Accepted
> Its deferral of WebSub is superseded by ADR-071 (v2.10.0), and "no feed caching" by the feed cache added in v1.1.2. RSS is now one of three feed formats (ADR-038) and is served at `/feed.rss` (`/feed.xml` still works). (status updated 2026-11-03)
> Item titles: since v2.16.0 only articles (notes with a title) have one; notes have none, and no first-line or date title is made up (ADR-076).
## Context
Phase 5 requires implementing RSS feed generation for syndicating published notes. We need to decide on the implementation approach, feed format, caching strategy, and technical details for generating a standards-compliant RSS feed.
@@ -3,6 +3,8 @@
## Status
Accepted (implemented in v1.3.0, reviewed in v2.8.0; was: Proposed)
> Since v2.16.0 `p-name` is the stored title and `p-summary` the author's summary or, in previews, the start of the text (ADR-076).
## Context
StarPunk currently implements basic microformats2 markup:
- h-entry on note articles
@@ -1,7 +1,7 @@
# ADR-062: Timestamp-Based Slug Format
## Status
Accepted (Supersedes ADR-007)
Accepted (Supersedes ADR-007). Articles take their slug from the title since v2.17.0 ([ADR-077](ADR-077-article-urls-and-title-slugs.md)).
## Context
+10 -1
View File
@@ -2,7 +2,7 @@
## Status
Accepted (v2.13.0, 2026-11-10)
Accepted (v2.13.0, 2026-11-10; amended v2.13.2, v2.13.3)
Implementation notes (v2.13.0):
@@ -235,6 +235,15 @@ While at it, the Threads client moves from `graph.threads.net` to
the same action on each crosspost. No web-action polyfill is loaded;
handlers such as browser extensions can use the markup.
## Amendments (v2.13.3, 2026-11-10)
- **Opted-out people are counted.** Their likes, reposts and quotes are
still not stored or named (Decision 5), but each check records how many
there were on the copy's `syndications` row (`likes_count`,
`reposts_count`, `quotes_count`, the columns Threads' counts use), and
the counts line adds them to the totals as "N on <silo>". Totals then
match the silo.
## Consequences
- Responses on all three silos show on the note within 15 minutes for new
+155
View File
@@ -0,0 +1,155 @@
# ADR-074: Notifications through Apprise
## Status
Accepted (v2.14.0, 2026-11-10)
## Context
Responses now reach a note from two places: received webmentions
(ADR-063) and backfeed from the crossposts (ADR-073). The owner only finds
out about them by opening the site or the admin. Crossposts that finally
fail, and silos that refuse a backfeed credential, are just as invisible.
Owners use different notification services. The production site uses
Gotify (read on iOS through iGotify). Others use ntfy, Pushover, email,
Matrix, Discord and so on. [Apprise](https://github.com/caronc/apprise)
covers about 100 of them behind one URL syntax (`gotifys://host/token`,
`ntfys://topic`, `mailtos://…`), both as a Python library and as a
separate HTTP server (apprise-api).
Options considered:
1. **Gotify only.** Simplest for this site, no use to anyone else.
2. **Post to an apprise-api server.** Standard, but every owner needs one
running, and messages take an extra hop.
3. **Embed the Apprise library.** Sends straight to each service; an owner
who prefers a central apprise-api can still use its `apprise://` /
`apprises://` URL.
4. **Gotify plus a generic JSON webhook, hand-written.** No dependency, but
every other service needs glue the owner has to write.
## Decision
### 1. Embed Apprise
StarPunk depends on the `apprise` package and sends with it directly.
`NOTIFY_URLS` holds one or more Apprise URLs (space- or comma-separated);
each message goes to all of them. With no URLs, notifications are off and
nothing else changes.
### 2. What can be notified
`NOTIFY_EVENTS` is a comma-separated list. Default:
`reply,mention,quote,problems`.
| Event | Sent when |
|---|---|
| `reply`, `mention`, `quote`, `like`, `repost`, `bookmark` | a response of that type is first shown on a note, or first awaits moderation (received webmention or backfeed alike) |
| `problems` | a crosspost fails for good (permanent error or last attempt), or a silo refuses the backfeed credential (ADR-073 Decision 9) |
### 3. Messages
Markdown, so services that render it (Gotify, iGotify, ntfy, Matrix…)
show links; others get the text.
- **Replies, mentions, quotes:** one message each. Title "↩ Reply from Ana
Lima via Bluesky" (or "💬 Mention from …", "❝ Quote from …"); body the
response text (plain, up to 300 characters), then a link to the note's
responses (`<permalink>#responses`).
- **Likes, reposts, bookmarks:** grouped per note and type in each pass:
"♥ 3 new likes on 'Starpunk now supports…'", with up to five names and a
link to the note.
- **Awaiting moderation:** when `WEBMENTION_AUTO_APPROVE` is off, the title
says so and the link goes to the admin webmention page instead.
- **Problems:** "⚠ Crosspost to Bluesky failed" / "⚠ Backfeed for Threads
stopped" with the error and a link to the note's edit page or the
Webmentions page.
Apprise's Gotify plugin has no click-through link, so the link is in the
text.
### 3a. Priority
Every message is sent with the same priority, `NOTIFY_PRIORITY` (default
`normal`). Apprise takes priority as a `priority` argument in each service
URL, with values that differ by service, so StarPunk adds
`priority=<NOTIFY_PRIORITY>` to each URL in `NOTIFY_URLS` that doesn't set
one itself; a URL can still set its own. Apprise's names (`low`,
`moderate`, `normal`, `high`, `emergency`) work for most services; Gotify
also takes 0–10 (it maps the names to 0, 3, 5, 8, 10). Different
priorities per event can come later.
### 4. Delivery
- New column `webmentions.notified_at`. A sweep finds responses with
`notified_at IS NULL`, `validation_status = 'valid'`, a moderation status
of `approved` or `pending`, and a type in `NOTIFY_EVENTS`, sends them,
and sets `notified_at`. Types not selected are marked without sending.
- The sweep runs in the webmention worker loop, which already runs in one
process at a time (its file lock), so nothing is sent twice by several
gunicorn workers. It handles received and backfed responses alike.
- A failed send leaves `notified_at` empty, so the next pass (about 10
seconds later) retries, with backoff up to 15 minutes. Responses older
than 24 hours that still haven't gone out are marked and dropped, with a
warning, so an outage doesn't end in a flood. The last error is shown on
the admin Webmentions page.
- Problems are sent at the moment they happen (from the syndication and
backfeed workers) and not retried: they are also in the log and admin.
- Sending never blocks or breaks validation, crossposting or backfeed.
### 5. No floods from history
- Migration 021 sets `notified_at` on every existing response.
- Responses stored by a backfill check (ADR-073 Decision 8) are stored with
`notified_at` set.
- So are responses found on a crosspost's first check when its note is more
than a day old (copies linked after the fact).
- Grouping (Decision 3) keeps a busy note to one like message per pass.
### 6. Admin
- **Send test notification** on the admin Webmentions page sends a test
message to every URL and reports each result.
- The page shows whether notifications are on, which events are selected,
and the last send error.
### 7. Configuration
| Setting | Default | Meaning |
|---|---|---|
| `NOTIFY_URLS` | empty (off) | Apprise URLs, space- or comma-separated |
| `NOTIFY_EVENTS` | `reply,mention,quote,problems` | which events are sent |
| `NOTIFY_PRIORITY` | `normal` | priority of every message, added to URLs that don't set one |
Settings stay in the environment, like the rest of StarPunk's (ADR-052).
URLs contain tokens and belong with the other secrets.
## Consequences
- One new dependency, `apprise`, with its own (requests, PyYAML, click,
Markdown, certifi). It must install on Python 3.14 in the container
image; this is checked before the release.
- Owners can send to almost any service without writing glue, directly or
through apprise-api.
- Each response is notified at most once, a few seconds after it appears,
and only if its type is selected.
- Apprise opens outbound connections to whatever URLs the owner configures,
outside the SSRF-safe fetcher. That is intended: the owner chooses these
URLs, as with the crosspost targets. A service behind the same reverse
proxy as StarPunk may need the deployment's internal-address entry
(`extra_hosts`), as GoToSocial does.
- Notification text includes other people's names and words, sent to the
owner's own services only.
## Alternatives considered
- **Gotify only / hand-written webhook.** See Context, options 1 and 4: no
dependency, but each service beyond Gotify needs glue the owner writes.
- **apprise-api only.** See option 2. Still possible through an
`apprises://` URL.
- **Sending from where responses are stored** (webmention worker,
backfeed) instead of a sweep. Two code paths, no retry, and backfeed
runs in every gunicorn worker. Rejected.
- **Choosing events in the admin UI.** Possible later; the environment is
where StarPunk's settings live today.
+121
View File
@@ -0,0 +1,121 @@
# ADR-075: MCP server for AI clients
## Status
Accepted (v2.15.0, 2026-11-11)
## Context
The owner wants AI assistants (claude.ai custom connectors, Claude Code and
other [Model Context Protocol](https://modelcontextprotocol.io/) clients) to
read, search and write notes. Micropub already covers writing, but MCP
clients don't speak Micropub: they discover tools over MCP and get tokens
through OAuth 2.1 with protected resource metadata (RFC 9728), resource
indicators (RFC 8707), and either dynamic client registration or a Client ID
Metadata Document (CIMD).
StarPunk already runs an OAuth/IndieAuth authorization server (ADR-064) with
PKCE, public clients only, and clients identified by a `client_id` URL whose
JSON document lists `redirect_uris`. That is what MCP calls CIMD. Claude uses
CIMD when the server's metadata says `client_id_metadata_document_supported:
true` and lists `none` in `token_endpoint_auth_methods_supported`.
StarPunk is Flask on gunicorn sync workers (WSGI). The official MCP Python
SDK is ASGI.
Options considered:
1. **Local stdio bridge** that calls `/micropub` with a stored token, like
existing Micropub MCP servers. No server changes, but every client needs
the bridge installed, claude.ai and mobile can't use it, and reading is
limited to what Micropub queries expose.
2. **Hand-written MCP route in Flask**, stateless JSON only. No new
dependencies or processes, but StarPunk would own the protocol details
(version negotiation, JSON-RPC, Origin checks, auth metadata) and track
the spec itself.
3. **Official SDK in a sidecar process** from the same image, behind the
same reverse proxy, sharing the data volume.
4. **Dynamic client registration** (RFC 7591) in the authorization server.
Every connection would register a new client, and nothing needs it once
CIMD works.
## Decision
### 1. A sidecar running the official SDK (option 3)
`starpunk/mcp_server/` builds an `mcp.server.MCPServer` (SDK 2.3) and serves
it with uvicorn (`uvicorn starpunk.mcp_server.app:app --port 8001`), as a
second container from the same image. The reverse proxy sends `/mcp*` and
`/.well-known/oauth-protected-resource*` to it and everything else to
gunicorn. The endpoint is `SITE_URL/mcp`.
The transport is stateless Streamable HTTP with plain JSON responses. No
session lives in the process, so restarts and several workers are harmless.
The sidecar builds the Flask app with `BACKGROUND_WORKERS_ENABLED=false`, so
only the main container runs the webmention, WebSub and syndication
workers. Each tool runs in a worker thread inside a Flask request context
for `SITE_URL`, and calls the same service functions as the web app.
SQLite (WAL) handles the second process like another gunicorn worker.
### 2. StarPunk is the authorization server; the sidecar only checks tokens
- The authorization server metadata adds
`client_id_metadata_document_supported: true` and
`token_endpoint_auth_methods_supported: ["none"]`, and `scopes_supported`
now includes `draft`.
- An authorization request may carry `resource` (RFC 8707). The only resource
StarPunk names is `SITE_URL/mcp`; anything else is `invalid_target`. The
resource is stored on the code and the token (migration 022), shown on the
consent screen, and must match if the token request repeats it.
- An MCP request with no `scope` is offered `create update delete media
draft`. The owner unticks what they don't want on the consent screen.
(The SDK publishes the resource's `scopes_supported` from its
required scopes, which StarPunk leaves empty so any token can connect.)
- The sidecar's `TokenVerifier` looks tokens up with `verify_access_token`
and reports their resource. With `validate_token_resource`, the SDK
refuses tokens not issued for `/mcp`, so a Micropub app's token can't be
replayed against MCP. MCP tokens still work at `/micropub`; both stay the
owner's own clients, and the Connected Apps page marks MCP tokens.
- DNS rebinding protection allows only the `SITE_URL` host and origin
(plus localhost in `DEV_MODE`).
### 3. Tools: reading and writing notes, nothing else
| Tool | Scope |
|---|---|
| `search_notes`, `list_notes`, `get_note`, `get_site_config` | any token; drafts need `create`, `update`, `delete` or `draft` |
| `list_tags` | `create`, `update`, `delete` or `draft` (as Micropub `q=category`) |
| `create_note` | `create` or `draft` (draft-only tokens always make drafts) |
| `update_note` | `update` |
| `delete_note`, `undelete_note` | `delete` |
| `upload_media` | `media` or `create` |
Writing tools build a Micropub JSON request and call `handle_create`,
`handle_update` and `handle_delete`. Scope rules, validation, crossposting,
webmentions and WebSub notifications are therefore identical to Micropub.
`create_note` makes a draft unless told otherwise. Tool annotations mark
reading tools read-only and update and delete as destructive.
Syndication management, webmention moderation and other admin functions are
not exposed.
## Consequences
- AI clients connect with just the URL: claude.ai via Settings → Connectors,
Claude Code via `claude mcp add --transport http starpunk SITE_URL/mcp`.
The owner approves them on the usual consent screen, and they appear (and
can be revoked) under Connected Apps.
- New runtime dependencies: `mcp` (with Starlette, pydantic and anyio) and
`uvicorn`. They are only imported by the sidecar.
- Deployments run a second container and need two proxy routes. A site that
doesn't run the sidecar just has no MCP; nothing else changes.
- There are still no refresh tokens. MCP clients re-authorize after
`INDIEAUTH_TOKEN_LIFETIME_DAYS`.
- SDK upgrades need care: 3.0 is expected to change defaults (for example
`validate_token_resource`, which StarPunk already sets).
## Alternatives
Options 1, 2 and 4 above. A stdio bridge could still be added later for
clients that can't do OAuth; it would use a token from this flow.
@@ -0,0 +1,74 @@
# ADR-076: Article titles and summaries stored apart from the text
## Status
Accepted (v2.16.0, 2026-11-11)
Article URLs (`/article/<slug>`) and title slugs: [ADR-077](ADR-077-article-urls-and-title-slugs.md) (v2.17.0).
Supersedes the v2.3.3 design ([micropub-title-and-source-url](../design/v2.3.3/2026-10-31-micropub-title-and-source-url.md))
and Q22's "a note has a title when its content starts with `#`". Amends
ADR-014 (feed item titles) and ADR-040 (`p-name` and `p-summary`).
## Context
An IndieWeb article is an h-entry with a `name` that isn't just the start of
its content (W3C Post Type Discovery). Since v1.2.0, StarPunk marked an article
by a `# Heading` first line in the Markdown, and v2.3.3 stored a Micropub `name`
as that heading. A review against IndieWeb practice found four gaps:
1. Feeds repeated an article's title: the item title, then the same heading
rendered at the top of the content.
2. Every feed item had a title, made from a note's first line (or the date),
so feed readers showed notes as headlines and couldn't tell them from
articles.
3. Micropub `summary` was used only when `content` was missing; sent
together with content, it was dropped.
4. Any note that happened to start with a heading became an article, and its
`q=source` changed from what the client sent. Micropub and mf2 treat `name`
and `content` as separate properties.
## Decision
- **Columns.** `notes.title` and `notes.summary` (migration 023). A note with
a title is an article; its Markdown file holds only the body. A heading in
the body is just a heading.
- **Moving existing titles.** Migration 023 has a Python data step
(`migrations.DATA_MIGRATIONS`) that runs once, in the migration's
transaction: for each note whose file starts with a heading (`split_title`,
trashed files included), it moves the heading into `title`, rewrites the file
and updates `content_hash`, then empties `notes_fts` so startup rebuilds it.
If anything fails, the changed files are written back and the transaction is
rolled back, so 023 is retried on the next start. Tying it to the migration
matters: run on every start it would turn new notes' headings into titles.
- **Micropub.** `name` → `title`; `summary` → `summary` when `content` is also
sent. A summary on its own is still stored as the content, which is how Post
Type Discovery reads it. `summary` can be replaced and deleted; `q=source`
returns `name` and `summary` when set. If the content starts with a heading
equal to `name`, that heading is dropped so it isn't shown twice.
- **Feeds.** Only articles have a title: RSS leaves `<title>` out (an item
needs a title or a description, and every item has a description), JSON Feed
leaves `title` out, and Atom, which requires `<title>`, has an empty one.
Atom `<summary>` and JSON Feed `summary` carry the author's summary. Content
no longer contains the title, so it isn't repeated.
- **Pages.** The article page has `p-name`, `p-summary` (when set) and the body
as `e-content`. On the homepage and tag pages an article shows its title,
summary (or the start of its text) and "Read more", without images. The
photos stay in the markup as hidden `u-photo` values so parsers read the same
post type there as on the article page.
- **Crossposting.** Articles go to GoToSocial, Bluesky and Threads as title,
summary (or the start of the text), link and hashtags, without images, the
usual POSSE form for articles. On Bluesky the link card is the article
itself. Notes are posted in full as before.
- **Editor.** Title and Summary fields; a titled note may have no body.
## Consequences
- The migration rewrites note files. Back up the database *and* `data/notes`
(and `.trash`) before deploying.
- `Note.title` is `None` for notes. Places that need a name for any post use
`Note.label` (title, else excerpt, else slug): notifications and the
Bluesky card. `Note.has_title` and `Note.body_html` are gone.
- Search indexes the title column; untitled notes are still listed by their
first line.
- A crossposted article that is edited is re-posted as a teaser.
@@ -0,0 +1,46 @@
# ADR-077: Article URLs and slugs from titles
## Status
Accepted (v2.17.0, 2026-11-12)
Amends ADR-062 (default slugs) for articles and ADR-076 (articles).
## Context
ADR-076 (v2.16.0) made a note with a title an article, but articles still got
a `/note/<timestamp>` URL like any other note. An article is not a note, and
its URL should say so. ADR-062 chose timestamp slugs because short notes have
no good words for a URL and their text may be private; an article has a title
written to be read, and that is the usual source of its slug on the IndieWeb.
## Decision
- **URLs.** Articles live at `/article/<slug>`, other notes at `/note/<slug>`.
`Note.permalink`, `Note.url`, `Note.endpoint` and `utils.post_path` /
`post_url` build them; nothing else writes `/note/` by hand. Slugs stay
unique across both, so a slug still names one post.
- **One URL each, the other redirects.** `/note/<slug>` for an article and
`/article/<slug>` for a note answer 301 to the right one. Adding or removing
a title changes the URL, and links made before (webmentions sent, crossposts,
bookmarks) keep working. A deleted post is 410 at either.
- **Slug from the title.** When a post is created with a title and no
`mp-slug`, the slug is the title made URL-safe (`sanitize_slug`: accents
folded, lowercase, hyphens), cut at a word boundary to 80 characters, `-2`,
`-3`… if taken, `-article` if reserved. A title with nothing usable (only
emoji, say) falls back to the timestamp slug. Notes keep timestamp slugs.
- **Slugs don't follow later title changes.** Editing an article's title, or
making a note an article, keeps its slug: a slug is the stable part of the
URL.
- **Micropub.** `Location` and `q=source` give the new URLs. Updates accept
either URL. An update that adds or removes a title answers 201 with the new
URL in `Location`, as the Micropub spec requires when a post's URL changes;
other updates are still 204.
- **Incoming webmentions** to either URL find the post.
## Consequences
- Production had no articles when this shipped, so no published URL changed.
- A note given a title later keeps its timestamp slug under `/article/`.
- Stored URLs (webmention sources, crosspost records) are not rewritten; the
redirects cover them.
+13 -9
View File
@@ -5,8 +5,8 @@ records one decision: its context, what was decided, and the consequences.
Superseded ADRs are kept as history; their Status line names the ADR that
replaced them.
**Total ADR files**: 69 (68 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-073, with two files numbered ADR-012). See [Numbering](#numbering).
**Total ADR files**: 73 (72 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-077, with two files numbered ADR-012). See [Numbering](#numbering).
Status values: **Accepted** (in force), **Superseded** (replaced by a later
ADR), **Deprecated** (withdrawn, nothing replaces it; none at present),
@@ -48,7 +48,7 @@ See also [database architecture](../architecture/database.md).
| ADR | Title | Status | Summary |
|---|---|---|---|
| [062](ADR-062-timestamp-based-slug-format.md) | Timestamp-Based Slug Format | Accepted | Default slugs are `YYYYMMDDHHMMSS` timestamps (v1.5.0); replaces ADR-007. |
| [062](ADR-062-timestamp-based-slug-format.md) | Timestamp-Based Slug Format | Accepted* | Default slugs are `YYYYMMDDHHMMSS` timestamps (v1.5.0); replaces ADR-007. Articles take theirs from the title (ADR-077). |
| [035](ADR-035-custom-slugs.md) | Custom Slugs in Micropub | Accepted* | Honour `mp-slug`, sanitized, with reserved names protected (v1.1.0). |
| [034](ADR-034-full-text-search.md) | Full-Text Search with SQLite FTS5 | Accepted | FTS5 search over note content (v1.1.0). |
| [057](ADR-057-media-attachment-model.md) | Media Attachment Model | Accepted | Social-media-style image attachments: up to 4 per note, ordered, with captions. |
@@ -58,7 +58,7 @@ See also [database architecture](../architecture/database.md).
| ADR | Title | Status | Summary |
|---|---|---|---|
| [014](ADR-014-rss-feed-implementation.md) | RSS Feed Implementation Strategy | Accepted* | RSS 2.0 feed; its WebSub deferral is superseded by ADR-071. |
| [014](ADR-014-rss-feed-implementation.md) | RSS Feed Implementation Strategy | Accepted* | RSS 2.0 feed; its WebSub deferral is superseded by ADR-071, and item titles by ADR-076. |
| [038](ADR-038-syndication-formats.md) | Multiple Syndication Format Support | Accepted | RSS, Atom and JSON Feed side by side (v1.1.2). |
| [071](ADR-071-websub.md) | WebSub publishing with a built-in hub | Accepted | Built-in WebSub hub at `/websub` for StarPunk's own feeds (v2.10.0, amended v2.10.1). |
| [059](ADR-059-full-feed-media-standardization.md) | Full Feed Media Standardization (Option 3) | Proposed | Full Media RSS, podcasts, video and thumbnails in feeds; not scheduled. |
@@ -67,7 +67,9 @@ See also [database architecture](../architecture/database.md).
| ADR | Title | Status | Summary |
|---|---|---|---|
| [040](ADR-040-microformats2-compliance.md) | Strict Microformats2 Compliance | Accepted | Full h-entry, h-card and h-feed markup (v1.3.0, reviewed v2.8.0). |
| [040](ADR-040-microformats2-compliance.md) | Strict Microformats2 Compliance | Accepted* | Full h-entry, h-card and h-feed markup (v1.3.0, reviewed v2.8.0); `p-name`/`p-summary` per ADR-076. |
| [076](ADR-076-note-title-and-summary-columns.md) | Article titles and summaries stored apart from the text | Accepted | `notes.title`/`summary` columns; feeds title only articles; article previews and crossposts are title + summary (v2.16.0). |
| [077](ADR-077-article-urls-and-title-slugs.md) | Article URLs and slugs from titles | Accepted | Articles at `/article/<slug>` with slugs from their titles; the other prefix redirects (v2.17.0). |
| [061](ADR-061-author-discovery.md) | Author Profile Discovery from IndieAuth | Accepted | The author h-card is read from `ADMIN_ME` and cached. |
| [063](ADR-063-webmention-implementation.md) | Webmention Implementation | Accepted | Async receiving (202) and sending via a background worker, SSRF guard, rate limits, moderation (v1.6.0, revised up to v2.10.1). *Reconstructed.* |
| [028](ADR-028-micropub-implementation.md) | Micropub Implementation Strategy | Accepted* | `/micropub` endpoint for creating posts (v1.0.0-rc.1); tokens now per ADR-064. |
@@ -85,7 +87,7 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
| [066](ADR-066-bluesky-syndication.md) | Crossposting to Bluesky | Accepted | Bluesky as a second crosspost target (v2.2.0). |
| [069](ADR-069-threads-syndication.md) | Crossposting to Threads | Accepted | Threads as a third target, with automatic token refresh (v2.6.0). |
| [070](ADR-070-crosspost-links-and-cards.md) | Links and link cards in crossposts | Accepted | Keep note links in crossposts and build link cards for the linked page (v2.6.3). |
| [073](ADR-073-backfeed.md) | Backfeed of responses to crossposts | Accepted | Poll GoToSocial, Bluesky and Threads for responses to crossposts and store them as webmentions; admin backfill (v2.13.0). |
| [073](ADR-073-backfeed.md) | Backfeed of responses to crossposts | Accepted | Poll GoToSocial, Bluesky and Threads for responses to crossposts and store them as webmentions; admin backfill (v2.13.0, amended v2.13.2–v2.13.3). |
### Authentication (current)
@@ -95,6 +97,7 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
| [010](ADR-010-authentication-module-design.md) | Authentication Module Design | Accepted* | Single `auth.py` module and server-side sessions; the login flow is now ADR-064. |
| [011](ADR-011-development-authentication-mechanism.md) | Development Authentication Mechanism | Accepted | `DEV_MODE` login bypass for local development only. |
| [022](ADR-022-auth-route-prefix-fix.md) | Fix IndieAuth Callback Route Mismatch | Accepted | Auth routes live under `/auth` (v0.9.2). |
| [075](ADR-075-mcp-server.md) | MCP server for AI clients | Accepted | `/mcp` served by the official MCP SDK in a sidecar; StarPunk's IndieAuth server issues resource-bound tokens via client ID metadata documents (v2.15.0). |
### Operations, errors and monitoring
@@ -105,6 +108,7 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
| [054](ADR-054-structured-logging-architecture.md) | Structured Logging Architecture | Accepted | Rotating log files with correlation IDs (v1.1.1). |
| [055](ADR-055-error-handling-philosophy.md) | Error Handling Philosophy | Accepted | Layered handling: graceful degradation, clear user messages, detailed logs. |
| [072](ADR-072-remove-metrics-dashboard.md) | Remove the metrics dashboard | Accepted | Drop the in-memory metrics system and dashboard; keep `/health`, `/admin/health` and logs (v2.12.0). |
| [074](ADR-074-notifications.md) | Notifications through Apprise | Accepted | Notify the owner of new responses and crosspost/backfeed problems via the embedded Apprise library (v2.14.0). |
See also [deployment](../architecture/deployment.md).
@@ -168,7 +172,7 @@ was carried out and is not itself superseded.
## How to create a new ADR
1. Use the next number after the highest existing one (currently ADR-074).
1. Use the next number after the highest existing one (currently ADR-077).
2. Name it `ADR-NNN-brief-descriptive-title.md`.
3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071):
@@ -201,5 +205,5 @@ was carried out and is not itself superseded.
---
**Last Updated**: 2026-11-03
**Total ADR files**: 67
**Last Updated**: 2026-11-11
**Total ADR files**: 72
+35 -9
View File
@@ -21,7 +21,13 @@ is in `../architecture/`; decisions are in `../decisions/` (ADRs).
## Contents
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [v2.12.0](#v2120) · [v2.12.1](#v2121) · [v2.13.0](#v2130) · [v2.13.1](#v2131) · [v2.13.2](#v2132) · [hotfix](#hotfix)
[v0.x](#v0x) · [v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [v2.12.0](#v2120) · [v2.12.1](#v2121) · [v2.13.0](#v2130) · [v2.13.1](#v2131) · [v2.13.2](#v2132) · [v2.13.3](#v2133) · [v2.13.4](#v2134) · [v2.14.0](#v2140) · [v2.14.1](#v2141) · [v2.14.2](#v2142) · [hotfix](#hotfix)
## v0.x
Not a release folder: documents from before v1.0.0 kept for history.
- [technology-stack-legacy.md](v0.x/technology-stack-legacy.md) — StarPunk Technology Stack - Quick Reference
## v1.0.0
@@ -408,31 +414,51 @@ OIDC admin login and built-in IndieAuth server.
## v2.10.2
- [2026-11-03-reply-context-fetch.md](v2.10.2/2026-11-03-reply-context-fetch.md) - Reply contexts fetched again (fetch_source returns three values since v2.7.0)
- [2026-11-03-reply-context-fetch.md](v2.10.2/2026-11-03-reply-context-fetch.md) — Reply context fetch
## v2.11.0
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) - Dependency upgrades (Python 3.14), foreign keys, CSRF, fetchers, debug default, shared rate limits
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) — Maintenance release
## v2.12.0
- [2026-11-03-remove-metrics.md](v2.12.0/2026-11-03-remove-metrics.md) - Metrics dashboard and `starpunk/monitoring` removed (ADR-072)
- [2026-11-03-remove-metrics.md](v2.12.0/2026-11-03-remove-metrics.md) — Remove the metrics dashboard
## v2.12.1
- [2026-11-10-category-unused-tags.md](v2.12.1/2026-11-10-category-unused-tags.md) - q=category lists only tags on posts that exist
- [2026-11-10-category-unused-tags.md](v2.12.1/2026-11-10-category-unused-tags.md) — q=category lists only tags in use
## v2.13.0
- [2026-11-10-backfeed.md](v2.13.0/2026-11-10-backfeed.md) - Backfeed of responses to crossposts (ADR-073), avatar cache, delete tombstones
- [2026-11-10-backfeed.md](v2.13.0/2026-11-10-backfeed.md) — Backfeed (implementation report)
## v2.13.1
- [2026-11-10-avatar-fetch.md](v2.13.1/2026-11-10-avatar-fetch.md) - Avatars from the own GoToSocial server and from Bluesky PDSes
- [2026-11-10-avatar-fetch.md](v2.13.1/2026-11-10-avatar-fetch.md) — Avatar fetching in production
## v2.13.2
- [2026-11-10-reaction-counts.md](v2.13.2/2026-11-10-reaction-counts.md) - Likes/reposts as a counts line with hover lists; web actions
- [2026-11-10-reaction-counts.md](v2.13.2/2026-11-10-reaction-counts.md) — Reaction counts and web actions
## v2.13.3
- [2026-11-10-count-opted-out.md](v2.13.3/2026-11-10-count-opted-out.md) — Count opted-out people without naming them
## v2.13.4
- [2026-11-10-list-counts.md](v2.13.4/2026-11-10-list-counts.md) — Counts on lists of notes, reply cards as links
## v2.14.0
- [2026-11-10-notifications.md](v2.14.0/2026-11-10-notifications.md) — Notifications (implementation report)
## v2.14.1
- [2026-11-10-tags-and-captions.md](v2.14.1/2026-11-10-tags-and-captions.md) — Clearing tags in the editor; captions escaped in feeds
## v2.14.2
- [2026-11-10-quotes-cleanup.md](v2.14.2/2026-11-10-quotes-cleanup.md) — Quote counts; dead files removed
## hotfix
@@ -453,4 +479,4 @@ IndieAuth PKCE endpoint discovery hotfix (2025-12-17, between v1.5.0 and v1.6.0;
---
**Last Updated**: 2026-11-03
**Last Updated**: 2026-11-10
@@ -3,7 +3,7 @@
> **Historical document.** This describes StarPunk's original v0.x technology
> choices (for example IndieLogin.com for admin login) and is kept for
> context only. It does not describe the current system; see
> [technology-stack.md](technology-stack.md) and [overview.md](overview.md).
> [technology-stack.md](../../architecture/technology-stack.md) and [overview.md](../../architecture/overview.md).
## Project Understanding
+1 -1
View File
@@ -1901,7 +1901,7 @@ class Webmention:
## References
- [ADR-004: File-Based Note Storage](../../decisions/ADR-004-file-based-note-storage.md)
- [Database Schema](../../../starpunk/monitoring/database.py)
- [Database Schema](../../../starpunk/database/schema.py)
- [Phase 1.1: Core Utilities](phase-1.1-core-utilities.md)
- [Python Coding Standards](../../standards/python-coding-standards.md)
- [Python Dataclasses Documentation](https://docs.python.org/3/library/dataclasses.html)
@@ -2061,7 +2061,7 @@ Phase 2.1 is complete when:
- [ADR-007: Slug Generation Algorithm](../../decisions/ADR-007-slug-generation-algorithm.md)
- [Phase 1.1: Core Utilities](phase-1.1-core-utilities.md)
- [Phase 1.2: Data Models](phase-1.2-data-models.md)
- [Database Schema](../../../starpunk/monitoring/database.py)
- [Database Schema](../../../starpunk/database/schema.py)
- [Python Coding Standards](../../standards/python-coding-standards.md)
- [Architecture Overview](../../architecture/overview.md)
- Security Architecture (`security.md`, not preserved)
@@ -343,6 +343,8 @@ Priority order:
3. First h-card found (fallback)
### p-name Only With Explicit Title
> Superseded in v2.16.0 (ADR-076): the title is `notes.title`, not a `#` first line. p-name still appears only for titled notes.
**Per Q22**:
- Detected by checking if content starts with `#`
- Matches note model's title extraction logic
@@ -0,0 +1,22 @@
# v2.13.3: Count opted-out people without naming them
## Why
A note's Bluesky copy had 1 like on Bluesky and 0 on the note. The liker's
account has the `!no-unauthenticated` label, so backfeed skipped them
(ADR-073 Decision 5). Every other Bluesky copy's like and repost counts
matched what StarPunk had stored.
## Change
`backfeed.hidden_counts(harvest)`: the silo's own counts when it gives no
names (Threads), else the number of opted-out likes, reposts and quotes in
the harvest. `check()` stores it on the copy's `syndications` row, and the
counts line (v2.13.2) already adds those columns to the totals as
"N on <silo>". Opted-out people are still not stored.
## Verification
`tests/test_backfeed.py::test_people_who_opt_out_are_counted_not_named`:
label, `#nobot` and `#nobridge` opt-outs counted (2 likes, 1 repost), not
stored, not named on the page; totals include them.
@@ -0,0 +1,24 @@
# v2.13.4: Counts on lists of notes, reply cards as links
## What changed
- `webmention_queries.response_counts(note_ids)`: likes, reposts and
replies per note in two grouped queries (shown webmentions, plus the
anonymous counts on synced `syndications` rows). `routes/public.py`
`attach_response_counts()` runs after `attach_contexts()` on the home,
tag and Likes pages.
- `partials/responses.html` `list_counts`: icons and numbers in the note's
meta line, linking to `<permalink>#responses` (an anchor on the note page).
- Reply cards: the date link (already the reply's own URL) stretches over
the card with `::after`; author and content links sit above it.
- v2.13.2's `.reaction*` classes clashed with the older `.reaction` lines
(Liked / Reposted / In reply to, `partials/reaction.html`): the counts
line now uses `.response-count*` / `.response-card*` / `.response-*`.
## Verification
- `tests/test_backfeed.py::test_home_page_shows_counts`; template, route,
microformats and likes/reposts tests pass.
- Rendered the home page, a note page and the Likes page in headless
Chromium: counts beside the date, quiet notes unchanged, the Liked line
laid out as before v2.13.2.
@@ -0,0 +1,38 @@
# v2.14.0: Notifications (implementation report)
Implements ADR-074.
## What changed
- `starpunk/notifications.py` (new): settings (`urls`, `events`,
`with_priority`), `send` (Apprise, Markdown, 15 s timeout, never raises,
errors captured with `apprise.LogCapture`), `problem`, and `Sweeper`,
which sends due responses, groups likes/reposts/bookmarks per note,
marks `notified_at`, backs off after a failure (10 s doubling to 15 min),
drops responses older than 24 hours, and records the last error in
`site_state.notify_last_error`.
- `migrations/021_notifications.sql`: `webmentions.notified_at`, set on all
existing rows.
- `webmention_worker.py`: runs the sweep each pass (one process holds the
worker lock).
- `backfeed.py`: `store(..., quiet=True)` inserts new rows as notified;
`check()` is quiet for backfill checks and a copy's first check of a note
older than a day. A refused credential sends a `problems` message.
- `syndication.py`: `_fail` sends a `problems` message when it gives up.
- Admin Webmentions page: status (events, services, priority, last error)
and **Send test notification** (`POST /admin/notifications/test`).
- Settings `NOTIFY_URLS`, `NOTIFY_EVENTS`, `NOTIFY_PRIORITY`; dependency
`apprise==2.2.*`.
## Verification
- `tests/test_notifications.py` (11 tests): settings parsing and priority
injection; one message per reply/mention, likes grouped, unselected types
marked, pending moderation linked to the admin; nothing sent twice;
failure backoff and retry, last error set and cleared; 24-hour drop; off
without URLs; quiet backfill and late first checks; problems (selected or
not) and the final crosspost failure; admin status and test button; a
real Apprise send (`json://`) to a local server; migration 021.
- Webmention, syndication, backfeed, admin and migration tests pass.
- Apprise 2.2.0 installs and imports on Python 3.14.
- Production Gotify (3.1.1) accepted a message with the StarPunk app token.
@@ -0,0 +1,27 @@
# v2.14.1: Clearing tags in the editor; captions escaped in feeds
Two bugs from the backlog (High), confirmed against the code at v2.14.0.
## Clearing tags
`routes/admin.py` (edit POST) passed `tags=tags if tags else None`, and
`notes.update_note` treats `None` as "unchanged", so an empty Tags field
kept the old tags although the form says "Leave blank to remove all tags".
It now passes the parsed list (possibly empty) whenever the form has a
`tags` field, and `None` only when it doesn't.
## Captions in feed HTML
The media `<img>` built for item HTML put the caption into `alt="…"`
unescaped in four places: `feeds/rss.py` (two), `feeds/atom.py` and
`feeds/json_feed.py`. A caption with `"`, `<` or `&` broke the markup. The
caption and the media URL are now `html.escape(..., quote=True)`d. The
Atom enclosure's `title` attribute was already escaped with `_escape_xml`
and is unchanged (escaping it again would double it).
## Verification
`tests/test_tags_and_captions.py` (4 tests, all failing before the fix):
caption escaped once in RSS, Atom (and the enclosure title not doubled) and
JSON Feed; the editor leaves tags alone without the field and clears them
with an empty one. Feed, admin, tag, public-route and media tests pass.
@@ -0,0 +1,31 @@
# v2.14.2: Quote counts; dead files removed
The two remaining items from the v2.14.0 documentation review.
## Quote counts
Backfeed stored anonymous quote counts (`syndications.quotes_count`: all
Threads quotes, and quotes by people who opted out elsewhere) that nothing
displayed. The counts line (`partials/responses.html` `reactions`) now has a
quote count after replies: named quotes plus those counts. Its card shows
only the anonymous part ("2 on Threads"); named quotes stay listed under
Responses. Quotes have no web action (there is no standard verb).
`webmention_queries.response_counts` adds `quotes`, so lists of notes show
them too.
## Removed
- `static/js/preview.js` (empty since the first commit) and the jsDelivr
`marked` script the editor pages loaded with it.
- `templates/feed.xml` (empty, never rendered) and `static/indielogin-test.html`
(a v1 IndieLogin test page).
- `syndication.get_targets()` read `THREADS_API` from the config, which
`config.py` never loads; the client now always uses the `THREADS_API`
constant.
## Verification
`tests/test_backfeed.py::test_quote_counts_include_anonymous_ones`; backfeed,
syndication, admin, template, public-route, microformats, notification,
webmention and feed tests pass. Rendered the counts line with the quote card
open in headless Chromium.
@@ -0,0 +1,42 @@
# v2.16.0: Article titles and summaries
Implements ADR-076. Prompted by a review of how StarPunk handles IndieWeb
articles: titles were repeated in feeds, every feed item had a made-up title,
Micropub `summary` was dropped, and any note starting with a heading became an
article.
## What changed
| Area | Change |
|---|---|
| Storage | Migration 023 adds `notes.title` and `notes.summary`; `create_note`/`update_note` take `title` and `summary` (`UNCHANGED` sentinel on update; `None`/"" clears) |
| Migration data step | `migrations.DATA_MIGRATIONS["023_note_titles.sql"] = _move_note_titles`; `run_migrations(..., data_dir=)` gets `DATA_PATH` from `init_db`. Moves `# Heading` first lines (via `split_title`) into `title`, finds trashed files with `utils.find_trashed_file` (shared with `undelete_note`), updates `content_hash`, empties `notes_fts`. Files are restored if the migration fails |
| Model | `Note.title`/`summary` fields; `Note.label` for places that need some name; `has_title`, `body_html` and the first-line `title` property removed |
| Micropub | `name` → title, `summary` → summary (only with content), `summary` updatable, `q=source` returns both; a heading repeating `name` is dropped (`drop_title_heading`) |
| Feeds | Titles only for articles (RSS none, JSON none, Atom empty); Atom/JSON summaries |
| Templates | Article page: `p-name`, `p-summary`, `e-content` without the title. Previews: title, summary or excerpt, "Read more", no images (hidden `u-photo` data kept for Post Type Discovery) |
| Crossposting | `compose_text` posts articles as title + `article_teaser` (summary, or 200 characters of plain text) + link + hashtags; no images for articles; Bluesky card is the article |
| Editor | Title and Summary fields on new/edit; content not required when titled |
| Search | FTS indexes `title`; untitled notes still use their first line |
| MCP | `create_note`/`update_note` take `summary`; note summaries include it |
## Fixes found along the way
- **Hashtag lines.** Python-Markdown made `#indieweb is fun` an `<h1>`; CommonMark needs a space after `#`. `markdown_ext.HashtagLineExtension` escapes the first `#` of a line where `#`s run straight into text, after fenced code and raw HTML are set aside. It is used for pages and feeds (`MARKDOWN_EXTENSIONS`) and crosspost plain text (`syndication._note_soup`), so the hashtag also reaches Bluesky as a tag. Search's first-line label keeps hashtags too.
- **Plain-text excerpt.** `Note.excerpt` only stripped `#` marks, so previews showed `**bold**` and `[link](url)`. It is now `syndication.plain_text` of the content, cut to 200 characters with `…`; `article_teaser` is the summary or this excerpt.
- **Inline images in Micropub articles.** (1) `extract_content` prefers `html` over `text` and marks it `content_format = 'html'`: clients send `text` as a plain fallback, so images, links and headings were lost. (2) `localize_inline_images` copies images from other sites (Markdown `![]()` or `<img>`) into the media library through the SSRF-checked fetcher used for `photo`, on create and on content replace, and rewrites their URLs; failures keep the original URL, at most 20 per post. Images stay inline (not `u-photo`), so the post remains an article. (3) `Note.html` sanitizes HTML content with nh3 directly instead of passing it through Markdown.
## Choices made with the owner
- Title column rather than an editor-only field, as the most standards-compliant option.
- Atom entries for notes keep an empty `<title>` (required by RFC 4287).
- Homepage and tag pages: articles show title and summary without images.
- Crossposts: articles as title + summary + link, no images.
## Tests
- `tests/test_articles.py` (new): feeds per format, article page mf2, homepage and tag-page previews, admin create/edit/clear, title-only article.
- `tests/test_migration_023.py` (new): titles moved (including trashed notes), hashtag first line untouched, hashes and FTS, runs once, files restored on failure, schema-only without `data_dir`.
- Updated for the new behaviour: `test_feed.py`, `test_feeds_{atom,json,rss}.py`, `test_routes_feed.py`, `test_models.py`, `test_microformats.py`, `test_microformats_v280.py`, `test_micropub.py`, `test_micropub_v240.py` (plus new summary and empty-body tests), `test_syndication.py` and `test_syndication_bluesky.py` (article crossposts).
- Locally: the 35 affected test files, 872+ tests, pass. CI runs the full suite on the tag.
@@ -0,0 +1,23 @@
# v2.17.0: Article URLs and title slugs
Implements ADR-077. Follow-up to v2.16.0 (ADR-076): articles had `/note/`
URLs and timestamp slugs.
## What changed
| Area | Change |
|---|---|
| URL helpers | `utils.post_path(slug, title)` and `post_url(slug, title)`; `Note.permalink` uses them, `Note.endpoint` (`public.article`/`public.note`) and `Note.url` (absolute) are new |
| Routes | `public.article` at `/article/<slug>`; `note` and `article` share `_permalink(slug, endpoint)`, which 301s to the other route when the post's type doesn't match. `_note_was_deleted` checks both URLs for hard-deleted posts |
| Slugs | `slug_utils.generate_title_slug(title, existing)`; `create_note` uses it for titled posts without `custom_slug`, else the timestamp slug |
| Micropub | `Location`, `q=source` url via `note.url`; `_note_from_url` accepts `/article/`; update answers 201 + `Location` when a title is added or removed |
| Other URL builders | Templates (`note.url`), webmention sending (`post_url`/`note.url`), crosspost permalink, backfeed target, notification links, search results (`article_title` from `notes.title`), MCP tools |
| Webmention receiving | `resolve_target_note` accepts `/note/` and `/article/` |
## Tests
`tests/test_articles.py::TestArticleUrls`: Micropub URL and slug, timestamp
slugs for notes, redirects both ways, updates via the old URL, slug rules
(accents, duplicates, reserved, `mp-slug`, emoji, long titles), 201 on
adding/removing a title, 410 for a deleted article, webmention targets and
homepage links, search result URLs.
@@ -1,5 +1,7 @@
# v2.3.3: Micropub titles, q=source URL, single title display
> Superseded in v2.16.0 by ADR-076: titles are stored in `notes.title`, not as a `# Heading` in the content, and `split_title` is used only to move old headings and drop a heading that repeats `name`.
Found while building tuipub, a terminal Micropub client, against StarPunk.
## Problems
+13 -15
View File
@@ -1,9 +1,9 @@
# StarPunk Backlog
**Last Updated**: 2026-11-03 (current release: v2.11.0)
**Last Updated**: 2026-11-10 (current release: v2.14.0)
These are the open features and bugs, sorted by priority. Each item was checked against the code
at v2.11.0. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` and in
at v2.14.0. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` and in
`CHANGELOG.md`.
## Priority Levels
@@ -29,16 +29,6 @@ None open.
## Medium
### Admin editor cannot clear all tags
- **Description**: When a note is saved with an empty tag field, the route passes `tags=None`. `update_note` treats that as "unchanged", so the last tags can't be removed from the editor.
- **Where**: `starpunk/routes/admin.py` (edit POST: `tags=tags if tags else None`), `starpunk/notes.py` (`update_note`)
- **Source**: Old backlog (v1.3.0 follow-up); still present
### Media captions not escaped in RSS HTML
- **Description**: The RSS description embeds `<img alt="{caption}">` without escaping the caption. A caption containing `"` or `<` breaks the item's HTML.
- **Where**: `starpunk/feeds/rss.py` (two places, ~lines 136 and 277)
- **Source**: Old backlog (Architect Review 2.2.10); still present
### No rate limiting on upload endpoints
- **Description**: Media uploads (`/admin/new` and the Micropub media endpoint) have no rate limit. Both require authentication, so the risk comes from a leaked token or session.
- **Where**: `starpunk/routes/admin.py`, `starpunk/routes/micropub.py`
@@ -75,7 +65,7 @@ None open.
- **Source**: Documentation refresh 2026-11-03
### Tag-filtered feeds
- **Description**: Tag pages are h-feeds and WebSub topics, but there are no RSS, Atom or JSON feeds per tag (for example `/tag/python/feed.atom`). Tag pages also have no pagination, and the editor has no tag autocomplete.
- **Description**: Tag pages are h-feeds and WebSub topics, but there are no RSS, Atom or JSON feeds per tag (for example `/tag/python/feed.atom`). Tag pages also have no pagination, and the editor has no tag autocomplete (Micropub `q=category`, v2.12.1, already gives the list a `<datalist>` could use).
- **Where**: `starpunk/routes/public.py`, `starpunk/feeds/`, `templates/admin/`
- **Source**: Old backlog (Tag enhancements)
@@ -154,7 +144,7 @@ None open.
- **Source**: Old backlog (v1.5.0 scoping)
### Unused settings and legacy tables
- **Description**: `NOTES_PATH` is read and its folder created, but notes are always written under `DATA_PATH/notes`. `SITE_AUTHOR` is required but shown nowhere. `INITIAL_SCHEMA_SQL` recreates the unused `tokens` and `authorization_codes` tables (dropped by migration 004) on every start, and `auth_state` is unused. Version fallbacks are stale (`author_discovery.py` "1.2.0", `base.html` "0.5.0").
- **Description**: `NOTES_PATH` is read and its folder created, but notes are always written under `DATA_PATH/notes`. `SITE_AUTHOR` is checked as required but defaults to "Unknown" and is shown nowhere. `INITIAL_SCHEMA_SQL` recreates the unused `tokens` and `authorization_codes` tables (dropped by migration 004) on every start, and `auth_state` is unused. Version fallbacks are stale (`author_discovery.py` "1.2.0", `base.html` "0.5.0").
- **Where**: `starpunk/config.py`, `starpunk/utils.py`, `starpunk/database/schema.py`, `starpunk/author_discovery.py`, `templates/base.html`
- **Source**: Documentation refresh 2026-11-03 (docs/architecture/database.md, deployment.md)
@@ -171,10 +161,18 @@ None open.
---
## Done since the last update (2025-12-17)
## Done (recent releases)
These items were removed from the backlog because they have shipped:
- v2.14.2: quote counts (with anonymous ones) shown; dead editor scripts, files and the `THREADS_API` read removed
- v2.14.1: the editor can clear all tags; image captions escaped in RSS, Atom and JSON Feed HTML
- v2.14.0: notifications of new responses and crosspost/backfeed problems through Apprise (ADR-074)
- v2.13.0–v2.13.4: backfeed of likes, reposts, quotes and replies from GoToSocial, Bluesky and Threads (ADR-073); counts line with web actions; counts on lists of notes; local avatar copies
- v2.12.1: Micropub `q=category` lists only tags in use
- v2.12.0: metrics dashboard and monitoring package removed (ADR-072)
- LICENSE: MIT, copyright Phil Skentelbery 2025-2026 (2026-11-03)
- v2.11.0: SQLite foreign keys enforced (orphans cleaned up by migration 018); CSRF tokens on admin, logout and consent forms; IndieAuth client and crosspost link-card fetches moved onto the SSRF-safe fetcher; `FLASK_DEBUG`/`FLASK_ENV` default to off/production; rate limits shared by all processes (and old windows pruned, so the webmention limiter no longer grows); dependencies upgraded (Python 3.14, no known vulnerabilities); stale `compose.yaml`, duplicate `dev_auth.py` and `test.ini` removed or fixed; `lxml`/`python-dateutil` pinned, `pytest` out of the runtime requirements
+16 -7
View File
@@ -4,19 +4,19 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
## Backlog
- **[BACKLOG.md](BACKLOG.md)** - Open features and bugs by priority (updated for v2.11.0)
- **[BACKLOG.md](BACKLOG.md)** - Open features and bugs by priority (updated for v2.14.0)
## Releases
- **[v1.0.0](v1.0.0/RELEASE.md)** (Released 2025-11-24)
- **[v1.1.0](v1.1.0/RELEASE.md)** (Released 2025-11-25)
- **[v1.1.1](v1.1.1/RELEASE.md)** (Released 2025-11-26)
- **[v1.1.2](v1.1.2/RELEASE.md)** (Released 2025-11-27)
- **[v1.1.1](v1.1.1/RELEASE.md)** (Released 2025-11-25)
- **[v1.1.2](v1.1.2/RELEASE.md)** (Released 2025-11-28)
- **[v1.2.0](v1.2.0/RELEASE.md)** (Released 2025-12-09)
- **[v1.3.0](v1.3.0/RELEASE.md)** (Complete)
- **[v1.3.1](v1.3.1/RELEASE.md)** - Feed Categories/Tags Support (Planning)
- **[v1.4.0](v1.4.0/RELEASE.md)** - Micropub Media Endpoint, Large Image Support, Enhanced Feed Media (Planning)
- **[v1.5.0](v1.5.0/RELEASE.md)** - Stability, Test Coverage, and Technical Debt Reduction (Approved)
- **[v1.3.0](v1.3.0/RELEASE.md)** (Released 2025-12-10)
- **[v1.3.1](v1.3.1/RELEASE.md)** - Feed Categories/Tags Support (Released 2025-12-10)
- **[v1.4.0](v1.4.0/RELEASE.md)** - Micropub Media Endpoint, Large Image Support, Enhanced Feed Media (Released 2025-12-16 as 1.4.0rc1, then 1.4.1 and 1.4.2)
- **[v1.5.0](v1.5.0/RELEASE.md)** - Stability, Test Coverage, and Technical Debt Reduction (Released 2025-12-17)
- **[v2.0.0](v2.0.0/RELEASE.md)** - OIDC admin login and built-in IndieAuth server (Released 2026-10-31)
- **[v2.0.1](v2.0.1/RELEASE.md)** - Absolute URLs behind HTTPS proxy (Released 2026-10-31)
- **[v2.0.2](v2.0.2/RELEASE.md)** - CI registry login (Released 2026-10-31)
@@ -49,3 +49,12 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
- **[v2.11.0](v2.11.0/RELEASE.md)** - Maintenance: dependency upgrades, High backlog items (Released 2026-11-03)
- **[v2.12.0](v2.12.0/RELEASE.md)** - Metrics dashboard removed (Released 2026-11-03)
- **[v2.12.1](v2.12.1/RELEASE.md)** - `q=category` lists only tags in use (Released 2026-11-10)
- **[v2.13.0](v2.13.0/RELEASE.md)** - Backfeed of responses to crossposts (Released 2026-11-10)
- **[v2.13.1](v2.13.1/RELEASE.md)** - Avatar fetching fixes (Released 2026-11-10)
- **[v2.13.2](v2.13.2/RELEASE.md)** - Counts line and web actions (Released 2026-11-10)
- **[v2.13.3](v2.13.3/RELEASE.md)** - Opted-out people counted without names (Released 2026-11-10)
- **[v2.13.4](v2.13.4/RELEASE.md)** - Counts on lists of notes, reply cards link out (Released 2026-11-10)
- **[v2.14.0](v2.14.0/RELEASE.md)** - Notifications through Apprise (Released 2026-11-10)
- **[v2.14.1](v2.14.1/RELEASE.md)** - Clearing tags in the editor; captions escaped in feeds (Released 2026-11-10)
- **[v2.14.2](v2.14.2/RELEASE.md)** - Quote counts; dead files removed (Released 2026-11-10)
+1 -1
View File
@@ -1,6 +1,6 @@
# StarPunk v1.1.1 Release
**Status**: Released 2025-11-26
**Status**: Released 2025-11-25
## Features
+1 -1
View File
@@ -1,6 +1,6 @@
# StarPunk v1.1.2 Release
**Status**: Released 2025-11-27
**Status**: Released 2025-11-28
**Codename**: Syndicate
## Features
+1 -1
View File
@@ -1,6 +1,6 @@
# StarPunk v1.3.0 Release
**Status**: Complete
**Status**: Released 2025-12-10
**Codename**: "Categories"
**Branch**: `feature/v1.3.0-tags-microformats`
**Approved**: 2025-12-10
+1 -1
View File
@@ -1,6 +1,6 @@
# StarPunk v1.3.1 Release
**Status**: Planning
**Status**: Released 2025-12-10
**Codename**: "Syndicate Tags"
**Focus**: Feed Categories/Tags Support
+1 -1
View File
@@ -1,6 +1,6 @@
# StarPunk v1.4.0 Release
**Status**: Planning
**Status**: Released 2025-12-16 (as 1.4.0rc1; then 1.4.1 and 1.4.2)
**Codename**: "Media"
**Focus**: Micropub Media Endpoint, Large Image Support, Enhanced Feed Media
+1 -1
View File
@@ -1,6 +1,6 @@
# StarPunk v1.5.0 Release Plan
**Status**: Approved
**Status**: Released 2025-12-17
**Codename**: "Trigger"
**Focus**: Stability, Test Coverage, and Technical Debt Reduction
**Last Updated**: 2025-12-17
+1 -1
View File
@@ -1,7 +1,7 @@
# v2.10.2
- **Date**: 2026-11-03
- **Status**: Released
- **Status**: Released 2026-11-03
## Bugs fixed
+1 -1
View File
@@ -1,7 +1,7 @@
# v2.11.0
- **Date**: 2026-11-03
- **Status**: Released
- **Status**: Released 2026-11-03
- **Focus**: Maintenance: dependency upgrades and the High backlog items
## Security
+1 -1
View File
@@ -1,7 +1,7 @@
# v2.12.0
- **Date**: 2026-11-03
- **Status**: Released
- **Status**: Released 2026-11-03
- **Focus**: Remove the metrics dashboard
## Removed
+14
View File
@@ -0,0 +1,14 @@
# v2.12.1
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Micropub `q=category` lists only tags in use
## Fixed
- `q=category` returns only tags on posts that exist (not ones an edit removed, nor ones only on deleted posts)
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [Design note](../../design/v2.12.1/2026-11-10-category-unused-tags.md)
+25
View File
@@ -0,0 +1,25 @@
# v2.13.0
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Backfeed of responses to crossposts
## Added
- Likes, reposts, quotes and replies to GoToSocial and Bluesky crossposts, and replies on Threads, shown on the note
- Age-based checking schedule up to `BACKFEED_MAX_AGE_DAYS` (90); admin backfill and per-crosspost Check now
- Threads like/repost/quote counts; a Quotes group; origin filter and Deleted view in the admin
- Settings `BACKFEED_ENABLED`, `BACKFEED_MAX_AGE_DAYS`; migration 020
## Changed
- Response avatars saved locally (`/avatars/`)
- Admin delete keeps a tombstone
- `#nobridge` / `#nobot` and Bluesky's logged-out label honoured; already backfed responses not shown twice from Bridgy
- Threads client on `graph.threads.com`
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [ADR-073](../../decisions/ADR-073-backfeed.md)
- [Design note](../../design/v2.13.0/2026-11-10-backfeed.md)
+15
View File
@@ -0,0 +1,15 @@
# v2.13.1
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Avatar fetching in production
## Fixed
- Avatars from the site's own GoToSocial (resolves to an internal address) and from Bluesky (CDN refuses some server IPv4 addresses; fetched from the PDS instead)
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [ADR-073](../../decisions/ADR-073-backfeed.md)
- [Design note](../../design/v2.13.1/2026-11-10-avatar-fetch.md)
+20
View File
@@ -0,0 +1,20 @@
# v2.13.2
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Counts line for likes and reposts; web actions
## Changed
- Likes, reposts and bookmarks shown as one line of counts under the note, with who on hover; Threads counts folded in
- `<indie-action>` web actions with links to the note's crossposts
## Fixed
- Avatars GoToSocial hasn't cached (redirects to the original server)
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [ADR-073](../../decisions/ADR-073-backfeed.md)
- [Design note](../../design/v2.13.2/2026-11-10-reaction-counts.md)
+15
View File
@@ -0,0 +1,15 @@
# v2.13.3
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Opted-out people counted without names
## Changed
- Likes and reposts by people who opted out of backfeed count in the totals, without name or avatar
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [ADR-073](../../decisions/ADR-073-backfeed.md)
- [Design note](../../design/v2.13.3/2026-11-10-count-opted-out.md)
+22
View File
@@ -0,0 +1,22 @@
# v2.13.4
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Counts on lists of notes; reply cards
## Added
- Like, repost and reply counts on the home, tag and Likes pages
## Changed
- A whole reply card opens the reply where it was posted
## Fixed
- The counts line's CSS clashed with the Liked / Reposted / In reply to lines (`.reaction`)
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [Design note](../../design/v2.13.4/2026-11-10-list-counts.md)
+16
View File
@@ -0,0 +1,16 @@
# v2.14.0
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Notifications through Apprise
## Added
- Notifications of new responses and crosspost/backfeed problems via the embedded Apprise library
- Settings `NOTIFY_URLS`, `NOTIFY_EVENTS`, `NOTIFY_PRIORITY`; admin status and Send test notification; migration 021
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [ADR-074](../../decisions/ADR-074-notifications.md)
- [Design note](../../design/v2.14.0/2026-11-10-notifications.md)
+15
View File
@@ -0,0 +1,15 @@
# v2.14.1
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: The two High backlog bugs
## Fixed
- The editor can clear all of a note's tags
- Image captions escaped in RSS, Atom and JSON Feed item HTML
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [Design note](../../design/v2.14.1/2026-11-10-tags-and-captions.md)
+18
View File
@@ -0,0 +1,18 @@
# v2.14.2
- **Date**: 2026-11-10
- **Status**: Released
- **Focus**: Quote counts; dead files removed
## Added
- Quote counts (with anonymous ones) in the counts line and on lists of notes
## Removed
- Unused editor scripts (`marked` from a CDN, empty `preview.js`), `templates/feed.xml`, `static/indielogin-test.html`, the `THREADS_API` config read
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [Design note](../../design/v2.14.2/2026-11-10-quotes-cleanup.md)
@@ -1,11 +1,11 @@
# Utility Function Patterns
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1; reviewed, content from 2025-11-18)
**Status**: Deprecated (2026-11-10). Superseded by [python-coding-standards.md](../python-coding-standards.md), which this document contradicts on type hints (`X | None`, built-in generics, hints on new and changed public functions).
**Last Updated**: 2026-11-03 (v2.10.1; content from 2025-11-18)
## Purpose
This document establishes coding patterns and conventions specifically for utility functions in StarPunk (`starpunk/utils.py`, `starpunk/slug_utils.py`, `starpunk/mf2_utils.py` and helpers in other modules). These patterns ensure utilities are consistent, testable, and maintainable. General rules, including project-wide patterns for configuration, database access and outbound HTTP, are in [python-coding-standards.md](python-coding-standards.md); where the two differ, that document wins.
This document establishes coding patterns and conventions specifically for utility functions in StarPunk (`starpunk/utils.py`, `starpunk/slug_utils.py`, `starpunk/mf2_utils.py` and helpers in other modules). These patterns ensure utilities are consistent, testable, and maintainable. General rules, including project-wide patterns for configuration, database access and outbound HTTP, are in [python-coding-standards.md](../python-coding-standards.md); where the two differ, that document wins.
The examples are illustrative. Several mirror real functions in `starpunk/utils.py` (`calculate_content_hash`, `write_note_file`, `validate_slug`, `generate_slug`, `validate_note_path`); others (`require_valid_slug`, `iter_note_files`) only show the pattern.
@@ -731,7 +731,7 @@ def generate_slug(content: str) -> str:
## References
- [Python Coding Standards](python-coding-standards.md)
- [Python Coding Standards](../python-coding-standards.md)
- [PEP 8 - Style Guide](https://peps.python.org/pep-0008/)
- [PEP 257 - Docstring Conventions](https://peps.python.org/pep-0257/)
- [PEP 484 - Type Hints](https://peps.python.org/pep-0484/)
+7 -7
View File
@@ -1,10 +1,10 @@
# Standards Index
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
Standards for developing and releasing StarPunk. Folder rules are in
[../DOCUMENTATION.md](../DOCUMENTATION.md). All standards below were checked
against the code at v2.10.1.
against the code at v2.14.0.
## Active standards
@@ -16,7 +16,6 @@ against the code at v2.10.1.
| [versioning-strategy.md](versioning-strategy.md) | Choosing MAJOR / MINOR / PATCH |
| [version-implementation-guide.md](version-implementation-guide.md) | Where the version lives, changelog entry and design note format |
| [python-coding-standards.md](python-coding-standards.md) | Style, naming, and project patterns (config, database, outbound HTTP, errors, routes, workers) |
| [utility-function-patterns.md](utility-function-patterns.md) | Patterns for small helper functions |
| [cookie-naming-convention.md](cookie-naming-convention.md) | `starpunk_` cookie prefix and the cookies in use |
| [documentation-organization.md](documentation-organization.md) | Templates and conventions for docs (folder rules: DOCUMENTATION.md) |
@@ -31,8 +30,8 @@ against the code at v2.10.1.
## Key rules in brief
- `uv` for environments, `uv run` for every command.
- Tests need `SESSION_SECRET`; the full suite takes about 6 minutes, 5 of
them in `tests/test_migration_race_condition.py`.
- Tests need `SESSION_SECRET`. Locally, run only the tests for the area you
changed; CI runs the full suite (about 6 minutes) on every `vX.Y.Z` tag.
- Every release: branch in a worktree, bump `__version__` and
`__version_info__`, changelog entry, design note (or ADR amendment),
`git merge --no-ff`, annotated `vX.Y.Z` tag; CI tests and publishes the
@@ -43,8 +42,9 @@ against the code at v2.10.1.
## Deprecated standards
Standards that no longer apply move to `DEPRECATED/` (none at
present). Earlier versions of the standards above are in git history.
Standards that no longer apply move to `DEPRECATED/`:
[utility-function-patterns.md](DEPRECATED/utility-function-patterns.md)
(superseded by python-coding-standards.md, 2026-11-10). Earlier versions of the standards above are in git history.
## Related
+1 -1
View File
@@ -2,7 +2,7 @@
**Status**: Active
**Date**: 2025-11-18
**Last Updated**: 2026-11-03 (v2.10.1; cookie list checked against the code)
**Last Updated**: 2026-11-10 (v2.14.0; cookie list checked against the code)
**Version**: 1.1
## Purpose
+9 -8
View File
@@ -1,7 +1,7 @@
# Development Setup
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
How to set up a local StarPunk environment, run the app and run the tests.
Each configuration variable is described in [`.env.example`](../../.env.example);
@@ -17,8 +17,9 @@ deployment is in [architecture/deployment.md](../architecture/deployment.md).
## Environment
The repository has no `pyproject.toml`; dependencies are in
`requirements.txt` (runtime) and `requirements-dev.txt` (adds pytest-cov,
pytest-mock, black, flake8, mypy).
`requirements.txt` (runtime) and `requirements-dev.txt` (adds
`requirements-test.txt` (pytest, pytest-mock), plus pytest-cov, black, flake8,
mypy and pip-audit).
```bash
git clone https://git.thesatelliteoflove.com/phil/StarPunk.git
@@ -100,9 +101,9 @@ docker run --rm -p 8000:8000 --env-file .env -v ./container-data:/data starpunk:
## Running the tests
```bash
SESSION_SECRET=test uv run pytest # full suite
SESSION_SECRET=test uv run pytest tests/test_websub.py # one file
SESSION_SECRET=test uv run pytest tests/test_websub.py # the area you changed
SESSION_SECRET=test uv run pytest tests/test_micropub.py -k delete -x
SESSION_SECRET=test uv run pytest # full suite (CI runs this; rarely needed locally)
```
- `SESSION_SECRET` must be set (in `.env` or the environment) because
@@ -113,9 +114,9 @@ SESSION_SECRET=test uv run pytest tests/test_micropub.py -k delete -x
`DATA_PATH` and database, so tests never touch `./data`. Under
`TESTING=True` the background workers don't start and
the production OIDC check is skipped.
- `tests/test_migration_race_condition.py` is slow (about 5 minutes). While
iterating, add `--ignore=tests/test_migration_race_condition.py`; run the
full suite before releasing.
- Locally, run the tests for the area you changed; the full suite, including
the slow `tests/test_migration_race_condition.py` (about 5 minutes), runs in
CI on the release tag.
- Coverage: `SESSION_SECRET=test uv run pytest --cov=starpunk --cov-report=term-missing`.
- The CI test stage can be reproduced exactly with
`docker build --target test -f Containerfile .` (see
+2 -2
View File
@@ -1,7 +1,7 @@
# Documentation Organization
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
**[docs/DOCUMENTATION.md](../DOCUMENTATION.md) is authoritative** for what
goes in which folder and supersedes this document. This standard only adds
@@ -94,5 +94,5 @@ top saying what replaced it, and removed from the active list in
- Architecture documents describing a design that no longer exists are
either rewritten or renamed with a `-legacy` suffix and a note at the top
(e.g. `architecture/technology-stack-legacy.md`).
(e.g. `design/v0.x/technology-stack-legacy.md`).
- Design notes and ADRs are historical records and stay where they are.
+5 -3
View File
@@ -1,7 +1,7 @@
# Git Branching and Release Process
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
How changes reach `main`, how releases are cut, and what happens after a
tag is pushed. This is the flow used for the v2 releases (see
@@ -27,6 +27,7 @@ differ, this document describes current practice.
| `feat/` | new behaviour (MINOR) | `feat/v2.10.0-websub` |
| `fix/` | bug and security fixes (PATCH) | `fix/v2.10.1-websub-hardening` |
| `ci/` | build and CI changes | `ci/v2.7.1-pytest` |
| `chore/` | maintenance, removals, licensing | `chore/remove-metrics` |
| `docs/` | documentation only, no release | `docs/refresh-v2.10.1` |
Release branches are named `<prefix>/vX.Y.Z-<short-description>`, so the
@@ -83,8 +84,9 @@ Code, tests and docs together:
- [ ] Add a design note `docs/design/vX.Y.Z/YYYY-MM-DD-<topic>.md` (what
changed and why, how it was verified). Small releases fully covered by
an ADR amendment and the changelog may skip it
- [ ] Full test suite passes:
`SESSION_SECRET=test uv run pytest`
- [ ] Tests for the changed area pass locally, e.g.
`SESSION_SECRET=test uv run pytest tests/test_notifications.py`;
CI runs the full suite on the tag (step 4)
Commit (usually one commit):
+2 -4
View File
@@ -1,12 +1,11 @@
# Python Coding Standards
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
How Python code in StarPunk is written. The project rule comes first:
"Every line of code must justify its existence. When in doubt, leave it
out." Patterns specific to small helper functions are in
[utility-function-patterns.md](utility-function-patterns.md).
out."
## Base style
@@ -147,5 +146,4 @@ functions or `Test...` classes with descriptive names
## References
- [PEP 8](https://peps.python.org/pep-0008/), [PEP 257](https://peps.python.org/pep-0257/), [PEP 484](https://peps.python.org/pep-0484/)
- [utility-function-patterns.md](utility-function-patterns.md)
- [ADR-055: Error handling philosophy](../decisions/ADR-055-error-handling-philosophy.md)
+12 -9
View File
@@ -1,7 +1,7 @@
# Testing Checklist
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
What to test, how to run the tests, and what CI does with them. Setup and
the environment variables the tests need are in
@@ -12,12 +12,14 @@ the environment variables the tests need are in
**Locally**
```bash
SESSION_SECRET=test uv run pytest # everything
SESSION_SECRET=test uv run pytest --ignore=tests/test_migration_race_condition.py # fast loop
SESSION_SECRET=test uv run pytest tests/test_websub.py tests/test_websub_hardening.py # the area you changed
SESSION_SECRET=test uv run pytest # everything (CI does this)
```
`tests/test_migration_race_condition.py` takes about 5 minutes; skip it
while iterating, never before a release.
Locally, run the tests for the area you changed. The full suite, including
the 5-minute `tests/test_migration_race_condition.py`, runs in CI. For wide
changes (migrations, `config.py`, `create_app`, `tests/conftest.py`) you can
build the CI test stage first: `docker build --target test -f Containerfile .`
**In CI** (`.gitea/workflows/build-container.yml`), on every pushed
`vX.Y.Z` tag (or a manual run):
@@ -30,16 +32,17 @@ while iterating, never before a release.
2. Only if that succeeds is the runtime image built and pushed as
`:vX.Y.Z` and `:latest`.
So a failing test blocks the release image; it does not block the merge or
the tag. Run the full suite before merging (see
[git-branching-strategy.md](git-branching-strategy.md#release-process)).
So a failing test blocks the release image, not the merge or the tag; CI is
where the full suite runs. If it fails, fix it in the next PATCH (see
[git-branching-strategy.md](git-branching-strategy.md#release-process),
step 4).
To reproduce CI locally: `docker build --target test -f Containerfile .`
(pytest-cov is not installed in that stage, so don't depend on it in
tests).
## Every change
- [ ] Full suite passes locally
- [ ] Tests for the changed area pass locally (CI runs the full suite)
- [ ] New behaviour has tests; a bug fix has a test that failed before the fix
- [ ] Tests use the `app` / `client` fixtures from `tests/conftest.py`
(temporary data directory, `TESTING=True`); nothing touches `./data`
@@ -1,7 +1,7 @@
# Version Implementation Guide
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
Where the version number lives and what to write for each release. Which
number to pick: [versioning-strategy.md](versioning-strategy.md). Git steps:
@@ -14,8 +14,8 @@ There is one source: `starpunk/__init__.py`, at the end of the file.
```python
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.10.1"
__version_info__ = (2, 10, 1)
__version__ = "X.Y.Z"
__version_info__ = (X, Y, Z)
```
Update both lines. Nothing else holds the number: there is no
+2 -2
View File
@@ -1,10 +1,10 @@
# Versioning Strategy
**Status**: Active
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-10 (v2.14.0)
StarPunk uses [Semantic Versioning 2.0.0](https://semver.org/):
`MAJOR.MINOR.PATCH`, currently **2.13.2**. Every release is a git tag
`MAJOR.MINOR.PATCH` (the current version is in `starpunk/__init__.py`). Every release is a git tag
`vX.Y.Z` and a `CHANGELOG.md` entry. Where the number is stored and how to
write the changelog entry is in
[version-implementation-guide.md](version-implementation-guide.md); the
+6
View File
@@ -0,0 +1,6 @@
-- Migration 021: Notifications (v2.14.0, ADR-074)
-- When the owner was notified of a response. Everything already stored
-- counts as notified, so turning notifications on doesn't send history.
ALTER TABLE webmentions ADD COLUMN notified_at TIMESTAMP;
UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP;
+6
View File
@@ -0,0 +1,6 @@
-- Migration 022: RFC 8707 resource indicators for IndieAuth codes and tokens
-- Version: 2.15.0 (ADR-075)
-- Tokens for the MCP endpoint are bound to it; Micropub tokens leave it NULL.
ALTER TABLE indieauth_codes ADD COLUMN resource TEXT;
ALTER TABLE indieauth_tokens ADD COLUMN resource TEXT;
+6
View File
@@ -0,0 +1,6 @@
-- Migration 023: Note titles and summaries (v2.16.0, ADR-076)
-- An article's title moves out of its Markdown ('# Heading' first line) into
-- notes.title; migrations.py moves existing headings right after this runs.
-- notes.summary holds an author-written summary (Micropub 'summary').
ALTER TABLE notes ADD COLUMN title TEXT;
ALTER TABLE notes ADD COLUMN summary TEXT;
+8 -1
View File
@@ -1,4 +1,4 @@
# StarPunk runtime dependencies (v2.11.0)
# StarPunk runtime dependencies (v2.15.0)
# Pinned to a minor series; see docs/architecture/technology-stack.md.
# Web framework and server
@@ -19,6 +19,13 @@ httpx==0.28.*
PyJWT[crypto]==2.15.*
python-dotenv==1.2.*
# Notifications (v2.14.0, ADR-074)
apprise==2.2.*
# Images
Pillow==13.0.*
pillow-heif==1.8.*
# MCP server sidecar (v2.15.0, ADR-075): official SDK, served by uvicorn
mcp==2.3.*
uvicorn==0.54.*
+8 -5
View File
@@ -325,9 +325,12 @@ def create_app(config=None):
register_error_handlers(app)
# Background workers, unless this process is the MCP sidecar (v2.15.0, ADR-075)
workers = app.config.get("BACKGROUND_WORKERS_ENABLED", True)
# Start webmention worker thread (v1.6.0 Phase 2)
# Per background-worker.md: Skip in test mode
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
if workers and app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
from starpunk.webmention_worker import WebmentionWorker
webmention_worker = WebmentionWorker(
app,
@@ -348,13 +351,13 @@ def create_app(config=None):
# Start WebSub worker (v2.10.0)
from starpunk.websub_worker import start_worker
app.websub_worker = start_worker(app)
app.websub_worker = start_worker(app) if workers else None
if app.websub_worker:
app.logger.info("WebSub worker started")
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
if workers and targets_configured(app.config) and not app.config.get("TESTING", False):
app.syndication_worker = SyndicationWorker(
app, poll_interval=app.config["SYNDICATION_POLL_INTERVAL"]
)
@@ -465,5 +468,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.13.2"
__version_info__ = (2, 13, 2)
__version__ = "2.17.0"
__version_info__ = (2, 17, 0)
+40 -10
View File
@@ -25,6 +25,7 @@ from typing import Dict, List, Optional
from flask import current_app
from starpunk.database import get_db
from starpunk.utils import post_path
DAY = 24 * 3600
SCHEDULE = [ # (note younger than, check every), in seconds
@@ -64,7 +65,9 @@ class Harvest:
"""What one check found. complete=False: some list was cut short."""
responses: List[Response]
complete: bool = True
counts: Optional[Dict[str, int]] = None # Threads likes/reposts/quotes
# Likes/reposts/quotes shown only as numbers: all of them on Threads
# (its API doesn't say who); elsewhere, those by people who opted out
counts: Optional[Dict[str, int]] = None
# --- Text -----------------------------------------------------------------
@@ -162,7 +165,7 @@ def _due(db, limit: int) -> list:
lease = "(s.backfeed_claimed_until IS NULL OR s.backfeed_claimed_until < datetime('now'))"
rows = db.execute(
f"""
SELECT s.*, n.created_at AS note_created, n.slug AS note_slug {_ELIGIBLE}
SELECT s.*, n.created_at AS note_created, n.slug AS note_slug, n.title AS note_title {_ELIGIBLE}
AND {lease}
AND n.created_at >= datetime('now', ?)
AND (s.backfeed_next_at IS NULL OR s.backfeed_next_at <= datetime('now'))
@@ -174,7 +177,7 @@ def _due(db, limit: int) -> list:
return rows
backfill = db.execute(
f"""
SELECT s.*, n.created_at AS note_created, n.slug AS note_slug {_ELIGIBLE}
SELECT s.*, n.created_at AS note_created, n.slug AS note_slug, n.title AS note_title {_ELIGIBLE}
AND {lease} AND s.backfeed_backfill = 1
AND (s.backfeed_next_at IS NULL OR s.backfeed_next_at <= datetime('now'))
ORDER BY s.id LIMIT 1
@@ -243,11 +246,27 @@ def _display_state(db, note_id: int) -> list:
)]
def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest) -> None:
def hidden_counts(harvest: Harvest) -> Dict[str, int]:
"""
Likes, reposts and quotes to show only as numbers: the silo's own counts
when it gives no names (Threads), else those by people who opted out
(v2.13.3: they're counted, not named)
"""
if harvest.counts is not None:
return harvest.counts
hidden = [r.kind for r in harvest.responses if _opted_out(r)]
return {k: hidden.count(kind) for k, kind in
(("likes", "like"), ("reposts", "repost"), ("quotes", "quote"))}
def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest,
quiet: bool = False) -> None:
"""
Upsert a check's responses for one note and silo. With a complete
harvest, that silo's responses for the note that weren't found are
marked invalid. Responses from people who opted out are removed.
marked invalid. Responses from people who opted out are removed (they
are counted instead, see `hidden_counts`). With `quiet`, new responses
are stored as already notified (ADR-074 Decision 5).
"""
from starpunk.webmention_processor import _snippet
@@ -273,8 +292,9 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest) -> N
INSERT INTO webmentions (
source_url, target_url, target_note_id, mention_type, author_name, author_url,
author_photo, content_snippet, validation_status, moderation_status,
validated_at, source_published, origin, remote_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'valid', ?, CURRENT_TIMESTAMP, ?, ?, ?)
validated_at, source_published, origin, remote_id, notified_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'valid', ?, CURRENT_TIMESTAMP, ?, ?, ?,
CASE WHEN ? THEN CURRENT_TIMESTAMP END)
ON CONFLICT(source_url, target_url) DO UPDATE SET
target_note_id = excluded.target_note_id,
mention_type = excluded.mention_type,
@@ -291,7 +311,8 @@ def store(db, note_id: int, target_url: str, origin: str, harvest: Harvest) -> N
updated_at = CURRENT_TIMESTAMP
""",
(r.source_url, target_url, note_id, r.kind, r.author_name or None, r.author_url or None,
r.author_photo or None, snippet, moderation, r.published, origin, r.remote_id),
r.author_photo or None, snippet, moderation, r.published, origin, r.remote_id,
1 if quiet else 0),
)
if dropped:
marks = ",".join("?" * len(dropped))
@@ -357,8 +378,12 @@ def check(row, target) -> None:
db = get_db(current_app)
site = current_app.config["SITE_URL"].rstrip("/")
harvest = target.client.responses(row["remote_id"], row["remote_url"] or "")
store(db, row["note_id"], f"{site}/note/{row['note_slug']}", row["target"], harvest)
_store_counts(db, row["id"], harvest.counts)
# No notifications for history: a backfill, or a copy's first check on an
# older note (e.g. one linked after the fact). ADR-074 Decision 5.
quiet = bool(row["backfeed_backfill"]) or (
row["backfeed_checked_at"] is None and _age_seconds(row["note_created"]) > DAY)
store(db, row["note_id"], site + post_path(row["note_slug"], row["note_title"]), row["target"], harvest, quiet=quiet)
_store_counts(db, row["id"], hidden_counts(harvest))
def process_due(limit: int = 3) -> int:
@@ -382,6 +407,11 @@ def process_due(limit: int = 3) -> int:
if _is_denied(e):
denied[row["target"]] = f"Needs {SCOPES.get(row['target'], 'read access')}: {e}"
current_app.logger.error(f"Backfeed for {row['target']} stopped: {denied[row['target']]}")
from starpunk import notifications
site = current_app.config["SITE_URL"].rstrip("/")
notifications.problem(current_app.config, current_app.config["DATABASE_PATH"],
f"Backfeed for {SILO_NAMES.get(row['target'], row['target'])} stopped",
denied[row["target"]], f"{site}/admin/webmentions")
db.execute("UPDATE syndications SET backfeed_claimed_until = NULL WHERE id = ?", (row["id"],))
db.commit()
continue
+6
View File
@@ -113,6 +113,8 @@ def load_config(app, config_override=None):
app.config["WEBSUB_MAX_SUBSCRIPTIONS"] = int(os.getenv("WEBSUB_MAX_SUBSCRIPTIONS", "1000"))
app.config["WEBSUB_MAX_PER_HOST"] = int(os.getenv("WEBSUB_MAX_PER_HOST", "25"))
app.config["WEBSUB_POLL_INTERVAL"] = int(os.getenv("WEBSUB_POLL_INTERVAL", "5"))
# Off in the MCP sidecar so it never runs a second set of workers (v2.15.0, ADR-075)
app.config["BACKGROUND_WORKERS_ENABLED"] = os.getenv("BACKGROUND_WORKERS_ENABLED", "true").lower() == "true"
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
@@ -149,6 +151,10 @@ def load_config(app, config_override=None):
# Backfeed of responses to crossposts (v2.13.0, ADR-073)
app.config["BACKFEED_ENABLED"] = os.getenv("BACKFEED_ENABLED", "false").lower() == "true"
app.config["BACKFEED_MAX_AGE_DAYS"] = int(os.getenv("BACKFEED_MAX_AGE_DAYS", "90"))
# Notifications through Apprise (v2.14.0, ADR-074)
app.config["NOTIFY_URLS"] = os.getenv("NOTIFY_URLS", "")
app.config["NOTIFY_EVENTS"] = os.getenv("NOTIFY_EVENTS", "reply,mention,quote,problems")
app.config["NOTIFY_PRIORITY"] = os.getenv("NOTIFY_PRIORITY", "normal")
# Apply overrides if provided
# CSRF protection for admin and consent forms (v2.11.0); on by default.
+3 -1
View File
@@ -17,10 +17,12 @@ def init_db(app=None):
if app:
db_path = app.config["DATABASE_PATH"]
logger = app.logger
data_dir = app.config.get("DATA_PATH")
else:
# Fallback to default path
db_path = Path("./data/starpunk.db")
logger = None
data_dir = None
# Ensure parent directory exists
db_path.parent.mkdir(parents=True, exist_ok=True)
@@ -41,4 +43,4 @@ def init_db(app=None):
# Run migrations
from starpunk.migrations import run_migrations
run_migrations(db_path, logger=logger)
run_migrations(db_path, logger=logger, data_dir=data_dir)
+7 -3
View File
@@ -16,6 +16,7 @@ Standards:
"""
# Standard library imports
from html import escape
from datetime import datetime, timezone
from typing import Optional
import xml.etree.ElementTree as ET
@@ -160,7 +161,10 @@ def generate_atom_streaming(
# Required elements
yield f' <id>{_escape_xml(permalink)}</id>\n'
yield f' <title>{_escape_xml(note.title)}</title>\n'
# atom:title is required but may be empty: untitled notes (ADR-076)
yield f' <title>{_escape_xml(note.title or "")}</title>\n'
if note.summary:
yield f' <summary>{_escape_xml(note.summary)}</summary>\n'
# Use created_at for both published and updated
# (Note model doesn't have updated_at tracking yet)
@@ -199,8 +203,8 @@ def generate_atom_streaming(
html_content += '<div class="media">'
for item in note.media:
media_url = f"{site_url}/media/{item['path']}"
caption = item.get('caption', '')
html_content += f'<img src="{media_url}" alt="{caption}" />'
caption = escape(item.get('caption') or '', quote=True) # v2.14.1
html_content += f'<img src="{escape(media_url, quote=True)}" alt="{caption}" />'
html_content += '</div>'
# Add text content below media
+9 -4
View File
@@ -16,6 +16,7 @@ Standards:
"""
# Standard library imports
from html import escape
from datetime import datetime, timezone
from typing import Optional, Dict, Any
import json
@@ -228,8 +229,12 @@ def _build_item_object(site_url: str, note: Note) -> Dict[str, Any]:
"url": permalink,
}
# Add title
item["title"] = note.title
# Title and summary only when the note has them: untitled notes have
# no title (JSON Feed 1.1; v2.16.0, ADR-076)
if note.title:
item["title"] = note.title
if note.summary:
item["summary"] = note.summary
# Add image field (URL of first/main image) - per JSON Feed 1.1 spec
# Per Q7: Field should be absent (not null) when no media
@@ -247,8 +252,8 @@ def _build_item_object(site_url: str, note: Note) -> Dict[str, Any]:
content_html += '<div class="media">'
for media_item in note.media:
media_url = f"{site_url}/media/{media_item['path']}"
caption = media_item.get('caption', '')
content_html += f'<img src="{media_url}" alt="{caption}" />'
caption = escape(media_item.get('caption') or '', quote=True) # v2.14.1
content_html += f'<img src="{escape(media_url, quote=True)}" alt="{caption}" />'
content_html += '</div>'
# Add text content below media
+19 -41
View File
@@ -20,6 +20,7 @@ Standards:
"""
# Standard library imports
from html import escape
from datetime import datetime, timezone
from typing import Optional
@@ -107,7 +108,9 @@ def generate_rss(
# Set required item elements
fe.id(permalink)
fe.title(get_note_title(note))
title = get_note_title(note)
if title: # untitled notes have only a description (RSS 2.0, ADR-076)
fe.title(title)
fe.link(href=permalink)
fe.guid(permalink, permalink=True)
@@ -127,8 +130,8 @@ def generate_rss(
html_content += '<div class="media">'
for item in note.media:
media_url = f"{site_url}/media/{item['path']}"
caption = item.get('caption', '')
html_content += f'<img src="{media_url}" alt="{caption}" />'
caption = escape(item.get('caption') or '', quote=True) # v2.14.1
html_content += f'<img src="{escape(media_url, quote=True)}" alt="{caption}" />'
html_content += '</div>'
# Add text content below media
@@ -254,16 +257,16 @@ def generate_rss_streaming(
html_content += '<div class="media">'
for item in note.media:
media_url = f"{site_url}/media/{item['path']}"
caption = item.get('caption', '')
html_content += f'<img src="{media_url}" alt="{caption}" />'
caption = escape(item.get('caption') or '', quote=True) # v2.14.1
html_content += f'<img src="{escape(media_url, quote=True)}" alt="{caption}" />'
html_content += '</div>'
# Add text content below media
html_content += clean_html_for_rss(note.feed_html)
# Build item XML
item_xml = f""" <item>
<title>{_escape_xml(title)}</title>
title_xml = f"\n <title>{_escape_xml(title)}</title>" if title else ""
item_xml = f""" <item>{title_xml}
<link>{_escape_xml(permalink)}</link>
<guid isPermaLink="true">{_escape_xml(permalink)}</guid>
<pubDate>{pub_date_str}</pubDate>"""
@@ -493,48 +496,23 @@ def format_rfc822_date(dt: datetime) -> str:
return dt.strftime("%a, %d %b %Y %H:%M:%S %z")
def get_note_title(note: Note) -> str:
def get_note_title(note: Note) -> Optional[str]:
"""
Extract title from note content
An item's title: the note's title (articles), cut to 100 characters
Attempts to extract a meaningful title from the note. Uses the first
line of content (stripped of markdown heading syntax) or falls back
to a formatted timestamp if content is unavailable.
Algorithm:
1. Try note.title property (first line, stripped of # syntax)
2. Fall back to timestamp if title is unavailable
Untitled notes get none (v2.16.0, ADR-076), so readers can tell notes
from articles; RSS 2.0 lets an item have just a description.
Args:
note: Note object
Returns:
Title string (max 100 chars, truncated if needed)
Examples:
>>> # Note with heading
>>> note = Note(...) # content: "# My First Note\\n\\n..."
>>> get_note_title(note)
'My First Note'
>>> # Note without heading (timestamp fallback)
>>> note = Note(...) # content: "Just some text"
>>> get_note_title(note)
'November 18, 2024 at 12:00 PM'
Title string, or None for an untitled note
"""
try:
# Use Note's title property (handles extraction logic)
title = note.title
# Truncate to 100 characters for RSS compatibility
if len(title) > 100:
title = title[:100].strip() + "..."
return title
except (FileNotFoundError, OSError, AttributeError):
# If title extraction fails, use timestamp
return note.created_at.strftime("%B %d, %Y at %I:%M %p")
title = note.title
if title and len(title) > 100:
title = title[:100].strip() + "..."
return title or None
def clean_html_for_rss(html: str) -> str:
+57 -17
View File
@@ -25,7 +25,11 @@ from starpunk.database import get_db
from starpunk.utils import absolute_url_for
CODE_LIFETIME_MINUTES = 10
SUPPORTED_SCOPES = ["create", "update", "delete", "media", "profile"]
SUPPORTED_SCOPES = ["create", "update", "delete", "media", "draft", "profile"]
# Scopes offered to an MCP client that asks for none (v2.15.0, ADR-075):
# the owner unticks what they don't want on the consent screen
MCP_DEFAULT_SCOPES = ["create", "update", "delete", "media", "draft"]
class AuthorizationRequestError(Exception):
@@ -56,6 +60,11 @@ def issuer() -> str:
return current_app.config["SITE_URL"].rstrip("/") + "/"
def mcp_resource() -> str:
"""The MCP endpoint's resource identifier (RFC 8707, v2.15.0)"""
return current_app.config["SITE_URL"].rstrip("/") + "/mcp"
def admin_me() -> str:
"""The single identity StarPunk issues tokens for"""
return current_app.config["ADMIN_ME"]
@@ -134,7 +143,7 @@ def validate_authorization_request(params: Dict[str, str]) -> Dict[str, Any]:
Returns:
Normalized request: client_id, redirect_uri, state, scopes,
code_challenge
code_challenge, resource
Raises:
AuthorizationRequestError
@@ -176,17 +185,31 @@ def validate_authorization_request(params: Dict[str, str]) -> Dict[str, Any]:
scopes = [s for s in (params.get("scope") or "").split() if s]
# RFC 8707 resource indicator: only the MCP endpoint is a named resource
# here (v2.15.0, ADR-075). Micropub clients don't send one.
resource = (params.get("resource") or "").strip() or None
if resource is not None:
if resource.rstrip("/") != mcp_resource():
raise AuthorizationRequestError(
"invalid_target", "Unknown resource; this server only issues tokens for its MCP endpoint"
)
resource = mcp_resource()
if not scopes:
scopes = list(MCP_DEFAULT_SCOPES)
return {
"client_id": client_id,
"redirect_uri": redirect_uri,
"state": params["state"],
"scopes": scopes,
"code_challenge": code_challenge,
"resource": resource,
}
def create_authorization_code(
client_id: str, redirect_uri: str, scope: str, code_challenge: str
client_id: str, redirect_uri: str, scope: str, code_challenge: str,
resource: Optional[str] = None,
) -> str:
"""Issue a single-use authorization code (stored hashed)"""
code = secrets.token_urlsafe(32)
@@ -195,8 +218,8 @@ def create_authorization_code(
db.execute(
"""
INSERT INTO indieauth_codes
(code_hash, client_id, redirect_uri, scope, code_challenge, expires_at)
VALUES (?, ?, ?, ?, ?, ?)
(code_hash, client_id, redirect_uri, scope, code_challenge, expires_at, resource)
VALUES (?, ?, ?, ?, ?, ?, ?)
""",
(
_hash(code),
@@ -205,6 +228,7 @@ def create_authorization_code(
scope,
code_challenge,
datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=CODE_LIFETIME_MINUTES),
resource,
),
)
db.commit()
@@ -218,7 +242,7 @@ def redeem_authorization_code(
Redeem an authorization code exactly once
Returns:
{"me": ..., "scope": ..., "client_id": ...}
{"me": ..., "scope": ..., "client_id": ..., "resource": ...}
Raises:
InvalidGrantError
@@ -240,7 +264,7 @@ def redeem_authorization_code(
raise InvalidGrantError("Authorization code is invalid, expired or already used")
row = db.execute(
"SELECT client_id, redirect_uri, scope, code_challenge FROM indieauth_codes WHERE code_hash = ?",
"SELECT client_id, redirect_uri, scope, code_challenge, resource FROM indieauth_codes WHERE code_hash = ?",
(_hash(code),),
).fetchone()
@@ -252,11 +276,20 @@ def redeem_authorization_code(
if not secrets.compare_digest(challenge, row["code_challenge"]):
raise InvalidGrantError("PKCE verification failed")
return {"me": admin_me(), "scope": row["scope"], "client_id": row["client_id"]}
return {
"me": admin_me(),
"scope": row["scope"],
"client_id": row["client_id"],
"resource": row["resource"],
}
def create_access_token(client_id: str, scope: str) -> Dict[str, Any]:
"""Issue an access token (stored hashed); returns the token response"""
def create_access_token(client_id: str, scope: str, resource: Optional[str] = None) -> Dict[str, Any]:
"""Issue an access token (stored hashed); returns the token response
resource is the RFC 8707 resource the token is for (the MCP endpoint),
or None for Micropub tokens.
"""
token = secrets.token_urlsafe(32)
lifetime_days = current_app.config.get("INDIEAUTH_TOKEN_LIFETIME_DAYS", 90)
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=lifetime_days)
@@ -264,10 +297,10 @@ def create_access_token(client_id: str, scope: str) -> Dict[str, Any]:
db = get_db(current_app)
db.execute(
"""
INSERT INTO indieauth_tokens (token_hash, client_id, scope, expires_at)
VALUES (?, ?, ?, ?)
INSERT INTO indieauth_tokens (token_hash, client_id, scope, expires_at, resource)
VALUES (?, ?, ?, ?, ?)
""",
(_hash(token), client_id, scope, expires_at),
(_hash(token), client_id, scope, expires_at, resource),
)
db.commit()
@@ -286,14 +319,15 @@ def verify_access_token(token: str) -> Optional[Dict[str, Any]]:
Check a bearer token issued by StarPunk
Returns:
{"me", "client_id", "scope"} if active, otherwise None
{"me", "client_id", "scope", "resource", "expires_at"} if active,
otherwise None
"""
if not token:
return None
db = get_db(current_app)
row = db.execute(
"""
SELECT id, client_id, scope FROM indieauth_tokens
SELECT id, client_id, scope, resource, expires_at FROM indieauth_tokens
WHERE token_hash = ? AND revoked_at IS NULL AND expires_at > datetime('now')
""",
(_hash(token),),
@@ -306,7 +340,13 @@ def verify_access_token(token: str) -> Optional[Dict[str, Any]]:
(row["id"],),
)
db.commit()
return {"me": admin_me(), "client_id": row["client_id"], "scope": row["scope"]}
return {
"me": admin_me(),
"client_id": row["client_id"],
"scope": row["scope"],
"resource": row["resource"],
"expires_at": row["expires_at"],
}
def revoke_access_token(token: str) -> None:
@@ -334,7 +374,7 @@ def list_active_tokens() -> List[Dict[str, Any]]:
db = get_db(current_app)
rows = db.execute(
"""
SELECT id, client_id, scope, created_at, expires_at, last_used_at
SELECT id, client_id, scope, resource, created_at, expires_at, last_used_at
FROM indieauth_tokens
WHERE revoked_at IS NULL AND expires_at > datetime('now')
ORDER BY created_at DESC
+25 -2
View File
@@ -1,7 +1,12 @@
"""
Markdown extension: turn bare URLs into links (v2.3.0)
Markdown extensions for notes
Python-Markdown only links <https://...>; notes often contain pasted URLs.
BareUrlExtension: turn bare URLs into links (v2.3.0). Python-Markdown only
links <https://...>; notes often contain pasted URLs.
HashtagLineExtension: a line starting with a hashtag stays text (v2.16.0).
Python-Markdown makes '#indieweb is fun' a heading; CommonMark (and every
other Markdown a client may use) needs a space after the '#'.
"""
import re
@@ -9,6 +14,7 @@ import xml.etree.ElementTree as etree
from markdown.extensions import Extension
from markdown.inlinepatterns import InlineProcessor
from markdown.preprocessors import Preprocessor
from markdown.treeprocessors import Treeprocessor
from markdown.util import AtomicString
@@ -40,3 +46,20 @@ class BareUrlExtension(Extension):
# Below links (160) and <autolinks> (120), so those win
md.inlinePatterns.register(_BareUrl(BARE_URL_RE, md), "bare_url", 105)
md.treeprocessors.register(_UnnestLinks(md), "unnest_links", 0)
# Up to 3 spaces, then '#'s straight followed by text: a hashtag, not a heading
_HASHTAG_LINE_RE = re.compile(r"^( {0,3})(?=#+[^\s#])")
class _HashtagLines(Preprocessor):
"""Escape the first '#' of such a line, so it renders as text"""
def run(self, lines):
return [_HASHTAG_LINE_RE.sub(r"\1\\", line) for line in lines]
class HashtagLineExtension(Extension):
def extendMarkdown(self, md):
# After fenced code (25) and raw HTML (20) are set aside, so code is untouched
md.preprocessors.register(_HashtagLines(md), "hashtag_lines", 15)
+13
View File
@@ -0,0 +1,13 @@
"""
MCP server for StarPunk (v2.15.0, ADR-075)
AI clients (claude.ai connectors, Claude Code, ...) read and write notes
through the Model Context Protocol at SITE_URL/mcp. It runs as a sidecar
process (uvicorn, the official MCP Python SDK) next to the Flask app and
calls the same service functions. StarPunk's own IndieAuth server issues the
tokens; this server only checks them (an OAuth resource server).
"""
from starpunk.mcp_server.server import create_asgi_app, create_mcp_server
__all__ = ["create_asgi_app", "create_mcp_server"]
+13
View File
@@ -0,0 +1,13 @@
"""
ASGI entry point for the MCP sidecar (v2.15.0, ADR-075)
uvicorn starpunk.mcp_server.app:app --host 0.0.0.0 --port 8001
Run it with BACKGROUND_WORKERS_ENABLED=false and the same environment and
data volume as the main app. Route /mcp and
/.well-known/oauth-protected-resource/mcp to it.
"""
from starpunk.mcp_server.server import create_asgi_app
app = create_asgi_app()
+56
View File
@@ -0,0 +1,56 @@
"""
Bearer token checks for the MCP endpoint (v2.15.0, ADR-075)
Tokens are StarPunk IndieAuth tokens bound to the MCP resource (RFC 8707).
The SDK's bearer middleware turns a missing or rejected token into a 401
that points the client at the protected resource metadata, and refuses
tokens whose resource isn't this endpoint (validate_token_resource).
"""
from datetime import datetime, timezone
from typing import Optional
import anyio.to_thread
from flask import Flask
from mcp.server.auth.provider import AccessToken
from starpunk.indieauth import verify_access_token
def _epoch(value) -> Optional[int]:
"""SQLite timestamp (datetime or 'YYYY-MM-DD HH:MM:SS' UTC) -> epoch seconds"""
if value is None:
return None
if isinstance(value, str):
try:
value = datetime.fromisoformat(value)
except ValueError:
return None
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return int(value.timestamp())
class StarPunkTokenVerifier:
"""TokenVerifier backed by the indieauth_tokens table"""
def __init__(self, app: Flask):
self.app = app
def _verify(self, token: str) -> Optional[AccessToken]:
with self.app.app_context():
info = verify_access_token(token)
if not info:
return None
return AccessToken(
token=token,
client_id=info["client_id"],
scopes=(info["scope"] or "").split(),
expires_at=_epoch(info.get("expires_at")),
resource=info.get("resource"),
subject=info["me"],
)
async def verify_token(self, token: str) -> Optional[AccessToken]:
# Database work happens off the event loop
return await anyio.to_thread.run_sync(self._verify, token)
+95
View File
@@ -0,0 +1,95 @@
"""
The MCP server and its ASGI app (v2.15.0, ADR-075)
Stateless Streamable HTTP with plain JSON responses: every request stands
alone, so any number of uvicorn workers can serve it and nothing is lost on
restart.
"""
from typing import Optional
from urllib.parse import urlparse
from flask import Flask
from mcp.server import MCPServer
from mcp.server.auth.settings import AuthSettings
from mcp.server.transport_security import TransportSecuritySettings
from pydantic import AnyHttpUrl
from starlette.applications import Starlette
from starpunk.mcp_server.auth import StarPunkTokenVerifier
from starpunk.mcp_server.tools import register_tools
INSTRUCTIONS = """\
StarPunk is an IndieWeb site: a personal blog of short notes and longer
articles, owned by one person. Notes are Markdown. A note whose first line
is '# Heading' is an article with that title. Notes can be drafts
(unpublished) or published; publishing may crosspost them to social
networks chosen with syndicate_to (see get_site_config). Prefer drafts unless
the owner asked to publish. Deleting is a soft delete that undelete_note
reverses.
"""
def _site(app: Flask) -> str:
return app.config["SITE_URL"].rstrip("/")
def create_mcp_server(app: Flask) -> MCPServer:
"""The MCP server, with StarPunk's tools and token checks"""
site = _site(app)
server = MCPServer(
name="StarPunk",
instructions=INSTRUCTIONS,
website_url=site,
version=app.config.get("VERSION", ""),
token_verifier=StarPunkTokenVerifier(app),
auth=AuthSettings(
# Same string as the issuer in /.well-known/oauth-authorization-server
issuer_url=AnyHttpUrl(site + "/"),
resource_server_url=AnyHttpUrl(site + "/mcp"),
# Tools check their own scopes; any StarPunk token may connect
required_scopes=[],
validate_token_resource=True,
),
)
register_tools(server, app)
return server
def _transport_security(app: Flask) -> TransportSecuritySettings:
"""Only requests for this site's host and from its origin (DNS rebinding)"""
site = urlparse(_site(app))
hosts = [site.netloc]
origins = [f"{site.scheme}://{site.netloc}"]
if app.config.get("DEV_MODE"):
hosts += ["localhost:*", "127.0.0.1:*"]
origins += ["http://localhost:*", "http://127.0.0.1:*"]
return TransportSecuritySettings(
enable_dns_rebinding_protection=True,
allowed_hosts=hosts,
allowed_origins=origins,
)
def create_asgi_app(app: Optional[Flask] = None) -> Starlette:
"""
ASGI app serving /mcp and its protected resource metadata
Args:
app: The Flask app to call into. Built from the environment if not
given, with background workers off (the main app runs those).
"""
if app is None:
from starpunk import create_app
app = create_app({"BACKGROUND_WORKERS_ENABLED": False})
server = create_mcp_server(app)
return server.streamable_http_app(
streamable_http_path="/mcp",
stateless_http=True,
json_response=True,
# Base64 photos for upload_media: 10 MB of image is ~14 MB encoded
max_request_body_size=16 * 1024 * 1024,
transport_security=_transport_security(app),
)
+370
View File
@@ -0,0 +1,370 @@
"""
MCP tools for StarPunk (v2.15.0, ADR-075)
Reading tools call the notes, tags and search services. Writing tools build
a Micropub request and hand it to the Micropub handlers, so an MCP post goes
through exactly the same checks and side effects (scopes, drafts-only
tokens, crossposting, webmentions, WebSub) as one from any Micropub client.
Scopes are the Micropub ones: reading needs any StarPunk token; drafts and
tags need create, update, delete or draft; writing needs create (or draft),
update, delete or media.
"""
import base64
import binascii
from pathlib import Path
from typing import Any, Callable, Optional
from urllib.parse import urlparse
import anyio.to_thread
from flask import Flask
from mcp.server import MCPServer
from mcp.server.auth.middleware.auth_context import get_access_token
from mcp.server.mcpserver.exceptions import ToolError
from mcp.types import ToolAnnotations
from pydantic import BaseModel, Field
from starpunk.indieauth import check_scope
from starpunk.micropub import MicropubError
DRAFT_SCOPES = ("create", "update", "delete", "draft")
MAX_LIMIT = 100
class Photo(BaseModel):
"""A photo for a note"""
url: str = Field(description="URL from upload_media, or any public image URL")
alt: str = Field(default="", description="Alt text describing the photo")
def _token_info() -> dict:
"""The caller's token as the Micropub handlers expect it"""
token = get_access_token()
if token is None:
raise ToolError("Not authenticated")
return {"me": token.subject, "client_id": token.client_id, "scope": " ".join(token.scopes)}
def _require(info: dict, *scopes: str) -> None:
if not any(check_scope(s, info["scope"]) for s in scopes):
raise ToolError(f"This needs the {' or '.join(scopes)} scope; reconnect and grant it")
def _sees_drafts(info: dict) -> bool:
return any(check_scope(s, info["scope"]) for s in DRAFT_SCOPES)
def _clamp(limit: int) -> int:
return max(1, min(int(limit), MAX_LIMIT))
def _note_url(slug: str, title=None) -> str:
from starpunk.utils import post_url
return post_url(slug, title)
def _summary(note) -> dict:
from starpunk.utils import iso_utc
return {
"url": note.url,
"slug": note.slug,
"title": note.title,
"summary": note.summary,
"kind": note.kind,
"status": "published" if note.published else "draft",
"published": iso_utc(note.created_at),
"excerpt": note.excerpt,
}
def _find_note(url_or_slug: str):
"""A note by its URL on this site or its slug"""
from starpunk.micropub import _note_from_url
from starpunk.notes import get_note
value = (url_or_slug or "").strip()
if urlparse(value).scheme:
return _note_from_url(value)
note = get_note(slug=value.strip("/").split("/")[-1]) if value else None
if note is None:
raise ToolError("Note not found")
return note
def _status_after(url: str) -> dict:
"""Summary of the note at url, as it is now"""
return _summary(_find_note(url))
def register_tools(server: MCPServer, app: Flask) -> None:
"""Add StarPunk's tools to the MCP server"""
site = app.config["SITE_URL"].rstrip("/")
async def run(fn: Callable[..., Any], *args, **kwargs) -> Any:
"""Run fn in a worker thread inside a Flask request context for SITE_URL"""
def call():
with app.test_request_context(base_url=site):
try:
return fn(*args, **kwargs)
except MicropubError as e:
raise ToolError(e.error_description or e.error) from e
except ValueError as e:
raise ToolError(str(e)) from e
return await anyio.to_thread.run_sync(call)
read_only = ToolAnnotations(read_only_hint=True, open_world_hint=False)
# Reading
@server.tool(annotations=read_only)
async def search_notes(query: str, include_drafts: bool = False, limit: int = 20) -> dict:
"""Full-text search of the site's notes. Returns matches with a snippet;
use get_note for the whole note."""
info = _token_info()
if include_drafts:
_require(info, *DRAFT_SCOPES)
def search():
from starpunk.search import has_fts_table, search_notes as fts
db_path = Path(app.config["DATABASE_PATH"])
if not has_fts_table(db_path):
raise ToolError("Search is not available on this server")
results = fts(query=query, db_path=db_path, published_only=not include_drafts, limit=_clamp(limit))
return {
"results": [
{"url": _note_url(r["slug"], r.get("article_title")), "slug": r["slug"], "title": r.get("title"),
"snippet": r.get("snippet")}
for r in results
]
}
return await run(search)
@server.tool(annotations=read_only)
async def list_notes(include_drafts: bool = False, limit: int = 20, offset: int = 0) -> dict:
"""The site's notes, newest first. Drafts are left out unless
include_drafts is true."""
info = _token_info()
if include_drafts:
_require(info, *DRAFT_SCOPES)
def notes():
from starpunk.notes import list_notes as list_all
found = list_all(published_only=not include_drafts, limit=_clamp(limit), offset=max(0, offset))
return {"notes": [_summary(n) for n in found], "offset": max(0, offset)}
return await run(notes)
@server.tool(annotations=read_only)
async def get_note(url_or_slug: str) -> dict:
"""One note in full: Markdown content, title (name), tags (category),
photos, status, reply/like/repost/bookmark target and crosspost links,
as Microformats2 properties."""
info = _token_info()
def one():
from starpunk.micropub import note_to_mf2
note = _find_note(url_or_slug)
if not note.published and not _sees_drafts(info):
raise ToolError("Note not found")
return {"url": note.url, "slug": note.slug,
"properties": note_to_mf2(note)["properties"]}
return await run(one)
@server.tool(annotations=read_only)
async def list_tags(filter: str = "") -> dict:
"""Tags in use on the site, most used first, optionally only those
containing filter."""
info = _token_info()
_require(info, *DRAFT_SCOPES)
def tags():
from starpunk.micropub import _all_categories
return {"tags": _all_categories(filter)}
return await run(tags)
@server.tool(annotations=read_only)
async def get_site_config() -> dict:
"""What the site supports: post types and the crosspost targets
(syndicate-to uids) that create_note's syndicate_to accepts."""
info = _token_info()
def config():
from starpunk.micropub import handle_query
response = handle_query({"q": "config"}, info)
# Flask views may return (response, status[, headers])
if isinstance(response, tuple):
response = response[0]
return response.get_json()
return await run(config)
# Writing
@server.tool(annotations=ToolAnnotations(destructive_hint=False, open_world_hint=True))
async def create_note(
content: str = "",
title: Optional[str] = None,
summary: Optional[str] = None,
tags: Optional[list[str]] = None,
draft: bool = True,
photos: Optional[list[Photo]] = None,
in_reply_to: Optional[str] = None,
like_of: Optional[str] = None,
repost_of: Optional[str] = None,
bookmark_of: Optional[str] = None,
syndicate_to: Optional[list[str]] = None,
slug: Optional[str] = None,
published_at: Optional[str] = None,
) -> dict:
"""Create a note (Markdown). It is a draft unless draft is false;
publishing makes it public and crossposts it to the syndicate_to
targets. A title makes it an article; summary is its short summary,
shown on the homepage and in feeds and crossposts. At most one of like_of,
repost_of and bookmark_of. published_at is an ISO 8601 date."""
info = _token_info()
_require(info, "create", "draft")
props: dict[str, list] = {"post-status": ["draft" if draft else "published"]}
if content:
props["content"] = [content]
if title:
props["name"] = [title]
if summary:
props["summary"] = [summary]
if tags:
props["category"] = list(tags)
if photos:
props["photo"] = [{"value": p.url, "alt": p.alt} for p in photos]
for name, value in (("in-reply-to", in_reply_to), ("like-of", like_of),
("repost-of", repost_of), ("bookmark-of", bookmark_of),
("published", published_at)):
if value:
props[name] = [value]
data: dict[str, Any] = {"type": ["h-entry"], "properties": props}
if syndicate_to:
data["mp-syndicate-to"] = list(syndicate_to)
if slug:
data["mp-slug"] = slug
def create():
from starpunk.micropub import handle_create
_, _, headers = handle_create(data, info)
return _status_after(headers["Location"])
return await run(create)
@server.tool(annotations=ToolAnnotations(destructive_hint=True, idempotent_hint=True, open_world_hint=True))
async def update_note(
url: str,
content: Optional[str] = None,
title: Optional[str] = None,
summary: Optional[str] = None,
tags: Optional[list[str]] = None,
add_tags: Optional[list[str]] = None,
remove_tags: Optional[list[str]] = None,
draft: Optional[bool] = None,
photos: Optional[list[Photo]] = None,
in_reply_to: Optional[str] = None,
) -> dict:
"""Change a note. Only the fields given change: content replaces the
text, title replaces the title ("" removes it), summary likewise, tags replaces all
tags, add_tags/remove_tags adjust them, photos replaces all photos,
draft false publishes a draft (and crossposts it) and true unpublishes.
Edits to published notes follow to their crossposts."""
info = _token_info()
_require(info, "update")
replace: dict[str, list] = {}
if content is not None:
replace["content"] = [content]
if title is not None:
replace["name"] = [title]
if summary is not None:
replace["summary"] = [summary]
if tags is not None:
replace["category"] = list(tags)
if draft is not None:
replace["post-status"] = ["draft" if draft else "published"]
if photos is not None:
replace["photo"] = [{"value": p.url, "alt": p.alt} for p in photos]
if in_reply_to is not None:
replace["in-reply-to"] = [in_reply_to]
data: dict[str, Any] = {"action": "update", "url": url, "replace": replace}
if add_tags:
data["add"] = {"category": list(add_tags)}
if remove_tags:
data["delete"] = {"category": list(remove_tags)}
def update():
from starpunk.micropub import handle_update
note = _find_note(url)
data["url"] = note.url
handle_update(data, info)
return _status_after(data["url"])
return await run(update)
@server.tool(annotations=ToolAnnotations(destructive_hint=True, idempotent_hint=True, open_world_hint=False))
async def delete_note(url: str) -> dict:
"""Delete a note. It can be restored with undelete_note."""
info = _token_info()
_require(info, "delete")
def delete():
from starpunk.micropub import handle_delete
note_url = _find_note(url).url
handle_delete({"url": note_url}, info)
return {"url": note_url, "status": "deleted"}
return await run(delete)
@server.tool(annotations=ToolAnnotations(destructive_hint=False, idempotent_hint=True, open_world_hint=False))
async def undelete_note(url: str) -> dict:
"""Restore a deleted note."""
info = _token_info()
_require(info, "delete")
note_url = url if urlparse(url).scheme else f"{site}/note/{url.strip('/')}"
def undelete():
from starpunk.micropub import handle_delete
handle_delete({"url": note_url}, info, undelete=True)
return _status_after(note_url)
return await run(undelete)
@server.tool(annotations=ToolAnnotations(destructive_hint=False, open_world_hint=False))
async def upload_media(filename: str, data_base64: str) -> dict:
"""Upload an image (JPEG, PNG, GIF, WebP or HEIC, base64-encoded) and
get its URL, to use as a photo in create_note or update_note."""
info = _token_info()
_require(info, "media", "create")
try:
data = base64.b64decode(data_base64, validate=True)
except (binascii.Error, ValueError):
raise ToolError("data_base64 is not valid base64")
def upload():
from starpunk.media import save_media
media = save_media(data, filename)
return {"url": f"{site}/media/{media['path']}"}
return await run(upload)
+98 -40
View File
@@ -7,7 +7,8 @@ interface for creating posts via external clients.
Functions:
normalize_properties: Convert form/JSON data to Micropub properties format
extract_content: Get content from Micropub properties
extract_title: Get the explicit title (name) from Micropub properties
extract_title: Get the title (name) from Micropub properties
extract_summary: Get the author's summary from Micropub properties
extract_tags: Get category tags from Micropub properties
handle_create: Process Micropub create action
handle_update: Process Micropub update action (v2.4.0)
@@ -204,10 +205,11 @@ def extract_content(properties: dict) -> str:
content = content_list[0]
# Handle structured content ({"html": "...", "text": "..."})
# Handle structured content ({"html": "...", "text": "..."}). The HTML is
# what was written; clients send text as a plain fallback, which loses
# images, links and headings (v2.16.0; before, text was preferred)
if isinstance(content, dict):
# Prefer text over html for markdown storage
content = content.get("text") or content.get("html", "")
content = content.get("html") or content.get("text", "")
if not isinstance(content, str) or not content.strip():
raise MicropubValidationError("Content cannot be empty")
@@ -216,9 +218,9 @@ def extract_content(properties: dict) -> str:
def content_format(properties: dict) -> Optional[str]:
"""'html' if the content was sent only as HTML (returned as such by q=source)"""
"""'html' if the content was sent as HTML (returned as such by q=source)"""
content = _first(properties.get("content"))
if isinstance(content, dict) and content.get("html") and not content.get("text"):
if isinstance(content, dict) and content.get("html"):
return "html"
return None
@@ -235,10 +237,7 @@ def _has_text(content) -> bool:
def extract_title(properties: dict) -> Optional[str]:
"""
Extract the explicit title (Micropub 'name' property)
StarPunk has no title field: a note's title is a '# Heading' first line
(see Note.title), so the caller stores the name as that heading.
Extract the title (Micropub 'name' property), stored in notes.title (ADR-076)
Args:
properties: Normalized Micropub properties dict
@@ -252,14 +251,22 @@ def extract_title(properties: dict) -> Optional[str]:
return " ".join(name.split()) or None
def add_title_heading(content: str, title: Optional[str]) -> str:
"""Prefix content with '# title' unless it already starts with that heading"""
if not title:
return content
first_line = content.lstrip().split("\n", 1)[0]
if first_line.startswith("#") and first_line.lstrip("#").strip() == title:
return content
return f"# {title}\n\n{content}"
def extract_summary(properties: dict) -> Optional[str]:
"""The author's summary (Micropub 'summary' property, v2.16.0), as plain text"""
summary = _first(properties.get("summary"))
if isinstance(summary, dict):
summary = summary.get("text") or summary.get("value") or ""
if not isinstance(summary, str):
return None
return summary.strip() or None
def drop_title_heading(content: str, title: Optional[str]) -> str:
"""Content without a '# heading' first line that only repeats the title"""
heading, rest = split_title(content)
if title and heading == title:
return rest
return content
def extract_tags(properties: dict) -> list[str]:
@@ -308,7 +315,7 @@ MAX_PHOTOS = 4 # ADR-057
MAX_REMOTE_PHOTO_BYTES = 20 * 1024 * 1024
# Properties we can store, and so can be updated (v2.4.0)
UPDATABLE = {"content", "name", "category", "in-reply-to", "post-status", "photo"}
UPDATABLE = {"content", "name", "summary", "category", "in-reply-to", "post-status", "photo"}
def _first(value):
@@ -407,9 +414,10 @@ def _public_address(host: str) -> bool:
return bool(infos)
def _fetch_remote_photo(url: str) -> Optional[int]:
def _fetch_remote_photo(url: str) -> Optional[dict]:
"""
Download a photo hosted elsewhere into the media library. Returns media id.
Download a photo hosted elsewhere into the media library. Returns the
saved media ({'id', 'path', ...}), or None if it couldn't be fetched.
Uses the webmention fetcher (v2.9.0): every redirect is checked against
private addresses and the connection is pinned to the checked address.
@@ -426,7 +434,7 @@ def _fetch_remote_photo(url: str) -> Optional[int]:
if result.status_code != 200:
raise ValueError(f"HTTP {result.status_code}")
name = urlparse(result.url).path.rsplit("/", 1)[-1] or "photo.jpg"
return save_media(result.content, name)["id"]
return save_media(result.content, name)
except Exception as e:
current_app.logger.warning(f"Could not fetch photo {url}: {e}")
return None
@@ -446,7 +454,35 @@ def _media_id_for_url(url: str) -> Optional[int]:
return row[0]
current_app.logger.warning(f"Photo URL not found in media: {url}")
return None
return _fetch_remote_photo(url)
media = _fetch_remote_photo(url)
return media["id"] if media else None
MAX_INLINE_IMAGES = 20
def localize_inline_images(content: str) -> str:
"""
Copy images embedded in the text from other sites into the media library
and point the text at the copies (v2.16.0), so an article doesn't depend
on the client's or a CDN's host staying up. Best effort: an image that
can't be fetched keeps its URL.
"""
from html import escape
from starpunk.syndication import inline_images
if not content or "<img" not in content and "![" not in content:
return content
site_url = current_app.config.get("SITE_URL", "").rstrip("/")
remote = [url for url, _ in inline_images(content)
if url.startswith(("http://", "https://")) and not url.startswith(site_url + "/")]
for url in list(dict.fromkeys(remote))[:MAX_INLINE_IMAGES]:
media = _fetch_remote_photo(url)
if media:
local = f"{site_url}/media/{media['path']}"
# As written in Markdown, and as it may be escaped in HTML attributes
content = content.replace(url, local).replace(escape(url), local)
return content
def _save_uploads(files) -> list[dict]:
@@ -513,7 +549,7 @@ def _check_type(data: dict) -> None:
def _note_from_url(url: Optional[str], deleted: bool = False):
"""Find the note a URL points at (/note/<slug> on this site)"""
"""Find the note a URL points at (/note/<slug> or /article/<slug> on this site)"""
from urllib.parse import urlparse
from starpunk.database import get_db
@@ -524,7 +560,7 @@ def _note_from_url(url: Optional[str], deleted: bool = False):
if parsed.netloc and site.netloc and parsed.netloc != site.netloc:
raise MicropubValidationError("URL is not on this site")
parts = [p for p in parsed.path.split("/") if p]
if len(parts) != 2 or parts[0] not in ("note", "notes"):
if len(parts) != 2 or parts[0] not in ("note", "notes", "article", "articles"):
raise MicropubValidationError("URL is not a post on this site")
slug = parts[1]
if deleted:
@@ -596,6 +632,10 @@ def handle_create(data: dict, token_info: dict, files=None):
if has_content or not (like_of or repost_of or bookmark_of or photos or title)
else ""
)
content = localize_inline_images(drop_title_heading(content, title))
# A summary with content is the author's summary; alone it is the
# content (Post Type Discovery), as extract_content does (v2.16.0)
summary = extract_summary(properties) if _has_text(properties.get("content")) else None
tags = extract_tags(properties)
published_date = extract_published_date(properties)
published = extract_post_status(properties) and not extract_visibility_draft(properties)
@@ -615,7 +655,9 @@ def handle_create(data: dict, token_info: dict, files=None):
note = None
try:
note = create_note(
content=add_title_heading(content, title),
content=content,
title=title,
summary=summary,
published=published,
created_at=published_date,
custom_slug=custom_slug,
@@ -636,7 +678,7 @@ def handle_create(data: dict, token_info: dict, files=None):
from starpunk.syndication import resolve_targets, select_targets
select_targets(note.id, resolve_targets(requested_targets))
return "", 201, {"Location": absolute_url_for("public.note", slug=note.slug)}
return "", 201, {"Location": note.url}
except InvalidNoteDataError as e:
raise MicropubValidationError(str(e))
@@ -686,7 +728,7 @@ def handle_update(data: dict, token_info: dict):
from starpunk.tags import get_note_tags
title, body = split_title(note.content)
title, summary, body = note.title, note.summary, note.content
tags = [t["display_name"] for t in get_note_tags(note.id, entered_order=True)]
photos = _note_photos(note)
changes = {}
@@ -696,6 +738,8 @@ def handle_update(data: dict, token_info: dict):
for prop in delete:
if prop == "name":
title = None
elif prop == "summary":
summary = None
elif prop == "content":
body = ""
elif prop == "category":
@@ -719,10 +763,12 @@ def handle_update(data: dict, token_info: dict):
if "content" in replace:
# Empty text is fine if the note keeps its photos; update_note checks (v2.6.5)
body = extract_content(replace) if _has_text(replace["content"]) else ""
body = localize_inline_images(extract_content(replace)) if _has_text(replace["content"]) else ""
changes["content_format"] = content_format(replace) if body else None
if "name" in replace:
title = extract_title(replace)
if "summary" in replace:
summary = extract_summary(replace)
if "category" in replace:
tags = extract_tags(replace)
if "photo" in replace:
@@ -738,17 +784,22 @@ def handle_update(data: dict, token_info: dict):
if "photo" in add:
photos += extract_photos(add)
# Text may go only if photos stay (or it's a like, repost or bookmark) (v2.6.5)
if not (body or "").strip() and not (photos or note.reacts_to):
raise MicropubValidationError("A post needs text or a photo")
# Text may go only if photos stay, it has a title, or it's a like, repost
# or bookmark (v2.6.5)
if not (body or "").strip() and not (photos or title or note.reacts_to):
raise MicropubValidationError("A post needs text, a title or a photo")
# Photos must all resolve before anything changes (v2.9.0)
if "photo" in touched:
media_ids, captions = _resolve_photos(photos)
new_content = add_title_heading(body, title)
if new_content != note.content:
changes["content"] = new_content
if title != note.title:
changes["title"] = title
if summary != note.summary:
changes["summary"] = summary
body = drop_title_heading(body, title)
if body != note.content:
changes["content"] = body
elif "content_format" in changes and changes["content_format"] == note.content_format:
del changes["content_format"]
if {"category"} & touched:
@@ -769,6 +820,11 @@ def handle_update(data: dict, token_info: dict):
current_app.logger.error(f"Failed to update note via Micropub: {e}")
raise MicropubError("server_error", "Failed to update post", status_code=500)
# Adding or removing a title moves the post between /note/ and /article/:
# the spec says to answer 201 with the new URL (v2.17.0)
if bool(title) != bool(note.title):
from starpunk.utils import post_url
return "", 201, {"Location": post_url(note.slug, title)}
return "", 204
@@ -800,20 +856,22 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
"""
A note as Microformats2 JSON, the inverse of create
A '# Heading' first line is the title, returned as name.
The title is returned as name and the author's summary as summary.
"""
from starpunk.syndication import get_note_syndications
title, content = split_title(note.content)
content = note.content
props = {
# HTML-authored content comes back as HTML (3.7.2, v2.9.0)
"content": [{"html": content} if note.content_format == "html" else content],
"published": [iso_utc(note.created_at)],
"url": [absolute_url_for("public.note", slug=note.slug)],
"url": [note.url],
"post-status": ["published" if note.published else "draft"],
}
if title:
props["name"] = [title]
if note.title:
props["name"] = [note.title]
if note.summary:
props["summary"] = [note.summary]
from starpunk.tags import get_note_tags
tags = get_note_tags(note.id, entered_order=True)
+67 -1
View File
@@ -344,7 +344,65 @@ def apply_migration(conn, migration_name, migration_path, logger=None):
raise MigrationError(error_msg)
def run_migrations(db_path, logger=None):
def _move_note_titles(conn, data_dir, logger):
"""
Migration 023 data step: move each note's '# Heading' title into notes.title
Runs once, inside the migration transaction. Rewrites the note files without
the heading and returns [(path, original text)] so a failed migration can
put them back (see _restore_files).
"""
from starpunk.models import split_title
from starpunk.utils import calculate_content_hash, find_trashed_file, write_note_file
data_dir = Path(data_dir)
written = []
try:
rows = conn.execute(
"SELECT id, slug, file_path FROM notes WHERE title IS NULL"
).fetchall()
for note_id, slug, file_path in rows:
path = data_dir / file_path
if not path.exists():
path = find_trashed_file(data_dir, file_path)
if path is None:
logger.warning(f"Migration 023: no file for note {slug}, title not moved")
continue
text = path.read_text(encoding="utf-8")
title, body = split_title(text)
if title is None:
continue
conn.execute(
"UPDATE notes SET title = ?, content_hash = ? WHERE id = ?",
(title, calculate_content_hash(body), note_id),
)
write_note_file(path, body)
written.append((path, text))
if written and table_exists(conn, "notes_fts"):
# Emptied so startup rebuilds it with the titles (starpunk/__init__.py)
conn.execute("DELETE FROM notes_fts")
logger.info(f"Migration 023: moved {len(written)} titles out of note files")
return written
except Exception:
_restore_files(written, logger)
raise
def _restore_files(written, logger):
"""Put back note files changed by a data step whose migration failed"""
for path, text in written:
try:
path.write_text(text, encoding="utf-8")
except OSError as e:
logger.critical(f"Could not restore {path} after failed migration: {e}")
# Python steps run right after a migration's SQL, in the same transaction.
# Each returns [(path, original text)] for files it changed.
DATA_MIGRATIONS = {"023_note_titles.sql": _move_note_titles}
def run_migrations(db_path, logger=None, data_dir=None):
"""
Run all pending database migrations with concurrency protection
@@ -374,6 +432,8 @@ def run_migrations(db_path, logger=None):
Args:
db_path: Path to SQLite database file
logger: Optional logger for output
data_dir: Data directory, for migrations that also change note files
(DATA_MIGRATIONS); skipped when None
Raises:
MigrationError: If any migration fails to apply or lock cannot be acquired
@@ -407,6 +467,7 @@ def run_migrations(db_path, logger=None):
# BEGIN IMMEDIATE acquires RESERVED lock, preventing other writes
# but allowing reads. Escalates to EXCLUSIVE during actual writes.
conn.execute("BEGIN IMMEDIATE")
written = [] # note files changed by DATA_MIGRATIONS, restored on failure
try:
# Ensure migrations tracking table exists
@@ -510,6 +571,9 @@ def run_migrations(db_path, logger=None):
# Execute migration inside the locked transaction
execute_in_transaction(conn, migration_sql)
data_step = DATA_MIGRATIONS.get(migration_name)
if data_step and data_dir is not None:
written += data_step(conn, data_dir, logger)
# Record migration as applied
conn.execute(
@@ -547,6 +611,7 @@ def run_migrations(db_path, logger=None):
except MigrationError:
# Migration error - rollback and re-raise
_restore_files(written, logger)
try:
conn.rollback()
except Exception as rollback_error:
@@ -556,6 +621,7 @@ def run_migrations(db_path, logger=None):
except Exception as e:
# Unexpected error during migration - rollback and wrap
_restore_files(written, logger)
try:
conn.rollback()
except Exception as rollback_error:
+54 -112
View File
@@ -36,9 +36,9 @@ DEFAULT_AUTH_STATE_EXPIRY_MINUTES = 5
DEFAULT_TOKEN_EXPIRY_DAYS = 90
# Constants - Markdown rendering
from starpunk.markdown_ext import BareUrlExtension
from starpunk.markdown_ext import BareUrlExtension, HashtagLineExtension
MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br", BareUrlExtension()]
MARKDOWN_EXTENSIONS = ["extra", "codehilite", "nl2br", BareUrlExtension(), HashtagLineExtension()]
# Constants - Content limits
MAX_TITLE_LENGTH = 200
@@ -76,11 +76,14 @@ EXCERPT_LENGTH = 200
def split_title(content: str) -> tuple[Optional[str], str]:
"""
Split a note's '# Heading' first line (its title, per Q22) off its content
Split a '# Heading' first line off Markdown content
Before v2.16.0 an article's title was its '# Heading' first line (Q22).
Titles now live in notes.title (ADR-076); migration 023 uses this to move
the old headings there, and Micropub to drop a heading that repeats 'name'.
The first line must be a Markdown heading: 1-6 '#' then a space and text.
'#indieweb ...' (a hashtag) or a bare '#' is not a title. This is the one
place that decides whether a note has a title (templates use has_title).
'#indieweb ...' (a hashtag) or a bare '#' is not a title.
Returns:
(title, remaining content); title is None if the note has no heading
@@ -171,6 +174,8 @@ class Note:
repost_of: Optional[str] = None # v2.5.0: URL reposted (this note is a repost)
bookmark_of: Optional[str] = None # v2.9.0: URL bookmarked (this note is a bookmark)
content_format: Optional[str] = None # v2.9.0: 'html' if sent as HTML via Micropub
title: Optional[str] = None # v2.16.0: an article's title (p-name), ADR-076
summary: Optional[str] = None # v2.16.0: author-written summary (p-summary)
_cached_content: Optional[str] = field(
default=None, repr=False, compare=False, init=False
)
@@ -233,6 +238,8 @@ class Note:
repost_of=data.get("repost_of"),
bookmark_of=data.get("bookmark_of"),
content_format=data.get("content_format"),
title=data.get("title"),
summary=data.get("summary"),
)
@property
@@ -291,8 +298,13 @@ class Note:
<p>This is my note content...</p>
"""
if self._cached_html is None:
# Render markdown to sanitized HTML (v2.9.0)
html = render_markdown(self.content) # triggers lazy load of content
if self.content_format == "html":
# Sent as HTML via Micropub: sanitize it as is. Through Markdown
# it picked up stray <br>s and empty paragraphs (v2.16.0)
html = nh3.clean(self.content, tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES)
else:
# Render markdown to sanitized HTML (v2.9.0)
html = render_markdown(self.content) # triggers lazy load of content
# Cache it
object.__setattr__(self, "_cached_html", html)
@@ -300,124 +312,40 @@ class Note:
return self._cached_html
@property
def body_html(self) -> str:
"""
Rendered HTML without the title heading
Pages show a titled note's title as their own heading, so the body
leaves it out to avoid showing it twice. Untitled notes: same as html.
"""
title, body = split_title(self.content)
if title is None:
return self.html
return render_markdown(body)
@property
def has_title(self) -> bool:
"""True if the note starts with a '# Heading' title (an article)"""
try:
return split_title(self.content)[0] is not None
except (FileNotFoundError, OSError):
return False
@property
def title(self) -> str:
"""
Extract title from content
Returns first line of content, or uses slug as fallback.
Strips markdown heading syntax (# ) if present.
Returns:
Title string
Examples:
>>> # Content: "# My First Note\\n\\nContent here..."
>>> note.title
'My First Note'
>>> # Content: "Just a note without heading"
>>> note.title
'Just a note without heading'
"""
try:
# Get content (may trigger lazy load)
content = self.content
# Split into lines and get first non-empty line
lines = content.strip().split("\n")
first_line = ""
for line in lines:
if line.strip():
first_line = line.strip()
break
if not first_line:
# No content, use slug
return self.slug
# Strip markdown heading syntax
if first_line.startswith("#"):
# Remove leading # characters and whitespace
first_line = first_line.lstrip("#").strip()
# Truncate to max length
if len(first_line) > MAX_TITLE_LENGTH:
first_line = first_line[:MAX_TITLE_LENGTH]
return first_line if first_line else self.slug
except (FileNotFoundError, OSError):
# If file doesn't exist, use slug
return self.slug
def label(self) -> str:
"""A name for places that always need one: the title, else the excerpt, else the slug"""
return self.title or " ".join(self.excerpt.split())[:MAX_TITLE_LENGTH] or self.slug
@property
def excerpt(self) -> str:
"""
Generate short excerpt for previews
Returns first 200 characters of content (plain text, no markdown).
Strips markdown formatting and adds ellipsis if truncated.
The first 200 characters of the note as plain text: Markdown (or
HTML) rendered and stripped, images left out, links as their text
(v2.16.0). Cut at a word boundary with an ellipsis if longer.
Returns:
Excerpt string
Examples:
>>> note.excerpt
'This is my note content. It has some interesting points...'
'This is my note content. It has some interesting points…'
"""
from starpunk.syndication import plain_text
try:
# Leave out the title; previews show it separately
_, content = split_title(self.content)
# Remove markdown heading syntax
lines = content.split("\n")
cleaned_lines = []
for line in lines:
if line.strip().startswith("#"):
# Remove heading syntax
cleaned_lines.append(line.lstrip("#").strip())
else:
cleaned_lines.append(line)
cleaned_content = "\n".join(cleaned_lines)
# Take first EXCERPT_LENGTH characters
excerpt = cleaned_content.strip()[:EXCERPT_LENGTH]
# Truncate to word boundary
if len(cleaned_content.strip()) > EXCERPT_LENGTH:
# Find last space in excerpt
text = " ".join(plain_text(self.content).split())
if len(text) > EXCERPT_LENGTH:
excerpt = text[:EXCERPT_LENGTH]
last_space = excerpt.rfind(" ")
if last_space > 0:
excerpt = excerpt[:last_space]
excerpt += "..."
text = (excerpt[:last_space] if last_space > 0 else excerpt).rstrip() + "…"
if not excerpt and self.reacts_to:
if not text and self.reacts_to:
# A like, repost or bookmark with no comment
return f"{REACTION_VERBS[self.kind]} {self.reacts_to}"
return excerpt
return text
except (FileNotFoundError, OSError):
# If file doesn't exist, return slug
@@ -455,14 +383,27 @@ class Note:
"""
Generate permalink (public URL path)
Returns:
URL path string (e.g., '/note/my-first-note')
Articles (notes with a title) live under /article/, other notes
under /note/ (v2.17.0).
Examples:
>>> note.permalink
'/note/my-first-note'
Returns:
URL path string (e.g., '/note/my-first-note', '/article/my-essay')
"""
return f"/note/{self.slug}"
from starpunk.utils import post_path
return post_path(self.slug, self.title)
@property
def endpoint(self) -> str:
"""The route that shows this note: public.article or public.note (v2.17.0)"""
return "public.article" if self.title else "public.note"
@property
def url(self) -> str:
"""Absolute permalink, built from SITE_URL (needs an app context)"""
from starpunk.utils import post_url
return post_url(self.slug, self.title)
@property
def is_published(self) -> bool:
@@ -533,6 +474,7 @@ class Note:
"id": self.id,
"slug": self.slug,
"title": self.title,
"summary": self.summary,
"published": self.published,
"created_at": self.created_at.strftime("%Y-%m-%dT%H:%M:%SZ"),
"updated_at": self.updated_at.strftime("%Y-%m-%dT%H:%M:%SZ"),
+49 -26
View File
@@ -40,6 +40,7 @@ from starpunk.utils import (
calculate_content_hash,
validate_note_path,
validate_slug,
post_url,
)
@@ -149,6 +150,8 @@ def create_note(
has_media: bool = False,
bookmark_of: Optional[str] = None,
content_format: Optional[str] = None,
title: Optional[str] = None,
summary: Optional[str] = None,
) -> Note:
"""
Create a new note
@@ -164,6 +167,8 @@ def create_note(
custom_slug: Optional custom slug (from Micropub mp-slug property)
tags: Optional list of tag display names (v1.3.0)
in_reply_to: Optional URL being replied to (v1.6.0)
title: Optional title; a titled note is an article (v2.16.0, ADR-076)
summary: Optional author-written summary (v2.16.0)
Returns:
Note object with all metadata and content loaded
@@ -176,7 +181,7 @@ def create_note(
Examples:
>>> # Create unpublished draft
>>> note = create_note("# My First Note\\n\\nContent here.", published=False)
>>> note = create_note("Content here.", title="My First Note", published=False)
>>> print(note.slug)
'my-first-note'
@@ -219,7 +224,9 @@ def create_note(
"like_of", like_of, "A note can only be one of a like, repost or bookmark"
)
content = content or ""
if not content.strip() and not (like_of or repost_of or bookmark_of or has_media):
title = (title or "").strip() or None
summary = (summary or "").strip() or None
if not content.strip() and not (like_of or repost_of or bookmark_of or has_media or title):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
@@ -244,9 +251,11 @@ def create_note(
if not success:
raise InvalidNoteDataError("slug", custom_slug, error)
else:
# Generate timestamp-based slug (ADR-062)
from starpunk.slug_utils import generate_timestamp_slug
slug = generate_timestamp_slug(created_at, existing_slugs)
# Articles take their slug from the title (v2.17.0); other notes, or a
# title with nothing usable in it, get a timestamp slug (ADR-062)
from starpunk.slug_utils import generate_timestamp_slug, generate_title_slug
slug = (generate_title_slug(title, existing_slugs) if title else None) \
or generate_timestamp_slug(created_at, existing_slugs)
# 4. GENERATE FILE PATH
note_path = generate_note_path(slug, created_at, data_dir)
@@ -281,11 +290,12 @@ def create_note(
db.execute(
"""
INSERT INTO notes (slug, file_path, published, created_at, updated_at, content_hash,
in_reply_to, like_of, repost_of, bookmark_of, content_format)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
in_reply_to, like_of, repost_of, bookmark_of, content_format,
title, summary)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(slug, file_path_rel, published, created_at, updated_at, content_hash,
in_reply_to, like_of, repost_of, bookmark_of, content_format),
in_reply_to, like_of, repost_of, bookmark_of, content_format, title, summary),
)
db.commit()
except Exception as e:
@@ -327,7 +337,7 @@ def create_note(
from starpunk.search import update_fts_index, has_fts_table
db_path = Path(current_app.config["DATABASE_PATH"])
if has_fts_table(db_path):
update_fts_index(db, note_id, slug, content)
update_fts_index(db, note_id, slug, content, title)
db.commit()
except Exception as e:
# FTS update failure should not prevent note creation
@@ -341,7 +351,7 @@ def create_note(
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
from starpunk.utils import absolute_url_for
note_url = absolute_url_for("public.note", slug=slug)
note_url = post_url(slug, title)
# Get rendered HTML from note
html_content = note.html
@@ -641,11 +651,14 @@ def update_note(
in_reply_to: Optional[str] = None,
has_media: bool = False,
content_format=UNCHANGED,
title=UNCHANGED,
summary=UNCHANGED,
) -> Note:
"""
Update a note's content and/or published status
content_format (v2.9.0): 'html' or None (Markdown); left alone unless given
title, summary (v2.16.0): new value, None or "" to remove; left alone unless given
Updates note content and/or metadata, maintaining atomic synchronization
between file and database. At least one of content or published must
@@ -713,8 +726,15 @@ def update_note(
if slug is not None and id is not None:
raise ValueError("Cannot provide both slug and id")
if content is None and published is None and tags is None and in_reply_to is None:
raise ValueError("Must provide at least one of content, published, tags, or in_reply_to to update")
if (content is None and published is None and tags is None and in_reply_to is None
and title is UNCHANGED and summary is UNCHANGED):
raise ValueError(
"Must provide at least one of content, published, tags, in_reply_to, title or summary to update"
)
if title is not UNCHANGED:
title = (title or "").strip() or None
if summary is not UNCHANGED:
summary = (summary or "").strip() or None
# Validate content if provided (checked against the note below for
# likes/reposts, whose comment may be empty)
@@ -729,7 +749,8 @@ def update_note(
raise NoteNotFoundError(identifier)
old_tag_names = _tag_names(existing_note.id) # for WebSub tag topics (v2.10.0)
if empty_content and not (existing_note.reacts_to or has_media):
new_title = existing_note.title if title is UNCHANGED else title
if empty_content and not (existing_note.reacts_to or has_media or new_title):
raise InvalidNoteDataError(
"content", content, "Content cannot be empty or whitespace-only"
)
@@ -797,6 +818,14 @@ def update_note(
update_fields.append("content_format = ?")
params.append(content_format)
if title is not UNCHANGED:
update_fields.append("title = ?")
params.append(title)
if summary is not UNCHANGED:
update_fields.append("summary = ?")
params.append(summary)
# Add WHERE clause parameter
if slug is not None:
where_clause = "slug = ?"
@@ -832,13 +861,14 @@ def update_note(
# Tag update failure should not prevent note update
current_app.logger.warning(f"Failed to update tags for note {existing_note.slug}: {e}")
# 7. UPDATE FTS INDEX (if available and content changed)
if content is not None:
# 7. UPDATE FTS INDEX (if available and content or title changed)
if content is not None or title is not UNCHANGED:
try:
from starpunk.search import update_fts_index, has_fts_table
db_path = Path(current_app.config["DATABASE_PATH"])
if has_fts_table(db_path):
update_fts_index(db, existing_note.id, existing_note.slug, content)
update_fts_index(db, existing_note.id, existing_note.slug,
existing_note.content if content is None else content, new_title)
db.commit()
except Exception as e:
# FTS update failure should not prevent note update
@@ -876,7 +906,7 @@ def update_note(
db_path = Path(current_app.config["DATABASE_PATH"])
site_url = current_app.config["SITE_URL"]
from starpunk.utils import absolute_url_for
note_url = absolute_url_for("public.note", slug=updated_note.slug)
note_url = updated_note.url
# Get rendered HTML from updated note (if published)
html_content = updated_note.html if updated_note.published else None
@@ -1129,7 +1159,7 @@ def undelete_note(slug: str) -> Note:
NoteNotFoundError: If there is no soft-deleted note with this slug
"""
import shutil
from starpunk.utils import TRASH_DIR_NAME
from starpunk.utils import find_trashed_file
db = get_db(current_app)
row = db.execute(
@@ -1142,14 +1172,7 @@ def undelete_note(slug: str) -> Note:
data_dir = Path(current_app.config["DATA_PATH"])
note_path = data_dir / row["file_path"]
if not note_path.exists():
# delete_note_file put it in .trash/<year>/<month>/<name>
parts = Path(row["file_path"]).parts
candidates = []
if "notes" in parts and len(parts) > parts.index("notes") + 2:
i = parts.index("notes")
candidates.append(data_dir / TRASH_DIR_NAME / parts[i + 1] / parts[i + 2] / note_path.name)
candidates += sorted((data_dir / TRASH_DIR_NAME).glob(f"*/*/{note_path.name}"))
source = next((c for c in candidates if c.exists()), None)
source = find_trashed_file(data_dir, row["file_path"])
if source is None:
raise NoteSyncError("undelete", f"Trashed file not found for {slug}", "Cannot restore note file")
note_path.parent.mkdir(parents=True, exist_ok=True)
+238
View File
@@ -0,0 +1,238 @@
"""
Notifications through Apprise (v2.14.0, ADR-074)
The owner hears about new responses (received webmentions and backfeed
alike) and about crossposting problems, on whatever service the Apprise
URLs in NOTIFY_URLS point at (Gotify, ntfy, Pushover, email, an
apprise-api server…).
Responses are sent by a sweep (`sweep`) in the webmention worker, which
runs in one process at a time: each response with `notified_at` empty is
sent once and marked. Likes, reposts and bookmarks are grouped per note.
Failed sends are retried with backoff; responses still unsent after 24
hours are dropped. Problems (`problem`) are sent at once, without retry.
Nothing here may break validation, crossposting or backfeed: every send is
wrapped, and failures are only logged and shown in the admin.
"""
import html
import logging
import re
import sqlite3
import time
from typing import Dict, List, Optional, Tuple
from urllib.parse import parse_qsl, urlsplit
from starpunk.utils import post_path
logger = logging.getLogger(__name__)
RESPONSE_EVENTS = ("reply", "mention", "quote", "like", "repost", "bookmark")
EVENTS = RESPONSE_EVENTS + ("problems",)
DEFAULT_EVENTS = "reply,mention,quote,problems"
GROUPED = ("like", "repost", "bookmark")
DROP_AFTER_HOURS = 24
MAX_BACKOFF = 15 * 60
SEND_TIMEOUT = 15
HEADINGS = {
"reply": "↩ Reply from {who}",
"mention": "💬 Mention from {who}",
"quote": "❝ Quote from {who}",
}
GROUP_HEADINGS = {
"like": ("♥", "like", "likes"),
"repost": ("🔁", "repost", "reposts"),
"bookmark": ("🔖", "bookmark", "bookmarks"),
}
# --- Settings ---------------------------------------------------------------------
def urls(config) -> List[str]:
"""NOTIFY_URLS as a list (space- or comma-separated)"""
return [u for u in re.split(r"[\s,]+", config.get("NOTIFY_URLS") or "") if u]
def events(config) -> set:
"""NOTIFY_EVENTS as a set of known event names"""
chosen = {e.strip().lower() for e in (config.get("NOTIFY_EVENTS") or DEFAULT_EVENTS).split(",")}
return chosen & set(EVENTS)
def enabled(config) -> bool:
return bool(urls(config))
def with_priority(url: str, priority: str) -> str:
"""The URL with `priority=` added, unless it sets one itself"""
if not priority or any(k == "priority" for k, _ in parse_qsl(urlsplit(url).query)):
return url
return url + ("&" if "?" in url else "?") + "priority=" + priority
# --- Sending ----------------------------------------------------------------------------
def _deliver(targets: List[str], title: str, body: str) -> Tuple[bool, str]:
"""Send one message to every URL. Returns (all sent, error text)."""
import apprise
sender = apprise.Apprise()
bad = [u.split("://", 1)[0] + "://…" for u in targets if not sender.add(u)]
if bad:
return False, f"Not a usable Apprise URL: {', '.join(bad)}"
with apprise.LogCapture(level=logging.WARNING, fmt="%(message)s") as log:
result = sender.notify(body, title=title, body_format=apprise.NotifyFormat.MARKDOWN,
timeout=SEND_TIMEOUT)
return bool(result), log.getvalue().strip()[:500]
def send(config, title: str, body: str) -> Tuple[bool, str]:
"""Send to all NOTIFY_URLS at NOTIFY_PRIORITY. Never raises."""
targets = [with_priority(u, config.get("NOTIFY_PRIORITY") or "normal") for u in urls(config)]
if not targets:
return False, "No NOTIFY_URLS set"
try:
ok, error = _deliver(targets, title, body)
except Exception as e: # a broken service mustn't break the caller
ok, error = False, f"{type(e).__name__}: {e}"
if not ok:
logger.warning(f"Notification '{title}' not sent: {error or 'unknown error'}")
return ok, error
def problem(config, db_path, title: str, detail: str, link: str) -> None:
"""A crosspost or backfeed problem, sent at once if 'problems' is selected"""
if not enabled(config) or "problems" not in events(config):
return
ok, error = send(config, f"⚠ {title}", f"{detail}\n\n[Open in StarPunk]({link})")
_record(db_path, ok, error)
def _record(db_path, ok: bool, error: str) -> None:
"""Remember the last failure for the admin page (cleared by a success)"""
try:
conn = sqlite3.connect(db_path)
try:
conn.execute("INSERT INTO site_state (key, value) VALUES ('notify_last_error', ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
("" if ok else time.strftime("%Y-%m-%d %H:%M:%S UTC ", time.gmtime())
+ (error or "unknown error"),))
conn.commit()
finally:
conn.close()
except sqlite3.Error as e:
logger.warning(f"Could not record notification result: {e}")
def last_error(db) -> str:
row = db.execute("SELECT value FROM site_state WHERE key = 'notify_last_error'").fetchone()
return row[0] if row else ""
# --- Messages -----------------------------------------------------------------------------
def _plain(snippet: Optional[str], limit: int = 300) -> str:
text = html.unescape(re.sub(r"<[^>]+>", " ", snippet or ""))
text = re.sub(r"\s+", " ", text).strip()
return text if len(text) <= limit else text[:limit - 1].rsplit(" ", 1)[0] + "…"
def _note_label(conn, note_id: int) -> str:
from starpunk.notes import get_note
try:
note = get_note(id=note_id, load_content=True)
label = note.label if note else ""
except Exception:
label = ""
return (label[:57] + "…") if len(label) > 60 else (label or "a note")
def _messages(rows, site_url: str, conn) -> List[Tuple[str, str, List[int]]]:
"""(title, body, webmention ids) for a batch of responses"""
site = site_url.rstrip("/")
names = {"gotosocial": "GoToSocial", "bluesky": "Bluesky", "threads": "Threads"}
messages, groups = [], {}
for row in rows:
via = names.get(row["origin"]) if row["origin"] != "webmention" else None
who = (row["author_name"] or "someone") + (f" via {via}" if via else "")
pending = row["moderation_status"] == "pending"
link = (f"{site}/admin/webmentions/{row['id']}" if pending
else f"{site}{post_path(row['slug'], row['note_title'])}#responses")
if row["mention_type"] in GROUPED:
groups.setdefault((row["target_note_id"], row["mention_type"], row["slug"], pending), []).append(
(row["id"], row["author_name"] or "someone", link))
continue
title = HEADINGS.get(row["mention_type"], "Response from {who}").format(who=who)
if pending:
title += " (awaiting moderation)"
text = _plain(row["content_snippet"])
body = (f"{text}\n\n" if text else "") + f"On “{_note_label(conn, row['target_note_id'])}”: [open]({link})"
messages.append((title, body, [row["id"]]))
for (note_id, kind, slug, pending), items in groups.items():
icon, one, many = GROUP_HEADINGS[kind]
n = len(items)
title = f"{icon} {n} new {one if n == 1 else many} on “{_note_label(conn, note_id)}”"
if pending:
title += " (awaiting moderation)"
people = ", ".join(name for _, name, _ in items[:5]) + (f" and {n - 5} more" if n > 5 else "")
messages.append((title, f"{people}\n\n[open]({items[0][2]})", [i for i, _, _ in items]))
return messages
# --- Sweep ----------------------------------------------------------------------------------
class Sweeper:
"""Sends new responses; one instance per webmention worker"""
def __init__(self):
self._failures = 0
self._next_try = 0.0
def run(self, config, db_path, site_url: str) -> int:
"""Send what's due. Returns how many responses were sent."""
if not enabled(config) or time.time() < self._next_try:
return 0
chosen = events(config)
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
try:
# Too old to be news (an outage, say): drop quietly
dropped = conn.execute(
f"""UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP
WHERE notified_at IS NULL
AND received_at < datetime('now', '-{DROP_AFTER_HOURS} hours')""").rowcount
if dropped:
logger.warning(f"Dropped {dropped} notification(s) older than {DROP_AFTER_HOURS} hours")
rows = conn.execute(
"""
SELECT w.*, n.slug, n.title AS note_title FROM webmentions w JOIN notes n ON n.id = w.target_note_id
WHERE w.notified_at IS NULL AND w.validation_status = 'valid'
AND w.moderation_status IN ('approved', 'pending')
ORDER BY w.id LIMIT 100
"""
).fetchall()
skip = [r["id"] for r in rows if r["mention_type"] not in chosen]
if skip:
conn.execute(f"UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP "
f"WHERE id IN ({','.join('?' * len(skip))})", skip)
conn.commit()
due = [r for r in rows if r["mention_type"] in chosen]
sent = 0
for title, body, ids in _messages(due, site_url, conn):
ok, error = send(config, title, body)
_record(db_path, ok, error)
if not ok:
self._failures += 1
self._next_try = time.time() + min(10 * 2 ** self._failures, MAX_BACKOFF)
break
self._failures = 0
conn.execute(f"UPDATE webmentions SET notified_at = CURRENT_TIMESTAMP "
f"WHERE id IN ({','.join('?' * len(ids))})", ids)
conn.commit()
sent += len(ids)
return sent
finally:
conn.close()
+51 -7
View File
@@ -128,6 +128,7 @@ def create_note_submit():
media_files: Multiple file upload (v1.2.0 Phase 3)
captions[]: Captions for each media file (v1.2.0 Phase 3)
tags: Comma-separated tag list (v1.3.0 Phase 3)
title, summary: Optional; a title makes it an article (v2.16.0)
Returns:
Redirect to dashboard on success, back to form on error
@@ -141,17 +142,19 @@ def create_note_submit():
published = "published" in request.form
custom_slug = request.form.get("custom_slug", "").strip()
tags_input = request.form.get("tags", "")
title = request.form.get("title", "").strip() or None
summary = request.form.get("summary", "").strip() or None
if not content:
flash("Content cannot be empty", "error")
# With photos, WebSub hears about the note once they're attached (v2.10.1)
has_uploads = any(f.filename for f in request.files.getlist('media_files'))
if not (content or title or has_uploads):
flash("A note needs text, a title or an image", "error")
return redirect(url_for("admin.new_note_form"))
# Parse tags (v1.3.0 Phase 3)
tags = parse_tag_input(tags_input)
# With photos, WebSub hears about the note once they're attached (v2.10.1)
has_uploads = any(f.filename for f in request.files.getlist('media_files'))
try:
# Create note first (per Q4)
note = create_note(
@@ -160,6 +163,8 @@ def create_note_submit():
custom_slug=custom_slug if custom_slug else None,
tags=tags if tags else None,
has_media=has_uploads,
title=title,
summary=summary,
)
# Handle media uploads (v1.2.0 Phase 3)
@@ -279,12 +284,14 @@ def update_note_submit(note_id: int):
content: Updated markdown content (required)
published: Checkbox for published status (optional)
tags: Comma-separated tag list (v1.3.0 Phase 3)
title, summary: Empty removes them; a form without the field leaves them (v2.16.0)
Returns:
Redirect to dashboard on success, back to form on error
Decorator: @require_auth
"""
from starpunk.notes import UNCHANGED
from starpunk.tags import parse_tag_input
# Check if note exists first
@@ -296,8 +303,12 @@ def update_note_submit(note_id: int):
content = request.form.get("content", "").strip()
published = "published" in request.form
tags_input = request.form.get("tags", "")
title = request.form["title"].strip() if "title" in request.form else UNCHANGED
summary = request.form["summary"].strip() if "summary" in request.form else UNCHANGED
has_title = existing_note.title if title is UNCHANGED else title
if not content and not existing_note.reacts_to: # likes/reposts may have no comment
# Likes/reposts may have no comment, articles no body
if not content and not (existing_note.reacts_to or has_title):
flash("Content cannot be empty", "error")
return redirect(url_for("admin.edit_note_form", note_id=note_id))
@@ -309,7 +320,11 @@ def update_note_submit(note_id: int):
id=note_id,
content=content,
published=published,
tags=tags if tags else None
# An empty Tags field removes all tags (v2.14.1); a form without
# the field leaves them alone
tags=tags if "tags" in request.form else None,
title=title,
summary=summary,
)
if "syndication_form" in request.form:
from starpunk.syndication import select_targets
@@ -515,6 +530,7 @@ def webmentions():
status_filter=status_filter,
origin_filter=origin_filter,
backfeed=backfeed.status(),
notify=_notify_status(),
page=page,
total_pages=total_pages,
user_me=g.me
@@ -681,6 +697,34 @@ def revoke_token(token_id):
return redirect(url_for("admin.tokens"))
def _notify_status() -> dict:
"""Notification settings and last error for the Webmentions page (v2.14.0)"""
from starpunk import notifications
from starpunk.database import get_db
return {
"enabled": notifications.enabled(current_app.config),
"events": sorted(notifications.events(current_app.config)),
"priority": current_app.config.get("NOTIFY_PRIORITY") or "normal",
"services": [u.split("://", 1)[0] for u in notifications.urls(current_app.config)],
"last_error": notifications.last_error(get_db(current_app)),
}
@bp.route("/notifications/test", methods=["POST"])
@require_auth
def notifications_test():
"""Send a test notification to every NOTIFY_URLS service (v2.14.0)"""
from starpunk import notifications
ok, error = notifications.send(current_app.config, "StarPunk test notification",
"Notifications from StarPunk are working.")
notifications._record(current_app.config["DATABASE_PATH"], ok, error)
flash("Test notification sent" if ok else f"Test notification failed: {error or 'unknown error'}",
"success" if ok else "error")
return redirect(url_for("admin.webmentions"))
@bp.route("/backfeed/backfill", methods=["POST"])
@require_auth
def backfeed_backfill():
+14 -2
View File
@@ -51,6 +51,7 @@ _REQUEST_FIELDS = (
"code_challenge",
"code_challenge_method",
"scope",
"resource",
)
@@ -95,6 +96,11 @@ def metadata():
"revocation_endpoint_auth_methods_supported": ["none"],
"scopes_supported": SUPPORTED_SCOPES,
"response_types_supported": ["code"],
# Public clients only (PKCE); clients identify with a client_id
# URL whose JSON metadata lists redirect_uris, which is what MCP
# calls a Client ID Metadata Document (v2.15.0, ADR-075)
"token_endpoint_auth_methods_supported": ["none"],
"client_id_metadata_document_supported": True,
"grant_types_supported": ["authorization_code"],
"code_challenge_methods_supported": ["S256"],
"authorization_response_iss_parameter_supported": True,
@@ -158,6 +164,7 @@ def approve():
auth_request["redirect_uri"],
" ".join(granted),
auth_request["code_challenge"],
resource=auth_request["resource"],
)
current_app.logger.info(
f"IndieAuth: approved {auth_request['client_id']} (scope: {' '.join(granted) or 'none'})"
@@ -209,7 +216,12 @@ def token():
# Spec: a code issued without scope cannot be used for an access token
return _oauth_error("invalid_grant", "No scope was granted; use the authorization endpoint")
return _json(create_access_token(result["client_id"], result["scope"]))
# RFC 8707: a resource named here must be the one the code was issued for
requested = (request.form.get("resource") or "").strip().rstrip("/")
if requested and requested != (result["resource"] or ""):
return _oauth_error("invalid_target", "resource does not match the authorization request")
return _json(create_access_token(result["client_id"], result["scope"], resource=result["resource"]))
@bp.route("/auth/token", methods=["GET"])
@@ -222,7 +234,7 @@ def token_verify():
info = verify_access_token(token)
if not info:
return _oauth_error("unauthorized", "Invalid or expired access token", 401)
return _json(info)
return _json({k: info[k] for k in ("me", "client_id", "scope")})
@bp.route("/auth/revoke", methods=["POST"])
+38 -4
View File
@@ -8,7 +8,7 @@ No authentication required for these routes.
import hashlib
from datetime import datetime, timedelta, timezone
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
from flask import Blueprint, abort, redirect, render_template, Response, current_app, request, send_from_directory, url_for
from starpunk.notes import list_notes, get_note
from starpunk.websub import embed_hub
@@ -292,9 +292,23 @@ def index():
object.__setattr__(note, '_cached_tags', tags)
attach_contexts(notes)
attach_response_counts(notes)
return render_template("index.html", notes=notes)
def attach_response_counts(notes) -> None:
"""Give listed notes their like, repost and reply counts (v2.13.4)"""
counts = {}
if current_app.config.get('WEBMENTION_ENABLED', False) and notes:
try:
from starpunk.webmention_queries import response_counts
counts = response_counts([n.id for n in notes])
except Exception as e:
current_app.logger.warning(f"Could not count responses: {e}")
for note in notes:
object.__setattr__(note, 'response_counts', counts.get(note.id))
def attach_contexts(notes) -> None:
"""Give likes and reposts the fetched context of the post they point at"""
from starpunk.webmention_queries import get_reply_context
@@ -320,6 +334,7 @@ def likes():
object.__setattr__(note, 'media', get_note_media(note.id))
object.__setattr__(note, '_cached_tags', get_note_tags(note.id))
attach_contexts(notes)
attach_response_counts(notes)
return render_template("likes.html", notes=notes)
@@ -339,18 +354,30 @@ def _note_was_deleted(slug: str) -> bool:
).fetchone():
return True
return db.execute(
"SELECT 1 FROM webmention_queue WHERE source_url = ? AND source_note_id IS NULL",
(absolute_url_for("public.note", slug=slug),),
"SELECT 1 FROM webmention_queue WHERE source_url IN (?, ?) AND source_note_id IS NULL",
(absolute_url_for("public.note", slug=slug), absolute_url_for("public.article", slug=slug)),
).fetchone() is not None
@bp.route("/note/<slug>")
def note(slug: str):
"""A note's permalink page; an article's /note/ URL redirects to /article/"""
return _permalink(slug, "public.note")
@bp.route("/article/<slug>")
def article(slug: str):
"""An article's permalink page (v2.17.0); a note's /article/ URL redirects to /note/"""
return _permalink(slug, "public.article")
def _permalink(slug: str, endpoint: str):
"""
Individual note permalink page
Individual note or article permalink page
Args:
slug: URL-safe note identifier
endpoint: The route the request came in on
Returns:
Rendered note template with full content
@@ -375,6 +402,12 @@ def note(slug: str):
abort(410)
abort(404)
# Each post has one URL: /article/ for articles, /note/ for notes (v2.17.0).
# The other one redirects, so links made before a title was added or
# removed keep working.
if note_obj.endpoint != endpoint:
return redirect(url_for(note_obj.endpoint, slug=slug), code=301)
# Get media for note (v1.2.0 Phase 3)
media = get_note_media(note_obj.id)
@@ -467,6 +500,7 @@ def tag(tag: str):
media = get_note_media(note.id)
object.__setattr__(note, 'media', media)
attach_contexts(notes)
attach_response_counts(notes)
return render_template(
"tag.html",
+3 -2
View File
@@ -10,6 +10,7 @@ from pathlib import Path
from flask import Blueprint, current_app, g, jsonify, render_template, request
from starpunk.search import has_fts_table, search_notes
from starpunk.utils import post_path
logger = logging.getLogger(__name__)
@@ -108,7 +109,7 @@ def api_search():
"title": r["title"] or f"Note from {r['created_at'][:10]}",
"excerpt": r["snippet"], # Already has <mark> tags
"published_at": r["created_at"],
"url": f"/note/{r['slug']}",
"url": post_path(r["slug"], r.get("article_title")),
}
for r in results
],
@@ -176,7 +177,7 @@ def search_page():
"title": r["title"] or f"Note from {r['created_at'][:10]}",
"excerpt": Markup(safe_snippet), # Mark as safe since we've escaped it ourselves
"published_at": r["created_at"],
"url": f"/note/{r['slug']}",
"url": post_path(r["slug"], r.get("article_title")),
})
results = formatted_results
except Exception as e:
+4 -3
View File
@@ -139,9 +139,10 @@ def resolve_target_note(target_url: str) -> int | None:
# Strip query params and fragments for matching
target_path = target_parsed.path.rstrip('/')
# Check if it's a note URL pattern: /note/{slug}
if target_path.startswith('/note/'):
slug = target_path[6:] # Remove '/note/' prefix
# A note or article URL: /note/{slug} or /article/{slug} (v2.17.0).
# Either prefix finds the post, as the page redirects to the right one.
prefix, _, slug = target_path.lstrip('/').partition('/')
if prefix in ('note', 'article') and '/' not in slug:
if slug:
try:
note = get_note(slug=slug)
+23 -20
View File
@@ -98,29 +98,32 @@ def has_fts_table(db_path: Path) -> bool:
return False
def update_fts_index(conn: sqlite3.Connection, note_id: int, slug: str, content: str):
def _first_line(content: str) -> str:
"""A note's first line without heading marks (result label for untitled notes).
A hashtag ('#indieweb ...') is kept."""
line = content.split('\n', 1)[0].strip()
return re.sub(r'^#{1,6}\s+', '', line)
def update_fts_index(conn: sqlite3.Connection, note_id: int, slug: str, content: str,
title: Optional[str] = None):
"""
Update FTS index for a note (insert or replace)
Extracts title from first line of content and updates the FTS index.
Uses REPLACE to handle both new notes and updates.
Indexes the note's title (articles, v2.16.0) or, for untitled notes, its
first line. Uses REPLACE to handle both new notes and updates.
Args:
conn: SQLite database connection
note_id: Note ID (used as FTS rowid)
slug: Note slug
content: Full markdown content
content: Markdown content
title: The note's title, if it has one
Raises:
sqlite3.Error: If FTS update fails
"""
# Extract title from first line
lines = content.split('\n', 1)
title = lines[0].strip() if lines else ''
# Remove markdown heading syntax (# ## ###)
if title.startswith('#'):
title = title.lstrip('#').strip()
title = title or _first_line(content)
# Limit title length
if len(title) > 100:
@@ -170,7 +173,7 @@ def rebuild_fts_index(db_path: Path, data_dir: Path):
# Get all non-deleted notes
cursor = conn.execute(
"SELECT id, slug, file_path FROM notes WHERE deleted_at IS NULL"
"SELECT id, slug, file_path, title FROM notes WHERE deleted_at IS NULL"
)
indexed_count = 0
@@ -188,7 +191,7 @@ def rebuild_fts_index(db_path: Path, data_dir: Path):
content = note_path.read_text(encoding='utf-8')
# Update FTS index
update_fts_index(conn, row['id'], row['slug'], content)
update_fts_index(conn, row['id'], row['slug'], content, row['title'])
indexed_count += 1
except Exception as e:
@@ -327,6 +330,7 @@ def search_notes_fts5(
notes.id,
notes.slug,
notes_fts.title,
notes.title AS article_title,
notes.published,
notes.created_at,
rank AS relevance,
@@ -353,6 +357,7 @@ def search_notes_fts5(
'id': row['id'],
'slug': row['slug'],
'title': row['title'],
'article_title': row['article_title'],
'snippet': row['snippet'], # Plain string - route must escape HTML while preserving <mark> tags
'relevance': row['relevance'],
'published': bool(row['published']),
@@ -408,6 +413,7 @@ def search_notes_fallback(
id,
slug,
file_path,
title,
published,
created_at
FROM notes
@@ -446,8 +452,8 @@ def search_notes_fallback(
file_path = data_dir / row['file_path']
content = read_note_file(file_path)
# Check if query matches content (case-insensitive)
content_lower = content.lower()
# Check if query matches title or content (case-insensitive)
content_lower = f"{row['title'] or ''}\n{content}".lower()
query_lower = query.lower()
matches = query_lower in content_lower
@@ -456,16 +462,13 @@ def search_notes_fallback(
matches = any(term.lower() in content_lower for term in terms)
if matches:
# Extract title from first line
lines = content.split('\n', 1)
title = lines[0].strip() if lines else row['slug']
if title.startswith('#'):
title = title.lstrip('#').strip()
title = row['title'] or _first_line(content) or row['slug']
results.append({
'id': row['id'],
'slug': row['slug'],
'title': title,
'article_title': row['title'],
'snippet': generate_snippet(content, query),
'relevance': 0.0, # No ranking in fallback mode
'published': bool(row['published']),
+29
View File
@@ -288,6 +288,35 @@ def generate_timestamp_slug(
return f"{base_slug}-{suffix}"
TITLE_SLUG_LENGTH = 80
def generate_title_slug(title: str, existing_slugs: Set[str]) -> Optional[str]:
"""
A slug made from an article's title (v2.17.0)
"Why I Left Twitter" becomes 'why-i-left-twitter'; long titles are cut at
a word boundary, and a taken slug gets -2, -3, ...
Returns:
The slug, or None when the title has nothing usable (e.g. only emoji),
so the caller falls back to a timestamp slug.
"""
slug = sanitize_slug(title or "")
if len(slug) > TITLE_SLUG_LENGTH:
cut = slug[:TITLE_SLUG_LENGTH + 1]
slug = cut.rsplit("-", 1)[0] if "-" in cut else slug[:TITLE_SLUG_LENGTH]
slug = slug.strip("-")
if not slug:
return None
if is_reserved_slug(slug):
slug = f"{slug}-article"
try:
return make_slug_unique_with_suffix(slug, existing_slugs)
except ValueError: # 99 articles with this title
return None
def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]) -> tuple[bool, Optional[str], Optional[str]]:
"""
Validate and sanitize a custom slug from Micropub
+35 -10
View File
@@ -357,7 +357,9 @@ class BlueskyClient:
def _record(self, post: "Post", text: Optional[str] = None, embed: Optional[dict] = None,
anchors: List[tuple] = ()) -> dict:
if text is None:
body, anchors = plain_text_links(post.note.content, links="anchor")
# An article is posted as a teaser (compose_text), so its links aren't in the text
body, anchors = (None, []) if post.note.title else \
plain_text_links(post.note.content, links="anchor")
text = compose_text(post.note, post.permalink, post.tags, post.max_chars, body=body)
record = {"$type": self.POST, "text": text, "createdAt": self._now()}
facets = bluesky_facets(text, anchors)
@@ -368,14 +370,16 @@ class BlueskyClient:
elif post.media:
record["embed"] = {"$type": "app.bsky.embed.images",
"images": [self._image(m) for m in post.media]}
elif note_links(post.note.content):
elif not post.note.title and note_links(post.note.content):
# The note links somewhere: the card shows that page, not the note
record["embed"] = self._link_card(note_links(post.note.content)[0])
else:
# Articles always get a card for the article itself (v2.16.0)
description = article_teaser(post.note) if post.note.title else plain_text(post.note.content)
record["embed"] = {"$type": "app.bsky.embed.external", "external": {
"uri": post.permalink,
"title": post.note.title[:300],
"description": plain_text(post.note.content)[:300],
"title": post.note.label[:300],
"description": description[:300],
}}
return record
@@ -1074,8 +1078,7 @@ def get_targets() -> Dict[str, Target]:
if token:
client = _client_cache.get("threads")
if client is None or client.config_token != token:
client = ThreadsClient(token, Path(config["DATA_PATH"]) / "threads_token.json",
api=config.get("THREADS_API") or THREADS_API)
client = ThreadsClient(token, Path(config["DATA_PATH"]) / "threads_token.json")
_client_cache["threads"] = client
username = (config.get("THREADS_USERNAME") or "").lstrip("@")
targets["threads"] = Target(
@@ -1186,19 +1189,30 @@ def _hashtag(display_name: str) -> str:
return f"#{word}" if word else ""
def article_teaser(note) -> str:
"""An article's summary, or the start of its text (v2.16.0)"""
return note.summary or note.excerpt
def compose_text(note, permalink: str, tags: List[str], max_chars: int,
body: Optional[str] = None) -> str:
"""
Full note (Markdown unless `body` is given), then the permalink and
hashtags; shortened at a word boundary if too long. On short targets the
hashtags may use at most a third of the space.
An article (a note with a title) is posted as its title and teaser
instead of its full text, the usual POSSE form for articles (v2.16.0).
"""
hashtags = ""
for h in (_hashtag(t) for t in tags):
if h and len(hashtags) + len(h) + 1 <= max(max_chars // 3, 1):
hashtags = f"{hashtags} {h}".strip()
tail = f"\n\n{permalink}" + (f"\n\n{hashtags}" if hashtags else "")
body = (note.content if body is None else body).strip()
if getattr(note, "title", None):
body = f"{note.title}\n\n{article_teaser(note)}".strip()
else:
body = (note.content if body is None else body).strip()
if getattr(note, "bookmark_of", None):
# A bookmark leads with what it bookmarks (v2.9.0)
body = f"🔖 {note.bookmark_of}\n\n{body}".strip()
@@ -1216,7 +1230,10 @@ def _note_soup(markdown_text: str):
import markdown
from bs4 import BeautifulSoup
return BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
from starpunk.markdown_ext import HashtagLineExtension
html = markdown.markdown(markdown_text, extensions=["fenced_code", HashtagLineExtension()])
return BeautifulSoup(html, "html.parser")
def _clean_url(url: str) -> str:
@@ -1464,6 +1481,13 @@ def _fail(db, row, error: SyndicationError) -> None:
f"Syndication {row['target']} for note {row['note_id']} failed "
f"(attempt {attempts}{', giving up' if failed else ''}): {error}"
)
if failed: # ADR-074: tell the owner, once, when we give up
from starpunk import notifications
target = get_targets().get(row["target"])
site = current_app.config["SITE_URL"].rstrip("/")
notifications.problem(current_app.config, current_app.config["DATABASE_PATH"],
f"Crosspost to {target.name if target else row['target']} failed",
str(error)[:300], f"{site}/admin/edit/{row['note_id']}")
def _target_title(url: Optional[str]) -> Optional[str]:
@@ -1500,11 +1524,12 @@ def _reconcile(row, target: Target) -> tuple:
site = current_app.config["SITE_URL"].rstrip("/")
post = Post(
note=note,
permalink=f"{site}/note/{note.slug}",
permalink=note.url,
tags=[t["display_name"] for t in get_note_tags(note.id, entered_order=True)],
max_chars=target.max_chars,
# Articles are posted as a teaser without images (v2.16.0)
media=_media_files(note.id, target.max_media, note.content)
if note.kind == "note" else [],
if note.kind == "note" and not note.title else [],
kind=note.kind,
target_url=note.reacts_to,
target_title=_target_title(note.reacts_to),
+31
View File
@@ -480,6 +480,27 @@ def validate_note_path(file_path: Path, data_dir: Path) -> bool:
# Atomic File Operations
def find_trashed_file(data_dir: Path, file_path: str) -> Optional[Path]:
"""
Where delete_note_file put a note's file: .trash/<year>/<month>/<name>
Args:
data_dir: Data directory
file_path: The note's file_path as stored in the database
Returns:
Path of the trashed file, or None if it can't be found
"""
parts = Path(file_path).parts
name = Path(file_path).name
candidates = []
if "notes" in parts and len(parts) > parts.index("notes") + 2:
i = parts.index("notes")
candidates.append(data_dir / TRASH_DIR_NAME / parts[i + 1] / parts[i + 2] / name)
candidates += sorted((data_dir / TRASH_DIR_NAME).glob(f"*/*/{name}"))
return next((c for c in candidates if c.exists()), None)
def write_note_file(file_path: Path, content: str) -> None:
"""
Write note content to file atomically
@@ -695,6 +716,16 @@ def absolute_url_for(endpoint: str, **values) -> str:
return current_app.config["SITE_URL"].rstrip("/") + url_for(endpoint, **values)
def post_path(slug: str, title: Optional[str] = None) -> str:
"""A post's URL path: /article/<slug> when it has a title, else /note/<slug> (v2.17.0)"""
return f"/{'article' if title else 'note'}/{slug}"
def post_url(slug: str, title: Optional[str] = None) -> str:
"""A post's absolute URL (see post_path)"""
return absolute_url_for("public.article" if title else "public.note", slug=slug)
def iso_utc(value) -> str:
"""
Format a timestamp as RFC 3339 UTC for dt-* and Micropub (v2.8.0)
+44
View File
@@ -110,6 +110,50 @@ def get_webmentions_for_note(note_id: int) -> Dict[str, List[Dict[str, Any]]]:
return grouped
def response_counts(note_ids: List[int]) -> Dict[int, Dict[str, int]]:
"""
Likes, reposts, replies and quotes per note, for lists of notes
(v2.13.4; quotes v2.14.2).
Shown responses (valid and approved) plus the counts silos give without
names (Threads; people who opted out of backfeed) on synced crossposts.
Notes with none are left out.
"""
if not note_ids:
return {}
db = get_db()
marks = ','.join('?' * len(note_ids))
counts: Dict[int, Dict[str, int]] = {}
def add(note_id, key, n):
if n:
counts.setdefault(note_id, {'likes': 0, 'reposts': 0, 'replies': 0, 'quotes': 0})[key] += n
for row in db.execute(
f"""
SELECT target_note_id, mention_type, COUNT(*) FROM webmentions
WHERE target_note_id IN ({marks}) AND validation_status = 'valid'
AND moderation_status = 'approved' AND mention_type IN ('like', 'repost', 'reply', 'quote')
GROUP BY target_note_id, mention_type
""",
note_ids,
):
add(row[0], {'like': 'likes', 'repost': 'reposts', 'reply': 'replies', 'quote': 'quotes'}[row[1]], row[2])
for row in db.execute(
f"""
SELECT note_id, SUM(COALESCE(likes_count, 0)), SUM(COALESCE(reposts_count, 0)),
SUM(COALESCE(quotes_count, 0))
FROM syndications WHERE note_id IN ({marks}) AND state = 'synced'
GROUP BY note_id
""",
note_ids,
):
add(row[0], 'likes', row[1])
add(row[0], 'reposts', row[2])
add(row[0], 'quotes', row[3])
return counts
def list_webmentions(
status_filter: Optional[str] = None,
page: int = 1,
+7
View File
@@ -83,7 +83,9 @@ class WebmentionWorker(threading.Thread):
self._last_processed = None
self._lock_file = None
from starpunk.avatars import Upkeep
from starpunk.notifications import Sweeper
self._avatar_upkeep = Upkeep()
self._notifier = Sweeper()
def run(self):
"""
@@ -118,6 +120,11 @@ class WebmentionWorker(threading.Thread):
# Local copies of response avatars (v2.13.0)
self._avatar_upkeep.run(self.db_path, self.app.config['DATA_PATH'],
self.site_url)
# New responses to the owner (v2.14.0, ADR-074)
try:
self._notifier.run(self.app.config, self.db_path, self.site_url)
except Exception as e:
logger.error(f"Notification sweep failed: {e}", exc_info=True)
# Update last processed time
if processed > 0:
Loaded 100 of 132 files, more files were not shown because too many files have changed in this diff. Show more