Merge branch 'maint/v2.11.0' for v2.11.0 release
Build Container / build (push) Successful in 7m57s

This commit is contained in:
Phil committed 2026-11-03 13:19:44 -07:00
commit 3f5e7dc735
67 files changed
+885 -498

No files matched your search

+8 -4
View File
@@ -73,10 +73,10 @@ DATABASE_PATH=./data/starpunk.db
# FLASK / LOGGING # FLASK / LOGGING
# ============================================================================= # =============================================================================
# development or production (default: development) # development or production (default: production)
FLASK_ENV=development FLASK_ENV=production
# 1 = debug on, 0 = off (default: 1). Set 0 in production. # 1 = debug on, 0 = off (default: 0). `flask run --debug` turns it on locally.
FLASK_DEBUG=1 FLASK_DEBUG=0
# DEBUG, INFO, WARNING, ERROR or CRITICAL (default: INFO) # DEBUG, INFO, WARNING, ERROR or CRITICAL (default: INFO)
LOG_LEVEL=INFO LOG_LEVEL=INFO
# Overrides the version shown in the footer and /health (default: the # Overrides the version shown in the footer and /health (default: the
@@ -104,6 +104,10 @@ LOG_LEVEL=INFO
# (default: false) # (default: false)
TRUSTED_PROXY=false TRUSTED_PROXY=false
# CSRF tokens on admin forms, logout and the IndieAuth consent (default: true).
# Leave on; tests turn it off unless they test it.
CSRF_ENABLED=true
# ============================================================================= # =============================================================================
# FEEDS # FEEDS
# ============================================================================= # =============================================================================
+24
View File
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [2.11.0] - 2026-11-03
Maintenance release: dependency upgrades and the High-priority backlog items.
### Security
- Every dependency is upgraded to its current release. The previous set had 97 published vulnerabilities (Pillow 64, PyJWT 22, gunicorn 4, and Flask, Markdown, python-dotenv, pytest and pillow-heif); the new set has none known. Notable versions: Flask 3.1, gunicorn 26, Pillow 13, PyJWT 2.15, httpx 0.28, nh3 0.3, Markdown 3.11, pillow-heif 1.8, psutil 7
- Admin forms, logout and the IndieAuth consent screen now carry a CSRF token (an HMAC of your login session, no storage needed); requests without it are refused. Endpoints that apps call with tokens (Micropub, Webmention, WebSub, the token endpoint) are unaffected
- Fetching an IndieAuth app's page and fetching pages and images for crosspost link cards now use the same protected fetcher as webmentions: every redirect is checked against private addresses, the connection goes to the checked address, and the whole fetch has a time limit
- Debug mode is off and the environment is production unless set otherwise (`flask run --debug` still turns debug on locally); the container image sets both explicitly
### Changed
- Runs on Python 3.14 (was 3.11, which reaches end of life in 2027)
- Rate limits on `/webmention` and `/websub` are counted in the database and shared by all server processes. Before, each of the four processes counted separately, so the real limit was four times the setting
- Test dependencies are in `requirements-test.txt` (used by CI); `pytest` is no longer installed in the production image, and `lxml` and `python-dateutil`, which StarPunk uses directly, are now pinned
- `uv` is pinned in the Containerfile instead of `latest`
### Fixed
- Deleting a note or a photo left its tag, photo and image-size rows behind, because SQLite only applies the schema's cascades when foreign keys are switched on. They now are; migration 018 removes rows left by earlier deletes (production had three)
- Deprecated `datetime.utcnow()` and sqlite3's default date adapter (both removed in a future Python) replaced; the stored formats are unchanged
- `compose.yaml` pinned version 0.6.0 (which would override the real version shown) and set an unused variable; a stray duplicate `dev_auth.py` and a `test.ini` with a private host name were removed
## [2.10.2] - 2026-11-03 ## [2.10.2] - 2026-11-03
### Fixed ### Fixed
+1 -1
View File
@@ -7,7 +7,7 @@ This file contains operational instructions for Claude agents working on this pr
- We use **uv** for Python virtual environment management - We use **uv** for Python virtual environment management
- All Python commands must be run with `uv run` prefix - All Python commands must be run with `uv run` prefix
- Example: `uv run pytest`, `uv run flask --app app.py run --debug` - Example: `uv run pytest`, `uv run flask --app app.py run --debug`
- There is no `pyproject.toml`: create the environment with `uv venv --python 3.11 && uv pip install -r requirements-dev.txt` - There is no `pyproject.toml`: create the environment with `uv venv --python 3.14 && uv pip install -r requirements-dev.txt`
- Tests need `SESSION_SECRET` set: `SESSION_SECRET=test uv run pytest`. The full suite takes about 6 minutes, 5 of them in `tests/test_migration_race_condition.py`; add `--ignore=tests/test_migration_race_condition.py` while iterating, but run everything before a release - Tests need `SESSION_SECRET` set: `SESSION_SECRET=test uv run pytest`. The full suite takes about 6 minutes, 5 of them in `tests/test_migration_race_condition.py`; add `--ignore=tests/test_migration_race_condition.py` while iterating, but run everything before a release
- Local login without OIDC: `DEV_MODE=true` and `DEV_ADMIN_ME` in `.env`, then `/dev/login` - Local login without OIDC: `DEV_MODE=true` and `DEV_ADMIN_ME` in `.env`, then `/dev/login`
- Setup details: `docs/standards/development-setup.md` - Setup details: `docs/standards/development-setup.md`
+9 -6
View File
@@ -6,10 +6,10 @@
# ============================================================================ # ============================================================================
# Build Stage - Install dependencies in virtual environment # Build Stage - Install dependencies in virtual environment
# ============================================================================ # ============================================================================
FROM python:3.11-slim AS builder FROM python:3.14-slim AS builder
# Install uv for fast dependency installation # Install uv for fast dependency installation
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv COPY --from=ghcr.io/astral-sh/uv:0.9.28 /uv /usr/local/bin/uv
WORKDIR /build WORKDIR /build
@@ -25,11 +25,12 @@ RUN uv venv /opt/venv && \
# ============================================================================ # ============================================================================
# Test Stage - Runs the test suite (CI builds it with --target test) # Test Stage - Runs the test suite (CI builds it with --target test)
# ============================================================================ # ============================================================================
FROM python:3.11-slim AS test FROM python:3.14-slim AS test
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv COPY --from=ghcr.io/astral-sh/uv:0.9.28 /uv /usr/local/bin/uv
COPY --from=builder /opt/venv /opt/venv COPY --from=builder /opt/venv /opt/venv
RUN uv pip install --no-cache --python /opt/venv/bin/python pytest pytest-mock COPY requirements-test.txt .
RUN uv pip install --no-cache --python /opt/venv/bin/python -r requirements-test.txt
ENV PATH="/opt/venv/bin:$PATH" \ ENV PATH="/opt/venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \ PYTHONUNBUFFERED=1 \
@@ -45,7 +46,7 @@ RUN python -m pytest -q -p no:cacheprovider tests
# ============================================================================ # ============================================================================
# Runtime Stage - Minimal production image (default target: keep it last) # Runtime Stage - Minimal production image (default target: keep it last)
# ============================================================================ # ============================================================================
FROM python:3.11-slim FROM python:3.14-slim
# Create non-root user for security # Create non-root user for security
# UID/GID 1000 is standard for first user on most systems # UID/GID 1000 is standard for first user on most systems
@@ -60,6 +61,8 @@ COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" \ ENV PATH="/opt/venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \ PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \ PYTHONDONTWRITEBYTECODE=1 \
FLASK_ENV=production \
FLASK_DEBUG=0 \
FLASK_APP=app.py \ FLASK_APP=app.py \
DATA_PATH=/data \ DATA_PATH=/data \
NOTES_PATH=/data/notes \ NOTES_PATH=/data/notes \
+1 -1
View File
@@ -69,7 +69,7 @@ redirect URI with your OIDC provider. Details:
### Local development ### Local development
```bash ```bash
uv venv --python 3.11 uv venv --python 3.14
uv pip install -r requirements-dev.txt uv pip install -r requirements-dev.txt
cp .env.example .env # set SESSION_SECRET; add DEV_MODE=true and DEV_ADMIN_ME cp .env.example .env # set SESSION_SECRET; add DEV_MODE=true and DEV_ADMIN_ME
uv run flask --app app.py run --debug # http://localhost:5000 uv run flask --app app.py run --debug # http://localhost:5000
+2 -7
View File
@@ -11,12 +11,11 @@
# docker compose logs -f # docker compose logs -f
# docker compose down # docker compose down
version: '3.8'
services: services:
starpunk: starpunk:
# Container configuration # Container configuration
image: starpunk:0.6.0 # Built locally from this checkout (release images: see README)
image: starpunk:local
container_name: starpunk container_name: starpunk
# Build configuration # Build configuration
@@ -49,10 +48,6 @@ services:
- NOTES_PATH=/data/notes - NOTES_PATH=/data/notes
- DATABASE_PATH=/data/starpunk.db - DATABASE_PATH=/data/starpunk.db
# Application metadata
- VERSION=0.6.0
- ENVIRONMENT=production
# Volume mounts for persistent data # Volume mounts for persistent data
# All application data stored in ./container-data on host # All application data stored in ./container-data on host
volumes: volumes:
-103
View File
@@ -1,103 +0,0 @@
"""
Development authentication module for StarPunk
WARNING: This module provides a development-only authentication mechanism
that bypasses IndieLogin. It should NEVER be enabled in production.
This module is separate from production auth (starpunk/auth.py) to maintain
clear architectural boundaries and enable easy security audits.
Security measures:
- Only active when DEV_MODE=true
- Returns 404 if DEV_MODE=false
- Requires DEV_ADMIN_ME configuration
- Logs prominent warnings
- Cannot coexist with production SITE_URL
- Visual warnings in UI
Functions:
is_dev_mode: Check if development mode is enabled
validate_dev_config: Validate development configuration
create_dev_session: Create session without authentication
"""
from flask import current_app
from starpunk.auth import create_session
def is_dev_mode() -> bool:
"""
Check if development mode is enabled
Returns:
True if DEV_MODE is enabled, False otherwise
Example:
>>> from starpunk.dev_auth import is_dev_mode
>>> if is_dev_mode():
... print("Development mode active")
"""
return current_app.config.get("DEV_MODE", False)
def validate_dev_config() -> None:
"""
Validate development mode configuration
Checks that DEV_MODE configuration is valid and safe:
- DEV_ADMIN_ME must be set if DEV_MODE is true
- Warns if DEV_MODE is enabled with production-like SITE_URL
Raises:
ValueError: If DEV_MODE is true but DEV_ADMIN_ME is not set
Logs:
WARNING: If DEV_MODE is enabled with HTTPS SITE_URL
"""
dev_mode = current_app.config.get("DEV_MODE", False)
if dev_mode:
# Require DEV_ADMIN_ME
dev_admin_me = current_app.config.get("DEV_ADMIN_ME")
if not dev_admin_me:
raise ValueError("DEV_MODE=true requires DEV_ADMIN_ME to be set")
# Warn if production-like configuration detected
site_url = current_app.config.get("SITE_URL", "")
if site_url.startswith("https://"):
current_app.logger.warning(
"WARNING: DEV_MODE is enabled with production SITE_URL. "
"This is likely a misconfiguration. "
"DEV_MODE should only be used in local development."
)
def create_dev_session(me: str) -> str:
"""
Create development session without authentication
WARNING: This bypasses IndieLogin authentication entirely.
Only use in development environments.
Args:
me: User identity URL (from DEV_ADMIN_ME config)
Returns:
Session token (same format as production sessions)
Logs:
WARNING: Logs that dev session was created without authentication
Example:
>>> token = create_dev_session("https://example.com")
>>> # Session created without IndieLogin verification
"""
current_app.logger.warning(
f"DEV MODE: Creating session for {me} without authentication. "
f"This should NEVER happen in production!"
)
# Use production session creation (same session format)
# This ensures dev sessions work identically to production
return create_session(me)
+1 -1
View File
@@ -11,7 +11,7 @@ per-release designs and reports are in [../design/](../design/).
|----------|-------------| |----------|-------------|
| [overview.md](overview.md) | System overview: purpose, request flow, modules and routes, background workers, storage, auth model, URL surface, caching, security posture | | [overview.md](overview.md) | System overview: purpose, request flow, modules and routes, background workers, storage, auth model, URL surface, caching, security posture |
| [technology-stack.md](technology-stack.md) | Current runtime, Python dependencies, standards implemented, testing and CI | | [technology-stack.md](technology-stack.md) | Current runtime, Python dependencies, standards implemented, testing and CI |
| [database.md](database.md) | SQLite schema after migrations 001–017, migration runner and its lock, connection pool | | [database.md](database.md) | SQLite schema after migrations 001–019, migration runner and its lock, connection pool |
| [deployment.md](deployment.md) | Container image stages, CI workflow, release process, `/data` layout, reverse proxy, all environment variables | | [deployment.md](deployment.md) | Container image stages, CI workflow, release process, `/data` layout, reverse proxy, all environment variables |
| [simplified-auth-architecture.md](simplified-auth-architecture.md) | Admin login via OpenID Connect and the built-in IndieAuth server that issues Micropub tokens (ADR-064) | | [simplified-auth-architecture.md](simplified-auth-architecture.md) | Admin login via OpenID Connect and the built-in IndieAuth server that issues Micropub tokens (ADR-064) |
| [indieweb-protocols.md](indieweb-protocols.md) | How Micropub, IndieAuth, Webmention, microformats2, WebSub and the feeds are implemented, with compliance status and known gaps | | [indieweb-protocols.md](indieweb-protocols.md) | How Micropub, IndieAuth, Webmention, microformats2, WebSub and the feeds are implemented, with compliance status and known gaps |
+10 -3
View File
@@ -1,8 +1,8 @@
# Database Architecture # Database Architecture
**Version**: 2.10.1 (2026-11-03) **Version**: 2.11.0 (2026-11-03)
**Status**: Current. Verified against a database created by `create_app()` **Status**: Current. Verified against a database created by `create_app()`
with migrations 001–017. with migrations 001–019.
StarPunk uses one SQLite file, `starpunk.db` under `DATA_PATH` (`/data` in StarPunk uses one SQLite file, `starpunk.db` under `DATA_PATH` (`/data` in
the container), in WAL mode. Note *content* is not in the database: it lives the container), in WAL mode. Note *content* is not in the database: it lives
@@ -71,6 +71,8 @@ Background: [ADR-031](../decisions/ADR-031-database-migration-system-redesign.md
| `015_bookmarks_and_content_format.sql` | 2.9 | `notes.bookmark_of`, `notes.content_format` | | `015_bookmarks_and_content_format.sql` | 2.9 | `notes.bookmark_of`, `notes.content_format` |
| `016_websub.sql` | 2.10.0 | `websub_subscriptions`, `websub_intents`, `websub_deliveries`, `site_state` | | `016_websub.sql` | 2.10.0 | `websub_subscriptions`, `websub_intents`, `websub_deliveries`, `site_state` |
| `017_websub_hardening.sql` | 2.10.1 | `websub_intents.request_id`, `requested_topic`, `callback_ip`; `websub_subscriptions.callback_ip` | | `017_websub_hardening.sql` | 2.10.1 | `websub_intents.request_id`, `requested_topic`, `callback_ip`; `websub_subscriptions.callback_ip` |
| `018_foreign_key_cleanup.sql` | 2.11.0 | Removes rows orphaned while foreign keys weren't enforced |
| `019_shared_rate_limits.sql` | 2.11.0 | `rate_limits` (counters shared by all processes) |
New migrations are numbered `NNN_description.sql`, must be safe to run inside New migrations are numbered `NNN_description.sql`, must be safe to run inside
one transaction, and should use `IF NOT EXISTS` where SQLite allows it. one transaction, and should use `IF NOT EXISTS` where SQLite allows it.
@@ -205,13 +207,18 @@ its cached feed data is stale.
**`schema_migrations`**: applied migrations (see above). **`schema_migrations`**: applied migrations (see above).
**`rate_limits`** (v2.11.0): request counts for `/webmention` and `/websub`,
one row per key (`<limiter>:<ip or domain>`) and time window, so every
server process enforces one shared limit. Old windows are pruned as it goes.
## Connections ## Connections
- **Request connections**: `starpunk/database/pool.py`. `init_pool()` creates - **Request connections**: `starpunk/database/pool.py`. `init_pool()` creates
one `ConnectionPool` per process (up to 5 idle connections, 10-second busy one `ConnectionPool` per process (up to 5 idle connections, 10-second busy
timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the
`create_app()` config dict). Connections use `sqlite3.Row` rows, `create_app()` config dict). Connections use `sqlite3.Row` rows,
`check_same_thread=False` and `PRAGMA journal_mode=WAL`, and are wrapped `check_same_thread=False`, `PRAGMA journal_mode=WAL` and (since v2.11.0)
`PRAGMA foreign_keys=ON`, so `ON DELETE CASCADE`/`SET NULL` run, and are wrapped
in `MonitoredConnection` for query timing when metrics are on. `get_db()` in `MonitoredConnection` for query timing when metrics are on. `get_db()`
stores one connection in `flask.g` per request or app context; on teardown stores one connection in `flask.g` per request or app context; on teardown
it goes back to the pool, rolled back first if a transaction was left it goes back to the pool, rolled back first if a transaction was left
+4 -4
View File
@@ -24,12 +24,12 @@ deployment repository, not here.
## Container image (`Containerfile`) ## Container image (`Containerfile`)
A multi-stage build on `python:3.11-slim`; it works with Podman and Docker. A multi-stage build on `python:3.14-slim`; it works with Podman and Docker.
| Stage | Purpose | | Stage | Purpose |
|-------|---------| |-------|---------|
| `builder` | Copies `uv` from `ghcr.io/astral-sh/uv`, creates `/opt/venv` and installs `requirements.txt` into it | | `builder` | Copies `uv` from `ghcr.io/astral-sh/uv`, creates `/opt/venv` and installs `requirements.txt` into it |
| `test` | Copies the venv, adds `pytest` and `pytest-mock`, copies the source and runs `python -m pytest -q -p no:cacheprovider tests` with test values for `SESSION_SECRET`, `SITE_URL` and `ADMIN_ME`. The stage only builds if the tests pass. | | `test` | Copies the venv, installs `requirements-test.txt`, copies the source and runs `python -m pytest -q -p no:cacheprovider tests` with test values for `SESSION_SECRET`, `SITE_URL` and `ADMIN_ME`. The stage only builds if the tests pass. |
| runtime (last, the default target) | Creates user `starpunk` (UID/GID 1000) and `/data/notes`, copies the venv and the application to `/app`, sets `DATA_PATH=/data`, `NOTES_PATH=/data/notes`, `DATABASE_PATH=/data/starpunk.db`, runs as `starpunk`, exposes port 8000 | | runtime (last, the default target) | Creates user `starpunk` (UID/GID 1000) and `/data/notes`, copies the venv and the application to `/app`, sets `DATA_PATH=/data`, `NOTES_PATH=/data/notes`, `DATABASE_PATH=/data/starpunk.db`, runs as `starpunk`, exposes port 8000 |
The runtime image's command is gunicorn: The runtime image's command is gunicorn:
@@ -172,8 +172,8 @@ ignored with a warning.
| `DATA_PATH` | `./data` (image: `/data`) | Root of all stored data | | `DATA_PATH` | `./data` (image: `/data`) | Root of all stored data |
| `DATABASE_PATH` | `./data/starpunk.db` (image: `/data/starpunk.db`) | | | `DATABASE_PATH` | `./data/starpunk.db` (image: `/data/starpunk.db`) | |
| `NOTES_PATH` | `./data/notes` (image: `/data/notes`) | Created at start-up, but note files are always written to `DATA_PATH/notes` | | `NOTES_PATH` | `./data/notes` (image: `/data/notes`) | Created at start-up, but note files are always written to `DATA_PATH/notes` |
| `FLASK_ENV` | `development` | Set `production` in production | | `FLASK_ENV` | `production` | Reported in `/health` and admin diagnostics; the image sets `production` |
| `FLASK_DEBUG` | `1` | Set `0` in production (the image does not set it) | | `FLASK_DEBUG` | `0` | Debug off unless `1` (`flask run --debug` sets it locally); the image sets `0` |
| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` | | `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
| `VERSION` | package `__version__` | Overrides the reported version; normally unset | | `VERSION` | package `__version__` | Overrides the reported version; normally unset |
+3 -3
View File
@@ -197,7 +197,8 @@ route isn't registered at all.
1. Rate limits: 60 requests per minute per client IP and 10 per minute per 1. Rate limits: 60 requests per minute per client IP and 10 per minute per
source domain (`WEBMENTION_RATE_LIMIT_*`). Over the limit gets 429 with source domain (`WEBMENTION_RATE_LIMIT_*`). Over the limit gets 429 with
`Retry-After`. The limiter is in memory, per process. The client IP `Retry-After`. Counts are shared by all server processes (`rate_limits`
table, v2.11.0). The client IP
comes from `X-Forwarded-For` only when `TRUSTED_PROXY=true`. comes from `X-Forwarded-For` only when `TRUSTED_PROXY=true`.
2. Form-encoded (JSON also accepted). `source` and `target` are required, 2. Form-encoded (JSON also accepted). `source` and `target` are required,
http(s), at most 2048 characters, and different from each other. http(s), at most 2048 characters, and different from each other.
@@ -445,7 +446,7 @@ Drafts that stay drafts change nothing.
Limits (429 when hit): Limits (429 when hit):
- 30 requests per minute per client IP (per process; needs - 30 requests per minute per client IP (shared by all processes; needs
`TRUSTED_PROXY=true` behind a proxy); `TRUSTED_PROXY=true` behind a proxy);
- at most 200 pending verifications; - at most 200 pending verifications;
- `WEBSUB_MAX_SUBSCRIPTIONS` (1000) in total; - `WEBSUB_MAX_SUBSCRIPTIONS` (1000) in total;
@@ -488,7 +489,6 @@ retried.
- The hub serves only this site's topics, by design. websub.rocks' hub - The hub serves only this site's topics, by design. websub.rocks' hub
tests use their own topics, so they can't be run against it. tests use their own topics, so they can't be run against it.
- Rate limiting is per process and in memory.
- Deliveries always carry the full topic content (no diffs). Fat pings - Deliveries always carry the full topic content (no diffs). Fat pings
aren't sent to external hubs. aren't sent to external hubs.
+11 -10
View File
@@ -293,13 +293,12 @@ their credentials are set. The full list with defaults is in
callbacks, remote Micropub photos, IndieAuth client pages, crosspost link callbacks, remote Micropub photos, IndieAuth client pages, crosspost link
cards) are checked first: every address the host resolves to must be public cards) are checked first: every address the host resolves to must be public
(private, loopback, link-local, CGNAT, multicast and reserved ranges are (private, loopback, link-local, CGNAT, multicast and reserved ranges are
refused). The fetcher in `webmention_ssrf.py`, used for webmentions, reply refused). Every fetch of a user-supplied URL (webmentions, reply contexts,
contexts, WebSub and remote Micropub photos, follows redirects by hand and WebSub, remote Micropub photos, IndieAuth client pages and, since v2.11.0,
re-checks each hop, connects to the address it checked (no second DNS crosspost link cards) uses the fetcher in `webmention_ssrf.py`: it follows
lookup) and limits response size (1 MB by default) and total time. redirects by hand and re-checks each hop, connects to the address it
Crosspost link cards use their own check that also re-checks each redirect. checked (no second DNS lookup) and limits response size (1 MB by default)
The IndieAuth client-page fetch checks only the initial URL. `DEV_MODE` and total time. `DEV_MODE` relaxes the checks for local testing.
relaxes the checks for local testing.
- **HTML sanitising**: rendered notes (Markdown or HTML sent by a Micropub - **HTML sanitising**: rendered notes (Markdown or HTML sent by a Micropub
client) and webmention content are cleaned with nh3 allow-lists; search client) and webmention content are cleaned with nh3 allow-lists; search
highlighting escapes everything except `<mark>`. highlighting escapes everything except `<mark>`.
@@ -307,15 +306,17 @@ their credentials are set. The full list with defaults is in
(defaults 60 and 10 per 60 seconds); `/websub` 30 requests per IP per (defaults 60 and 10 per 60 seconds); `/websub` 30 requests per IP per
minute, plus limits on subscriptions in total, per callback host and per minute, plus limits on subscriptions in total, per callback host and per
callback address, and on requests waiting for verification. Request-rate callback address, and on requests waiting for verification. Request-rate
counters are in memory and per process. counters live in the `rate_limits` table, shared by all server processes
(v2.11.0).
- **Client IP behind a proxy**: with `TRUSTED_PROXY=true` the first address in - **Client IP behind a proxy**: with `TRUSTED_PROXY=true` the first address in
`X-Forwarded-For` is used as the client IP for rate limiting. Set it only `X-Forwarded-For` is used as the client IP for rate limiting. Set it only
when the app is reachable solely through a proxy that sets that header when the app is reachable solely through a proxy that sets that header
(Caddy does); without it, every client shares the proxy's address. (Caddy does); without it, every client shares the proxy's address.
- **Auth**: hashed session and token storage, single-use login state, PKCE - **Auth**: hashed session and token storage, single-use login state, PKCE
for both OIDC and IndieAuth, redirect URIs checked against the client, for both OIDC and IndieAuth, redirect URIs checked against the client,
post-login redirects limited to this site. Admin form POSTs rely on the post-login redirects limited to this site. Admin form POSTs, logout and
SameSite=Lax session cookie (there are no per-form CSRF tokens). the IndieAuth consent carry a CSRF token: an HMAC of the session cookie,
checked by `starpunk/csrf.py` (v2.11.0), on top of the SameSite=Lax cookie.
- **Uploads**: request bodies are capped at 50 MB; images are processed by - **Uploads**: request bodies are capped at 50 MB; images are processed by
Pillow and stored under random (UUID) names. Pillow and stored under random (UUID) names.
- **Container**: runs as a non-root user. HTTPS and security headers (HSTS, - **Container**: runs as a non-root user. HTTPS and security headers (HSTS,
@@ -328,14 +328,16 @@ In place:
untrusted redirect URIs never followed. untrusted redirect URIs never followed.
- Post-login redirects are limited to this site. - Post-login redirects are limited to this site.
CSRF (v2.11.0): admin form POSTs, `/auth/logout` and the consent form's
approve button need a `csrf_token` field (or `X-CSRF-Token` header) equal to
an HMAC-SHA256 of the `starpunk_session` cookie under `SECRET_KEY`; anything
else gets 400. The token changes with every login and needs no storage.
Endpoints apps call with bearer tokens are not affected. The client metadata
fetch uses `webmention_ssrf.safe_get` (every redirect hop checked, the
connection pinned, a 5-second limit).
Known gaps: Known gaps:
- Admin forms and the consent form have no CSRF token. They rely on the
`SameSite=Lax` session cookie, which cross-site POSTs don't carry.
- The client metadata fetch checks the `client_id` URL against private
addresses, but then makes a plain `httpx.get` that follows redirects.
Redirect hops are not re-checked and the connection is not pinned, unlike
`webmention_ssrf.safe_get`.
- `scopes_supported` leaves out `draft`, although Micropub honours it and - `scopes_supported` leaves out `draft`, although Micropub honours it and
the consent screen grants it when requested. Unknown scopes are also shown the consent screen grants it when requested. Unknown scopes are also shown
and granted as requested; they just have no effect. and granted as requested; they just have no effect.
@@ -246,10 +246,6 @@ to itself (v2.8.0).
- Bluesky and Threads copies don't follow edits (platform limitation). - Bluesky and Threads copies don't follow edits (platform limitation).
Unticking and re-ticking a target makes a new copy. Unticking and re-ticking a target makes a new copy.
- Link-card and Threads image fetches check every redirect hop against
private addresses but don't pin DNS, and use a per-request rather than an
overall time limit. Webmention and WebSub fetching use the stricter
`webmention_ssrf` fetcher.
- No per-target choice for Micropub updates: `mp-syndicate-to` is read only - No per-target choice for Micropub updates: `mp-syndicate-to` is read only
on create. on create.
- Likes of ordinary web pages, and all likes on Threads, stay on the site. - Likes of ordinary web pages, and all likes on Threads, stay on the site.
+28 -24
View File
@@ -14,9 +14,9 @@ the pieces fit together.
| Component | Choice | | Component | Choice |
|-----------|--------| |-----------|--------|
| Language | Python 3.11 (`python:3.11-slim` base image) | | Language | Python 3.14 (`python:3.14-slim` base image; since v2.11.0) |
| Web framework | Flask 3.0 (Werkzeug, Jinja2, itsdangerous, click come with it) | | Web framework | Flask 3.1 (Werkzeug, Jinja2, itsdangerous, click come with it) |
| WSGI server | gunicorn 21.2, 4 sync workers (see [deployment.md](deployment.md)) | | WSGI server | gunicorn 26.2, 4 sync workers (see [deployment.md](deployment.md)) |
| Database | SQLite (standard-library `sqlite3`), WAL mode, FTS5 for search | | Database | SQLite (standard-library `sqlite3`), WAL mode, FTS5 for search |
| Content storage | Markdown files on disk | | Content storage | Markdown files on disk |
| Background work | `threading` daemon threads; `fcntl.flock` file locks to pick one process | | Background work | `threading` daemon threads; `fcntl.flock` file locks to pick one process |
@@ -28,32 +28,36 @@ the pieces fit together.
## Python dependencies (`requirements.txt`) ## Python dependencies (`requirements.txt`)
Pinned to a minor series. Upgraded in v2.11.0, when the set had no known
vulnerabilities (`pip-audit`); check again before each release.
| Package | Pin | Used for | | Package | Pin | Used for |
|---------|-----|----------| |---------|-----|----------|
| Flask | `3.0.*` | Web framework | | Flask | `3.1.*` | Web framework |
| gunicorn | `21.2.*` | Production WSGI server | | gunicorn | `26.2.*` | Production WSGI server |
| markdown | `3.5.*` | Rendering notes (`extra`, `codehilite`, `nl2br` and a bare-URL linking extension) | | markdown | `3.11.*` | Rendering notes (`extra`, `codehilite`, `nl2br` and a bare-URL linking extension) |
| feedgen | `1.0.*` | RSS 2.0 feed (`starpunk/feeds/rss.py`); Atom and JSON Feed are generated by StarPunk's own code | | nh3 | `0.3.*` | HTML sanitising (rendered notes, webmention content) |
| httpx | `0.27.*` | All outbound HTTP: OIDC, IndieAuth client discovery, webmention and WebSub requests (through the SSRF-safe wrapper in `webmention_ssrf.py`), crossposting APIs, health check | | beautifulsoup4 | `4.15.*` | HTML parsing: IndieAuth client pages, crosspost text and link cards |
| python-dotenv | `1.0.*` | Loading `.env` | | lxml | `6.1.*` | Outgoing link extraction, webmention endpoint discovery and source checks |
| beautifulsoup4 | `4.12.*` | HTML parsing: IndieAuth client pages, crosspost text and link cards |
| mf2py | `2.0.*` | Parsing microformats2 (webmention sources, reply contexts, author h-card) | | mf2py | `2.0.*` | Parsing microformats2 (webmention sources, reply contexts, author h-card) |
| pytest | `8.0.*` | Test runner (listed here as well as in the image's test stage) | | python-dateutil | `4.0.*` | Dates in reply contexts |
| psutil | `5.9.*` | Memory monitor and system metrics | | feedgen | `1.0.*` | RSS 2.0 feed (`starpunk/feeds/rss.py`); Atom and JSON Feed are generated by StarPunk's own code |
| Pillow | `10.1.*` | Image validation, resizing and variants | | httpx | `0.28.*` | All outbound HTTP; requests to user-supplied URLs go through `webmention_ssrf.safe_get`/`safe_post` |
| pillow-heif | `0.18.*` | HEIC/HEIF uploads (iPhone photos) | | PyJWT[crypto] | `2.15.*` | Verifying OIDC ID tokens (JWKS, RS/ES signatures via `cryptography`) |
| nh3 | `0.2.*` | HTML sanitising (rendered notes, webmention content) | | python-dotenv | `1.2.*` | Loading `.env` |
| PyJWT[crypto] | `2.10.*` | Verifying OIDC ID tokens (JWKS, RS/ES signatures via `cryptography`) | | Pillow | `13.0.*` | Image validation, resizing and variants |
| pillow-heif | `1.8.*` | HEIC/HEIF uploads (iPhone photos) |
| psutil | `7.2.*` | Memory monitor and system metrics |
Packages imported directly but installed only as dependencies of the above: `markupsafe` comes with Flask/Jinja2.
`lxml` (outgoing link extraction, webmention endpoint discovery and source checks; from feedgen) and `python-dateutil` (dates in
reply contexts; from feedgen). `markupsafe` comes with Flask/Jinja2.
### Development dependencies (`requirements-dev.txt`) ### Test and development dependencies
Includes `requirements.txt`, plus `pytest-cov`, `pytest-mock`, `black`, - `requirements-test.txt`: `pytest` 9.2 and `pytest-mock` 3.15, used by the
`flake8`, `mypy` and `gunicorn`. Coding standards are in Containerfile test stage (CI).
[../standards/](../standards/). - `requirements-dev.txt`: runtime and test requirements plus `pytest-cov`,
`black`, `flake8`, `mypy` (not enforced) and `pip-audit`. Coding standards
are in [../standards/](../standards/).
### Deliberately not used ### Deliberately not used
@@ -94,7 +98,7 @@ Protocol details: [indieweb-protocols.md](indieweb-protocols.md),
``` ```
- **In CI**: the `test` stage of the `Containerfile` installs the production - **In CI**: the `test` stage of the `Containerfile` installs the production
requirements plus pytest and pytest-mock and runs the whole suite. The requirements plus `requirements-test.txt` and runs the whole suite. The
image is only built and pushed if it passes. Checklist: image is only built and pushed if it passes. Checklist:
[../standards/testing-checklist.md](../standards/testing-checklist.md). [../standards/testing-checklist.md](../standards/testing-checklist.md).
+5 -1
View File
@@ -21,7 +21,7 @@ is in `../architecture/`; decisions are in `../decisions/` (ADRs).
## Contents ## Contents
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [hotfix](#hotfix) [v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [hotfix](#hotfix)
## v1.0.0 ## v1.0.0
@@ -410,6 +410,10 @@ OIDC admin login and built-in IndieAuth server.
- [2026-11-03-reply-context-fetch.md](v2.10.2/2026-11-03-reply-context-fetch.md) - Reply contexts fetched again (fetch_source returns three values since v2.7.0) - [2026-11-03-reply-context-fetch.md](v2.10.2/2026-11-03-reply-context-fetch.md) - Reply contexts fetched again (fetch_source returns three values since v2.7.0)
## v2.11.0
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) - Dependency upgrades (Python 3.14), foreign keys, CSRF, fetchers, debug default, shared rate limits
## hotfix ## hotfix
Not a release folder. Not a release folder.
@@ -0,0 +1,60 @@
# v2.11.0: Maintenance release
Dependency upgrades and the five High items in the backlog (all re-checked
against the code first and still present).
## Dependencies
`pip-audit` on the v2.10.2 set found 97 published vulnerabilities in eight
packages (Pillow 64, PyJWT 22, gunicorn 4, Flask 2, Markdown 2,
python-dotenv 2, pytest 2, pillow-heif 1). All direct dependencies were moved
to their current minor series (`requirements.txt`); the new set has no known
vulnerabilities. Python moved from 3.11 to 3.14 (`python:3.14-slim`); the
whole suite passes with `-W error::DeprecationWarning`.
- `requirements-test.txt` (pytest, pytest-mock) is installed by the
Containerfile test stage; pytest left the runtime requirements.
- `lxml` and `python-dateutil` are pinned (used directly, previously only
transitive). `uv` is pinned to 0.9.28 in the Containerfile.
- `datetime.utcnow()` (34 calls in `starpunk/`, more in tests) became
`datetime.now(timezone.utc).replace(tzinfo=None)`: same naive-UTC values.
sqlite3's default datetime adapters, deprecated in 3.12, are registered
explicitly in `starpunk/database/__init__.py` with the same output.
## High backlog items
1. **Foreign keys.** `PRAGMA foreign_keys=ON` on every pooled connection
(`database/pool.py`); all deletes of notes, media and WebSub subscriptions
go through the pool. Every foreign key is `CASCADE` or `SET NULL`, so no
delete can be blocked. Migration 018 removes orphans (production: three
`note_tags` rows from hard-deleted notes). Crosspost rows still have no
foreign key on purpose.
2. **CSRF.** `starpunk/csrf.py`: token = HMAC-SHA256(SECRET_KEY,
"csrf:" + starpunk_session cookie); `before_request` checks the
`csrf_token` field or `X-CSRF-Token` header on every admin POST,
`auth.logout` and `indieauth.approve`; 400 otherwise. All 12 POST forms
carry it. `CSRF_ENABLED` (default true); test apps turn it off unless they
set it.
3. **Fetchers.** `indieauth.fetch_client_redirect_uris` and
`syndication._fetch_public` use `webmention_ssrf.safe_get` (per-hop
checks, DNS pinning, hard deadline). `safe_get` gained `truncate` for
reads that stop at the size limit.
4. **Debug default.** `FLASK_DEBUG` defaults to 0 and `FLASK_ENV` to
production; the runtime image sets both.
5. **Shared rate limits.** `SharedRateLimiter` keeps sliding-window counters
in `rate_limits` (migration 019): current window plus the previous one
weighted by overlap. Used by `/webmention` (per IP and per source domain)
and `/websub`; old windows are pruned every 200 calls.
## Housekeeping
`compose.yaml` no longer pins 0.6.0 or sets `VERSION`/`ENVIRONMENT`; the
unused root `dev_auth.py` and the committed `test.ini` were removed.
## Tests
New: `tests/test_csrf.py`, `tests/test_foreign_keys.py`,
`tests/test_shared_rate_limit.py`, IndieAuth client-metadata tests through the
fetcher (including a redirect to a private address). Crosspost tests route
link-card fetches through the fetcher's transport. Full suite: 1264 passed
on Python 3.14.
+9 -35
View File
@@ -1,9 +1,9 @@
# StarPunk Backlog # StarPunk Backlog
**Last Updated**: 2026-11-03 (current release: v2.10.1) **Last Updated**: 2026-11-03 (current release: v2.11.0)
These are the open features and bugs, sorted by priority. Each item was checked against the code These are the open features and bugs, sorted by priority. Each item was checked against the code
at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` and in at v2.11.0. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` and in
`CHANGELOG.md`. `CHANGELOG.md`.
## Priority Levels ## Priority Levels
@@ -23,30 +23,7 @@ at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` a
## High ## High
### SQLite foreign keys are not enforced (ON DELETE CASCADE is inert) None open.
- **Description**: No connection runs `PRAGMA foreign_keys = ON`, so the `ON DELETE CASCADE` clauses in the schema do nothing. When a note is deleted from the admin (a hard delete), its `note_tags`, `note_media`, `syndications` and webmention rows stay behind. `delete_media()` relies on the cascade to remove `media_variants` and `note_media` rows, so those are left orphaned too. Before turning the pragma on, check for existing orphan rows and add a migration that cleans them up. Each cascade also needs tests.
- **Where**: `starpunk/database/` (connection setup/pool), `starpunk/notes.py` (`delete_note`, hard delete), `starpunk/media.py` (`delete_media`), `migrations/007`–`010`, `016`
- **Source**: v2.10.x review; confirmed in code (there is no `foreign_keys` pragma anywhere in `starpunk/`)
### No CSRF tokens on admin and consent forms
- **Description**: Admin forms (create/edit/delete notes, webmention moderation, token revocation) and the IndieAuth consent form rely only on the `SameSite=Lax` session cookie. Lax blocks cross-site POSTs in current browsers, but a per-session CSRF token is the standard defence and covers older clients and same-site subdomains.
- **Where**: `starpunk/routes/admin.py`, `starpunk/routes/indieauth.py`, `templates/admin/*`
- **Source**: Documentation refresh 2026-11-03 (architecture review)
### Two outbound fetches don't use the hardened fetcher
- **Description**: IndieAuth client-metadata discovery (`indieauth.fetch_client_redirect_uris`) checks only the first URL, then uses `httpx.get(follow_redirects=True)`: redirect hops aren't checked and DNS isn't pinned. Crosspost link cards and Threads image fetches (`syndication._fetch_public`) check each hop but don't pin DNS, and their 10 s timeout is per read. Move both to `webmention_ssrf.safe_get`, as Webmention, WebSub and Micropub photo downloads already do.
- **Where**: `starpunk/indieauth.py`, `starpunk/syndication.py`
- **Source**: Documentation refresh 2026-11-03
### `FLASK_DEBUG` defaults to on
- **Description**: `config.py` reads `FLASK_DEBUG` with default `"1"` (and `FLASK_ENV` defaults to development); the runtime image sets neither. Production is safe only because the deployment sets `FLASK_DEBUG=0`. Default to off, or set it in the `Containerfile` runtime stage.
- **Where**: `starpunk/config.py`, `Containerfile`
- **Source**: Documentation refresh 2026-11-03 (production verified: DEBUG is False there)
### Rate limiters are per process
- **Description**: `RateLimiter` keeps its counters in memory, and gunicorn runs 4 workers. The WebSub hub limit of 30/min is therefore really 4×30/min, and the webmention endpoint limit is multiplied the same way. Consider a counter stored in SQLite that all workers share.
- **Where**: `starpunk/rate_limiter.py`, `starpunk/routes/websub.py`, `starpunk/routes/webmention.py`, `Containerfile` (`--workers 4`)
- **Source**: ADR-071 (Consequences: rate limiting bounds subscriber-chosen requests); v2.10.1 review
--- ---
@@ -92,20 +69,15 @@ at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` a
- **Where**: `starpunk/webmention_processor.py`, `starpunk/webmention_worker.py` - **Where**: `starpunk/webmention_processor.py`, `starpunk/webmention_worker.py`
- **Source**: v1.6.0 design reconstruction (docs/design/v1.6.0/background-worker.md) - **Source**: v1.6.0 design reconstruction (docs/design/v1.6.0/background-worker.md)
### Webmention rate limiter never cleaned up
- **Description**: `cleanup_rate_limiters()` is never called, so per-IP and per-domain keys accumulate for the life of each process.
- **Where**: `starpunk/rate_limiter.py`, `starpunk/routes/webmention.py`
- **Source**: v1.6.0 design reconstruction
### Atom and JSON Feed not linked from pages ### Atom and JSON Feed not linked from pages
- **Description**: `base.html` advertises only `/feed` (as RSS) and OPML with `rel=alternate`; add `<link rel="alternate">` for `/feed.atom` and `/feed.json` so readers can pick a format. - **Description**: `base.html` advertises only `/feed` (as RSS) and OPML with `rel=alternate`; add `<link rel="alternate">` for `/feed.atom` and `/feed.json` so readers can pick a format.
- **Where**: `templates/base.html` - **Where**: `templates/base.html`
- **Source**: Documentation refresh 2026-11-03 - **Source**: Documentation refresh 2026-11-03
### Housekeeping in the repository ### LICENSE placeholder
- **Description**: `compose.yaml` is stale (`image: starpunk:0.6.0`, `VERSION=0.6.0` would override the real version, unused `ENVIRONMENT`); the root `dev_auth.py` duplicates `starpunk/dev_auth.py` and is unused; `LICENSE` says "Copyright (c) 2024 [Your Name]"; `test.ini` contains a private host name; `lxml` and `python-dateutil` are used directly but only installed as dependencies of other packages, while `pytest` is pinned in the production `requirements.txt`. - **Description**: `LICENSE` still reads "Copyright (c) 2024 [Your Name]". Fill in the holder and year.
- **Where**: `compose.yaml`, `dev_auth.py`, `LICENSE`, `test.ini`, `requirements*.txt` - **Where**: `LICENSE`
- **Source**: Documentation refresh 2026-11-03 - **Source**: Documentation refresh 2026-11-03 (rest of that housekeeping item done in v2.11.0)
### Backfeed responses to crossposts ### Backfeed responses to crossposts
- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy. - **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy.
@@ -213,6 +185,8 @@ at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` a
These items were removed from the backlog because they have shipped: These items were removed from the backlog because they have shipped:
- v2.11.0: SQLite foreign keys enforced (orphans cleaned up by migration 018); CSRF tokens on admin, logout and consent forms; IndieAuth client and crosspost link-card fetches moved onto the SSRF-safe fetcher; `FLASK_DEBUG`/`FLASK_ENV` default to off/production; rate limits shared by all processes (and old windows pruned, so the webmention limiter no longer grows); dependencies upgraded (Python 3.14, no known vulnerabilities); stale `compose.yaml`, duplicate `dev_auth.py` and `test.ini` removed or fixed; `lxml`/`python-dateutil` pinned, `pytest` out of the runtime requirements
- POSSE: GoToSocial (v2.1.0), Bluesky (v2.2.0), Threads (v2.6.0) - POSSE: GoToSocial (v2.1.0), Bluesky (v2.2.0), Threads (v2.6.0)
- Webmentions: receive, send, display and moderation (v1.6.0); fixed and made spec-compliant in v2.6.8–v2.7.2 - Webmentions: receive, send, display and moderation (v1.6.0); fixed and made spec-compliant in v2.6.8–v2.7.2
- Reply contexts, replies, likes, reposts and bookmarks (v2.4.0, v2.5.0, v2.9.0) - Reply contexts, replies, likes, reposts and bookmarks (v2.4.0, v2.5.0, v2.9.0)
+2 -1
View File
@@ -4,7 +4,7 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
## Backlog ## Backlog
- **[BACKLOG.md](BACKLOG.md)** - Open features and bugs by priority (updated for v2.10.1) - **[BACKLOG.md](BACKLOG.md)** - Open features and bugs by priority (updated for v2.11.0)
## Releases ## Releases
@@ -47,3 +47,4 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
- **[v2.10.0](v2.10.0/RELEASE.md)** - WebSub (Released 2026-11-03) - **[v2.10.0](v2.10.0/RELEASE.md)** - WebSub (Released 2026-11-03)
- **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03) - **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03)
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03) - **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
- **[v2.11.0](v2.11.0/RELEASE.md)** - Maintenance: dependency upgrades, High backlog items (Released 2026-11-03)
+28
View File
@@ -0,0 +1,28 @@
# v2.11.0
- **Date**: 2026-11-03
- **Status**: Released
- **Focus**: Maintenance: dependency upgrades and the High backlog items
## Security
- All dependencies upgraded (97 known vulnerabilities before, none after); Python 3.14
- CSRF tokens on admin forms, logout and the IndieAuth consent
- IndieAuth client and crosspost link-card fetches use the SSRF-safe fetcher
- Debug off and production environment by default
## Changes
- Rate limits shared by all server processes
- `requirements-test.txt`; pytest out of the production image; `lxml` and `python-dateutil` pinned; `uv` pinned
## Bugs fixed
- Foreign keys enforced, so deletes cascade; orphan rows removed (migration 018)
- Deprecated `datetime.utcnow()` and sqlite3 adapters replaced
- Stale `compose.yaml`, duplicate `dev_auth.py` and `test.ini` cleaned up
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [Design note](../../design/v2.11.0/2026-11-03-maintenance-release.md)
+2 -2
View File
@@ -9,7 +9,7 @@ deployment is in [architecture/deployment.md](../architecture/deployment.md).
## Requirements ## Requirements
- Python 3.11 (the container image uses `python:3.11-slim`) - Python 3.14 (the container image uses `python:3.14-slim`)
- [uv](https://docs.astral.sh/uv/) for the virtual environment and packages ([ADR-006](../decisions/ADR-006-python-virtual-environment-uv.md)) - [uv](https://docs.astral.sh/uv/) for the virtual environment and packages ([ADR-006](../decisions/ADR-006-python-virtual-environment-uv.md))
- git - git
- Optional: Docker or Podman, to build the image and run the CI test stage locally - Optional: Docker or Podman, to build the image and run the CI test stage locally
@@ -23,7 +23,7 @@ pytest-mock, black, flake8, mypy).
```bash ```bash
git clone https://git.thesatelliteoflove.com/phil/StarPunk.git git clone https://git.thesatelliteoflove.com/phil/StarPunk.git
cd StarPunk cd StarPunk
uv venv --python 3.11 # creates .venv/ (gitignored) uv venv --python 3.14 # creates .venv/ (gitignored)
uv pip install -r requirements-dev.txt uv pip install -r requirements-dev.txt
``` ```
+1 -1
View File
@@ -63,7 +63,7 @@ cd ~/Documents/projects/StarPunk # main checkout, on main
git pull git pull
git worktree add ../StarPunk-worktrees/v2.10.1 -b fix/v2.10.1-websub-hardening main git worktree add ../StarPunk-worktrees/v2.10.1 -b fix/v2.10.1-websub-hardening main
cd ../StarPunk-worktrees/v2.10.1 cd ../StarPunk-worktrees/v2.10.1
uv venv --python 3.11 && uv pip install -r requirements-dev.txt uv venv --python 3.14 && uv pip install -r requirements-dev.txt
``` ```
### 2. Make the change ### 2. Make the change
+1 -1
View File
@@ -10,7 +10,7 @@ out." Patterns specific to small helper functions are in
## Base style ## Base style
- [PEP 8](https://peps.python.org/pep-0008/) naming and layout, Python 3.11. - [PEP 8](https://peps.python.org/pep-0008/) naming and layout, Python 3.14. Use `datetime.now(timezone.utc)`, not the deprecated `datetime.utcnow()`.
- **Match the file you are editing.** 4-space indents everywhere; quote - **Match the file you are editing.** 4-space indents everywhere; quote
style varies between modules (newer code uses double quotes); keep lines style varies between modules (newer code uses double quotes); keep lines
to roughly 100 characters. to roughly 100 characters.
+12
View File
@@ -0,0 +1,12 @@
-- Migration 018: Remove rows orphaned while foreign keys weren't enforced (v2.11.0)
-- From v2.11.0 every pooled connection runs PRAGMA foreign_keys=ON, so these
-- cascades happen on delete. This removes what earlier deletes left behind.
DELETE FROM note_tags WHERE note_id NOT IN (SELECT id FROM notes);
DELETE FROM note_tags WHERE tag_id NOT IN (SELECT id FROM tags);
DELETE FROM note_media WHERE note_id NOT IN (SELECT id FROM notes);
DELETE FROM note_media WHERE media_id NOT IN (SELECT id FROM media);
DELETE FROM media_variants WHERE media_id NOT IN (SELECT id FROM media);
DELETE FROM webmention_queue WHERE source_note_id IS NOT NULL AND source_note_id NOT IN (SELECT id FROM notes);
UPDATE webmentions SET target_note_id = NULL WHERE target_note_id IS NOT NULL AND target_note_id NOT IN (SELECT id FROM notes);
DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND subscription_id NOT IN (SELECT id FROM websub_subscriptions);
+9
View File
@@ -0,0 +1,9 @@
-- Migration 019: Rate limit counters shared by all server processes (v2.11.0)
-- One row per key and time window; see starpunk/rate_limiter.py SharedRateLimiter.
CREATE TABLE IF NOT EXISTS rate_limits (
key TEXT NOT NULL,
window INTEGER NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (key, window)
);
+7 -15
View File
@@ -1,17 +1,9 @@
# StarPunk Development Dependencies # StarPunk development dependencies
# Includes code quality and testing tools
# Include production dependencies
-r requirements.txt -r requirements.txt
-r requirements-test.txt
# Testing pytest-cov==7.* # Coverage reports
pytest-cov>=5.0.0 # Test coverage reporting black==26.* # Formatting (not enforced, see docs/standards)
pytest-mock>=3.12.0 # Mocking for tests flake8==7.* # Linting (not enforced)
mypy==1.* # Type checking (not enforced)
# Code Quality pip-audit==2.* # Known-vulnerability check of the dependency set
black>=24.0.0 # Code formatting
flake8>=7.0.0 # Linting
mypy>=1.8.0 # Type checking
# WSGI Server (for production-like testing)
gunicorn>=21.2.0
+3
View File
@@ -0,0 +1,3 @@
# Test dependencies: used by the Containerfile test stage and CI (v2.11.0)
pytest==9.2.*
pytest-mock==3.15.*
+21 -36
View File
@@ -1,42 +1,27 @@
# StarPunk Production Dependencies # StarPunk runtime dependencies (v2.11.0)
# Python 3.11+ required # Pinned to a minor series; see docs/architecture/technology-stack.md.
# Web Framework # Web framework and server
Flask==3.0.* Flask==3.1.*
gunicorn==26.2.*
# WSGI Server (Production) # Content
gunicorn==21.2.* markdown==3.11.*
nh3==0.3.*
# Content Processing beautifulsoup4==4.15.*
markdown==3.5.* lxml==6.1.*
mf2py==2.0.*
# Feed Generation python-dateutil==4.0.*
feedgen==1.0.* feedgen==1.0.*
# HTTP Client (for IndieAuth) # HTTP and auth
httpx==0.27.* httpx==0.28.*
PyJWT[crypto]==2.15.*
python-dotenv==1.2.*
# Configuration Management # Images
python-dotenv==1.0.* Pillow==13.0.*
pillow-heif==1.8.*
# HTML Parsing (for IndieAuth endpoint discovery) # Monitoring
beautifulsoup4==4.12.* psutil==7.2.*
# Microformats2 Parsing (v1.2.0)
mf2py==2.0.*
# Testing Framework
pytest==8.0.*
# System Monitoring (v1.1.2)
psutil==5.9.*
# Image Processing (v1.2.0, updated v1.4.2 for HEIC support)
Pillow==10.1.*
# HEIC/HEIF Support (v1.4.2 - iPhone photos)
pillow-heif==0.18.*
# HTML Sanitization (v1.6.0 - Webmention content)
nh3==0.2.*
PyJWT[crypto]==2.10.*
+6 -2
View File
@@ -309,6 +309,10 @@ def create_app(config=None):
response.headers.add('Link', '</webmention>; rel="webmention"') response.headers.add('Link', '</webmention>; rel="webmention"')
return response return response
# CSRF tokens on admin and consent forms (v2.11.0)
from starpunk import csrf
csrf.init_app(app)
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0) # WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
from starpunk.websub import add_discovery_headers, template_links from starpunk.websub import add_discovery_headers, template_links
app.after_request(add_discovery_headers) app.after_request(add_discovery_headers)
@@ -470,5 +474,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0) # Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details # See docs/standards/versioning-strategy.md for details
__version__ = "2.10.2" __version__ = "2.11.0"
__version_info__ = (2, 10, 2) __version_info__ = (2, 11, 0)
+3 -3
View File
@@ -32,7 +32,7 @@ import base64
import hashlib import hashlib
import secrets import secrets
import time import time
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from functools import wraps from functools import wraps
from typing import Any, Dict, Optional, Tuple from typing import Any, Dict, Optional, Tuple
from urllib.parse import urlencode from urllib.parse import urlencode
@@ -223,7 +223,7 @@ def initiate_login(next_url: Optional[str] = None) -> str:
nonce, nonce,
code_verifier, code_verifier,
_safe_next_url(next_url), _safe_next_url(next_url),
datetime.utcnow() + timedelta(minutes=LOGIN_STATE_MINUTES), datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=LOGIN_STATE_MINUTES),
), ),
) )
db.commit() db.commit()
@@ -361,7 +361,7 @@ def create_session(me: str) -> str:
# Calculate expiry (use configured session lifetime or default to 30 days) # Calculate expiry (use configured session lifetime or default to 30 days)
session_lifetime = current_app.config.get("SESSION_LIFETIME", 30) session_lifetime = current_app.config.get("SESSION_LIFETIME", 30)
expires_at = datetime.utcnow() + timedelta(days=session_lifetime) expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=session_lifetime)
current_app.logger.debug(f"Auth: Session expiry: {expires_at} ({session_lifetime} days)") current_app.logger.debug(f"Auth: Session expiry: {expires_at} ({session_lifetime} days)")
+3 -3
View File
@@ -22,7 +22,7 @@ Fallback Behavior (per Q14):
import json import json
import logging import logging
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from typing import Dict, Optional from typing import Dict, Optional
from urllib.parse import urlparse from urllib.parse import urlparse
@@ -236,7 +236,7 @@ def get_author_profile(me_url: str, refresh: bool = False) -> Dict:
if cached: if cached:
# Check if cache is still valid # Check if cache is still valid
cached_until = datetime.fromisoformat(cached['cached_until']) cached_until = datetime.fromisoformat(cached['cached_until'])
if datetime.utcnow() < cached_until: if datetime.now(timezone.utc).replace(tzinfo=None) < cached_until:
current_app.logger.debug(f"Using cached author profile for {me_url}") current_app.logger.debug(f"Using cached author profile for {me_url}")
# Parse rel_me_links from JSON # Parse rel_me_links from JSON
@@ -326,7 +326,7 @@ def save_author_profile(me_url: str, profile: Dict) -> None:
db = get_db(current_app) db = get_db(current_app)
# Calculate cache expiry (24 hours from now) # Calculate cache expiry (24 hours from now)
cached_until = datetime.utcnow() + timedelta(hours=CACHE_TTL_HOURS) cached_until = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(hours=CACHE_TTL_HOURS)
# Convert rel_me_links to JSON (per Q17) # Convert rel_me_links to JSON (per Q17)
rel_me_json = json.dumps(profile.get('rel_me_links', [])) rel_me_json = json.dumps(profile.get('rel_me_links', []))
+10 -2
View File
@@ -71,8 +71,10 @@ def load_config(app, config_override=None):
app.config["DATABASE_PATH"] = Path(os.getenv("DATABASE_PATH", "./data/starpunk.db")) app.config["DATABASE_PATH"] = Path(os.getenv("DATABASE_PATH", "./data/starpunk.db"))
# Flask environment # Flask environment
app.config["ENV"] = os.getenv("FLASK_ENV", "development") # Safe defaults (v2.11.0): production, debug off. `flask run --debug`
app.config["DEBUG"] = os.getenv("FLASK_DEBUG", "1") == "1" # sets FLASK_DEBUG=1 for local development.
app.config["ENV"] = os.getenv("FLASK_ENV", "production")
app.config["DEBUG"] = os.getenv("FLASK_DEBUG", "0") == "1"
# Logging # Logging
app.config["LOG_LEVEL"] = os.getenv("LOG_LEVEL", "INFO") app.config["LOG_LEVEL"] = os.getenv("LOG_LEVEL", "INFO")
@@ -153,8 +155,14 @@ def load_config(app, config_override=None):
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10")) app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
# Apply overrides if provided # Apply overrides if provided
# CSRF protection for admin and consent forms (v2.11.0); on by default.
# Tests turn it off unless they set CSRF_ENABLED themselves.
app.config["CSRF_ENABLED"] = os.getenv("CSRF_ENABLED", "true").lower() == "true"
if config_override: if config_override:
app.config.update(config_override) app.config.update(config_override)
if app.config.get("TESTING") and "CSRF_ENABLED" not in config_override:
app.config["CSRF_ENABLED"] = False
# Normalize SITE_URL trailing slash (in case override provided URL without slash) # Normalize SITE_URL trailing slash (in case override provided URL without slash)
if "SITE_URL" in app.config: if "SITE_URL" in app.config:
+56
View File
@@ -0,0 +1,56 @@
"""
CSRF protection for the admin and consent forms (v2.11.0)
The token is an HMAC of the login session cookie under the app secret: no
storage, tied to the session, and different after every login. Forms carry
it as a hidden `csrf_token` field (template global `csrf_token()`); POSTs to
protected endpoints without the right token get 400.
Protected: every admin POST, logout, and the IndieAuth consent ("approve").
Endpoints used by apps with bearer tokens (Micropub, token, revoke,
Webmention, WebSub) are not: they have no cookie-based authority.
"""
import hashlib
import hmac
from flask import abort, current_app, request
SESSION_COOKIE = "starpunk_session"
FIELD = "csrf_token"
HEADER = "X-CSRF-Token"
PROTECTED_ENDPOINTS = {"auth.logout", "indieauth.approve"}
def csrf_token() -> str:
"""The CSRF token for the current login session ('' when not logged in)"""
session = request.cookies.get(SESSION_COOKIE)
if not session:
return ""
key = str(current_app.config["SECRET_KEY"]).encode("utf-8")
return hmac.new(key, b"csrf:" + session.encode("utf-8"), hashlib.sha256).hexdigest()
def _protected() -> bool:
return request.blueprint == "admin" or request.endpoint in PROTECTED_ENDPOINTS
def check_csrf():
"""before_request: refuse state-changing requests without a valid token"""
if request.method in ("GET", "HEAD", "OPTIONS") or not current_app.config.get("CSRF_ENABLED", True):
return None
if not _protected():
return None
expected = csrf_token()
if not expected:
return None # not logged in: the route's own auth check refuses it
sent = request.form.get(FIELD) or request.headers.get(HEADER) or ""
if not hmac.compare_digest(sent, expected):
current_app.logger.warning(f"CSRF token missing or wrong for {request.endpoint}")
abort(400, description="The form expired or came from another site. Reload the page and try again.")
return None
def init_app(app) -> None:
app.before_request(check_csrf)
app.add_template_global(csrf_token, "csrf_token")
+8
View File
@@ -13,4 +13,12 @@ from starpunk.database.init import init_db
from starpunk.database.pool import init_pool, get_db, get_pool_stats from starpunk.database.pool import init_pool, get_db, get_pool_stats
from starpunk.database.schema import INITIAL_SCHEMA_SQL from starpunk.database.schema import INITIAL_SCHEMA_SQL
# Explicit datetime adapters (v2.11.0). Python 3.12 deprecated sqlite3's
# built-in ones; these store exactly what they did ("YYYY-MM-DD HH:MM:SS[.ffffff]").
import datetime as _datetime
import sqlite3 as _sqlite3
_sqlite3.register_adapter(_datetime.datetime, lambda value: value.isoformat(" "))
_sqlite3.register_adapter(_datetime.date, lambda value: value.isoformat())
__all__ = ['init_db', 'init_pool', 'get_db', 'get_pool_stats', 'INITIAL_SCHEMA_SQL'] __all__ = ['init_db', 'init_pool', 'get_db', 'get_pool_stats', 'INITIAL_SCHEMA_SQL']
+3
View File
@@ -68,6 +68,9 @@ class ConnectionPool:
# Enable WAL mode for better concurrency # Enable WAL mode for better concurrency
conn.execute("PRAGMA journal_mode=WAL") conn.execute("PRAGMA journal_mode=WAL")
# Enforce foreign keys, so ON DELETE CASCADE / SET NULL actually run
# (SQLite leaves them off per connection; v2.11.0)
conn.execute("PRAGMA foreign_keys=ON")
self._stats['connections_created'] += 1 self._stats['connections_created'] += 1
+2 -2
View File
@@ -14,7 +14,7 @@ Per v1.1.2 Phase 3:
Specification: http://opml.org/spec2.opml Specification: http://opml.org/spec2.opml
""" """
from datetime import datetime from datetime import datetime, timezone
from xml.sax.saxutils import escape from xml.sax.saxutils import escape
@@ -57,7 +57,7 @@ def generate_opml(site_url: str, site_name: str) -> str:
safe_site_name = escape(site_name) safe_site_name = escape(site_name)
# RFC 822 date format (required by OPML spec) # RFC 822 date format (required by OPML spec)
creation_date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT') creation_date = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%a, %d %b %Y %H:%M:%S GMT')
# Build OPML document # Build OPML document
opml_lines = [ opml_lines = [
+18 -17
View File
@@ -13,7 +13,7 @@ current IndieAuth spec: https://indieauth.spec.indieweb.org/
import base64 import base64
import hashlib import hashlib
import secrets import secrets
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional from typing import Any, Dict, List, Optional
from urllib.parse import urljoin, urlparse from urllib.parse import urljoin, urlparse
@@ -83,41 +83,42 @@ def fetch_client_redirect_uris(client_id: str) -> List[str]:
Supports a JSON client metadata document ("redirect_uris") and the older Supports a JSON client metadata document ("redirect_uris") and the older
HTML/Link header form (rel="redirect_uri"). Returns [] on any failure. HTML/Link header form (rel="redirect_uri"). Returns [] on any failure.
""" """
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch import json
from starpunk.webmention_ssrf import FetchTooLargeError, SSRFBlockedError, safe_get, user_agent
# Every redirect hop is checked and the connection pinned to the checked
# address, with a hard time limit (v2.11.0; DEV_MODE allows localhost)
try: try:
if not current_app.config.get("DEV_MODE"): result = safe_get(
validate_url_for_fetch(client_id)
response = httpx.get(
client_id, client_id,
timeout=5.0, headers={"Accept": "application/json, text/html;q=0.9",
follow_redirects=True, "User-Agent": user_agent(current_app.config.get("SITE_URL", ""))},
headers={"Accept": "application/json, text/html;q=0.9"}, max_bytes=500_000, timeout=5.0, truncate=True,
) )
except (httpx.HTTPError, SSRFBlockedError, ValueError) as e: except (httpx.HTTPError, SSRFBlockedError, ValueError, FetchTooLargeError) as e:
current_app.logger.info(f"IndieAuth: could not fetch client {client_id}: {e}") current_app.logger.info(f"IndieAuth: could not fetch client {client_id}: {e}")
return [] return []
if response.status_code != 200: if result.status_code != 200:
return [] return []
base = str(response.url) base = result.url
uris: List[str] = [] uris: List[str] = []
if "json" in response.headers.get("Content-Type", ""): if "json" in result.headers.get("Content-Type", ""):
try: try:
data = response.json() data = json.loads(result.text)
except ValueError: except ValueError:
return [] return []
if isinstance(data, dict) and isinstance(data.get("redirect_uris"), list): if isinstance(data, dict) and isinstance(data.get("redirect_uris"), list):
uris.extend(u for u in data["redirect_uris"] if isinstance(u, str)) uris.extend(u for u in data["redirect_uris"] if isinstance(u, str))
return uris return uris
link = response.links.get("redirect_uri") link = httpx.Response(200, headers=result.headers).links.get("redirect_uri")
if link and link.get("url"): if link and link.get("url"):
uris.append(urljoin(base, link["url"])) uris.append(urljoin(base, link["url"]))
soup = BeautifulSoup(response.text[:500_000], "html.parser") soup = BeautifulSoup(result.text, "html.parser")
for tag in soup.find_all(["link", "a"], href=True): for tag in soup.find_all(["link", "a"], href=True):
if "redirect_uri" in (tag.get("rel") or []): if "redirect_uri" in (tag.get("rel") or []):
uris.append(urljoin(base, tag["href"])) uris.append(urljoin(base, tag["href"]))
@@ -203,7 +204,7 @@ def create_authorization_code(
redirect_uri, redirect_uri,
scope, scope,
code_challenge, code_challenge,
datetime.utcnow() + timedelta(minutes=CODE_LIFETIME_MINUTES), datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=CODE_LIFETIME_MINUTES),
), ),
) )
db.commit() db.commit()
@@ -258,7 +259,7 @@ def create_access_token(client_id: str, scope: str) -> Dict[str, Any]:
"""Issue an access token (stored hashed); returns the token response""" """Issue an access token (stored hashed); returns the token response"""
token = secrets.token_urlsafe(32) token = secrets.token_urlsafe(32)
lifetime_days = current_app.config.get("INDIEAUTH_TOKEN_LIFETIME_DAYS", 90) lifetime_days = current_app.config.get("INDIEAUTH_TOKEN_LIFETIME_DAYS", 90)
expires_at = datetime.utcnow() + timedelta(days=lifetime_days) expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=lifetime_days)
db = get_db(current_app) db = get_db(current_app)
db.execute( db.execute(
+9 -9
View File
@@ -8,7 +8,7 @@ state. All models are immutable and use dataclasses for clean structure.
# Standard library imports # Standard library imports
from dataclasses import dataclass, field, replace from dataclasses import dataclass, field, replace
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
from typing import Any, Optional from typing import Any, Optional
import re import re
@@ -698,7 +698,7 @@ class Session:
>>> session.is_expired >>> session.is_expired
False False
""" """
return datetime.utcnow() >= self.expires_at return datetime.now(timezone.utc).replace(tzinfo=None) >= self.expires_at
@property @property
def is_active(self) -> bool: def is_active(self) -> bool:
@@ -726,7 +726,7 @@ class Session:
>>> session.age >>> session.age
datetime.timedelta(days=2, seconds=3600) datetime.timedelta(days=2, seconds=3600)
""" """
return datetime.utcnow() - self.created_at return datetime.now(timezone.utc).replace(tzinfo=None) - self.created_at
@property @property
def time_until_expiry(self) -> timedelta: def time_until_expiry(self) -> timedelta:
@@ -740,7 +740,7 @@ class Session:
>>> session.time_until_expiry >>> session.time_until_expiry
datetime.timedelta(days=28) datetime.timedelta(days=28)
""" """
return self.expires_at - datetime.utcnow() return self.expires_at - datetime.now(timezone.utc).replace(tzinfo=None)
def is_valid(self) -> bool: def is_valid(self) -> bool:
""" """
@@ -787,7 +787,7 @@ class Session:
>>> updated.last_used_at >>> updated.last_used_at
datetime.datetime(2024, 11, 18, 15, 30, 0) datetime.datetime(2024, 11, 18, 15, 30, 0)
""" """
return replace(self, last_used_at=datetime.utcnow()) return replace(self, last_used_at=datetime.now(timezone.utc).replace(tzinfo=None))
def to_dict(self) -> dict[str, Any]: def to_dict(self) -> dict[str, Any]:
""" """
@@ -863,7 +863,7 @@ class Token:
me: str me: str
client_id: Optional[str] = None client_id: Optional[str] = None
scope: Optional[str] = None scope: Optional[str] = None
created_at: datetime = field(default_factory=lambda: datetime.utcnow()) created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc).replace(tzinfo=None))
expires_at: Optional[datetime] = None expires_at: Optional[datetime] = None
@classmethod @classmethod
@@ -926,7 +926,7 @@ class Token:
# No expiry set, token never expires # No expiry set, token never expires
return False return False
return datetime.utcnow() >= self.expires_at return datetime.now(timezone.utc).replace(tzinfo=None) >= self.expires_at
@property @property
def is_active(self) -> bool: def is_active(self) -> bool:
@@ -1145,7 +1145,7 @@ class AuthState:
>>> auth_state.is_expired >>> auth_state.is_expired
False False
""" """
return datetime.utcnow() >= self.expires_at return datetime.now(timezone.utc).replace(tzinfo=None) >= self.expires_at
@property @property
def is_active(self) -> bool: def is_active(self) -> bool:
@@ -1169,7 +1169,7 @@ class AuthState:
>>> auth_state.age >>> auth_state.age
datetime.timedelta(seconds=120) datetime.timedelta(seconds=120)
""" """
return datetime.utcnow() - self.created_at return datetime.now(timezone.utc).replace(tzinfo=None) - self.created_at
def is_valid(self) -> bool: def is_valid(self) -> bool:
""" """
+2 -2
View File
@@ -24,7 +24,7 @@ import random
import time import time
from collections import deque from collections import deque
from dataclasses import dataclass, field, asdict from dataclasses import dataclass, field, asdict
from datetime import datetime from datetime import datetime, timezone
from threading import Lock from threading import Lock
from typing import Any, Deque, Dict, List, Literal, Optional, Union from typing import Any, Deque, Dict, List, Literal, Optional, Union
@@ -159,7 +159,7 @@ class MetricsBuffer:
metric = Metric( metric = Metric(
operation_type=operation_type, operation_type=operation_type,
operation_name=operation_name, operation_name=operation_name,
timestamp=datetime.utcnow().isoformat() + "Z", timestamp=datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
duration_ms=duration_ms, duration_ms=duration_ms,
process_id=self._process_id, process_id=self._process_id,
metadata=metadata or {} metadata=metadata or {}
+4 -4
View File
@@ -20,7 +20,7 @@ Exceptions:
""" """
# Standard library imports # Standard library imports
from datetime import datetime from datetime import datetime, timezone
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
@@ -226,7 +226,7 @@ def create_note(
# 2. SETUP # 2. SETUP
if created_at is None: if created_at is None:
created_at = datetime.utcnow() created_at = datetime.now(timezone.utc).replace(tzinfo=None)
updated_at = created_at # Same as created_at for new notes updated_at = created_at # Same as created_at for new notes
@@ -735,7 +735,7 @@ def update_note(
) )
# 3. SETUP # 3. SETUP
updated_at = datetime.utcnow() updated_at = datetime.now(timezone.utc).replace(tzinfo=None)
data_dir = Path(current_app.config["DATA_PATH"]) data_dir = Path(current_app.config["DATA_PATH"])
note_path = data_dir / existing_note.file_path note_path = data_dir / existing_note.file_path
@@ -1052,7 +1052,7 @@ def delete_note(
if soft: if soft:
# SOFT DELETE: Mark as deleted in database # SOFT DELETE: Mark as deleted in database
deleted_at = datetime.utcnow() deleted_at = datetime.now(timezone.utc).replace(tzinfo=None)
try: try:
db.execute( db.execute(
+76 -2
View File
@@ -138,6 +138,79 @@ _ip_limiter: RateLimiter | None = None
_domain_limiter: RateLimiter | None = None _domain_limiter: RateLimiter | None = None
class SharedRateLimiter:
"""
Rate limiter whose counts live in SQLite, shared by every gunicorn
process (v2.11.0). In-memory counters let each of the 4 workers allow the
full limit, so the real limit was four times the setting.
Sliding-window counter: the current window's count plus the previous
window's, weighted by how much of it still overlaps the last
`window_seconds`. Same interface as RateLimiter; needs an app context.
"""
CLEANUP_EVERY = 200 # calls between removals of old windows
def __init__(self, limit: int, window_seconds: int, prefix: str):
self.limit = limit
self.window_seconds = window_seconds
self.prefix = prefix
self._calls = 0
def _db(self):
from flask import current_app
from starpunk.database import get_db
return get_db(current_app)
def _windows(self):
now = time.time()
current = int(now // self.window_seconds)
overlap = 1 - (now % self.window_seconds) / self.window_seconds
return now, current, overlap
def _estimate(self, db, key: str) -> float:
_, current, overlap = self._windows()
counts = dict(db.execute(
"SELECT window, count FROM rate_limits WHERE key = ? AND window IN (?, ?)",
(f"{self.prefix}:{key}", current - 1, current),
).fetchall())
return counts.get(current - 1, 0) * overlap + counts.get(current, 0)
def is_allowed(self, key: str) -> bool:
db = self._db()
_, current, _ = self._windows()
if self._estimate(db, key) >= self.limit:
return False
db.execute(
"INSERT INTO rate_limits (key, window, count) VALUES (?, ?, 1) "
"ON CONFLICT(key, window) DO UPDATE SET count = count + 1",
(f"{self.prefix}:{key}", current),
)
db.commit()
self._calls += 1
if self._calls % self.CLEANUP_EVERY == 0:
self.cleanup()
return True
def get_retry_after(self, key: str) -> int:
now, current, _ = self._windows()
return max(1, int((current + 1) * self.window_seconds - now) + 1)
def get_request_count(self, key: str) -> int:
return int(self._estimate(self._db(), key))
def cleanup(self, max_idle_seconds: int = 300) -> int:
"""Remove windows too old to matter (this limiter's keys only)"""
_, current, _ = self._windows()
db = self._db()
cursor = db.execute(
"DELETE FROM rate_limits WHERE key LIKE ? AND window < ?",
(f"{self.prefix}:%", current - 1),
)
db.commit()
return cursor.rowcount
def init_rate_limiters(app) -> None: def init_rate_limiters(app) -> None:
""" """
Initialize rate limiters with Flask app configuration. Initialize rate limiters with Flask app configuration.
@@ -157,8 +230,9 @@ def init_rate_limiters(app) -> None:
domain_limit = app.config.get('WEBMENTION_RATE_LIMIT_DOMAIN', 10) domain_limit = app.config.get('WEBMENTION_RATE_LIMIT_DOMAIN', 10)
window = app.config.get('WEBMENTION_RATE_LIMIT_WINDOW', 60) window = app.config.get('WEBMENTION_RATE_LIMIT_WINDOW', 60)
_ip_limiter = RateLimiter(ip_limit, window) # Shared by all server processes (v2.11.0)
_domain_limiter = RateLimiter(domain_limit, window) _ip_limiter = SharedRateLimiter(ip_limit, window, "webmention-ip")
_domain_limiter = SharedRateLimiter(domain_limit, window, "webmention-domain")
app.logger.info( app.logger.info(
f"Webmention rate limiter initialized: {ip_limit}/min IP, " f"Webmention rate limiter initialized: {ip_limit}/min IP, "
+2 -2
View File
@@ -9,7 +9,7 @@ from flask import Blueprint, flash, g, jsonify, redirect, render_template, reque
import logging import logging
import os import os
import shutil import shutil
from datetime import datetime from datetime import datetime, timezone
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -531,7 +531,7 @@ def metrics():
from starpunk.monitoring.business import get_feed_statistics from starpunk.monitoring.business import get_feed_statistics
response = { response = {
"timestamp": datetime.utcnow().isoformat() + "Z", "timestamp": datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
"process_id": os.getpid(), "process_id": os.getpid(),
"database": {}, "database": {},
"performance": {}, "performance": {},
+2 -2
View File
@@ -6,7 +6,7 @@ No authentication required for these routes.
""" """
import hashlib import hashlib
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
@@ -54,7 +54,7 @@ def _get_cached_notes():
# Get cache duration from config (in seconds) # Get cache duration from config (in seconds)
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300) cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
cache_duration = timedelta(seconds=cache_seconds) cache_duration = timedelta(seconds=cache_seconds)
now = datetime.utcnow() now = datetime.now(timezone.utc).replace(tzinfo=None)
# The cached list is stale once any public note changed, in any process # The cached list is stale once any public note changed, in any process
# (v2.10.0): every change bumps content_version in the database # (v2.10.0): every change bumps content_version in the database
+4 -9
View File
@@ -9,16 +9,15 @@ optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
from flask import Blueprint, current_app, request from flask import Blueprint, current_app, request
from starpunk.database import get_db from starpunk.database import get_db
from starpunk.rate_limiter import RateLimiter from starpunk.rate_limiter import SharedRateLimiter
from starpunk import websub from starpunk import websub
bp = Blueprint("websub", __name__) bp = Blueprint("websub", __name__)
# Subscription requests per client IP per minute # Subscription requests per client IP per minute
# (per gunicorn process; needs TRUSTED_PROXY=true behind a reverse proxy, # Shared by all server processes (v2.11.0). Needs TRUSTED_PROXY=true behind a
# or every client shares the proxy's address) # reverse proxy, or every client shares the proxy's address.
_limiter = RateLimiter(limit=30, window_seconds=60) _limiter = SharedRateLimiter(limit=30, window_seconds=60, prefix="websub")
_requests = 0
def _text(message: str, status: int): def _text(message: str, status: int):
@@ -31,10 +30,6 @@ def hub():
return _text("WebSub is not enabled on this site", 404) return _text("WebSub is not enabled on this site", 404)
from starpunk.routes.webmention import get_client_ip from starpunk.routes.webmention import get_client_ip
global _requests
_requests += 1
if _requests % 500 == 0:
_limiter.cleanup() # forget idle clients so memory doesn't grow
if not _limiter.is_allowed(get_client_ip()): if not _limiter.is_allowed(get_client_ip()):
return _text("Too many requests", 429) return _text("Too many requests", 429)
+6 -6
View File
@@ -14,7 +14,7 @@ Per developer Q&A Q8:
import re import re
import unicodedata import unicodedata
import logging import logging
from datetime import datetime from datetime import datetime, timezone
from typing import Optional, Set from typing import Optional, Set
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -138,7 +138,7 @@ def sanitize_slug(slug: str, allow_timestamp_fallback: bool = False) -> str:
# Check if normalization resulted in empty slug # Check if normalization resulted in empty slug
if not slug and allow_timestamp_fallback: if not slug and allow_timestamp_fallback:
# Per Q8: Use timestamp-based fallback # Per Q8: Use timestamp-based fallback
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S') timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
slug = timestamp slug = timestamp
logger.warning( logger.warning(
f"Slug normalization failed for input '{original_slug}' " f"Slug normalization failed for input '{original_slug}' "
@@ -268,7 +268,7 @@ def generate_timestamp_slug(
'20251216143052-2' '20251216143052-2'
""" """
if created_at is None: if created_at is None:
created_at = datetime.utcnow() created_at = datetime.now(timezone.utc).replace(tzinfo=None)
if existing_slugs is None: if existing_slugs is None:
existing_slugs = set() existing_slugs = set()
@@ -340,7 +340,7 @@ def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]
if not sanitized: if not sanitized:
# This should never happen with allow_timestamp_fallback=True # This should never happen with allow_timestamp_fallback=True
# but handle it just in case # but handle it just in case
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S') timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
sanitized = timestamp sanitized = timestamp
logger.error( logger.error(
f"Unexpected empty slug after sanitization with fallback. " f"Unexpected empty slug after sanitization with fallback. "
@@ -361,7 +361,7 @@ def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]
if not validate_slug(sanitized): if not validate_slug(sanitized):
# This should rarely happen after sanitization # This should rarely happen after sanitization
# but if it does, use timestamp fallback # but if it does, use timestamp fallback
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S') timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
logger.warning( logger.warning(
f"Slug '{sanitized}' (from '{custom_slug}') failed validation. " f"Slug '{sanitized}' (from '{custom_slug}') failed validation. "
f"Using timestamp fallback: {timestamp}" f"Using timestamp fallback: {timestamp}"
@@ -375,7 +375,7 @@ def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]
except ValueError as e: except ValueError as e:
# This should rarely happen, but if it does, use timestamp # This should rarely happen, but if it does, use timestamp
# Per Q8: Never fail Micropub request # Per Q8: Never fail Micropub request
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S') timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
logger.error( logger.error(
f"Could not create unique slug from '{custom_slug}'. " f"Could not create unique slug from '{custom_slug}'. "
f"Using timestamp: {timestamp}. Error: {e}" f"Using timestamp: {timestamp}. Error: {e}"
+30 -32
View File
@@ -444,39 +444,37 @@ CARD_IMAGE_LIMIT = 5 * 1024 * 1024
def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple: def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple:
""" """
GET a public web URL, checking every redirect hop (no requests into the GET a public web URL (link card pages, images to re-upload).
LAN). Reads at most `limit` bytes: stops there if `truncate`, else fails.
Returns (data, content_type, final_url).
"""
from urllib.parse import urljoin, urlparse
from starpunk.micropub import _public_address
site = current_app.config.get("SITE_URL", "") Uses the shared SSRF-safe fetcher (v2.11.0): every redirect hop checked,
for _ in range(5): the connection pinned to the checked address, a hard 10 s limit. Reads at
parsed = urlparse(url) most `limit` bytes: stops there if `truncate`, else fails.
if parsed.scheme not in ("http", "https") or not parsed.hostname: Returns (data, content_type, final_url). `http` is no longer used.
raise SyndicationError(f"not a web URL: {url}", permanent=True) """
if not current_app.config.get("TESTING") and not _public_address(parsed.hostname): from urllib.parse import urlparse
raise SyndicationError(f"not a public address: {url}", permanent=True) from starpunk.webmention_ssrf import (
try: FetchTooLargeError, ResolutionError, SSRFBlockedError, safe_get, user_agent,
with http.stream("GET", url, follow_redirects=False, timeout=10.0, )
headers={"User-Agent": f"StarPunk (+{site})"}) as response:
if response.is_redirect and "location" in response.headers: parsed = urlparse(url)
url = urljoin(url, response.headers["location"]) if parsed.scheme not in ("http", "https") or not parsed.hostname:
continue raise SyndicationError(f"not a web URL: {url}", permanent=True)
if response.status_code >= 400: try:
raise SyndicationError(f"{url}: HTTP {response.status_code}") result = safe_get(url, headers={"User-Agent": user_agent(current_app.config.get("SITE_URL", ""))},
data = bytearray() max_bytes=limit, max_redirects=5, timeout=10.0, truncate=truncate)
for chunk in response.iter_bytes(): except ResolutionError as e:
data += chunk raise SyndicationError(f"{url}: {e}")
if len(data) > limit: except (SSRFBlockedError, ValueError) as e:
if truncate: raise SyndicationError(f"not a public address: {url} ({e})", permanent=True)
break except FetchTooLargeError:
raise SyndicationError(f"{url}: too large") raise SyndicationError(f"{url}: too large")
return bytes(data), response.headers.get("content-type", ""), url except httpx.TooManyRedirects:
except httpx.HTTPError as e: raise SyndicationError(f"{url}: too many redirects")
raise SyndicationError(f"{url}: {type(e).__name__}") except httpx.HTTPError as e:
raise SyndicationError(f"{url}: too many redirects") raise SyndicationError(f"{url}: {type(e).__name__}")
if result.status_code >= 400:
raise SyndicationError(f"{url}: HTTP {result.status_code}")
return result.content, result.headers.get("content-type", ""), result.url
def page_card(http: httpx.Client, url: str) -> dict: def page_card(http: httpx.Client, url: str) -> dict:
+2 -2
View File
@@ -217,7 +217,7 @@ def generate_slug(content: str, created_at: Optional[datetime] = None) -> str:
# If slug is empty or too short, use timestamp fallback # If slug is empty or too short, use timestamp fallback
if len(slug) < MIN_SLUG_LENGTH: if len(slug) < MIN_SLUG_LENGTH:
if created_at is None: if created_at is None:
created_at = datetime.utcnow() created_at = datetime.now(timezone.utc).replace(tzinfo=None)
slug = created_at.strftime("%Y%m%d-%H%M%S") slug = created_at.strftime("%Y%m%d-%H%M%S")
# Truncate to maximum length # Truncate to maximum length
@@ -592,7 +592,7 @@ def delete_note_file(
month = parts[notes_idx + 2] month = parts[notes_idx + 2]
except (ValueError, IndexError): except (ValueError, IndexError):
# If path doesn't follow expected structure, use current date # If path doesn't follow expected structure, use current date
now = datetime.utcnow() now = datetime.now(timezone.utc).replace(tzinfo=None)
year = now.strftime("%Y") year = now.strftime("%Y")
month = now.strftime("%m") month = now.strftime("%m")
+8 -2
View File
@@ -250,7 +250,8 @@ def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = N
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES, def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult: max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT,
truncate: bool = False) -> FetchResult:
""" """
GET a URL, following redirects by hand and checking each hop. GET a URL, following redirects by hand and checking each hop.
@@ -258,6 +259,8 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
url: URL to fetch url: URL to fetch
headers: Request headers headers: Request headers
max_bytes: Response body limit max_bytes: Response body limit
truncate: Stop at max_bytes and return what was read, instead of
raising FetchTooLargeError (v2.11.0)
max_redirects: Redirect limit max_redirects: Redirect limit
timeout: Total time limit in seconds timeout: Total time limit in seconds
@@ -287,13 +290,16 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
continue continue
content_length = response.headers.get('Content-Length', '') content_length = response.headers.get('Content-Length', '')
if content_length.isdigit() and int(content_length) > max_bytes: if not truncate and content_length.isdigit() and int(content_length) > max_bytes:
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)") raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
content = b'' content = b''
for chunk in response.iter_bytes(chunk_size=8192): for chunk in response.iter_bytes(chunk_size=8192):
content += chunk content += chunk
if len(content) > max_bytes: if len(content) > max_bytes:
if truncate:
content = content[:max_bytes]
break
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)") raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
if time.monotonic() > deadline: if time.monotonic() > deadline:
raise httpx.TimeoutException("Fetch exceeded time limit") raise httpx.TimeoutException("Fetch exceeded time limit")
+3 -3
View File
@@ -19,7 +19,7 @@ import threading
import time import time
import sqlite3 import sqlite3
import httpx import httpx
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
from typing import Dict, Any, Optional from typing import Dict, Any, Optional
from urllib.parse import urljoin from urllib.parse import urljoin
@@ -116,7 +116,7 @@ class WebmentionWorker(threading.Thread):
# Update last processed time # Update last processed time
if processed > 0: if processed > 0:
self._last_processed = datetime.utcnow() self._last_processed = datetime.now(timezone.utc).replace(tzinfo=None)
# Sleep if nothing processed # Sleep if nothing processed
if processed == 0: if processed == 0:
@@ -790,7 +790,7 @@ class WebmentionWorker(threading.Thread):
} }
delay = delays.get(new_retry_count) delay = delays.get(new_retry_count)
if delay: if delay:
next_retry_at = datetime.utcnow() + delay next_retry_at = datetime.now(timezone.utc).replace(tzinfo=None) + delay
try: try:
conn = sqlite3.connect(self.db_path) conn = sqlite3.connect(self.db_path)
+5 -5
View File
@@ -25,7 +25,7 @@ import hashlib
import hmac import hmac
import logging import logging
import threading import threading
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
from typing import Optional from typing import Optional
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
@@ -53,11 +53,11 @@ def signature(secret: str, body: bytes) -> str:
def _now() -> str: def _now() -> str:
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S") return datetime.now(timezone.utc).replace(tzinfo=None).strftime("%Y-%m-%d %H:%M:%S")
def _later(seconds: int) -> str: def _later(seconds: int) -> str:
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S") return (datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
class WebSubWorker(threading.Thread): class WebSubWorker(threading.Thread):
@@ -132,7 +132,7 @@ class WebSubWorker(threading.Thread):
).fetchall(): ).fetchall():
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop) self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
done += 1 done += 1
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY: if self._last_prune is None or datetime.now(timezone.utc).replace(tzinfo=None) - self._last_prune > PRUNE_EVERY:
self.prune(db) self.prune(db)
return done return done
@@ -348,7 +348,7 @@ class WebSubWorker(threading.Thread):
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND " db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)") "subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
db.commit() db.commit()
self._last_prune = datetime.utcnow() self._last_prune = datetime.now(timezone.utc).replace(tzinfo=None)
if cursor.rowcount: if cursor.rowcount:
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions") logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
return cursor.rowcount return cursor.rowcount
+1 -1
View File
@@ -16,7 +16,7 @@
</p> </p>
<p><small>After approval you will be sent to {{ auth_request.redirect_uri }}</small></p> <p><small>After approval you will be sent to {{ auth_request.redirect_uri }}</small></p>
<form action="{{ url_for('indieauth.approve') }}" method="POST" class="login-form"> <form action="{{ url_for('indieauth.approve') }}" method="POST" class="login-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
{% for name, value in fields.items() %} {% for name, value in fields.items() %}
<input type="hidden" name="{{ name }}" value="{{ value }}"> <input type="hidden" name="{{ name }}" value="{{ value }}">
{% endfor %} {% endfor %}
+1 -1
View File
@@ -10,7 +10,7 @@
<a href="{{ url_for('admin.new_note_form') }}">New Note</a> <a href="{{ url_for('admin.new_note_form') }}">New Note</a>
<a href="{{ url_for('admin.metrics_dashboard') }}">Metrics</a> <a href="{{ url_for('admin.metrics_dashboard') }}">Metrics</a>
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a> <a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"> <form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button button-secondary">Logout</button> <button type="submit" class="button button-secondary">Logout</button>
</form> </form>
</nav> </nav>
+1 -1
View File
@@ -74,7 +74,7 @@
<a href="{{ url_for('admin.edit_note_form', note_id=note.id) }}" class="button button-small"> <a href="{{ url_for('admin.edit_note_form', note_id=note.id) }}" class="button button-small">
Edit Edit
</a> </a>
<form action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note?');"> <form action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note?');"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<input type="hidden" name="confirm" value="yes"> <input type="hidden" name="confirm" value="yes">
<button type="submit" class="button button-small button-danger">Delete</button> <button type="submit" class="button button-small button-danger">Delete</button>
</form> </form>
+3 -3
View File
@@ -10,7 +10,7 @@
Created: {{ note.created_at.strftime('%B %d, %Y at %I:%M %p') }} Created: {{ note.created_at.strftime('%B %d, %Y at %I:%M %p') }}
</p> </p>
<form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form"> <form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<div class="form-group"> <div class="form-group">
{% if note.reacts_to %} {% if note.reacts_to %}
<p class="note-meta">{{ {'like': '★ Like of', 'repost': '🔁 Repost of', 'bookmark': '🔖 Bookmark of'}[note.kind] }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p> <p class="note-meta">{{ {'like': '★ Like of', 'repost': '🔁 Repost of', 'bookmark': '🔖 Bookmark of'}[note.kind] }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
@@ -76,7 +76,7 @@
</form> </form>
{# Outside the edit form: nested forms are not allowed in HTML #} {# Outside the edit form: nested forms are not allowed in HTML #}
<form id="delete-note-form" action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');"> <form id="delete-note-form" action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<input type="hidden" name="confirm" value="yes"> <input type="hidden" name="confirm" value="yes">
</form> </form>
@@ -98,7 +98,7 @@
{% endif %} {% endif %}
{% if s.last_error %}<br><small>{{ s.last_error }}</small>{% endif %} {% if s.last_error %}<br><small>{{ s.last_error }}</small>{% endif %}
{% if s.state != 'synced' %} {% if s.state != 'synced' %}
<form action="{{ url_for('admin.retry_syndication', syndication_id=s.id) }}" method="POST" style="display: inline;"> <form action="{{ url_for('admin.retry_syndication', syndication_id=s.id) }}" method="POST" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button button-secondary">Retry now</button> <button type="submit" class="button button-secondary">Retry now</button>
</form> </form>
{% endif %} {% endif %}
+1 -1
View File
@@ -6,7 +6,7 @@
<div class="note-editor"> <div class="note-editor">
<h2>Create New Note</h2> <h2>Create New Note</h2>
<form action="{{ url_for('admin.create_note_submit') }}" method="POST" enctype="multipart/form-data" class="note-form"> <form action="{{ url_for('admin.create_note_submit') }}" method="POST" enctype="multipart/form-data" class="note-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<div class="form-group"> <div class="form-group">
<label for="content">Content (Markdown)</label> <label for="content">Content (Markdown)</label>
<textarea <textarea
+1 -1
View File
@@ -29,7 +29,7 @@
<td>{{ t.last_used_at or "never" }}</td> <td>{{ t.last_used_at or "never" }}</td>
<td>{{ t.expires_at }}</td> <td>{{ t.expires_at }}</td>
<td> <td>
<form method="POST" action="{{ url_for('admin.revoke_token', token_id=t.id) }}"> <form method="POST" action="{{ url_for('admin.revoke_token', token_id=t.id) }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button button-danger" <button type="submit" class="button button-danger"
onclick="return confirm('Revoke this app\'s access?')">Revoke</button> onclick="return confirm('Revoke this app\'s access?')">Revoke</button>
</form> </form>
+3 -3
View File
@@ -83,21 +83,21 @@
{# Moderation Actions #} {# Moderation Actions #}
<div class="moderation-actions"> <div class="moderation-actions">
<h3>Actions</h3> <h3>Actions</h3>
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"> <form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<input type="hidden" name="action" value="approve"> <input type="hidden" name="action" value="approve">
<button type="submit" class="button button-primary" {% if webmention.moderation_status == 'approved' %}disabled{% endif %}> <button type="submit" class="button button-primary" {% if webmention.moderation_status == 'approved' %}disabled{% endif %}>
Approve Approve
</button> </button>
</form> </form>
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"> <form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<input type="hidden" name="action" value="hide"> <input type="hidden" name="action" value="hide">
<button type="submit" class="button" {% if webmention.moderation_status == 'hidden' %}disabled{% endif %}> <button type="submit" class="button" {% if webmention.moderation_status == 'hidden' %}disabled{% endif %}>
Hide Hide
</button> </button>
</form> </form>
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"> <form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<input type="hidden" name="action" value="delete"> <input type="hidden" name="action" value="delete">
<button type="submit" class="button button-danger" onclick="return confirm('Delete this webmention permanently?')"> <button type="submit" class="button button-danger" onclick="return confirm('Delete this webmention permanently?')">
Delete Delete
+1 -1
View File
@@ -29,7 +29,7 @@
</div> </div>
{% if webmentions %} {% if webmentions %}
<form method="POST" action="{{ url_for('admin.webmention_bulk_action') }}"> <form method="POST" action="{{ url_for('admin.webmention_bulk_action') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<div class="bulk-actions"> <div class="bulk-actions">
<button type="submit" name="action" value="approve" class="button"> <button type="submit" name="action" value="approve" class="button">
Approve Selected Approve Selected
-1
View File
@@ -1 +0,0 @@
cogitator-01.porgy-porgy.ts.net
+6 -6
View File
@@ -4,7 +4,7 @@ Tests for authentication module (starpunk/auth.py)
import hashlib import hashlib
import secrets import secrets
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
import httpx import httpx
@@ -102,7 +102,7 @@ class TestCleanup:
with app.app_context(): with app.app_context():
# Create expired session # Create expired session
token_hash = _hash_token("expired-token") token_hash = _hash_token("expired-token")
expires_at = datetime.utcnow() - timedelta(days=1) expires_at = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1)
db.execute( db.execute(
""" """
@@ -126,7 +126,7 @@ class TestCleanup:
"""Test that cleanup keeps valid sessions""" """Test that cleanup keeps valid sessions"""
with app.app_context(): with app.app_context():
token_hash = _hash_token("valid-token") token_hash = _hash_token("valid-token")
expires_at = datetime.utcnow() + timedelta(days=30) expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30)
db.execute( db.execute(
""" """
@@ -226,7 +226,7 @@ class TestVerifySession:
# Create expired session # Create expired session
token = secrets.token_urlsafe(32) token = secrets.token_urlsafe(32)
token_hash = _hash_token(token) token_hash = _hash_token(token)
expires_at = datetime.utcnow() - timedelta(days=1) expires_at = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1)
db.execute( db.execute(
""" """
@@ -340,7 +340,7 @@ class TestRequireAuthDecorator:
with app.app_context(): with app.app_context():
token = secrets.token_urlsafe(32) token = secrets.token_urlsafe(32)
token_hash = _hash_token(token) token_hash = _hash_token(token)
expires_at = datetime.utcnow() - timedelta(days=1) expires_at = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1)
db.execute( db.execute(
""" """
@@ -397,7 +397,7 @@ class TestSecurityFeatures:
).fetchone() ).fetchone()
expires_at = datetime.fromisoformat(result["expires_at"]) expires_at = datetime.fromisoformat(result["expires_at"])
created_at = datetime.utcnow() created_at = datetime.now(timezone.utc).replace(tzinfo=None)
# Should expire approximately 1 day from now # Should expire approximately 1 day from now
# (allow for minor timing differences) # (allow for minor timing differences)
+4 -4
View File
@@ -8,7 +8,7 @@ Per v1.2.0 Phase 2 and developer Q&A Q31-Q35:
""" """
import json import json
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from unittest.mock import Mock, patch from unittest.mock import Mock, patch
import pytest import pytest
@@ -203,7 +203,7 @@ class TestGetAuthorProfile:
from starpunk.database import get_db from starpunk.database import get_db
db = get_db(app) db = get_db(app)
expired_time = (datetime.utcnow() - timedelta(hours=48)).isoformat() expired_time = (datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=48)).isoformat()
db.execute( db.execute(
""" """
@@ -296,9 +296,9 @@ class TestSaveAuthorProfile:
'rel_me_links': [], 'rel_me_links': [],
} }
before_save = datetime.utcnow() before_save = datetime.now(timezone.utc).replace(tzinfo=None)
save_author_profile('https://cache.example.com', profile) save_author_profile('https://cache.example.com', profile)
after_save = datetime.utcnow() after_save = datetime.now(timezone.utc).replace(tzinfo=None)
# Check cache expiry # Check cache expiry
db = get_db(app) db = get_db(app)
+80
View File
@@ -0,0 +1,80 @@
"""
CSRF protection on admin and consent forms (v2.11.0)
"""
import re
import pytest
from starpunk import create_app
from starpunk.auth import create_session
@pytest.fixture
def app(tmp_path):
return create_app({
"TESTING": True, "CSRF_ENABLED": True, # the real check, unlike other tests
"SITE_URL": "http://localhost:5000", "ADMIN_ME": "https://test.example.com",
"SESSION_SECRET": "x", "DATA_PATH": tmp_path, "NOTES_PATH": tmp_path / "notes",
"DATABASE_PATH": tmp_path / "starpunk.db",
})
@pytest.fixture
def admin(app):
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session(app.config["ADMIN_ME"]))
return client
def token_from(page: bytes) -> str:
return re.search(rb'name="csrf_token" value="([0-9a-f]{64})"', page).group(1).decode()
def test_forms_carry_a_token(admin):
assert token_from(admin.get("/admin/new").data)
def test_post_without_token_is_refused(app, admin):
response = admin.post("/admin/new", data={"content": "Forged"})
assert response.status_code == 400
from starpunk.notes import list_notes
with app.app_context():
assert list_notes() == []
def test_post_with_wrong_token_is_refused(admin):
assert admin.post("/admin/new", data={"content": "x", "csrf_token": "0" * 64}).status_code == 400
def test_post_with_token_works(app, admin):
token = token_from(admin.get("/admin/new").data)
response = admin.post("/admin/new", data={"content": "Real", "csrf_token": token, "published": "on"})
assert response.status_code == 302
from starpunk.notes import list_notes
with app.app_context():
assert [n.content.strip() for n in list_notes()] == ["Real"]
def test_token_changes_with_the_session(app):
tokens = []
for _ in range(2):
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session(app.config["ADMIN_ME"]))
tokens.append(token_from(client.get("/admin/new").data))
assert tokens[0] != tokens[1]
def test_logout_and_consent_are_protected(admin):
assert admin.post("/auth/logout").status_code == 400
assert admin.post("/auth/authorize/approve", data={"client_id": "https://app.example/"}).status_code == 400
def test_app_endpoints_are_not_affected(app):
client = app.test_client()
# Bearer-token and public endpoints answer normally, without a CSRF token
assert client.post("/micropub", data={"h": "entry", "content": "x"}).status_code == 401
assert client.post("/auth/token", data={"grant_type": "authorization_code"}).status_code == 400
assert client.post("/webmention", data={}).status_code == 400
+66
View File
@@ -0,0 +1,66 @@
"""
Foreign keys are enforced (v2.11.0)
SQLite leaves foreign keys off per connection, so ON DELETE CASCADE never ran
and hard deletes left tag, media and variant rows behind.
"""
import io
import sqlite3
from pathlib import Path
from PIL import Image
from starpunk.database import get_db
from starpunk.migrations import execute_in_transaction
def png():
buf = io.BytesIO()
Image.new("RGB", (30, 20), "red").save(buf, format="PNG")
return buf.getvalue()
def count(app, sql):
return sqlite3.connect(app.config["DATABASE_PATH"]).execute(sql).fetchone()[0]
def test_pooled_connections_enforce_foreign_keys(app):
with app.app_context():
assert get_db(app).execute("PRAGMA foreign_keys").fetchone()[0] == 1
def test_hard_delete_removes_tags_and_media_links(app):
from starpunk.media import attach_media_to_note, save_media
from starpunk.notes import create_note, delete_note
with app.test_request_context():
note = create_note("Tagged with a photo", published=True, tags=["one", "two"], has_media=True)
media = save_media(png(), "a.png")
attach_media_to_note(note.id, [media["id"]], [""])
delete_note(id=note.id, soft=False)
assert count(app, "SELECT COUNT(*) FROM note_tags") == 0
assert count(app, "SELECT COUNT(*) FROM note_media") == 0
def test_deleting_media_removes_variants(app):
from starpunk.media import delete_media, save_media
with app.test_request_context():
media = save_media(png(), "b.png")
delete_media(media["id"])
assert count(app, "SELECT COUNT(*) FROM media_variants") == 0
def test_migration_018_removes_existing_orphans(app):
conn = sqlite3.connect(app.config["DATABASE_PATH"]) # foreign keys off here
conn.execute("INSERT INTO tags (name, display_name) VALUES ('t', 't')")
conn.execute("INSERT INTO note_tags (note_id, tag_id) VALUES (999, 1)")
conn.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) "
"VALUES ('https://s.example/', 'http://localhost:5000/note/x', 999)")
conn.commit()
sql = (Path(__file__).parent.parent / "migrations" / "018_foreign_key_cleanup.sql").read_text()
conn.execute("BEGIN")
execute_in_transaction(conn, sql)
conn.commit()
assert conn.execute("SELECT COUNT(*) FROM note_tags").fetchone()[0] == 0
assert conn.execute("SELECT target_note_id FROM webmentions").fetchone()[0] is None
assert conn.execute("PRAGMA foreign_key_check").fetchall() == []
+41 -14
View File
@@ -129,27 +129,54 @@ class TestValidation:
class TestClientMetadata: class TestClientMetadata:
def _response(self, content_type, body, links=None): """Client metadata is fetched through the SSRF-safe fetcher (v2.11.0)"""
response = MagicMock(status_code=200, text=body, url="https://quill.example/")
response.headers = {"Content-Type": content_type}
response.links = links or {}
response.json.return_value = __import__("json").loads(body) if "json" in content_type else None
return response
def test_json_metadata(self, app): @staticmethod
def _serve(monkeypatch, handler, private=()):
import httpx
def resolve(host, *args, **kwargs):
return [(2, 1, 6, "", ("10.0.0.5" if host in private else "93.184.216.34", 0))]
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
def test_json_metadata(self, app, monkeypatch):
import httpx
body = '{"client_id": "https://quill.example/", "redirect_uris": ["app://cb"]}' body = '{"client_id": "https://quill.example/", "redirect_uris": ["app://cb"]}'
with app.test_request_context(), patch( self._serve(monkeypatch, lambda r: httpx.Response(
"starpunk.webmention_ssrf.validate_url_for_fetch" 200, headers={"Content-Type": "application/json"}, text=body))
), patch("starpunk.indieauth.httpx.get", return_value=self._response("application/json", body)): with app.test_request_context():
assert fetch_client_redirect_uris(CLIENT_ID) == ["app://cb"] assert fetch_client_redirect_uris(CLIENT_ID) == ["app://cb"]
def test_html_link(self, app): def test_html_link(self, app, monkeypatch):
import httpx
body = '<html><head><link rel="redirect_uri" href="/cb"></head></html>' body = '<html><head><link rel="redirect_uri" href="/cb"></head></html>'
with app.test_request_context(), patch( self._serve(monkeypatch, lambda r: httpx.Response(
"starpunk.webmention_ssrf.validate_url_for_fetch" 200, headers={"Content-Type": "text/html"}, text=body))
), patch("starpunk.indieauth.httpx.get", return_value=self._response("text/html", body)): with app.test_request_context():
assert fetch_client_redirect_uris(CLIENT_ID) == ["https://quill.example/cb"] assert fetch_client_redirect_uris(CLIENT_ID) == ["https://quill.example/cb"]
def test_link_header(self, app, monkeypatch):
import httpx
self._serve(monkeypatch, lambda r: httpx.Response(
200, headers={"Content-Type": "text/html", "Link": '</cb2>; rel="redirect_uri"'}, text="<p>hi</p>"))
with app.test_request_context():
assert fetch_client_redirect_uris(CLIENT_ID) == ["https://quill.example/cb2"]
def test_redirect_to_private_address_refused(self, app, monkeypatch):
import httpx
fetched = []
def handler(request):
fetched.append(request.headers["host"])
return httpx.Response(302, headers={"Location": "http://internal.example/"})
self._serve(monkeypatch, handler, private={"internal.example"})
with app.test_request_context():
assert fetch_client_redirect_uris(CLIENT_ID) == []
assert "internal.example" not in fetched
class TestCodesAndTokens: class TestCodesAndTokens:
def test_full_exchange(self, app): def test_full_exchange(self, app):
+74 -74
View File
@@ -9,7 +9,7 @@ Organized by model:
""" """
import pytest import pytest
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from pathlib import Path from pathlib import Path
from starpunk.models import Note, Session, Token, AuthState, split_title from starpunk.models import Note, Session, Token, AuthState, split_title
from starpunk.utils import calculate_content_hash from starpunk.utils import calculate_content_hash
@@ -68,8 +68,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -89,8 +89,8 @@ class TestNoteModel:
slug="missing", slug="missing",
file_path="notes/2024/11/missing.md", file_path="notes/2024/11/missing.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -109,8 +109,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -130,8 +130,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -150,8 +150,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -168,8 +168,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -182,8 +182,8 @@ class TestNoteModel:
slug="my-test-note", slug="my-test-note",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=Path("/nonexistent"), _data_dir=Path("/nonexistent"),
) )
@@ -202,8 +202,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -222,8 +222,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -238,8 +238,8 @@ class TestNoteModel:
slug="my-note", slug="my-note",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=Path("/nonexistent"), _data_dir=Path("/nonexistent"),
) )
@@ -254,8 +254,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -289,8 +289,8 @@ class TestNoteModel:
slug="my-note", slug="my-note",
file_path="notes/2024/11/my-note.md", file_path="notes/2024/11/my-note.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=Path("data"), _data_dir=Path("data"),
) )
@@ -303,8 +303,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=Path("data"), _data_dir=Path("data"),
) )
@@ -342,8 +342,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -362,8 +362,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -386,8 +386,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
content_hash=content_hash, content_hash=content_hash,
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -410,8 +410,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
content_hash=content_hash, content_hash=content_hash,
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -433,8 +433,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
content_hash=None, content_hash=None,
_data_dir=tmp_path, _data_dir=tmp_path,
) )
@@ -449,8 +449,8 @@ class TestNoteModel:
slug="test", slug="test",
file_path="notes/2024/11/test.md", file_path="notes/2024/11/test.md",
published=True, published=True,
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
updated_at=datetime.utcnow(), updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
content_hash="abc123", content_hash="abc123",
_data_dir=Path("/nonexistent"), _data_dir=Path("/nonexistent"),
) )
@@ -499,8 +499,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
) )
assert session.is_expired is False assert session.is_expired is False
assert session.is_active is True assert session.is_active is True
@@ -511,21 +511,21 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow() - timedelta(days=31), created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=31),
expires_at=datetime.utcnow() - timedelta(days=1), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
) )
assert session.is_expired is True assert session.is_expired is True
assert session.is_active is False assert session.is_active is False
def test_age(self): def test_age(self):
"""Test age calculation""" """Test age calculation"""
created = datetime.utcnow() - timedelta(hours=2) created = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=2)
session = Session( session = Session(
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=created, created_at=created,
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
) )
age = session.age age = session.age
# Age should be at least 2 hours (account for test execution time) # Age should be at least 2 hours (account for test execution time)
@@ -537,8 +537,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
) )
time_left = session.time_until_expiry time_left = session.time_until_expiry
# Should be approximately 30 days # Should be approximately 30 days
@@ -550,8 +550,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow() - timedelta(days=31), created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=31),
expires_at=datetime.utcnow() - timedelta(days=1), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
) )
time_left = session.time_until_expiry time_left = session.time_until_expiry
assert time_left.total_seconds() < 0 assert time_left.total_seconds() < 0
@@ -562,8 +562,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
) )
assert session.is_valid() is True assert session.is_valid() is True
@@ -573,8 +573,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow() - timedelta(days=31), created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=31),
expires_at=datetime.utcnow() - timedelta(days=1), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
) )
assert session.is_valid() is False assert session.is_valid() is False
@@ -584,8 +584,8 @@ class TestSessionModel:
id=1, id=1,
session_token="", session_token="",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
) )
assert session.is_valid() is False assert session.is_valid() is False
@@ -595,8 +595,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="", me="",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
) )
assert session.is_valid() is False assert session.is_valid() is False
@@ -606,8 +606,8 @@ class TestSessionModel:
id=1, id=1,
session_token="abc123", session_token="abc123",
me="https://alice.example.com", me="https://alice.example.com",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(days=30), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
last_used_at=None, last_used_at=None,
) )
@@ -721,7 +721,7 @@ class TestTokenModel:
token = Token( token = Token(
token="xyz789", token="xyz789",
me="https://alice.example.com", me="https://alice.example.com",
expires_at=datetime.utcnow() + timedelta(days=90), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=90),
) )
assert token.is_expired is False assert token.is_expired is False
assert token.is_active is True assert token.is_active is True
@@ -731,7 +731,7 @@ class TestTokenModel:
token = Token( token = Token(
token="xyz789", token="xyz789",
me="https://alice.example.com", me="https://alice.example.com",
expires_at=datetime.utcnow() - timedelta(days=1), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
) )
assert token.is_expired is True assert token.is_expired is True
assert token.is_active is False assert token.is_active is False
@@ -746,7 +746,7 @@ class TestTokenModel:
token = Token( token = Token(
token="xyz789", token="xyz789",
me="https://alice.example.com", me="https://alice.example.com",
expires_at=datetime.utcnow() - timedelta(days=1), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
) )
assert token.is_valid() is False assert token.is_valid() is False
@@ -827,8 +827,8 @@ class TestAuthStateModel:
"""Test is_expired returns False for active state""" """Test is_expired returns False for active state"""
auth_state = AuthState( auth_state = AuthState(
state="random123", state="random123",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(minutes=5), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=5),
) )
assert auth_state.is_expired is False assert auth_state.is_expired is False
assert auth_state.is_active is True assert auth_state.is_active is True
@@ -837,19 +837,19 @@ class TestAuthStateModel:
"""Test is_expired returns True for expired state""" """Test is_expired returns True for expired state"""
auth_state = AuthState( auth_state = AuthState(
state="random123", state="random123",
created_at=datetime.utcnow() - timedelta(minutes=10), created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=10),
expires_at=datetime.utcnow() - timedelta(minutes=5), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=5),
) )
assert auth_state.is_expired is True assert auth_state.is_expired is True
assert auth_state.is_active is False assert auth_state.is_active is False
def test_age(self): def test_age(self):
"""Test age calculation""" """Test age calculation"""
created = datetime.utcnow() - timedelta(minutes=2) created = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=2)
auth_state = AuthState( auth_state = AuthState(
state="random123", state="random123",
created_at=created, created_at=created,
expires_at=datetime.utcnow() + timedelta(minutes=3), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=3),
) )
age = auth_state.age age = auth_state.age
# Age should be at least 2 minutes # Age should be at least 2 minutes
@@ -859,8 +859,8 @@ class TestAuthStateModel:
"""Test validation succeeds for active state""" """Test validation succeeds for active state"""
auth_state = AuthState( auth_state = AuthState(
state="random123", state="random123",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(minutes=5), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=5),
) )
assert auth_state.is_valid() is True assert auth_state.is_valid() is True
@@ -868,8 +868,8 @@ class TestAuthStateModel:
"""Test validation fails for expired state""" """Test validation fails for expired state"""
auth_state = AuthState( auth_state = AuthState(
state="random123", state="random123",
created_at=datetime.utcnow() - timedelta(minutes=10), created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=10),
expires_at=datetime.utcnow() - timedelta(minutes=5), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=5),
) )
assert auth_state.is_valid() is False assert auth_state.is_valid() is False
@@ -877,8 +877,8 @@ class TestAuthStateModel:
"""Test validation fails for empty state""" """Test validation fails for empty state"""
auth_state = AuthState( auth_state = AuthState(
state="", state="",
created_at=datetime.utcnow(), created_at=datetime.now(timezone.utc).replace(tzinfo=None),
expires_at=datetime.utcnow() + timedelta(minutes=5), expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=5),
) )
assert auth_state.is_valid() is False assert auth_state.is_valid() is False
+56
View File
@@ -0,0 +1,56 @@
"""
Rate limits are shared by all server processes (v2.11.0)
Each gunicorn process used to keep its own counts, so with 4 workers a client
got four times the limit. Two SharedRateLimiter instances on one database
stand in for two processes.
"""
import time
from unittest.mock import patch
from starpunk.rate_limiter import SharedRateLimiter
def test_two_processes_share_one_limit(app):
first = SharedRateLimiter(limit=3, window_seconds=60, prefix="t")
second = SharedRateLimiter(limit=3, window_seconds=60, prefix="t")
with app.app_context(), patch("starpunk.rate_limiter.time.time", return_value=6000.0):
results = [limiter.is_allowed("1.2.3.4") for limiter in (first, second, first, second)]
assert results == [True, True, True, False]
def test_keys_and_prefixes_are_separate(app):
a = SharedRateLimiter(limit=1, window_seconds=60, prefix="a")
b = SharedRateLimiter(limit=1, window_seconds=60, prefix="b")
with app.app_context(), patch("starpunk.rate_limiter.time.time", return_value=6000.0):
assert a.is_allowed("x") and a.is_allowed("y") and b.is_allowed("x")
assert not a.is_allowed("x")
def test_window_slides(app):
limiter = SharedRateLimiter(limit=2, window_seconds=60, prefix="s")
with app.app_context():
# Window 100 (6000-6059 s): two requests use the limit
with patch("starpunk.rate_limiter.time.time", return_value=6000.0):
assert limiter.is_allowed("k") and limiter.is_allowed("k")
assert not limiter.is_allowed("k")
# 6030 s is still window 100: still limited
with patch("starpunk.rate_limiter.time.time", return_value=6030.0):
assert not limiter.is_allowed("k")
# 6090 s is halfway through window 101: window 100 counts half (2 x 0.5 = 1)
with patch("starpunk.rate_limiter.time.time", return_value=6090.0):
assert limiter.is_allowed("k") # 1 + 0 < 2
assert not limiter.is_allowed("k") # 1 + 1 = 2
def test_cleanup_removes_old_windows(app):
import sqlite3
limiter = SharedRateLimiter(limit=5, window_seconds=60, prefix="c")
with app.app_context():
with patch("starpunk.rate_limiter.time.time", return_value=6000.0):
limiter.is_allowed("k")
with patch("starpunk.rate_limiter.time.time", return_value=9000.0):
limiter.cleanup()
rows = sqlite3.connect(app.config["DATABASE_PATH"]).execute("SELECT COUNT(*) FROM rate_limits").fetchone()[0]
assert rows == 0
+14 -9
View File
@@ -99,7 +99,8 @@ class FakeWeb:
self.fetched = [] self.fetched = []
def __call__(self, request): def __call__(self, request):
url = str(request.url) # Pinned requests go to the checked IP; the Host header names the site
url = f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
self.fetched.append(url) self.fetched.append(url)
status, headers, body = self.pages.get(url, (404, {}, b"nope")) status, headers, body = self.pages.get(url, (404, {}, b"nope"))
return httpx.Response(status, headers=headers, content=body) return httpx.Response(status, headers=headers, content=body)
@@ -112,11 +113,20 @@ def web():
def routed(pds, web): def routed(pds, web):
return httpx.MockTransport( return httpx.MockTransport(
lambda r: pds(r) if r.url.host in ("bsky.example", "pds.example") else web(r)) lambda r: pds(r) if r.headers.get("host", r.url.host) in ("bsky.example", "pds.example") else web(r))
def resolve(host, *args, **kwargs):
"""Every test host is public, except internal.example"""
ip = "10.0.0.5" if host == "internal.example" else "93.184.216.34"
return [(2, 1, 6, "", (ip, 0))]
@pytest.fixture @pytest.fixture
def app(tmp_path, pds, web): def app(tmp_path, pds, web, monkeypatch):
# Pages and images for link cards go through the shared SSRF-safe fetcher (v2.11.0)
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", routed(pds, web))
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
data = tmp_path / "data" data = tmp_path / "data"
app = create_app({ app = create_app({
"TESTING": True, "TESTING": True,
@@ -328,12 +338,7 @@ class TestLinkCards:
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch): def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"") web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
monkeypatch.setattr("starpunk.micropub._public_address", lambda host: host != "internal.example") publish(app, content=f"See {ARTICLE}") # internal.example resolves to 10.0.0.5
app.config["TESTING"] = False
try:
publish(app, content=f"See {ARTICLE}")
finally:
app.config["TESTING"] = True
assert "http://internal.example/admin" not in web.fetched assert "http://internal.example/admin" not in web.fetched
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE assert only_record(pds)["embed"]["external"]["title"] == ARTICLE