maint: Dependency upgrades and High backlog items (v2.11.0)
- All dependencies to current releases (97 known vulnerabilities -> 0), Python 3.14, uv pinned, requirements-test.txt, lxml/dateutil pinned - CSRF tokens on admin forms, logout and IndieAuth consent - IndieAuth client and crosspost link-card fetches via safe_get - FLASK_DEBUG/FLASK_ENV default off/production; image sets them - Foreign keys enforced on pooled connections; migration 018 removes orphans - Rate limits shared across processes (rate_limits, migration 019) - datetime.utcnow() and sqlite3 default adapters replaced - compose.yaml fixed; duplicate dev_auth.py and test.ini removed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9f17e145ff
commit
75946be7b2
67 files changed
+885
-498
No files matched your search
+8
-4
@@ -73,10 +73,10 @@ DATABASE_PATH=./data/starpunk.db
|
|||||||
# FLASK / LOGGING
|
# FLASK / LOGGING
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
||||||
# development or production (default: development)
|
# development or production (default: production)
|
||||||
FLASK_ENV=development
|
FLASK_ENV=production
|
||||||
# 1 = debug on, 0 = off (default: 1). Set 0 in production.
|
# 1 = debug on, 0 = off (default: 0). `flask run --debug` turns it on locally.
|
||||||
FLASK_DEBUG=1
|
FLASK_DEBUG=0
|
||||||
# DEBUG, INFO, WARNING, ERROR or CRITICAL (default: INFO)
|
# DEBUG, INFO, WARNING, ERROR or CRITICAL (default: INFO)
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
# Overrides the version shown in the footer and /health (default: the
|
# Overrides the version shown in the footer and /health (default: the
|
||||||
@@ -104,6 +104,10 @@ LOG_LEVEL=INFO
|
|||||||
# (default: false)
|
# (default: false)
|
||||||
TRUSTED_PROXY=false
|
TRUSTED_PROXY=false
|
||||||
|
|
||||||
|
# CSRF tokens on admin forms, logout and the IndieAuth consent (default: true).
|
||||||
|
# Leave on; tests turn it off unless they test it.
|
||||||
|
CSRF_ENABLED=true
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# FEEDS
|
# FEEDS
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|||||||
@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [2.11.0] - 2026-11-03
|
||||||
|
|
||||||
|
Maintenance release: dependency upgrades and the High-priority backlog items.
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
- Every dependency is upgraded to its current release. The previous set had 97 published vulnerabilities (Pillow 64, PyJWT 22, gunicorn 4, and Flask, Markdown, python-dotenv, pytest and pillow-heif); the new set has none known. Notable versions: Flask 3.1, gunicorn 26, Pillow 13, PyJWT 2.15, httpx 0.28, nh3 0.3, Markdown 3.11, pillow-heif 1.8, psutil 7
|
||||||
|
- Admin forms, logout and the IndieAuth consent screen now carry a CSRF token (an HMAC of your login session, no storage needed); requests without it are refused. Endpoints that apps call with tokens (Micropub, Webmention, WebSub, the token endpoint) are unaffected
|
||||||
|
- Fetching an IndieAuth app's page and fetching pages and images for crosspost link cards now use the same protected fetcher as webmentions: every redirect is checked against private addresses, the connection goes to the checked address, and the whole fetch has a time limit
|
||||||
|
- Debug mode is off and the environment is production unless set otherwise (`flask run --debug` still turns debug on locally); the container image sets both explicitly
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Runs on Python 3.14 (was 3.11, which reaches end of life in 2027)
|
||||||
|
- Rate limits on `/webmention` and `/websub` are counted in the database and shared by all server processes. Before, each of the four processes counted separately, so the real limit was four times the setting
|
||||||
|
- Test dependencies are in `requirements-test.txt` (used by CI); `pytest` is no longer installed in the production image, and `lxml` and `python-dateutil`, which StarPunk uses directly, are now pinned
|
||||||
|
- `uv` is pinned in the Containerfile instead of `latest`
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Deleting a note or a photo left its tag, photo and image-size rows behind, because SQLite only applies the schema's cascades when foreign keys are switched on. They now are; migration 018 removes rows left by earlier deletes (production had three)
|
||||||
|
- Deprecated `datetime.utcnow()` and sqlite3's default date adapter (both removed in a future Python) replaced; the stored formats are unchanged
|
||||||
|
- `compose.yaml` pinned version 0.6.0 (which would override the real version shown) and set an unused variable; a stray duplicate `dev_auth.py` and a `test.ini` with a private host name were removed
|
||||||
|
|
||||||
## [2.10.2] - 2026-11-03
|
## [2.10.2] - 2026-11-03
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ This file contains operational instructions for Claude agents working on this pr
|
|||||||
- We use **uv** for Python virtual environment management
|
- We use **uv** for Python virtual environment management
|
||||||
- All Python commands must be run with `uv run` prefix
|
- All Python commands must be run with `uv run` prefix
|
||||||
- Example: `uv run pytest`, `uv run flask --app app.py run --debug`
|
- Example: `uv run pytest`, `uv run flask --app app.py run --debug`
|
||||||
- There is no `pyproject.toml`: create the environment with `uv venv --python 3.11 && uv pip install -r requirements-dev.txt`
|
- There is no `pyproject.toml`: create the environment with `uv venv --python 3.14 && uv pip install -r requirements-dev.txt`
|
||||||
- Tests need `SESSION_SECRET` set: `SESSION_SECRET=test uv run pytest`. The full suite takes about 6 minutes, 5 of them in `tests/test_migration_race_condition.py`; add `--ignore=tests/test_migration_race_condition.py` while iterating, but run everything before a release
|
- Tests need `SESSION_SECRET` set: `SESSION_SECRET=test uv run pytest`. The full suite takes about 6 minutes, 5 of them in `tests/test_migration_race_condition.py`; add `--ignore=tests/test_migration_race_condition.py` while iterating, but run everything before a release
|
||||||
- Local login without OIDC: `DEV_MODE=true` and `DEV_ADMIN_ME` in `.env`, then `/dev/login`
|
- Local login without OIDC: `DEV_MODE=true` and `DEV_ADMIN_ME` in `.env`, then `/dev/login`
|
||||||
- Setup details: `docs/standards/development-setup.md`
|
- Setup details: `docs/standards/development-setup.md`
|
||||||
|
|||||||
+9
-6
@@ -6,10 +6,10 @@
|
|||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Build Stage - Install dependencies in virtual environment
|
# Build Stage - Install dependencies in virtual environment
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
FROM python:3.11-slim AS builder
|
FROM python:3.14-slim AS builder
|
||||||
|
|
||||||
# Install uv for fast dependency installation
|
# Install uv for fast dependency installation
|
||||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
COPY --from=ghcr.io/astral-sh/uv:0.9.28 /uv /usr/local/bin/uv
|
||||||
|
|
||||||
WORKDIR /build
|
WORKDIR /build
|
||||||
|
|
||||||
@@ -25,11 +25,12 @@ RUN uv venv /opt/venv && \
|
|||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Test Stage - Runs the test suite (CI builds it with --target test)
|
# Test Stage - Runs the test suite (CI builds it with --target test)
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
FROM python:3.11-slim AS test
|
FROM python:3.14-slim AS test
|
||||||
|
|
||||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
COPY --from=ghcr.io/astral-sh/uv:0.9.28 /uv /usr/local/bin/uv
|
||||||
COPY --from=builder /opt/venv /opt/venv
|
COPY --from=builder /opt/venv /opt/venv
|
||||||
RUN uv pip install --no-cache --python /opt/venv/bin/python pytest pytest-mock
|
COPY requirements-test.txt .
|
||||||
|
RUN uv pip install --no-cache --python /opt/venv/bin/python -r requirements-test.txt
|
||||||
|
|
||||||
ENV PATH="/opt/venv/bin:$PATH" \
|
ENV PATH="/opt/venv/bin:$PATH" \
|
||||||
PYTHONUNBUFFERED=1 \
|
PYTHONUNBUFFERED=1 \
|
||||||
@@ -45,7 +46,7 @@ RUN python -m pytest -q -p no:cacheprovider tests
|
|||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Runtime Stage - Minimal production image (default target: keep it last)
|
# Runtime Stage - Minimal production image (default target: keep it last)
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
FROM python:3.11-slim
|
FROM python:3.14-slim
|
||||||
|
|
||||||
# Create non-root user for security
|
# Create non-root user for security
|
||||||
# UID/GID 1000 is standard for first user on most systems
|
# UID/GID 1000 is standard for first user on most systems
|
||||||
@@ -60,6 +61,8 @@ COPY --from=builder /opt/venv /opt/venv
|
|||||||
ENV PATH="/opt/venv/bin:$PATH" \
|
ENV PATH="/opt/venv/bin:$PATH" \
|
||||||
PYTHONUNBUFFERED=1 \
|
PYTHONUNBUFFERED=1 \
|
||||||
PYTHONDONTWRITEBYTECODE=1 \
|
PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
FLASK_ENV=production \
|
||||||
|
FLASK_DEBUG=0 \
|
||||||
FLASK_APP=app.py \
|
FLASK_APP=app.py \
|
||||||
DATA_PATH=/data \
|
DATA_PATH=/data \
|
||||||
NOTES_PATH=/data/notes \
|
NOTES_PATH=/data/notes \
|
||||||
|
|||||||
@@ -69,7 +69,7 @@ redirect URI with your OIDC provider. Details:
|
|||||||
### Local development
|
### Local development
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv venv --python 3.11
|
uv venv --python 3.14
|
||||||
uv pip install -r requirements-dev.txt
|
uv pip install -r requirements-dev.txt
|
||||||
cp .env.example .env # set SESSION_SECRET; add DEV_MODE=true and DEV_ADMIN_ME
|
cp .env.example .env # set SESSION_SECRET; add DEV_MODE=true and DEV_ADMIN_ME
|
||||||
uv run flask --app app.py run --debug # http://localhost:5000
|
uv run flask --app app.py run --debug # http://localhost:5000
|
||||||
|
|||||||
+2
-7
@@ -11,12 +11,11 @@
|
|||||||
# docker compose logs -f
|
# docker compose logs -f
|
||||||
# docker compose down
|
# docker compose down
|
||||||
|
|
||||||
version: '3.8'
|
|
||||||
|
|
||||||
services:
|
services:
|
||||||
starpunk:
|
starpunk:
|
||||||
# Container configuration
|
# Container configuration
|
||||||
image: starpunk:0.6.0
|
# Built locally from this checkout (release images: see README)
|
||||||
|
image: starpunk:local
|
||||||
container_name: starpunk
|
container_name: starpunk
|
||||||
|
|
||||||
# Build configuration
|
# Build configuration
|
||||||
@@ -49,10 +48,6 @@ services:
|
|||||||
- NOTES_PATH=/data/notes
|
- NOTES_PATH=/data/notes
|
||||||
- DATABASE_PATH=/data/starpunk.db
|
- DATABASE_PATH=/data/starpunk.db
|
||||||
|
|
||||||
# Application metadata
|
|
||||||
- VERSION=0.6.0
|
|
||||||
- ENVIRONMENT=production
|
|
||||||
|
|
||||||
# Volume mounts for persistent data
|
# Volume mounts for persistent data
|
||||||
# All application data stored in ./container-data on host
|
# All application data stored in ./container-data on host
|
||||||
volumes:
|
volumes:
|
||||||
|
|||||||
-103
@@ -1,103 +0,0 @@
|
|||||||
"""
|
|
||||||
Development authentication module for StarPunk
|
|
||||||
|
|
||||||
WARNING: This module provides a development-only authentication mechanism
|
|
||||||
that bypasses IndieLogin. It should NEVER be enabled in production.
|
|
||||||
|
|
||||||
This module is separate from production auth (starpunk/auth.py) to maintain
|
|
||||||
clear architectural boundaries and enable easy security audits.
|
|
||||||
|
|
||||||
Security measures:
|
|
||||||
- Only active when DEV_MODE=true
|
|
||||||
- Returns 404 if DEV_MODE=false
|
|
||||||
- Requires DEV_ADMIN_ME configuration
|
|
||||||
- Logs prominent warnings
|
|
||||||
- Cannot coexist with production SITE_URL
|
|
||||||
- Visual warnings in UI
|
|
||||||
|
|
||||||
Functions:
|
|
||||||
is_dev_mode: Check if development mode is enabled
|
|
||||||
validate_dev_config: Validate development configuration
|
|
||||||
create_dev_session: Create session without authentication
|
|
||||||
"""
|
|
||||||
|
|
||||||
from flask import current_app
|
|
||||||
|
|
||||||
from starpunk.auth import create_session
|
|
||||||
|
|
||||||
|
|
||||||
def is_dev_mode() -> bool:
|
|
||||||
"""
|
|
||||||
Check if development mode is enabled
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
True if DEV_MODE is enabled, False otherwise
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> from starpunk.dev_auth import is_dev_mode
|
|
||||||
>>> if is_dev_mode():
|
|
||||||
... print("Development mode active")
|
|
||||||
"""
|
|
||||||
return current_app.config.get("DEV_MODE", False)
|
|
||||||
|
|
||||||
|
|
||||||
def validate_dev_config() -> None:
|
|
||||||
"""
|
|
||||||
Validate development mode configuration
|
|
||||||
|
|
||||||
Checks that DEV_MODE configuration is valid and safe:
|
|
||||||
- DEV_ADMIN_ME must be set if DEV_MODE is true
|
|
||||||
- Warns if DEV_MODE is enabled with production-like SITE_URL
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
ValueError: If DEV_MODE is true but DEV_ADMIN_ME is not set
|
|
||||||
|
|
||||||
Logs:
|
|
||||||
WARNING: If DEV_MODE is enabled with HTTPS SITE_URL
|
|
||||||
"""
|
|
||||||
dev_mode = current_app.config.get("DEV_MODE", False)
|
|
||||||
|
|
||||||
if dev_mode:
|
|
||||||
# Require DEV_ADMIN_ME
|
|
||||||
dev_admin_me = current_app.config.get("DEV_ADMIN_ME")
|
|
||||||
if not dev_admin_me:
|
|
||||||
raise ValueError("DEV_MODE=true requires DEV_ADMIN_ME to be set")
|
|
||||||
|
|
||||||
# Warn if production-like configuration detected
|
|
||||||
site_url = current_app.config.get("SITE_URL", "")
|
|
||||||
if site_url.startswith("https://"):
|
|
||||||
current_app.logger.warning(
|
|
||||||
"WARNING: DEV_MODE is enabled with production SITE_URL. "
|
|
||||||
"This is likely a misconfiguration. "
|
|
||||||
"DEV_MODE should only be used in local development."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def create_dev_session(me: str) -> str:
|
|
||||||
"""
|
|
||||||
Create development session without authentication
|
|
||||||
|
|
||||||
WARNING: This bypasses IndieLogin authentication entirely.
|
|
||||||
Only use in development environments.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
me: User identity URL (from DEV_ADMIN_ME config)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Session token (same format as production sessions)
|
|
||||||
|
|
||||||
Logs:
|
|
||||||
WARNING: Logs that dev session was created without authentication
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> token = create_dev_session("https://example.com")
|
|
||||||
>>> # Session created without IndieLogin verification
|
|
||||||
"""
|
|
||||||
current_app.logger.warning(
|
|
||||||
f"DEV MODE: Creating session for {me} without authentication. "
|
|
||||||
f"This should NEVER happen in production!"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Use production session creation (same session format)
|
|
||||||
# This ensures dev sessions work identically to production
|
|
||||||
return create_session(me)
|
|
||||||
@@ -11,7 +11,7 @@ per-release designs and reports are in [../design/](../design/).
|
|||||||
|----------|-------------|
|
|----------|-------------|
|
||||||
| [overview.md](overview.md) | System overview: purpose, request flow, modules and routes, background workers, storage, auth model, URL surface, caching, security posture |
|
| [overview.md](overview.md) | System overview: purpose, request flow, modules and routes, background workers, storage, auth model, URL surface, caching, security posture |
|
||||||
| [technology-stack.md](technology-stack.md) | Current runtime, Python dependencies, standards implemented, testing and CI |
|
| [technology-stack.md](technology-stack.md) | Current runtime, Python dependencies, standards implemented, testing and CI |
|
||||||
| [database.md](database.md) | SQLite schema after migrations 001–017, migration runner and its lock, connection pool |
|
| [database.md](database.md) | SQLite schema after migrations 001–019, migration runner and its lock, connection pool |
|
||||||
| [deployment.md](deployment.md) | Container image stages, CI workflow, release process, `/data` layout, reverse proxy, all environment variables |
|
| [deployment.md](deployment.md) | Container image stages, CI workflow, release process, `/data` layout, reverse proxy, all environment variables |
|
||||||
| [simplified-auth-architecture.md](simplified-auth-architecture.md) | Admin login via OpenID Connect and the built-in IndieAuth server that issues Micropub tokens (ADR-064) |
|
| [simplified-auth-architecture.md](simplified-auth-architecture.md) | Admin login via OpenID Connect and the built-in IndieAuth server that issues Micropub tokens (ADR-064) |
|
||||||
| [indieweb-protocols.md](indieweb-protocols.md) | How Micropub, IndieAuth, Webmention, microformats2, WebSub and the feeds are implemented, with compliance status and known gaps |
|
| [indieweb-protocols.md](indieweb-protocols.md) | How Micropub, IndieAuth, Webmention, microformats2, WebSub and the feeds are implemented, with compliance status and known gaps |
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# Database Architecture
|
# Database Architecture
|
||||||
|
|
||||||
**Version**: 2.10.1 (2026-11-03)
|
**Version**: 2.11.0 (2026-11-03)
|
||||||
**Status**: Current. Verified against a database created by `create_app()`
|
**Status**: Current. Verified against a database created by `create_app()`
|
||||||
with migrations 001–017.
|
with migrations 001–019.
|
||||||
|
|
||||||
StarPunk uses one SQLite file, `starpunk.db` under `DATA_PATH` (`/data` in
|
StarPunk uses one SQLite file, `starpunk.db` under `DATA_PATH` (`/data` in
|
||||||
the container), in WAL mode. Note *content* is not in the database: it lives
|
the container), in WAL mode. Note *content* is not in the database: it lives
|
||||||
@@ -71,6 +71,8 @@ Background: [ADR-031](../decisions/ADR-031-database-migration-system-redesign.md
|
|||||||
| `015_bookmarks_and_content_format.sql` | 2.9 | `notes.bookmark_of`, `notes.content_format` |
|
| `015_bookmarks_and_content_format.sql` | 2.9 | `notes.bookmark_of`, `notes.content_format` |
|
||||||
| `016_websub.sql` | 2.10.0 | `websub_subscriptions`, `websub_intents`, `websub_deliveries`, `site_state` |
|
| `016_websub.sql` | 2.10.0 | `websub_subscriptions`, `websub_intents`, `websub_deliveries`, `site_state` |
|
||||||
| `017_websub_hardening.sql` | 2.10.1 | `websub_intents.request_id`, `requested_topic`, `callback_ip`; `websub_subscriptions.callback_ip` |
|
| `017_websub_hardening.sql` | 2.10.1 | `websub_intents.request_id`, `requested_topic`, `callback_ip`; `websub_subscriptions.callback_ip` |
|
||||||
|
| `018_foreign_key_cleanup.sql` | 2.11.0 | Removes rows orphaned while foreign keys weren't enforced |
|
||||||
|
| `019_shared_rate_limits.sql` | 2.11.0 | `rate_limits` (counters shared by all processes) |
|
||||||
|
|
||||||
New migrations are numbered `NNN_description.sql`, must be safe to run inside
|
New migrations are numbered `NNN_description.sql`, must be safe to run inside
|
||||||
one transaction, and should use `IF NOT EXISTS` where SQLite allows it.
|
one transaction, and should use `IF NOT EXISTS` where SQLite allows it.
|
||||||
@@ -205,13 +207,18 @@ its cached feed data is stale.
|
|||||||
|
|
||||||
**`schema_migrations`**: applied migrations (see above).
|
**`schema_migrations`**: applied migrations (see above).
|
||||||
|
|
||||||
|
**`rate_limits`** (v2.11.0): request counts for `/webmention` and `/websub`,
|
||||||
|
one row per key (`<limiter>:<ip or domain>`) and time window, so every
|
||||||
|
server process enforces one shared limit. Old windows are pruned as it goes.
|
||||||
|
|
||||||
## Connections
|
## Connections
|
||||||
|
|
||||||
- **Request connections**: `starpunk/database/pool.py`. `init_pool()` creates
|
- **Request connections**: `starpunk/database/pool.py`. `init_pool()` creates
|
||||||
one `ConnectionPool` per process (up to 5 idle connections, 10-second busy
|
one `ConnectionPool` per process (up to 5 idle connections, 10-second busy
|
||||||
timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the
|
timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the
|
||||||
`create_app()` config dict). Connections use `sqlite3.Row` rows,
|
`create_app()` config dict). Connections use `sqlite3.Row` rows,
|
||||||
`check_same_thread=False` and `PRAGMA journal_mode=WAL`, and are wrapped
|
`check_same_thread=False`, `PRAGMA journal_mode=WAL` and (since v2.11.0)
|
||||||
|
`PRAGMA foreign_keys=ON`, so `ON DELETE CASCADE`/`SET NULL` run, and are wrapped
|
||||||
in `MonitoredConnection` for query timing when metrics are on. `get_db()`
|
in `MonitoredConnection` for query timing when metrics are on. `get_db()`
|
||||||
stores one connection in `flask.g` per request or app context; on teardown
|
stores one connection in `flask.g` per request or app context; on teardown
|
||||||
it goes back to the pool, rolled back first if a transaction was left
|
it goes back to the pool, rolled back first if a transaction was left
|
||||||
|
|||||||
@@ -24,12 +24,12 @@ deployment repository, not here.
|
|||||||
|
|
||||||
## Container image (`Containerfile`)
|
## Container image (`Containerfile`)
|
||||||
|
|
||||||
A multi-stage build on `python:3.11-slim`; it works with Podman and Docker.
|
A multi-stage build on `python:3.14-slim`; it works with Podman and Docker.
|
||||||
|
|
||||||
| Stage | Purpose |
|
| Stage | Purpose |
|
||||||
|-------|---------|
|
|-------|---------|
|
||||||
| `builder` | Copies `uv` from `ghcr.io/astral-sh/uv`, creates `/opt/venv` and installs `requirements.txt` into it |
|
| `builder` | Copies `uv` from `ghcr.io/astral-sh/uv`, creates `/opt/venv` and installs `requirements.txt` into it |
|
||||||
| `test` | Copies the venv, adds `pytest` and `pytest-mock`, copies the source and runs `python -m pytest -q -p no:cacheprovider tests` with test values for `SESSION_SECRET`, `SITE_URL` and `ADMIN_ME`. The stage only builds if the tests pass. |
|
| `test` | Copies the venv, installs `requirements-test.txt`, copies the source and runs `python -m pytest -q -p no:cacheprovider tests` with test values for `SESSION_SECRET`, `SITE_URL` and `ADMIN_ME`. The stage only builds if the tests pass. |
|
||||||
| runtime (last, the default target) | Creates user `starpunk` (UID/GID 1000) and `/data/notes`, copies the venv and the application to `/app`, sets `DATA_PATH=/data`, `NOTES_PATH=/data/notes`, `DATABASE_PATH=/data/starpunk.db`, runs as `starpunk`, exposes port 8000 |
|
| runtime (last, the default target) | Creates user `starpunk` (UID/GID 1000) and `/data/notes`, copies the venv and the application to `/app`, sets `DATA_PATH=/data`, `NOTES_PATH=/data/notes`, `DATABASE_PATH=/data/starpunk.db`, runs as `starpunk`, exposes port 8000 |
|
||||||
|
|
||||||
The runtime image's command is gunicorn:
|
The runtime image's command is gunicorn:
|
||||||
@@ -172,8 +172,8 @@ ignored with a warning.
|
|||||||
| `DATA_PATH` | `./data` (image: `/data`) | Root of all stored data |
|
| `DATA_PATH` | `./data` (image: `/data`) | Root of all stored data |
|
||||||
| `DATABASE_PATH` | `./data/starpunk.db` (image: `/data/starpunk.db`) | |
|
| `DATABASE_PATH` | `./data/starpunk.db` (image: `/data/starpunk.db`) | |
|
||||||
| `NOTES_PATH` | `./data/notes` (image: `/data/notes`) | Created at start-up, but note files are always written to `DATA_PATH/notes` |
|
| `NOTES_PATH` | `./data/notes` (image: `/data/notes`) | Created at start-up, but note files are always written to `DATA_PATH/notes` |
|
||||||
| `FLASK_ENV` | `development` | Set `production` in production |
|
| `FLASK_ENV` | `production` | Reported in `/health` and admin diagnostics; the image sets `production` |
|
||||||
| `FLASK_DEBUG` | `1` | Set `0` in production (the image does not set it) |
|
| `FLASK_DEBUG` | `0` | Debug off unless `1` (`flask run --debug` sets it locally); the image sets `0` |
|
||||||
| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
|
| `LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
|
||||||
| `VERSION` | package `__version__` | Overrides the reported version; normally unset |
|
| `VERSION` | package `__version__` | Overrides the reported version; normally unset |
|
||||||
|
|
||||||
|
|||||||
@@ -197,7 +197,8 @@ route isn't registered at all.
|
|||||||
|
|
||||||
1. Rate limits: 60 requests per minute per client IP and 10 per minute per
|
1. Rate limits: 60 requests per minute per client IP and 10 per minute per
|
||||||
source domain (`WEBMENTION_RATE_LIMIT_*`). Over the limit gets 429 with
|
source domain (`WEBMENTION_RATE_LIMIT_*`). Over the limit gets 429 with
|
||||||
`Retry-After`. The limiter is in memory, per process. The client IP
|
`Retry-After`. Counts are shared by all server processes (`rate_limits`
|
||||||
|
table, v2.11.0). The client IP
|
||||||
comes from `X-Forwarded-For` only when `TRUSTED_PROXY=true`.
|
comes from `X-Forwarded-For` only when `TRUSTED_PROXY=true`.
|
||||||
2. Form-encoded (JSON also accepted). `source` and `target` are required,
|
2. Form-encoded (JSON also accepted). `source` and `target` are required,
|
||||||
http(s), at most 2048 characters, and different from each other.
|
http(s), at most 2048 characters, and different from each other.
|
||||||
@@ -445,7 +446,7 @@ Drafts that stay drafts change nothing.
|
|||||||
|
|
||||||
Limits (429 when hit):
|
Limits (429 when hit):
|
||||||
|
|
||||||
- 30 requests per minute per client IP (per process; needs
|
- 30 requests per minute per client IP (shared by all processes; needs
|
||||||
`TRUSTED_PROXY=true` behind a proxy);
|
`TRUSTED_PROXY=true` behind a proxy);
|
||||||
- at most 200 pending verifications;
|
- at most 200 pending verifications;
|
||||||
- `WEBSUB_MAX_SUBSCRIPTIONS` (1000) in total;
|
- `WEBSUB_MAX_SUBSCRIPTIONS` (1000) in total;
|
||||||
@@ -488,7 +489,6 @@ retried.
|
|||||||
|
|
||||||
- The hub serves only this site's topics, by design. websub.rocks' hub
|
- The hub serves only this site's topics, by design. websub.rocks' hub
|
||||||
tests use their own topics, so they can't be run against it.
|
tests use their own topics, so they can't be run against it.
|
||||||
- Rate limiting is per process and in memory.
|
|
||||||
- Deliveries always carry the full topic content (no diffs). Fat pings
|
- Deliveries always carry the full topic content (no diffs). Fat pings
|
||||||
aren't sent to external hubs.
|
aren't sent to external hubs.
|
||||||
|
|
||||||
|
|||||||
@@ -293,13 +293,12 @@ their credentials are set. The full list with defaults is in
|
|||||||
callbacks, remote Micropub photos, IndieAuth client pages, crosspost link
|
callbacks, remote Micropub photos, IndieAuth client pages, crosspost link
|
||||||
cards) are checked first: every address the host resolves to must be public
|
cards) are checked first: every address the host resolves to must be public
|
||||||
(private, loopback, link-local, CGNAT, multicast and reserved ranges are
|
(private, loopback, link-local, CGNAT, multicast and reserved ranges are
|
||||||
refused). The fetcher in `webmention_ssrf.py`, used for webmentions, reply
|
refused). Every fetch of a user-supplied URL (webmentions, reply contexts,
|
||||||
contexts, WebSub and remote Micropub photos, follows redirects by hand and
|
WebSub, remote Micropub photos, IndieAuth client pages and, since v2.11.0,
|
||||||
re-checks each hop, connects to the address it checked (no second DNS
|
crosspost link cards) uses the fetcher in `webmention_ssrf.py`: it follows
|
||||||
lookup) and limits response size (1 MB by default) and total time.
|
redirects by hand and re-checks each hop, connects to the address it
|
||||||
Crosspost link cards use their own check that also re-checks each redirect.
|
checked (no second DNS lookup) and limits response size (1 MB by default)
|
||||||
The IndieAuth client-page fetch checks only the initial URL. `DEV_MODE`
|
and total time. `DEV_MODE` relaxes the checks for local testing.
|
||||||
relaxes the checks for local testing.
|
|
||||||
- **HTML sanitising**: rendered notes (Markdown or HTML sent by a Micropub
|
- **HTML sanitising**: rendered notes (Markdown or HTML sent by a Micropub
|
||||||
client) and webmention content are cleaned with nh3 allow-lists; search
|
client) and webmention content are cleaned with nh3 allow-lists; search
|
||||||
highlighting escapes everything except `<mark>`.
|
highlighting escapes everything except `<mark>`.
|
||||||
@@ -307,15 +306,17 @@ their credentials are set. The full list with defaults is in
|
|||||||
(defaults 60 and 10 per 60 seconds); `/websub` 30 requests per IP per
|
(defaults 60 and 10 per 60 seconds); `/websub` 30 requests per IP per
|
||||||
minute, plus limits on subscriptions in total, per callback host and per
|
minute, plus limits on subscriptions in total, per callback host and per
|
||||||
callback address, and on requests waiting for verification. Request-rate
|
callback address, and on requests waiting for verification. Request-rate
|
||||||
counters are in memory and per process.
|
counters live in the `rate_limits` table, shared by all server processes
|
||||||
|
(v2.11.0).
|
||||||
- **Client IP behind a proxy**: with `TRUSTED_PROXY=true` the first address in
|
- **Client IP behind a proxy**: with `TRUSTED_PROXY=true` the first address in
|
||||||
`X-Forwarded-For` is used as the client IP for rate limiting. Set it only
|
`X-Forwarded-For` is used as the client IP for rate limiting. Set it only
|
||||||
when the app is reachable solely through a proxy that sets that header
|
when the app is reachable solely through a proxy that sets that header
|
||||||
(Caddy does); without it, every client shares the proxy's address.
|
(Caddy does); without it, every client shares the proxy's address.
|
||||||
- **Auth**: hashed session and token storage, single-use login state, PKCE
|
- **Auth**: hashed session and token storage, single-use login state, PKCE
|
||||||
for both OIDC and IndieAuth, redirect URIs checked against the client,
|
for both OIDC and IndieAuth, redirect URIs checked against the client,
|
||||||
post-login redirects limited to this site. Admin form POSTs rely on the
|
post-login redirects limited to this site. Admin form POSTs, logout and
|
||||||
SameSite=Lax session cookie (there are no per-form CSRF tokens).
|
the IndieAuth consent carry a CSRF token: an HMAC of the session cookie,
|
||||||
|
checked by `starpunk/csrf.py` (v2.11.0), on top of the SameSite=Lax cookie.
|
||||||
- **Uploads**: request bodies are capped at 50 MB; images are processed by
|
- **Uploads**: request bodies are capped at 50 MB; images are processed by
|
||||||
Pillow and stored under random (UUID) names.
|
Pillow and stored under random (UUID) names.
|
||||||
- **Container**: runs as a non-root user. HTTPS and security headers (HSTS,
|
- **Container**: runs as a non-root user. HTTPS and security headers (HSTS,
|
||||||
|
|||||||
@@ -328,14 +328,16 @@ In place:
|
|||||||
untrusted redirect URIs never followed.
|
untrusted redirect URIs never followed.
|
||||||
- Post-login redirects are limited to this site.
|
- Post-login redirects are limited to this site.
|
||||||
|
|
||||||
|
CSRF (v2.11.0): admin form POSTs, `/auth/logout` and the consent form's
|
||||||
|
approve button need a `csrf_token` field (or `X-CSRF-Token` header) equal to
|
||||||
|
an HMAC-SHA256 of the `starpunk_session` cookie under `SECRET_KEY`; anything
|
||||||
|
else gets 400. The token changes with every login and needs no storage.
|
||||||
|
Endpoints apps call with bearer tokens are not affected. The client metadata
|
||||||
|
fetch uses `webmention_ssrf.safe_get` (every redirect hop checked, the
|
||||||
|
connection pinned, a 5-second limit).
|
||||||
|
|
||||||
Known gaps:
|
Known gaps:
|
||||||
|
|
||||||
- Admin forms and the consent form have no CSRF token. They rely on the
|
|
||||||
`SameSite=Lax` session cookie, which cross-site POSTs don't carry.
|
|
||||||
- The client metadata fetch checks the `client_id` URL against private
|
|
||||||
addresses, but then makes a plain `httpx.get` that follows redirects.
|
|
||||||
Redirect hops are not re-checked and the connection is not pinned, unlike
|
|
||||||
`webmention_ssrf.safe_get`.
|
|
||||||
- `scopes_supported` leaves out `draft`, although Micropub honours it and
|
- `scopes_supported` leaves out `draft`, although Micropub honours it and
|
||||||
the consent screen grants it when requested. Unknown scopes are also shown
|
the consent screen grants it when requested. Unknown scopes are also shown
|
||||||
and granted as requested; they just have no effect.
|
and granted as requested; they just have no effect.
|
||||||
|
|||||||
@@ -246,10 +246,6 @@ to itself (v2.8.0).
|
|||||||
|
|
||||||
- Bluesky and Threads copies don't follow edits (platform limitation).
|
- Bluesky and Threads copies don't follow edits (platform limitation).
|
||||||
Unticking and re-ticking a target makes a new copy.
|
Unticking and re-ticking a target makes a new copy.
|
||||||
- Link-card and Threads image fetches check every redirect hop against
|
|
||||||
private addresses but don't pin DNS, and use a per-request rather than an
|
|
||||||
overall time limit. Webmention and WebSub fetching use the stricter
|
|
||||||
`webmention_ssrf` fetcher.
|
|
||||||
- No per-target choice for Micropub updates: `mp-syndicate-to` is read only
|
- No per-target choice for Micropub updates: `mp-syndicate-to` is read only
|
||||||
on create.
|
on create.
|
||||||
- Likes of ordinary web pages, and all likes on Threads, stay on the site.
|
- Likes of ordinary web pages, and all likes on Threads, stay on the site.
|
||||||
@@ -14,9 +14,9 @@ the pieces fit together.
|
|||||||
|
|
||||||
| Component | Choice |
|
| Component | Choice |
|
||||||
|-----------|--------|
|
|-----------|--------|
|
||||||
| Language | Python 3.11 (`python:3.11-slim` base image) |
|
| Language | Python 3.14 (`python:3.14-slim` base image; since v2.11.0) |
|
||||||
| Web framework | Flask 3.0 (Werkzeug, Jinja2, itsdangerous, click come with it) |
|
| Web framework | Flask 3.1 (Werkzeug, Jinja2, itsdangerous, click come with it) |
|
||||||
| WSGI server | gunicorn 21.2, 4 sync workers (see [deployment.md](deployment.md)) |
|
| WSGI server | gunicorn 26.2, 4 sync workers (see [deployment.md](deployment.md)) |
|
||||||
| Database | SQLite (standard-library `sqlite3`), WAL mode, FTS5 for search |
|
| Database | SQLite (standard-library `sqlite3`), WAL mode, FTS5 for search |
|
||||||
| Content storage | Markdown files on disk |
|
| Content storage | Markdown files on disk |
|
||||||
| Background work | `threading` daemon threads; `fcntl.flock` file locks to pick one process |
|
| Background work | `threading` daemon threads; `fcntl.flock` file locks to pick one process |
|
||||||
@@ -28,32 +28,36 @@ the pieces fit together.
|
|||||||
|
|
||||||
## Python dependencies (`requirements.txt`)
|
## Python dependencies (`requirements.txt`)
|
||||||
|
|
||||||
|
Pinned to a minor series. Upgraded in v2.11.0, when the set had no known
|
||||||
|
vulnerabilities (`pip-audit`); check again before each release.
|
||||||
|
|
||||||
| Package | Pin | Used for |
|
| Package | Pin | Used for |
|
||||||
|---------|-----|----------|
|
|---------|-----|----------|
|
||||||
| Flask | `3.0.*` | Web framework |
|
| Flask | `3.1.*` | Web framework |
|
||||||
| gunicorn | `21.2.*` | Production WSGI server |
|
| gunicorn | `26.2.*` | Production WSGI server |
|
||||||
| markdown | `3.5.*` | Rendering notes (`extra`, `codehilite`, `nl2br` and a bare-URL linking extension) |
|
| markdown | `3.11.*` | Rendering notes (`extra`, `codehilite`, `nl2br` and a bare-URL linking extension) |
|
||||||
| feedgen | `1.0.*` | RSS 2.0 feed (`starpunk/feeds/rss.py`); Atom and JSON Feed are generated by StarPunk's own code |
|
| nh3 | `0.3.*` | HTML sanitising (rendered notes, webmention content) |
|
||||||
| httpx | `0.27.*` | All outbound HTTP: OIDC, IndieAuth client discovery, webmention and WebSub requests (through the SSRF-safe wrapper in `webmention_ssrf.py`), crossposting APIs, health check |
|
| beautifulsoup4 | `4.15.*` | HTML parsing: IndieAuth client pages, crosspost text and link cards |
|
||||||
| python-dotenv | `1.0.*` | Loading `.env` |
|
| lxml | `6.1.*` | Outgoing link extraction, webmention endpoint discovery and source checks |
|
||||||
| beautifulsoup4 | `4.12.*` | HTML parsing: IndieAuth client pages, crosspost text and link cards |
|
|
||||||
| mf2py | `2.0.*` | Parsing microformats2 (webmention sources, reply contexts, author h-card) |
|
| mf2py | `2.0.*` | Parsing microformats2 (webmention sources, reply contexts, author h-card) |
|
||||||
| pytest | `8.0.*` | Test runner (listed here as well as in the image's test stage) |
|
| python-dateutil | `4.0.*` | Dates in reply contexts |
|
||||||
| psutil | `5.9.*` | Memory monitor and system metrics |
|
| feedgen | `1.0.*` | RSS 2.0 feed (`starpunk/feeds/rss.py`); Atom and JSON Feed are generated by StarPunk's own code |
|
||||||
| Pillow | `10.1.*` | Image validation, resizing and variants |
|
| httpx | `0.28.*` | All outbound HTTP; requests to user-supplied URLs go through `webmention_ssrf.safe_get`/`safe_post` |
|
||||||
| pillow-heif | `0.18.*` | HEIC/HEIF uploads (iPhone photos) |
|
| PyJWT[crypto] | `2.15.*` | Verifying OIDC ID tokens (JWKS, RS/ES signatures via `cryptography`) |
|
||||||
| nh3 | `0.2.*` | HTML sanitising (rendered notes, webmention content) |
|
| python-dotenv | `1.2.*` | Loading `.env` |
|
||||||
| PyJWT[crypto] | `2.10.*` | Verifying OIDC ID tokens (JWKS, RS/ES signatures via `cryptography`) |
|
| Pillow | `13.0.*` | Image validation, resizing and variants |
|
||||||
|
| pillow-heif | `1.8.*` | HEIC/HEIF uploads (iPhone photos) |
|
||||||
|
| psutil | `7.2.*` | Memory monitor and system metrics |
|
||||||
|
|
||||||
Packages imported directly but installed only as dependencies of the above:
|
`markupsafe` comes with Flask/Jinja2.
|
||||||
`lxml` (outgoing link extraction, webmention endpoint discovery and source checks; from feedgen) and `python-dateutil` (dates in
|
|
||||||
reply contexts; from feedgen). `markupsafe` comes with Flask/Jinja2.
|
|
||||||
|
|
||||||
### Development dependencies (`requirements-dev.txt`)
|
### Test and development dependencies
|
||||||
|
|
||||||
Includes `requirements.txt`, plus `pytest-cov`, `pytest-mock`, `black`,
|
- `requirements-test.txt`: `pytest` 9.2 and `pytest-mock` 3.15, used by the
|
||||||
`flake8`, `mypy` and `gunicorn`. Coding standards are in
|
Containerfile test stage (CI).
|
||||||
[../standards/](../standards/).
|
- `requirements-dev.txt`: runtime and test requirements plus `pytest-cov`,
|
||||||
|
`black`, `flake8`, `mypy` (not enforced) and `pip-audit`. Coding standards
|
||||||
|
are in [../standards/](../standards/).
|
||||||
|
|
||||||
### Deliberately not used
|
### Deliberately not used
|
||||||
|
|
||||||
@@ -94,7 +98,7 @@ Protocol details: [indieweb-protocols.md](indieweb-protocols.md),
|
|||||||
```
|
```
|
||||||
|
|
||||||
- **In CI**: the `test` stage of the `Containerfile` installs the production
|
- **In CI**: the `test` stage of the `Containerfile` installs the production
|
||||||
requirements plus pytest and pytest-mock and runs the whole suite. The
|
requirements plus `requirements-test.txt` and runs the whole suite. The
|
||||||
image is only built and pushed if it passes. Checklist:
|
image is only built and pushed if it passes. Checklist:
|
||||||
[../standards/testing-checklist.md](../standards/testing-checklist.md).
|
[../standards/testing-checklist.md](../standards/testing-checklist.md).
|
||||||
|
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ is in `../architecture/`; decisions are in `../decisions/` (ADRs).
|
|||||||
|
|
||||||
## Contents
|
## Contents
|
||||||
|
|
||||||
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [hotfix](#hotfix)
|
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [hotfix](#hotfix)
|
||||||
|
|
||||||
## v1.0.0
|
## v1.0.0
|
||||||
|
|
||||||
@@ -410,6 +410,10 @@ OIDC admin login and built-in IndieAuth server.
|
|||||||
|
|
||||||
- [2026-11-03-reply-context-fetch.md](v2.10.2/2026-11-03-reply-context-fetch.md) - Reply contexts fetched again (fetch_source returns three values since v2.7.0)
|
- [2026-11-03-reply-context-fetch.md](v2.10.2/2026-11-03-reply-context-fetch.md) - Reply contexts fetched again (fetch_source returns three values since v2.7.0)
|
||||||
|
|
||||||
|
## v2.11.0
|
||||||
|
|
||||||
|
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) - Dependency upgrades (Python 3.14), foreign keys, CSRF, fetchers, debug default, shared rate limits
|
||||||
|
|
||||||
## hotfix
|
## hotfix
|
||||||
|
|
||||||
Not a release folder.
|
Not a release folder.
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# v2.11.0: Maintenance release
|
||||||
|
|
||||||
|
Dependency upgrades and the five High items in the backlog (all re-checked
|
||||||
|
against the code first and still present).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
`pip-audit` on the v2.10.2 set found 97 published vulnerabilities in eight
|
||||||
|
packages (Pillow 64, PyJWT 22, gunicorn 4, Flask 2, Markdown 2,
|
||||||
|
python-dotenv 2, pytest 2, pillow-heif 1). All direct dependencies were moved
|
||||||
|
to their current minor series (`requirements.txt`); the new set has no known
|
||||||
|
vulnerabilities. Python moved from 3.11 to 3.14 (`python:3.14-slim`); the
|
||||||
|
whole suite passes with `-W error::DeprecationWarning`.
|
||||||
|
|
||||||
|
- `requirements-test.txt` (pytest, pytest-mock) is installed by the
|
||||||
|
Containerfile test stage; pytest left the runtime requirements.
|
||||||
|
- `lxml` and `python-dateutil` are pinned (used directly, previously only
|
||||||
|
transitive). `uv` is pinned to 0.9.28 in the Containerfile.
|
||||||
|
- `datetime.utcnow()` (34 calls in `starpunk/`, more in tests) became
|
||||||
|
`datetime.now(timezone.utc).replace(tzinfo=None)`: same naive-UTC values.
|
||||||
|
sqlite3's default datetime adapters, deprecated in 3.12, are registered
|
||||||
|
explicitly in `starpunk/database/__init__.py` with the same output.
|
||||||
|
|
||||||
|
## High backlog items
|
||||||
|
|
||||||
|
1. **Foreign keys.** `PRAGMA foreign_keys=ON` on every pooled connection
|
||||||
|
(`database/pool.py`); all deletes of notes, media and WebSub subscriptions
|
||||||
|
go through the pool. Every foreign key is `CASCADE` or `SET NULL`, so no
|
||||||
|
delete can be blocked. Migration 018 removes orphans (production: three
|
||||||
|
`note_tags` rows from hard-deleted notes). Crosspost rows still have no
|
||||||
|
foreign key on purpose.
|
||||||
|
2. **CSRF.** `starpunk/csrf.py`: token = HMAC-SHA256(SECRET_KEY,
|
||||||
|
"csrf:" + starpunk_session cookie); `before_request` checks the
|
||||||
|
`csrf_token` field or `X-CSRF-Token` header on every admin POST,
|
||||||
|
`auth.logout` and `indieauth.approve`; 400 otherwise. All 12 POST forms
|
||||||
|
carry it. `CSRF_ENABLED` (default true); test apps turn it off unless they
|
||||||
|
set it.
|
||||||
|
3. **Fetchers.** `indieauth.fetch_client_redirect_uris` and
|
||||||
|
`syndication._fetch_public` use `webmention_ssrf.safe_get` (per-hop
|
||||||
|
checks, DNS pinning, hard deadline). `safe_get` gained `truncate` for
|
||||||
|
reads that stop at the size limit.
|
||||||
|
4. **Debug default.** `FLASK_DEBUG` defaults to 0 and `FLASK_ENV` to
|
||||||
|
production; the runtime image sets both.
|
||||||
|
5. **Shared rate limits.** `SharedRateLimiter` keeps sliding-window counters
|
||||||
|
in `rate_limits` (migration 019): current window plus the previous one
|
||||||
|
weighted by overlap. Used by `/webmention` (per IP and per source domain)
|
||||||
|
and `/websub`; old windows are pruned every 200 calls.
|
||||||
|
|
||||||
|
## Housekeeping
|
||||||
|
|
||||||
|
`compose.yaml` no longer pins 0.6.0 or sets `VERSION`/`ENVIRONMENT`; the
|
||||||
|
unused root `dev_auth.py` and the committed `test.ini` were removed.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
New: `tests/test_csrf.py`, `tests/test_foreign_keys.py`,
|
||||||
|
`tests/test_shared_rate_limit.py`, IndieAuth client-metadata tests through the
|
||||||
|
fetcher (including a redirect to a private address). Crosspost tests route
|
||||||
|
link-card fetches through the fetcher's transport. Full suite: 1264 passed
|
||||||
|
on Python 3.14.
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
# StarPunk Backlog
|
# StarPunk Backlog
|
||||||
|
|
||||||
**Last Updated**: 2026-11-03 (current release: v2.10.1)
|
**Last Updated**: 2026-11-03 (current release: v2.11.0)
|
||||||
|
|
||||||
These are the open features and bugs, sorted by priority. Each item was checked against the code
|
These are the open features and bugs, sorted by priority. Each item was checked against the code
|
||||||
at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` and in
|
at v2.11.0. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` and in
|
||||||
`CHANGELOG.md`.
|
`CHANGELOG.md`.
|
||||||
|
|
||||||
## Priority Levels
|
## Priority Levels
|
||||||
@@ -23,30 +23,7 @@ at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` a
|
|||||||
|
|
||||||
## High
|
## High
|
||||||
|
|
||||||
### SQLite foreign keys are not enforced (ON DELETE CASCADE is inert)
|
None open.
|
||||||
- **Description**: No connection runs `PRAGMA foreign_keys = ON`, so the `ON DELETE CASCADE` clauses in the schema do nothing. When a note is deleted from the admin (a hard delete), its `note_tags`, `note_media`, `syndications` and webmention rows stay behind. `delete_media()` relies on the cascade to remove `media_variants` and `note_media` rows, so those are left orphaned too. Before turning the pragma on, check for existing orphan rows and add a migration that cleans them up. Each cascade also needs tests.
|
|
||||||
- **Where**: `starpunk/database/` (connection setup/pool), `starpunk/notes.py` (`delete_note`, hard delete), `starpunk/media.py` (`delete_media`), `migrations/007`–`010`, `016`
|
|
||||||
- **Source**: v2.10.x review; confirmed in code (there is no `foreign_keys` pragma anywhere in `starpunk/`)
|
|
||||||
|
|
||||||
### No CSRF tokens on admin and consent forms
|
|
||||||
- **Description**: Admin forms (create/edit/delete notes, webmention moderation, token revocation) and the IndieAuth consent form rely only on the `SameSite=Lax` session cookie. Lax blocks cross-site POSTs in current browsers, but a per-session CSRF token is the standard defence and covers older clients and same-site subdomains.
|
|
||||||
- **Where**: `starpunk/routes/admin.py`, `starpunk/routes/indieauth.py`, `templates/admin/*`
|
|
||||||
- **Source**: Documentation refresh 2026-11-03 (architecture review)
|
|
||||||
|
|
||||||
### Two outbound fetches don't use the hardened fetcher
|
|
||||||
- **Description**: IndieAuth client-metadata discovery (`indieauth.fetch_client_redirect_uris`) checks only the first URL, then uses `httpx.get(follow_redirects=True)`: redirect hops aren't checked and DNS isn't pinned. Crosspost link cards and Threads image fetches (`syndication._fetch_public`) check each hop but don't pin DNS, and their 10 s timeout is per read. Move both to `webmention_ssrf.safe_get`, as Webmention, WebSub and Micropub photo downloads already do.
|
|
||||||
- **Where**: `starpunk/indieauth.py`, `starpunk/syndication.py`
|
|
||||||
- **Source**: Documentation refresh 2026-11-03
|
|
||||||
|
|
||||||
### `FLASK_DEBUG` defaults to on
|
|
||||||
- **Description**: `config.py` reads `FLASK_DEBUG` with default `"1"` (and `FLASK_ENV` defaults to development); the runtime image sets neither. Production is safe only because the deployment sets `FLASK_DEBUG=0`. Default to off, or set it in the `Containerfile` runtime stage.
|
|
||||||
- **Where**: `starpunk/config.py`, `Containerfile`
|
|
||||||
- **Source**: Documentation refresh 2026-11-03 (production verified: DEBUG is False there)
|
|
||||||
|
|
||||||
### Rate limiters are per process
|
|
||||||
- **Description**: `RateLimiter` keeps its counters in memory, and gunicorn runs 4 workers. The WebSub hub limit of 30/min is therefore really 4×30/min, and the webmention endpoint limit is multiplied the same way. Consider a counter stored in SQLite that all workers share.
|
|
||||||
- **Where**: `starpunk/rate_limiter.py`, `starpunk/routes/websub.py`, `starpunk/routes/webmention.py`, `Containerfile` (`--workers 4`)
|
|
||||||
- **Source**: ADR-071 (Consequences: rate limiting bounds subscriber-chosen requests); v2.10.1 review
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -92,20 +69,15 @@ at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` a
|
|||||||
- **Where**: `starpunk/webmention_processor.py`, `starpunk/webmention_worker.py`
|
- **Where**: `starpunk/webmention_processor.py`, `starpunk/webmention_worker.py`
|
||||||
- **Source**: v1.6.0 design reconstruction (docs/design/v1.6.0/background-worker.md)
|
- **Source**: v1.6.0 design reconstruction (docs/design/v1.6.0/background-worker.md)
|
||||||
|
|
||||||
### Webmention rate limiter never cleaned up
|
|
||||||
- **Description**: `cleanup_rate_limiters()` is never called, so per-IP and per-domain keys accumulate for the life of each process.
|
|
||||||
- **Where**: `starpunk/rate_limiter.py`, `starpunk/routes/webmention.py`
|
|
||||||
- **Source**: v1.6.0 design reconstruction
|
|
||||||
|
|
||||||
### Atom and JSON Feed not linked from pages
|
### Atom and JSON Feed not linked from pages
|
||||||
- **Description**: `base.html` advertises only `/feed` (as RSS) and OPML with `rel=alternate`; add `<link rel="alternate">` for `/feed.atom` and `/feed.json` so readers can pick a format.
|
- **Description**: `base.html` advertises only `/feed` (as RSS) and OPML with `rel=alternate`; add `<link rel="alternate">` for `/feed.atom` and `/feed.json` so readers can pick a format.
|
||||||
- **Where**: `templates/base.html`
|
- **Where**: `templates/base.html`
|
||||||
- **Source**: Documentation refresh 2026-11-03
|
- **Source**: Documentation refresh 2026-11-03
|
||||||
|
|
||||||
### Housekeeping in the repository
|
### LICENSE placeholder
|
||||||
- **Description**: `compose.yaml` is stale (`image: starpunk:0.6.0`, `VERSION=0.6.0` would override the real version, unused `ENVIRONMENT`); the root `dev_auth.py` duplicates `starpunk/dev_auth.py` and is unused; `LICENSE` says "Copyright (c) 2024 [Your Name]"; `test.ini` contains a private host name; `lxml` and `python-dateutil` are used directly but only installed as dependencies of other packages, while `pytest` is pinned in the production `requirements.txt`.
|
- **Description**: `LICENSE` still reads "Copyright (c) 2024 [Your Name]". Fill in the holder and year.
|
||||||
- **Where**: `compose.yaml`, `dev_auth.py`, `LICENSE`, `test.ini`, `requirements*.txt`
|
- **Where**: `LICENSE`
|
||||||
- **Source**: Documentation refresh 2026-11-03
|
- **Source**: Documentation refresh 2026-11-03 (rest of that housekeeping item done in v2.11.0)
|
||||||
|
|
||||||
### Backfeed responses to crossposts
|
### Backfeed responses to crossposts
|
||||||
- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy.
|
- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy.
|
||||||
@@ -213,6 +185,8 @@ at v2.10.1. Work that has shipped is listed per release in `vX.Y.Z/RELEASE.md` a
|
|||||||
|
|
||||||
These items were removed from the backlog because they have shipped:
|
These items were removed from the backlog because they have shipped:
|
||||||
|
|
||||||
|
- v2.11.0: SQLite foreign keys enforced (orphans cleaned up by migration 018); CSRF tokens on admin, logout and consent forms; IndieAuth client and crosspost link-card fetches moved onto the SSRF-safe fetcher; `FLASK_DEBUG`/`FLASK_ENV` default to off/production; rate limits shared by all processes (and old windows pruned, so the webmention limiter no longer grows); dependencies upgraded (Python 3.14, no known vulnerabilities); stale `compose.yaml`, duplicate `dev_auth.py` and `test.ini` removed or fixed; `lxml`/`python-dateutil` pinned, `pytest` out of the runtime requirements
|
||||||
|
|
||||||
- POSSE: GoToSocial (v2.1.0), Bluesky (v2.2.0), Threads (v2.6.0)
|
- POSSE: GoToSocial (v2.1.0), Bluesky (v2.2.0), Threads (v2.6.0)
|
||||||
- Webmentions: receive, send, display and moderation (v1.6.0); fixed and made spec-compliant in v2.6.8–v2.7.2
|
- Webmentions: receive, send, display and moderation (v1.6.0); fixed and made spec-compliant in v2.6.8–v2.7.2
|
||||||
- Reply contexts, replies, likes, reposts and bookmarks (v2.4.0, v2.5.0, v2.9.0)
|
- Reply contexts, replies, likes, reposts and bookmarks (v2.4.0, v2.5.0, v2.9.0)
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
|
|||||||
|
|
||||||
## Backlog
|
## Backlog
|
||||||
|
|
||||||
- **[BACKLOG.md](BACKLOG.md)** - Open features and bugs by priority (updated for v2.10.1)
|
- **[BACKLOG.md](BACKLOG.md)** - Open features and bugs by priority (updated for v2.11.0)
|
||||||
|
|
||||||
## Releases
|
## Releases
|
||||||
|
|
||||||
@@ -47,3 +47,4 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
|
|||||||
- **[v2.10.0](v2.10.0/RELEASE.md)** - WebSub (Released 2026-11-03)
|
- **[v2.10.0](v2.10.0/RELEASE.md)** - WebSub (Released 2026-11-03)
|
||||||
- **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03)
|
- **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03)
|
||||||
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
|
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
|
||||||
|
- **[v2.11.0](v2.11.0/RELEASE.md)** - Maintenance: dependency upgrades, High backlog items (Released 2026-11-03)
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# v2.11.0
|
||||||
|
|
||||||
|
- **Date**: 2026-11-03
|
||||||
|
- **Status**: Released
|
||||||
|
- **Focus**: Maintenance: dependency upgrades and the High backlog items
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- All dependencies upgraded (97 known vulnerabilities before, none after); Python 3.14
|
||||||
|
- CSRF tokens on admin forms, logout and the IndieAuth consent
|
||||||
|
- IndieAuth client and crosspost link-card fetches use the SSRF-safe fetcher
|
||||||
|
- Debug off and production environment by default
|
||||||
|
|
||||||
|
## Changes
|
||||||
|
|
||||||
|
- Rate limits shared by all server processes
|
||||||
|
- `requirements-test.txt`; pytest out of the production image; `lxml` and `python-dateutil` pinned; `uv` pinned
|
||||||
|
|
||||||
|
## Bugs fixed
|
||||||
|
|
||||||
|
- Foreign keys enforced, so deletes cascade; orphan rows removed (migration 018)
|
||||||
|
- Deprecated `datetime.utcnow()` and sqlite3 adapters replaced
|
||||||
|
- Stale `compose.yaml`, duplicate `dev_auth.py` and `test.ini` cleaned up
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [CHANGELOG](../../../CHANGELOG.md)
|
||||||
|
- [Design note](../../design/v2.11.0/2026-11-03-maintenance-release.md)
|
||||||
@@ -9,7 +9,7 @@ deployment is in [architecture/deployment.md](../architecture/deployment.md).
|
|||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Python 3.11 (the container image uses `python:3.11-slim`)
|
- Python 3.14 (the container image uses `python:3.14-slim`)
|
||||||
- [uv](https://docs.astral.sh/uv/) for the virtual environment and packages ([ADR-006](../decisions/ADR-006-python-virtual-environment-uv.md))
|
- [uv](https://docs.astral.sh/uv/) for the virtual environment and packages ([ADR-006](../decisions/ADR-006-python-virtual-environment-uv.md))
|
||||||
- git
|
- git
|
||||||
- Optional: Docker or Podman, to build the image and run the CI test stage locally
|
- Optional: Docker or Podman, to build the image and run the CI test stage locally
|
||||||
@@ -23,7 +23,7 @@ pytest-mock, black, flake8, mypy).
|
|||||||
```bash
|
```bash
|
||||||
git clone https://git.thesatelliteoflove.com/phil/StarPunk.git
|
git clone https://git.thesatelliteoflove.com/phil/StarPunk.git
|
||||||
cd StarPunk
|
cd StarPunk
|
||||||
uv venv --python 3.11 # creates .venv/ (gitignored)
|
uv venv --python 3.14 # creates .venv/ (gitignored)
|
||||||
uv pip install -r requirements-dev.txt
|
uv pip install -r requirements-dev.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ cd ~/Documents/projects/StarPunk # main checkout, on main
|
|||||||
git pull
|
git pull
|
||||||
git worktree add ../StarPunk-worktrees/v2.10.1 -b fix/v2.10.1-websub-hardening main
|
git worktree add ../StarPunk-worktrees/v2.10.1 -b fix/v2.10.1-websub-hardening main
|
||||||
cd ../StarPunk-worktrees/v2.10.1
|
cd ../StarPunk-worktrees/v2.10.1
|
||||||
uv venv --python 3.11 && uv pip install -r requirements-dev.txt
|
uv venv --python 3.14 && uv pip install -r requirements-dev.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Make the change
|
### 2. Make the change
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ out." Patterns specific to small helper functions are in
|
|||||||
|
|
||||||
## Base style
|
## Base style
|
||||||
|
|
||||||
- [PEP 8](https://peps.python.org/pep-0008/) naming and layout, Python 3.11.
|
- [PEP 8](https://peps.python.org/pep-0008/) naming and layout, Python 3.14. Use `datetime.now(timezone.utc)`, not the deprecated `datetime.utcnow()`.
|
||||||
- **Match the file you are editing.** 4-space indents everywhere; quote
|
- **Match the file you are editing.** 4-space indents everywhere; quote
|
||||||
style varies between modules (newer code uses double quotes); keep lines
|
style varies between modules (newer code uses double quotes); keep lines
|
||||||
to roughly 100 characters.
|
to roughly 100 characters.
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
-- Migration 018: Remove rows orphaned while foreign keys weren't enforced (v2.11.0)
|
||||||
|
-- From v2.11.0 every pooled connection runs PRAGMA foreign_keys=ON, so these
|
||||||
|
-- cascades happen on delete. This removes what earlier deletes left behind.
|
||||||
|
|
||||||
|
DELETE FROM note_tags WHERE note_id NOT IN (SELECT id FROM notes);
|
||||||
|
DELETE FROM note_tags WHERE tag_id NOT IN (SELECT id FROM tags);
|
||||||
|
DELETE FROM note_media WHERE note_id NOT IN (SELECT id FROM notes);
|
||||||
|
DELETE FROM note_media WHERE media_id NOT IN (SELECT id FROM media);
|
||||||
|
DELETE FROM media_variants WHERE media_id NOT IN (SELECT id FROM media);
|
||||||
|
DELETE FROM webmention_queue WHERE source_note_id IS NOT NULL AND source_note_id NOT IN (SELECT id FROM notes);
|
||||||
|
UPDATE webmentions SET target_note_id = NULL WHERE target_note_id IS NOT NULL AND target_note_id NOT IN (SELECT id FROM notes);
|
||||||
|
DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND subscription_id NOT IN (SELECT id FROM websub_subscriptions);
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
-- Migration 019: Rate limit counters shared by all server processes (v2.11.0)
|
||||||
|
-- One row per key and time window; see starpunk/rate_limiter.py SharedRateLimiter.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS rate_limits (
|
||||||
|
key TEXT NOT NULL,
|
||||||
|
window INTEGER NOT NULL,
|
||||||
|
count INTEGER NOT NULL DEFAULT 0,
|
||||||
|
PRIMARY KEY (key, window)
|
||||||
|
);
|
||||||
+7
-15
@@ -1,17 +1,9 @@
|
|||||||
# StarPunk Development Dependencies
|
# StarPunk development dependencies
|
||||||
# Includes code quality and testing tools
|
|
||||||
|
|
||||||
# Include production dependencies
|
|
||||||
-r requirements.txt
|
-r requirements.txt
|
||||||
|
-r requirements-test.txt
|
||||||
|
|
||||||
# Testing
|
pytest-cov==7.* # Coverage reports
|
||||||
pytest-cov>=5.0.0 # Test coverage reporting
|
black==26.* # Formatting (not enforced, see docs/standards)
|
||||||
pytest-mock>=3.12.0 # Mocking for tests
|
flake8==7.* # Linting (not enforced)
|
||||||
|
mypy==1.* # Type checking (not enforced)
|
||||||
# Code Quality
|
pip-audit==2.* # Known-vulnerability check of the dependency set
|
||||||
black>=24.0.0 # Code formatting
|
|
||||||
flake8>=7.0.0 # Linting
|
|
||||||
mypy>=1.8.0 # Type checking
|
|
||||||
|
|
||||||
# WSGI Server (for production-like testing)
|
|
||||||
gunicorn>=21.2.0
|
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# Test dependencies: used by the Containerfile test stage and CI (v2.11.0)
|
||||||
|
pytest==9.2.*
|
||||||
|
pytest-mock==3.15.*
|
||||||
+21
-36
@@ -1,42 +1,27 @@
|
|||||||
# StarPunk Production Dependencies
|
# StarPunk runtime dependencies (v2.11.0)
|
||||||
# Python 3.11+ required
|
# Pinned to a minor series; see docs/architecture/technology-stack.md.
|
||||||
|
|
||||||
# Web Framework
|
# Web framework and server
|
||||||
Flask==3.0.*
|
Flask==3.1.*
|
||||||
|
gunicorn==26.2.*
|
||||||
|
|
||||||
# WSGI Server (Production)
|
# Content
|
||||||
gunicorn==21.2.*
|
markdown==3.11.*
|
||||||
|
nh3==0.3.*
|
||||||
# Content Processing
|
beautifulsoup4==4.15.*
|
||||||
markdown==3.5.*
|
lxml==6.1.*
|
||||||
|
mf2py==2.0.*
|
||||||
# Feed Generation
|
python-dateutil==4.0.*
|
||||||
feedgen==1.0.*
|
feedgen==1.0.*
|
||||||
|
|
||||||
# HTTP Client (for IndieAuth)
|
# HTTP and auth
|
||||||
httpx==0.27.*
|
httpx==0.28.*
|
||||||
|
PyJWT[crypto]==2.15.*
|
||||||
|
python-dotenv==1.2.*
|
||||||
|
|
||||||
# Configuration Management
|
# Images
|
||||||
python-dotenv==1.0.*
|
Pillow==13.0.*
|
||||||
|
pillow-heif==1.8.*
|
||||||
|
|
||||||
# HTML Parsing (for IndieAuth endpoint discovery)
|
# Monitoring
|
||||||
beautifulsoup4==4.12.*
|
psutil==7.2.*
|
||||||
|
|
||||||
# Microformats2 Parsing (v1.2.0)
|
|
||||||
mf2py==2.0.*
|
|
||||||
|
|
||||||
# Testing Framework
|
|
||||||
pytest==8.0.*
|
|
||||||
|
|
||||||
# System Monitoring (v1.1.2)
|
|
||||||
psutil==5.9.*
|
|
||||||
|
|
||||||
# Image Processing (v1.2.0, updated v1.4.2 for HEIC support)
|
|
||||||
Pillow==10.1.*
|
|
||||||
|
|
||||||
# HEIC/HEIF Support (v1.4.2 - iPhone photos)
|
|
||||||
pillow-heif==0.18.*
|
|
||||||
|
|
||||||
# HTML Sanitization (v1.6.0 - Webmention content)
|
|
||||||
nh3==0.2.*
|
|
||||||
PyJWT[crypto]==2.10.*
|
|
||||||
@@ -309,6 +309,10 @@ def create_app(config=None):
|
|||||||
response.headers.add('Link', '</webmention>; rel="webmention"')
|
response.headers.add('Link', '</webmention>; rel="webmention"')
|
||||||
return response
|
return response
|
||||||
|
|
||||||
|
# CSRF tokens on admin and consent forms (v2.11.0)
|
||||||
|
from starpunk import csrf
|
||||||
|
csrf.init_app(app)
|
||||||
|
|
||||||
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
|
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
|
||||||
from starpunk.websub import add_discovery_headers, template_links
|
from starpunk.websub import add_discovery_headers, template_links
|
||||||
app.after_request(add_discovery_headers)
|
app.after_request(add_discovery_headers)
|
||||||
@@ -470,5 +474,5 @@ def create_app(config=None):
|
|||||||
|
|
||||||
# Package version (Semantic Versioning 2.0.0)
|
# Package version (Semantic Versioning 2.0.0)
|
||||||
# See docs/standards/versioning-strategy.md for details
|
# See docs/standards/versioning-strategy.md for details
|
||||||
__version__ = "2.10.2"
|
__version__ = "2.11.0"
|
||||||
__version_info__ = (2, 10, 2)
|
__version_info__ = (2, 11, 0)
|
||||||
+3
-3
@@ -32,7 +32,7 @@ import base64
|
|||||||
import hashlib
|
import hashlib
|
||||||
import secrets
|
import secrets
|
||||||
import time
|
import time
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from functools import wraps
|
from functools import wraps
|
||||||
from typing import Any, Dict, Optional, Tuple
|
from typing import Any, Dict, Optional, Tuple
|
||||||
from urllib.parse import urlencode
|
from urllib.parse import urlencode
|
||||||
@@ -223,7 +223,7 @@ def initiate_login(next_url: Optional[str] = None) -> str:
|
|||||||
nonce,
|
nonce,
|
||||||
code_verifier,
|
code_verifier,
|
||||||
_safe_next_url(next_url),
|
_safe_next_url(next_url),
|
||||||
datetime.utcnow() + timedelta(minutes=LOGIN_STATE_MINUTES),
|
datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=LOGIN_STATE_MINUTES),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
@@ -361,7 +361,7 @@ def create_session(me: str) -> str:
|
|||||||
|
|
||||||
# Calculate expiry (use configured session lifetime or default to 30 days)
|
# Calculate expiry (use configured session lifetime or default to 30 days)
|
||||||
session_lifetime = current_app.config.get("SESSION_LIFETIME", 30)
|
session_lifetime = current_app.config.get("SESSION_LIFETIME", 30)
|
||||||
expires_at = datetime.utcnow() + timedelta(days=session_lifetime)
|
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=session_lifetime)
|
||||||
|
|
||||||
current_app.logger.debug(f"Auth: Session expiry: {expires_at} ({session_lifetime} days)")
|
current_app.logger.debug(f"Auth: Session expiry: {expires_at} ({session_lifetime} days)")
|
||||||
|
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ Fallback Behavior (per Q14):
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from typing import Dict, Optional
|
from typing import Dict, Optional
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
@@ -236,7 +236,7 @@ def get_author_profile(me_url: str, refresh: bool = False) -> Dict:
|
|||||||
if cached:
|
if cached:
|
||||||
# Check if cache is still valid
|
# Check if cache is still valid
|
||||||
cached_until = datetime.fromisoformat(cached['cached_until'])
|
cached_until = datetime.fromisoformat(cached['cached_until'])
|
||||||
if datetime.utcnow() < cached_until:
|
if datetime.now(timezone.utc).replace(tzinfo=None) < cached_until:
|
||||||
current_app.logger.debug(f"Using cached author profile for {me_url}")
|
current_app.logger.debug(f"Using cached author profile for {me_url}")
|
||||||
|
|
||||||
# Parse rel_me_links from JSON
|
# Parse rel_me_links from JSON
|
||||||
@@ -326,7 +326,7 @@ def save_author_profile(me_url: str, profile: Dict) -> None:
|
|||||||
db = get_db(current_app)
|
db = get_db(current_app)
|
||||||
|
|
||||||
# Calculate cache expiry (24 hours from now)
|
# Calculate cache expiry (24 hours from now)
|
||||||
cached_until = datetime.utcnow() + timedelta(hours=CACHE_TTL_HOURS)
|
cached_until = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(hours=CACHE_TTL_HOURS)
|
||||||
|
|
||||||
# Convert rel_me_links to JSON (per Q17)
|
# Convert rel_me_links to JSON (per Q17)
|
||||||
rel_me_json = json.dumps(profile.get('rel_me_links', []))
|
rel_me_json = json.dumps(profile.get('rel_me_links', []))
|
||||||
|
|||||||
+10
-2
@@ -71,8 +71,10 @@ def load_config(app, config_override=None):
|
|||||||
app.config["DATABASE_PATH"] = Path(os.getenv("DATABASE_PATH", "./data/starpunk.db"))
|
app.config["DATABASE_PATH"] = Path(os.getenv("DATABASE_PATH", "./data/starpunk.db"))
|
||||||
|
|
||||||
# Flask environment
|
# Flask environment
|
||||||
app.config["ENV"] = os.getenv("FLASK_ENV", "development")
|
# Safe defaults (v2.11.0): production, debug off. `flask run --debug`
|
||||||
app.config["DEBUG"] = os.getenv("FLASK_DEBUG", "1") == "1"
|
# sets FLASK_DEBUG=1 for local development.
|
||||||
|
app.config["ENV"] = os.getenv("FLASK_ENV", "production")
|
||||||
|
app.config["DEBUG"] = os.getenv("FLASK_DEBUG", "0") == "1"
|
||||||
|
|
||||||
# Logging
|
# Logging
|
||||||
app.config["LOG_LEVEL"] = os.getenv("LOG_LEVEL", "INFO")
|
app.config["LOG_LEVEL"] = os.getenv("LOG_LEVEL", "INFO")
|
||||||
@@ -153,8 +155,14 @@ def load_config(app, config_override=None):
|
|||||||
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
|
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
|
||||||
|
|
||||||
# Apply overrides if provided
|
# Apply overrides if provided
|
||||||
|
# CSRF protection for admin and consent forms (v2.11.0); on by default.
|
||||||
|
# Tests turn it off unless they set CSRF_ENABLED themselves.
|
||||||
|
app.config["CSRF_ENABLED"] = os.getenv("CSRF_ENABLED", "true").lower() == "true"
|
||||||
|
|
||||||
if config_override:
|
if config_override:
|
||||||
app.config.update(config_override)
|
app.config.update(config_override)
|
||||||
|
if app.config.get("TESTING") and "CSRF_ENABLED" not in config_override:
|
||||||
|
app.config["CSRF_ENABLED"] = False
|
||||||
|
|
||||||
# Normalize SITE_URL trailing slash (in case override provided URL without slash)
|
# Normalize SITE_URL trailing slash (in case override provided URL without slash)
|
||||||
if "SITE_URL" in app.config:
|
if "SITE_URL" in app.config:
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""
|
||||||
|
CSRF protection for the admin and consent forms (v2.11.0)
|
||||||
|
|
||||||
|
The token is an HMAC of the login session cookie under the app secret: no
|
||||||
|
storage, tied to the session, and different after every login. Forms carry
|
||||||
|
it as a hidden `csrf_token` field (template global `csrf_token()`); POSTs to
|
||||||
|
protected endpoints without the right token get 400.
|
||||||
|
|
||||||
|
Protected: every admin POST, logout, and the IndieAuth consent ("approve").
|
||||||
|
Endpoints used by apps with bearer tokens (Micropub, token, revoke,
|
||||||
|
Webmention, WebSub) are not: they have no cookie-based authority.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
|
||||||
|
from flask import abort, current_app, request
|
||||||
|
|
||||||
|
SESSION_COOKIE = "starpunk_session"
|
||||||
|
FIELD = "csrf_token"
|
||||||
|
HEADER = "X-CSRF-Token"
|
||||||
|
PROTECTED_ENDPOINTS = {"auth.logout", "indieauth.approve"}
|
||||||
|
|
||||||
|
|
||||||
|
def csrf_token() -> str:
|
||||||
|
"""The CSRF token for the current login session ('' when not logged in)"""
|
||||||
|
session = request.cookies.get(SESSION_COOKIE)
|
||||||
|
if not session:
|
||||||
|
return ""
|
||||||
|
key = str(current_app.config["SECRET_KEY"]).encode("utf-8")
|
||||||
|
return hmac.new(key, b"csrf:" + session.encode("utf-8"), hashlib.sha256).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _protected() -> bool:
|
||||||
|
return request.blueprint == "admin" or request.endpoint in PROTECTED_ENDPOINTS
|
||||||
|
|
||||||
|
|
||||||
|
def check_csrf():
|
||||||
|
"""before_request: refuse state-changing requests without a valid token"""
|
||||||
|
if request.method in ("GET", "HEAD", "OPTIONS") or not current_app.config.get("CSRF_ENABLED", True):
|
||||||
|
return None
|
||||||
|
if not _protected():
|
||||||
|
return None
|
||||||
|
expected = csrf_token()
|
||||||
|
if not expected:
|
||||||
|
return None # not logged in: the route's own auth check refuses it
|
||||||
|
sent = request.form.get(FIELD) or request.headers.get(HEADER) or ""
|
||||||
|
if not hmac.compare_digest(sent, expected):
|
||||||
|
current_app.logger.warning(f"CSRF token missing or wrong for {request.endpoint}")
|
||||||
|
abort(400, description="The form expired or came from another site. Reload the page and try again.")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def init_app(app) -> None:
|
||||||
|
app.before_request(check_csrf)
|
||||||
|
app.add_template_global(csrf_token, "csrf_token")
|
||||||
@@ -13,4 +13,12 @@ from starpunk.database.init import init_db
|
|||||||
from starpunk.database.pool import init_pool, get_db, get_pool_stats
|
from starpunk.database.pool import init_pool, get_db, get_pool_stats
|
||||||
from starpunk.database.schema import INITIAL_SCHEMA_SQL
|
from starpunk.database.schema import INITIAL_SCHEMA_SQL
|
||||||
|
|
||||||
|
# Explicit datetime adapters (v2.11.0). Python 3.12 deprecated sqlite3's
|
||||||
|
# built-in ones; these store exactly what they did ("YYYY-MM-DD HH:MM:SS[.ffffff]").
|
||||||
|
import datetime as _datetime
|
||||||
|
import sqlite3 as _sqlite3
|
||||||
|
|
||||||
|
_sqlite3.register_adapter(_datetime.datetime, lambda value: value.isoformat(" "))
|
||||||
|
_sqlite3.register_adapter(_datetime.date, lambda value: value.isoformat())
|
||||||
|
|
||||||
__all__ = ['init_db', 'init_pool', 'get_db', 'get_pool_stats', 'INITIAL_SCHEMA_SQL']
|
__all__ = ['init_db', 'init_pool', 'get_db', 'get_pool_stats', 'INITIAL_SCHEMA_SQL']
|
||||||
@@ -68,6 +68,9 @@ class ConnectionPool:
|
|||||||
|
|
||||||
# Enable WAL mode for better concurrency
|
# Enable WAL mode for better concurrency
|
||||||
conn.execute("PRAGMA journal_mode=WAL")
|
conn.execute("PRAGMA journal_mode=WAL")
|
||||||
|
# Enforce foreign keys, so ON DELETE CASCADE / SET NULL actually run
|
||||||
|
# (SQLite leaves them off per connection; v2.11.0)
|
||||||
|
conn.execute("PRAGMA foreign_keys=ON")
|
||||||
|
|
||||||
self._stats['connections_created'] += 1
|
self._stats['connections_created'] += 1
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Per v1.1.2 Phase 3:
|
|||||||
Specification: http://opml.org/spec2.opml
|
Specification: http://opml.org/spec2.opml
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime, timezone
|
||||||
from xml.sax.saxutils import escape
|
from xml.sax.saxutils import escape
|
||||||
|
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ def generate_opml(site_url: str, site_name: str) -> str:
|
|||||||
safe_site_name = escape(site_name)
|
safe_site_name = escape(site_name)
|
||||||
|
|
||||||
# RFC 822 date format (required by OPML spec)
|
# RFC 822 date format (required by OPML spec)
|
||||||
creation_date = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
|
creation_date = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%a, %d %b %Y %H:%M:%S GMT')
|
||||||
|
|
||||||
# Build OPML document
|
# Build OPML document
|
||||||
opml_lines = [
|
opml_lines = [
|
||||||
|
|||||||
+18
-17
@@ -13,7 +13,7 @@ current IndieAuth spec: https://indieauth.spec.indieweb.org/
|
|||||||
import base64
|
import base64
|
||||||
import hashlib
|
import hashlib
|
||||||
import secrets
|
import secrets
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from typing import Any, Dict, List, Optional
|
from typing import Any, Dict, List, Optional
|
||||||
from urllib.parse import urljoin, urlparse
|
from urllib.parse import urljoin, urlparse
|
||||||
|
|
||||||
@@ -83,41 +83,42 @@ def fetch_client_redirect_uris(client_id: str) -> List[str]:
|
|||||||
Supports a JSON client metadata document ("redirect_uris") and the older
|
Supports a JSON client metadata document ("redirect_uris") and the older
|
||||||
HTML/Link header form (rel="redirect_uri"). Returns [] on any failure.
|
HTML/Link header form (rel="redirect_uri"). Returns [] on any failure.
|
||||||
"""
|
"""
|
||||||
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
|
import json
|
||||||
|
from starpunk.webmention_ssrf import FetchTooLargeError, SSRFBlockedError, safe_get, user_agent
|
||||||
|
|
||||||
|
# Every redirect hop is checked and the connection pinned to the checked
|
||||||
|
# address, with a hard time limit (v2.11.0; DEV_MODE allows localhost)
|
||||||
try:
|
try:
|
||||||
if not current_app.config.get("DEV_MODE"):
|
result = safe_get(
|
||||||
validate_url_for_fetch(client_id)
|
|
||||||
response = httpx.get(
|
|
||||||
client_id,
|
client_id,
|
||||||
timeout=5.0,
|
headers={"Accept": "application/json, text/html;q=0.9",
|
||||||
follow_redirects=True,
|
"User-Agent": user_agent(current_app.config.get("SITE_URL", ""))},
|
||||||
headers={"Accept": "application/json, text/html;q=0.9"},
|
max_bytes=500_000, timeout=5.0, truncate=True,
|
||||||
)
|
)
|
||||||
except (httpx.HTTPError, SSRFBlockedError, ValueError) as e:
|
except (httpx.HTTPError, SSRFBlockedError, ValueError, FetchTooLargeError) as e:
|
||||||
current_app.logger.info(f"IndieAuth: could not fetch client {client_id}: {e}")
|
current_app.logger.info(f"IndieAuth: could not fetch client {client_id}: {e}")
|
||||||
return []
|
return []
|
||||||
|
|
||||||
if response.status_code != 200:
|
if result.status_code != 200:
|
||||||
return []
|
return []
|
||||||
|
|
||||||
base = str(response.url)
|
base = result.url
|
||||||
uris: List[str] = []
|
uris: List[str] = []
|
||||||
|
|
||||||
if "json" in response.headers.get("Content-Type", ""):
|
if "json" in result.headers.get("Content-Type", ""):
|
||||||
try:
|
try:
|
||||||
data = response.json()
|
data = json.loads(result.text)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return []
|
return []
|
||||||
if isinstance(data, dict) and isinstance(data.get("redirect_uris"), list):
|
if isinstance(data, dict) and isinstance(data.get("redirect_uris"), list):
|
||||||
uris.extend(u for u in data["redirect_uris"] if isinstance(u, str))
|
uris.extend(u for u in data["redirect_uris"] if isinstance(u, str))
|
||||||
return uris
|
return uris
|
||||||
|
|
||||||
link = response.links.get("redirect_uri")
|
link = httpx.Response(200, headers=result.headers).links.get("redirect_uri")
|
||||||
if link and link.get("url"):
|
if link and link.get("url"):
|
||||||
uris.append(urljoin(base, link["url"]))
|
uris.append(urljoin(base, link["url"]))
|
||||||
|
|
||||||
soup = BeautifulSoup(response.text[:500_000], "html.parser")
|
soup = BeautifulSoup(result.text, "html.parser")
|
||||||
for tag in soup.find_all(["link", "a"], href=True):
|
for tag in soup.find_all(["link", "a"], href=True):
|
||||||
if "redirect_uri" in (tag.get("rel") or []):
|
if "redirect_uri" in (tag.get("rel") or []):
|
||||||
uris.append(urljoin(base, tag["href"]))
|
uris.append(urljoin(base, tag["href"]))
|
||||||
@@ -203,7 +204,7 @@ def create_authorization_code(
|
|||||||
redirect_uri,
|
redirect_uri,
|
||||||
scope,
|
scope,
|
||||||
code_challenge,
|
code_challenge,
|
||||||
datetime.utcnow() + timedelta(minutes=CODE_LIFETIME_MINUTES),
|
datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=CODE_LIFETIME_MINUTES),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
@@ -258,7 +259,7 @@ def create_access_token(client_id: str, scope: str) -> Dict[str, Any]:
|
|||||||
"""Issue an access token (stored hashed); returns the token response"""
|
"""Issue an access token (stored hashed); returns the token response"""
|
||||||
token = secrets.token_urlsafe(32)
|
token = secrets.token_urlsafe(32)
|
||||||
lifetime_days = current_app.config.get("INDIEAUTH_TOKEN_LIFETIME_DAYS", 90)
|
lifetime_days = current_app.config.get("INDIEAUTH_TOKEN_LIFETIME_DAYS", 90)
|
||||||
expires_at = datetime.utcnow() + timedelta(days=lifetime_days)
|
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=lifetime_days)
|
||||||
|
|
||||||
db = get_db(current_app)
|
db = get_db(current_app)
|
||||||
db.execute(
|
db.execute(
|
||||||
|
|||||||
+9
-9
@@ -8,7 +8,7 @@ state. All models are immutable and use dataclasses for clean structure.
|
|||||||
|
|
||||||
# Standard library imports
|
# Standard library imports
|
||||||
from dataclasses import dataclass, field, replace
|
from dataclasses import dataclass, field, replace
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
import re
|
import re
|
||||||
@@ -698,7 +698,7 @@ class Session:
|
|||||||
>>> session.is_expired
|
>>> session.is_expired
|
||||||
False
|
False
|
||||||
"""
|
"""
|
||||||
return datetime.utcnow() >= self.expires_at
|
return datetime.now(timezone.utc).replace(tzinfo=None) >= self.expires_at
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def is_active(self) -> bool:
|
def is_active(self) -> bool:
|
||||||
@@ -726,7 +726,7 @@ class Session:
|
|||||||
>>> session.age
|
>>> session.age
|
||||||
datetime.timedelta(days=2, seconds=3600)
|
datetime.timedelta(days=2, seconds=3600)
|
||||||
"""
|
"""
|
||||||
return datetime.utcnow() - self.created_at
|
return datetime.now(timezone.utc).replace(tzinfo=None) - self.created_at
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def time_until_expiry(self) -> timedelta:
|
def time_until_expiry(self) -> timedelta:
|
||||||
@@ -740,7 +740,7 @@ class Session:
|
|||||||
>>> session.time_until_expiry
|
>>> session.time_until_expiry
|
||||||
datetime.timedelta(days=28)
|
datetime.timedelta(days=28)
|
||||||
"""
|
"""
|
||||||
return self.expires_at - datetime.utcnow()
|
return self.expires_at - datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
def is_valid(self) -> bool:
|
def is_valid(self) -> bool:
|
||||||
"""
|
"""
|
||||||
@@ -787,7 +787,7 @@ class Session:
|
|||||||
>>> updated.last_used_at
|
>>> updated.last_used_at
|
||||||
datetime.datetime(2024, 11, 18, 15, 30, 0)
|
datetime.datetime(2024, 11, 18, 15, 30, 0)
|
||||||
"""
|
"""
|
||||||
return replace(self, last_used_at=datetime.utcnow())
|
return replace(self, last_used_at=datetime.now(timezone.utc).replace(tzinfo=None))
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
def to_dict(self) -> dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
@@ -863,7 +863,7 @@ class Token:
|
|||||||
me: str
|
me: str
|
||||||
client_id: Optional[str] = None
|
client_id: Optional[str] = None
|
||||||
scope: Optional[str] = None
|
scope: Optional[str] = None
|
||||||
created_at: datetime = field(default_factory=lambda: datetime.utcnow())
|
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc).replace(tzinfo=None))
|
||||||
expires_at: Optional[datetime] = None
|
expires_at: Optional[datetime] = None
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -926,7 +926,7 @@ class Token:
|
|||||||
# No expiry set, token never expires
|
# No expiry set, token never expires
|
||||||
return False
|
return False
|
||||||
|
|
||||||
return datetime.utcnow() >= self.expires_at
|
return datetime.now(timezone.utc).replace(tzinfo=None) >= self.expires_at
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def is_active(self) -> bool:
|
def is_active(self) -> bool:
|
||||||
@@ -1145,7 +1145,7 @@ class AuthState:
|
|||||||
>>> auth_state.is_expired
|
>>> auth_state.is_expired
|
||||||
False
|
False
|
||||||
"""
|
"""
|
||||||
return datetime.utcnow() >= self.expires_at
|
return datetime.now(timezone.utc).replace(tzinfo=None) >= self.expires_at
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def is_active(self) -> bool:
|
def is_active(self) -> bool:
|
||||||
@@ -1169,7 +1169,7 @@ class AuthState:
|
|||||||
>>> auth_state.age
|
>>> auth_state.age
|
||||||
datetime.timedelta(seconds=120)
|
datetime.timedelta(seconds=120)
|
||||||
"""
|
"""
|
||||||
return datetime.utcnow() - self.created_at
|
return datetime.now(timezone.utc).replace(tzinfo=None) - self.created_at
|
||||||
|
|
||||||
def is_valid(self) -> bool:
|
def is_valid(self) -> bool:
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ import random
|
|||||||
import time
|
import time
|
||||||
from collections import deque
|
from collections import deque
|
||||||
from dataclasses import dataclass, field, asdict
|
from dataclasses import dataclass, field, asdict
|
||||||
from datetime import datetime
|
from datetime import datetime, timezone
|
||||||
from threading import Lock
|
from threading import Lock
|
||||||
from typing import Any, Deque, Dict, List, Literal, Optional, Union
|
from typing import Any, Deque, Dict, List, Literal, Optional, Union
|
||||||
|
|
||||||
@@ -159,7 +159,7 @@ class MetricsBuffer:
|
|||||||
metric = Metric(
|
metric = Metric(
|
||||||
operation_type=operation_type,
|
operation_type=operation_type,
|
||||||
operation_name=operation_name,
|
operation_name=operation_name,
|
||||||
timestamp=datetime.utcnow().isoformat() + "Z",
|
timestamp=datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
|
||||||
duration_ms=duration_ms,
|
duration_ms=duration_ms,
|
||||||
process_id=self._process_id,
|
process_id=self._process_id,
|
||||||
metadata=metadata or {}
|
metadata=metadata or {}
|
||||||
|
|||||||
+4
-4
@@ -20,7 +20,7 @@ Exceptions:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
# Standard library imports
|
# Standard library imports
|
||||||
from datetime import datetime
|
from datetime import datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
@@ -226,7 +226,7 @@ def create_note(
|
|||||||
|
|
||||||
# 2. SETUP
|
# 2. SETUP
|
||||||
if created_at is None:
|
if created_at is None:
|
||||||
created_at = datetime.utcnow()
|
created_at = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
updated_at = created_at # Same as created_at for new notes
|
updated_at = created_at # Same as created_at for new notes
|
||||||
|
|
||||||
@@ -735,7 +735,7 @@ def update_note(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# 3. SETUP
|
# 3. SETUP
|
||||||
updated_at = datetime.utcnow()
|
updated_at = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
data_dir = Path(current_app.config["DATA_PATH"])
|
data_dir = Path(current_app.config["DATA_PATH"])
|
||||||
note_path = data_dir / existing_note.file_path
|
note_path = data_dir / existing_note.file_path
|
||||||
|
|
||||||
@@ -1052,7 +1052,7 @@ def delete_note(
|
|||||||
|
|
||||||
if soft:
|
if soft:
|
||||||
# SOFT DELETE: Mark as deleted in database
|
# SOFT DELETE: Mark as deleted in database
|
||||||
deleted_at = datetime.utcnow()
|
deleted_at = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
db.execute(
|
db.execute(
|
||||||
|
|||||||
@@ -138,6 +138,79 @@ _ip_limiter: RateLimiter | None = None
|
|||||||
_domain_limiter: RateLimiter | None = None
|
_domain_limiter: RateLimiter | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class SharedRateLimiter:
|
||||||
|
"""
|
||||||
|
Rate limiter whose counts live in SQLite, shared by every gunicorn
|
||||||
|
process (v2.11.0). In-memory counters let each of the 4 workers allow the
|
||||||
|
full limit, so the real limit was four times the setting.
|
||||||
|
|
||||||
|
Sliding-window counter: the current window's count plus the previous
|
||||||
|
window's, weighted by how much of it still overlaps the last
|
||||||
|
`window_seconds`. Same interface as RateLimiter; needs an app context.
|
||||||
|
"""
|
||||||
|
|
||||||
|
CLEANUP_EVERY = 200 # calls between removals of old windows
|
||||||
|
|
||||||
|
def __init__(self, limit: int, window_seconds: int, prefix: str):
|
||||||
|
self.limit = limit
|
||||||
|
self.window_seconds = window_seconds
|
||||||
|
self.prefix = prefix
|
||||||
|
self._calls = 0
|
||||||
|
|
||||||
|
def _db(self):
|
||||||
|
from flask import current_app
|
||||||
|
from starpunk.database import get_db
|
||||||
|
return get_db(current_app)
|
||||||
|
|
||||||
|
def _windows(self):
|
||||||
|
now = time.time()
|
||||||
|
current = int(now // self.window_seconds)
|
||||||
|
overlap = 1 - (now % self.window_seconds) / self.window_seconds
|
||||||
|
return now, current, overlap
|
||||||
|
|
||||||
|
def _estimate(self, db, key: str) -> float:
|
||||||
|
_, current, overlap = self._windows()
|
||||||
|
counts = dict(db.execute(
|
||||||
|
"SELECT window, count FROM rate_limits WHERE key = ? AND window IN (?, ?)",
|
||||||
|
(f"{self.prefix}:{key}", current - 1, current),
|
||||||
|
).fetchall())
|
||||||
|
return counts.get(current - 1, 0) * overlap + counts.get(current, 0)
|
||||||
|
|
||||||
|
def is_allowed(self, key: str) -> bool:
|
||||||
|
db = self._db()
|
||||||
|
_, current, _ = self._windows()
|
||||||
|
if self._estimate(db, key) >= self.limit:
|
||||||
|
return False
|
||||||
|
db.execute(
|
||||||
|
"INSERT INTO rate_limits (key, window, count) VALUES (?, ?, 1) "
|
||||||
|
"ON CONFLICT(key, window) DO UPDATE SET count = count + 1",
|
||||||
|
(f"{self.prefix}:{key}", current),
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
self._calls += 1
|
||||||
|
if self._calls % self.CLEANUP_EVERY == 0:
|
||||||
|
self.cleanup()
|
||||||
|
return True
|
||||||
|
|
||||||
|
def get_retry_after(self, key: str) -> int:
|
||||||
|
now, current, _ = self._windows()
|
||||||
|
return max(1, int((current + 1) * self.window_seconds - now) + 1)
|
||||||
|
|
||||||
|
def get_request_count(self, key: str) -> int:
|
||||||
|
return int(self._estimate(self._db(), key))
|
||||||
|
|
||||||
|
def cleanup(self, max_idle_seconds: int = 300) -> int:
|
||||||
|
"""Remove windows too old to matter (this limiter's keys only)"""
|
||||||
|
_, current, _ = self._windows()
|
||||||
|
db = self._db()
|
||||||
|
cursor = db.execute(
|
||||||
|
"DELETE FROM rate_limits WHERE key LIKE ? AND window < ?",
|
||||||
|
(f"{self.prefix}:%", current - 1),
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
return cursor.rowcount
|
||||||
|
|
||||||
|
|
||||||
def init_rate_limiters(app) -> None:
|
def init_rate_limiters(app) -> None:
|
||||||
"""
|
"""
|
||||||
Initialize rate limiters with Flask app configuration.
|
Initialize rate limiters with Flask app configuration.
|
||||||
@@ -157,8 +230,9 @@ def init_rate_limiters(app) -> None:
|
|||||||
domain_limit = app.config.get('WEBMENTION_RATE_LIMIT_DOMAIN', 10)
|
domain_limit = app.config.get('WEBMENTION_RATE_LIMIT_DOMAIN', 10)
|
||||||
window = app.config.get('WEBMENTION_RATE_LIMIT_WINDOW', 60)
|
window = app.config.get('WEBMENTION_RATE_LIMIT_WINDOW', 60)
|
||||||
|
|
||||||
_ip_limiter = RateLimiter(ip_limit, window)
|
# Shared by all server processes (v2.11.0)
|
||||||
_domain_limiter = RateLimiter(domain_limit, window)
|
_ip_limiter = SharedRateLimiter(ip_limit, window, "webmention-ip")
|
||||||
|
_domain_limiter = SharedRateLimiter(domain_limit, window, "webmention-domain")
|
||||||
|
|
||||||
app.logger.info(
|
app.logger.info(
|
||||||
f"Webmention rate limiter initialized: {ip_limit}/min IP, "
|
f"Webmention rate limiter initialized: {ip_limit}/min IP, "
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ from flask import Blueprint, flash, g, jsonify, redirect, render_template, reque
|
|||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
from datetime import datetime
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -531,7 +531,7 @@ def metrics():
|
|||||||
from starpunk.monitoring.business import get_feed_statistics
|
from starpunk.monitoring.business import get_feed_statistics
|
||||||
|
|
||||||
response = {
|
response = {
|
||||||
"timestamp": datetime.utcnow().isoformat() + "Z",
|
"timestamp": datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
|
||||||
"process_id": os.getpid(),
|
"process_id": os.getpid(),
|
||||||
"database": {},
|
"database": {},
|
||||||
"performance": {},
|
"performance": {},
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ No authentication required for these routes.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
|
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
|
||||||
|
|
||||||
@@ -54,7 +54,7 @@ def _get_cached_notes():
|
|||||||
# Get cache duration from config (in seconds)
|
# Get cache duration from config (in seconds)
|
||||||
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
|
cache_seconds = current_app.config.get("FEED_CACHE_SECONDS", 300)
|
||||||
cache_duration = timedelta(seconds=cache_seconds)
|
cache_duration = timedelta(seconds=cache_seconds)
|
||||||
now = datetime.utcnow()
|
now = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
# The cached list is stale once any public note changed, in any process
|
# The cached list is stale once any public note changed, in any process
|
||||||
# (v2.10.0): every change bumps content_version in the database
|
# (v2.10.0): every change bumps content_version in the database
|
||||||
|
|||||||
@@ -9,16 +9,15 @@ optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
|
|||||||
from flask import Blueprint, current_app, request
|
from flask import Blueprint, current_app, request
|
||||||
|
|
||||||
from starpunk.database import get_db
|
from starpunk.database import get_db
|
||||||
from starpunk.rate_limiter import RateLimiter
|
from starpunk.rate_limiter import SharedRateLimiter
|
||||||
from starpunk import websub
|
from starpunk import websub
|
||||||
|
|
||||||
bp = Blueprint("websub", __name__)
|
bp = Blueprint("websub", __name__)
|
||||||
|
|
||||||
# Subscription requests per client IP per minute
|
# Subscription requests per client IP per minute
|
||||||
# (per gunicorn process; needs TRUSTED_PROXY=true behind a reverse proxy,
|
# Shared by all server processes (v2.11.0). Needs TRUSTED_PROXY=true behind a
|
||||||
# or every client shares the proxy's address)
|
# reverse proxy, or every client shares the proxy's address.
|
||||||
_limiter = RateLimiter(limit=30, window_seconds=60)
|
_limiter = SharedRateLimiter(limit=30, window_seconds=60, prefix="websub")
|
||||||
_requests = 0
|
|
||||||
|
|
||||||
|
|
||||||
def _text(message: str, status: int):
|
def _text(message: str, status: int):
|
||||||
@@ -31,10 +30,6 @@ def hub():
|
|||||||
return _text("WebSub is not enabled on this site", 404)
|
return _text("WebSub is not enabled on this site", 404)
|
||||||
|
|
||||||
from starpunk.routes.webmention import get_client_ip
|
from starpunk.routes.webmention import get_client_ip
|
||||||
global _requests
|
|
||||||
_requests += 1
|
|
||||||
if _requests % 500 == 0:
|
|
||||||
_limiter.cleanup() # forget idle clients so memory doesn't grow
|
|
||||||
if not _limiter.is_allowed(get_client_ip()):
|
if not _limiter.is_allowed(get_client_ip()):
|
||||||
return _text("Too many requests", 429)
|
return _text("Too many requests", 429)
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Per developer Q&A Q8:
|
|||||||
import re
|
import re
|
||||||
import unicodedata
|
import unicodedata
|
||||||
import logging
|
import logging
|
||||||
from datetime import datetime
|
from datetime import datetime, timezone
|
||||||
from typing import Optional, Set
|
from typing import Optional, Set
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
@@ -138,7 +138,7 @@ def sanitize_slug(slug: str, allow_timestamp_fallback: bool = False) -> str:
|
|||||||
# Check if normalization resulted in empty slug
|
# Check if normalization resulted in empty slug
|
||||||
if not slug and allow_timestamp_fallback:
|
if not slug and allow_timestamp_fallback:
|
||||||
# Per Q8: Use timestamp-based fallback
|
# Per Q8: Use timestamp-based fallback
|
||||||
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S')
|
timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
|
||||||
slug = timestamp
|
slug = timestamp
|
||||||
logger.warning(
|
logger.warning(
|
||||||
f"Slug normalization failed for input '{original_slug}' "
|
f"Slug normalization failed for input '{original_slug}' "
|
||||||
@@ -268,7 +268,7 @@ def generate_timestamp_slug(
|
|||||||
'20251216143052-2'
|
'20251216143052-2'
|
||||||
"""
|
"""
|
||||||
if created_at is None:
|
if created_at is None:
|
||||||
created_at = datetime.utcnow()
|
created_at = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
if existing_slugs is None:
|
if existing_slugs is None:
|
||||||
existing_slugs = set()
|
existing_slugs = set()
|
||||||
@@ -340,7 +340,7 @@ def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]
|
|||||||
if not sanitized:
|
if not sanitized:
|
||||||
# This should never happen with allow_timestamp_fallback=True
|
# This should never happen with allow_timestamp_fallback=True
|
||||||
# but handle it just in case
|
# but handle it just in case
|
||||||
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S')
|
timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
|
||||||
sanitized = timestamp
|
sanitized = timestamp
|
||||||
logger.error(
|
logger.error(
|
||||||
f"Unexpected empty slug after sanitization with fallback. "
|
f"Unexpected empty slug after sanitization with fallback. "
|
||||||
@@ -361,7 +361,7 @@ def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]
|
|||||||
if not validate_slug(sanitized):
|
if not validate_slug(sanitized):
|
||||||
# This should rarely happen after sanitization
|
# This should rarely happen after sanitization
|
||||||
# but if it does, use timestamp fallback
|
# but if it does, use timestamp fallback
|
||||||
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S')
|
timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
|
||||||
logger.warning(
|
logger.warning(
|
||||||
f"Slug '{sanitized}' (from '{custom_slug}') failed validation. "
|
f"Slug '{sanitized}' (from '{custom_slug}') failed validation. "
|
||||||
f"Using timestamp fallback: {timestamp}"
|
f"Using timestamp fallback: {timestamp}"
|
||||||
@@ -375,7 +375,7 @@ def validate_and_sanitize_custom_slug(custom_slug: str, existing_slugs: Set[str]
|
|||||||
except ValueError as e:
|
except ValueError as e:
|
||||||
# This should rarely happen, but if it does, use timestamp
|
# This should rarely happen, but if it does, use timestamp
|
||||||
# Per Q8: Never fail Micropub request
|
# Per Q8: Never fail Micropub request
|
||||||
timestamp = datetime.utcnow().strftime('%Y%m%d-%H%M%S')
|
timestamp = datetime.now(timezone.utc).replace(tzinfo=None).strftime('%Y%m%d-%H%M%S')
|
||||||
logger.error(
|
logger.error(
|
||||||
f"Could not create unique slug from '{custom_slug}'. "
|
f"Could not create unique slug from '{custom_slug}'. "
|
||||||
f"Using timestamp: {timestamp}. Error: {e}"
|
f"Using timestamp: {timestamp}. Error: {e}"
|
||||||
|
|||||||
+30
-32
@@ -444,39 +444,37 @@ CARD_IMAGE_LIMIT = 5 * 1024 * 1024
|
|||||||
|
|
||||||
def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple:
|
def _fetch_public(http: httpx.Client, url: str, limit: int, truncate: bool) -> tuple:
|
||||||
"""
|
"""
|
||||||
GET a public web URL, checking every redirect hop (no requests into the
|
GET a public web URL (link card pages, images to re-upload).
|
||||||
LAN). Reads at most `limit` bytes: stops there if `truncate`, else fails.
|
|
||||||
Returns (data, content_type, final_url).
|
|
||||||
"""
|
|
||||||
from urllib.parse import urljoin, urlparse
|
|
||||||
from starpunk.micropub import _public_address
|
|
||||||
|
|
||||||
site = current_app.config.get("SITE_URL", "")
|
Uses the shared SSRF-safe fetcher (v2.11.0): every redirect hop checked,
|
||||||
for _ in range(5):
|
the connection pinned to the checked address, a hard 10 s limit. Reads at
|
||||||
parsed = urlparse(url)
|
most `limit` bytes: stops there if `truncate`, else fails.
|
||||||
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
Returns (data, content_type, final_url). `http` is no longer used.
|
||||||
raise SyndicationError(f"not a web URL: {url}", permanent=True)
|
"""
|
||||||
if not current_app.config.get("TESTING") and not _public_address(parsed.hostname):
|
from urllib.parse import urlparse
|
||||||
raise SyndicationError(f"not a public address: {url}", permanent=True)
|
from starpunk.webmention_ssrf import (
|
||||||
try:
|
FetchTooLargeError, ResolutionError, SSRFBlockedError, safe_get, user_agent,
|
||||||
with http.stream("GET", url, follow_redirects=False, timeout=10.0,
|
)
|
||||||
headers={"User-Agent": f"StarPunk (+{site})"}) as response:
|
|
||||||
if response.is_redirect and "location" in response.headers:
|
parsed = urlparse(url)
|
||||||
url = urljoin(url, response.headers["location"])
|
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||||
continue
|
raise SyndicationError(f"not a web URL: {url}", permanent=True)
|
||||||
if response.status_code >= 400:
|
try:
|
||||||
raise SyndicationError(f"{url}: HTTP {response.status_code}")
|
result = safe_get(url, headers={"User-Agent": user_agent(current_app.config.get("SITE_URL", ""))},
|
||||||
data = bytearray()
|
max_bytes=limit, max_redirects=5, timeout=10.0, truncate=truncate)
|
||||||
for chunk in response.iter_bytes():
|
except ResolutionError as e:
|
||||||
data += chunk
|
raise SyndicationError(f"{url}: {e}")
|
||||||
if len(data) > limit:
|
except (SSRFBlockedError, ValueError) as e:
|
||||||
if truncate:
|
raise SyndicationError(f"not a public address: {url} ({e})", permanent=True)
|
||||||
break
|
except FetchTooLargeError:
|
||||||
raise SyndicationError(f"{url}: too large")
|
raise SyndicationError(f"{url}: too large")
|
||||||
return bytes(data), response.headers.get("content-type", ""), url
|
except httpx.TooManyRedirects:
|
||||||
except httpx.HTTPError as e:
|
raise SyndicationError(f"{url}: too many redirects")
|
||||||
raise SyndicationError(f"{url}: {type(e).__name__}")
|
except httpx.HTTPError as e:
|
||||||
raise SyndicationError(f"{url}: too many redirects")
|
raise SyndicationError(f"{url}: {type(e).__name__}")
|
||||||
|
if result.status_code >= 400:
|
||||||
|
raise SyndicationError(f"{url}: HTTP {result.status_code}")
|
||||||
|
return result.content, result.headers.get("content-type", ""), result.url
|
||||||
|
|
||||||
|
|
||||||
def page_card(http: httpx.Client, url: str) -> dict:
|
def page_card(http: httpx.Client, url: str) -> dict:
|
||||||
|
|||||||
+2
-2
@@ -217,7 +217,7 @@ def generate_slug(content: str, created_at: Optional[datetime] = None) -> str:
|
|||||||
# If slug is empty or too short, use timestamp fallback
|
# If slug is empty or too short, use timestamp fallback
|
||||||
if len(slug) < MIN_SLUG_LENGTH:
|
if len(slug) < MIN_SLUG_LENGTH:
|
||||||
if created_at is None:
|
if created_at is None:
|
||||||
created_at = datetime.utcnow()
|
created_at = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
slug = created_at.strftime("%Y%m%d-%H%M%S")
|
slug = created_at.strftime("%Y%m%d-%H%M%S")
|
||||||
|
|
||||||
# Truncate to maximum length
|
# Truncate to maximum length
|
||||||
@@ -592,7 +592,7 @@ def delete_note_file(
|
|||||||
month = parts[notes_idx + 2]
|
month = parts[notes_idx + 2]
|
||||||
except (ValueError, IndexError):
|
except (ValueError, IndexError):
|
||||||
# If path doesn't follow expected structure, use current date
|
# If path doesn't follow expected structure, use current date
|
||||||
now = datetime.utcnow()
|
now = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
year = now.strftime("%Y")
|
year = now.strftime("%Y")
|
||||||
month = now.strftime("%m")
|
month = now.strftime("%m")
|
||||||
|
|
||||||
|
|||||||
@@ -250,7 +250,8 @@ def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = N
|
|||||||
|
|
||||||
|
|
||||||
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||||
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
|
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT,
|
||||||
|
truncate: bool = False) -> FetchResult:
|
||||||
"""
|
"""
|
||||||
GET a URL, following redirects by hand and checking each hop.
|
GET a URL, following redirects by hand and checking each hop.
|
||||||
|
|
||||||
@@ -258,6 +259,8 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
|||||||
url: URL to fetch
|
url: URL to fetch
|
||||||
headers: Request headers
|
headers: Request headers
|
||||||
max_bytes: Response body limit
|
max_bytes: Response body limit
|
||||||
|
truncate: Stop at max_bytes and return what was read, instead of
|
||||||
|
raising FetchTooLargeError (v2.11.0)
|
||||||
max_redirects: Redirect limit
|
max_redirects: Redirect limit
|
||||||
timeout: Total time limit in seconds
|
timeout: Total time limit in seconds
|
||||||
|
|
||||||
@@ -287,13 +290,16 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
content_length = response.headers.get('Content-Length', '')
|
content_length = response.headers.get('Content-Length', '')
|
||||||
if content_length.isdigit() and int(content_length) > max_bytes:
|
if not truncate and content_length.isdigit() and int(content_length) > max_bytes:
|
||||||
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
|
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
|
||||||
|
|
||||||
content = b''
|
content = b''
|
||||||
for chunk in response.iter_bytes(chunk_size=8192):
|
for chunk in response.iter_bytes(chunk_size=8192):
|
||||||
content += chunk
|
content += chunk
|
||||||
if len(content) > max_bytes:
|
if len(content) > max_bytes:
|
||||||
|
if truncate:
|
||||||
|
content = content[:max_bytes]
|
||||||
|
break
|
||||||
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
|
raise FetchTooLargeError(f"Response exceeds size limit ({max_bytes} bytes)")
|
||||||
if time.monotonic() > deadline:
|
if time.monotonic() > deadline:
|
||||||
raise httpx.TimeoutException("Fetch exceeded time limit")
|
raise httpx.TimeoutException("Fetch exceeded time limit")
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ import threading
|
|||||||
import time
|
import time
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import httpx
|
import httpx
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Dict, Any, Optional
|
from typing import Dict, Any, Optional
|
||||||
from urllib.parse import urljoin
|
from urllib.parse import urljoin
|
||||||
@@ -116,7 +116,7 @@ class WebmentionWorker(threading.Thread):
|
|||||||
|
|
||||||
# Update last processed time
|
# Update last processed time
|
||||||
if processed > 0:
|
if processed > 0:
|
||||||
self._last_processed = datetime.utcnow()
|
self._last_processed = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
# Sleep if nothing processed
|
# Sleep if nothing processed
|
||||||
if processed == 0:
|
if processed == 0:
|
||||||
@@ -790,7 +790,7 @@ class WebmentionWorker(threading.Thread):
|
|||||||
}
|
}
|
||||||
delay = delays.get(new_retry_count)
|
delay = delays.get(new_retry_count)
|
||||||
if delay:
|
if delay:
|
||||||
next_retry_at = datetime.utcnow() + delay
|
next_retry_at = datetime.now(timezone.utc).replace(tzinfo=None) + delay
|
||||||
|
|
||||||
try:
|
try:
|
||||||
conn = sqlite3.connect(self.db_path)
|
conn = sqlite3.connect(self.db_path)
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ import hashlib
|
|||||||
import hmac
|
import hmac
|
||||||
import logging
|
import logging
|
||||||
import threading
|
import threading
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
|
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
|
||||||
@@ -53,11 +53,11 @@ def signature(secret: str, body: bytes) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _now() -> str:
|
def _now() -> str:
|
||||||
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
|
return datetime.now(timezone.utc).replace(tzinfo=None).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
def _later(seconds: int) -> str:
|
def _later(seconds: int) -> str:
|
||||||
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
|
return (datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
|
|
||||||
class WebSubWorker(threading.Thread):
|
class WebSubWorker(threading.Thread):
|
||||||
@@ -132,7 +132,7 @@ class WebSubWorker(threading.Thread):
|
|||||||
).fetchall():
|
).fetchall():
|
||||||
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
|
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
|
||||||
done += 1
|
done += 1
|
||||||
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
|
if self._last_prune is None or datetime.now(timezone.utc).replace(tzinfo=None) - self._last_prune > PRUNE_EVERY:
|
||||||
self.prune(db)
|
self.prune(db)
|
||||||
return done
|
return done
|
||||||
|
|
||||||
@@ -348,7 +348,7 @@ class WebSubWorker(threading.Thread):
|
|||||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
|
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
|
||||||
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
|
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
|
||||||
db.commit()
|
db.commit()
|
||||||
self._last_prune = datetime.utcnow()
|
self._last_prune = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
if cursor.rowcount:
|
if cursor.rowcount:
|
||||||
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
|
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
|
||||||
return cursor.rowcount
|
return cursor.rowcount
|
||||||
|
|||||||
@@ -16,7 +16,7 @@
|
|||||||
</p>
|
</p>
|
||||||
<p><small>After approval you will be sent to {{ auth_request.redirect_uri }}</small></p>
|
<p><small>After approval you will be sent to {{ auth_request.redirect_uri }}</small></p>
|
||||||
|
|
||||||
<form action="{{ url_for('indieauth.approve') }}" method="POST" class="login-form">
|
<form action="{{ url_for('indieauth.approve') }}" method="POST" class="login-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
{% for name, value in fields.items() %}
|
{% for name, value in fields.items() %}
|
||||||
<input type="hidden" name="{{ name }}" value="{{ value }}">
|
<input type="hidden" name="{{ name }}" value="{{ value }}">
|
||||||
{% endfor %}
|
{% endfor %}
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
<a href="{{ url_for('admin.new_note_form') }}">New Note</a>
|
<a href="{{ url_for('admin.new_note_form') }}">New Note</a>
|
||||||
<a href="{{ url_for('admin.metrics_dashboard') }}">Metrics</a>
|
<a href="{{ url_for('admin.metrics_dashboard') }}">Metrics</a>
|
||||||
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
|
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
|
||||||
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form">
|
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<button type="submit" class="button button-secondary">Logout</button>
|
<button type="submit" class="button button-secondary">Logout</button>
|
||||||
</form>
|
</form>
|
||||||
</nav>
|
</nav>
|
||||||
|
|||||||
@@ -74,7 +74,7 @@
|
|||||||
<a href="{{ url_for('admin.edit_note_form', note_id=note.id) }}" class="button button-small">
|
<a href="{{ url_for('admin.edit_note_form', note_id=note.id) }}" class="button button-small">
|
||||||
Edit
|
Edit
|
||||||
</a>
|
</a>
|
||||||
<form action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note?');">
|
<form action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note?');"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<input type="hidden" name="confirm" value="yes">
|
<input type="hidden" name="confirm" value="yes">
|
||||||
<button type="submit" class="button button-small button-danger">Delete</button>
|
<button type="submit" class="button button-small button-danger">Delete</button>
|
||||||
</form>
|
</form>
|
||||||
|
|||||||
@@ -10,7 +10,7 @@
|
|||||||
Created: {{ note.created_at.strftime('%B %d, %Y at %I:%M %p') }}
|
Created: {{ note.created_at.strftime('%B %d, %Y at %I:%M %p') }}
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form">
|
<form action="{{ url_for('admin.update_note_submit', note_id=note.id) }}" method="POST" class="note-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<div class="form-group">
|
<div class="form-group">
|
||||||
{% if note.reacts_to %}
|
{% if note.reacts_to %}
|
||||||
<p class="note-meta">{{ {'like': '★ Like of', 'repost': '🔁 Repost of', 'bookmark': '🔖 Bookmark of'}[note.kind] }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
|
<p class="note-meta">{{ {'like': '★ Like of', 'repost': '🔁 Repost of', 'bookmark': '🔖 Bookmark of'}[note.kind] }} <a href="{{ note.reacts_to }}">{{ note.reacts_to }}</a></p>
|
||||||
@@ -76,7 +76,7 @@
|
|||||||
</form>
|
</form>
|
||||||
|
|
||||||
{# Outside the edit form: nested forms are not allowed in HTML #}
|
{# Outside the edit form: nested forms are not allowed in HTML #}
|
||||||
<form id="delete-note-form" action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');">
|
<form id="delete-note-form" action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<input type="hidden" name="confirm" value="yes">
|
<input type="hidden" name="confirm" value="yes">
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
@@ -98,7 +98,7 @@
|
|||||||
{% endif %}
|
{% endif %}
|
||||||
{% if s.last_error %}<br><small>{{ s.last_error }}</small>{% endif %}
|
{% if s.last_error %}<br><small>{{ s.last_error }}</small>{% endif %}
|
||||||
{% if s.state != 'synced' %}
|
{% if s.state != 'synced' %}
|
||||||
<form action="{{ url_for('admin.retry_syndication', syndication_id=s.id) }}" method="POST" style="display: inline;">
|
<form action="{{ url_for('admin.retry_syndication', syndication_id=s.id) }}" method="POST" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<button type="submit" class="button button-secondary">Retry now</button>
|
<button type="submit" class="button button-secondary">Retry now</button>
|
||||||
</form>
|
</form>
|
||||||
{% endif %}
|
{% endif %}
|
||||||
|
|||||||
@@ -6,7 +6,7 @@
|
|||||||
<div class="note-editor">
|
<div class="note-editor">
|
||||||
<h2>Create New Note</h2>
|
<h2>Create New Note</h2>
|
||||||
|
|
||||||
<form action="{{ url_for('admin.create_note_submit') }}" method="POST" enctype="multipart/form-data" class="note-form">
|
<form action="{{ url_for('admin.create_note_submit') }}" method="POST" enctype="multipart/form-data" class="note-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<div class="form-group">
|
<div class="form-group">
|
||||||
<label for="content">Content (Markdown)</label>
|
<label for="content">Content (Markdown)</label>
|
||||||
<textarea
|
<textarea
|
||||||
|
|||||||
@@ -29,7 +29,7 @@
|
|||||||
<td>{{ t.last_used_at or "never" }}</td>
|
<td>{{ t.last_used_at or "never" }}</td>
|
||||||
<td>{{ t.expires_at }}</td>
|
<td>{{ t.expires_at }}</td>
|
||||||
<td>
|
<td>
|
||||||
<form method="POST" action="{{ url_for('admin.revoke_token', token_id=t.id) }}">
|
<form method="POST" action="{{ url_for('admin.revoke_token', token_id=t.id) }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<button type="submit" class="button button-danger"
|
<button type="submit" class="button button-danger"
|
||||||
onclick="return confirm('Revoke this app\'s access?')">Revoke</button>
|
onclick="return confirm('Revoke this app\'s access?')">Revoke</button>
|
||||||
</form>
|
</form>
|
||||||
|
|||||||
@@ -83,21 +83,21 @@
|
|||||||
{# Moderation Actions #}
|
{# Moderation Actions #}
|
||||||
<div class="moderation-actions">
|
<div class="moderation-actions">
|
||||||
<h3>Actions</h3>
|
<h3>Actions</h3>
|
||||||
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;">
|
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<input type="hidden" name="action" value="approve">
|
<input type="hidden" name="action" value="approve">
|
||||||
<button type="submit" class="button button-primary" {% if webmention.moderation_status == 'approved' %}disabled{% endif %}>
|
<button type="submit" class="button button-primary" {% if webmention.moderation_status == 'approved' %}disabled{% endif %}>
|
||||||
Approve
|
Approve
|
||||||
</button>
|
</button>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;">
|
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<input type="hidden" name="action" value="hide">
|
<input type="hidden" name="action" value="hide">
|
||||||
<button type="submit" class="button" {% if webmention.moderation_status == 'hidden' %}disabled{% endif %}>
|
<button type="submit" class="button" {% if webmention.moderation_status == 'hidden' %}disabled{% endif %}>
|
||||||
Hide
|
Hide
|
||||||
</button>
|
</button>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;">
|
<form method="POST" action="{{ url_for('admin.webmention_action', id=webmention.id) }}" style="display: inline;"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<input type="hidden" name="action" value="delete">
|
<input type="hidden" name="action" value="delete">
|
||||||
<button type="submit" class="button button-danger" onclick="return confirm('Delete this webmention permanently?')">
|
<button type="submit" class="button button-danger" onclick="return confirm('Delete this webmention permanently?')">
|
||||||
Delete
|
Delete
|
||||||
|
|||||||
@@ -29,7 +29,7 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
{% if webmentions %}
|
{% if webmentions %}
|
||||||
<form method="POST" action="{{ url_for('admin.webmention_bulk_action') }}">
|
<form method="POST" action="{{ url_for('admin.webmention_bulk_action') }}"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<div class="bulk-actions">
|
<div class="bulk-actions">
|
||||||
<button type="submit" name="action" value="approve" class="button">
|
<button type="submit" name="action" value="approve" class="button">
|
||||||
Approve Selected
|
Approve Selected
|
||||||
|
|||||||
+6
-6
@@ -4,7 +4,7 @@ Tests for authentication module (starpunk/auth.py)
|
|||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import secrets
|
import secrets
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from unittest.mock import MagicMock, patch
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
@@ -102,7 +102,7 @@ class TestCleanup:
|
|||||||
with app.app_context():
|
with app.app_context():
|
||||||
# Create expired session
|
# Create expired session
|
||||||
token_hash = _hash_token("expired-token")
|
token_hash = _hash_token("expired-token")
|
||||||
expires_at = datetime.utcnow() - timedelta(days=1)
|
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1)
|
||||||
|
|
||||||
db.execute(
|
db.execute(
|
||||||
"""
|
"""
|
||||||
@@ -126,7 +126,7 @@ class TestCleanup:
|
|||||||
"""Test that cleanup keeps valid sessions"""
|
"""Test that cleanup keeps valid sessions"""
|
||||||
with app.app_context():
|
with app.app_context():
|
||||||
token_hash = _hash_token("valid-token")
|
token_hash = _hash_token("valid-token")
|
||||||
expires_at = datetime.utcnow() + timedelta(days=30)
|
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30)
|
||||||
|
|
||||||
db.execute(
|
db.execute(
|
||||||
"""
|
"""
|
||||||
@@ -226,7 +226,7 @@ class TestVerifySession:
|
|||||||
# Create expired session
|
# Create expired session
|
||||||
token = secrets.token_urlsafe(32)
|
token = secrets.token_urlsafe(32)
|
||||||
token_hash = _hash_token(token)
|
token_hash = _hash_token(token)
|
||||||
expires_at = datetime.utcnow() - timedelta(days=1)
|
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1)
|
||||||
|
|
||||||
db.execute(
|
db.execute(
|
||||||
"""
|
"""
|
||||||
@@ -340,7 +340,7 @@ class TestRequireAuthDecorator:
|
|||||||
with app.app_context():
|
with app.app_context():
|
||||||
token = secrets.token_urlsafe(32)
|
token = secrets.token_urlsafe(32)
|
||||||
token_hash = _hash_token(token)
|
token_hash = _hash_token(token)
|
||||||
expires_at = datetime.utcnow() - timedelta(days=1)
|
expires_at = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1)
|
||||||
|
|
||||||
db.execute(
|
db.execute(
|
||||||
"""
|
"""
|
||||||
@@ -397,7 +397,7 @@ class TestSecurityFeatures:
|
|||||||
).fetchone()
|
).fetchone()
|
||||||
|
|
||||||
expires_at = datetime.fromisoformat(result["expires_at"])
|
expires_at = datetime.fromisoformat(result["expires_at"])
|
||||||
created_at = datetime.utcnow()
|
created_at = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
# Should expire approximately 1 day from now
|
# Should expire approximately 1 day from now
|
||||||
# (allow for minor timing differences)
|
# (allow for minor timing differences)
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ Per v1.2.0 Phase 2 and developer Q&A Q31-Q35:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from unittest.mock import Mock, patch
|
from unittest.mock import Mock, patch
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
@@ -203,7 +203,7 @@ class TestGetAuthorProfile:
|
|||||||
from starpunk.database import get_db
|
from starpunk.database import get_db
|
||||||
db = get_db(app)
|
db = get_db(app)
|
||||||
|
|
||||||
expired_time = (datetime.utcnow() - timedelta(hours=48)).isoformat()
|
expired_time = (datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=48)).isoformat()
|
||||||
|
|
||||||
db.execute(
|
db.execute(
|
||||||
"""
|
"""
|
||||||
@@ -296,9 +296,9 @@ class TestSaveAuthorProfile:
|
|||||||
'rel_me_links': [],
|
'rel_me_links': [],
|
||||||
}
|
}
|
||||||
|
|
||||||
before_save = datetime.utcnow()
|
before_save = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
save_author_profile('https://cache.example.com', profile)
|
save_author_profile('https://cache.example.com', profile)
|
||||||
after_save = datetime.utcnow()
|
after_save = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
# Check cache expiry
|
# Check cache expiry
|
||||||
db = get_db(app)
|
db = get_db(app)
|
||||||
|
|||||||
@@ -0,0 +1,80 @@
|
|||||||
|
"""
|
||||||
|
CSRF protection on admin and consent forms (v2.11.0)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from starpunk import create_app
|
||||||
|
from starpunk.auth import create_session
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def app(tmp_path):
|
||||||
|
return create_app({
|
||||||
|
"TESTING": True, "CSRF_ENABLED": True, # the real check, unlike other tests
|
||||||
|
"SITE_URL": "http://localhost:5000", "ADMIN_ME": "https://test.example.com",
|
||||||
|
"SESSION_SECRET": "x", "DATA_PATH": tmp_path, "NOTES_PATH": tmp_path / "notes",
|
||||||
|
"DATABASE_PATH": tmp_path / "starpunk.db",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def admin(app):
|
||||||
|
client = app.test_client()
|
||||||
|
with app.test_request_context():
|
||||||
|
client.set_cookie("starpunk_session", create_session(app.config["ADMIN_ME"]))
|
||||||
|
return client
|
||||||
|
|
||||||
|
|
||||||
|
def token_from(page: bytes) -> str:
|
||||||
|
return re.search(rb'name="csrf_token" value="([0-9a-f]{64})"', page).group(1).decode()
|
||||||
|
|
||||||
|
|
||||||
|
def test_forms_carry_a_token(admin):
|
||||||
|
assert token_from(admin.get("/admin/new").data)
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_without_token_is_refused(app, admin):
|
||||||
|
response = admin.post("/admin/new", data={"content": "Forged"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
from starpunk.notes import list_notes
|
||||||
|
with app.app_context():
|
||||||
|
assert list_notes() == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_with_wrong_token_is_refused(admin):
|
||||||
|
assert admin.post("/admin/new", data={"content": "x", "csrf_token": "0" * 64}).status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_with_token_works(app, admin):
|
||||||
|
token = token_from(admin.get("/admin/new").data)
|
||||||
|
response = admin.post("/admin/new", data={"content": "Real", "csrf_token": token, "published": "on"})
|
||||||
|
assert response.status_code == 302
|
||||||
|
from starpunk.notes import list_notes
|
||||||
|
with app.app_context():
|
||||||
|
assert [n.content.strip() for n in list_notes()] == ["Real"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_token_changes_with_the_session(app):
|
||||||
|
tokens = []
|
||||||
|
for _ in range(2):
|
||||||
|
client = app.test_client()
|
||||||
|
with app.test_request_context():
|
||||||
|
client.set_cookie("starpunk_session", create_session(app.config["ADMIN_ME"]))
|
||||||
|
tokens.append(token_from(client.get("/admin/new").data))
|
||||||
|
assert tokens[0] != tokens[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_logout_and_consent_are_protected(admin):
|
||||||
|
assert admin.post("/auth/logout").status_code == 400
|
||||||
|
assert admin.post("/auth/authorize/approve", data={"client_id": "https://app.example/"}).status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_app_endpoints_are_not_affected(app):
|
||||||
|
client = app.test_client()
|
||||||
|
# Bearer-token and public endpoints answer normally, without a CSRF token
|
||||||
|
assert client.post("/micropub", data={"h": "entry", "content": "x"}).status_code == 401
|
||||||
|
assert client.post("/auth/token", data={"grant_type": "authorization_code"}).status_code == 400
|
||||||
|
assert client.post("/webmention", data={}).status_code == 400
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""
|
||||||
|
Foreign keys are enforced (v2.11.0)
|
||||||
|
|
||||||
|
SQLite leaves foreign keys off per connection, so ON DELETE CASCADE never ran
|
||||||
|
and hard deletes left tag, media and variant rows behind.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from starpunk.database import get_db
|
||||||
|
from starpunk.migrations import execute_in_transaction
|
||||||
|
|
||||||
|
|
||||||
|
def png():
|
||||||
|
buf = io.BytesIO()
|
||||||
|
Image.new("RGB", (30, 20), "red").save(buf, format="PNG")
|
||||||
|
return buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def count(app, sql):
|
||||||
|
return sqlite3.connect(app.config["DATABASE_PATH"]).execute(sql).fetchone()[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_pooled_connections_enforce_foreign_keys(app):
|
||||||
|
with app.app_context():
|
||||||
|
assert get_db(app).execute("PRAGMA foreign_keys").fetchone()[0] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_hard_delete_removes_tags_and_media_links(app):
|
||||||
|
from starpunk.media import attach_media_to_note, save_media
|
||||||
|
from starpunk.notes import create_note, delete_note
|
||||||
|
with app.test_request_context():
|
||||||
|
note = create_note("Tagged with a photo", published=True, tags=["one", "two"], has_media=True)
|
||||||
|
media = save_media(png(), "a.png")
|
||||||
|
attach_media_to_note(note.id, [media["id"]], [""])
|
||||||
|
delete_note(id=note.id, soft=False)
|
||||||
|
assert count(app, "SELECT COUNT(*) FROM note_tags") == 0
|
||||||
|
assert count(app, "SELECT COUNT(*) FROM note_media") == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_deleting_media_removes_variants(app):
|
||||||
|
from starpunk.media import delete_media, save_media
|
||||||
|
with app.test_request_context():
|
||||||
|
media = save_media(png(), "b.png")
|
||||||
|
delete_media(media["id"])
|
||||||
|
assert count(app, "SELECT COUNT(*) FROM media_variants") == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_migration_018_removes_existing_orphans(app):
|
||||||
|
conn = sqlite3.connect(app.config["DATABASE_PATH"]) # foreign keys off here
|
||||||
|
conn.execute("INSERT INTO tags (name, display_name) VALUES ('t', 't')")
|
||||||
|
conn.execute("INSERT INTO note_tags (note_id, tag_id) VALUES (999, 1)")
|
||||||
|
conn.execute("INSERT INTO webmentions (source_url, target_url, target_note_id) "
|
||||||
|
"VALUES ('https://s.example/', 'http://localhost:5000/note/x', 999)")
|
||||||
|
conn.commit()
|
||||||
|
sql = (Path(__file__).parent.parent / "migrations" / "018_foreign_key_cleanup.sql").read_text()
|
||||||
|
conn.execute("BEGIN")
|
||||||
|
execute_in_transaction(conn, sql)
|
||||||
|
conn.commit()
|
||||||
|
assert conn.execute("SELECT COUNT(*) FROM note_tags").fetchone()[0] == 0
|
||||||
|
assert conn.execute("SELECT target_note_id FROM webmentions").fetchone()[0] is None
|
||||||
|
assert conn.execute("PRAGMA foreign_key_check").fetchall() == []
|
||||||
+41
-14
@@ -129,27 +129,54 @@ class TestValidation:
|
|||||||
|
|
||||||
|
|
||||||
class TestClientMetadata:
|
class TestClientMetadata:
|
||||||
def _response(self, content_type, body, links=None):
|
"""Client metadata is fetched through the SSRF-safe fetcher (v2.11.0)"""
|
||||||
response = MagicMock(status_code=200, text=body, url="https://quill.example/")
|
|
||||||
response.headers = {"Content-Type": content_type}
|
|
||||||
response.links = links or {}
|
|
||||||
response.json.return_value = __import__("json").loads(body) if "json" in content_type else None
|
|
||||||
return response
|
|
||||||
|
|
||||||
def test_json_metadata(self, app):
|
@staticmethod
|
||||||
|
def _serve(monkeypatch, handler, private=()):
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
def resolve(host, *args, **kwargs):
|
||||||
|
return [(2, 1, 6, "", ("10.0.0.5" if host in private else "93.184.216.34", 0))]
|
||||||
|
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
|
||||||
|
|
||||||
|
def test_json_metadata(self, app, monkeypatch):
|
||||||
|
import httpx
|
||||||
body = '{"client_id": "https://quill.example/", "redirect_uris": ["app://cb"]}'
|
body = '{"client_id": "https://quill.example/", "redirect_uris": ["app://cb"]}'
|
||||||
with app.test_request_context(), patch(
|
self._serve(monkeypatch, lambda r: httpx.Response(
|
||||||
"starpunk.webmention_ssrf.validate_url_for_fetch"
|
200, headers={"Content-Type": "application/json"}, text=body))
|
||||||
), patch("starpunk.indieauth.httpx.get", return_value=self._response("application/json", body)):
|
with app.test_request_context():
|
||||||
assert fetch_client_redirect_uris(CLIENT_ID) == ["app://cb"]
|
assert fetch_client_redirect_uris(CLIENT_ID) == ["app://cb"]
|
||||||
|
|
||||||
def test_html_link(self, app):
|
def test_html_link(self, app, monkeypatch):
|
||||||
|
import httpx
|
||||||
body = '<html><head><link rel="redirect_uri" href="/cb"></head></html>'
|
body = '<html><head><link rel="redirect_uri" href="/cb"></head></html>'
|
||||||
with app.test_request_context(), patch(
|
self._serve(monkeypatch, lambda r: httpx.Response(
|
||||||
"starpunk.webmention_ssrf.validate_url_for_fetch"
|
200, headers={"Content-Type": "text/html"}, text=body))
|
||||||
), patch("starpunk.indieauth.httpx.get", return_value=self._response("text/html", body)):
|
with app.test_request_context():
|
||||||
assert fetch_client_redirect_uris(CLIENT_ID) == ["https://quill.example/cb"]
|
assert fetch_client_redirect_uris(CLIENT_ID) == ["https://quill.example/cb"]
|
||||||
|
|
||||||
|
def test_link_header(self, app, monkeypatch):
|
||||||
|
import httpx
|
||||||
|
self._serve(monkeypatch, lambda r: httpx.Response(
|
||||||
|
200, headers={"Content-Type": "text/html", "Link": '</cb2>; rel="redirect_uri"'}, text="<p>hi</p>"))
|
||||||
|
with app.test_request_context():
|
||||||
|
assert fetch_client_redirect_uris(CLIENT_ID) == ["https://quill.example/cb2"]
|
||||||
|
|
||||||
|
def test_redirect_to_private_address_refused(self, app, monkeypatch):
|
||||||
|
import httpx
|
||||||
|
fetched = []
|
||||||
|
|
||||||
|
def handler(request):
|
||||||
|
fetched.append(request.headers["host"])
|
||||||
|
return httpx.Response(302, headers={"Location": "http://internal.example/"})
|
||||||
|
|
||||||
|
self._serve(monkeypatch, handler, private={"internal.example"})
|
||||||
|
with app.test_request_context():
|
||||||
|
assert fetch_client_redirect_uris(CLIENT_ID) == []
|
||||||
|
assert "internal.example" not in fetched
|
||||||
|
|
||||||
|
|
||||||
class TestCodesAndTokens:
|
class TestCodesAndTokens:
|
||||||
def test_full_exchange(self, app):
|
def test_full_exchange(self, app):
|
||||||
|
|||||||
+74
-74
@@ -9,7 +9,7 @@ Organized by model:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from datetime import datetime, timedelta
|
from datetime import datetime, timedelta, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from starpunk.models import Note, Session, Token, AuthState, split_title
|
from starpunk.models import Note, Session, Token, AuthState, split_title
|
||||||
from starpunk.utils import calculate_content_hash
|
from starpunk.utils import calculate_content_hash
|
||||||
@@ -68,8 +68,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -89,8 +89,8 @@ class TestNoteModel:
|
|||||||
slug="missing",
|
slug="missing",
|
||||||
file_path="notes/2024/11/missing.md",
|
file_path="notes/2024/11/missing.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -109,8 +109,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -130,8 +130,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -150,8 +150,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -168,8 +168,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -182,8 +182,8 @@ class TestNoteModel:
|
|||||||
slug="my-test-note",
|
slug="my-test-note",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=Path("/nonexistent"),
|
_data_dir=Path("/nonexistent"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -202,8 +202,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -222,8 +222,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -238,8 +238,8 @@ class TestNoteModel:
|
|||||||
slug="my-note",
|
slug="my-note",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=Path("/nonexistent"),
|
_data_dir=Path("/nonexistent"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -254,8 +254,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -289,8 +289,8 @@ class TestNoteModel:
|
|||||||
slug="my-note",
|
slug="my-note",
|
||||||
file_path="notes/2024/11/my-note.md",
|
file_path="notes/2024/11/my-note.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=Path("data"),
|
_data_dir=Path("data"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -303,8 +303,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=Path("data"),
|
_data_dir=Path("data"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -342,8 +342,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -362,8 +362,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -386,8 +386,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
content_hash=content_hash,
|
content_hash=content_hash,
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
@@ -410,8 +410,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
content_hash=content_hash,
|
content_hash=content_hash,
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
@@ -433,8 +433,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
content_hash=None,
|
content_hash=None,
|
||||||
_data_dir=tmp_path,
|
_data_dir=tmp_path,
|
||||||
)
|
)
|
||||||
@@ -449,8 +449,8 @@ class TestNoteModel:
|
|||||||
slug="test",
|
slug="test",
|
||||||
file_path="notes/2024/11/test.md",
|
file_path="notes/2024/11/test.md",
|
||||||
published=True,
|
published=True,
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
updated_at=datetime.utcnow(),
|
updated_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
content_hash="abc123",
|
content_hash="abc123",
|
||||||
_data_dir=Path("/nonexistent"),
|
_data_dir=Path("/nonexistent"),
|
||||||
)
|
)
|
||||||
@@ -499,8 +499,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
)
|
)
|
||||||
assert session.is_expired is False
|
assert session.is_expired is False
|
||||||
assert session.is_active is True
|
assert session.is_active is True
|
||||||
@@ -511,21 +511,21 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow() - timedelta(days=31),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=31),
|
||||||
expires_at=datetime.utcnow() - timedelta(days=1),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
|
||||||
)
|
)
|
||||||
assert session.is_expired is True
|
assert session.is_expired is True
|
||||||
assert session.is_active is False
|
assert session.is_active is False
|
||||||
|
|
||||||
def test_age(self):
|
def test_age(self):
|
||||||
"""Test age calculation"""
|
"""Test age calculation"""
|
||||||
created = datetime.utcnow() - timedelta(hours=2)
|
created = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(hours=2)
|
||||||
session = Session(
|
session = Session(
|
||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=created,
|
created_at=created,
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
)
|
)
|
||||||
age = session.age
|
age = session.age
|
||||||
# Age should be at least 2 hours (account for test execution time)
|
# Age should be at least 2 hours (account for test execution time)
|
||||||
@@ -537,8 +537,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
)
|
)
|
||||||
time_left = session.time_until_expiry
|
time_left = session.time_until_expiry
|
||||||
# Should be approximately 30 days
|
# Should be approximately 30 days
|
||||||
@@ -550,8 +550,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow() - timedelta(days=31),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=31),
|
||||||
expires_at=datetime.utcnow() - timedelta(days=1),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
|
||||||
)
|
)
|
||||||
time_left = session.time_until_expiry
|
time_left = session.time_until_expiry
|
||||||
assert time_left.total_seconds() < 0
|
assert time_left.total_seconds() < 0
|
||||||
@@ -562,8 +562,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
)
|
)
|
||||||
assert session.is_valid() is True
|
assert session.is_valid() is True
|
||||||
|
|
||||||
@@ -573,8 +573,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow() - timedelta(days=31),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=31),
|
||||||
expires_at=datetime.utcnow() - timedelta(days=1),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
|
||||||
)
|
)
|
||||||
assert session.is_valid() is False
|
assert session.is_valid() is False
|
||||||
|
|
||||||
@@ -584,8 +584,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="",
|
session_token="",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
)
|
)
|
||||||
assert session.is_valid() is False
|
assert session.is_valid() is False
|
||||||
|
|
||||||
@@ -595,8 +595,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="",
|
me="",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
)
|
)
|
||||||
assert session.is_valid() is False
|
assert session.is_valid() is False
|
||||||
|
|
||||||
@@ -606,8 +606,8 @@ class TestSessionModel:
|
|||||||
id=1,
|
id=1,
|
||||||
session_token="abc123",
|
session_token="abc123",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(days=30),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=30),
|
||||||
last_used_at=None,
|
last_used_at=None,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -721,7 +721,7 @@ class TestTokenModel:
|
|||||||
token = Token(
|
token = Token(
|
||||||
token="xyz789",
|
token="xyz789",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
expires_at=datetime.utcnow() + timedelta(days=90),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(days=90),
|
||||||
)
|
)
|
||||||
assert token.is_expired is False
|
assert token.is_expired is False
|
||||||
assert token.is_active is True
|
assert token.is_active is True
|
||||||
@@ -731,7 +731,7 @@ class TestTokenModel:
|
|||||||
token = Token(
|
token = Token(
|
||||||
token="xyz789",
|
token="xyz789",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
expires_at=datetime.utcnow() - timedelta(days=1),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
|
||||||
)
|
)
|
||||||
assert token.is_expired is True
|
assert token.is_expired is True
|
||||||
assert token.is_active is False
|
assert token.is_active is False
|
||||||
@@ -746,7 +746,7 @@ class TestTokenModel:
|
|||||||
token = Token(
|
token = Token(
|
||||||
token="xyz789",
|
token="xyz789",
|
||||||
me="https://alice.example.com",
|
me="https://alice.example.com",
|
||||||
expires_at=datetime.utcnow() - timedelta(days=1),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(days=1),
|
||||||
)
|
)
|
||||||
assert token.is_valid() is False
|
assert token.is_valid() is False
|
||||||
|
|
||||||
@@ -827,8 +827,8 @@ class TestAuthStateModel:
|
|||||||
"""Test is_expired returns False for active state"""
|
"""Test is_expired returns False for active state"""
|
||||||
auth_state = AuthState(
|
auth_state = AuthState(
|
||||||
state="random123",
|
state="random123",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(minutes=5),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=5),
|
||||||
)
|
)
|
||||||
assert auth_state.is_expired is False
|
assert auth_state.is_expired is False
|
||||||
assert auth_state.is_active is True
|
assert auth_state.is_active is True
|
||||||
@@ -837,19 +837,19 @@ class TestAuthStateModel:
|
|||||||
"""Test is_expired returns True for expired state"""
|
"""Test is_expired returns True for expired state"""
|
||||||
auth_state = AuthState(
|
auth_state = AuthState(
|
||||||
state="random123",
|
state="random123",
|
||||||
created_at=datetime.utcnow() - timedelta(minutes=10),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=10),
|
||||||
expires_at=datetime.utcnow() - timedelta(minutes=5),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=5),
|
||||||
)
|
)
|
||||||
assert auth_state.is_expired is True
|
assert auth_state.is_expired is True
|
||||||
assert auth_state.is_active is False
|
assert auth_state.is_active is False
|
||||||
|
|
||||||
def test_age(self):
|
def test_age(self):
|
||||||
"""Test age calculation"""
|
"""Test age calculation"""
|
||||||
created = datetime.utcnow() - timedelta(minutes=2)
|
created = datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=2)
|
||||||
auth_state = AuthState(
|
auth_state = AuthState(
|
||||||
state="random123",
|
state="random123",
|
||||||
created_at=created,
|
created_at=created,
|
||||||
expires_at=datetime.utcnow() + timedelta(minutes=3),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=3),
|
||||||
)
|
)
|
||||||
age = auth_state.age
|
age = auth_state.age
|
||||||
# Age should be at least 2 minutes
|
# Age should be at least 2 minutes
|
||||||
@@ -859,8 +859,8 @@ class TestAuthStateModel:
|
|||||||
"""Test validation succeeds for active state"""
|
"""Test validation succeeds for active state"""
|
||||||
auth_state = AuthState(
|
auth_state = AuthState(
|
||||||
state="random123",
|
state="random123",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(minutes=5),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=5),
|
||||||
)
|
)
|
||||||
assert auth_state.is_valid() is True
|
assert auth_state.is_valid() is True
|
||||||
|
|
||||||
@@ -868,8 +868,8 @@ class TestAuthStateModel:
|
|||||||
"""Test validation fails for expired state"""
|
"""Test validation fails for expired state"""
|
||||||
auth_state = AuthState(
|
auth_state = AuthState(
|
||||||
state="random123",
|
state="random123",
|
||||||
created_at=datetime.utcnow() - timedelta(minutes=10),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=10),
|
||||||
expires_at=datetime.utcnow() - timedelta(minutes=5),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) - timedelta(minutes=5),
|
||||||
)
|
)
|
||||||
assert auth_state.is_valid() is False
|
assert auth_state.is_valid() is False
|
||||||
|
|
||||||
@@ -877,8 +877,8 @@ class TestAuthStateModel:
|
|||||||
"""Test validation fails for empty state"""
|
"""Test validation fails for empty state"""
|
||||||
auth_state = AuthState(
|
auth_state = AuthState(
|
||||||
state="",
|
state="",
|
||||||
created_at=datetime.utcnow(),
|
created_at=datetime.now(timezone.utc).replace(tzinfo=None),
|
||||||
expires_at=datetime.utcnow() + timedelta(minutes=5),
|
expires_at=datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(minutes=5),
|
||||||
)
|
)
|
||||||
assert auth_state.is_valid() is False
|
assert auth_state.is_valid() is False
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""
|
||||||
|
Rate limits are shared by all server processes (v2.11.0)
|
||||||
|
|
||||||
|
Each gunicorn process used to keep its own counts, so with 4 workers a client
|
||||||
|
got four times the limit. Two SharedRateLimiter instances on one database
|
||||||
|
stand in for two processes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import time
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from starpunk.rate_limiter import SharedRateLimiter
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_processes_share_one_limit(app):
|
||||||
|
first = SharedRateLimiter(limit=3, window_seconds=60, prefix="t")
|
||||||
|
second = SharedRateLimiter(limit=3, window_seconds=60, prefix="t")
|
||||||
|
with app.app_context(), patch("starpunk.rate_limiter.time.time", return_value=6000.0):
|
||||||
|
results = [limiter.is_allowed("1.2.3.4") for limiter in (first, second, first, second)]
|
||||||
|
assert results == [True, True, True, False]
|
||||||
|
|
||||||
|
|
||||||
|
def test_keys_and_prefixes_are_separate(app):
|
||||||
|
a = SharedRateLimiter(limit=1, window_seconds=60, prefix="a")
|
||||||
|
b = SharedRateLimiter(limit=1, window_seconds=60, prefix="b")
|
||||||
|
with app.app_context(), patch("starpunk.rate_limiter.time.time", return_value=6000.0):
|
||||||
|
assert a.is_allowed("x") and a.is_allowed("y") and b.is_allowed("x")
|
||||||
|
assert not a.is_allowed("x")
|
||||||
|
|
||||||
|
|
||||||
|
def test_window_slides(app):
|
||||||
|
limiter = SharedRateLimiter(limit=2, window_seconds=60, prefix="s")
|
||||||
|
with app.app_context():
|
||||||
|
# Window 100 (6000-6059 s): two requests use the limit
|
||||||
|
with patch("starpunk.rate_limiter.time.time", return_value=6000.0):
|
||||||
|
assert limiter.is_allowed("k") and limiter.is_allowed("k")
|
||||||
|
assert not limiter.is_allowed("k")
|
||||||
|
# 6030 s is still window 100: still limited
|
||||||
|
with patch("starpunk.rate_limiter.time.time", return_value=6030.0):
|
||||||
|
assert not limiter.is_allowed("k")
|
||||||
|
# 6090 s is halfway through window 101: window 100 counts half (2 x 0.5 = 1)
|
||||||
|
with patch("starpunk.rate_limiter.time.time", return_value=6090.0):
|
||||||
|
assert limiter.is_allowed("k") # 1 + 0 < 2
|
||||||
|
assert not limiter.is_allowed("k") # 1 + 1 = 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_cleanup_removes_old_windows(app):
|
||||||
|
import sqlite3
|
||||||
|
limiter = SharedRateLimiter(limit=5, window_seconds=60, prefix="c")
|
||||||
|
with app.app_context():
|
||||||
|
with patch("starpunk.rate_limiter.time.time", return_value=6000.0):
|
||||||
|
limiter.is_allowed("k")
|
||||||
|
with patch("starpunk.rate_limiter.time.time", return_value=9000.0):
|
||||||
|
limiter.cleanup()
|
||||||
|
rows = sqlite3.connect(app.config["DATABASE_PATH"]).execute("SELECT COUNT(*) FROM rate_limits").fetchone()[0]
|
||||||
|
assert rows == 0
|
||||||
@@ -99,7 +99,8 @@ class FakeWeb:
|
|||||||
self.fetched = []
|
self.fetched = []
|
||||||
|
|
||||||
def __call__(self, request):
|
def __call__(self, request):
|
||||||
url = str(request.url)
|
# Pinned requests go to the checked IP; the Host header names the site
|
||||||
|
url = f"{request.url.scheme}://{request.headers['host']}{request.url.raw_path.decode()}"
|
||||||
self.fetched.append(url)
|
self.fetched.append(url)
|
||||||
status, headers, body = self.pages.get(url, (404, {}, b"nope"))
|
status, headers, body = self.pages.get(url, (404, {}, b"nope"))
|
||||||
return httpx.Response(status, headers=headers, content=body)
|
return httpx.Response(status, headers=headers, content=body)
|
||||||
@@ -112,11 +113,20 @@ def web():
|
|||||||
|
|
||||||
def routed(pds, web):
|
def routed(pds, web):
|
||||||
return httpx.MockTransport(
|
return httpx.MockTransport(
|
||||||
lambda r: pds(r) if r.url.host in ("bsky.example", "pds.example") else web(r))
|
lambda r: pds(r) if r.headers.get("host", r.url.host) in ("bsky.example", "pds.example") else web(r))
|
||||||
|
|
||||||
|
|
||||||
|
def resolve(host, *args, **kwargs):
|
||||||
|
"""Every test host is public, except internal.example"""
|
||||||
|
ip = "10.0.0.5" if host == "internal.example" else "93.184.216.34"
|
||||||
|
return [(2, 1, 6, "", (ip, 0))]
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def app(tmp_path, pds, web):
|
def app(tmp_path, pds, web, monkeypatch):
|
||||||
|
# Pages and images for link cards go through the shared SSRF-safe fetcher (v2.11.0)
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", routed(pds, web))
|
||||||
|
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", resolve)
|
||||||
data = tmp_path / "data"
|
data = tmp_path / "data"
|
||||||
app = create_app({
|
app = create_app({
|
||||||
"TESTING": True,
|
"TESTING": True,
|
||||||
@@ -328,12 +338,7 @@ class TestLinkCards:
|
|||||||
|
|
||||||
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
|
def test_redirect_to_private_address_is_not_followed(self, app, pds, web, monkeypatch):
|
||||||
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
|
web.pages[ARTICLE] = (302, {"location": "http://internal.example/admin"}, b"")
|
||||||
monkeypatch.setattr("starpunk.micropub._public_address", lambda host: host != "internal.example")
|
publish(app, content=f"See {ARTICLE}") # internal.example resolves to 10.0.0.5
|
||||||
app.config["TESTING"] = False
|
|
||||||
try:
|
|
||||||
publish(app, content=f"See {ARTICLE}")
|
|
||||||
finally:
|
|
||||||
app.config["TESTING"] = True
|
|
||||||
assert "http://internal.example/admin" not in web.fetched
|
assert "http://internal.example/admin" not in web.fetched
|
||||||
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE
|
assert only_record(pds)["embed"]["external"]["title"] == ARTICLE
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user