Files
StarPunk/docs/standards/testing-checklist.md
T
PhilandClaude Opus 5.5 ceef048003 docs: documentation review for v2.14.0
Architecture, README, standards, CLAUDE.md, settings reference, indexes,
backlog, release plans and CHANGELOG checked against the code at v2.14.0:
backfeed, notifications, counts line and web actions documented; test
policy (changed area locally, full suite in CI); stale versions, statuses
and links fixed; legacy and superseded docs moved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:51:57 -07:00

5.6 KiB

Testing Checklist

Status: Active Last Updated: 2026-11-10 (v2.14.0)

What to test, how to run the tests, and what CI does with them. Setup and the environment variables the tests need are in development-setup.md.

How tests run

Locally

SESSION_SECRET=test uv run pytest tests/test_websub.py tests/test_websub_hardening.py  # the area you changed
SESSION_SECRET=test uv run pytest                                            # everything (CI does this)

Locally, run the tests for the area you changed. The full suite, including the 5-minute tests/test_migration_race_condition.py, runs in CI. For wide changes (migrations, config.py, create_app, tests/conftest.py) you can build the CI test stage first: docker build --target test -f Containerfile .

In CI (.gitea/workflows/build-container.yml), on every pushed vX.Y.Z tag (or a manual run):

  1. docker buildx build --target test -f Containerfile . builds the Containerfile's test stage: the runtime dependencies plus pytest and pytest-mock, with SESSION_SECRET=ci-test-secret, SITE_URL=http://localhost:5000 and ADMIN_ME=https://test.example.com, then python -m pytest -q -p no:cacheprovider tests.
  2. Only if that succeeds is the runtime image built and pushed as :vX.Y.Z and :latest.

So a failing test blocks the release image, not the merge or the tag; CI is where the full suite runs. If it fails, fix it in the next PATCH (see git-branching-strategy.md, step 4). To reproduce CI locally: docker build --target test -f Containerfile . (pytest-cov is not installed in that stage, so don't depend on it in tests).

Every change

  • Tests for the changed area pass locally (CI runs the full suite)
  • New behaviour has tests; a bug fix has a test that failed before the fix
  • Tests use the app / client fixtures from tests/conftest.py (temporary data directory, TESTING=True); nothing touches ./data
  • No test makes real network requests (see below)
  • Schema changes: a numbered migration in migrations/, plus a test that it applies to an existing database (tests/test_migrations.py)

Conventions

  • Where: one file per module or feature (tests/test_websub.py). Work driven by a spec review or an incident gets its own file named for the release, with a docstring saying what it covers, e.g. test_micropub_v290.py, test_microformats_v280.py, test_websub_hardening.py ("one test per finding of the v2.10.0 review").

  • Outbound HTTP: everything StarPunk fetches from or sends to other servers goes through starpunk/webmention_ssrf.py. Tests replace its transport and DNS instead of mocking callers:

    monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
    monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
    

    (see the web fixture in tests/test_websub.py). Test the private address rejection by making getaddrinfo return e.g. 10.0.0.2.

  • Background workers don't run under TESTING=True. Drive them directly, e.g. WebSubWorker(app) inside app.app_context() (run_worker in tests/test_websub.py).

  • Crossposting targets are configured per test with app config overrides and mocked HTTP; never with real credentials.

  • Concurrency fixes need a test that reproduces the race with real processes or threads (tests/test_migration_concurrency.py).

New or changed protocol work

StarPunk implements Micropub, IndieAuth, Webmention, WebSub and microformats2. For changes to any of them:

  • Read the W3C Recommendation (or the living spec) and cite section numbers in the ADR and in test names or docstrings
  • Test each MUST that the change touches, including the error responses (status codes, error values, headers) and not only the happy path
  • Test discovery: Link headers and <link> elements still advertise the endpoint on every page and feed that should have them
  • Test what a strict client sends: form-encoded and JSON (Micropub), case and percent-encoding variants (WebSub topics), relative URLs (Webmention sources)
  • Test that requests to user-supplied URLs are refused for private addresses and time out
  • Rendered markup: parse the page with mf2py and assert on the parsed properties (tests/test_microformats*.py), not on raw HTML strings
  • Manual check against the live site after deploy with the external validators below

Validation tools (manual, after deploy)

Protocol Tool
Micropub micropub.rocks, a real client (Quill, Indigenous)
IndieAuth Sign in to a Micropub client; indieauth.rocks
Webmention webmention.rocks
WebSub websub.rocks (publisher and hub tests)
Microformats2 pin13.net/mf2, IndieWebify.me
Feeds W3C Feed Validator (RSS, Atom), JSON Feed validator
HTML W3C HTML validator

Specifications