Files
StarPunk/docs/standards/testing-checklist.md
T
PhilandClaude Opus 5.5 0716984958 docs: Bring all documentation up to date for v2.10.1
- architecture/: overview, technology stack, auth (OIDC + built-in
  IndieAuth), syndication (POSSE) rewritten; new database.md,
  deployment.md and indieweb-protocols.md; legacy stack marked historical
- decisions/: reconstructed ADR-060 and ADR-063; 31 status corrections
  (superseded auth ADRs, proposed -> accepted); INDEX rebuilt
- design/: reconstructed the v1.6.0 documents the webmention code cites;
  reports for 18 v2 releases without a design folder; INDEX rebuilt
- projectplan/: BACKLOG rewritten (shipped items removed, open issues from
  the reviews added); RELEASE.md for v2.0.0-v2.10.1; INDEX
- standards/: development setup (uv, tests, DEV_MODE), testing (CI test
  stage), git and release flow, versioning, coding standards, cookies
- README, CLAUDE.md, .env.example (every setting the code reads, with
  defaults) brought up to date
- 137 broken links repaired (old absolute paths, moved files); stale ADR
  references in code comments corrected (comments only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:40:19 -07:00

5.3 KiB

Testing Checklist

Status: Active Last Updated: 2026-11-03 (v2.10.1)

What to test, how to run the tests, and what CI does with them. Setup and the environment variables the tests need are in development-setup.md.

How tests run

Locally

SESSION_SECRET=test uv run pytest                                            # everything
SESSION_SECRET=test uv run pytest --ignore=tests/test_migration_race_condition.py  # fast loop

tests/test_migration_race_condition.py takes about 5 minutes; skip it while iterating, never before a release.

In CI (.gitea/workflows/build-container.yml), on every pushed vX.Y.Z tag (or a manual run):

  1. docker buildx build --target test -f Containerfile . builds the Containerfile's test stage: the runtime dependencies plus pytest and pytest-mock, with SESSION_SECRET=ci-test-secret, SITE_URL=http://localhost:5000 and ADMIN_ME=https://test.example.com, then python -m pytest -q -p no:cacheprovider tests.
  2. Only if that succeeds is the runtime image built and pushed as :vX.Y.Z and :latest.

So a failing test blocks the release image; it does not block the merge or the tag. Run the full suite before merging (see git-branching-strategy.md). To reproduce CI locally: docker build --target test -f Containerfile . (pytest-cov is not installed in that stage, so don't depend on it in tests).

Every change

  • Full suite passes locally
  • New behaviour has tests; a bug fix has a test that failed before the fix
  • Tests use the app / client fixtures from tests/conftest.py (temporary data directory, TESTING=True); nothing touches ./data
  • No test makes real network requests (see below)
  • Schema changes: a numbered migration in migrations/, plus a test that it applies to an existing database (tests/test_migrations.py)

Conventions

  • Where: one file per module or feature (tests/test_websub.py). Work driven by a spec review or an incident gets its own file named for the release, with a docstring saying what it covers, e.g. test_micropub_v290.py, test_microformats_v280.py, test_websub_hardening.py ("one test per finding of the v2.10.0 review").

  • Outbound HTTP: everything StarPunk fetches from or sends to other servers goes through starpunk/webmention_ssrf.py. Tests replace its transport and DNS instead of mocking callers:

    monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
    monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
    

    (see the web fixture in tests/test_websub.py). Test the private address rejection by making getaddrinfo return e.g. 10.0.0.2.

  • Background workers don't run under TESTING=True. Drive them directly, e.g. WebSubWorker(app) inside app.app_context() (run_worker in tests/test_websub.py).

  • Crossposting targets are configured per test with app config overrides and mocked HTTP; never with real credentials.

  • Concurrency fixes need a test that reproduces the race with real processes or threads (tests/test_migration_concurrency.py).

New or changed protocol work

StarPunk implements Micropub, IndieAuth, Webmention, WebSub and microformats2. For changes to any of them:

  • Read the W3C Recommendation (or the living spec) and cite section numbers in the ADR and in test names or docstrings
  • Test each MUST that the change touches, including the error responses (status codes, error values, headers) and not only the happy path
  • Test discovery: Link headers and <link> elements still advertise the endpoint on every page and feed that should have them
  • Test what a strict client sends: form-encoded and JSON (Micropub), case and percent-encoding variants (WebSub topics), relative URLs (Webmention sources)
  • Test that requests to user-supplied URLs are refused for private addresses and time out
  • Rendered markup: parse the page with mf2py and assert on the parsed properties (tests/test_microformats*.py), not on raw HTML strings
  • Manual check against the live site after deploy with the external validators below

Validation tools (manual, after deploy)

Protocol Tool
Micropub micropub.rocks, a real client (Quill, Indigenous)
IndieAuth Sign in to a Micropub client; indieauth.rocks
Webmention webmention.rocks
WebSub websub.rocks (publisher and hub tests)
Microformats2 pin13.net/mf2, IndieWebify.me
Feeds W3C Feed Validator (RSS, Atom), JSON Feed validator
HTML W3C HTML validator

Specifications