Compare commits

...
10 Commits
Author SHA1 Message Date
Phil afef1fc9a5 Merge branch 'fix/v2.12.1-category-unused-tags' for v2.12.1 release
Build Container / build (push) Successful in 8m3s
2026-11-10 11:28:42 -07:00
PhilandClaude Opus 5.5 b450c0c648 fix: q=category lists only tags on existing posts (v2.12.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:28:32 -07:00
Phil 13936749de Merge branch 'chore/remove-metrics': remove metrics dashboard (v2.12.0)
Build Container / build (push) Successful in 7m53s
2026-11-03 16:07:48 -07:00
PhilandClaude Opus 5.5 742e2e3ec3 Remove the metrics dashboard and monitoring package (v2.12.0)
The in-memory metrics system covered one gunicorn process at a time, was
lost on restart and was never used. Remove /admin/metrics-dashboard,
/admin/metrics, starpunk/monitoring, the HTTP/query timing, memory monitor,
feed counters, METRICS_* settings and psutil. Keep /health, /admin/health
(without its metrics block), pool stats and the feed cache. ADR-072.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 16:07:47 -07:00
Phil d6a916ba7d Merge branch 'chore/license-mit': MIT license 2026-11-03 13:43:39 -07:00
PhilandClaude Opus 5.5 e654dd84de chore: License StarPunk under MIT
MIT suits software better than CC BY (Creative Commons advises against its
licenses for code). Copyright Phil Skentelbery 2025-2026.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:43:39 -07:00
Phil abb75f080e Merge branch 'chore/license-cc-by': CC BY 4.0 license 2026-11-03 13:43:17 -07:00
PhilandClaude Opus 5.5 d5157a6059 chore: License StarPunk under CC BY 4.0
Replace the CC0 dedication with the Creative Commons Attribution 4.0
International license (official legal code), credited to Phil Skentelbery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:43:17 -07:00
Phil 393228002b Merge branch 'chore/license-cc0': CC0 license 2026-11-03 13:42:39 -07:00
PhilandClaude Opus 5.5 874465e103 chore: Dedicate StarPunk to the public domain (CC0 1.0)
Replace the placeholder MIT notice ("Copyright (c) 2024 [Your Name]") with
the official CC0 1.0 Universal legal code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:42:39 -07:00
42 changed files with 215 additions and 2974 deletions

No files matched your search

+1 -11
View File
@@ -199,19 +199,9 @@ THREADS_USERNAME=
# SYNDICATION_POLL_INTERVAL=10
# =============================================================================
# MONITORING
# DEBUGGING
# =============================================================================
# Metrics shown on /admin/metrics-dashboard (default: true)
# METRICS_ENABLED=true
# Queries slower than this many seconds are logged (default: 1.0)
# METRICS_SLOW_QUERY_THRESHOLD=1.0
# Fraction of operations recorded, 0.0-1.0 (default: 1.0)
# METRICS_SAMPLING_RATE=1.0
# METRICS_BUFFER_SIZE=1000
# Seconds between memory samples (default: 30)
# METRICS_MEMORY_INTERVAL=30
# Keep uploads that fail processing in DATA_PATH/debug for diagnosis
# (default: false); cleaned up by age (default: 7 days) and total size
# (default: 100 MB)
+17
View File
@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.12.1] - 2026-11-10
### Fixed
- Micropub `q=category` (the tag list apps use for autocomplete) only lists tags on posts that still exist. It used to include tags that an edit had removed from every post, and tags only used on deleted posts. Drafts still count; a deleted post's tags come back if it is undeleted
## [2.12.0] - 2026-11-03
The metrics dashboard is gone (ADR-072).
### Removed
- The admin metrics dashboard (`/admin/metrics-dashboard`, the "Metrics" admin link) and its JSON endpoint `/admin/metrics`. Its numbers were kept in memory per server process, so they covered a quarter of the traffic and vanished on every restart
- Everything that only fed it: request and database-query timing, the memory monitor thread, feed-generation counters (`starpunk/monitoring/`), and the `X-Request-ID` response header. Slow queries are no longer logged
- The `METRICS_ENABLED`, `METRICS_SLOW_QUERY_THRESHOLD`, `METRICS_SAMPLING_RATE`, `METRICS_BUFFER_SIZE` and `METRICS_MEMORY_INTERVAL` settings (ignored if still set), and the `psutil` dependency
- The `metrics` section of `/admin/health`; its database, pool, filesystem and disk checks are unchanged, as is `/health`
## [2.11.0] - 2026-11-03
Maintenance release: dependency upgrades and the High-priority backlog items.
+1 -1
View File
@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2024 [Your Name]
Copyright (c) 2025-2026 Phil Skentelbery
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+3 -2
View File
@@ -41,7 +41,7 @@ what changed in each release is in [CHANGELOG.md](CHANGELOG.md).
**Operations**
- Migrations applied automatically at startup
- `/health` endpoint, admin metrics dashboard, structured logs
- `/health` endpoint, `/admin/health` diagnostics, structured logs
- Fetches of user-supplied URLs are limited to public addresses, size and
time
@@ -113,4 +113,5 @@ up regularly, and always before upgrading to a release with a migration.
## License
MIT, see [LICENSE](LICENSE).
[MIT](LICENSE): use, change and share it for anything, including
commercially, as long as the copyright and license notice stay with it.
+1 -2
View File
@@ -218,8 +218,7 @@ server process enforces one shared limit. Old windows are pruned as it goes.
timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the
`create_app()` config dict). Connections use `sqlite3.Row` rows,
`check_same_thread=False`, `PRAGMA journal_mode=WAL` and (since v2.11.0)
`PRAGMA foreign_keys=ON`, so `ON DELETE CASCADE`/`SET NULL` run, and are wrapped
in `MonitoredConnection` for query timing when metrics are on. `get_db()`
`PRAGMA foreign_keys=ON`, so `ON DELETE CASCADE`/`SET NULL` run. `get_db()`
stores one connection in `flask.g` per request or app context; on teardown
it goes back to the pool, rolled back first if a transaction was left
open.
+3 -6
View File
@@ -235,15 +235,12 @@ starts only if at least one is.
| `THREADS_MAX_MEDIA` | `4` | |
| `SYNDICATION_POLL_INTERVAL` | `10` | Seconds |
### Metrics and debugging
### Debugging
The `METRICS_*` variables were removed in v2.12.0 and are ignored if set.
| Variable | Default | Notes |
|----------|---------|-------|
| `METRICS_ENABLED` | `true` | HTTP/database metrics and the memory monitor |
| `METRICS_SLOW_QUERY_THRESHOLD` | `1.0` | Seconds |
| `METRICS_SAMPLING_RATE` | `1.0` | |
| `METRICS_BUFFER_SIZE` | `1000` | |
| `METRICS_MEMORY_INTERVAL` | `30` | Seconds |
| `DEBUG_SAVE_FAILED_UPLOADS` | `false` | Keep failed uploads in `debug/` |
| `DEBUG_FILE_MAX_AGE_DAYS` | `7` | |
| `DEBUG_FILE_MAX_SIZE_MB` | `100` | |
+6 -15
View File
@@ -43,8 +43,6 @@ Key decisions:
[ADR-058](../decisions/ADR-058-image-optimization-strategy.md) (media),
[ADR-034](../decisions/ADR-034-full-text-search.md) (search),
[ADR-052](../decisions/ADR-052-configuration-system-architecture.md) (configuration),
[ADR-053](../decisions/ADR-053-performance-monitoring-strategy.md) and
[ADR-060](../decisions/ADR-060-production-hotfix-metrics-dashboard.md) (metrics),
[ADR-054](../decisions/ADR-054-structured-logging-architecture.md) (logging),
[ADR-055](../decisions/ADR-055-error-handling-philosophy.md) (errors).
@@ -65,7 +63,6 @@ Key decisions:
│ webmention worker works its queues only while holding a lock │
│ websub worker works its queues only while holding a lock │
│ syndication worker claims rows with a lease in SQLite │
│ memory monitor per-process metrics │
└───────────────┬──────────────────────────────┬─────────────────────┘
│ │
┌───────▼────────┐ ┌──────────▼──────────────────┐
@@ -86,7 +83,7 @@ Key decisions:
validates configuration, configures logging, cleans up old debug and
temporary upload files, creates the schema and runs migrations (under a
lock, see [database.md](database.md)), creates the connection pool,
installs the HTTP metrics middleware, configures the feed cache and rate
configures the feed cache and rate
limiters, fills the FTS index if it is empty, registers blueprints, error
handlers and template helpers, and starts the background threads.
3. `before_request` assigns a correlation ID (included in every log line) and,
@@ -94,8 +91,7 @@ Key decisions:
4. The blueprint handler runs. Database access goes through `get_db()`, which
hands out a pooled, request-scoped SQLite connection.
5. `after_request` hooks add `Link: </webmention>; rel="webmention"` to HTML
responses, WebSub `Link rel=hub` / `rel=self` headers to topic responses,
and request metrics with an `X-Request-ID` header.
responses, WebSub `Link rel=hub` / `rel=self` headers to topic responses.
6. Errors are rendered by the central handlers in `starpunk/errors.py`
(JSON for Micropub, HTML pages from `templates/<code>.html` elsewhere).
@@ -141,7 +137,6 @@ wait for outbound network calls; the workers make them later.
| `errors.py` | Central error handlers |
| `feeds/` | `rss.py` (feedgen), `atom.py`, `json_feed.py`, `opml.py`, `negotiation.py` (Accept header), `cache.py` (LRU + TTL feed cache with ETags) |
| `feed.py` | Backward-compatibility re-export of `feeds.rss` |
| `monitoring/` | Metrics buffer, HTTP middleware, database timing wrapper, memory monitor, business metrics |
| `database/` | `schema.py` (`INITIAL_SCHEMA_SQL`), `init.py` (`init_db`), `pool.py` (connection pool) |
| `migrations.py` | Migration runner for `migrations/NNN_*.sql` |
@@ -196,7 +191,7 @@ Everything works without JavaScript.
| `/admin/webmentions`, `/admin/webmentions/<id>`, `…/action`, `…/bulk-action` | Webmention moderation |
| `/admin/tokens`, `/admin/tokens/<id>/revoke` | Issued IndieAuth tokens |
| `/admin/syndication/<id>/retry` | Retry a failed crosspost |
| `/admin/metrics-dashboard`, `/admin/metrics`, `/admin/health` | Monitoring (HTML dashboard, JSON metrics, full diagnostics) |
| `/admin/health` | Full diagnostics (JSON): database, pool statistics, filesystem, disk |
| `/health` | Public liveness check (`{"status": "ok", "version": …}`); `?detailed=true` requires login |
## Background workers
@@ -210,7 +205,6 @@ coordinated through the data volume:
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries |
| WebSub worker (`websub_worker.py`) | `WEBSUB_ENABLED` and not testing | Same pattern with `websub-worker.lock` | Verifies subscription intents, delivers topic content (signed with `X-Hub-Signature` when the subscriber gave a secret), retries, expires leases; pings an external hub instead if `WEBSUB_HUB_URL` is set |
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts |
| Memory monitor (`monitoring/memory.py`) | `METRICS_ENABLED` and not testing | None (per-process metrics) | Samples process memory every `METRICS_MEMORY_INTERVAL` seconds |
All workers run inside an app context, are daemon threads (they end with the
process) and sleep for their poll interval when there is nothing to do. Under
@@ -326,14 +320,11 @@ their credentials are set. The full list with defaults is in
- Logs go to stdout and `logs/starpunk.log` (10 MB × 10 files); each line
carries the request's correlation ID.
- Metrics (`monitoring/`): HTTP request timings, database query timings
(with a slow-query threshold), memory, pool statistics and business
counters, in an in-memory buffer per process, shown at
`/admin/metrics-dashboard` and `/admin/metrics`.
- `/health` reports status, version and webmention-worker statistics;
`/health?detailed=true` checks database, filesystem and disk space;
`/admin/health` adds metrics, pool statistics and system information.
`/admin/health` adds pool statistics and filesystem details. (The metrics
dashboard was removed in v2.12.0, see ADR-072.)
---
**Last Updated**: 2026-11-03 (v2.10.1)
**Last Updated**: 2026-11-03 (v2.12.0)
-1
View File
@@ -47,7 +47,6 @@ vulnerabilities (`pip-audit`); check again before each release.
| python-dotenv | `1.2.*` | Loading `.env` |
| Pillow | `13.0.*` | Image validation, resizing and variants |
| pillow-heif | `1.8.*` | HEIC/HEIF uploads (iPhone photos) |
| psutil | `7.2.*` | Memory monitor and system metrics |
`markupsafe` comes with Flask/Jinja2.
@@ -1,7 +1,9 @@
# ADR-053: Performance Monitoring Strategy
## Status
Accepted
Superseded by ADR-072 (was: Accepted, v1.1.1)
> The metrics system was removed in v2.12.0. (status updated 2026-11-03)
## Context
StarPunk v1.1.1 introduces performance monitoring to help operators understand system behavior in production. Currently, we have no visibility into:
@@ -2,7 +2,9 @@
## Status
Accepted (v1.1.1-rc.2, 2025-11-25)
Superseded by ADR-072 (was: Accepted, v1.1.1-rc.2, 2025-11-25)
> The metrics dashboard was removed in v2.12.0. (status updated 2026-11-03)
> Reconstructed on 2026-11-03 from code, changelog and related documents;
> the original was lost. The original
@@ -0,0 +1,51 @@
# ADR-072: Remove the metrics dashboard
## Status
Accepted (v2.12.0, 2026-11-03). Supersedes ADR-053 and ADR-060.
## Context
v1.1.1 and v1.1.2 added built-in performance monitoring (ADR-053): HTTP
request timing middleware, a wrapper on every database connection that
timed queries, a memory-sampling thread, feed-generation counters, an
in-memory buffer of the results, and an HTML dashboard at
`/admin/metrics-dashboard` with a JSON twin at `/admin/metrics` (ADR-060
covered adapting the data for the dashboard template).
In practice it told us little:
- The buffer lives in memory in each of the four gunicorn processes, so the
dashboard showed whichever process served the request, and everything was
lost on restart or deploy.
- Nothing in it was ever used to find or fix a problem. Logs (with
correlation IDs), `/health` and `/admin/health` were.
- It cost something on every request and every query, kept a thread per
process, and needed `psutil`, about 1,300 lines of code and three test
files.
## Decision
Remove the dashboard and everything that only fed it: the `starpunk/monitoring`
package, the HTTP metrics middleware (including its `X-Request-ID`
response header), `MonitoredConnection`, the memory monitor, the
feed-generation tracking calls, `/admin/metrics-dashboard`,
`/admin/metrics`, the `METRICS_*` settings and the `psutil` dependency.
Keep:
- `/health` and `/health?detailed=true`;
- `/admin/health`, without its `metrics` section (database, pool
statistics, filesystem and disk checks stay);
- connection pool statistics (`get_pool_stats`) and the feed cache, which
do work of their own;
- structured logs with correlation IDs (ADR-054).
## Consequences
- Less code and one fewer dependency; no timing work on each request or
query; one fewer thread per process.
- Slow database queries are no longer logged. If that is ever needed, a
query timer can be added at the pool level, or an external tool used.
- `METRICS_*` environment variables are ignored if still set.
- Bookmarks to `/admin/metrics-dashboard` now get a 404.
+6 -5
View File
@@ -5,8 +5,8 @@ records one decision: its context, what was decided, and the consequences.
Superseded ADRs are kept as history; their Status line names the ADR that
replaced them.
**Total ADR files**: 67 (66 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-071, with two files numbered ADR-012). See [Numbering](#numbering).
**Total ADR files**: 68 (67 numbers: ADR-001 to ADR-044 and ADR-050 to
ADR-072, with two files numbered ADR-012). See [Numbering](#numbering).
Status values: **Accepted** (in force), **Superseded** (replaced by a later
ADR), **Deprecated** (withdrawn, nothing replaces it; none at present),
@@ -101,10 +101,9 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
|---|---|---|---|
| [012](ADR-012-http-error-handling-policy.md) | HTTP Error Handling Policy | Accepted | Correct HTTP status codes (e.g. 404 for missing notes) across routes. |
| [052](ADR-052-configuration-system-architecture.md) | Configuration System Architecture | Accepted | Environment variables with defaults, loaded in one module. |
| [053](ADR-053-performance-monitoring-strategy.md) | Performance Monitoring Strategy | Accepted | Built-in, standard-library performance metrics (v1.1.1). |
| [054](ADR-054-structured-logging-architecture.md) | Structured Logging Architecture | Accepted | Rotating log files with correlation IDs (v1.1.1). |
| [055](ADR-055-error-handling-philosophy.md) | Error Handling Philosophy | Accepted | Layered handling: graceful degradation, clear user messages, detailed logs. |
| [060](ADR-060-production-hotfix-metrics-dashboard.md) | Route Adapter for the Metrics Dashboard | Accepted | Adapt monitoring data to the dashboard template in the route (v1.1.1-rc.2). *Reconstructed.* |
| [072](ADR-072-remove-metrics-dashboard.md) | Remove the metrics dashboard | Accepted | Drop the in-memory metrics system and dashboard; keep `/health`, `/admin/health` and logs (v2.12.0). |
See also [deployment](../architecture/deployment.md).
@@ -117,6 +116,8 @@ designs of v0.x–v1.6.0, all replaced by ADR-064 in v2.0.0.
|---|---|---|---|
| [005](ADR-005-indielogin-authentication.md) | IndieLogin Authentication Integration | Superseded by 064 | Admin login through indielogin.com. |
| [007](ADR-007-slug-generation-algorithm.md) | Slug Generation Algorithm | Superseded by 062 | Slugs from the first words of the note. |
| [053](ADR-053-performance-monitoring-strategy.md) | Performance Monitoring Strategy | Superseded by 072 | Built-in, standard-library performance metrics (v1.1.1). |
| [060](ADR-060-production-hotfix-metrics-dashboard.md) | Route Adapter for the Metrics Dashboard | Superseded by 072 | Adapt monitoring data to the dashboard template in the route (v1.1.1-rc.2). *Reconstructed.* |
| [016](ADR-016-indieauth-client-discovery.md) | IndieAuth Client Discovery Mechanism | Superseded by 019 | h-app markup for client discovery. |
| [017](ADR-017-oauth-client-metadata-document.md) | OAuth Client ID Metadata Document Implementation | Superseded by 019 | OAuth client metadata endpoint. |
| [018](ADR-018-indieauth-detailed-logging.md) | IndieAuth Detailed Logging Strategy | Superseded by 064 | Redacted HTTP logging of the IndieAuth login flow. |
@@ -166,7 +167,7 @@ was carried out and is not itself superseded.
## How to create a new ADR
1. Use the next number after the highest existing one (currently ADR-072).
1. Use the next number after the highest existing one (currently ADR-073).
2. Name it `ADR-NNN-brief-descriptive-title.md`.
3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071):
+9 -1
View File
@@ -21,7 +21,7 @@ is in `../architecture/`; decisions are in `../decisions/` (ADRs).
## Contents
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [hotfix](#hotfix)
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [v2.12.0](#v2120) · [v2.12.1](#v2121) · [hotfix](#hotfix)
## v1.0.0
@@ -414,6 +414,14 @@ OIDC admin login and built-in IndieAuth server.
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) - Dependency upgrades (Python 3.14), foreign keys, CSRF, fetchers, debug default, shared rate limits
## v2.12.0
- [2026-11-03-remove-metrics.md](v2.12.0/2026-11-03-remove-metrics.md) - Metrics dashboard and `starpunk/monitoring` removed (ADR-072)
## v2.12.1
- [2026-11-10-category-unused-tags.md](v2.12.1/2026-11-10-category-unused-tags.md) - q=category lists only tags on posts that exist
## hotfix
Not a release folder.
@@ -0,0 +1,28 @@
# v2.12.0: Remove the metrics dashboard
Decision: [ADR-072](../../decisions/ADR-072-remove-metrics-dashboard.md).
## Removed
| What | Where |
|---|---|
| Metrics buffer, HTTP middleware, database timing wrapper, memory monitor, business metrics | `starpunk/monitoring/` (deleted) |
| HTTP middleware and memory monitor start-up | `starpunk/__init__.py` |
| `MonitoredConnection` wrapping, `slow_query_threshold` and `metrics_enabled` arguments | `starpunk/database/pool.py` |
| `track_feed_generated` calls and their timing code | `starpunk/feeds/atom.py`, `json_feed.py`, `rss.py` |
| `/admin/metrics-dashboard`, `/admin/metrics`, `transform_metrics_for_template`, the `metrics` block of `/admin/health` | `starpunk/routes/admin.py` |
| Dashboard template and the admin nav link | `templates/admin/metrics_dashboard.html`, `templates/admin/base.html` |
| `METRICS_ENABLED`, `METRICS_SLOW_QUERY_THRESHOLD`, `METRICS_SAMPLING_RATE`, `METRICS_BUFFER_SIZE`, `METRICS_MEMORY_INTERVAL` | `starpunk/config.py`, `.env.example` |
| `psutil` | `requirements.txt` |
| Tests | `tests/test_monitoring.py`, `tests/test_monitoring_feed_statistics.py`, `tests/test_admin_feed_statistics.py` |
## Kept
`/health`, `/health?detailed=true`, `/admin/health` (database, pool,
filesystem, disk), pool statistics, the feed cache and its own hit/miss
counters, and correlation IDs in logs.
## Upgrade
No migration. `METRICS_*` variables can be deleted from the environment;
they are ignored if left.
@@ -0,0 +1,28 @@
# v2.12.1: q=category lists only tags in use
## Problem
`q=category` (added in v2.4.0) is what Micropub clients use for tag
autocomplete. It read every row of `tags` with a `LEFT JOIN` on
`note_tags`, so it also returned:
- tags no post uses any more (a `tags` row stays after an edit removes
the tag from its last post), and
- tags only used on soft-deleted posts (`note_tags` rows stay so that
undelete can restore them).
Clients then offered tags that appear nowhere on the site.
## Fix
`_all_categories()` in `starpunk/micropub.py` inner-joins `note_tags` and
`notes` with `deleted_at IS NULL`. Order is unchanged: most used first,
then by name. Drafts count, since they are the owner's posts. Unused
`tags` rows are left in place; nothing else reads them as a list.
## Verification
`tests/test_micropub_v240.py::TestQueries::test_category_skips_unused_tags`
retags one post and deletes another, checks only the live tag is listed,
then undeletes and checks the deleted post's tag returns. It fails on
v2.12.0.
+2 -5
View File
@@ -74,11 +74,6 @@ None open.
- **Where**: `templates/base.html`
- **Source**: Documentation refresh 2026-11-03
### LICENSE placeholder
- **Description**: `LICENSE` still reads "Copyright (c) 2024 [Your Name]". Fill in the holder and year.
- **Where**: `LICENSE`
- **Source**: Documentation refresh 2026-11-03 (rest of that housekeeping item done in v2.11.0)
### Backfeed responses to crossposts
- **Description**: Idea: bring likes, boosts and replies to StarPunk's own crossposts back to the note as responses. Two options: poll GoToSocial (the token would need a `read` scope), or rely on Bridgy, which sends webmentions. Since v2.8.0, StarPunk ignores webmentions *from* a note's own crossposts, so that rule would need to change to work with Bridgy.
- **Where**: `starpunk/syndication.py`, `starpunk/webmention_worker.py`
@@ -185,6 +180,8 @@ None open.
These items were removed from the backlog because they have shipped:
- LICENSE: MIT, copyright Phil Skentelbery 2025-2026 (2026-11-03)
- v2.11.0: SQLite foreign keys enforced (orphans cleaned up by migration 018); CSRF tokens on admin, logout and consent forms; IndieAuth client and crosspost link-card fetches moved onto the SSRF-safe fetcher; `FLASK_DEBUG`/`FLASK_ENV` default to off/production; rate limits shared by all processes (and old windows pruned, so the webmention limiter no longer grows); dependencies upgraded (Python 3.14, no known vulnerabilities); stale `compose.yaml`, duplicate `dev_auth.py` and `test.ini` removed or fixed; `lxml`/`python-dateutil` pinned, `pytest` out of the runtime requirements
- POSSE: GoToSocial (v2.1.0), Bluesky (v2.2.0), Threads (v2.6.0)
+1
View File
@@ -48,3 +48,4 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
- **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03)
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
- **[v2.11.0](v2.11.0/RELEASE.md)** - Maintenance: dependency upgrades, High backlog items (Released 2026-11-03)
- **[v2.12.0](v2.12.0/RELEASE.md)** - Metrics dashboard removed (Released 2026-11-03)
+21
View File
@@ -0,0 +1,21 @@
# v2.12.0
- **Date**: 2026-11-03
- **Status**: Released
- **Focus**: Remove the metrics dashboard
## Removed
- Admin metrics dashboard (`/admin/metrics-dashboard`) and `/admin/metrics`
- The `starpunk/monitoring` package: request timing, query timing, memory monitor thread, feed counters
- `METRICS_*` settings and the `psutil` dependency
## Kept
- `/health`, `/admin/health` (without the metrics section), logs with correlation IDs
## Related
- [CHANGELOG](../../../CHANGELOG.md)
- [ADR-072](../../decisions/ADR-072-remove-metrics-dashboard.md)
- [Design note](../../design/v2.12.0/2026-11-03-remove-metrics.md)
+4 -5
View File
@@ -84,10 +84,9 @@ banner to every page. It sets the `starpunk_session` cookie for
server.
Each app process also starts background threads: the webmention worker,
the WebSub hub worker, the memory monitor and, when a crossposting target
is configured, the syndication worker. Turn off what you don't need locally
with `WEBMENTION_WORKER_ENABLED=false`, `WEBSUB_ENABLED=false` and
`METRICS_ENABLED=false`. Leave `GOTOSOCIAL_*`, `BLUESKY_*` and `THREADS_*`
the WebSub hub worker and, when a crossposting target is configured, the
syndication worker. Turn off what you don't need locally with
`WEBMENTION_WORKER_ENABLED=false` and `WEBSUB_ENABLED=false`. Leave `GOTOSOCIAL_*`, `BLUESKY_*` and `THREADS_*`
empty unless you mean to post to those accounts.
Production runs the same app under gunicorn (`gunicorn app:app`, see `CMD`
@@ -112,7 +111,7 @@ SESSION_SECRET=test uv run pytest tests/test_micropub.py -k delete -x
`ADMIN_ME=https://test.example.com`; set them too to reproduce CI exactly.
- `tests/conftest.py` builds each app with `TESTING=True` and a temporary
`DATA_PATH` and database, so tests never touch `./data`. Under
`TESTING=True` the background workers and memory monitor don't start and
`TESTING=True` the background workers don't start and
the production OIDC check is skipped.
- `tests/test_migration_race_condition.py` is slow (about 5 minutes). While
iterating, add `--ignore=tests/test_migration_race_condition.py`; run the
+1 -1
View File
@@ -4,7 +4,7 @@
**Last Updated**: 2026-11-03 (v2.10.1)
StarPunk uses [Semantic Versioning 2.0.0](https://semver.org/):
`MAJOR.MINOR.PATCH`, currently **2.10.1**. Every release is a git tag
`MAJOR.MINOR.PATCH`, currently **2.12.1**. Every release is a git tag
`vX.Y.Z` and a `CHANGELOG.md` entry. Where the number is stored and how to
write the changelog entry is in
[version-implementation-guide.md](version-implementation-guide.md); the
-3
View File
@@ -22,6 +22,3 @@ python-dotenv==1.2.*
# Images
Pillow==13.0.*
pillow-heif==1.8.*
# Monitoring
psutil==7.2.*
+2 -19
View File
@@ -142,12 +142,6 @@ def create_app(config=None):
# Initialize connection pool
init_pool(app)
# Setup HTTP metrics middleware (v1.1.2 Phase 1)
if app.config.get('METRICS_ENABLED', True):
from starpunk.monitoring import setup_http_metrics
setup_http_metrics(app)
app.logger.info("HTTP metrics middleware enabled")
# Initialize feed cache (v1.1.2 Phase 3)
if app.config.get('FEED_CACHE_ENABLED', True):
from starpunk.feeds import configure_cache
@@ -323,17 +317,6 @@ def create_app(config=None):
register_error_handlers(app)
# Start memory monitor thread (v1.1.2 Phase 1)
# Per CQ5: Skip in test mode
if app.config.get('METRICS_ENABLED', True) and not app.config.get('TESTING', False):
from starpunk.monitoring import MemoryMonitor
memory_monitor = MemoryMonitor(interval=app.config.get('METRICS_MEMORY_INTERVAL', 30))
memory_monitor.start()
app.memory_monitor = memory_monitor
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
# No teardown_appcontext stop: that ran after every request and stopped
# the monitor straight away (v2.10.0). It is a daemon thread.
# Start webmention worker thread (v1.6.0 Phase 2)
# Per background-worker.md: Skip in test mode
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
@@ -474,5 +457,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.11.0"
__version_info__ = (2, 11, 0)
__version__ = "2.12.1"
__version_info__ = (2, 12, 1)
-7
View File
@@ -99,13 +99,6 @@ def load_config(app, config_override=None):
# Flask MAX_CONTENT_LENGTH limits request body size (matches media.py MAX_FILE_SIZE)
app.config["MAX_CONTENT_LENGTH"] = 50 * 1024 * 1024 # 50MB
# Metrics configuration (v1.1.2 Phase 1)
app.config["METRICS_ENABLED"] = os.getenv("METRICS_ENABLED", "true").lower() == "true"
app.config["METRICS_SLOW_QUERY_THRESHOLD"] = float(os.getenv("METRICS_SLOW_QUERY_THRESHOLD", "1.0"))
app.config["METRICS_SAMPLING_RATE"] = float(os.getenv("METRICS_SAMPLING_RATE", "1.0"))
app.config["METRICS_BUFFER_SIZE"] = int(os.getenv("METRICS_BUFFER_SIZE", "1000"))
app.config["METRICS_MEMORY_INTERVAL"] = int(os.getenv("METRICS_MEMORY_INTERVAL", "30"))
# Debug file configuration (v1.5.0 Phase 2)
app.config["DEBUG_SAVE_FAILED_UPLOADS"] = os.getenv("DEBUG_SAVE_FAILED_UPLOADS", "false").lower() == "true"
app.config["DEBUG_FILE_MAX_AGE_DAYS"] = int(os.getenv("DEBUG_FILE_MAX_AGE_DAYS", "7"))
+1 -1
View File
@@ -6,7 +6,7 @@ Provides database initialization and connection pooling
Per v1.1.1 Phase 1:
- Connection pooling for improved performance (ADR-053)
- Request-scoped connections via Flask's g object
- Pool statistics for monitoring
- Pool statistics
"""
from starpunk.database.init import init_db
+4 -30
View File
@@ -5,7 +5,7 @@ Per ADR-053 and developer Q&A Q2, CQ1:
- Provides connection pooling for improved performance
- Integrates with Flask's g object for request-scoped connections
- Maintains same interface as get_db() for transparency
- Pool statistics available for metrics
- Pool statistics (shown in /admin/health)
- Wraps connections with MonitoredConnection for timing (v1.1.2 Phase 1)
Note: Migrations use direct connections (not pooled) for isolation
@@ -27,7 +27,7 @@ class ConnectionPool:
but this provides connection reuse and request-scoped connection management.
"""
def __init__(self, db_path, pool_size=5, timeout=10.0, slow_query_threshold=1.0, metrics_enabled=True):
def __init__(self, db_path, pool_size=5, timeout=10.0):
"""
Initialize connection pool
@@ -35,14 +35,10 @@ class ConnectionPool:
db_path: Path to SQLite database file
pool_size: Maximum number of connections in pool
timeout: Timeout for getting connection (seconds)
slow_query_threshold: Threshold in seconds for slow query detection (v1.1.2)
metrics_enabled: Whether to enable metrics collection (v1.1.2)
"""
self.db_path = Path(db_path)
self.pool_size = pool_size
self.timeout = timeout
self.slow_query_threshold = slow_query_threshold
self.metrics_enabled = metrics_enabled
self._pool = deque(maxlen=pool_size)
self._lock = Lock()
self._stats = {
@@ -56,8 +52,6 @@ class ConnectionPool:
def _create_connection(self):
"""
Create a new database connection
Per CQ1: Wraps connection with MonitoredConnection if metrics enabled
"""
conn = sqlite3.connect(
self.db_path,
@@ -73,12 +67,6 @@ class ConnectionPool:
conn.execute("PRAGMA foreign_keys=ON")
self._stats['connections_created'] += 1
# Wrap with monitoring if enabled (v1.1.2 Phase 1)
if self.metrics_enabled:
from starpunk.monitoring import MonitoredConnection
return MonitoredConnection(conn, self.slow_query_threshold)
return conn
def get_connection(self):
@@ -171,8 +159,6 @@ def init_pool(app):
"""
Initialize the connection pool
Per CQ2: Passes metrics configuration from app config
Args:
app: Flask application instance
"""
@@ -181,20 +167,8 @@ def init_pool(app):
db_path = app.config['DATABASE_PATH']
pool_size = app.config.get('DB_POOL_SIZE', 5)
timeout = app.config.get('DB_TIMEOUT', 10.0)
slow_query_threshold = app.config.get('METRICS_SLOW_QUERY_THRESHOLD', 1.0)
metrics_enabled = app.config.get('METRICS_ENABLED', True)
_pool = ConnectionPool(
db_path,
pool_size,
timeout,
slow_query_threshold,
metrics_enabled
)
app.logger.info(
f"Database connection pool initialized "
f"(size={pool_size}, metrics={'enabled' if metrics_enabled else 'disabled'})"
)
_pool = ConnectionPool(db_path, pool_size, timeout)
app.logger.info(f"Database connection pool initialized (size={pool_size})")
# Register teardown handler
@app.teardown_appcontext
-17
View File
@@ -18,12 +18,10 @@ Standards:
# Standard library imports
from datetime import datetime, timezone
from typing import Optional
import time
import xml.etree.ElementTree as ET
# Local imports
from starpunk.models import Note
from starpunk.monitoring.business import track_feed_generated
# ATOM namespace
@@ -127,10 +125,6 @@ def generate_atom_streaming(
# Remove trailing slash from site_url for consistency
site_url = site_url.rstrip("/")
# Track feed generation timing
start_time = time.time()
item_count = 0
# Current timestamp for updated
now = datetime.now(timezone.utc)
@@ -159,8 +153,6 @@ def generate_atom_streaming(
# Yield entries (newest first)
# Notes from database are already in DESC order (newest first)
for note in notes[:limit]:
item_count += 1
# Build permalink URL
permalink = f"{site_url}{note.permalink}"
@@ -229,15 +221,6 @@ def generate_atom_streaming(
# Yield closing tag
yield '</feed>\n'
# Track feed generation metrics
duration_ms = (time.time() - start_time) * 1000
track_feed_generated(
format='atom',
item_count=item_count,
duration_ms=duration_ms,
cached=False
)
def _escape_xml(text: str) -> str:
"""
-28
View File
@@ -18,12 +18,10 @@ Standards:
# Standard library imports
from datetime import datetime, timezone
from typing import Optional, Dict, Any
import time
import json
# Local imports
from starpunk.models import Note
from starpunk.monitoring.business import track_feed_generated
def generate_json_feed(
@@ -74,9 +72,6 @@ def generate_json_feed(
# Remove trailing slash from site_url for consistency
site_url = site_url.rstrip("/")
# Track feed generation timing
start_time = time.time()
# Build feed object
feed = _build_feed_object(
site_url=site_url,
@@ -88,14 +83,6 @@ def generate_json_feed(
# Serialize to JSON (pretty-printed)
feed_json = json.dumps(feed, ensure_ascii=False, indent=2)
# Track feed generation metrics
duration_ms = (time.time() - start_time) * 1000
track_feed_generated(
format='json',
item_count=min(len(notes), limit),
duration_ms=duration_ms,
cached=False
)
return feed_json
@@ -147,10 +134,6 @@ def generate_json_feed_streaming(
# Remove trailing slash from site_url for consistency
site_url = site_url.rstrip("/")
# Track feed generation timing
start_time = time.time()
item_count = 0
# Start feed object
yield '{\n'
yield f' "version": "https://jsonfeed.org/version/1.1",\n'
@@ -170,8 +153,6 @@ def generate_json_feed_streaming(
# Notes from database are already in DESC order (newest first)
items = notes[:limit]
for i, note in enumerate(items):
item_count += 1
# Build item object
item = _build_item_object(site_url, note)
@@ -193,15 +174,6 @@ def generate_json_feed_streaming(
yield ' ]\n'
yield '}\n'
# Track feed generation metrics
duration_ms = (time.time() - start_time) * 1000
track_feed_generated(
format='json',
item_count=item_count,
duration_ms=duration_ms,
cached=False
)
def _build_feed_object(
site_url: str,
-28
View File
@@ -22,14 +22,12 @@ Standards:
# Standard library imports
from datetime import datetime, timezone
from typing import Optional
import time
# Third-party imports
from feedgen.feed import FeedGenerator
# Local imports
from starpunk.models import Note
from starpunk.monitoring.business import track_feed_generated
def generate_rss(
@@ -98,9 +96,6 @@ def generate_rss(
# Set last build date to now
fg.lastBuildDate(datetime.now(timezone.utc))
# Track feed generation timing
start_time = time.time()
# Add items (limit to configured maximum, newest first)
# Notes from database are DESC but feedgen reverses them, so we reverse back
for note in reversed(notes[:limit]):
@@ -165,14 +160,6 @@ def generate_rss(
# We need to inject media:content and media:thumbnail elements
feed_xml = _inject_media_rss_elements(feed_xml, site_url, notes[:limit])
# Track feed generation metrics
duration_ms = (time.time() - start_time) * 1000
track_feed_generated(
format='rss',
item_count=min(len(notes), limit),
duration_ms=duration_ms,
cached=False
)
return feed_xml
@@ -227,10 +214,6 @@ def generate_rss_streaming(
# Remove trailing slash from site_url for consistency
site_url = site_url.rstrip("/")
# Track feed generation timing
start_time = time.time()
item_count = 0
# Current timestamp for lastBuildDate
now = datetime.now(timezone.utc)
last_build = format_rfc822_date(now)
@@ -251,8 +234,6 @@ def generate_rss_streaming(
# Yield items (newest first)
# Notes from database are already in DESC order (newest first)
for note in notes[:limit]:
item_count += 1
# Build permalink URL
permalink = f"{site_url}{note.permalink}"
@@ -371,15 +352,6 @@ def generate_rss_streaming(
yield " </channel>\n"
yield "</rss>\n"
# Track feed generation metrics
duration_ms = (time.time() - start_time) * 1000
track_feed_generated(
format='rss',
item_count=item_count,
duration_ms=duration_ms,
cached=False
)
def _inject_media_rss_elements(feed_xml: str, site_url: str, notes: list[Note]) -> str:
"""
+6 -3
View File
@@ -842,13 +842,16 @@ def note_to_mf2(note, wanted: Optional[list] = None) -> dict:
def _all_categories(prefix: str = "") -> list:
"""Tags in use, most used first (q=category)"""
"""Tags on posts that exist (drafts included), most used first (q=category)"""
from starpunk.database import get_db
# Inner joins: tags left behind by edits or soft-deleted posts don't count
rows = get_db(current_app).execute(
"""
SELECT t.display_name, COUNT(nt.note_id) AS uses
FROM tags t LEFT JOIN note_tags nt ON nt.tag_id = t.id
SELECT t.display_name, COUNT(*) AS uses
FROM tags t
JOIN note_tags nt ON nt.tag_id = t.id
JOIN notes n ON n.id = nt.note_id AND n.deleted_at IS NULL
GROUP BY t.id ORDER BY uses DESC, t.display_name
"""
).fetchall()
-35
View File
@@ -1,35 +0,0 @@
"""
Performance monitoring for StarPunk
This package provides performance monitoring capabilities including:
- Metrics collection with circular buffers
- Operation timing (database, HTTP, rendering)
- Per-process metrics with aggregation
- Configurable sampling rates
- Database query monitoring (v1.1.2 Phase 1)
- HTTP request/response metrics (v1.1.2 Phase 1)
- Memory monitoring (v1.1.2 Phase 1)
Per ADR-053 and developer Q&A Q6, Q12:
- Each process maintains its own circular buffer
- Buffers store recent metrics (default 1000 entries)
- Metrics include process ID for multi-process deployment
- Sampling rates are configurable per operation type
"""
from starpunk.monitoring.metrics import MetricsBuffer, record_metric, get_metrics, get_metrics_stats
from starpunk.monitoring.database import MonitoredConnection
from starpunk.monitoring.http import setup_http_metrics
from starpunk.monitoring.memory import MemoryMonitor
from starpunk.monitoring import business
__all__ = [
"MetricsBuffer",
"record_metric",
"get_metrics",
"get_metrics_stats",
"MonitoredConnection",
"setup_http_metrics",
"MemoryMonitor",
"business",
]
-298
View File
@@ -1,298 +0,0 @@
"""
Business metrics for StarPunk operations
Per v1.1.2 Phase 1:
- Track note operations (create, update, delete)
- Track feed generation and cache hits/misses
- Track content statistics
Per v1.1.2 Phase 3:
- Track feed statistics by format
- Track feed cache hit/miss rates
- Provide feed statistics dashboard
Example usage:
>>> from starpunk.monitoring.business import track_note_created
>>> track_note_created(note_id=123, content_length=500)
"""
from typing import Optional, Dict, Any
from starpunk.monitoring.metrics import record_metric, get_metrics_stats
def track_note_created(note_id: int, content_length: int, has_media: bool = False) -> None:
"""
Track note creation event
Args:
note_id: ID of created note
content_length: Length of note content in characters
has_media: Whether note has media attachments
"""
metadata = {
'note_id': note_id,
'content_length': content_length,
'has_media': has_media,
}
record_metric(
'render', # Use 'render' for business metrics
'note_created',
content_length,
metadata,
force=True # Always track business events
)
def track_note_updated(note_id: int, content_length: int, fields_changed: Optional[list] = None) -> None:
"""
Track note update event
Args:
note_id: ID of updated note
content_length: New length of note content
fields_changed: List of fields that were changed
"""
metadata = {
'note_id': note_id,
'content_length': content_length,
}
if fields_changed:
metadata['fields_changed'] = ','.join(fields_changed)
record_metric(
'render',
'note_updated',
content_length,
metadata,
force=True
)
def track_note_deleted(note_id: int) -> None:
"""
Track note deletion event
Args:
note_id: ID of deleted note
"""
metadata = {
'note_id': note_id,
}
record_metric(
'render',
'note_deleted',
0, # No meaningful duration for deletion
metadata,
force=True
)
def track_feed_generated(format: str, item_count: int, duration_ms: float, cached: bool = False) -> None:
"""
Track feed generation event
Args:
format: Feed format (rss, atom, json)
item_count: Number of items in feed
duration_ms: Time taken to generate feed
cached: Whether feed was served from cache
"""
metadata = {
'format': format,
'item_count': item_count,
'cached': cached,
}
operation = f'feed_{format}{"_cached" if cached else "_generated"}'
record_metric(
'render',
operation,
duration_ms,
metadata,
force=True # Always track feed operations
)
def track_cache_hit(cache_type: str, key: str) -> None:
"""
Track cache hit event
Args:
cache_type: Type of cache (feed, etc.)
key: Cache key that was hit
"""
metadata = {
'cache_type': cache_type,
'key': key,
}
record_metric(
'render',
f'{cache_type}_cache_hit',
0,
metadata,
force=True
)
def track_cache_miss(cache_type: str, key: str) -> None:
"""
Track cache miss event
Args:
cache_type: Type of cache (feed, etc.)
key: Cache key that was missed
"""
metadata = {
'cache_type': cache_type,
'key': key,
}
record_metric(
'render',
f'{cache_type}_cache_miss',
0,
metadata,
force=True
)
def get_feed_statistics() -> Dict[str, Any]:
"""
Get aggregated feed statistics from metrics buffer and feed cache.
Analyzes metrics to provide feed-specific statistics including:
- Total requests by format (RSS, ATOM, JSON)
- Cache hit/miss rates by format
- Feed generation times by format
- Format popularity (percentage breakdown)
- Feed cache internal statistics
Returns:
Dictionary with feed statistics:
{
'by_format': {
'rss': {'generated': int, 'cached': int, 'total': int, 'avg_duration_ms': float},
'atom': {...},
'json': {...}
},
'cache': {
'hits': int,
'misses': int,
'hit_rate': float (0.0-1.0),
'entries': int,
'evictions': int
},
'total_requests': int,
'format_percentages': {
'rss': float,
'atom': float,
'json': float
}
}
Example:
>>> stats = get_feed_statistics()
>>> print(f"RSS requests: {stats['by_format']['rss']['total']}")
>>> print(f"Cache hit rate: {stats['cache']['hit_rate']:.2%}")
"""
# Get all metrics
all_metrics = get_metrics_stats()
# Initialize result structure
result = {
'by_format': {
'rss': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
'atom': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
'json': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
},
'cache': {
'hits': 0,
'misses': 0,
'hit_rate': 0.0,
},
'total_requests': 0,
'format_percentages': {
'rss': 0.0,
'atom': 0.0,
'json': 0.0,
},
}
# Get by_operation metrics if available
by_operation = all_metrics.get('by_operation', {})
# Count feed operations by format
for operation_name, op_stats in by_operation.items():
# Feed operations are named: feed_rss_generated, feed_rss_cached, etc.
if operation_name.startswith('feed_'):
parts = operation_name.split('_')
if len(parts) >= 3:
format_name = parts[1] # rss, atom, or json
operation_type = parts[2] # generated or cached
if format_name in result['by_format']:
count = op_stats.get('count', 0)
if operation_type == 'generated':
result['by_format'][format_name]['generated'] = count
# Track average duration for generated feeds
result['by_format'][format_name]['avg_duration_ms'] = op_stats.get('avg_duration_ms', 0.0)
elif operation_type == 'cached':
result['by_format'][format_name]['cached'] = count
# Update total for this format
result['by_format'][format_name]['total'] = (
result['by_format'][format_name]['generated'] +
result['by_format'][format_name]['cached']
)
# Track cache hits/misses
elif operation_name == 'feed_cache_hit':
result['cache']['hits'] = op_stats.get('count', 0)
elif operation_name == 'feed_cache_miss':
result['cache']['misses'] = op_stats.get('count', 0)
# Calculate total requests across all formats
result['total_requests'] = sum(
fmt['total'] for fmt in result['by_format'].values()
)
# Calculate cache hit rate
total_cache_requests = result['cache']['hits'] + result['cache']['misses']
if total_cache_requests > 0:
result['cache']['hit_rate'] = result['cache']['hits'] / total_cache_requests
# Calculate format percentages
if result['total_requests'] > 0:
for format_name, fmt_stats in result['by_format'].items():
result['format_percentages'][format_name] = (
fmt_stats['total'] / result['total_requests']
)
# Get feed cache statistics if available
try:
from starpunk.feeds import get_cache
feed_cache = get_cache()
cache_stats = feed_cache.get_stats()
# Merge cache stats (prefer FeedCache internal stats over metrics)
result['cache']['entries'] = cache_stats.get('entries', 0)
result['cache']['evictions'] = cache_stats.get('evictions', 0)
# Use FeedCache hit rate if available and more accurate
if cache_stats.get('hits', 0) + cache_stats.get('misses', 0) > 0:
result['cache']['hits'] = cache_stats.get('hits', 0)
result['cache']['misses'] = cache_stats.get('misses', 0)
result['cache']['hit_rate'] = cache_stats.get('hit_rate', 0.0)
except ImportError:
# Feed cache not available, use defaults
pass
return result
-236
View File
@@ -1,236 +0,0 @@
"""
Database operation monitoring wrapper
Per ADR-053, v1.1.2 Phase 1, and developer Q&A CQ1, IQ1, IQ3:
- Wraps SQLite connections at the pool level
- Times all database operations
- Extracts query type and table name (best effort)
- Detects slow queries based on configurable threshold
- Records metrics to the metrics collector
Example usage:
>>> from starpunk.monitoring.database import MonitoredConnection
>>> conn = sqlite3.connect(':memory:')
>>> monitored = MonitoredConnection(conn, metrics_collector)
>>> cursor = monitored.execute('SELECT * FROM notes')
"""
import re
import sqlite3
import time
from typing import Optional, Any, Tuple
from starpunk.monitoring.metrics import record_metric
class MonitoredConnection:
"""
Wrapper for SQLite connections that monitors performance
Per CQ1: Wraps connections at the pool level
Per IQ1: Uses simple regex for table name extraction
Per IQ3: Single configurable slow query threshold
"""
def __init__(self, connection: sqlite3.Connection, slow_query_threshold: float = 1.0):
"""
Initialize monitored connection wrapper
Args:
connection: SQLite connection to wrap
slow_query_threshold: Threshold in seconds for slow query detection
"""
self._connection = connection
self._slow_query_threshold = slow_query_threshold
def execute(self, query: str, parameters: Optional[Tuple] = None) -> sqlite3.Cursor:
"""
Execute a query with performance monitoring
Args:
query: SQL query to execute
parameters: Optional query parameters
Returns:
sqlite3.Cursor: Query cursor
"""
start_time = time.perf_counter()
query_type = self._get_query_type(query)
table_name = self._extract_table_name(query)
try:
if parameters:
cursor = self._connection.execute(query, parameters)
else:
cursor = self._connection.execute(query)
duration_sec = time.perf_counter() - start_time
duration_ms = duration_sec * 1000
# Record metric (forced if slow query)
is_slow = duration_sec >= self._slow_query_threshold
metadata = {
'query_type': query_type,
'table': table_name,
'is_slow': is_slow,
}
# Add query text for slow queries (for debugging)
if is_slow:
# Truncate query to avoid storing huge queries
metadata['query'] = query[:200] if len(query) > 200 else query
record_metric(
'database',
f'{query_type} {table_name}',
duration_ms,
metadata,
force=is_slow # Always record slow queries
)
return cursor
except Exception as e:
duration_sec = time.perf_counter() - start_time
duration_ms = duration_sec * 1000
# Record error metric
metadata = {
'query_type': query_type,
'table': table_name,
'error': str(e),
'query': query[:200] if len(query) > 200 else query
}
record_metric(
'database',
f'{query_type} {table_name} ERROR',
duration_ms,
metadata,
force=True # Always record errors
)
raise
def executemany(self, query: str, parameters) -> sqlite3.Cursor:
"""
Execute a query with multiple parameter sets
Args:
query: SQL query to execute
parameters: Sequence of parameter tuples
Returns:
sqlite3.Cursor: Query cursor
"""
start_time = time.perf_counter()
query_type = self._get_query_type(query)
table_name = self._extract_table_name(query)
try:
cursor = self._connection.executemany(query, parameters)
duration_ms = (time.perf_counter() - start_time) * 1000
# Record metric
metadata = {
'query_type': query_type,
'table': table_name,
'batch': True,
}
record_metric(
'database',
f'{query_type} {table_name} BATCH',
duration_ms,
metadata
)
return cursor
except Exception as e:
duration_ms = (time.perf_counter() - start_time) * 1000
metadata = {
'query_type': query_type,
'table': table_name,
'error': str(e),
'batch': True
}
record_metric(
'database',
f'{query_type} {table_name} BATCH ERROR',
duration_ms,
metadata,
force=True
)
raise
def _get_query_type(self, query: str) -> str:
"""
Extract query type from SQL statement
Args:
query: SQL query
Returns:
Query type (SELECT, INSERT, UPDATE, DELETE, etc.)
"""
query_upper = query.strip().upper()
for query_type in ['SELECT', 'INSERT', 'UPDATE', 'DELETE', 'CREATE', 'DROP', 'ALTER', 'PRAGMA']:
if query_upper.startswith(query_type):
return query_type
return 'OTHER'
def _extract_table_name(self, query: str) -> str:
"""
Extract table name from query (best effort)
Per IQ1: Keep it simple with basic regex patterns.
Returns "unknown" for complex queries.
Note: Complex queries (JOINs, subqueries, CTEs) return "unknown".
This covers 90% of queries accurately.
Args:
query: SQL query
Returns:
Table name or "unknown"
"""
query_lower = query.lower().strip()
# Simple patterns that cover 90% of cases
patterns = [
r'from\s+(\w+)',
r'update\s+(\w+)',
r'insert\s+into\s+(\w+)',
r'delete\s+from\s+(\w+)',
r'create\s+table\s+(?:if\s+not\s+exists\s+)?(\w+)',
r'drop\s+table\s+(?:if\s+exists\s+)?(\w+)',
r'alter\s+table\s+(\w+)',
]
for pattern in patterns:
match = re.search(pattern, query_lower)
if match:
return match.group(1)
# Complex queries (JOINs, subqueries, CTEs)
return "unknown"
# Delegate all other connection methods to the wrapped connection
def __getattr__(self, name: str) -> Any:
"""Delegate all other methods to the wrapped connection"""
return getattr(self._connection, name)
def __enter__(self):
"""Support context manager protocol"""
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Support context manager protocol"""
return self._connection.__exit__(exc_type, exc_val, exc_tb)
-133
View File
@@ -1,133 +0,0 @@
"""
HTTP request/response metrics middleware
Per v1.1.2 Phase 1 and developer Q&A IQ2:
- Times all HTTP requests
- Generates request IDs for tracking (IQ2)
- Records status codes, methods, routes
- Tracks request and response sizes
- Adds X-Request-ID header to all responses (not just debug mode)
Example usage:
>>> from starpunk.monitoring.http import setup_http_metrics
>>> app = Flask(__name__)
>>> setup_http_metrics(app)
"""
import time
import uuid
from flask import g, request, Flask
from typing import Any
from starpunk.monitoring.metrics import record_metric
def setup_http_metrics(app: Flask) -> None:
"""
Setup HTTP metrics collection for Flask app
Per IQ2: Generates request IDs and adds X-Request-ID header in all modes
Args:
app: Flask application instance
"""
@app.before_request
def start_request_metrics():
"""
Initialize request metrics tracking
Per IQ2: Generate UUID request ID and store in g
"""
# Generate request ID (IQ2: in all modes, not just debug)
g.request_id = str(uuid.uuid4())
# Store request start time and metadata
g.request_start_time = time.perf_counter()
g.request_metadata = {
'method': request.method,
'endpoint': request.endpoint or 'unknown',
'path': request.path,
'content_length': request.content_length or 0,
}
@app.after_request
def record_response_metrics(response):
"""
Record HTTP response metrics
Args:
response: Flask response object
Returns:
Modified response with X-Request-ID header
"""
# Skip if metrics not initialized (shouldn't happen in normal flow)
if not hasattr(g, 'request_start_time'):
return response
# Calculate request duration
duration_sec = time.perf_counter() - g.request_start_time
duration_ms = duration_sec * 1000
# Get response size
response_size = 0
# Check if response is in direct passthrough mode (streaming)
if hasattr(response, 'direct_passthrough') and response.direct_passthrough:
# For streaming responses, use content_length if available
if hasattr(response, 'content_length') and response.content_length:
response_size = response.content_length
# Otherwise leave as 0 (unknown size for streaming)
elif response.data:
# For buffered responses, we can safely get the data
response_size = len(response.data)
elif hasattr(response, 'content_length') and response.content_length:
response_size = response.content_length
# Build metadata
metadata = {
**g.request_metadata,
'status_code': response.status_code,
'response_size': response_size,
}
# Record metric
operation_name = f"{g.request_metadata['method']} {g.request_metadata['endpoint']}"
record_metric(
'http',
operation_name,
duration_ms,
metadata
)
# Add request ID header (IQ2: in all modes)
response.headers['X-Request-ID'] = g.request_id
return response
@app.teardown_request
def record_error_metrics(error=None):
"""
Record metrics for requests that result in errors
Args:
error: Exception if request failed
"""
if error and hasattr(g, 'request_start_time'):
duration_ms = (time.perf_counter() - g.request_start_time) * 1000
metadata = {
**g.request_metadata,
'error': str(error),
'error_type': type(error).__name__,
}
operation_name = f"{g.request_metadata['method']} {g.request_metadata['endpoint']} ERROR"
record_metric(
'http',
operation_name,
duration_ms,
metadata,
force=True # Always record errors
)
-191
View File
@@ -1,191 +0,0 @@
"""
Memory monitoring background thread
Per v1.1.2 Phase 1 and developer Q&A CQ5, IQ8:
- Background daemon thread for continuous memory monitoring
- Tracks RSS and VMS memory usage
- Detects memory growth and potential leaks
- 5-second baseline period after startup (IQ8)
- Skipped in test mode (CQ5)
Example usage:
>>> from starpunk.monitoring.memory import MemoryMonitor
>>> monitor = MemoryMonitor(interval=30)
>>> monitor.start() # Runs as daemon thread
>>> # ... application runs ...
>>> monitor.stop()
"""
import gc
import logging
import os
import sys
import threading
import time
from typing import Dict, Any
import psutil
from starpunk.monitoring.metrics import record_metric
logger = logging.getLogger(__name__)
class MemoryMonitor(threading.Thread):
"""
Background thread for memory monitoring
Per CQ5: Daemon thread that auto-terminates with main process
Per IQ8: 5-second baseline period after startup
"""
def __init__(self, interval: int = 30):
"""
Initialize memory monitor thread
Args:
interval: Monitoring interval in seconds (default: 30)
"""
super().__init__(daemon=True) # CQ5: daemon thread
self.interval = interval
self._stop_event = threading.Event()
self._process = psutil.Process()
self._baseline_memory = None
self._high_water_mark = 0
def run(self):
"""
Main monitoring loop
Per IQ8: Wait 5 seconds for app initialization before setting baseline
"""
try:
# Wait for app initialization (IQ8: 5 seconds)
time.sleep(5)
# Set baseline memory
memory_info = self._get_memory_info()
self._baseline_memory = memory_info['rss_mb']
logger.info(f"Memory monitor baseline set: {self._baseline_memory:.2f} MB RSS")
# Start monitoring loop
while not self._stop_event.is_set():
try:
self._collect_metrics()
except Exception as e:
logger.error(f"Memory monitoring error: {e}", exc_info=True)
# Wait for interval or until stop event
self._stop_event.wait(self.interval)
except Exception as e:
logger.error(f"Memory monitor thread failed: {e}", exc_info=True)
def _collect_metrics(self):
"""Collect and record memory metrics"""
memory_info = self._get_memory_info()
gc_stats = self._get_gc_stats()
# Update high water mark
if memory_info['rss_mb'] > self._high_water_mark:
self._high_water_mark = memory_info['rss_mb']
# Calculate growth rate (MB/hour) if baseline is set
growth_rate = 0.0
if self._baseline_memory:
growth_rate = memory_info['rss_mb'] - self._baseline_memory
# Record metrics
metadata = {
'rss_mb': memory_info['rss_mb'],
'vms_mb': memory_info['vms_mb'],
'percent': memory_info['percent'],
'high_water_mb': self._high_water_mark,
'growth_mb': growth_rate,
'gc_collections': gc_stats['collections'],
'gc_collected': gc_stats['collected'],
}
record_metric(
'render', # Use 'render' operation type for memory metrics
'memory_usage',
memory_info['rss_mb'],
metadata,
force=True # Always record memory metrics
)
# Warn if significant growth detected (>10MB growth from baseline)
if growth_rate > 10.0:
logger.warning(
f"Memory growth detected: +{growth_rate:.2f} MB from baseline "
f"(current: {memory_info['rss_mb']:.2f} MB, baseline: {self._baseline_memory:.2f} MB)"
)
def _get_memory_info(self) -> Dict[str, float]:
"""
Get current process memory usage
Returns:
Dict with memory info in MB
"""
memory = self._process.memory_info()
return {
'rss_mb': memory.rss / (1024 * 1024), # Resident Set Size
'vms_mb': memory.vms / (1024 * 1024), # Virtual Memory Size
'percent': self._process.memory_percent(),
}
def _get_gc_stats(self) -> Dict[str, Any]:
"""
Get garbage collection statistics
Returns:
Dict with GC stats
"""
# Get collection counts per generation
counts = gc.get_count()
# Perform a quick gen 0 collection and count collected objects
collected = gc.collect(0)
return {
'collections': {
'gen0': counts[0],
'gen1': counts[1],
'gen2': counts[2],
},
'collected': collected,
'uncollectable': len(gc.garbage),
}
def stop(self):
"""
Stop the monitoring thread gracefully
Sets the stop event to signal the thread to exit
"""
logger.info("Stopping memory monitor")
self._stop_event.set()
def get_stats(self) -> Dict[str, Any]:
"""
Get current memory statistics
Returns:
Dict with current memory stats
"""
if not self._baseline_memory:
return {'status': 'initializing'}
memory_info = self._get_memory_info()
return {
'status': 'running',
'current_rss_mb': memory_info['rss_mb'],
'baseline_rss_mb': self._baseline_memory,
'growth_mb': memory_info['rss_mb'] - self._baseline_memory,
'high_water_mb': self._high_water_mark,
'percent': memory_info['percent'],
}
-425
View File
@@ -1,425 +0,0 @@
"""
Metrics collection and buffering for performance monitoring
Per ADR-053 and developer Q&A Q6, Q12:
- Per-process circular buffers using deque
- Configurable buffer size (default 1000 entries)
- Include process ID in all metrics
- Configuration-based sampling rates
- Operation types: database, http, render
Example usage:
>>> from starpunk.monitoring import record_metric, get_metrics
>>>
>>> # Record a database operation
>>> record_metric('database', 'query', duration_ms=45.2, query='SELECT * FROM notes')
>>>
>>> # Get all metrics
>>> metrics = get_metrics()
>>> print(f"Collected {len(metrics)} metrics")
"""
import os
import random
import time
from collections import deque
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from threading import Lock
from typing import Any, Deque, Dict, List, Literal, Optional, Union
# Operation types for categorizing metrics
OperationType = Literal["database", "http", "render"]
# Module-level circular buffer (per-process)
# Each process in a multi-process deployment maintains its own buffer
_metrics_buffer: Optional["MetricsBuffer"] = None
_buffer_lock = Lock()
@dataclass
class Metric:
"""
Represents a single performance metric
Attributes:
operation_type: Type of operation (database/http/render)
operation_name: Name/description of operation
timestamp: When the metric was recorded (ISO format)
duration_ms: Duration in milliseconds
process_id: Process ID that recorded the metric
metadata: Additional operation-specific data
"""
operation_type: OperationType
operation_name: str
timestamp: str
duration_ms: float
process_id: int
metadata: Dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> Dict[str, Any]:
"""Convert metric to dictionary for serialization"""
return asdict(self)
class MetricsBuffer:
"""
Circular buffer for storing performance metrics
Per developer Q&A Q6:
- Uses deque for efficient circular buffer
- Per-process storage (not shared across workers)
- Thread-safe with locking
- Configurable max size (default 1000)
- Automatic eviction of oldest entries when full
Per developer Q&A Q12:
- Configurable sampling rates per operation type
- Default 100% sampling (suitable for low-traffic sites)
- Slow queries always logged regardless of sampling
Example:
>>> buffer = MetricsBuffer(max_size=1000)
>>> buffer.record('database', 'query', 45.2, {'query': 'SELECT ...'})
>>> metrics = buffer.get_all()
"""
def __init__(
self,
max_size: int = 1000,
sampling_rates: Optional[Union[Dict[OperationType, float], float]] = None
):
"""
Initialize metrics buffer
Args:
max_size: Maximum number of metrics to store
sampling_rates: Either:
- float: Global sampling rate for all operation types (0.0-1.0)
- dict: Mapping operation type to sampling rate
Default: 1.0 (100% sampling)
"""
self.max_size = max_size
self._buffer: Deque[Metric] = deque(maxlen=max_size)
self._lock = Lock()
self._process_id = os.getpid()
# Handle different sampling_rates types
if sampling_rates is None:
# Default to 100% sampling for all types
self._sampling_rates = {
"database": 1.0,
"http": 1.0,
"render": 1.0,
}
elif isinstance(sampling_rates, (int, float)):
# Global rate for all types
rate = float(sampling_rates)
self._sampling_rates = {
"database": rate,
"http": rate,
"render": rate,
}
else:
# Dict with per-type rates
self._sampling_rates = sampling_rates
def record(
self,
operation_type: OperationType,
operation_name: str,
duration_ms: float,
metadata: Optional[Dict[str, Any]] = None,
force: bool = False
) -> bool:
"""
Record a performance metric
Args:
operation_type: Type of operation (database/http/render)
operation_name: Name/description of operation
duration_ms: Duration in milliseconds
metadata: Additional operation-specific data
force: If True, bypass sampling (for slow query logging)
Returns:
True if metric was recorded, False if skipped due to sampling
Example:
>>> buffer.record('database', 'SELECT notes', 45.2,
... {'query': 'SELECT * FROM notes LIMIT 10'})
True
"""
# Apply sampling (unless forced)
if not force:
sampling_rate = self._sampling_rates.get(operation_type, 0.1)
if random.random() > sampling_rate:
return False
metric = Metric(
operation_type=operation_type,
operation_name=operation_name,
timestamp=datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
duration_ms=duration_ms,
process_id=self._process_id,
metadata=metadata or {}
)
with self._lock:
self._buffer.append(metric)
return True
def get_all(self) -> List[Metric]:
"""
Get all metrics from buffer
Returns:
List of metrics (oldest to newest)
Example:
>>> metrics = buffer.get_all()
>>> len(metrics)
1000
"""
with self._lock:
return list(self._buffer)
def get_recent(self, count: int) -> List[Metric]:
"""
Get most recent N metrics
Args:
count: Number of recent metrics to return
Returns:
List of most recent metrics (newest first)
Example:
>>> recent = buffer.get_recent(10)
>>> len(recent)
10
"""
with self._lock:
# Convert to list, reverse to get newest first, then slice
all_metrics = list(self._buffer)
all_metrics.reverse()
return all_metrics[:count]
def get_by_type(self, operation_type: OperationType) -> List[Metric]:
"""
Get all metrics of a specific type
Args:
operation_type: Type to filter by (database/http/render)
Returns:
List of metrics matching the type
Example:
>>> db_metrics = buffer.get_by_type('database')
"""
with self._lock:
return [m for m in self._buffer if m.operation_type == operation_type]
def get_slow_operations(
self,
threshold_ms: float = 1000.0,
operation_type: Optional[OperationType] = None
) -> List[Metric]:
"""
Get operations that exceeded a duration threshold
Args:
threshold_ms: Duration threshold in milliseconds
operation_type: Optional type filter
Returns:
List of slow operations
Example:
>>> slow_queries = buffer.get_slow_operations(1000, 'database')
"""
with self._lock:
metrics = list(self._buffer)
# Filter by type if specified
if operation_type:
metrics = [m for m in metrics if m.operation_type == operation_type]
# Filter by duration threshold
return [m for m in metrics if m.duration_ms >= threshold_ms]
def get_stats(self) -> Dict[str, Any]:
"""
Get statistics about the buffer
Returns:
Dict with buffer statistics
Example:
>>> stats = buffer.get_stats()
>>> stats['total_count']
1000
"""
with self._lock:
metrics = list(self._buffer)
# Calculate stats per operation type
type_stats = {}
for op_type in ["database", "http", "render"]:
type_metrics = [m for m in metrics if m.operation_type == op_type]
if type_metrics:
durations = [m.duration_ms for m in type_metrics]
type_stats[op_type] = {
"count": len(type_metrics),
"avg_duration_ms": sum(durations) / len(durations),
"min_duration_ms": min(durations),
"max_duration_ms": max(durations),
}
else:
type_stats[op_type] = {
"count": 0,
"avg_duration_ms": 0.0,
"min_duration_ms": 0.0,
"max_duration_ms": 0.0,
}
return {
"total_count": len(metrics),
"max_size": self.max_size,
"process_id": self._process_id,
"sampling_rates": self._sampling_rates,
"by_type": type_stats,
}
def clear(self) -> None:
"""
Clear all metrics from buffer
Example:
>>> buffer.clear()
"""
with self._lock:
self._buffer.clear()
def set_sampling_rate(
self,
operation_type: OperationType,
rate: float
) -> None:
"""
Update sampling rate for an operation type
Args:
operation_type: Type to update
rate: New sampling rate (0.0-1.0)
Example:
>>> buffer.set_sampling_rate('database', 0.5) # 50% sampling
"""
if not 0.0 <= rate <= 1.0:
raise ValueError("Sampling rate must be between 0.0 and 1.0")
with self._lock:
self._sampling_rates[operation_type] = rate
def get_buffer() -> MetricsBuffer:
"""
Get or create the module-level metrics buffer
This ensures a single buffer per process. In multi-process deployments
(e.g., gunicorn), each worker process will have its own buffer.
Returns:
MetricsBuffer instance for this process
Example:
>>> buffer = get_buffer()
>>> buffer.record('database', 'query', 45.2)
"""
global _metrics_buffer
if _metrics_buffer is None:
with _buffer_lock:
# Double-check locking pattern
if _metrics_buffer is None:
# Get configuration from Flask app if available
try:
from flask import current_app
max_size = current_app.config.get('METRICS_BUFFER_SIZE', 1000)
sampling_rate = current_app.config.get('METRICS_SAMPLING_RATE', 1.0)
except (ImportError, RuntimeError):
# Flask not available or no app context
max_size = 1000
sampling_rate = 1.0 # Default to 100%
_metrics_buffer = MetricsBuffer(
max_size=max_size,
sampling_rates=sampling_rate
)
return _metrics_buffer
def record_metric(
operation_type: OperationType,
operation_name: str,
duration_ms: float,
metadata: Optional[Dict[str, Any]] = None,
force: bool = False
) -> bool:
"""
Record a metric using the module-level buffer
Convenience function that uses get_buffer() internally.
Args:
operation_type: Type of operation (database/http/render)
operation_name: Name/description of operation
duration_ms: Duration in milliseconds
metadata: Additional operation-specific data
force: If True, bypass sampling (for slow query logging)
Returns:
True if metric was recorded, False if skipped due to sampling
Example:
>>> record_metric('database', 'SELECT notes', 45.2,
... {'query': 'SELECT * FROM notes LIMIT 10'})
True
"""
buffer = get_buffer()
return buffer.record(operation_type, operation_name, duration_ms, metadata, force)
def get_metrics() -> List[Metric]:
"""
Get all metrics from the module-level buffer
Returns:
List of metrics (oldest to newest)
Example:
>>> metrics = get_metrics()
>>> len(metrics)
1000
"""
buffer = get_buffer()
return buffer.get_all()
def get_metrics_stats() -> Dict[str, Any]:
"""
Get statistics from the module-level buffer
Returns:
Dict with buffer statistics
Example:
>>> stats = get_metrics_stats()
>>> print(f"Total metrics: {stats['total_count']}")
"""
buffer = get_buffer()
return buffer.get_stats()
+1 -207
View File
@@ -9,7 +9,6 @@ from flask import Blueprint, flash, g, jsonify, redirect, render_template, reque
import logging
import os
import shutil
from datetime import datetime, timezone
logger = logging.getLogger(__name__)
@@ -366,202 +365,6 @@ def delete_note_submit(note_id: int):
return redirect(url_for("admin.dashboard"))
def transform_metrics_for_template(metrics_stats):
"""
Transform metrics stats to match template structure
The template expects direct access to metrics.database.count, but
get_metrics_stats() returns metrics.by_type.database.count.
This function adapts the data structure to match template expectations.
Args:
metrics_stats: Dict from get_metrics_stats() with nested by_type structure
Returns:
Dict with flattened structure matching template expectations
Per ADR-060: Route Adapter Pattern for template compatibility
"""
transformed = {}
# Map by_type to direct access
for op_type in ['database', 'http', 'render']:
if 'by_type' in metrics_stats and op_type in metrics_stats['by_type']:
type_data = metrics_stats['by_type'][op_type]
transformed[op_type] = {
'count': type_data.get('count', 0),
'avg': type_data.get('avg_duration_ms', 0),
'min': type_data.get('min_duration_ms', 0),
'max': type_data.get('max_duration_ms', 0)
}
else:
# Provide defaults for missing types or when by_type doesn't exist
transformed[op_type] = {
'count': 0,
'avg': 0,
'min': 0,
'max': 0
}
# Keep other top-level stats
transformed['total_count'] = metrics_stats.get('total_count', 0)
transformed['max_size'] = metrics_stats.get('max_size', 1000)
transformed['process_id'] = metrics_stats.get('process_id', 0)
return transformed
@bp.route("/metrics-dashboard")
@require_auth
def metrics_dashboard():
"""
Metrics visualization dashboard (Phase 3)
Displays performance metrics, database statistics, feed statistics,
and system health with visual charts and auto-refresh capability.
Per Q19 requirements:
- Server-side rendering with Jinja2
- htmx for auto-refresh
- Chart.js from CDN for graphs
- Progressive enhancement (works without JS)
Per v1.1.2 Phase 3:
- Feed statistics by format
- Cache hit/miss rates
- Format popularity breakdown
Returns:
Rendered dashboard template with metrics
Decorator: @require_auth
Template: templates/admin/metrics_dashboard.html
"""
# Defensive imports with graceful degradation for missing modules
try:
from starpunk.database.pool import get_pool_stats
from starpunk.monitoring import get_metrics_stats
from starpunk.monitoring.business import get_feed_statistics
monitoring_available = True
except ImportError:
monitoring_available = False
# Provide fallback functions that return error messages
def get_pool_stats():
return {"error": "Database pool monitoring not available"}
def get_metrics_stats():
return {"error": "Monitoring module not implemented"}
def get_feed_statistics():
return {"error": "Feed statistics not available"}
# Get current metrics for initial page load
metrics_data = {}
pool_stats = {}
feed_stats = {}
try:
raw_metrics = get_metrics_stats()
metrics_data = transform_metrics_for_template(raw_metrics)
except Exception as e:
flash(f"Error loading metrics: {e}", "warning")
# Provide safe defaults matching template expectations
metrics_data = {
'database': {'count': 0, 'avg': 0, 'min': 0, 'max': 0},
'http': {'count': 0, 'avg': 0, 'min': 0, 'max': 0},
'render': {'count': 0, 'avg': 0, 'min': 0, 'max': 0},
'total_count': 0,
'max_size': 1000,
'process_id': 0
}
try:
pool_stats = get_pool_stats()
except Exception as e:
flash(f"Error loading pool stats: {e}", "warning")
try:
feed_stats = get_feed_statistics()
except Exception as e:
flash(f"Error loading feed stats: {e}", "warning")
# Provide safe defaults
feed_stats = {
'by_format': {
'rss': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
'atom': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
'json': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
},
'cache': {'hits': 0, 'misses': 0, 'hit_rate': 0.0, 'entries': 0, 'evictions': 0},
'total_requests': 0,
'format_percentages': {'rss': 0.0, 'atom': 0.0, 'json': 0.0},
}
return render_template(
"admin/metrics_dashboard.html",
metrics=metrics_data,
pool=pool_stats,
feeds=feed_stats,
user_me=g.me
)
@bp.route("/metrics")
@require_auth
def metrics():
"""
Performance metrics and database pool statistics endpoint
Per Phase 2 requirements:
- Expose database pool statistics
- Show performance metrics from MetricsBuffer
- Requires authentication
Per v1.1.2 Phase 3:
- Include feed statistics
Returns:
JSON with metrics, pool statistics, and feed statistics
Response codes:
200: Metrics retrieved successfully
Decorator: @require_auth
"""
from flask import current_app
from starpunk.database.pool import get_pool_stats
from starpunk.monitoring import get_metrics_stats
from starpunk.monitoring.business import get_feed_statistics
response = {
"timestamp": datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
"process_id": os.getpid(),
"database": {},
"performance": {},
"feeds": {}
}
# Get database pool statistics
try:
pool_stats = get_pool_stats()
response["database"]["pool"] = pool_stats
except Exception as e:
response["database"]["pool"] = {"error": str(e)}
# Get performance metrics
try:
metrics_stats = get_metrics_stats()
response["performance"] = metrics_stats
except Exception as e:
response["performance"] = {"error": str(e)}
# Get feed statistics
try:
feed_stats = get_feed_statistics()
response["feeds"] = feed_stats
except Exception as e:
response["feeds"] = {"error": str(e)}
return jsonify(response), 200
@bp.route("/health")
@require_auth
def health_diagnostics():
@@ -571,7 +374,7 @@ def health_diagnostics():
Per developer Q&A Q10:
- Always requires authentication
- Provides comprehensive diagnostics
- Includes metrics, database pool statistics, and system info
- Includes database pool statistics and system info
Returns:
JSON with complete system diagnostics
@@ -591,7 +394,6 @@ def health_diagnostics():
"environment": current_app.config.get("ENV", "unknown"),
"process_id": os.getpid(),
"checks": {},
"metrics": {},
"database": {}
}
@@ -665,14 +467,6 @@ def health_diagnostics():
}
overall_healthy = False
# Performance metrics
try:
from starpunk.monitoring import get_metrics_stats
metrics_stats = get_metrics_stats()
diagnostics["metrics"] = metrics_stats
except Exception as e:
diagnostics["metrics"] = {"error": str(e)}
# Update overall status
diagnostics["status"] = "healthy" if overall_healthy else "unhealthy"
-1
View File
@@ -8,7 +8,6 @@
<nav class="admin-nav">
<a href="{{ url_for('admin.dashboard') }}">Dashboard</a>
<a href="{{ url_for('admin.new_note_form') }}">New Note</a>
<a href="{{ url_for('admin.metrics_dashboard') }}">Metrics</a>
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
<button type="submit" class="button button-secondary">Logout</button>
-585
View File
@@ -1,585 +0,0 @@
{% extends "admin/base.html" %}
{% block title %}Metrics Dashboard - StarPunk Admin{% endblock %}
{% block head %}
{{ super() }}
<!-- Chart.js from CDN for visualizations -->
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js" crossorigin="anonymous"></script>
<!-- htmx for auto-refresh -->
<script src="https://unpkg.com/htmx.org@1.9.10" crossorigin="anonymous"></script>
<style>
.metrics-dashboard {
max-width: 1200px;
}
.metrics-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 20px;
margin-bottom: 30px;
}
.metric-card {
background: #fff;
border: 1px solid #ddd;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 4px rgba(0,0,0,0.05);
}
.metric-card h3 {
margin-top: 0;
font-size: 1.1em;
color: #333;
border-bottom: 2px solid #007bff;
padding-bottom: 10px;
margin-bottom: 15px;
}
.metric-value {
font-size: 2em;
font-weight: bold;
color: #007bff;
margin: 10px 0;
}
.metric-label {
color: #666;
font-size: 0.9em;
margin-bottom: 5px;
}
.metric-detail {
display: flex;
justify-content: space-between;
padding: 8px 0;
border-bottom: 1px solid #f0f0f0;
}
.metric-detail:last-child {
border-bottom: none;
}
.metric-detail-label {
color: #666;
}
.metric-detail-value {
font-weight: bold;
}
.chart-container {
position: relative;
height: 300px;
margin-top: 20px;
}
.status-indicator {
display: inline-block;
width: 12px;
height: 12px;
border-radius: 50%;
margin-right: 8px;
}
.status-healthy {
background-color: #28a745;
}
.status-warning {
background-color: #ffc107;
}
.status-error {
background-color: #dc3545;
}
.refresh-info {
color: #666;
font-size: 0.9em;
text-align: center;
margin-top: 20px;
padding: 10px;
background-color: #f8f9fa;
border-radius: 4px;
}
.no-js-message {
display: none;
background-color: #fff3cd;
border: 1px solid #ffeaa7;
color: #856404;
padding: 15px;
border-radius: 4px;
margin-bottom: 20px;
}
noscript .no-js-message {
display: block;
}
</style>
{% endblock %}
{% block admin_content %}
<div class="metrics-dashboard">
<h2>Metrics Dashboard</h2>
<noscript>
<div class="no-js-message">
Note: Auto-refresh and charts require JavaScript. Data is displayed below in text format.
</div>
</noscript>
<!-- Auto-refresh container -->
<div hx-get="{{ url_for('admin.metrics') }}" hx-trigger="every 10s" hx-swap="none" hx-on::after-request="updateDashboard(event)"></div>
<!-- Database Pool Statistics -->
<div class="metrics-grid">
<div class="metric-card">
<h3>Database Connection Pool</h3>
<div class="metric-detail">
<span class="metric-detail-label">Active Connections</span>
<span class="metric-detail-value" id="pool-active">{{ pool.active_connections|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Idle Connections</span>
<span class="metric-detail-value" id="pool-idle">{{ pool.idle_connections|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Total Connections</span>
<span class="metric-detail-value" id="pool-total">{{ pool.total_connections|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Pool Size</span>
<span class="metric-detail-value" id="pool-size">{{ pool.pool_size|default(5) }}</span>
</div>
</div>
<div class="metric-card">
<h3>Database Operations</h3>
<div class="metric-detail">
<span class="metric-detail-label">Total Queries</span>
<span class="metric-detail-value" id="db-total">{{ metrics.database.count|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Average Time</span>
<span class="metric-detail-value" id="db-avg">{{ "%.2f"|format(metrics.database.avg|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Min Time</span>
<span class="metric-detail-value" id="db-min">{{ "%.2f"|format(metrics.database.min|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Max Time</span>
<span class="metric-detail-value" id="db-max">{{ "%.2f"|format(metrics.database.max|default(0)) }} ms</span>
</div>
</div>
<div class="metric-card">
<h3>HTTP Requests</h3>
<div class="metric-detail">
<span class="metric-detail-label">Total Requests</span>
<span class="metric-detail-value" id="http-total">{{ metrics.http.count|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Average Time</span>
<span class="metric-detail-value" id="http-avg">{{ "%.2f"|format(metrics.http.avg|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Min Time</span>
<span class="metric-detail-value" id="http-min">{{ "%.2f"|format(metrics.http.min|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Max Time</span>
<span class="metric-detail-value" id="http-max">{{ "%.2f"|format(metrics.http.max|default(0)) }} ms</span>
</div>
</div>
<div class="metric-card">
<h3>Template Rendering</h3>
<div class="metric-detail">
<span class="metric-detail-label">Total Renders</span>
<span class="metric-detail-value" id="render-total">{{ metrics.render.count|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Average Time</span>
<span class="metric-detail-value" id="render-avg">{{ "%.2f"|format(metrics.render.avg|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Min Time</span>
<span class="metric-detail-value" id="render-min">{{ "%.2f"|format(metrics.render.min|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Max Time</span>
<span class="metric-detail-value" id="render-max">{{ "%.2f"|format(metrics.render.max|default(0)) }} ms</span>
</div>
</div>
</div>
<!-- Charts -->
<div class="metrics-grid">
<div class="metric-card">
<h3>Connection Pool Usage</h3>
<div class="chart-container">
<canvas id="poolChart"></canvas>
</div>
</div>
<div class="metric-card">
<h3>Performance Overview</h3>
<div class="chart-container">
<canvas id="performanceChart"></canvas>
</div>
</div>
</div>
<!-- Feed Statistics (Phase 3) -->
<h2 style="margin-top: 40px;">Feed Statistics</h2>
<div class="metrics-grid">
<div class="metric-card">
<h3>Feed Requests by Format</h3>
<div class="metric-detail">
<span class="metric-detail-label">RSS</span>
<span class="metric-detail-value" id="feed-rss-total">{{ feeds.by_format.rss.total|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">ATOM</span>
<span class="metric-detail-value" id="feed-atom-total">{{ feeds.by_format.atom.total|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">JSON Feed</span>
<span class="metric-detail-value" id="feed-json-total">{{ feeds.by_format.json.total|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Total Requests</span>
<span class="metric-detail-value" id="feed-total">{{ feeds.total_requests|default(0) }}</span>
</div>
</div>
<div class="metric-card">
<h3>Feed Cache Statistics</h3>
<div class="metric-detail">
<span class="metric-detail-label">Cache Hits</span>
<span class="metric-detail-value" id="feed-cache-hits">{{ feeds.cache.hits|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Cache Misses</span>
<span class="metric-detail-value" id="feed-cache-misses">{{ feeds.cache.misses|default(0) }}</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Hit Rate</span>
<span class="metric-detail-value" id="feed-cache-hit-rate">{{ "%.1f"|format(feeds.cache.hit_rate|default(0) * 100) }}%</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">Cached Entries</span>
<span class="metric-detail-value" id="feed-cache-entries">{{ feeds.cache.entries|default(0) }}</span>
</div>
</div>
<div class="metric-card">
<h3>Feed Generation Performance</h3>
<div class="metric-detail">
<span class="metric-detail-label">RSS Avg Time</span>
<span class="metric-detail-value" id="feed-rss-avg">{{ "%.2f"|format(feeds.by_format.rss.avg_duration_ms|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">ATOM Avg Time</span>
<span class="metric-detail-value" id="feed-atom-avg">{{ "%.2f"|format(feeds.by_format.atom.avg_duration_ms|default(0)) }} ms</span>
</div>
<div class="metric-detail">
<span class="metric-detail-label">JSON Avg Time</span>
<span class="metric-detail-value" id="feed-json-avg">{{ "%.2f"|format(feeds.by_format.json.avg_duration_ms|default(0)) }} ms</span>
</div>
</div>
</div>
<!-- Feed Charts -->
<div class="metrics-grid">
<div class="metric-card">
<h3>Format Popularity</h3>
<div class="chart-container">
<canvas id="feedFormatChart"></canvas>
</div>
</div>
<div class="metric-card">
<h3>Cache Efficiency</h3>
<div class="chart-container">
<canvas id="feedCacheChart"></canvas>
</div>
</div>
</div>
<div class="refresh-info">
Auto-refresh every 10 seconds (requires JavaScript)
</div>
</div>
<script>
// Initialize charts with current data
let poolChart, performanceChart, feedFormatChart, feedCacheChart;
function initCharts() {
// Pool usage chart (doughnut)
const poolCtx = document.getElementById('poolChart');
if (poolCtx && !poolChart) {
poolChart = new Chart(poolCtx, {
type: 'doughnut',
data: {
labels: ['Active', 'Idle'],
datasets: [{
data: [
{{ pool.active_connections|default(0) }},
{{ pool.idle_connections|default(0) }}
],
backgroundColor: ['#007bff', '#6c757d'],
borderWidth: 1
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
plugins: {
legend: {
position: 'bottom'
},
title: {
display: true,
text: 'Connection Distribution'
}
}
}
});
}
// Performance chart (bar)
const perfCtx = document.getElementById('performanceChart');
if (perfCtx && !performanceChart) {
performanceChart = new Chart(perfCtx, {
type: 'bar',
data: {
labels: ['Database', 'HTTP', 'Render'],
datasets: [{
label: 'Average Time (ms)',
data: [
{{ metrics.database.avg|default(0) }},
{{ metrics.http.avg|default(0) }},
{{ metrics.render.avg|default(0) }}
],
backgroundColor: ['#007bff', '#28a745', '#ffc107'],
borderWidth: 1
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
scales: {
y: {
beginAtZero: true,
title: {
display: true,
text: 'Milliseconds'
}
}
},
plugins: {
legend: {
display: false
},
title: {
display: true,
text: 'Average Response Times'
}
}
}
});
}
// Feed format chart (pie)
const feedFormatCtx = document.getElementById('feedFormatChart');
if (feedFormatCtx && !feedFormatChart) {
feedFormatChart = new Chart(feedFormatCtx, {
type: 'pie',
data: {
labels: ['RSS', 'ATOM', 'JSON Feed'],
datasets: [{
data: [
{{ feeds.by_format.rss.total|default(0) }},
{{ feeds.by_format.atom.total|default(0) }},
{{ feeds.by_format.json.total|default(0) }}
],
backgroundColor: ['#ff6384', '#36a2eb', '#ffce56'],
borderWidth: 1
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
plugins: {
legend: {
position: 'bottom'
},
title: {
display: true,
text: 'Feed Format Distribution'
}
}
}
});
}
// Feed cache chart (doughnut)
const feedCacheCtx = document.getElementById('feedCacheChart');
if (feedCacheCtx && !feedCacheChart) {
feedCacheChart = new Chart(feedCacheCtx, {
type: 'doughnut',
data: {
labels: ['Cache Hits', 'Cache Misses'],
datasets: [{
data: [
{{ feeds.cache.hits|default(0) }},
{{ feeds.cache.misses|default(0) }}
],
backgroundColor: ['#28a745', '#dc3545'],
borderWidth: 1
}]
},
options: {
responsive: true,
maintainAspectRatio: false,
plugins: {
legend: {
position: 'bottom'
},
title: {
display: true,
text: 'Cache Hit/Miss Ratio'
}
}
}
});
}
}
// Update dashboard with new data from htmx
function updateDashboard(event) {
if (!event.detail.xhr) return;
try {
const data = JSON.parse(event.detail.xhr.responseText);
// Update pool statistics
if (data.database && data.database.pool) {
const pool = data.database.pool;
document.getElementById('pool-active').textContent = pool.active_connections || 0;
document.getElementById('pool-idle').textContent = pool.idle_connections || 0;
document.getElementById('pool-total').textContent = pool.total_connections || 0;
document.getElementById('pool-size').textContent = pool.pool_size || 5;
// Update pool chart
if (poolChart) {
poolChart.data.datasets[0].data = [
pool.active_connections || 0,
pool.idle_connections || 0
];
poolChart.update();
}
}
// Update performance metrics
if (data.performance) {
const perf = data.performance;
// Database
if (perf.database) {
document.getElementById('db-total').textContent = perf.database.count || 0;
document.getElementById('db-avg').textContent = (perf.database.avg || 0).toFixed(2) + ' ms';
document.getElementById('db-min').textContent = (perf.database.min || 0).toFixed(2) + ' ms';
document.getElementById('db-max').textContent = (perf.database.max || 0).toFixed(2) + ' ms';
}
// HTTP
if (perf.http) {
document.getElementById('http-total').textContent = perf.http.count || 0;
document.getElementById('http-avg').textContent = (perf.http.avg || 0).toFixed(2) + ' ms';
document.getElementById('http-min').textContent = (perf.http.min || 0).toFixed(2) + ' ms';
document.getElementById('http-max').textContent = (perf.http.max || 0).toFixed(2) + ' ms';
}
// Render
if (perf.render) {
document.getElementById('render-total').textContent = perf.render.count || 0;
document.getElementById('render-avg').textContent = (perf.render.avg || 0).toFixed(2) + ' ms';
document.getElementById('render-min').textContent = (perf.render.min || 0).toFixed(2) + ' ms';
document.getElementById('render-max').textContent = (perf.render.max || 0).toFixed(2) + ' ms';
}
// Update performance chart
if (performanceChart && perf.database && perf.http && perf.render) {
performanceChart.data.datasets[0].data = [
perf.database.avg || 0,
perf.http.avg || 0,
perf.render.avg || 0
];
performanceChart.update();
}
}
// Update feed statistics
if (data.feeds) {
const feeds = data.feeds;
// Feed requests by format
if (feeds.by_format) {
document.getElementById('feed-rss-total').textContent = feeds.by_format.rss?.total || 0;
document.getElementById('feed-atom-total').textContent = feeds.by_format.atom?.total || 0;
document.getElementById('feed-json-total').textContent = feeds.by_format.json?.total || 0;
document.getElementById('feed-total').textContent = feeds.total_requests || 0;
// Feed generation performance
document.getElementById('feed-rss-avg').textContent = (feeds.by_format.rss?.avg_duration_ms || 0).toFixed(2) + ' ms';
document.getElementById('feed-atom-avg').textContent = (feeds.by_format.atom?.avg_duration_ms || 0).toFixed(2) + ' ms';
document.getElementById('feed-json-avg').textContent = (feeds.by_format.json?.avg_duration_ms || 0).toFixed(2) + ' ms';
// Update feed format chart
if (feedFormatChart) {
feedFormatChart.data.datasets[0].data = [
feeds.by_format.rss?.total || 0,
feeds.by_format.atom?.total || 0,
feeds.by_format.json?.total || 0
];
feedFormatChart.update();
}
}
// Feed cache statistics
if (feeds.cache) {
document.getElementById('feed-cache-hits').textContent = feeds.cache.hits || 0;
document.getElementById('feed-cache-misses').textContent = feeds.cache.misses || 0;
document.getElementById('feed-cache-hit-rate').textContent = ((feeds.cache.hit_rate || 0) * 100).toFixed(1) + '%';
document.getElementById('feed-cache-entries').textContent = feeds.cache.entries || 0;
// Update feed cache chart
if (feedCacheChart) {
feedCacheChart.data.datasets[0].data = [
feeds.cache.hits || 0,
feeds.cache.misses || 0
];
feedCacheChart.update();
}
}
}
} catch (e) {
console.error('Error updating dashboard:', e);
}
}
// Initialize charts when DOM is ready
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', initCharts);
} else {
initCharts();
}
</script>
{% endblock %}
-108
View File
@@ -1,108 +0,0 @@
"""
Integration tests for feed statistics in admin dashboard
Tests the feed statistics features in /admin/metrics-dashboard and /admin/metrics
per v1.1.2 Phase 3.
"""
import pytest
from starpunk.auth import create_session
@pytest.fixture
def authenticated_client(app, client):
"""Client with authenticated session"""
with app.test_request_context():
# Create a session for the test user
session_token = create_session(app.config["ADMIN_ME"])
# Set session cookie
client.set_cookie("starpunk_session", session_token)
return client
def test_feed_statistics_dashboard_endpoint(authenticated_client):
"""Test metrics dashboard includes feed statistics section"""
response = authenticated_client.get("/admin/metrics-dashboard")
assert response.status_code == 200
# Should contain feed statistics section
assert b"Feed Statistics" in response.data
assert b"Feed Requests by Format" in response.data
assert b"Feed Cache Statistics" in response.data
assert b"Feed Generation Performance" in response.data
# Should have chart canvases
assert b'id="feedFormatChart"' in response.data
assert b'id="feedCacheChart"' in response.data
def test_feed_statistics_metrics_endpoint(authenticated_client):
"""Test /admin/metrics endpoint includes feed statistics"""
response = authenticated_client.get("/admin/metrics")
assert response.status_code == 200
data = response.get_json()
# Should have feeds key
assert "feeds" in data
# Should have expected structure
feeds = data["feeds"]
if "error" not in feeds:
assert "by_format" in feeds
assert "cache" in feeds
assert "total_requests" in feeds
assert "format_percentages" in feeds
# Check format structure
for format_name in ["rss", "atom", "json"]:
assert format_name in feeds["by_format"]
fmt = feeds["by_format"][format_name]
assert "generated" in fmt
assert "cached" in fmt
assert "total" in fmt
assert "avg_duration_ms" in fmt
# Check cache structure
assert "hits" in feeds["cache"]
assert "misses" in feeds["cache"]
assert "hit_rate" in feeds["cache"]
def test_feed_statistics_after_feed_request(authenticated_client):
"""Test feed statistics track actual feed requests"""
# Make a feed request
response = authenticated_client.get("/feed.rss")
assert response.status_code == 200
# Check metrics endpoint now has data
response = authenticated_client.get("/admin/metrics")
assert response.status_code == 200
data = response.get_json()
# Should have feeds data
assert "feeds" in data
feeds = data["feeds"]
# May have requests tracked (depends on metrics buffer timing)
# Just verify structure is correct
assert "total_requests" in feeds
assert feeds["total_requests"] >= 0
def test_dashboard_requires_auth_for_feed_stats(client):
"""Test dashboard requires authentication (even for feed stats)"""
response = client.get("/admin/metrics-dashboard")
# Should redirect to auth or return 401/403
assert response.status_code in [302, 401, 403]
def test_metrics_endpoint_requires_auth_for_feed_stats(client):
"""Test metrics endpoint requires authentication"""
response = client.get("/admin/metrics")
# Should redirect to auth or return 401/403
assert response.status_code in [302, 401, 403]
+12
View File
@@ -198,6 +198,18 @@ class TestQueries:
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["IndieWeb", "python"]
assert client.get("/micropub?q=category&filter=py", headers=h).get_json()["categories"] == ["python"]
def test_category_skips_unused_tags(self, client, h):
"""Tags dropped by an edit or on deleted posts aren't offered (v2.12.1)"""
create(client, h, category=["kept"])
retagged = create(client, h, category=["dropped"])
deleted = create(client, h, category=["trashed"])
body = {"action": "update", "url": retagged, "replace": {"category": ["kept"]}}
assert client.post("/micropub", json=body, headers=h).status_code in (200, 204)
assert client.post("/micropub", json={"action": "delete", "url": deleted}, headers=h).status_code == 204
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["kept"]
assert client.post("/micropub", json={"action": "undelete", "url": deleted}, headers=h).status_code < 300
assert client.get("/micropub?q=category", headers=h).get_json()["categories"] == ["kept", "trashed"]
def test_config_lists_queries(self, client, h):
config = client.get("/micropub?q=config", headers=h).get_json()
assert "category" in config["q"] and "media-endpoint" in config
-459
View File
@@ -1,459 +0,0 @@
"""
Tests for metrics instrumentation (v1.1.2 Phase 1)
Tests database monitoring, HTTP metrics, memory monitoring, and business metrics.
"""
import pytest
import sqlite3
import time
import threading
from unittest.mock import Mock, patch, MagicMock
from starpunk.monitoring import (
MonitoredConnection,
MemoryMonitor,
get_metrics,
get_metrics_stats,
business,
)
from starpunk.monitoring.metrics import get_buffer
from starpunk.monitoring.http import setup_http_metrics
class TestMonitoredConnection:
"""Tests for database operation monitoring"""
def test_execute_records_metric(self):
"""Test that execute() records a metric"""
# Create in-memory database
conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE test (id INTEGER, name TEXT)')
# Wrap with monitoring
monitored = MonitoredConnection(conn, slow_query_threshold=1.0)
# Clear metrics buffer
get_buffer().clear()
# Execute query
monitored.execute('SELECT * FROM test')
# Check metric was recorded
metrics = get_metrics()
# Note: May not be recorded due to sampling, but slow queries are forced
# So we'll check stats instead
stats = get_metrics_stats()
assert stats['total_count'] >= 0 # May be 0 due to sampling
def test_slow_query_always_recorded(self):
"""Test that slow queries are always recorded regardless of sampling"""
# Create in-memory database
conn = sqlite3.connect(':memory:')
# Set very low threshold so any query is "slow"
monitored = MonitoredConnection(conn, slow_query_threshold=0.0)
# Clear metrics buffer
get_buffer().clear()
# Execute query (will be considered slow)
monitored.execute('SELECT 1')
# Check metric was recorded (forced due to being slow)
metrics = get_metrics()
assert len(metrics) > 0
# Check that is_slow is True in metadata
assert any(m.metadata.get('is_slow', False) is True for m in metrics)
def test_extract_table_name_select(self):
"""Test table name extraction from SELECT query"""
conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE notes (id INTEGER)')
monitored = MonitoredConnection(conn)
table_name = monitored._extract_table_name('SELECT * FROM notes WHERE id = 1')
assert table_name == 'notes'
def test_extract_table_name_insert(self):
"""Test table name extraction from INSERT query"""
conn = sqlite3.connect(':memory:')
monitored = MonitoredConnection(conn)
table_name = monitored._extract_table_name('INSERT INTO users (name) VALUES (?)')
assert table_name == 'users'
def test_extract_table_name_update(self):
"""Test table name extraction from UPDATE query"""
conn = sqlite3.connect(':memory:')
monitored = MonitoredConnection(conn)
table_name = monitored._extract_table_name('UPDATE posts SET title = ?')
assert table_name == 'posts'
def test_extract_table_name_unknown(self):
"""Test that complex queries return 'unknown'"""
conn = sqlite3.connect(':memory:')
monitored = MonitoredConnection(conn)
# Complex query with JOIN
table_name = monitored._extract_table_name(
'SELECT a.* FROM notes a JOIN users b ON a.user_id = b.id'
)
# Our simple regex will find 'notes' from the first FROM
assert table_name in ['notes', 'unknown']
def test_get_query_type(self):
"""Test query type extraction"""
conn = sqlite3.connect(':memory:')
monitored = MonitoredConnection(conn)
assert monitored._get_query_type('SELECT * FROM notes') == 'SELECT'
assert monitored._get_query_type('INSERT INTO notes VALUES (?)') == 'INSERT'
assert monitored._get_query_type('UPDATE notes SET x = 1') == 'UPDATE'
assert monitored._get_query_type('DELETE FROM notes') == 'DELETE'
assert monitored._get_query_type('CREATE TABLE test (id INT)') == 'CREATE'
assert monitored._get_query_type('PRAGMA journal_mode=WAL') == 'PRAGMA'
def test_execute_with_parameters(self):
"""Test execute with query parameters"""
conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE test (id INTEGER, name TEXT)')
monitored = MonitoredConnection(conn, slow_query_threshold=1.0)
# Execute with parameters
monitored.execute('INSERT INTO test (id, name) VALUES (?, ?)', (1, 'test'))
# Verify data was inserted
cursor = monitored.execute('SELECT * FROM test WHERE id = ?', (1,))
rows = cursor.fetchall()
assert len(rows) == 1
def test_executemany(self):
"""Test executemany batch operations"""
conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE test (id INTEGER, name TEXT)')
monitored = MonitoredConnection(conn)
# Clear metrics
get_buffer().clear()
# Execute batch insert
data = [(1, 'first'), (2, 'second'), (3, 'third')]
monitored.executemany('INSERT INTO test (id, name) VALUES (?, ?)', data)
# Check metric was recorded
metrics = get_metrics()
# May not be recorded due to sampling
stats = get_metrics_stats()
assert stats is not None
def test_error_recording(self):
"""Test that errors are recorded in metrics"""
conn = sqlite3.connect(':memory:')
monitored = MonitoredConnection(conn)
# Clear metrics
get_buffer().clear()
# Execute invalid query
with pytest.raises(sqlite3.OperationalError):
monitored.execute('SELECT * FROM nonexistent_table')
# Check error was recorded (forced)
metrics = get_metrics()
assert len(metrics) > 0
assert any('ERROR' in m.operation_name for m in metrics)
class TestHTTPMetrics:
"""Tests for HTTP request/response monitoring"""
def test_setup_http_metrics(self, app):
"""Test HTTP metrics middleware setup"""
# Add a simple test route
@app.route('/test')
def test_route():
return 'OK', 200
setup_http_metrics(app)
# Clear metrics
get_buffer().clear()
# Make a request
with app.test_client() as client:
response = client.get('/test')
assert response.status_code == 200
# Check request ID header was added
assert 'X-Request-ID' in response.headers
# Check metrics were recorded
metrics = get_metrics()
# May be sampled, so just check structure
stats = get_metrics_stats()
assert stats is not None
def test_request_id_generation(self, app):
"""Test that unique request IDs are generated"""
# Add a simple test route
@app.route('/test')
def test_route():
return 'OK', 200
setup_http_metrics(app)
request_ids = set()
with app.test_client() as client:
for _ in range(5):
response = client.get('/test')
request_id = response.headers.get('X-Request-ID')
assert request_id is not None
request_ids.add(request_id)
# All request IDs should be unique
assert len(request_ids) == 5
def test_error_metrics_recorded(self, app):
"""Test that errors are recorded in metrics"""
# Add a simple test route
@app.route('/test')
def test_route():
return 'OK', 200
setup_http_metrics(app)
# Clear metrics
get_buffer().clear()
with app.test_client() as client:
# Request non-existent endpoint
response = client.get('/this-does-not-exist')
assert response.status_code == 404
# Error metrics should be recorded (forced)
# Note: 404 is not necessarily an error in the teardown handler
# but will be in metrics as a 404 status code
metrics = get_metrics()
stats = get_metrics_stats()
assert stats is not None
class TestMemoryMonitor:
"""Tests for memory monitoring thread"""
def test_memory_monitor_initialization(self):
"""Test memory monitor can be initialized"""
monitor = MemoryMonitor(interval=1)
assert monitor.interval == 1
assert monitor.daemon is True # Per CQ5
def test_memory_monitor_starts_and_stops(self):
"""Test memory monitor thread lifecycle"""
monitor = MemoryMonitor(interval=1)
# Start monitor
monitor.start()
assert monitor.is_alive()
# Wait a bit for initialization
time.sleep(0.5)
# Stop monitor gracefully
monitor.stop()
# Give it time to finish gracefully
time.sleep(1.0)
monitor.join(timeout=5)
# Thread should have stopped
# Note: In rare cases daemon thread may still be cleaning up
if monitor.is_alive():
# Give it one more second
time.sleep(1.0)
assert not monitor.is_alive()
def test_memory_monitor_collects_metrics(self):
"""Test that memory monitor collects metrics"""
# Clear metrics
get_buffer().clear()
monitor = MemoryMonitor(interval=1)
monitor.start()
# Wait for baseline + one collection
time.sleep(7) # 5s baseline + 2s for collection
# Stop monitor
monitor.stop()
monitor.join(timeout=2)
# Check metrics were collected
metrics = get_metrics()
memory_metrics = [m for m in metrics if 'memory' in m.operation_name.lower()]
# Should have at least one memory metric
assert len(memory_metrics) > 0
def test_memory_monitor_stats(self):
"""Test memory monitor statistics"""
monitor = MemoryMonitor(interval=1)
monitor.start()
# Wait for baseline
time.sleep(6)
# Get stats
stats = monitor.get_stats()
assert stats['status'] == 'running'
assert 'current_rss_mb' in stats
assert 'baseline_rss_mb' in stats
assert stats['baseline_rss_mb'] > 0
monitor.stop()
monitor.join(timeout=2)
class TestBusinessMetrics:
"""Tests for business metrics tracking"""
def test_track_note_created(self):
"""Test note creation tracking"""
get_buffer().clear()
business.track_note_created(note_id=123, content_length=500, has_media=False)
metrics = get_metrics()
assert len(metrics) > 0
note_metrics = [m for m in metrics if 'note_created' in m.operation_name]
assert len(note_metrics) > 0
assert note_metrics[0].metadata['note_id'] == 123
assert note_metrics[0].metadata['content_length'] == 500
def test_track_note_updated(self):
"""Test note update tracking"""
get_buffer().clear()
business.track_note_updated(
note_id=456,
content_length=750,
fields_changed=['title', 'content']
)
metrics = get_metrics()
note_metrics = [m for m in metrics if 'note_updated' in m.operation_name]
assert len(note_metrics) > 0
assert note_metrics[0].metadata['note_id'] == 456
def test_track_note_deleted(self):
"""Test note deletion tracking"""
get_buffer().clear()
business.track_note_deleted(note_id=789)
metrics = get_metrics()
note_metrics = [m for m in metrics if 'note_deleted' in m.operation_name]
assert len(note_metrics) > 0
assert note_metrics[0].metadata['note_id'] == 789
def test_track_feed_generated(self):
"""Test feed generation tracking"""
get_buffer().clear()
business.track_feed_generated(
format='rss',
item_count=50,
duration_ms=45.2,
cached=False
)
metrics = get_metrics()
feed_metrics = [m for m in metrics if 'feed_rss' in m.operation_name]
assert len(feed_metrics) > 0
assert feed_metrics[0].metadata['format'] == 'rss'
assert feed_metrics[0].metadata['item_count'] == 50
def test_track_cache_hit(self):
"""Test cache hit tracking"""
get_buffer().clear()
business.track_cache_hit(cache_type='feed', key='rss:latest')
metrics = get_metrics()
cache_metrics = [m for m in metrics if 'cache_hit' in m.operation_name]
assert len(cache_metrics) > 0
def test_track_cache_miss(self):
"""Test cache miss tracking"""
get_buffer().clear()
business.track_cache_miss(cache_type='feed', key='atom:latest')
metrics = get_metrics()
cache_metrics = [m for m in metrics if 'cache_miss' in m.operation_name]
assert len(cache_metrics) > 0
class TestMetricsConfiguration:
"""Tests for metrics configuration"""
def test_metrics_can_be_disabled(self, app):
"""Test that metrics can be disabled via configuration"""
# This would be tested by setting METRICS_ENABLED=False
# and verifying no metrics are collected
assert 'METRICS_ENABLED' in app.config
def test_slow_query_threshold_configurable(self, app):
"""Test that slow query threshold is configurable"""
assert 'METRICS_SLOW_QUERY_THRESHOLD' in app.config
assert isinstance(app.config['METRICS_SLOW_QUERY_THRESHOLD'], float)
def test_sampling_rate_configurable(self, app):
"""Test that sampling rate is configurable"""
assert 'METRICS_SAMPLING_RATE' in app.config
assert isinstance(app.config['METRICS_SAMPLING_RATE'], float)
assert 0.0 <= app.config['METRICS_SAMPLING_RATE'] <= 1.0
def test_buffer_size_configurable(self, app):
"""Test that buffer size is configurable"""
assert 'METRICS_BUFFER_SIZE' in app.config
assert isinstance(app.config['METRICS_BUFFER_SIZE'], int)
assert app.config['METRICS_BUFFER_SIZE'] > 0
def test_memory_interval_configurable(self, app):
"""Test that memory monitor interval is configurable"""
assert 'METRICS_MEMORY_INTERVAL' in app.config
assert isinstance(app.config['METRICS_MEMORY_INTERVAL'], int)
assert app.config['METRICS_MEMORY_INTERVAL'] > 0
@pytest.fixture
def app():
"""Create test Flask app with minimal configuration"""
from flask import Flask
from pathlib import Path
import tempfile
app = Flask(__name__)
# Create temp directory for testing
temp_dir = tempfile.mkdtemp()
temp_path = Path(temp_dir)
# Minimal configuration to avoid migration issues
app.config.update({
'TESTING': True,
'DATABASE_PATH': temp_path / 'test.db',
'DATA_PATH': temp_path,
'NOTES_PATH': temp_path / 'notes',
'SESSION_SECRET': 'test-secret',
'ADMIN_ME': 'https://test.example.com',
'METRICS_ENABLED': True,
'METRICS_SLOW_QUERY_THRESHOLD': 1.0,
'METRICS_SAMPLING_RATE': 1.0,
'METRICS_BUFFER_SIZE': 1000,
'METRICS_MEMORY_INTERVAL': 30,
})
return app
-103
View File
@@ -1,103 +0,0 @@
"""
Tests for feed statistics tracking
Tests feed statistics aggregation per v1.1.2 Phase 3.
"""
import pytest
from starpunk.monitoring.business import get_feed_statistics, track_feed_generated
def test_get_feed_statistics_returns_structure():
"""Test get_feed_statistics returns expected structure"""
stats = get_feed_statistics()
# Check top-level keys
assert "by_format" in stats
assert "cache" in stats
assert "total_requests" in stats
assert "format_percentages" in stats
# Check by_format structure
assert "rss" in stats["by_format"]
assert "atom" in stats["by_format"]
assert "json" in stats["by_format"]
# Check format stats structure
for format_name in ["rss", "atom", "json"]:
fmt_stats = stats["by_format"][format_name]
assert "generated" in fmt_stats
assert "cached" in fmt_stats
assert "total" in fmt_stats
assert "avg_duration_ms" in fmt_stats
# Check cache structure
assert "hits" in stats["cache"]
assert "misses" in stats["cache"]
assert "hit_rate" in stats["cache"]
def test_get_feed_statistics_empty_metrics():
"""Test get_feed_statistics with no metrics returns zeros"""
stats = get_feed_statistics()
# All values should be zero or empty
assert stats["total_requests"] >= 0
assert stats["cache"]["hit_rate"] >= 0.0
assert stats["cache"]["hit_rate"] <= 1.0
def test_feed_statistics_cache_hit_rate_calculation():
"""Test cache hit rate is calculated correctly"""
stats = get_feed_statistics()
# Hit rate should be between 0 and 1
assert 0.0 <= stats["cache"]["hit_rate"] <= 1.0
# If there are hits and misses, hit rate should be hits / (hits + misses)
if stats["cache"]["hits"] + stats["cache"]["misses"] > 0:
expected_rate = stats["cache"]["hits"] / (
stats["cache"]["hits"] + stats["cache"]["misses"]
)
assert abs(stats["cache"]["hit_rate"] - expected_rate) < 0.001
def test_feed_statistics_format_percentages():
"""Test format percentages sum to 1.0 when there are requests"""
stats = get_feed_statistics()
if stats["total_requests"] > 0:
total_percentage = sum(stats["format_percentages"].values())
# Should sum to approximately 1.0 (allowing for floating point errors)
assert abs(total_percentage - 1.0) < 0.001
def test_feed_statistics_total_requests_sum():
"""Test total_requests equals sum of all format totals"""
stats = get_feed_statistics()
format_total = sum(
fmt["total"] for fmt in stats["by_format"].values()
)
assert stats["total_requests"] == format_total
def test_track_feed_generated_records_metrics():
"""Test track_feed_generated creates metrics entries"""
# Note: This test just verifies the function runs without error.
# Actual metrics tracking is tested in integration tests.
track_feed_generated(
format="rss",
item_count=10,
duration_ms=50.5,
cached=False
)
# Get statistics - may be empty if metrics buffer hasn't persisted yet
stats = get_feed_statistics()
# Verify structure is correct
assert "total_requests" in stats
assert "by_format" in stats
assert "cache" in stats