Merge branch 'chore/remove-metrics': remove metrics dashboard (v2.12.0)
Build Container / build (push) Successful in 7m53s
Build Container / build (push) Successful in 7m53s
This commit is contained in:
commit
13936749de
36 files changed
+153
-2963
No files matched your search
+1
-11
@@ -199,19 +199,9 @@ THREADS_USERNAME=
|
|||||||
# SYNDICATION_POLL_INTERVAL=10
|
# SYNDICATION_POLL_INTERVAL=10
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# MONITORING
|
# DEBUGGING
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
||||||
# Metrics shown on /admin/metrics-dashboard (default: true)
|
|
||||||
# METRICS_ENABLED=true
|
|
||||||
# Queries slower than this many seconds are logged (default: 1.0)
|
|
||||||
# METRICS_SLOW_QUERY_THRESHOLD=1.0
|
|
||||||
# Fraction of operations recorded, 0.0-1.0 (default: 1.0)
|
|
||||||
# METRICS_SAMPLING_RATE=1.0
|
|
||||||
# METRICS_BUFFER_SIZE=1000
|
|
||||||
# Seconds between memory samples (default: 30)
|
|
||||||
# METRICS_MEMORY_INTERVAL=30
|
|
||||||
|
|
||||||
# Keep uploads that fail processing in DATA_PATH/debug for diagnosis
|
# Keep uploads that fail processing in DATA_PATH/debug for diagnosis
|
||||||
# (default: false); cleaned up by age (default: 7 days) and total size
|
# (default: false); cleaned up by age (default: 7 days) and total size
|
||||||
# (default: 100 MB)
|
# (default: 100 MB)
|
||||||
|
|||||||
@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [2.12.0] - 2026-11-03
|
||||||
|
|
||||||
|
The metrics dashboard is gone (ADR-072).
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
|
||||||
|
- The admin metrics dashboard (`/admin/metrics-dashboard`, the "Metrics" admin link) and its JSON endpoint `/admin/metrics`. Its numbers were kept in memory per server process, so they covered a quarter of the traffic and vanished on every restart
|
||||||
|
- Everything that only fed it: request and database-query timing, the memory monitor thread, feed-generation counters (`starpunk/monitoring/`), and the `X-Request-ID` response header. Slow queries are no longer logged
|
||||||
|
- The `METRICS_ENABLED`, `METRICS_SLOW_QUERY_THRESHOLD`, `METRICS_SAMPLING_RATE`, `METRICS_BUFFER_SIZE` and `METRICS_MEMORY_INTERVAL` settings (ignored if still set), and the `psutil` dependency
|
||||||
|
- The `metrics` section of `/admin/health`; its database, pool, filesystem and disk checks are unchanged, as is `/health`
|
||||||
|
|
||||||
## [2.11.0] - 2026-11-03
|
## [2.11.0] - 2026-11-03
|
||||||
|
|
||||||
Maintenance release: dependency upgrades and the High-priority backlog items.
|
Maintenance release: dependency upgrades and the High-priority backlog items.
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ what changed in each release is in [CHANGELOG.md](CHANGELOG.md).
|
|||||||
|
|
||||||
**Operations**
|
**Operations**
|
||||||
- Migrations applied automatically at startup
|
- Migrations applied automatically at startup
|
||||||
- `/health` endpoint, admin metrics dashboard, structured logs
|
- `/health` endpoint, `/admin/health` diagnostics, structured logs
|
||||||
- Fetches of user-supplied URLs are limited to public addresses, size and
|
- Fetches of user-supplied URLs are limited to public addresses, size and
|
||||||
time
|
time
|
||||||
|
|
||||||
|
|||||||
@@ -218,8 +218,7 @@ server process enforces one shared limit. Old windows are pruned as it goes.
|
|||||||
timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the
|
timeout; `DB_POOL_SIZE` and `DB_TIMEOUT` can be set only through the
|
||||||
`create_app()` config dict). Connections use `sqlite3.Row` rows,
|
`create_app()` config dict). Connections use `sqlite3.Row` rows,
|
||||||
`check_same_thread=False`, `PRAGMA journal_mode=WAL` and (since v2.11.0)
|
`check_same_thread=False`, `PRAGMA journal_mode=WAL` and (since v2.11.0)
|
||||||
`PRAGMA foreign_keys=ON`, so `ON DELETE CASCADE`/`SET NULL` run, and are wrapped
|
`PRAGMA foreign_keys=ON`, so `ON DELETE CASCADE`/`SET NULL` run. `get_db()`
|
||||||
in `MonitoredConnection` for query timing when metrics are on. `get_db()`
|
|
||||||
stores one connection in `flask.g` per request or app context; on teardown
|
stores one connection in `flask.g` per request or app context; on teardown
|
||||||
it goes back to the pool, rolled back first if a transaction was left
|
it goes back to the pool, rolled back first if a transaction was left
|
||||||
open.
|
open.
|
||||||
|
|||||||
@@ -235,15 +235,12 @@ starts only if at least one is.
|
|||||||
| `THREADS_MAX_MEDIA` | `4` | |
|
| `THREADS_MAX_MEDIA` | `4` | |
|
||||||
| `SYNDICATION_POLL_INTERVAL` | `10` | Seconds |
|
| `SYNDICATION_POLL_INTERVAL` | `10` | Seconds |
|
||||||
|
|
||||||
### Metrics and debugging
|
### Debugging
|
||||||
|
|
||||||
|
The `METRICS_*` variables were removed in v2.12.0 and are ignored if set.
|
||||||
|
|
||||||
| Variable | Default | Notes |
|
| Variable | Default | Notes |
|
||||||
|----------|---------|-------|
|
|----------|---------|-------|
|
||||||
| `METRICS_ENABLED` | `true` | HTTP/database metrics and the memory monitor |
|
|
||||||
| `METRICS_SLOW_QUERY_THRESHOLD` | `1.0` | Seconds |
|
|
||||||
| `METRICS_SAMPLING_RATE` | `1.0` | |
|
|
||||||
| `METRICS_BUFFER_SIZE` | `1000` | |
|
|
||||||
| `METRICS_MEMORY_INTERVAL` | `30` | Seconds |
|
|
||||||
| `DEBUG_SAVE_FAILED_UPLOADS` | `false` | Keep failed uploads in `debug/` |
|
| `DEBUG_SAVE_FAILED_UPLOADS` | `false` | Keep failed uploads in `debug/` |
|
||||||
| `DEBUG_FILE_MAX_AGE_DAYS` | `7` | |
|
| `DEBUG_FILE_MAX_AGE_DAYS` | `7` | |
|
||||||
| `DEBUG_FILE_MAX_SIZE_MB` | `100` | |
|
| `DEBUG_FILE_MAX_SIZE_MB` | `100` | |
|
||||||
|
|||||||
@@ -43,8 +43,6 @@ Key decisions:
|
|||||||
[ADR-058](../decisions/ADR-058-image-optimization-strategy.md) (media),
|
[ADR-058](../decisions/ADR-058-image-optimization-strategy.md) (media),
|
||||||
[ADR-034](../decisions/ADR-034-full-text-search.md) (search),
|
[ADR-034](../decisions/ADR-034-full-text-search.md) (search),
|
||||||
[ADR-052](../decisions/ADR-052-configuration-system-architecture.md) (configuration),
|
[ADR-052](../decisions/ADR-052-configuration-system-architecture.md) (configuration),
|
||||||
[ADR-053](../decisions/ADR-053-performance-monitoring-strategy.md) and
|
|
||||||
[ADR-060](../decisions/ADR-060-production-hotfix-metrics-dashboard.md) (metrics),
|
|
||||||
[ADR-054](../decisions/ADR-054-structured-logging-architecture.md) (logging),
|
[ADR-054](../decisions/ADR-054-structured-logging-architecture.md) (logging),
|
||||||
[ADR-055](../decisions/ADR-055-error-handling-philosophy.md) (errors).
|
[ADR-055](../decisions/ADR-055-error-handling-philosophy.md) (errors).
|
||||||
|
|
||||||
@@ -65,7 +63,6 @@ Key decisions:
|
|||||||
│ webmention worker works its queues only while holding a lock │
|
│ webmention worker works its queues only while holding a lock │
|
||||||
│ websub worker works its queues only while holding a lock │
|
│ websub worker works its queues only while holding a lock │
|
||||||
│ syndication worker claims rows with a lease in SQLite │
|
│ syndication worker claims rows with a lease in SQLite │
|
||||||
│ memory monitor per-process metrics │
|
|
||||||
└───────────────┬──────────────────────────────┬─────────────────────┘
|
└───────────────┬──────────────────────────────┬─────────────────────┘
|
||||||
│ │
|
│ │
|
||||||
┌───────▼────────┐ ┌──────────▼──────────────────┐
|
┌───────▼────────┐ ┌──────────▼──────────────────┐
|
||||||
@@ -86,7 +83,7 @@ Key decisions:
|
|||||||
validates configuration, configures logging, cleans up old debug and
|
validates configuration, configures logging, cleans up old debug and
|
||||||
temporary upload files, creates the schema and runs migrations (under a
|
temporary upload files, creates the schema and runs migrations (under a
|
||||||
lock, see [database.md](database.md)), creates the connection pool,
|
lock, see [database.md](database.md)), creates the connection pool,
|
||||||
installs the HTTP metrics middleware, configures the feed cache and rate
|
configures the feed cache and rate
|
||||||
limiters, fills the FTS index if it is empty, registers blueprints, error
|
limiters, fills the FTS index if it is empty, registers blueprints, error
|
||||||
handlers and template helpers, and starts the background threads.
|
handlers and template helpers, and starts the background threads.
|
||||||
3. `before_request` assigns a correlation ID (included in every log line) and,
|
3. `before_request` assigns a correlation ID (included in every log line) and,
|
||||||
@@ -94,8 +91,7 @@ Key decisions:
|
|||||||
4. The blueprint handler runs. Database access goes through `get_db()`, which
|
4. The blueprint handler runs. Database access goes through `get_db()`, which
|
||||||
hands out a pooled, request-scoped SQLite connection.
|
hands out a pooled, request-scoped SQLite connection.
|
||||||
5. `after_request` hooks add `Link: </webmention>; rel="webmention"` to HTML
|
5. `after_request` hooks add `Link: </webmention>; rel="webmention"` to HTML
|
||||||
responses, WebSub `Link rel=hub` / `rel=self` headers to topic responses,
|
responses, WebSub `Link rel=hub` / `rel=self` headers to topic responses.
|
||||||
and request metrics with an `X-Request-ID` header.
|
|
||||||
6. Errors are rendered by the central handlers in `starpunk/errors.py`
|
6. Errors are rendered by the central handlers in `starpunk/errors.py`
|
||||||
(JSON for Micropub, HTML pages from `templates/<code>.html` elsewhere).
|
(JSON for Micropub, HTML pages from `templates/<code>.html` elsewhere).
|
||||||
|
|
||||||
@@ -141,7 +137,6 @@ wait for outbound network calls; the workers make them later.
|
|||||||
| `errors.py` | Central error handlers |
|
| `errors.py` | Central error handlers |
|
||||||
| `feeds/` | `rss.py` (feedgen), `atom.py`, `json_feed.py`, `opml.py`, `negotiation.py` (Accept header), `cache.py` (LRU + TTL feed cache with ETags) |
|
| `feeds/` | `rss.py` (feedgen), `atom.py`, `json_feed.py`, `opml.py`, `negotiation.py` (Accept header), `cache.py` (LRU + TTL feed cache with ETags) |
|
||||||
| `feed.py` | Backward-compatibility re-export of `feeds.rss` |
|
| `feed.py` | Backward-compatibility re-export of `feeds.rss` |
|
||||||
| `monitoring/` | Metrics buffer, HTTP middleware, database timing wrapper, memory monitor, business metrics |
|
|
||||||
| `database/` | `schema.py` (`INITIAL_SCHEMA_SQL`), `init.py` (`init_db`), `pool.py` (connection pool) |
|
| `database/` | `schema.py` (`INITIAL_SCHEMA_SQL`), `init.py` (`init_db`), `pool.py` (connection pool) |
|
||||||
| `migrations.py` | Migration runner for `migrations/NNN_*.sql` |
|
| `migrations.py` | Migration runner for `migrations/NNN_*.sql` |
|
||||||
|
|
||||||
@@ -196,7 +191,7 @@ Everything works without JavaScript.
|
|||||||
| `/admin/webmentions`, `/admin/webmentions/<id>`, `…/action`, `…/bulk-action` | Webmention moderation |
|
| `/admin/webmentions`, `/admin/webmentions/<id>`, `…/action`, `…/bulk-action` | Webmention moderation |
|
||||||
| `/admin/tokens`, `/admin/tokens/<id>/revoke` | Issued IndieAuth tokens |
|
| `/admin/tokens`, `/admin/tokens/<id>/revoke` | Issued IndieAuth tokens |
|
||||||
| `/admin/syndication/<id>/retry` | Retry a failed crosspost |
|
| `/admin/syndication/<id>/retry` | Retry a failed crosspost |
|
||||||
| `/admin/metrics-dashboard`, `/admin/metrics`, `/admin/health` | Monitoring (HTML dashboard, JSON metrics, full diagnostics) |
|
| `/admin/health` | Full diagnostics (JSON): database, pool statistics, filesystem, disk |
|
||||||
| `/health` | Public liveness check (`{"status": "ok", "version": …}`); `?detailed=true` requires login |
|
| `/health` | Public liveness check (`{"status": "ok", "version": …}`); `?detailed=true` requires login |
|
||||||
|
|
||||||
## Background workers
|
## Background workers
|
||||||
@@ -210,7 +205,6 @@ coordinated through the data volume:
|
|||||||
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries |
|
| Webmention worker (`webmention_worker.py`) | `WEBMENTION_WORKER_ENABLED` and not testing | Non-blocking `fcntl.flock` on `webmention-worker.lock` next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in `sending`. | Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries |
|
||||||
| WebSub worker (`websub_worker.py`) | `WEBSUB_ENABLED` and not testing | Same pattern with `websub-worker.lock` | Verifies subscription intents, delivers topic content (signed with `X-Hub-Signature` when the subscriber gave a secret), retries, expires leases; pings an external hub instead if `WEBSUB_HUB_URL` is set |
|
| WebSub worker (`websub_worker.py`) | `WEBSUB_ENABLED` and not testing | Same pattern with `websub-worker.lock` | Verifies subscription intents, delivers topic content (signed with `X-Hub-Signature` when the subscriber gave a secret), retries, expires leases; pings an external hub instead if `WEBSUB_HUB_URL` is set |
|
||||||
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts |
|
| Syndication worker (`syndication.py`) | At least one crosspost target configured, and not testing | No file lock. Rows in `syndications` are claimed with a 5-minute `claimed_until` lease by an atomic `UPDATE`, so any process may work a row but only one at a time; a `version` column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to `threads_token.json`. | Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts |
|
||||||
| Memory monitor (`monitoring/memory.py`) | `METRICS_ENABLED` and not testing | None (per-process metrics) | Samples process memory every `METRICS_MEMORY_INTERVAL` seconds |
|
|
||||||
|
|
||||||
All workers run inside an app context, are daemon threads (they end with the
|
All workers run inside an app context, are daemon threads (they end with the
|
||||||
process) and sleep for their poll interval when there is nothing to do. Under
|
process) and sleep for their poll interval when there is nothing to do. Under
|
||||||
@@ -326,14 +320,11 @@ their credentials are set. The full list with defaults is in
|
|||||||
|
|
||||||
- Logs go to stdout and `logs/starpunk.log` (10 MB × 10 files); each line
|
- Logs go to stdout and `logs/starpunk.log` (10 MB × 10 files); each line
|
||||||
carries the request's correlation ID.
|
carries the request's correlation ID.
|
||||||
- Metrics (`monitoring/`): HTTP request timings, database query timings
|
|
||||||
(with a slow-query threshold), memory, pool statistics and business
|
|
||||||
counters, in an in-memory buffer per process, shown at
|
|
||||||
`/admin/metrics-dashboard` and `/admin/metrics`.
|
|
||||||
- `/health` reports status, version and webmention-worker statistics;
|
- `/health` reports status, version and webmention-worker statistics;
|
||||||
`/health?detailed=true` checks database, filesystem and disk space;
|
`/health?detailed=true` checks database, filesystem and disk space;
|
||||||
`/admin/health` adds metrics, pool statistics and system information.
|
`/admin/health` adds pool statistics and filesystem details. (The metrics
|
||||||
|
dashboard was removed in v2.12.0, see ADR-072.)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
**Last Updated**: 2026-11-03 (v2.10.1)
|
**Last Updated**: 2026-11-03 (v2.12.0)
|
||||||
@@ -47,7 +47,6 @@ vulnerabilities (`pip-audit`); check again before each release.
|
|||||||
| python-dotenv | `1.2.*` | Loading `.env` |
|
| python-dotenv | `1.2.*` | Loading `.env` |
|
||||||
| Pillow | `13.0.*` | Image validation, resizing and variants |
|
| Pillow | `13.0.*` | Image validation, resizing and variants |
|
||||||
| pillow-heif | `1.8.*` | HEIC/HEIF uploads (iPhone photos) |
|
| pillow-heif | `1.8.*` | HEIC/HEIF uploads (iPhone photos) |
|
||||||
| psutil | `7.2.*` | Memory monitor and system metrics |
|
|
||||||
|
|
||||||
`markupsafe` comes with Flask/Jinja2.
|
`markupsafe` comes with Flask/Jinja2.
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
# ADR-053: Performance Monitoring Strategy
|
# ADR-053: Performance Monitoring Strategy
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
Accepted
|
Superseded by ADR-072 (was: Accepted, v1.1.1)
|
||||||
|
|
||||||
|
> The metrics system was removed in v2.12.0. (status updated 2026-11-03)
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
StarPunk v1.1.1 introduces performance monitoring to help operators understand system behavior in production. Currently, we have no visibility into:
|
StarPunk v1.1.1 introduces performance monitoring to help operators understand system behavior in production. Currently, we have no visibility into:
|
||||||
|
|||||||
@@ -2,7 +2,9 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Accepted (v1.1.1-rc.2, 2025-11-25)
|
Superseded by ADR-072 (was: Accepted, v1.1.1-rc.2, 2025-11-25)
|
||||||
|
|
||||||
|
> The metrics dashboard was removed in v2.12.0. (status updated 2026-11-03)
|
||||||
|
|
||||||
> Reconstructed on 2026-11-03 from code, changelog and related documents;
|
> Reconstructed on 2026-11-03 from code, changelog and related documents;
|
||||||
> the original was lost. The original
|
> the original was lost. The original
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# ADR-072: Remove the metrics dashboard
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted (v2.12.0, 2026-11-03). Supersedes ADR-053 and ADR-060.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
v1.1.1 and v1.1.2 added built-in performance monitoring (ADR-053): HTTP
|
||||||
|
request timing middleware, a wrapper on every database connection that
|
||||||
|
timed queries, a memory-sampling thread, feed-generation counters, an
|
||||||
|
in-memory buffer of the results, and an HTML dashboard at
|
||||||
|
`/admin/metrics-dashboard` with a JSON twin at `/admin/metrics` (ADR-060
|
||||||
|
covered adapting the data for the dashboard template).
|
||||||
|
|
||||||
|
In practice it told us little:
|
||||||
|
|
||||||
|
- The buffer lives in memory in each of the four gunicorn processes, so the
|
||||||
|
dashboard showed whichever process served the request, and everything was
|
||||||
|
lost on restart or deploy.
|
||||||
|
- Nothing in it was ever used to find or fix a problem. Logs (with
|
||||||
|
correlation IDs), `/health` and `/admin/health` were.
|
||||||
|
- It cost something on every request and every query, kept a thread per
|
||||||
|
process, and needed `psutil`, about 1,300 lines of code and three test
|
||||||
|
files.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Remove the dashboard and everything that only fed it: the `starpunk/monitoring`
|
||||||
|
package, the HTTP metrics middleware (including its `X-Request-ID`
|
||||||
|
response header), `MonitoredConnection`, the memory monitor, the
|
||||||
|
feed-generation tracking calls, `/admin/metrics-dashboard`,
|
||||||
|
`/admin/metrics`, the `METRICS_*` settings and the `psutil` dependency.
|
||||||
|
|
||||||
|
Keep:
|
||||||
|
|
||||||
|
- `/health` and `/health?detailed=true`;
|
||||||
|
- `/admin/health`, without its `metrics` section (database, pool
|
||||||
|
statistics, filesystem and disk checks stay);
|
||||||
|
- connection pool statistics (`get_pool_stats`) and the feed cache, which
|
||||||
|
do work of their own;
|
||||||
|
- structured logs with correlation IDs (ADR-054).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Less code and one fewer dependency; no timing work on each request or
|
||||||
|
query; one fewer thread per process.
|
||||||
|
- Slow database queries are no longer logged. If that is ever needed, a
|
||||||
|
query timer can be added at the pool level, or an external tool used.
|
||||||
|
- `METRICS_*` environment variables are ignored if still set.
|
||||||
|
- Bookmarks to `/admin/metrics-dashboard` now get a 404.
|
||||||
@@ -5,8 +5,8 @@ records one decision: its context, what was decided, and the consequences.
|
|||||||
Superseded ADRs are kept as history; their Status line names the ADR that
|
Superseded ADRs are kept as history; their Status line names the ADR that
|
||||||
replaced them.
|
replaced them.
|
||||||
|
|
||||||
**Total ADR files**: 67 (66 numbers: ADR-001 to ADR-044 and ADR-050 to
|
**Total ADR files**: 68 (67 numbers: ADR-001 to ADR-044 and ADR-050 to
|
||||||
ADR-071, with two files numbered ADR-012). See [Numbering](#numbering).
|
ADR-072, with two files numbered ADR-012). See [Numbering](#numbering).
|
||||||
|
|
||||||
Status values: **Accepted** (in force), **Superseded** (replaced by a later
|
Status values: **Accepted** (in force), **Superseded** (replaced by a later
|
||||||
ADR), **Deprecated** (withdrawn, nothing replaces it; none at present),
|
ADR), **Deprecated** (withdrawn, nothing replaces it; none at present),
|
||||||
@@ -101,10 +101,9 @@ See also [IndieWeb protocols](../architecture/indieweb-protocols.md).
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| [012](ADR-012-http-error-handling-policy.md) | HTTP Error Handling Policy | Accepted | Correct HTTP status codes (e.g. 404 for missing notes) across routes. |
|
| [012](ADR-012-http-error-handling-policy.md) | HTTP Error Handling Policy | Accepted | Correct HTTP status codes (e.g. 404 for missing notes) across routes. |
|
||||||
| [052](ADR-052-configuration-system-architecture.md) | Configuration System Architecture | Accepted | Environment variables with defaults, loaded in one module. |
|
| [052](ADR-052-configuration-system-architecture.md) | Configuration System Architecture | Accepted | Environment variables with defaults, loaded in one module. |
|
||||||
| [053](ADR-053-performance-monitoring-strategy.md) | Performance Monitoring Strategy | Accepted | Built-in, standard-library performance metrics (v1.1.1). |
|
|
||||||
| [054](ADR-054-structured-logging-architecture.md) | Structured Logging Architecture | Accepted | Rotating log files with correlation IDs (v1.1.1). |
|
| [054](ADR-054-structured-logging-architecture.md) | Structured Logging Architecture | Accepted | Rotating log files with correlation IDs (v1.1.1). |
|
||||||
| [055](ADR-055-error-handling-philosophy.md) | Error Handling Philosophy | Accepted | Layered handling: graceful degradation, clear user messages, detailed logs. |
|
| [055](ADR-055-error-handling-philosophy.md) | Error Handling Philosophy | Accepted | Layered handling: graceful degradation, clear user messages, detailed logs. |
|
||||||
| [060](ADR-060-production-hotfix-metrics-dashboard.md) | Route Adapter for the Metrics Dashboard | Accepted | Adapt monitoring data to the dashboard template in the route (v1.1.1-rc.2). *Reconstructed.* |
|
| [072](ADR-072-remove-metrics-dashboard.md) | Remove the metrics dashboard | Accepted | Drop the in-memory metrics system and dashboard; keep `/health`, `/admin/health` and logs (v2.12.0). |
|
||||||
|
|
||||||
See also [deployment](../architecture/deployment.md).
|
See also [deployment](../architecture/deployment.md).
|
||||||
|
|
||||||
@@ -117,6 +116,8 @@ designs of v0.x–v1.6.0, all replaced by ADR-064 in v2.0.0.
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| [005](ADR-005-indielogin-authentication.md) | IndieLogin Authentication Integration | Superseded by 064 | Admin login through indielogin.com. |
|
| [005](ADR-005-indielogin-authentication.md) | IndieLogin Authentication Integration | Superseded by 064 | Admin login through indielogin.com. |
|
||||||
| [007](ADR-007-slug-generation-algorithm.md) | Slug Generation Algorithm | Superseded by 062 | Slugs from the first words of the note. |
|
| [007](ADR-007-slug-generation-algorithm.md) | Slug Generation Algorithm | Superseded by 062 | Slugs from the first words of the note. |
|
||||||
|
| [053](ADR-053-performance-monitoring-strategy.md) | Performance Monitoring Strategy | Superseded by 072 | Built-in, standard-library performance metrics (v1.1.1). |
|
||||||
|
| [060](ADR-060-production-hotfix-metrics-dashboard.md) | Route Adapter for the Metrics Dashboard | Superseded by 072 | Adapt monitoring data to the dashboard template in the route (v1.1.1-rc.2). *Reconstructed.* |
|
||||||
| [016](ADR-016-indieauth-client-discovery.md) | IndieAuth Client Discovery Mechanism | Superseded by 019 | h-app markup for client discovery. |
|
| [016](ADR-016-indieauth-client-discovery.md) | IndieAuth Client Discovery Mechanism | Superseded by 019 | h-app markup for client discovery. |
|
||||||
| [017](ADR-017-oauth-client-metadata-document.md) | OAuth Client ID Metadata Document Implementation | Superseded by 019 | OAuth client metadata endpoint. |
|
| [017](ADR-017-oauth-client-metadata-document.md) | OAuth Client ID Metadata Document Implementation | Superseded by 019 | OAuth client metadata endpoint. |
|
||||||
| [018](ADR-018-indieauth-detailed-logging.md) | IndieAuth Detailed Logging Strategy | Superseded by 064 | Redacted HTTP logging of the IndieAuth login flow. |
|
| [018](ADR-018-indieauth-detailed-logging.md) | IndieAuth Detailed Logging Strategy | Superseded by 064 | Redacted HTTP logging of the IndieAuth login flow. |
|
||||||
@@ -166,7 +167,7 @@ was carried out and is not itself superseded.
|
|||||||
|
|
||||||
## How to create a new ADR
|
## How to create a new ADR
|
||||||
|
|
||||||
1. Use the next number after the highest existing one (currently ADR-072).
|
1. Use the next number after the highest existing one (currently ADR-073).
|
||||||
2. Name it `ADR-NNN-brief-descriptive-title.md`.
|
2. Name it `ADR-NNN-brief-descriptive-title.md`.
|
||||||
3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071):
|
3. Follow the format of recent ADRs (e.g. ADR-070, ADR-071):
|
||||||
|
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ is in `../architecture/`; decisions are in `../decisions/` (ADRs).
|
|||||||
|
|
||||||
## Contents
|
## Contents
|
||||||
|
|
||||||
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [hotfix](#hotfix)
|
[v1.0.0](#v100) · [v1.1.1](#v111) · [v1.1.2](#v112) · [v1.2.0](#v120) · [v1.3.0](#v130) · [v1.3.1](#v131) · [v1.4.0](#v140) · [v1.4.1](#v141) · [v1.4.2](#v142) · [v1.5.0](#v150) · [v1.6.0](#v160) · [v2.0.0](#v200) · [v2.0.1](#v201) · [v2.0.2](#v202) · [v2.1.0](#v210) · [v2.2.0](#v220) · [v2.3.0](#v230) · [v2.3.1](#v231) · [v2.3.2](#v232) · [v2.3.3](#v233) · [v2.4.0](#v240) · [v2.5.0](#v250) · [v2.6.0](#v260) · [v2.6.1](#v261) · [v2.6.2](#v262) · [v2.6.3](#v263) · [v2.6.4](#v264) · [v2.6.5](#v265) · [v2.6.6](#v266) · [v2.6.7](#v267) · [v2.6.8](#v268) · [v2.6.9](#v269) · [v2.7.0](#v270) · [v2.7.1](#v271) · [v2.7.2](#v272) · [v2.8.0](#v280) · [v2.8.1](#v281) · [v2.9.0](#v290) · [v2.10.0](#v2100) · [v2.10.1](#v2101) · [v2.10.2](#v2102) · [v2.11.0](#v2110) · [v2.12.0](#v2120) · [hotfix](#hotfix)
|
||||||
|
|
||||||
## v1.0.0
|
## v1.0.0
|
||||||
|
|
||||||
@@ -414,6 +414,10 @@ OIDC admin login and built-in IndieAuth server.
|
|||||||
|
|
||||||
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) - Dependency upgrades (Python 3.14), foreign keys, CSRF, fetchers, debug default, shared rate limits
|
- [2026-11-03-maintenance-release.md](v2.11.0/2026-11-03-maintenance-release.md) - Dependency upgrades (Python 3.14), foreign keys, CSRF, fetchers, debug default, shared rate limits
|
||||||
|
|
||||||
|
## v2.12.0
|
||||||
|
|
||||||
|
- [2026-11-03-remove-metrics.md](v2.12.0/2026-11-03-remove-metrics.md) - Metrics dashboard and `starpunk/monitoring` removed (ADR-072)
|
||||||
|
|
||||||
## hotfix
|
## hotfix
|
||||||
|
|
||||||
Not a release folder.
|
Not a release folder.
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# v2.12.0: Remove the metrics dashboard
|
||||||
|
|
||||||
|
Decision: [ADR-072](../../decisions/ADR-072-remove-metrics-dashboard.md).
|
||||||
|
|
||||||
|
## Removed
|
||||||
|
|
||||||
|
| What | Where |
|
||||||
|
|---|---|
|
||||||
|
| Metrics buffer, HTTP middleware, database timing wrapper, memory monitor, business metrics | `starpunk/monitoring/` (deleted) |
|
||||||
|
| HTTP middleware and memory monitor start-up | `starpunk/__init__.py` |
|
||||||
|
| `MonitoredConnection` wrapping, `slow_query_threshold` and `metrics_enabled` arguments | `starpunk/database/pool.py` |
|
||||||
|
| `track_feed_generated` calls and their timing code | `starpunk/feeds/atom.py`, `json_feed.py`, `rss.py` |
|
||||||
|
| `/admin/metrics-dashboard`, `/admin/metrics`, `transform_metrics_for_template`, the `metrics` block of `/admin/health` | `starpunk/routes/admin.py` |
|
||||||
|
| Dashboard template and the admin nav link | `templates/admin/metrics_dashboard.html`, `templates/admin/base.html` |
|
||||||
|
| `METRICS_ENABLED`, `METRICS_SLOW_QUERY_THRESHOLD`, `METRICS_SAMPLING_RATE`, `METRICS_BUFFER_SIZE`, `METRICS_MEMORY_INTERVAL` | `starpunk/config.py`, `.env.example` |
|
||||||
|
| `psutil` | `requirements.txt` |
|
||||||
|
| Tests | `tests/test_monitoring.py`, `tests/test_monitoring_feed_statistics.py`, `tests/test_admin_feed_statistics.py` |
|
||||||
|
|
||||||
|
## Kept
|
||||||
|
|
||||||
|
`/health`, `/health?detailed=true`, `/admin/health` (database, pool,
|
||||||
|
filesystem, disk), pool statistics, the feed cache and its own hit/miss
|
||||||
|
counters, and correlation IDs in logs.
|
||||||
|
|
||||||
|
## Upgrade
|
||||||
|
|
||||||
|
No migration. `METRICS_*` variables can be deleted from the environment;
|
||||||
|
they are ignored if left.
|
||||||
@@ -48,3 +48,4 @@ Future work and per-release plans. See `../DOCUMENTATION.md` (PROJECTPLAN).
|
|||||||
- **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03)
|
- **[v2.10.1](v2.10.1/RELEASE.md)** - WebSub hardening (Released 2026-11-03)
|
||||||
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
|
- **[v2.10.2](v2.10.2/RELEASE.md)** - Reply context fetch fix (Released 2026-11-03)
|
||||||
- **[v2.11.0](v2.11.0/RELEASE.md)** - Maintenance: dependency upgrades, High backlog items (Released 2026-11-03)
|
- **[v2.11.0](v2.11.0/RELEASE.md)** - Maintenance: dependency upgrades, High backlog items (Released 2026-11-03)
|
||||||
|
- **[v2.12.0](v2.12.0/RELEASE.md)** - Metrics dashboard removed (Released 2026-11-03)
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# v2.12.0
|
||||||
|
|
||||||
|
- **Date**: 2026-11-03
|
||||||
|
- **Status**: Released
|
||||||
|
- **Focus**: Remove the metrics dashboard
|
||||||
|
|
||||||
|
## Removed
|
||||||
|
|
||||||
|
- Admin metrics dashboard (`/admin/metrics-dashboard`) and `/admin/metrics`
|
||||||
|
- The `starpunk/monitoring` package: request timing, query timing, memory monitor thread, feed counters
|
||||||
|
- `METRICS_*` settings and the `psutil` dependency
|
||||||
|
|
||||||
|
## Kept
|
||||||
|
|
||||||
|
- `/health`, `/admin/health` (without the metrics section), logs with correlation IDs
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [CHANGELOG](../../../CHANGELOG.md)
|
||||||
|
- [ADR-072](../../decisions/ADR-072-remove-metrics-dashboard.md)
|
||||||
|
- [Design note](../../design/v2.12.0/2026-11-03-remove-metrics.md)
|
||||||
@@ -84,10 +84,9 @@ banner to every page. It sets the `starpunk_session` cookie for
|
|||||||
server.
|
server.
|
||||||
|
|
||||||
Each app process also starts background threads: the webmention worker,
|
Each app process also starts background threads: the webmention worker,
|
||||||
the WebSub hub worker, the memory monitor and, when a crossposting target
|
the WebSub hub worker and, when a crossposting target is configured, the
|
||||||
is configured, the syndication worker. Turn off what you don't need locally
|
syndication worker. Turn off what you don't need locally with
|
||||||
with `WEBMENTION_WORKER_ENABLED=false`, `WEBSUB_ENABLED=false` and
|
`WEBMENTION_WORKER_ENABLED=false` and `WEBSUB_ENABLED=false`. Leave `GOTOSOCIAL_*`, `BLUESKY_*` and `THREADS_*`
|
||||||
`METRICS_ENABLED=false`. Leave `GOTOSOCIAL_*`, `BLUESKY_*` and `THREADS_*`
|
|
||||||
empty unless you mean to post to those accounts.
|
empty unless you mean to post to those accounts.
|
||||||
|
|
||||||
Production runs the same app under gunicorn (`gunicorn app:app`, see `CMD`
|
Production runs the same app under gunicorn (`gunicorn app:app`, see `CMD`
|
||||||
@@ -112,7 +111,7 @@ SESSION_SECRET=test uv run pytest tests/test_micropub.py -k delete -x
|
|||||||
`ADMIN_ME=https://test.example.com`; set them too to reproduce CI exactly.
|
`ADMIN_ME=https://test.example.com`; set them too to reproduce CI exactly.
|
||||||
- `tests/conftest.py` builds each app with `TESTING=True` and a temporary
|
- `tests/conftest.py` builds each app with `TESTING=True` and a temporary
|
||||||
`DATA_PATH` and database, so tests never touch `./data`. Under
|
`DATA_PATH` and database, so tests never touch `./data`. Under
|
||||||
`TESTING=True` the background workers and memory monitor don't start and
|
`TESTING=True` the background workers don't start and
|
||||||
the production OIDC check is skipped.
|
the production OIDC check is skipped.
|
||||||
- `tests/test_migration_race_condition.py` is slow (about 5 minutes). While
|
- `tests/test_migration_race_condition.py` is slow (about 5 minutes). While
|
||||||
iterating, add `--ignore=tests/test_migration_race_condition.py`; run the
|
iterating, add `--ignore=tests/test_migration_race_condition.py`; run the
|
||||||
|
|||||||
@@ -22,6 +22,3 @@ python-dotenv==1.2.*
|
|||||||
# Images
|
# Images
|
||||||
Pillow==13.0.*
|
Pillow==13.0.*
|
||||||
pillow-heif==1.8.*
|
pillow-heif==1.8.*
|
||||||
|
|
||||||
# Monitoring
|
|
||||||
psutil==7.2.*
|
|
||||||
+2
-19
@@ -142,12 +142,6 @@ def create_app(config=None):
|
|||||||
# Initialize connection pool
|
# Initialize connection pool
|
||||||
init_pool(app)
|
init_pool(app)
|
||||||
|
|
||||||
# Setup HTTP metrics middleware (v1.1.2 Phase 1)
|
|
||||||
if app.config.get('METRICS_ENABLED', True):
|
|
||||||
from starpunk.monitoring import setup_http_metrics
|
|
||||||
setup_http_metrics(app)
|
|
||||||
app.logger.info("HTTP metrics middleware enabled")
|
|
||||||
|
|
||||||
# Initialize feed cache (v1.1.2 Phase 3)
|
# Initialize feed cache (v1.1.2 Phase 3)
|
||||||
if app.config.get('FEED_CACHE_ENABLED', True):
|
if app.config.get('FEED_CACHE_ENABLED', True):
|
||||||
from starpunk.feeds import configure_cache
|
from starpunk.feeds import configure_cache
|
||||||
@@ -323,17 +317,6 @@ def create_app(config=None):
|
|||||||
|
|
||||||
register_error_handlers(app)
|
register_error_handlers(app)
|
||||||
|
|
||||||
# Start memory monitor thread (v1.1.2 Phase 1)
|
|
||||||
# Per CQ5: Skip in test mode
|
|
||||||
if app.config.get('METRICS_ENABLED', True) and not app.config.get('TESTING', False):
|
|
||||||
from starpunk.monitoring import MemoryMonitor
|
|
||||||
memory_monitor = MemoryMonitor(interval=app.config.get('METRICS_MEMORY_INTERVAL', 30))
|
|
||||||
memory_monitor.start()
|
|
||||||
app.memory_monitor = memory_monitor
|
|
||||||
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
|
|
||||||
# No teardown_appcontext stop: that ran after every request and stopped
|
|
||||||
# the monitor straight away (v2.10.0). It is a daemon thread.
|
|
||||||
|
|
||||||
# Start webmention worker thread (v1.6.0 Phase 2)
|
# Start webmention worker thread (v1.6.0 Phase 2)
|
||||||
# Per background-worker.md: Skip in test mode
|
# Per background-worker.md: Skip in test mode
|
||||||
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
|
if app.config.get('WEBMENTION_WORKER_ENABLED', True) and not app.config.get('TESTING', False):
|
||||||
@@ -474,5 +457,5 @@ def create_app(config=None):
|
|||||||
|
|
||||||
# Package version (Semantic Versioning 2.0.0)
|
# Package version (Semantic Versioning 2.0.0)
|
||||||
# See docs/standards/versioning-strategy.md for details
|
# See docs/standards/versioning-strategy.md for details
|
||||||
__version__ = "2.11.0"
|
__version__ = "2.12.0"
|
||||||
__version_info__ = (2, 11, 0)
|
__version_info__ = (2, 12, 0)
|
||||||
@@ -99,13 +99,6 @@ def load_config(app, config_override=None):
|
|||||||
# Flask MAX_CONTENT_LENGTH limits request body size (matches media.py MAX_FILE_SIZE)
|
# Flask MAX_CONTENT_LENGTH limits request body size (matches media.py MAX_FILE_SIZE)
|
||||||
app.config["MAX_CONTENT_LENGTH"] = 50 * 1024 * 1024 # 50MB
|
app.config["MAX_CONTENT_LENGTH"] = 50 * 1024 * 1024 # 50MB
|
||||||
|
|
||||||
# Metrics configuration (v1.1.2 Phase 1)
|
|
||||||
app.config["METRICS_ENABLED"] = os.getenv("METRICS_ENABLED", "true").lower() == "true"
|
|
||||||
app.config["METRICS_SLOW_QUERY_THRESHOLD"] = float(os.getenv("METRICS_SLOW_QUERY_THRESHOLD", "1.0"))
|
|
||||||
app.config["METRICS_SAMPLING_RATE"] = float(os.getenv("METRICS_SAMPLING_RATE", "1.0"))
|
|
||||||
app.config["METRICS_BUFFER_SIZE"] = int(os.getenv("METRICS_BUFFER_SIZE", "1000"))
|
|
||||||
app.config["METRICS_MEMORY_INTERVAL"] = int(os.getenv("METRICS_MEMORY_INTERVAL", "30"))
|
|
||||||
|
|
||||||
# Debug file configuration (v1.5.0 Phase 2)
|
# Debug file configuration (v1.5.0 Phase 2)
|
||||||
app.config["DEBUG_SAVE_FAILED_UPLOADS"] = os.getenv("DEBUG_SAVE_FAILED_UPLOADS", "false").lower() == "true"
|
app.config["DEBUG_SAVE_FAILED_UPLOADS"] = os.getenv("DEBUG_SAVE_FAILED_UPLOADS", "false").lower() == "true"
|
||||||
app.config["DEBUG_FILE_MAX_AGE_DAYS"] = int(os.getenv("DEBUG_FILE_MAX_AGE_DAYS", "7"))
|
app.config["DEBUG_FILE_MAX_AGE_DAYS"] = int(os.getenv("DEBUG_FILE_MAX_AGE_DAYS", "7"))
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ Provides database initialization and connection pooling
|
|||||||
Per v1.1.1 Phase 1:
|
Per v1.1.1 Phase 1:
|
||||||
- Connection pooling for improved performance (ADR-053)
|
- Connection pooling for improved performance (ADR-053)
|
||||||
- Request-scoped connections via Flask's g object
|
- Request-scoped connections via Flask's g object
|
||||||
- Pool statistics for monitoring
|
- Pool statistics
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from starpunk.database.init import init_db
|
from starpunk.database.init import init_db
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ Per ADR-053 and developer Q&A Q2, CQ1:
|
|||||||
- Provides connection pooling for improved performance
|
- Provides connection pooling for improved performance
|
||||||
- Integrates with Flask's g object for request-scoped connections
|
- Integrates with Flask's g object for request-scoped connections
|
||||||
- Maintains same interface as get_db() for transparency
|
- Maintains same interface as get_db() for transparency
|
||||||
- Pool statistics available for metrics
|
- Pool statistics (shown in /admin/health)
|
||||||
- Wraps connections with MonitoredConnection for timing (v1.1.2 Phase 1)
|
- Wraps connections with MonitoredConnection for timing (v1.1.2 Phase 1)
|
||||||
|
|
||||||
Note: Migrations use direct connections (not pooled) for isolation
|
Note: Migrations use direct connections (not pooled) for isolation
|
||||||
@@ -27,7 +27,7 @@ class ConnectionPool:
|
|||||||
but this provides connection reuse and request-scoped connection management.
|
but this provides connection reuse and request-scoped connection management.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, db_path, pool_size=5, timeout=10.0, slow_query_threshold=1.0, metrics_enabled=True):
|
def __init__(self, db_path, pool_size=5, timeout=10.0):
|
||||||
"""
|
"""
|
||||||
Initialize connection pool
|
Initialize connection pool
|
||||||
|
|
||||||
@@ -35,14 +35,10 @@ class ConnectionPool:
|
|||||||
db_path: Path to SQLite database file
|
db_path: Path to SQLite database file
|
||||||
pool_size: Maximum number of connections in pool
|
pool_size: Maximum number of connections in pool
|
||||||
timeout: Timeout for getting connection (seconds)
|
timeout: Timeout for getting connection (seconds)
|
||||||
slow_query_threshold: Threshold in seconds for slow query detection (v1.1.2)
|
|
||||||
metrics_enabled: Whether to enable metrics collection (v1.1.2)
|
|
||||||
"""
|
"""
|
||||||
self.db_path = Path(db_path)
|
self.db_path = Path(db_path)
|
||||||
self.pool_size = pool_size
|
self.pool_size = pool_size
|
||||||
self.timeout = timeout
|
self.timeout = timeout
|
||||||
self.slow_query_threshold = slow_query_threshold
|
|
||||||
self.metrics_enabled = metrics_enabled
|
|
||||||
self._pool = deque(maxlen=pool_size)
|
self._pool = deque(maxlen=pool_size)
|
||||||
self._lock = Lock()
|
self._lock = Lock()
|
||||||
self._stats = {
|
self._stats = {
|
||||||
@@ -56,8 +52,6 @@ class ConnectionPool:
|
|||||||
def _create_connection(self):
|
def _create_connection(self):
|
||||||
"""
|
"""
|
||||||
Create a new database connection
|
Create a new database connection
|
||||||
|
|
||||||
Per CQ1: Wraps connection with MonitoredConnection if metrics enabled
|
|
||||||
"""
|
"""
|
||||||
conn = sqlite3.connect(
|
conn = sqlite3.connect(
|
||||||
self.db_path,
|
self.db_path,
|
||||||
@@ -73,12 +67,6 @@ class ConnectionPool:
|
|||||||
conn.execute("PRAGMA foreign_keys=ON")
|
conn.execute("PRAGMA foreign_keys=ON")
|
||||||
|
|
||||||
self._stats['connections_created'] += 1
|
self._stats['connections_created'] += 1
|
||||||
|
|
||||||
# Wrap with monitoring if enabled (v1.1.2 Phase 1)
|
|
||||||
if self.metrics_enabled:
|
|
||||||
from starpunk.monitoring import MonitoredConnection
|
|
||||||
return MonitoredConnection(conn, self.slow_query_threshold)
|
|
||||||
|
|
||||||
return conn
|
return conn
|
||||||
|
|
||||||
def get_connection(self):
|
def get_connection(self):
|
||||||
@@ -171,8 +159,6 @@ def init_pool(app):
|
|||||||
"""
|
"""
|
||||||
Initialize the connection pool
|
Initialize the connection pool
|
||||||
|
|
||||||
Per CQ2: Passes metrics configuration from app config
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
app: Flask application instance
|
app: Flask application instance
|
||||||
"""
|
"""
|
||||||
@@ -181,20 +167,8 @@ def init_pool(app):
|
|||||||
db_path = app.config['DATABASE_PATH']
|
db_path = app.config['DATABASE_PATH']
|
||||||
pool_size = app.config.get('DB_POOL_SIZE', 5)
|
pool_size = app.config.get('DB_POOL_SIZE', 5)
|
||||||
timeout = app.config.get('DB_TIMEOUT', 10.0)
|
timeout = app.config.get('DB_TIMEOUT', 10.0)
|
||||||
slow_query_threshold = app.config.get('METRICS_SLOW_QUERY_THRESHOLD', 1.0)
|
_pool = ConnectionPool(db_path, pool_size, timeout)
|
||||||
metrics_enabled = app.config.get('METRICS_ENABLED', True)
|
app.logger.info(f"Database connection pool initialized (size={pool_size})")
|
||||||
|
|
||||||
_pool = ConnectionPool(
|
|
||||||
db_path,
|
|
||||||
pool_size,
|
|
||||||
timeout,
|
|
||||||
slow_query_threshold,
|
|
||||||
metrics_enabled
|
|
||||||
)
|
|
||||||
app.logger.info(
|
|
||||||
f"Database connection pool initialized "
|
|
||||||
f"(size={pool_size}, metrics={'enabled' if metrics_enabled else 'disabled'})"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Register teardown handler
|
# Register teardown handler
|
||||||
@app.teardown_appcontext
|
@app.teardown_appcontext
|
||||||
|
|||||||
@@ -18,12 +18,10 @@ Standards:
|
|||||||
# Standard library imports
|
# Standard library imports
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
import time
|
|
||||||
import xml.etree.ElementTree as ET
|
import xml.etree.ElementTree as ET
|
||||||
|
|
||||||
# Local imports
|
# Local imports
|
||||||
from starpunk.models import Note
|
from starpunk.models import Note
|
||||||
from starpunk.monitoring.business import track_feed_generated
|
|
||||||
|
|
||||||
|
|
||||||
# ATOM namespace
|
# ATOM namespace
|
||||||
@@ -127,10 +125,6 @@ def generate_atom_streaming(
|
|||||||
# Remove trailing slash from site_url for consistency
|
# Remove trailing slash from site_url for consistency
|
||||||
site_url = site_url.rstrip("/")
|
site_url = site_url.rstrip("/")
|
||||||
|
|
||||||
# Track feed generation timing
|
|
||||||
start_time = time.time()
|
|
||||||
item_count = 0
|
|
||||||
|
|
||||||
# Current timestamp for updated
|
# Current timestamp for updated
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
|
|
||||||
@@ -159,8 +153,6 @@ def generate_atom_streaming(
|
|||||||
# Yield entries (newest first)
|
# Yield entries (newest first)
|
||||||
# Notes from database are already in DESC order (newest first)
|
# Notes from database are already in DESC order (newest first)
|
||||||
for note in notes[:limit]:
|
for note in notes[:limit]:
|
||||||
item_count += 1
|
|
||||||
|
|
||||||
# Build permalink URL
|
# Build permalink URL
|
||||||
permalink = f"{site_url}{note.permalink}"
|
permalink = f"{site_url}{note.permalink}"
|
||||||
|
|
||||||
@@ -229,15 +221,6 @@ def generate_atom_streaming(
|
|||||||
# Yield closing tag
|
# Yield closing tag
|
||||||
yield '</feed>\n'
|
yield '</feed>\n'
|
||||||
|
|
||||||
# Track feed generation metrics
|
|
||||||
duration_ms = (time.time() - start_time) * 1000
|
|
||||||
track_feed_generated(
|
|
||||||
format='atom',
|
|
||||||
item_count=item_count,
|
|
||||||
duration_ms=duration_ms,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _escape_xml(text: str) -> str:
|
def _escape_xml(text: str) -> str:
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -18,12 +18,10 @@ Standards:
|
|||||||
# Standard library imports
|
# Standard library imports
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from typing import Optional, Dict, Any
|
from typing import Optional, Dict, Any
|
||||||
import time
|
|
||||||
import json
|
import json
|
||||||
|
|
||||||
# Local imports
|
# Local imports
|
||||||
from starpunk.models import Note
|
from starpunk.models import Note
|
||||||
from starpunk.monitoring.business import track_feed_generated
|
|
||||||
|
|
||||||
|
|
||||||
def generate_json_feed(
|
def generate_json_feed(
|
||||||
@@ -74,9 +72,6 @@ def generate_json_feed(
|
|||||||
# Remove trailing slash from site_url for consistency
|
# Remove trailing slash from site_url for consistency
|
||||||
site_url = site_url.rstrip("/")
|
site_url = site_url.rstrip("/")
|
||||||
|
|
||||||
# Track feed generation timing
|
|
||||||
start_time = time.time()
|
|
||||||
|
|
||||||
# Build feed object
|
# Build feed object
|
||||||
feed = _build_feed_object(
|
feed = _build_feed_object(
|
||||||
site_url=site_url,
|
site_url=site_url,
|
||||||
@@ -88,14 +83,6 @@ def generate_json_feed(
|
|||||||
# Serialize to JSON (pretty-printed)
|
# Serialize to JSON (pretty-printed)
|
||||||
feed_json = json.dumps(feed, ensure_ascii=False, indent=2)
|
feed_json = json.dumps(feed, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
# Track feed generation metrics
|
|
||||||
duration_ms = (time.time() - start_time) * 1000
|
|
||||||
track_feed_generated(
|
|
||||||
format='json',
|
|
||||||
item_count=min(len(notes), limit),
|
|
||||||
duration_ms=duration_ms,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
return feed_json
|
return feed_json
|
||||||
|
|
||||||
@@ -147,10 +134,6 @@ def generate_json_feed_streaming(
|
|||||||
# Remove trailing slash from site_url for consistency
|
# Remove trailing slash from site_url for consistency
|
||||||
site_url = site_url.rstrip("/")
|
site_url = site_url.rstrip("/")
|
||||||
|
|
||||||
# Track feed generation timing
|
|
||||||
start_time = time.time()
|
|
||||||
item_count = 0
|
|
||||||
|
|
||||||
# Start feed object
|
# Start feed object
|
||||||
yield '{\n'
|
yield '{\n'
|
||||||
yield f' "version": "https://jsonfeed.org/version/1.1",\n'
|
yield f' "version": "https://jsonfeed.org/version/1.1",\n'
|
||||||
@@ -170,8 +153,6 @@ def generate_json_feed_streaming(
|
|||||||
# Notes from database are already in DESC order (newest first)
|
# Notes from database are already in DESC order (newest first)
|
||||||
items = notes[:limit]
|
items = notes[:limit]
|
||||||
for i, note in enumerate(items):
|
for i, note in enumerate(items):
|
||||||
item_count += 1
|
|
||||||
|
|
||||||
# Build item object
|
# Build item object
|
||||||
item = _build_item_object(site_url, note)
|
item = _build_item_object(site_url, note)
|
||||||
|
|
||||||
@@ -193,15 +174,6 @@ def generate_json_feed_streaming(
|
|||||||
yield ' ]\n'
|
yield ' ]\n'
|
||||||
yield '}\n'
|
yield '}\n'
|
||||||
|
|
||||||
# Track feed generation metrics
|
|
||||||
duration_ms = (time.time() - start_time) * 1000
|
|
||||||
track_feed_generated(
|
|
||||||
format='json',
|
|
||||||
item_count=item_count,
|
|
||||||
duration_ms=duration_ms,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _build_feed_object(
|
def _build_feed_object(
|
||||||
site_url: str,
|
site_url: str,
|
||||||
|
|||||||
@@ -22,14 +22,12 @@ Standards:
|
|||||||
# Standard library imports
|
# Standard library imports
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
import time
|
|
||||||
|
|
||||||
# Third-party imports
|
# Third-party imports
|
||||||
from feedgen.feed import FeedGenerator
|
from feedgen.feed import FeedGenerator
|
||||||
|
|
||||||
# Local imports
|
# Local imports
|
||||||
from starpunk.models import Note
|
from starpunk.models import Note
|
||||||
from starpunk.monitoring.business import track_feed_generated
|
|
||||||
|
|
||||||
|
|
||||||
def generate_rss(
|
def generate_rss(
|
||||||
@@ -98,9 +96,6 @@ def generate_rss(
|
|||||||
# Set last build date to now
|
# Set last build date to now
|
||||||
fg.lastBuildDate(datetime.now(timezone.utc))
|
fg.lastBuildDate(datetime.now(timezone.utc))
|
||||||
|
|
||||||
# Track feed generation timing
|
|
||||||
start_time = time.time()
|
|
||||||
|
|
||||||
# Add items (limit to configured maximum, newest first)
|
# Add items (limit to configured maximum, newest first)
|
||||||
# Notes from database are DESC but feedgen reverses them, so we reverse back
|
# Notes from database are DESC but feedgen reverses them, so we reverse back
|
||||||
for note in reversed(notes[:limit]):
|
for note in reversed(notes[:limit]):
|
||||||
@@ -165,14 +160,6 @@ def generate_rss(
|
|||||||
# We need to inject media:content and media:thumbnail elements
|
# We need to inject media:content and media:thumbnail elements
|
||||||
feed_xml = _inject_media_rss_elements(feed_xml, site_url, notes[:limit])
|
feed_xml = _inject_media_rss_elements(feed_xml, site_url, notes[:limit])
|
||||||
|
|
||||||
# Track feed generation metrics
|
|
||||||
duration_ms = (time.time() - start_time) * 1000
|
|
||||||
track_feed_generated(
|
|
||||||
format='rss',
|
|
||||||
item_count=min(len(notes), limit),
|
|
||||||
duration_ms=duration_ms,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
return feed_xml
|
return feed_xml
|
||||||
|
|
||||||
@@ -227,10 +214,6 @@ def generate_rss_streaming(
|
|||||||
# Remove trailing slash from site_url for consistency
|
# Remove trailing slash from site_url for consistency
|
||||||
site_url = site_url.rstrip("/")
|
site_url = site_url.rstrip("/")
|
||||||
|
|
||||||
# Track feed generation timing
|
|
||||||
start_time = time.time()
|
|
||||||
item_count = 0
|
|
||||||
|
|
||||||
# Current timestamp for lastBuildDate
|
# Current timestamp for lastBuildDate
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
last_build = format_rfc822_date(now)
|
last_build = format_rfc822_date(now)
|
||||||
@@ -251,8 +234,6 @@ def generate_rss_streaming(
|
|||||||
# Yield items (newest first)
|
# Yield items (newest first)
|
||||||
# Notes from database are already in DESC order (newest first)
|
# Notes from database are already in DESC order (newest first)
|
||||||
for note in notes[:limit]:
|
for note in notes[:limit]:
|
||||||
item_count += 1
|
|
||||||
|
|
||||||
# Build permalink URL
|
# Build permalink URL
|
||||||
permalink = f"{site_url}{note.permalink}"
|
permalink = f"{site_url}{note.permalink}"
|
||||||
|
|
||||||
@@ -371,15 +352,6 @@ def generate_rss_streaming(
|
|||||||
yield " </channel>\n"
|
yield " </channel>\n"
|
||||||
yield "</rss>\n"
|
yield "</rss>\n"
|
||||||
|
|
||||||
# Track feed generation metrics
|
|
||||||
duration_ms = (time.time() - start_time) * 1000
|
|
||||||
track_feed_generated(
|
|
||||||
format='rss',
|
|
||||||
item_count=item_count,
|
|
||||||
duration_ms=duration_ms,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _inject_media_rss_elements(feed_xml: str, site_url: str, notes: list[Note]) -> str:
|
def _inject_media_rss_elements(feed_xml: str, site_url: str, notes: list[Note]) -> str:
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -1,35 +0,0 @@
|
|||||||
"""
|
|
||||||
Performance monitoring for StarPunk
|
|
||||||
|
|
||||||
This package provides performance monitoring capabilities including:
|
|
||||||
- Metrics collection with circular buffers
|
|
||||||
- Operation timing (database, HTTP, rendering)
|
|
||||||
- Per-process metrics with aggregation
|
|
||||||
- Configurable sampling rates
|
|
||||||
- Database query monitoring (v1.1.2 Phase 1)
|
|
||||||
- HTTP request/response metrics (v1.1.2 Phase 1)
|
|
||||||
- Memory monitoring (v1.1.2 Phase 1)
|
|
||||||
|
|
||||||
Per ADR-053 and developer Q&A Q6, Q12:
|
|
||||||
- Each process maintains its own circular buffer
|
|
||||||
- Buffers store recent metrics (default 1000 entries)
|
|
||||||
- Metrics include process ID for multi-process deployment
|
|
||||||
- Sampling rates are configurable per operation type
|
|
||||||
"""
|
|
||||||
|
|
||||||
from starpunk.monitoring.metrics import MetricsBuffer, record_metric, get_metrics, get_metrics_stats
|
|
||||||
from starpunk.monitoring.database import MonitoredConnection
|
|
||||||
from starpunk.monitoring.http import setup_http_metrics
|
|
||||||
from starpunk.monitoring.memory import MemoryMonitor
|
|
||||||
from starpunk.monitoring import business
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"MetricsBuffer",
|
|
||||||
"record_metric",
|
|
||||||
"get_metrics",
|
|
||||||
"get_metrics_stats",
|
|
||||||
"MonitoredConnection",
|
|
||||||
"setup_http_metrics",
|
|
||||||
"MemoryMonitor",
|
|
||||||
"business",
|
|
||||||
]
|
|
||||||
@@ -1,298 +0,0 @@
|
|||||||
"""
|
|
||||||
Business metrics for StarPunk operations
|
|
||||||
|
|
||||||
Per v1.1.2 Phase 1:
|
|
||||||
- Track note operations (create, update, delete)
|
|
||||||
- Track feed generation and cache hits/misses
|
|
||||||
- Track content statistics
|
|
||||||
|
|
||||||
Per v1.1.2 Phase 3:
|
|
||||||
- Track feed statistics by format
|
|
||||||
- Track feed cache hit/miss rates
|
|
||||||
- Provide feed statistics dashboard
|
|
||||||
|
|
||||||
Example usage:
|
|
||||||
>>> from starpunk.monitoring.business import track_note_created
|
|
||||||
>>> track_note_created(note_id=123, content_length=500)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from typing import Optional, Dict, Any
|
|
||||||
|
|
||||||
from starpunk.monitoring.metrics import record_metric, get_metrics_stats
|
|
||||||
|
|
||||||
|
|
||||||
def track_note_created(note_id: int, content_length: int, has_media: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Track note creation event
|
|
||||||
|
|
||||||
Args:
|
|
||||||
note_id: ID of created note
|
|
||||||
content_length: Length of note content in characters
|
|
||||||
has_media: Whether note has media attachments
|
|
||||||
"""
|
|
||||||
metadata = {
|
|
||||||
'note_id': note_id,
|
|
||||||
'content_length': content_length,
|
|
||||||
'has_media': has_media,
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render', # Use 'render' for business metrics
|
|
||||||
'note_created',
|
|
||||||
content_length,
|
|
||||||
metadata,
|
|
||||||
force=True # Always track business events
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def track_note_updated(note_id: int, content_length: int, fields_changed: Optional[list] = None) -> None:
|
|
||||||
"""
|
|
||||||
Track note update event
|
|
||||||
|
|
||||||
Args:
|
|
||||||
note_id: ID of updated note
|
|
||||||
content_length: New length of note content
|
|
||||||
fields_changed: List of fields that were changed
|
|
||||||
"""
|
|
||||||
metadata = {
|
|
||||||
'note_id': note_id,
|
|
||||||
'content_length': content_length,
|
|
||||||
}
|
|
||||||
|
|
||||||
if fields_changed:
|
|
||||||
metadata['fields_changed'] = ','.join(fields_changed)
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render',
|
|
||||||
'note_updated',
|
|
||||||
content_length,
|
|
||||||
metadata,
|
|
||||||
force=True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def track_note_deleted(note_id: int) -> None:
|
|
||||||
"""
|
|
||||||
Track note deletion event
|
|
||||||
|
|
||||||
Args:
|
|
||||||
note_id: ID of deleted note
|
|
||||||
"""
|
|
||||||
metadata = {
|
|
||||||
'note_id': note_id,
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render',
|
|
||||||
'note_deleted',
|
|
||||||
0, # No meaningful duration for deletion
|
|
||||||
metadata,
|
|
||||||
force=True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def track_feed_generated(format: str, item_count: int, duration_ms: float, cached: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Track feed generation event
|
|
||||||
|
|
||||||
Args:
|
|
||||||
format: Feed format (rss, atom, json)
|
|
||||||
item_count: Number of items in feed
|
|
||||||
duration_ms: Time taken to generate feed
|
|
||||||
cached: Whether feed was served from cache
|
|
||||||
"""
|
|
||||||
metadata = {
|
|
||||||
'format': format,
|
|
||||||
'item_count': item_count,
|
|
||||||
'cached': cached,
|
|
||||||
}
|
|
||||||
|
|
||||||
operation = f'feed_{format}{"_cached" if cached else "_generated"}'
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render',
|
|
||||||
operation,
|
|
||||||
duration_ms,
|
|
||||||
metadata,
|
|
||||||
force=True # Always track feed operations
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def track_cache_hit(cache_type: str, key: str) -> None:
|
|
||||||
"""
|
|
||||||
Track cache hit event
|
|
||||||
|
|
||||||
Args:
|
|
||||||
cache_type: Type of cache (feed, etc.)
|
|
||||||
key: Cache key that was hit
|
|
||||||
"""
|
|
||||||
metadata = {
|
|
||||||
'cache_type': cache_type,
|
|
||||||
'key': key,
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render',
|
|
||||||
f'{cache_type}_cache_hit',
|
|
||||||
0,
|
|
||||||
metadata,
|
|
||||||
force=True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def track_cache_miss(cache_type: str, key: str) -> None:
|
|
||||||
"""
|
|
||||||
Track cache miss event
|
|
||||||
|
|
||||||
Args:
|
|
||||||
cache_type: Type of cache (feed, etc.)
|
|
||||||
key: Cache key that was missed
|
|
||||||
"""
|
|
||||||
metadata = {
|
|
||||||
'cache_type': cache_type,
|
|
||||||
'key': key,
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render',
|
|
||||||
f'{cache_type}_cache_miss',
|
|
||||||
0,
|
|
||||||
metadata,
|
|
||||||
force=True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def get_feed_statistics() -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Get aggregated feed statistics from metrics buffer and feed cache.
|
|
||||||
|
|
||||||
Analyzes metrics to provide feed-specific statistics including:
|
|
||||||
- Total requests by format (RSS, ATOM, JSON)
|
|
||||||
- Cache hit/miss rates by format
|
|
||||||
- Feed generation times by format
|
|
||||||
- Format popularity (percentage breakdown)
|
|
||||||
- Feed cache internal statistics
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dictionary with feed statistics:
|
|
||||||
{
|
|
||||||
'by_format': {
|
|
||||||
'rss': {'generated': int, 'cached': int, 'total': int, 'avg_duration_ms': float},
|
|
||||||
'atom': {...},
|
|
||||||
'json': {...}
|
|
||||||
},
|
|
||||||
'cache': {
|
|
||||||
'hits': int,
|
|
||||||
'misses': int,
|
|
||||||
'hit_rate': float (0.0-1.0),
|
|
||||||
'entries': int,
|
|
||||||
'evictions': int
|
|
||||||
},
|
|
||||||
'total_requests': int,
|
|
||||||
'format_percentages': {
|
|
||||||
'rss': float,
|
|
||||||
'atom': float,
|
|
||||||
'json': float
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> stats = get_feed_statistics()
|
|
||||||
>>> print(f"RSS requests: {stats['by_format']['rss']['total']}")
|
|
||||||
>>> print(f"Cache hit rate: {stats['cache']['hit_rate']:.2%}")
|
|
||||||
"""
|
|
||||||
# Get all metrics
|
|
||||||
all_metrics = get_metrics_stats()
|
|
||||||
|
|
||||||
# Initialize result structure
|
|
||||||
result = {
|
|
||||||
'by_format': {
|
|
||||||
'rss': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
|
|
||||||
'atom': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
|
|
||||||
'json': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
|
|
||||||
},
|
|
||||||
'cache': {
|
|
||||||
'hits': 0,
|
|
||||||
'misses': 0,
|
|
||||||
'hit_rate': 0.0,
|
|
||||||
},
|
|
||||||
'total_requests': 0,
|
|
||||||
'format_percentages': {
|
|
||||||
'rss': 0.0,
|
|
||||||
'atom': 0.0,
|
|
||||||
'json': 0.0,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
# Get by_operation metrics if available
|
|
||||||
by_operation = all_metrics.get('by_operation', {})
|
|
||||||
|
|
||||||
# Count feed operations by format
|
|
||||||
for operation_name, op_stats in by_operation.items():
|
|
||||||
# Feed operations are named: feed_rss_generated, feed_rss_cached, etc.
|
|
||||||
if operation_name.startswith('feed_'):
|
|
||||||
parts = operation_name.split('_')
|
|
||||||
if len(parts) >= 3:
|
|
||||||
format_name = parts[1] # rss, atom, or json
|
|
||||||
operation_type = parts[2] # generated or cached
|
|
||||||
|
|
||||||
if format_name in result['by_format']:
|
|
||||||
count = op_stats.get('count', 0)
|
|
||||||
|
|
||||||
if operation_type == 'generated':
|
|
||||||
result['by_format'][format_name]['generated'] = count
|
|
||||||
# Track average duration for generated feeds
|
|
||||||
result['by_format'][format_name]['avg_duration_ms'] = op_stats.get('avg_duration_ms', 0.0)
|
|
||||||
elif operation_type == 'cached':
|
|
||||||
result['by_format'][format_name]['cached'] = count
|
|
||||||
|
|
||||||
# Update total for this format
|
|
||||||
result['by_format'][format_name]['total'] = (
|
|
||||||
result['by_format'][format_name]['generated'] +
|
|
||||||
result['by_format'][format_name]['cached']
|
|
||||||
)
|
|
||||||
|
|
||||||
# Track cache hits/misses
|
|
||||||
elif operation_name == 'feed_cache_hit':
|
|
||||||
result['cache']['hits'] = op_stats.get('count', 0)
|
|
||||||
elif operation_name == 'feed_cache_miss':
|
|
||||||
result['cache']['misses'] = op_stats.get('count', 0)
|
|
||||||
|
|
||||||
# Calculate total requests across all formats
|
|
||||||
result['total_requests'] = sum(
|
|
||||||
fmt['total'] for fmt in result['by_format'].values()
|
|
||||||
)
|
|
||||||
|
|
||||||
# Calculate cache hit rate
|
|
||||||
total_cache_requests = result['cache']['hits'] + result['cache']['misses']
|
|
||||||
if total_cache_requests > 0:
|
|
||||||
result['cache']['hit_rate'] = result['cache']['hits'] / total_cache_requests
|
|
||||||
|
|
||||||
# Calculate format percentages
|
|
||||||
if result['total_requests'] > 0:
|
|
||||||
for format_name, fmt_stats in result['by_format'].items():
|
|
||||||
result['format_percentages'][format_name] = (
|
|
||||||
fmt_stats['total'] / result['total_requests']
|
|
||||||
)
|
|
||||||
|
|
||||||
# Get feed cache statistics if available
|
|
||||||
try:
|
|
||||||
from starpunk.feeds import get_cache
|
|
||||||
feed_cache = get_cache()
|
|
||||||
cache_stats = feed_cache.get_stats()
|
|
||||||
|
|
||||||
# Merge cache stats (prefer FeedCache internal stats over metrics)
|
|
||||||
result['cache']['entries'] = cache_stats.get('entries', 0)
|
|
||||||
result['cache']['evictions'] = cache_stats.get('evictions', 0)
|
|
||||||
|
|
||||||
# Use FeedCache hit rate if available and more accurate
|
|
||||||
if cache_stats.get('hits', 0) + cache_stats.get('misses', 0) > 0:
|
|
||||||
result['cache']['hits'] = cache_stats.get('hits', 0)
|
|
||||||
result['cache']['misses'] = cache_stats.get('misses', 0)
|
|
||||||
result['cache']['hit_rate'] = cache_stats.get('hit_rate', 0.0)
|
|
||||||
|
|
||||||
except ImportError:
|
|
||||||
# Feed cache not available, use defaults
|
|
||||||
pass
|
|
||||||
|
|
||||||
return result
|
|
||||||
@@ -1,236 +0,0 @@
|
|||||||
"""
|
|
||||||
Database operation monitoring wrapper
|
|
||||||
|
|
||||||
Per ADR-053, v1.1.2 Phase 1, and developer Q&A CQ1, IQ1, IQ3:
|
|
||||||
- Wraps SQLite connections at the pool level
|
|
||||||
- Times all database operations
|
|
||||||
- Extracts query type and table name (best effort)
|
|
||||||
- Detects slow queries based on configurable threshold
|
|
||||||
- Records metrics to the metrics collector
|
|
||||||
|
|
||||||
Example usage:
|
|
||||||
>>> from starpunk.monitoring.database import MonitoredConnection
|
|
||||||
>>> conn = sqlite3.connect(':memory:')
|
|
||||||
>>> monitored = MonitoredConnection(conn, metrics_collector)
|
|
||||||
>>> cursor = monitored.execute('SELECT * FROM notes')
|
|
||||||
"""
|
|
||||||
|
|
||||||
import re
|
|
||||||
import sqlite3
|
|
||||||
import time
|
|
||||||
from typing import Optional, Any, Tuple
|
|
||||||
|
|
||||||
from starpunk.monitoring.metrics import record_metric
|
|
||||||
|
|
||||||
|
|
||||||
class MonitoredConnection:
|
|
||||||
"""
|
|
||||||
Wrapper for SQLite connections that monitors performance
|
|
||||||
|
|
||||||
Per CQ1: Wraps connections at the pool level
|
|
||||||
Per IQ1: Uses simple regex for table name extraction
|
|
||||||
Per IQ3: Single configurable slow query threshold
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, connection: sqlite3.Connection, slow_query_threshold: float = 1.0):
|
|
||||||
"""
|
|
||||||
Initialize monitored connection wrapper
|
|
||||||
|
|
||||||
Args:
|
|
||||||
connection: SQLite connection to wrap
|
|
||||||
slow_query_threshold: Threshold in seconds for slow query detection
|
|
||||||
"""
|
|
||||||
self._connection = connection
|
|
||||||
self._slow_query_threshold = slow_query_threshold
|
|
||||||
|
|
||||||
def execute(self, query: str, parameters: Optional[Tuple] = None) -> sqlite3.Cursor:
|
|
||||||
"""
|
|
||||||
Execute a query with performance monitoring
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: SQL query to execute
|
|
||||||
parameters: Optional query parameters
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
sqlite3.Cursor: Query cursor
|
|
||||||
"""
|
|
||||||
start_time = time.perf_counter()
|
|
||||||
query_type = self._get_query_type(query)
|
|
||||||
table_name = self._extract_table_name(query)
|
|
||||||
|
|
||||||
try:
|
|
||||||
if parameters:
|
|
||||||
cursor = self._connection.execute(query, parameters)
|
|
||||||
else:
|
|
||||||
cursor = self._connection.execute(query)
|
|
||||||
|
|
||||||
duration_sec = time.perf_counter() - start_time
|
|
||||||
duration_ms = duration_sec * 1000
|
|
||||||
|
|
||||||
# Record metric (forced if slow query)
|
|
||||||
is_slow = duration_sec >= self._slow_query_threshold
|
|
||||||
metadata = {
|
|
||||||
'query_type': query_type,
|
|
||||||
'table': table_name,
|
|
||||||
'is_slow': is_slow,
|
|
||||||
}
|
|
||||||
|
|
||||||
# Add query text for slow queries (for debugging)
|
|
||||||
if is_slow:
|
|
||||||
# Truncate query to avoid storing huge queries
|
|
||||||
metadata['query'] = query[:200] if len(query) > 200 else query
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'database',
|
|
||||||
f'{query_type} {table_name}',
|
|
||||||
duration_ms,
|
|
||||||
metadata,
|
|
||||||
force=is_slow # Always record slow queries
|
|
||||||
)
|
|
||||||
|
|
||||||
return cursor
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
duration_sec = time.perf_counter() - start_time
|
|
||||||
duration_ms = duration_sec * 1000
|
|
||||||
|
|
||||||
# Record error metric
|
|
||||||
metadata = {
|
|
||||||
'query_type': query_type,
|
|
||||||
'table': table_name,
|
|
||||||
'error': str(e),
|
|
||||||
'query': query[:200] if len(query) > 200 else query
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'database',
|
|
||||||
f'{query_type} {table_name} ERROR',
|
|
||||||
duration_ms,
|
|
||||||
metadata,
|
|
||||||
force=True # Always record errors
|
|
||||||
)
|
|
||||||
|
|
||||||
raise
|
|
||||||
|
|
||||||
def executemany(self, query: str, parameters) -> sqlite3.Cursor:
|
|
||||||
"""
|
|
||||||
Execute a query with multiple parameter sets
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: SQL query to execute
|
|
||||||
parameters: Sequence of parameter tuples
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
sqlite3.Cursor: Query cursor
|
|
||||||
"""
|
|
||||||
start_time = time.perf_counter()
|
|
||||||
query_type = self._get_query_type(query)
|
|
||||||
table_name = self._extract_table_name(query)
|
|
||||||
|
|
||||||
try:
|
|
||||||
cursor = self._connection.executemany(query, parameters)
|
|
||||||
duration_ms = (time.perf_counter() - start_time) * 1000
|
|
||||||
|
|
||||||
# Record metric
|
|
||||||
metadata = {
|
|
||||||
'query_type': query_type,
|
|
||||||
'table': table_name,
|
|
||||||
'batch': True,
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'database',
|
|
||||||
f'{query_type} {table_name} BATCH',
|
|
||||||
duration_ms,
|
|
||||||
metadata
|
|
||||||
)
|
|
||||||
|
|
||||||
return cursor
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
duration_ms = (time.perf_counter() - start_time) * 1000
|
|
||||||
|
|
||||||
metadata = {
|
|
||||||
'query_type': query_type,
|
|
||||||
'table': table_name,
|
|
||||||
'error': str(e),
|
|
||||||
'batch': True
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'database',
|
|
||||||
f'{query_type} {table_name} BATCH ERROR',
|
|
||||||
duration_ms,
|
|
||||||
metadata,
|
|
||||||
force=True
|
|
||||||
)
|
|
||||||
|
|
||||||
raise
|
|
||||||
|
|
||||||
def _get_query_type(self, query: str) -> str:
|
|
||||||
"""
|
|
||||||
Extract query type from SQL statement
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: SQL query
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Query type (SELECT, INSERT, UPDATE, DELETE, etc.)
|
|
||||||
"""
|
|
||||||
query_upper = query.strip().upper()
|
|
||||||
|
|
||||||
for query_type in ['SELECT', 'INSERT', 'UPDATE', 'DELETE', 'CREATE', 'DROP', 'ALTER', 'PRAGMA']:
|
|
||||||
if query_upper.startswith(query_type):
|
|
||||||
return query_type
|
|
||||||
|
|
||||||
return 'OTHER'
|
|
||||||
|
|
||||||
def _extract_table_name(self, query: str) -> str:
|
|
||||||
"""
|
|
||||||
Extract table name from query (best effort)
|
|
||||||
|
|
||||||
Per IQ1: Keep it simple with basic regex patterns.
|
|
||||||
Returns "unknown" for complex queries.
|
|
||||||
|
|
||||||
Note: Complex queries (JOINs, subqueries, CTEs) return "unknown".
|
|
||||||
This covers 90% of queries accurately.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: SQL query
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Table name or "unknown"
|
|
||||||
"""
|
|
||||||
query_lower = query.lower().strip()
|
|
||||||
|
|
||||||
# Simple patterns that cover 90% of cases
|
|
||||||
patterns = [
|
|
||||||
r'from\s+(\w+)',
|
|
||||||
r'update\s+(\w+)',
|
|
||||||
r'insert\s+into\s+(\w+)',
|
|
||||||
r'delete\s+from\s+(\w+)',
|
|
||||||
r'create\s+table\s+(?:if\s+not\s+exists\s+)?(\w+)',
|
|
||||||
r'drop\s+table\s+(?:if\s+exists\s+)?(\w+)',
|
|
||||||
r'alter\s+table\s+(\w+)',
|
|
||||||
]
|
|
||||||
|
|
||||||
for pattern in patterns:
|
|
||||||
match = re.search(pattern, query_lower)
|
|
||||||
if match:
|
|
||||||
return match.group(1)
|
|
||||||
|
|
||||||
# Complex queries (JOINs, subqueries, CTEs)
|
|
||||||
return "unknown"
|
|
||||||
|
|
||||||
# Delegate all other connection methods to the wrapped connection
|
|
||||||
def __getattr__(self, name: str) -> Any:
|
|
||||||
"""Delegate all other methods to the wrapped connection"""
|
|
||||||
return getattr(self._connection, name)
|
|
||||||
|
|
||||||
def __enter__(self):
|
|
||||||
"""Support context manager protocol"""
|
|
||||||
return self
|
|
||||||
|
|
||||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
|
||||||
"""Support context manager protocol"""
|
|
||||||
return self._connection.__exit__(exc_type, exc_val, exc_tb)
|
|
||||||
@@ -1,133 +0,0 @@
|
|||||||
"""
|
|
||||||
HTTP request/response metrics middleware
|
|
||||||
|
|
||||||
Per v1.1.2 Phase 1 and developer Q&A IQ2:
|
|
||||||
- Times all HTTP requests
|
|
||||||
- Generates request IDs for tracking (IQ2)
|
|
||||||
- Records status codes, methods, routes
|
|
||||||
- Tracks request and response sizes
|
|
||||||
- Adds X-Request-ID header to all responses (not just debug mode)
|
|
||||||
|
|
||||||
Example usage:
|
|
||||||
>>> from starpunk.monitoring.http import setup_http_metrics
|
|
||||||
>>> app = Flask(__name__)
|
|
||||||
>>> setup_http_metrics(app)
|
|
||||||
"""
|
|
||||||
|
|
||||||
import time
|
|
||||||
import uuid
|
|
||||||
from flask import g, request, Flask
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from starpunk.monitoring.metrics import record_metric
|
|
||||||
|
|
||||||
|
|
||||||
def setup_http_metrics(app: Flask) -> None:
|
|
||||||
"""
|
|
||||||
Setup HTTP metrics collection for Flask app
|
|
||||||
|
|
||||||
Per IQ2: Generates request IDs and adds X-Request-ID header in all modes
|
|
||||||
|
|
||||||
Args:
|
|
||||||
app: Flask application instance
|
|
||||||
"""
|
|
||||||
|
|
||||||
@app.before_request
|
|
||||||
def start_request_metrics():
|
|
||||||
"""
|
|
||||||
Initialize request metrics tracking
|
|
||||||
|
|
||||||
Per IQ2: Generate UUID request ID and store in g
|
|
||||||
"""
|
|
||||||
# Generate request ID (IQ2: in all modes, not just debug)
|
|
||||||
g.request_id = str(uuid.uuid4())
|
|
||||||
|
|
||||||
# Store request start time and metadata
|
|
||||||
g.request_start_time = time.perf_counter()
|
|
||||||
g.request_metadata = {
|
|
||||||
'method': request.method,
|
|
||||||
'endpoint': request.endpoint or 'unknown',
|
|
||||||
'path': request.path,
|
|
||||||
'content_length': request.content_length or 0,
|
|
||||||
}
|
|
||||||
|
|
||||||
@app.after_request
|
|
||||||
def record_response_metrics(response):
|
|
||||||
"""
|
|
||||||
Record HTTP response metrics
|
|
||||||
|
|
||||||
Args:
|
|
||||||
response: Flask response object
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Modified response with X-Request-ID header
|
|
||||||
"""
|
|
||||||
# Skip if metrics not initialized (shouldn't happen in normal flow)
|
|
||||||
if not hasattr(g, 'request_start_time'):
|
|
||||||
return response
|
|
||||||
|
|
||||||
# Calculate request duration
|
|
||||||
duration_sec = time.perf_counter() - g.request_start_time
|
|
||||||
duration_ms = duration_sec * 1000
|
|
||||||
|
|
||||||
# Get response size
|
|
||||||
response_size = 0
|
|
||||||
|
|
||||||
# Check if response is in direct passthrough mode (streaming)
|
|
||||||
if hasattr(response, 'direct_passthrough') and response.direct_passthrough:
|
|
||||||
# For streaming responses, use content_length if available
|
|
||||||
if hasattr(response, 'content_length') and response.content_length:
|
|
||||||
response_size = response.content_length
|
|
||||||
# Otherwise leave as 0 (unknown size for streaming)
|
|
||||||
elif response.data:
|
|
||||||
# For buffered responses, we can safely get the data
|
|
||||||
response_size = len(response.data)
|
|
||||||
elif hasattr(response, 'content_length') and response.content_length:
|
|
||||||
response_size = response.content_length
|
|
||||||
|
|
||||||
# Build metadata
|
|
||||||
metadata = {
|
|
||||||
**g.request_metadata,
|
|
||||||
'status_code': response.status_code,
|
|
||||||
'response_size': response_size,
|
|
||||||
}
|
|
||||||
|
|
||||||
# Record metric
|
|
||||||
operation_name = f"{g.request_metadata['method']} {g.request_metadata['endpoint']}"
|
|
||||||
record_metric(
|
|
||||||
'http',
|
|
||||||
operation_name,
|
|
||||||
duration_ms,
|
|
||||||
metadata
|
|
||||||
)
|
|
||||||
|
|
||||||
# Add request ID header (IQ2: in all modes)
|
|
||||||
response.headers['X-Request-ID'] = g.request_id
|
|
||||||
|
|
||||||
return response
|
|
||||||
|
|
||||||
@app.teardown_request
|
|
||||||
def record_error_metrics(error=None):
|
|
||||||
"""
|
|
||||||
Record metrics for requests that result in errors
|
|
||||||
|
|
||||||
Args:
|
|
||||||
error: Exception if request failed
|
|
||||||
"""
|
|
||||||
if error and hasattr(g, 'request_start_time'):
|
|
||||||
duration_ms = (time.perf_counter() - g.request_start_time) * 1000
|
|
||||||
|
|
||||||
metadata = {
|
|
||||||
**g.request_metadata,
|
|
||||||
'error': str(error),
|
|
||||||
'error_type': type(error).__name__,
|
|
||||||
}
|
|
||||||
|
|
||||||
operation_name = f"{g.request_metadata['method']} {g.request_metadata['endpoint']} ERROR"
|
|
||||||
record_metric(
|
|
||||||
'http',
|
|
||||||
operation_name,
|
|
||||||
duration_ms,
|
|
||||||
metadata,
|
|
||||||
force=True # Always record errors
|
|
||||||
)
|
|
||||||
@@ -1,191 +0,0 @@
|
|||||||
"""
|
|
||||||
Memory monitoring background thread
|
|
||||||
|
|
||||||
Per v1.1.2 Phase 1 and developer Q&A CQ5, IQ8:
|
|
||||||
- Background daemon thread for continuous memory monitoring
|
|
||||||
- Tracks RSS and VMS memory usage
|
|
||||||
- Detects memory growth and potential leaks
|
|
||||||
- 5-second baseline period after startup (IQ8)
|
|
||||||
- Skipped in test mode (CQ5)
|
|
||||||
|
|
||||||
Example usage:
|
|
||||||
>>> from starpunk.monitoring.memory import MemoryMonitor
|
|
||||||
>>> monitor = MemoryMonitor(interval=30)
|
|
||||||
>>> monitor.start() # Runs as daemon thread
|
|
||||||
>>> # ... application runs ...
|
|
||||||
>>> monitor.stop()
|
|
||||||
"""
|
|
||||||
|
|
||||||
import gc
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import threading
|
|
||||||
import time
|
|
||||||
from typing import Dict, Any
|
|
||||||
|
|
||||||
import psutil
|
|
||||||
|
|
||||||
from starpunk.monitoring.metrics import record_metric
|
|
||||||
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class MemoryMonitor(threading.Thread):
|
|
||||||
"""
|
|
||||||
Background thread for memory monitoring
|
|
||||||
|
|
||||||
Per CQ5: Daemon thread that auto-terminates with main process
|
|
||||||
Per IQ8: 5-second baseline period after startup
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, interval: int = 30):
|
|
||||||
"""
|
|
||||||
Initialize memory monitor thread
|
|
||||||
|
|
||||||
Args:
|
|
||||||
interval: Monitoring interval in seconds (default: 30)
|
|
||||||
"""
|
|
||||||
super().__init__(daemon=True) # CQ5: daemon thread
|
|
||||||
self.interval = interval
|
|
||||||
self._stop_event = threading.Event()
|
|
||||||
self._process = psutil.Process()
|
|
||||||
self._baseline_memory = None
|
|
||||||
self._high_water_mark = 0
|
|
||||||
|
|
||||||
def run(self):
|
|
||||||
"""
|
|
||||||
Main monitoring loop
|
|
||||||
|
|
||||||
Per IQ8: Wait 5 seconds for app initialization before setting baseline
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
# Wait for app initialization (IQ8: 5 seconds)
|
|
||||||
time.sleep(5)
|
|
||||||
|
|
||||||
# Set baseline memory
|
|
||||||
memory_info = self._get_memory_info()
|
|
||||||
self._baseline_memory = memory_info['rss_mb']
|
|
||||||
logger.info(f"Memory monitor baseline set: {self._baseline_memory:.2f} MB RSS")
|
|
||||||
|
|
||||||
# Start monitoring loop
|
|
||||||
while not self._stop_event.is_set():
|
|
||||||
try:
|
|
||||||
self._collect_metrics()
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Memory monitoring error: {e}", exc_info=True)
|
|
||||||
|
|
||||||
# Wait for interval or until stop event
|
|
||||||
self._stop_event.wait(self.interval)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Memory monitor thread failed: {e}", exc_info=True)
|
|
||||||
|
|
||||||
def _collect_metrics(self):
|
|
||||||
"""Collect and record memory metrics"""
|
|
||||||
memory_info = self._get_memory_info()
|
|
||||||
gc_stats = self._get_gc_stats()
|
|
||||||
|
|
||||||
# Update high water mark
|
|
||||||
if memory_info['rss_mb'] > self._high_water_mark:
|
|
||||||
self._high_water_mark = memory_info['rss_mb']
|
|
||||||
|
|
||||||
# Calculate growth rate (MB/hour) if baseline is set
|
|
||||||
growth_rate = 0.0
|
|
||||||
if self._baseline_memory:
|
|
||||||
growth_rate = memory_info['rss_mb'] - self._baseline_memory
|
|
||||||
|
|
||||||
# Record metrics
|
|
||||||
metadata = {
|
|
||||||
'rss_mb': memory_info['rss_mb'],
|
|
||||||
'vms_mb': memory_info['vms_mb'],
|
|
||||||
'percent': memory_info['percent'],
|
|
||||||
'high_water_mb': self._high_water_mark,
|
|
||||||
'growth_mb': growth_rate,
|
|
||||||
'gc_collections': gc_stats['collections'],
|
|
||||||
'gc_collected': gc_stats['collected'],
|
|
||||||
}
|
|
||||||
|
|
||||||
record_metric(
|
|
||||||
'render', # Use 'render' operation type for memory metrics
|
|
||||||
'memory_usage',
|
|
||||||
memory_info['rss_mb'],
|
|
||||||
metadata,
|
|
||||||
force=True # Always record memory metrics
|
|
||||||
)
|
|
||||||
|
|
||||||
# Warn if significant growth detected (>10MB growth from baseline)
|
|
||||||
if growth_rate > 10.0:
|
|
||||||
logger.warning(
|
|
||||||
f"Memory growth detected: +{growth_rate:.2f} MB from baseline "
|
|
||||||
f"(current: {memory_info['rss_mb']:.2f} MB, baseline: {self._baseline_memory:.2f} MB)"
|
|
||||||
)
|
|
||||||
|
|
||||||
def _get_memory_info(self) -> Dict[str, float]:
|
|
||||||
"""
|
|
||||||
Get current process memory usage
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with memory info in MB
|
|
||||||
"""
|
|
||||||
memory = self._process.memory_info()
|
|
||||||
|
|
||||||
return {
|
|
||||||
'rss_mb': memory.rss / (1024 * 1024), # Resident Set Size
|
|
||||||
'vms_mb': memory.vms / (1024 * 1024), # Virtual Memory Size
|
|
||||||
'percent': self._process.memory_percent(),
|
|
||||||
}
|
|
||||||
|
|
||||||
def _get_gc_stats(self) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Get garbage collection statistics
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with GC stats
|
|
||||||
"""
|
|
||||||
# Get collection counts per generation
|
|
||||||
counts = gc.get_count()
|
|
||||||
|
|
||||||
# Perform a quick gen 0 collection and count collected objects
|
|
||||||
collected = gc.collect(0)
|
|
||||||
|
|
||||||
return {
|
|
||||||
'collections': {
|
|
||||||
'gen0': counts[0],
|
|
||||||
'gen1': counts[1],
|
|
||||||
'gen2': counts[2],
|
|
||||||
},
|
|
||||||
'collected': collected,
|
|
||||||
'uncollectable': len(gc.garbage),
|
|
||||||
}
|
|
||||||
|
|
||||||
def stop(self):
|
|
||||||
"""
|
|
||||||
Stop the monitoring thread gracefully
|
|
||||||
|
|
||||||
Sets the stop event to signal the thread to exit
|
|
||||||
"""
|
|
||||||
logger.info("Stopping memory monitor")
|
|
||||||
self._stop_event.set()
|
|
||||||
|
|
||||||
def get_stats(self) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Get current memory statistics
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with current memory stats
|
|
||||||
"""
|
|
||||||
if not self._baseline_memory:
|
|
||||||
return {'status': 'initializing'}
|
|
||||||
|
|
||||||
memory_info = self._get_memory_info()
|
|
||||||
|
|
||||||
return {
|
|
||||||
'status': 'running',
|
|
||||||
'current_rss_mb': memory_info['rss_mb'],
|
|
||||||
'baseline_rss_mb': self._baseline_memory,
|
|
||||||
'growth_mb': memory_info['rss_mb'] - self._baseline_memory,
|
|
||||||
'high_water_mb': self._high_water_mark,
|
|
||||||
'percent': memory_info['percent'],
|
|
||||||
}
|
|
||||||
@@ -1,425 +0,0 @@
|
|||||||
"""
|
|
||||||
Metrics collection and buffering for performance monitoring
|
|
||||||
|
|
||||||
Per ADR-053 and developer Q&A Q6, Q12:
|
|
||||||
- Per-process circular buffers using deque
|
|
||||||
- Configurable buffer size (default 1000 entries)
|
|
||||||
- Include process ID in all metrics
|
|
||||||
- Configuration-based sampling rates
|
|
||||||
- Operation types: database, http, render
|
|
||||||
|
|
||||||
Example usage:
|
|
||||||
>>> from starpunk.monitoring import record_metric, get_metrics
|
|
||||||
>>>
|
|
||||||
>>> # Record a database operation
|
|
||||||
>>> record_metric('database', 'query', duration_ms=45.2, query='SELECT * FROM notes')
|
|
||||||
>>>
|
|
||||||
>>> # Get all metrics
|
|
||||||
>>> metrics = get_metrics()
|
|
||||||
>>> print(f"Collected {len(metrics)} metrics")
|
|
||||||
"""
|
|
||||||
|
|
||||||
import os
|
|
||||||
import random
|
|
||||||
import time
|
|
||||||
from collections import deque
|
|
||||||
from dataclasses import dataclass, field, asdict
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from threading import Lock
|
|
||||||
from typing import Any, Deque, Dict, List, Literal, Optional, Union
|
|
||||||
|
|
||||||
# Operation types for categorizing metrics
|
|
||||||
OperationType = Literal["database", "http", "render"]
|
|
||||||
|
|
||||||
# Module-level circular buffer (per-process)
|
|
||||||
# Each process in a multi-process deployment maintains its own buffer
|
|
||||||
_metrics_buffer: Optional["MetricsBuffer"] = None
|
|
||||||
_buffer_lock = Lock()
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Metric:
|
|
||||||
"""
|
|
||||||
Represents a single performance metric
|
|
||||||
|
|
||||||
Attributes:
|
|
||||||
operation_type: Type of operation (database/http/render)
|
|
||||||
operation_name: Name/description of operation
|
|
||||||
timestamp: When the metric was recorded (ISO format)
|
|
||||||
duration_ms: Duration in milliseconds
|
|
||||||
process_id: Process ID that recorded the metric
|
|
||||||
metadata: Additional operation-specific data
|
|
||||||
"""
|
|
||||||
operation_type: OperationType
|
|
||||||
operation_name: str
|
|
||||||
timestamp: str
|
|
||||||
duration_ms: float
|
|
||||||
process_id: int
|
|
||||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
def to_dict(self) -> Dict[str, Any]:
|
|
||||||
"""Convert metric to dictionary for serialization"""
|
|
||||||
return asdict(self)
|
|
||||||
|
|
||||||
|
|
||||||
class MetricsBuffer:
|
|
||||||
"""
|
|
||||||
Circular buffer for storing performance metrics
|
|
||||||
|
|
||||||
Per developer Q&A Q6:
|
|
||||||
- Uses deque for efficient circular buffer
|
|
||||||
- Per-process storage (not shared across workers)
|
|
||||||
- Thread-safe with locking
|
|
||||||
- Configurable max size (default 1000)
|
|
||||||
- Automatic eviction of oldest entries when full
|
|
||||||
|
|
||||||
Per developer Q&A Q12:
|
|
||||||
- Configurable sampling rates per operation type
|
|
||||||
- Default 100% sampling (suitable for low-traffic sites)
|
|
||||||
- Slow queries always logged regardless of sampling
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> buffer = MetricsBuffer(max_size=1000)
|
|
||||||
>>> buffer.record('database', 'query', 45.2, {'query': 'SELECT ...'})
|
|
||||||
>>> metrics = buffer.get_all()
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
max_size: int = 1000,
|
|
||||||
sampling_rates: Optional[Union[Dict[OperationType, float], float]] = None
|
|
||||||
):
|
|
||||||
"""
|
|
||||||
Initialize metrics buffer
|
|
||||||
|
|
||||||
Args:
|
|
||||||
max_size: Maximum number of metrics to store
|
|
||||||
sampling_rates: Either:
|
|
||||||
- float: Global sampling rate for all operation types (0.0-1.0)
|
|
||||||
- dict: Mapping operation type to sampling rate
|
|
||||||
Default: 1.0 (100% sampling)
|
|
||||||
"""
|
|
||||||
self.max_size = max_size
|
|
||||||
self._buffer: Deque[Metric] = deque(maxlen=max_size)
|
|
||||||
self._lock = Lock()
|
|
||||||
self._process_id = os.getpid()
|
|
||||||
|
|
||||||
# Handle different sampling_rates types
|
|
||||||
if sampling_rates is None:
|
|
||||||
# Default to 100% sampling for all types
|
|
||||||
self._sampling_rates = {
|
|
||||||
"database": 1.0,
|
|
||||||
"http": 1.0,
|
|
||||||
"render": 1.0,
|
|
||||||
}
|
|
||||||
elif isinstance(sampling_rates, (int, float)):
|
|
||||||
# Global rate for all types
|
|
||||||
rate = float(sampling_rates)
|
|
||||||
self._sampling_rates = {
|
|
||||||
"database": rate,
|
|
||||||
"http": rate,
|
|
||||||
"render": rate,
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
# Dict with per-type rates
|
|
||||||
self._sampling_rates = sampling_rates
|
|
||||||
|
|
||||||
def record(
|
|
||||||
self,
|
|
||||||
operation_type: OperationType,
|
|
||||||
operation_name: str,
|
|
||||||
duration_ms: float,
|
|
||||||
metadata: Optional[Dict[str, Any]] = None,
|
|
||||||
force: bool = False
|
|
||||||
) -> bool:
|
|
||||||
"""
|
|
||||||
Record a performance metric
|
|
||||||
|
|
||||||
Args:
|
|
||||||
operation_type: Type of operation (database/http/render)
|
|
||||||
operation_name: Name/description of operation
|
|
||||||
duration_ms: Duration in milliseconds
|
|
||||||
metadata: Additional operation-specific data
|
|
||||||
force: If True, bypass sampling (for slow query logging)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
True if metric was recorded, False if skipped due to sampling
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> buffer.record('database', 'SELECT notes', 45.2,
|
|
||||||
... {'query': 'SELECT * FROM notes LIMIT 10'})
|
|
||||||
True
|
|
||||||
"""
|
|
||||||
# Apply sampling (unless forced)
|
|
||||||
if not force:
|
|
||||||
sampling_rate = self._sampling_rates.get(operation_type, 0.1)
|
|
||||||
if random.random() > sampling_rate:
|
|
||||||
return False
|
|
||||||
|
|
||||||
metric = Metric(
|
|
||||||
operation_type=operation_type,
|
|
||||||
operation_name=operation_name,
|
|
||||||
timestamp=datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
|
|
||||||
duration_ms=duration_ms,
|
|
||||||
process_id=self._process_id,
|
|
||||||
metadata=metadata or {}
|
|
||||||
)
|
|
||||||
|
|
||||||
with self._lock:
|
|
||||||
self._buffer.append(metric)
|
|
||||||
|
|
||||||
return True
|
|
||||||
|
|
||||||
def get_all(self) -> List[Metric]:
|
|
||||||
"""
|
|
||||||
Get all metrics from buffer
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of metrics (oldest to newest)
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> metrics = buffer.get_all()
|
|
||||||
>>> len(metrics)
|
|
||||||
1000
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
return list(self._buffer)
|
|
||||||
|
|
||||||
def get_recent(self, count: int) -> List[Metric]:
|
|
||||||
"""
|
|
||||||
Get most recent N metrics
|
|
||||||
|
|
||||||
Args:
|
|
||||||
count: Number of recent metrics to return
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of most recent metrics (newest first)
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> recent = buffer.get_recent(10)
|
|
||||||
>>> len(recent)
|
|
||||||
10
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
# Convert to list, reverse to get newest first, then slice
|
|
||||||
all_metrics = list(self._buffer)
|
|
||||||
all_metrics.reverse()
|
|
||||||
return all_metrics[:count]
|
|
||||||
|
|
||||||
def get_by_type(self, operation_type: OperationType) -> List[Metric]:
|
|
||||||
"""
|
|
||||||
Get all metrics of a specific type
|
|
||||||
|
|
||||||
Args:
|
|
||||||
operation_type: Type to filter by (database/http/render)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of metrics matching the type
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> db_metrics = buffer.get_by_type('database')
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
return [m for m in self._buffer if m.operation_type == operation_type]
|
|
||||||
|
|
||||||
def get_slow_operations(
|
|
||||||
self,
|
|
||||||
threshold_ms: float = 1000.0,
|
|
||||||
operation_type: Optional[OperationType] = None
|
|
||||||
) -> List[Metric]:
|
|
||||||
"""
|
|
||||||
Get operations that exceeded a duration threshold
|
|
||||||
|
|
||||||
Args:
|
|
||||||
threshold_ms: Duration threshold in milliseconds
|
|
||||||
operation_type: Optional type filter
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of slow operations
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> slow_queries = buffer.get_slow_operations(1000, 'database')
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
metrics = list(self._buffer)
|
|
||||||
|
|
||||||
# Filter by type if specified
|
|
||||||
if operation_type:
|
|
||||||
metrics = [m for m in metrics if m.operation_type == operation_type]
|
|
||||||
|
|
||||||
# Filter by duration threshold
|
|
||||||
return [m for m in metrics if m.duration_ms >= threshold_ms]
|
|
||||||
|
|
||||||
def get_stats(self) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Get statistics about the buffer
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with buffer statistics
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> stats = buffer.get_stats()
|
|
||||||
>>> stats['total_count']
|
|
||||||
1000
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
metrics = list(self._buffer)
|
|
||||||
|
|
||||||
# Calculate stats per operation type
|
|
||||||
type_stats = {}
|
|
||||||
for op_type in ["database", "http", "render"]:
|
|
||||||
type_metrics = [m for m in metrics if m.operation_type == op_type]
|
|
||||||
if type_metrics:
|
|
||||||
durations = [m.duration_ms for m in type_metrics]
|
|
||||||
type_stats[op_type] = {
|
|
||||||
"count": len(type_metrics),
|
|
||||||
"avg_duration_ms": sum(durations) / len(durations),
|
|
||||||
"min_duration_ms": min(durations),
|
|
||||||
"max_duration_ms": max(durations),
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
type_stats[op_type] = {
|
|
||||||
"count": 0,
|
|
||||||
"avg_duration_ms": 0.0,
|
|
||||||
"min_duration_ms": 0.0,
|
|
||||||
"max_duration_ms": 0.0,
|
|
||||||
}
|
|
||||||
|
|
||||||
return {
|
|
||||||
"total_count": len(metrics),
|
|
||||||
"max_size": self.max_size,
|
|
||||||
"process_id": self._process_id,
|
|
||||||
"sampling_rates": self._sampling_rates,
|
|
||||||
"by_type": type_stats,
|
|
||||||
}
|
|
||||||
|
|
||||||
def clear(self) -> None:
|
|
||||||
"""
|
|
||||||
Clear all metrics from buffer
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> buffer.clear()
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
self._buffer.clear()
|
|
||||||
|
|
||||||
def set_sampling_rate(
|
|
||||||
self,
|
|
||||||
operation_type: OperationType,
|
|
||||||
rate: float
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
Update sampling rate for an operation type
|
|
||||||
|
|
||||||
Args:
|
|
||||||
operation_type: Type to update
|
|
||||||
rate: New sampling rate (0.0-1.0)
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> buffer.set_sampling_rate('database', 0.5) # 50% sampling
|
|
||||||
"""
|
|
||||||
if not 0.0 <= rate <= 1.0:
|
|
||||||
raise ValueError("Sampling rate must be between 0.0 and 1.0")
|
|
||||||
|
|
||||||
with self._lock:
|
|
||||||
self._sampling_rates[operation_type] = rate
|
|
||||||
|
|
||||||
|
|
||||||
def get_buffer() -> MetricsBuffer:
|
|
||||||
"""
|
|
||||||
Get or create the module-level metrics buffer
|
|
||||||
|
|
||||||
This ensures a single buffer per process. In multi-process deployments
|
|
||||||
(e.g., gunicorn), each worker process will have its own buffer.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
MetricsBuffer instance for this process
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> buffer = get_buffer()
|
|
||||||
>>> buffer.record('database', 'query', 45.2)
|
|
||||||
"""
|
|
||||||
global _metrics_buffer
|
|
||||||
|
|
||||||
if _metrics_buffer is None:
|
|
||||||
with _buffer_lock:
|
|
||||||
# Double-check locking pattern
|
|
||||||
if _metrics_buffer is None:
|
|
||||||
# Get configuration from Flask app if available
|
|
||||||
try:
|
|
||||||
from flask import current_app
|
|
||||||
max_size = current_app.config.get('METRICS_BUFFER_SIZE', 1000)
|
|
||||||
sampling_rate = current_app.config.get('METRICS_SAMPLING_RATE', 1.0)
|
|
||||||
except (ImportError, RuntimeError):
|
|
||||||
# Flask not available or no app context
|
|
||||||
max_size = 1000
|
|
||||||
sampling_rate = 1.0 # Default to 100%
|
|
||||||
|
|
||||||
_metrics_buffer = MetricsBuffer(
|
|
||||||
max_size=max_size,
|
|
||||||
sampling_rates=sampling_rate
|
|
||||||
)
|
|
||||||
|
|
||||||
return _metrics_buffer
|
|
||||||
|
|
||||||
|
|
||||||
def record_metric(
|
|
||||||
operation_type: OperationType,
|
|
||||||
operation_name: str,
|
|
||||||
duration_ms: float,
|
|
||||||
metadata: Optional[Dict[str, Any]] = None,
|
|
||||||
force: bool = False
|
|
||||||
) -> bool:
|
|
||||||
"""
|
|
||||||
Record a metric using the module-level buffer
|
|
||||||
|
|
||||||
Convenience function that uses get_buffer() internally.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
operation_type: Type of operation (database/http/render)
|
|
||||||
operation_name: Name/description of operation
|
|
||||||
duration_ms: Duration in milliseconds
|
|
||||||
metadata: Additional operation-specific data
|
|
||||||
force: If True, bypass sampling (for slow query logging)
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
True if metric was recorded, False if skipped due to sampling
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> record_metric('database', 'SELECT notes', 45.2,
|
|
||||||
... {'query': 'SELECT * FROM notes LIMIT 10'})
|
|
||||||
True
|
|
||||||
"""
|
|
||||||
buffer = get_buffer()
|
|
||||||
return buffer.record(operation_type, operation_name, duration_ms, metadata, force)
|
|
||||||
|
|
||||||
|
|
||||||
def get_metrics() -> List[Metric]:
|
|
||||||
"""
|
|
||||||
Get all metrics from the module-level buffer
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of metrics (oldest to newest)
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> metrics = get_metrics()
|
|
||||||
>>> len(metrics)
|
|
||||||
1000
|
|
||||||
"""
|
|
||||||
buffer = get_buffer()
|
|
||||||
return buffer.get_all()
|
|
||||||
|
|
||||||
|
|
||||||
def get_metrics_stats() -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Get statistics from the module-level buffer
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with buffer statistics
|
|
||||||
|
|
||||||
Example:
|
|
||||||
>>> stats = get_metrics_stats()
|
|
||||||
>>> print(f"Total metrics: {stats['total_count']}")
|
|
||||||
"""
|
|
||||||
buffer = get_buffer()
|
|
||||||
return buffer.get_stats()
|
|
||||||
+1
-207
@@ -9,7 +9,6 @@ from flask import Blueprint, flash, g, jsonify, redirect, render_template, reque
|
|||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -366,202 +365,6 @@ def delete_note_submit(note_id: int):
|
|||||||
return redirect(url_for("admin.dashboard"))
|
return redirect(url_for("admin.dashboard"))
|
||||||
|
|
||||||
|
|
||||||
def transform_metrics_for_template(metrics_stats):
|
|
||||||
"""
|
|
||||||
Transform metrics stats to match template structure
|
|
||||||
|
|
||||||
The template expects direct access to metrics.database.count, but
|
|
||||||
get_metrics_stats() returns metrics.by_type.database.count.
|
|
||||||
This function adapts the data structure to match template expectations.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
metrics_stats: Dict from get_metrics_stats() with nested by_type structure
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with flattened structure matching template expectations
|
|
||||||
|
|
||||||
Per ADR-060: Route Adapter Pattern for template compatibility
|
|
||||||
"""
|
|
||||||
transformed = {}
|
|
||||||
|
|
||||||
# Map by_type to direct access
|
|
||||||
for op_type in ['database', 'http', 'render']:
|
|
||||||
if 'by_type' in metrics_stats and op_type in metrics_stats['by_type']:
|
|
||||||
type_data = metrics_stats['by_type'][op_type]
|
|
||||||
transformed[op_type] = {
|
|
||||||
'count': type_data.get('count', 0),
|
|
||||||
'avg': type_data.get('avg_duration_ms', 0),
|
|
||||||
'min': type_data.get('min_duration_ms', 0),
|
|
||||||
'max': type_data.get('max_duration_ms', 0)
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
# Provide defaults for missing types or when by_type doesn't exist
|
|
||||||
transformed[op_type] = {
|
|
||||||
'count': 0,
|
|
||||||
'avg': 0,
|
|
||||||
'min': 0,
|
|
||||||
'max': 0
|
|
||||||
}
|
|
||||||
|
|
||||||
# Keep other top-level stats
|
|
||||||
transformed['total_count'] = metrics_stats.get('total_count', 0)
|
|
||||||
transformed['max_size'] = metrics_stats.get('max_size', 1000)
|
|
||||||
transformed['process_id'] = metrics_stats.get('process_id', 0)
|
|
||||||
|
|
||||||
return transformed
|
|
||||||
|
|
||||||
|
|
||||||
@bp.route("/metrics-dashboard")
|
|
||||||
@require_auth
|
|
||||||
def metrics_dashboard():
|
|
||||||
"""
|
|
||||||
Metrics visualization dashboard (Phase 3)
|
|
||||||
|
|
||||||
Displays performance metrics, database statistics, feed statistics,
|
|
||||||
and system health with visual charts and auto-refresh capability.
|
|
||||||
|
|
||||||
Per Q19 requirements:
|
|
||||||
- Server-side rendering with Jinja2
|
|
||||||
- htmx for auto-refresh
|
|
||||||
- Chart.js from CDN for graphs
|
|
||||||
- Progressive enhancement (works without JS)
|
|
||||||
|
|
||||||
Per v1.1.2 Phase 3:
|
|
||||||
- Feed statistics by format
|
|
||||||
- Cache hit/miss rates
|
|
||||||
- Format popularity breakdown
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Rendered dashboard template with metrics
|
|
||||||
|
|
||||||
Decorator: @require_auth
|
|
||||||
Template: templates/admin/metrics_dashboard.html
|
|
||||||
"""
|
|
||||||
# Defensive imports with graceful degradation for missing modules
|
|
||||||
try:
|
|
||||||
from starpunk.database.pool import get_pool_stats
|
|
||||||
from starpunk.monitoring import get_metrics_stats
|
|
||||||
from starpunk.monitoring.business import get_feed_statistics
|
|
||||||
monitoring_available = True
|
|
||||||
except ImportError:
|
|
||||||
monitoring_available = False
|
|
||||||
# Provide fallback functions that return error messages
|
|
||||||
def get_pool_stats():
|
|
||||||
return {"error": "Database pool monitoring not available"}
|
|
||||||
def get_metrics_stats():
|
|
||||||
return {"error": "Monitoring module not implemented"}
|
|
||||||
def get_feed_statistics():
|
|
||||||
return {"error": "Feed statistics not available"}
|
|
||||||
|
|
||||||
# Get current metrics for initial page load
|
|
||||||
metrics_data = {}
|
|
||||||
pool_stats = {}
|
|
||||||
feed_stats = {}
|
|
||||||
|
|
||||||
try:
|
|
||||||
raw_metrics = get_metrics_stats()
|
|
||||||
metrics_data = transform_metrics_for_template(raw_metrics)
|
|
||||||
except Exception as e:
|
|
||||||
flash(f"Error loading metrics: {e}", "warning")
|
|
||||||
# Provide safe defaults matching template expectations
|
|
||||||
metrics_data = {
|
|
||||||
'database': {'count': 0, 'avg': 0, 'min': 0, 'max': 0},
|
|
||||||
'http': {'count': 0, 'avg': 0, 'min': 0, 'max': 0},
|
|
||||||
'render': {'count': 0, 'avg': 0, 'min': 0, 'max': 0},
|
|
||||||
'total_count': 0,
|
|
||||||
'max_size': 1000,
|
|
||||||
'process_id': 0
|
|
||||||
}
|
|
||||||
|
|
||||||
try:
|
|
||||||
pool_stats = get_pool_stats()
|
|
||||||
except Exception as e:
|
|
||||||
flash(f"Error loading pool stats: {e}", "warning")
|
|
||||||
|
|
||||||
try:
|
|
||||||
feed_stats = get_feed_statistics()
|
|
||||||
except Exception as e:
|
|
||||||
flash(f"Error loading feed stats: {e}", "warning")
|
|
||||||
# Provide safe defaults
|
|
||||||
feed_stats = {
|
|
||||||
'by_format': {
|
|
||||||
'rss': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
|
|
||||||
'atom': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
|
|
||||||
'json': {'generated': 0, 'cached': 0, 'total': 0, 'avg_duration_ms': 0.0},
|
|
||||||
},
|
|
||||||
'cache': {'hits': 0, 'misses': 0, 'hit_rate': 0.0, 'entries': 0, 'evictions': 0},
|
|
||||||
'total_requests': 0,
|
|
||||||
'format_percentages': {'rss': 0.0, 'atom': 0.0, 'json': 0.0},
|
|
||||||
}
|
|
||||||
|
|
||||||
return render_template(
|
|
||||||
"admin/metrics_dashboard.html",
|
|
||||||
metrics=metrics_data,
|
|
||||||
pool=pool_stats,
|
|
||||||
feeds=feed_stats,
|
|
||||||
user_me=g.me
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@bp.route("/metrics")
|
|
||||||
@require_auth
|
|
||||||
def metrics():
|
|
||||||
"""
|
|
||||||
Performance metrics and database pool statistics endpoint
|
|
||||||
|
|
||||||
Per Phase 2 requirements:
|
|
||||||
- Expose database pool statistics
|
|
||||||
- Show performance metrics from MetricsBuffer
|
|
||||||
- Requires authentication
|
|
||||||
|
|
||||||
Per v1.1.2 Phase 3:
|
|
||||||
- Include feed statistics
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
JSON with metrics, pool statistics, and feed statistics
|
|
||||||
|
|
||||||
Response codes:
|
|
||||||
200: Metrics retrieved successfully
|
|
||||||
|
|
||||||
Decorator: @require_auth
|
|
||||||
"""
|
|
||||||
from flask import current_app
|
|
||||||
from starpunk.database.pool import get_pool_stats
|
|
||||||
from starpunk.monitoring import get_metrics_stats
|
|
||||||
from starpunk.monitoring.business import get_feed_statistics
|
|
||||||
|
|
||||||
response = {
|
|
||||||
"timestamp": datetime.now(timezone.utc).replace(tzinfo=None).isoformat() + "Z",
|
|
||||||
"process_id": os.getpid(),
|
|
||||||
"database": {},
|
|
||||||
"performance": {},
|
|
||||||
"feeds": {}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Get database pool statistics
|
|
||||||
try:
|
|
||||||
pool_stats = get_pool_stats()
|
|
||||||
response["database"]["pool"] = pool_stats
|
|
||||||
except Exception as e:
|
|
||||||
response["database"]["pool"] = {"error": str(e)}
|
|
||||||
|
|
||||||
# Get performance metrics
|
|
||||||
try:
|
|
||||||
metrics_stats = get_metrics_stats()
|
|
||||||
response["performance"] = metrics_stats
|
|
||||||
except Exception as e:
|
|
||||||
response["performance"] = {"error": str(e)}
|
|
||||||
|
|
||||||
# Get feed statistics
|
|
||||||
try:
|
|
||||||
feed_stats = get_feed_statistics()
|
|
||||||
response["feeds"] = feed_stats
|
|
||||||
except Exception as e:
|
|
||||||
response["feeds"] = {"error": str(e)}
|
|
||||||
|
|
||||||
return jsonify(response), 200
|
|
||||||
|
|
||||||
|
|
||||||
@bp.route("/health")
|
@bp.route("/health")
|
||||||
@require_auth
|
@require_auth
|
||||||
def health_diagnostics():
|
def health_diagnostics():
|
||||||
@@ -571,7 +374,7 @@ def health_diagnostics():
|
|||||||
Per developer Q&A Q10:
|
Per developer Q&A Q10:
|
||||||
- Always requires authentication
|
- Always requires authentication
|
||||||
- Provides comprehensive diagnostics
|
- Provides comprehensive diagnostics
|
||||||
- Includes metrics, database pool statistics, and system info
|
- Includes database pool statistics and system info
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
JSON with complete system diagnostics
|
JSON with complete system diagnostics
|
||||||
@@ -591,7 +394,6 @@ def health_diagnostics():
|
|||||||
"environment": current_app.config.get("ENV", "unknown"),
|
"environment": current_app.config.get("ENV", "unknown"),
|
||||||
"process_id": os.getpid(),
|
"process_id": os.getpid(),
|
||||||
"checks": {},
|
"checks": {},
|
||||||
"metrics": {},
|
|
||||||
"database": {}
|
"database": {}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -665,14 +467,6 @@ def health_diagnostics():
|
|||||||
}
|
}
|
||||||
overall_healthy = False
|
overall_healthy = False
|
||||||
|
|
||||||
# Performance metrics
|
|
||||||
try:
|
|
||||||
from starpunk.monitoring import get_metrics_stats
|
|
||||||
metrics_stats = get_metrics_stats()
|
|
||||||
diagnostics["metrics"] = metrics_stats
|
|
||||||
except Exception as e:
|
|
||||||
diagnostics["metrics"] = {"error": str(e)}
|
|
||||||
|
|
||||||
# Update overall status
|
# Update overall status
|
||||||
diagnostics["status"] = "healthy" if overall_healthy else "unhealthy"
|
diagnostics["status"] = "healthy" if overall_healthy else "unhealthy"
|
||||||
|
|
||||||
|
|||||||
@@ -8,7 +8,6 @@
|
|||||||
<nav class="admin-nav">
|
<nav class="admin-nav">
|
||||||
<a href="{{ url_for('admin.dashboard') }}">Dashboard</a>
|
<a href="{{ url_for('admin.dashboard') }}">Dashboard</a>
|
||||||
<a href="{{ url_for('admin.new_note_form') }}">New Note</a>
|
<a href="{{ url_for('admin.new_note_form') }}">New Note</a>
|
||||||
<a href="{{ url_for('admin.metrics_dashboard') }}">Metrics</a>
|
|
||||||
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
|
<a href="{{ url_for('admin.tokens') }}">Connected Apps</a>
|
||||||
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
<form action="{{ url_for('auth.logout') }}" method="POST" class="logout-form"><input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
|
||||||
<button type="submit" class="button button-secondary">Logout</button>
|
<button type="submit" class="button button-secondary">Logout</button>
|
||||||
|
|||||||
@@ -1,585 +0,0 @@
|
|||||||
{% extends "admin/base.html" %}
|
|
||||||
|
|
||||||
{% block title %}Metrics Dashboard - StarPunk Admin{% endblock %}
|
|
||||||
|
|
||||||
{% block head %}
|
|
||||||
{{ super() }}
|
|
||||||
<!-- Chart.js from CDN for visualizations -->
|
|
||||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js" crossorigin="anonymous"></script>
|
|
||||||
<!-- htmx for auto-refresh -->
|
|
||||||
<script src="https://unpkg.com/htmx.org@1.9.10" crossorigin="anonymous"></script>
|
|
||||||
<style>
|
|
||||||
.metrics-dashboard {
|
|
||||||
max-width: 1200px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metrics-grid {
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
|
|
||||||
gap: 20px;
|
|
||||||
margin-bottom: 30px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-card {
|
|
||||||
background: #fff;
|
|
||||||
border: 1px solid #ddd;
|
|
||||||
border-radius: 8px;
|
|
||||||
padding: 20px;
|
|
||||||
box-shadow: 0 2px 4px rgba(0,0,0,0.05);
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-card h3 {
|
|
||||||
margin-top: 0;
|
|
||||||
font-size: 1.1em;
|
|
||||||
color: #333;
|
|
||||||
border-bottom: 2px solid #007bff;
|
|
||||||
padding-bottom: 10px;
|
|
||||||
margin-bottom: 15px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-value {
|
|
||||||
font-size: 2em;
|
|
||||||
font-weight: bold;
|
|
||||||
color: #007bff;
|
|
||||||
margin: 10px 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-label {
|
|
||||||
color: #666;
|
|
||||||
font-size: 0.9em;
|
|
||||||
margin-bottom: 5px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-detail {
|
|
||||||
display: flex;
|
|
||||||
justify-content: space-between;
|
|
||||||
padding: 8px 0;
|
|
||||||
border-bottom: 1px solid #f0f0f0;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-detail:last-child {
|
|
||||||
border-bottom: none;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-detail-label {
|
|
||||||
color: #666;
|
|
||||||
}
|
|
||||||
|
|
||||||
.metric-detail-value {
|
|
||||||
font-weight: bold;
|
|
||||||
}
|
|
||||||
|
|
||||||
.chart-container {
|
|
||||||
position: relative;
|
|
||||||
height: 300px;
|
|
||||||
margin-top: 20px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.status-indicator {
|
|
||||||
display: inline-block;
|
|
||||||
width: 12px;
|
|
||||||
height: 12px;
|
|
||||||
border-radius: 50%;
|
|
||||||
margin-right: 8px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.status-healthy {
|
|
||||||
background-color: #28a745;
|
|
||||||
}
|
|
||||||
|
|
||||||
.status-warning {
|
|
||||||
background-color: #ffc107;
|
|
||||||
}
|
|
||||||
|
|
||||||
.status-error {
|
|
||||||
background-color: #dc3545;
|
|
||||||
}
|
|
||||||
|
|
||||||
.refresh-info {
|
|
||||||
color: #666;
|
|
||||||
font-size: 0.9em;
|
|
||||||
text-align: center;
|
|
||||||
margin-top: 20px;
|
|
||||||
padding: 10px;
|
|
||||||
background-color: #f8f9fa;
|
|
||||||
border-radius: 4px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.no-js-message {
|
|
||||||
display: none;
|
|
||||||
background-color: #fff3cd;
|
|
||||||
border: 1px solid #ffeaa7;
|
|
||||||
color: #856404;
|
|
||||||
padding: 15px;
|
|
||||||
border-radius: 4px;
|
|
||||||
margin-bottom: 20px;
|
|
||||||
}
|
|
||||||
|
|
||||||
noscript .no-js-message {
|
|
||||||
display: block;
|
|
||||||
}
|
|
||||||
</style>
|
|
||||||
{% endblock %}
|
|
||||||
|
|
||||||
{% block admin_content %}
|
|
||||||
<div class="metrics-dashboard">
|
|
||||||
<h2>Metrics Dashboard</h2>
|
|
||||||
|
|
||||||
<noscript>
|
|
||||||
<div class="no-js-message">
|
|
||||||
Note: Auto-refresh and charts require JavaScript. Data is displayed below in text format.
|
|
||||||
</div>
|
|
||||||
</noscript>
|
|
||||||
|
|
||||||
<!-- Auto-refresh container -->
|
|
||||||
<div hx-get="{{ url_for('admin.metrics') }}" hx-trigger="every 10s" hx-swap="none" hx-on::after-request="updateDashboard(event)"></div>
|
|
||||||
|
|
||||||
<!-- Database Pool Statistics -->
|
|
||||||
<div class="metrics-grid">
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Database Connection Pool</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Active Connections</span>
|
|
||||||
<span class="metric-detail-value" id="pool-active">{{ pool.active_connections|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Idle Connections</span>
|
|
||||||
<span class="metric-detail-value" id="pool-idle">{{ pool.idle_connections|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Total Connections</span>
|
|
||||||
<span class="metric-detail-value" id="pool-total">{{ pool.total_connections|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Pool Size</span>
|
|
||||||
<span class="metric-detail-value" id="pool-size">{{ pool.pool_size|default(5) }}</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Database Operations</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Total Queries</span>
|
|
||||||
<span class="metric-detail-value" id="db-total">{{ metrics.database.count|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Average Time</span>
|
|
||||||
<span class="metric-detail-value" id="db-avg">{{ "%.2f"|format(metrics.database.avg|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Min Time</span>
|
|
||||||
<span class="metric-detail-value" id="db-min">{{ "%.2f"|format(metrics.database.min|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Max Time</span>
|
|
||||||
<span class="metric-detail-value" id="db-max">{{ "%.2f"|format(metrics.database.max|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>HTTP Requests</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Total Requests</span>
|
|
||||||
<span class="metric-detail-value" id="http-total">{{ metrics.http.count|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Average Time</span>
|
|
||||||
<span class="metric-detail-value" id="http-avg">{{ "%.2f"|format(metrics.http.avg|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Min Time</span>
|
|
||||||
<span class="metric-detail-value" id="http-min">{{ "%.2f"|format(metrics.http.min|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Max Time</span>
|
|
||||||
<span class="metric-detail-value" id="http-max">{{ "%.2f"|format(metrics.http.max|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Template Rendering</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Total Renders</span>
|
|
||||||
<span class="metric-detail-value" id="render-total">{{ metrics.render.count|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Average Time</span>
|
|
||||||
<span class="metric-detail-value" id="render-avg">{{ "%.2f"|format(metrics.render.avg|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Min Time</span>
|
|
||||||
<span class="metric-detail-value" id="render-min">{{ "%.2f"|format(metrics.render.min|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Max Time</span>
|
|
||||||
<span class="metric-detail-value" id="render-max">{{ "%.2f"|format(metrics.render.max|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- Charts -->
|
|
||||||
<div class="metrics-grid">
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Connection Pool Usage</h3>
|
|
||||||
<div class="chart-container">
|
|
||||||
<canvas id="poolChart"></canvas>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Performance Overview</h3>
|
|
||||||
<div class="chart-container">
|
|
||||||
<canvas id="performanceChart"></canvas>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- Feed Statistics (Phase 3) -->
|
|
||||||
<h2 style="margin-top: 40px;">Feed Statistics</h2>
|
|
||||||
<div class="metrics-grid">
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Feed Requests by Format</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">RSS</span>
|
|
||||||
<span class="metric-detail-value" id="feed-rss-total">{{ feeds.by_format.rss.total|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">ATOM</span>
|
|
||||||
<span class="metric-detail-value" id="feed-atom-total">{{ feeds.by_format.atom.total|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">JSON Feed</span>
|
|
||||||
<span class="metric-detail-value" id="feed-json-total">{{ feeds.by_format.json.total|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Total Requests</span>
|
|
||||||
<span class="metric-detail-value" id="feed-total">{{ feeds.total_requests|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Feed Cache Statistics</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Cache Hits</span>
|
|
||||||
<span class="metric-detail-value" id="feed-cache-hits">{{ feeds.cache.hits|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Cache Misses</span>
|
|
||||||
<span class="metric-detail-value" id="feed-cache-misses">{{ feeds.cache.misses|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Hit Rate</span>
|
|
||||||
<span class="metric-detail-value" id="feed-cache-hit-rate">{{ "%.1f"|format(feeds.cache.hit_rate|default(0) * 100) }}%</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">Cached Entries</span>
|
|
||||||
<span class="metric-detail-value" id="feed-cache-entries">{{ feeds.cache.entries|default(0) }}</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Feed Generation Performance</h3>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">RSS Avg Time</span>
|
|
||||||
<span class="metric-detail-value" id="feed-rss-avg">{{ "%.2f"|format(feeds.by_format.rss.avg_duration_ms|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">ATOM Avg Time</span>
|
|
||||||
<span class="metric-detail-value" id="feed-atom-avg">{{ "%.2f"|format(feeds.by_format.atom.avg_duration_ms|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
<div class="metric-detail">
|
|
||||||
<span class="metric-detail-label">JSON Avg Time</span>
|
|
||||||
<span class="metric-detail-value" id="feed-json-avg">{{ "%.2f"|format(feeds.by_format.json.avg_duration_ms|default(0)) }} ms</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- Feed Charts -->
|
|
||||||
<div class="metrics-grid">
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Format Popularity</h3>
|
|
||||||
<div class="chart-container">
|
|
||||||
<canvas id="feedFormatChart"></canvas>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="metric-card">
|
|
||||||
<h3>Cache Efficiency</h3>
|
|
||||||
<div class="chart-container">
|
|
||||||
<canvas id="feedCacheChart"></canvas>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="refresh-info">
|
|
||||||
Auto-refresh every 10 seconds (requires JavaScript)
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<script>
|
|
||||||
// Initialize charts with current data
|
|
||||||
let poolChart, performanceChart, feedFormatChart, feedCacheChart;
|
|
||||||
|
|
||||||
function initCharts() {
|
|
||||||
// Pool usage chart (doughnut)
|
|
||||||
const poolCtx = document.getElementById('poolChart');
|
|
||||||
if (poolCtx && !poolChart) {
|
|
||||||
poolChart = new Chart(poolCtx, {
|
|
||||||
type: 'doughnut',
|
|
||||||
data: {
|
|
||||||
labels: ['Active', 'Idle'],
|
|
||||||
datasets: [{
|
|
||||||
data: [
|
|
||||||
{{ pool.active_connections|default(0) }},
|
|
||||||
{{ pool.idle_connections|default(0) }}
|
|
||||||
],
|
|
||||||
backgroundColor: ['#007bff', '#6c757d'],
|
|
||||||
borderWidth: 1
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
options: {
|
|
||||||
responsive: true,
|
|
||||||
maintainAspectRatio: false,
|
|
||||||
plugins: {
|
|
||||||
legend: {
|
|
||||||
position: 'bottom'
|
|
||||||
},
|
|
||||||
title: {
|
|
||||||
display: true,
|
|
||||||
text: 'Connection Distribution'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
// Performance chart (bar)
|
|
||||||
const perfCtx = document.getElementById('performanceChart');
|
|
||||||
if (perfCtx && !performanceChart) {
|
|
||||||
performanceChart = new Chart(perfCtx, {
|
|
||||||
type: 'bar',
|
|
||||||
data: {
|
|
||||||
labels: ['Database', 'HTTP', 'Render'],
|
|
||||||
datasets: [{
|
|
||||||
label: 'Average Time (ms)',
|
|
||||||
data: [
|
|
||||||
{{ metrics.database.avg|default(0) }},
|
|
||||||
{{ metrics.http.avg|default(0) }},
|
|
||||||
{{ metrics.render.avg|default(0) }}
|
|
||||||
],
|
|
||||||
backgroundColor: ['#007bff', '#28a745', '#ffc107'],
|
|
||||||
borderWidth: 1
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
options: {
|
|
||||||
responsive: true,
|
|
||||||
maintainAspectRatio: false,
|
|
||||||
scales: {
|
|
||||||
y: {
|
|
||||||
beginAtZero: true,
|
|
||||||
title: {
|
|
||||||
display: true,
|
|
||||||
text: 'Milliseconds'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
plugins: {
|
|
||||||
legend: {
|
|
||||||
display: false
|
|
||||||
},
|
|
||||||
title: {
|
|
||||||
display: true,
|
|
||||||
text: 'Average Response Times'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
// Feed format chart (pie)
|
|
||||||
const feedFormatCtx = document.getElementById('feedFormatChart');
|
|
||||||
if (feedFormatCtx && !feedFormatChart) {
|
|
||||||
feedFormatChart = new Chart(feedFormatCtx, {
|
|
||||||
type: 'pie',
|
|
||||||
data: {
|
|
||||||
labels: ['RSS', 'ATOM', 'JSON Feed'],
|
|
||||||
datasets: [{
|
|
||||||
data: [
|
|
||||||
{{ feeds.by_format.rss.total|default(0) }},
|
|
||||||
{{ feeds.by_format.atom.total|default(0) }},
|
|
||||||
{{ feeds.by_format.json.total|default(0) }}
|
|
||||||
],
|
|
||||||
backgroundColor: ['#ff6384', '#36a2eb', '#ffce56'],
|
|
||||||
borderWidth: 1
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
options: {
|
|
||||||
responsive: true,
|
|
||||||
maintainAspectRatio: false,
|
|
||||||
plugins: {
|
|
||||||
legend: {
|
|
||||||
position: 'bottom'
|
|
||||||
},
|
|
||||||
title: {
|
|
||||||
display: true,
|
|
||||||
text: 'Feed Format Distribution'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
// Feed cache chart (doughnut)
|
|
||||||
const feedCacheCtx = document.getElementById('feedCacheChart');
|
|
||||||
if (feedCacheCtx && !feedCacheChart) {
|
|
||||||
feedCacheChart = new Chart(feedCacheCtx, {
|
|
||||||
type: 'doughnut',
|
|
||||||
data: {
|
|
||||||
labels: ['Cache Hits', 'Cache Misses'],
|
|
||||||
datasets: [{
|
|
||||||
data: [
|
|
||||||
{{ feeds.cache.hits|default(0) }},
|
|
||||||
{{ feeds.cache.misses|default(0) }}
|
|
||||||
],
|
|
||||||
backgroundColor: ['#28a745', '#dc3545'],
|
|
||||||
borderWidth: 1
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
options: {
|
|
||||||
responsive: true,
|
|
||||||
maintainAspectRatio: false,
|
|
||||||
plugins: {
|
|
||||||
legend: {
|
|
||||||
position: 'bottom'
|
|
||||||
},
|
|
||||||
title: {
|
|
||||||
display: true,
|
|
||||||
text: 'Cache Hit/Miss Ratio'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Update dashboard with new data from htmx
|
|
||||||
function updateDashboard(event) {
|
|
||||||
if (!event.detail.xhr) return;
|
|
||||||
|
|
||||||
try {
|
|
||||||
const data = JSON.parse(event.detail.xhr.responseText);
|
|
||||||
|
|
||||||
// Update pool statistics
|
|
||||||
if (data.database && data.database.pool) {
|
|
||||||
const pool = data.database.pool;
|
|
||||||
document.getElementById('pool-active').textContent = pool.active_connections || 0;
|
|
||||||
document.getElementById('pool-idle').textContent = pool.idle_connections || 0;
|
|
||||||
document.getElementById('pool-total').textContent = pool.total_connections || 0;
|
|
||||||
document.getElementById('pool-size').textContent = pool.pool_size || 5;
|
|
||||||
|
|
||||||
// Update pool chart
|
|
||||||
if (poolChart) {
|
|
||||||
poolChart.data.datasets[0].data = [
|
|
||||||
pool.active_connections || 0,
|
|
||||||
pool.idle_connections || 0
|
|
||||||
];
|
|
||||||
poolChart.update();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Update performance metrics
|
|
||||||
if (data.performance) {
|
|
||||||
const perf = data.performance;
|
|
||||||
|
|
||||||
// Database
|
|
||||||
if (perf.database) {
|
|
||||||
document.getElementById('db-total').textContent = perf.database.count || 0;
|
|
||||||
document.getElementById('db-avg').textContent = (perf.database.avg || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('db-min').textContent = (perf.database.min || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('db-max').textContent = (perf.database.max || 0).toFixed(2) + ' ms';
|
|
||||||
}
|
|
||||||
|
|
||||||
// HTTP
|
|
||||||
if (perf.http) {
|
|
||||||
document.getElementById('http-total').textContent = perf.http.count || 0;
|
|
||||||
document.getElementById('http-avg').textContent = (perf.http.avg || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('http-min').textContent = (perf.http.min || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('http-max').textContent = (perf.http.max || 0).toFixed(2) + ' ms';
|
|
||||||
}
|
|
||||||
|
|
||||||
// Render
|
|
||||||
if (perf.render) {
|
|
||||||
document.getElementById('render-total').textContent = perf.render.count || 0;
|
|
||||||
document.getElementById('render-avg').textContent = (perf.render.avg || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('render-min').textContent = (perf.render.min || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('render-max').textContent = (perf.render.max || 0).toFixed(2) + ' ms';
|
|
||||||
}
|
|
||||||
|
|
||||||
// Update performance chart
|
|
||||||
if (performanceChart && perf.database && perf.http && perf.render) {
|
|
||||||
performanceChart.data.datasets[0].data = [
|
|
||||||
perf.database.avg || 0,
|
|
||||||
perf.http.avg || 0,
|
|
||||||
perf.render.avg || 0
|
|
||||||
];
|
|
||||||
performanceChart.update();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Update feed statistics
|
|
||||||
if (data.feeds) {
|
|
||||||
const feeds = data.feeds;
|
|
||||||
|
|
||||||
// Feed requests by format
|
|
||||||
if (feeds.by_format) {
|
|
||||||
document.getElementById('feed-rss-total').textContent = feeds.by_format.rss?.total || 0;
|
|
||||||
document.getElementById('feed-atom-total').textContent = feeds.by_format.atom?.total || 0;
|
|
||||||
document.getElementById('feed-json-total').textContent = feeds.by_format.json?.total || 0;
|
|
||||||
document.getElementById('feed-total').textContent = feeds.total_requests || 0;
|
|
||||||
|
|
||||||
// Feed generation performance
|
|
||||||
document.getElementById('feed-rss-avg').textContent = (feeds.by_format.rss?.avg_duration_ms || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('feed-atom-avg').textContent = (feeds.by_format.atom?.avg_duration_ms || 0).toFixed(2) + ' ms';
|
|
||||||
document.getElementById('feed-json-avg').textContent = (feeds.by_format.json?.avg_duration_ms || 0).toFixed(2) + ' ms';
|
|
||||||
|
|
||||||
// Update feed format chart
|
|
||||||
if (feedFormatChart) {
|
|
||||||
feedFormatChart.data.datasets[0].data = [
|
|
||||||
feeds.by_format.rss?.total || 0,
|
|
||||||
feeds.by_format.atom?.total || 0,
|
|
||||||
feeds.by_format.json?.total || 0
|
|
||||||
];
|
|
||||||
feedFormatChart.update();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Feed cache statistics
|
|
||||||
if (feeds.cache) {
|
|
||||||
document.getElementById('feed-cache-hits').textContent = feeds.cache.hits || 0;
|
|
||||||
document.getElementById('feed-cache-misses').textContent = feeds.cache.misses || 0;
|
|
||||||
document.getElementById('feed-cache-hit-rate').textContent = ((feeds.cache.hit_rate || 0) * 100).toFixed(1) + '%';
|
|
||||||
document.getElementById('feed-cache-entries').textContent = feeds.cache.entries || 0;
|
|
||||||
|
|
||||||
// Update feed cache chart
|
|
||||||
if (feedCacheChart) {
|
|
||||||
feedCacheChart.data.datasets[0].data = [
|
|
||||||
feeds.cache.hits || 0,
|
|
||||||
feeds.cache.misses || 0
|
|
||||||
];
|
|
||||||
feedCacheChart.update();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (e) {
|
|
||||||
console.error('Error updating dashboard:', e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Initialize charts when DOM is ready
|
|
||||||
if (document.readyState === 'loading') {
|
|
||||||
document.addEventListener('DOMContentLoaded', initCharts);
|
|
||||||
} else {
|
|
||||||
initCharts();
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
{% endblock %}
|
|
||||||
@@ -1,108 +0,0 @@
|
|||||||
"""
|
|
||||||
Integration tests for feed statistics in admin dashboard
|
|
||||||
|
|
||||||
Tests the feed statistics features in /admin/metrics-dashboard and /admin/metrics
|
|
||||||
per v1.1.2 Phase 3.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from starpunk.auth import create_session
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def authenticated_client(app, client):
|
|
||||||
"""Client with authenticated session"""
|
|
||||||
with app.test_request_context():
|
|
||||||
# Create a session for the test user
|
|
||||||
session_token = create_session(app.config["ADMIN_ME"])
|
|
||||||
|
|
||||||
# Set session cookie
|
|
||||||
client.set_cookie("starpunk_session", session_token)
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
def test_feed_statistics_dashboard_endpoint(authenticated_client):
|
|
||||||
"""Test metrics dashboard includes feed statistics section"""
|
|
||||||
response = authenticated_client.get("/admin/metrics-dashboard")
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
|
|
||||||
# Should contain feed statistics section
|
|
||||||
assert b"Feed Statistics" in response.data
|
|
||||||
assert b"Feed Requests by Format" in response.data
|
|
||||||
assert b"Feed Cache Statistics" in response.data
|
|
||||||
assert b"Feed Generation Performance" in response.data
|
|
||||||
|
|
||||||
# Should have chart canvases
|
|
||||||
assert b'id="feedFormatChart"' in response.data
|
|
||||||
assert b'id="feedCacheChart"' in response.data
|
|
||||||
|
|
||||||
|
|
||||||
def test_feed_statistics_metrics_endpoint(authenticated_client):
|
|
||||||
"""Test /admin/metrics endpoint includes feed statistics"""
|
|
||||||
response = authenticated_client.get("/admin/metrics")
|
|
||||||
|
|
||||||
assert response.status_code == 200
|
|
||||||
data = response.get_json()
|
|
||||||
|
|
||||||
# Should have feeds key
|
|
||||||
assert "feeds" in data
|
|
||||||
|
|
||||||
# Should have expected structure
|
|
||||||
feeds = data["feeds"]
|
|
||||||
if "error" not in feeds:
|
|
||||||
assert "by_format" in feeds
|
|
||||||
assert "cache" in feeds
|
|
||||||
assert "total_requests" in feeds
|
|
||||||
assert "format_percentages" in feeds
|
|
||||||
|
|
||||||
# Check format structure
|
|
||||||
for format_name in ["rss", "atom", "json"]:
|
|
||||||
assert format_name in feeds["by_format"]
|
|
||||||
fmt = feeds["by_format"][format_name]
|
|
||||||
assert "generated" in fmt
|
|
||||||
assert "cached" in fmt
|
|
||||||
assert "total" in fmt
|
|
||||||
assert "avg_duration_ms" in fmt
|
|
||||||
|
|
||||||
# Check cache structure
|
|
||||||
assert "hits" in feeds["cache"]
|
|
||||||
assert "misses" in feeds["cache"]
|
|
||||||
assert "hit_rate" in feeds["cache"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_feed_statistics_after_feed_request(authenticated_client):
|
|
||||||
"""Test feed statistics track actual feed requests"""
|
|
||||||
# Make a feed request
|
|
||||||
response = authenticated_client.get("/feed.rss")
|
|
||||||
assert response.status_code == 200
|
|
||||||
|
|
||||||
# Check metrics endpoint now has data
|
|
||||||
response = authenticated_client.get("/admin/metrics")
|
|
||||||
assert response.status_code == 200
|
|
||||||
data = response.get_json()
|
|
||||||
|
|
||||||
# Should have feeds data
|
|
||||||
assert "feeds" in data
|
|
||||||
feeds = data["feeds"]
|
|
||||||
|
|
||||||
# May have requests tracked (depends on metrics buffer timing)
|
|
||||||
# Just verify structure is correct
|
|
||||||
assert "total_requests" in feeds
|
|
||||||
assert feeds["total_requests"] >= 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_dashboard_requires_auth_for_feed_stats(client):
|
|
||||||
"""Test dashboard requires authentication (even for feed stats)"""
|
|
||||||
response = client.get("/admin/metrics-dashboard")
|
|
||||||
|
|
||||||
# Should redirect to auth or return 401/403
|
|
||||||
assert response.status_code in [302, 401, 403]
|
|
||||||
|
|
||||||
|
|
||||||
def test_metrics_endpoint_requires_auth_for_feed_stats(client):
|
|
||||||
"""Test metrics endpoint requires authentication"""
|
|
||||||
response = client.get("/admin/metrics")
|
|
||||||
|
|
||||||
# Should redirect to auth or return 401/403
|
|
||||||
assert response.status_code in [302, 401, 403]
|
|
||||||
@@ -1,459 +0,0 @@
|
|||||||
"""
|
|
||||||
Tests for metrics instrumentation (v1.1.2 Phase 1)
|
|
||||||
|
|
||||||
Tests database monitoring, HTTP metrics, memory monitoring, and business metrics.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
import sqlite3
|
|
||||||
import time
|
|
||||||
import threading
|
|
||||||
from unittest.mock import Mock, patch, MagicMock
|
|
||||||
|
|
||||||
from starpunk.monitoring import (
|
|
||||||
MonitoredConnection,
|
|
||||||
MemoryMonitor,
|
|
||||||
get_metrics,
|
|
||||||
get_metrics_stats,
|
|
||||||
business,
|
|
||||||
)
|
|
||||||
from starpunk.monitoring.metrics import get_buffer
|
|
||||||
from starpunk.monitoring.http import setup_http_metrics
|
|
||||||
|
|
||||||
|
|
||||||
class TestMonitoredConnection:
|
|
||||||
"""Tests for database operation monitoring"""
|
|
||||||
|
|
||||||
def test_execute_records_metric(self):
|
|
||||||
"""Test that execute() records a metric"""
|
|
||||||
# Create in-memory database
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
conn.execute('CREATE TABLE test (id INTEGER, name TEXT)')
|
|
||||||
|
|
||||||
# Wrap with monitoring
|
|
||||||
monitored = MonitoredConnection(conn, slow_query_threshold=1.0)
|
|
||||||
|
|
||||||
# Clear metrics buffer
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
# Execute query
|
|
||||||
monitored.execute('SELECT * FROM test')
|
|
||||||
|
|
||||||
# Check metric was recorded
|
|
||||||
metrics = get_metrics()
|
|
||||||
# Note: May not be recorded due to sampling, but slow queries are forced
|
|
||||||
# So we'll check stats instead
|
|
||||||
stats = get_metrics_stats()
|
|
||||||
assert stats['total_count'] >= 0 # May be 0 due to sampling
|
|
||||||
|
|
||||||
def test_slow_query_always_recorded(self):
|
|
||||||
"""Test that slow queries are always recorded regardless of sampling"""
|
|
||||||
# Create in-memory database
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
|
|
||||||
# Set very low threshold so any query is "slow"
|
|
||||||
monitored = MonitoredConnection(conn, slow_query_threshold=0.0)
|
|
||||||
|
|
||||||
# Clear metrics buffer
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
# Execute query (will be considered slow)
|
|
||||||
monitored.execute('SELECT 1')
|
|
||||||
|
|
||||||
# Check metric was recorded (forced due to being slow)
|
|
||||||
metrics = get_metrics()
|
|
||||||
assert len(metrics) > 0
|
|
||||||
# Check that is_slow is True in metadata
|
|
||||||
assert any(m.metadata.get('is_slow', False) is True for m in metrics)
|
|
||||||
|
|
||||||
def test_extract_table_name_select(self):
|
|
||||||
"""Test table name extraction from SELECT query"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
conn.execute('CREATE TABLE notes (id INTEGER)')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
table_name = monitored._extract_table_name('SELECT * FROM notes WHERE id = 1')
|
|
||||||
assert table_name == 'notes'
|
|
||||||
|
|
||||||
def test_extract_table_name_insert(self):
|
|
||||||
"""Test table name extraction from INSERT query"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
table_name = monitored._extract_table_name('INSERT INTO users (name) VALUES (?)')
|
|
||||||
assert table_name == 'users'
|
|
||||||
|
|
||||||
def test_extract_table_name_update(self):
|
|
||||||
"""Test table name extraction from UPDATE query"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
table_name = monitored._extract_table_name('UPDATE posts SET title = ?')
|
|
||||||
assert table_name == 'posts'
|
|
||||||
|
|
||||||
def test_extract_table_name_unknown(self):
|
|
||||||
"""Test that complex queries return 'unknown'"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
# Complex query with JOIN
|
|
||||||
table_name = monitored._extract_table_name(
|
|
||||||
'SELECT a.* FROM notes a JOIN users b ON a.user_id = b.id'
|
|
||||||
)
|
|
||||||
# Our simple regex will find 'notes' from the first FROM
|
|
||||||
assert table_name in ['notes', 'unknown']
|
|
||||||
|
|
||||||
def test_get_query_type(self):
|
|
||||||
"""Test query type extraction"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
assert monitored._get_query_type('SELECT * FROM notes') == 'SELECT'
|
|
||||||
assert monitored._get_query_type('INSERT INTO notes VALUES (?)') == 'INSERT'
|
|
||||||
assert monitored._get_query_type('UPDATE notes SET x = 1') == 'UPDATE'
|
|
||||||
assert monitored._get_query_type('DELETE FROM notes') == 'DELETE'
|
|
||||||
assert monitored._get_query_type('CREATE TABLE test (id INT)') == 'CREATE'
|
|
||||||
assert monitored._get_query_type('PRAGMA journal_mode=WAL') == 'PRAGMA'
|
|
||||||
|
|
||||||
def test_execute_with_parameters(self):
|
|
||||||
"""Test execute with query parameters"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
conn.execute('CREATE TABLE test (id INTEGER, name TEXT)')
|
|
||||||
monitored = MonitoredConnection(conn, slow_query_threshold=1.0)
|
|
||||||
|
|
||||||
# Execute with parameters
|
|
||||||
monitored.execute('INSERT INTO test (id, name) VALUES (?, ?)', (1, 'test'))
|
|
||||||
|
|
||||||
# Verify data was inserted
|
|
||||||
cursor = monitored.execute('SELECT * FROM test WHERE id = ?', (1,))
|
|
||||||
rows = cursor.fetchall()
|
|
||||||
assert len(rows) == 1
|
|
||||||
|
|
||||||
def test_executemany(self):
|
|
||||||
"""Test executemany batch operations"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
conn.execute('CREATE TABLE test (id INTEGER, name TEXT)')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
# Clear metrics
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
# Execute batch insert
|
|
||||||
data = [(1, 'first'), (2, 'second'), (3, 'third')]
|
|
||||||
monitored.executemany('INSERT INTO test (id, name) VALUES (?, ?)', data)
|
|
||||||
|
|
||||||
# Check metric was recorded
|
|
||||||
metrics = get_metrics()
|
|
||||||
# May not be recorded due to sampling
|
|
||||||
stats = get_metrics_stats()
|
|
||||||
assert stats is not None
|
|
||||||
|
|
||||||
def test_error_recording(self):
|
|
||||||
"""Test that errors are recorded in metrics"""
|
|
||||||
conn = sqlite3.connect(':memory:')
|
|
||||||
monitored = MonitoredConnection(conn)
|
|
||||||
|
|
||||||
# Clear metrics
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
# Execute invalid query
|
|
||||||
with pytest.raises(sqlite3.OperationalError):
|
|
||||||
monitored.execute('SELECT * FROM nonexistent_table')
|
|
||||||
|
|
||||||
# Check error was recorded (forced)
|
|
||||||
metrics = get_metrics()
|
|
||||||
assert len(metrics) > 0
|
|
||||||
assert any('ERROR' in m.operation_name for m in metrics)
|
|
||||||
|
|
||||||
|
|
||||||
class TestHTTPMetrics:
|
|
||||||
"""Tests for HTTP request/response monitoring"""
|
|
||||||
|
|
||||||
def test_setup_http_metrics(self, app):
|
|
||||||
"""Test HTTP metrics middleware setup"""
|
|
||||||
# Add a simple test route
|
|
||||||
@app.route('/test')
|
|
||||||
def test_route():
|
|
||||||
return 'OK', 200
|
|
||||||
|
|
||||||
setup_http_metrics(app)
|
|
||||||
|
|
||||||
# Clear metrics
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
# Make a request
|
|
||||||
with app.test_client() as client:
|
|
||||||
response = client.get('/test')
|
|
||||||
assert response.status_code == 200
|
|
||||||
|
|
||||||
# Check request ID header was added
|
|
||||||
assert 'X-Request-ID' in response.headers
|
|
||||||
|
|
||||||
# Check metrics were recorded
|
|
||||||
metrics = get_metrics()
|
|
||||||
# May be sampled, so just check structure
|
|
||||||
stats = get_metrics_stats()
|
|
||||||
assert stats is not None
|
|
||||||
|
|
||||||
def test_request_id_generation(self, app):
|
|
||||||
"""Test that unique request IDs are generated"""
|
|
||||||
# Add a simple test route
|
|
||||||
@app.route('/test')
|
|
||||||
def test_route():
|
|
||||||
return 'OK', 200
|
|
||||||
|
|
||||||
setup_http_metrics(app)
|
|
||||||
|
|
||||||
request_ids = set()
|
|
||||||
|
|
||||||
with app.test_client() as client:
|
|
||||||
for _ in range(5):
|
|
||||||
response = client.get('/test')
|
|
||||||
request_id = response.headers.get('X-Request-ID')
|
|
||||||
assert request_id is not None
|
|
||||||
request_ids.add(request_id)
|
|
||||||
|
|
||||||
# All request IDs should be unique
|
|
||||||
assert len(request_ids) == 5
|
|
||||||
|
|
||||||
def test_error_metrics_recorded(self, app):
|
|
||||||
"""Test that errors are recorded in metrics"""
|
|
||||||
# Add a simple test route
|
|
||||||
@app.route('/test')
|
|
||||||
def test_route():
|
|
||||||
return 'OK', 200
|
|
||||||
|
|
||||||
setup_http_metrics(app)
|
|
||||||
|
|
||||||
# Clear metrics
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
with app.test_client() as client:
|
|
||||||
# Request non-existent endpoint
|
|
||||||
response = client.get('/this-does-not-exist')
|
|
||||||
assert response.status_code == 404
|
|
||||||
|
|
||||||
# Error metrics should be recorded (forced)
|
|
||||||
# Note: 404 is not necessarily an error in the teardown handler
|
|
||||||
# but will be in metrics as a 404 status code
|
|
||||||
metrics = get_metrics()
|
|
||||||
stats = get_metrics_stats()
|
|
||||||
assert stats is not None
|
|
||||||
|
|
||||||
|
|
||||||
class TestMemoryMonitor:
|
|
||||||
"""Tests for memory monitoring thread"""
|
|
||||||
|
|
||||||
def test_memory_monitor_initialization(self):
|
|
||||||
"""Test memory monitor can be initialized"""
|
|
||||||
monitor = MemoryMonitor(interval=1)
|
|
||||||
assert monitor.interval == 1
|
|
||||||
assert monitor.daemon is True # Per CQ5
|
|
||||||
|
|
||||||
def test_memory_monitor_starts_and_stops(self):
|
|
||||||
"""Test memory monitor thread lifecycle"""
|
|
||||||
monitor = MemoryMonitor(interval=1)
|
|
||||||
|
|
||||||
# Start monitor
|
|
||||||
monitor.start()
|
|
||||||
assert monitor.is_alive()
|
|
||||||
|
|
||||||
# Wait a bit for initialization
|
|
||||||
time.sleep(0.5)
|
|
||||||
|
|
||||||
# Stop monitor gracefully
|
|
||||||
monitor.stop()
|
|
||||||
# Give it time to finish gracefully
|
|
||||||
time.sleep(1.0)
|
|
||||||
monitor.join(timeout=5)
|
|
||||||
# Thread should have stopped
|
|
||||||
# Note: In rare cases daemon thread may still be cleaning up
|
|
||||||
if monitor.is_alive():
|
|
||||||
# Give it one more second
|
|
||||||
time.sleep(1.0)
|
|
||||||
assert not monitor.is_alive()
|
|
||||||
|
|
||||||
def test_memory_monitor_collects_metrics(self):
|
|
||||||
"""Test that memory monitor collects metrics"""
|
|
||||||
# Clear metrics
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
monitor = MemoryMonitor(interval=1)
|
|
||||||
monitor.start()
|
|
||||||
|
|
||||||
# Wait for baseline + one collection
|
|
||||||
time.sleep(7) # 5s baseline + 2s for collection
|
|
||||||
|
|
||||||
# Stop monitor
|
|
||||||
monitor.stop()
|
|
||||||
monitor.join(timeout=2)
|
|
||||||
|
|
||||||
# Check metrics were collected
|
|
||||||
metrics = get_metrics()
|
|
||||||
memory_metrics = [m for m in metrics if 'memory' in m.operation_name.lower()]
|
|
||||||
|
|
||||||
# Should have at least one memory metric
|
|
||||||
assert len(memory_metrics) > 0
|
|
||||||
|
|
||||||
def test_memory_monitor_stats(self):
|
|
||||||
"""Test memory monitor statistics"""
|
|
||||||
monitor = MemoryMonitor(interval=1)
|
|
||||||
monitor.start()
|
|
||||||
|
|
||||||
# Wait for baseline
|
|
||||||
time.sleep(6)
|
|
||||||
|
|
||||||
# Get stats
|
|
||||||
stats = monitor.get_stats()
|
|
||||||
assert stats['status'] == 'running'
|
|
||||||
assert 'current_rss_mb' in stats
|
|
||||||
assert 'baseline_rss_mb' in stats
|
|
||||||
assert stats['baseline_rss_mb'] > 0
|
|
||||||
|
|
||||||
monitor.stop()
|
|
||||||
monitor.join(timeout=2)
|
|
||||||
|
|
||||||
|
|
||||||
class TestBusinessMetrics:
|
|
||||||
"""Tests for business metrics tracking"""
|
|
||||||
|
|
||||||
def test_track_note_created(self):
|
|
||||||
"""Test note creation tracking"""
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
business.track_note_created(note_id=123, content_length=500, has_media=False)
|
|
||||||
|
|
||||||
metrics = get_metrics()
|
|
||||||
assert len(metrics) > 0
|
|
||||||
|
|
||||||
note_metrics = [m for m in metrics if 'note_created' in m.operation_name]
|
|
||||||
assert len(note_metrics) > 0
|
|
||||||
assert note_metrics[0].metadata['note_id'] == 123
|
|
||||||
assert note_metrics[0].metadata['content_length'] == 500
|
|
||||||
|
|
||||||
def test_track_note_updated(self):
|
|
||||||
"""Test note update tracking"""
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
business.track_note_updated(
|
|
||||||
note_id=456,
|
|
||||||
content_length=750,
|
|
||||||
fields_changed=['title', 'content']
|
|
||||||
)
|
|
||||||
|
|
||||||
metrics = get_metrics()
|
|
||||||
note_metrics = [m for m in metrics if 'note_updated' in m.operation_name]
|
|
||||||
assert len(note_metrics) > 0
|
|
||||||
assert note_metrics[0].metadata['note_id'] == 456
|
|
||||||
|
|
||||||
def test_track_note_deleted(self):
|
|
||||||
"""Test note deletion tracking"""
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
business.track_note_deleted(note_id=789)
|
|
||||||
|
|
||||||
metrics = get_metrics()
|
|
||||||
note_metrics = [m for m in metrics if 'note_deleted' in m.operation_name]
|
|
||||||
assert len(note_metrics) > 0
|
|
||||||
assert note_metrics[0].metadata['note_id'] == 789
|
|
||||||
|
|
||||||
def test_track_feed_generated(self):
|
|
||||||
"""Test feed generation tracking"""
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
business.track_feed_generated(
|
|
||||||
format='rss',
|
|
||||||
item_count=50,
|
|
||||||
duration_ms=45.2,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
metrics = get_metrics()
|
|
||||||
feed_metrics = [m for m in metrics if 'feed_rss' in m.operation_name]
|
|
||||||
assert len(feed_metrics) > 0
|
|
||||||
assert feed_metrics[0].metadata['format'] == 'rss'
|
|
||||||
assert feed_metrics[0].metadata['item_count'] == 50
|
|
||||||
|
|
||||||
def test_track_cache_hit(self):
|
|
||||||
"""Test cache hit tracking"""
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
business.track_cache_hit(cache_type='feed', key='rss:latest')
|
|
||||||
|
|
||||||
metrics = get_metrics()
|
|
||||||
cache_metrics = [m for m in metrics if 'cache_hit' in m.operation_name]
|
|
||||||
assert len(cache_metrics) > 0
|
|
||||||
|
|
||||||
def test_track_cache_miss(self):
|
|
||||||
"""Test cache miss tracking"""
|
|
||||||
get_buffer().clear()
|
|
||||||
|
|
||||||
business.track_cache_miss(cache_type='feed', key='atom:latest')
|
|
||||||
|
|
||||||
metrics = get_metrics()
|
|
||||||
cache_metrics = [m for m in metrics if 'cache_miss' in m.operation_name]
|
|
||||||
assert len(cache_metrics) > 0
|
|
||||||
|
|
||||||
|
|
||||||
class TestMetricsConfiguration:
|
|
||||||
"""Tests for metrics configuration"""
|
|
||||||
|
|
||||||
def test_metrics_can_be_disabled(self, app):
|
|
||||||
"""Test that metrics can be disabled via configuration"""
|
|
||||||
# This would be tested by setting METRICS_ENABLED=False
|
|
||||||
# and verifying no metrics are collected
|
|
||||||
assert 'METRICS_ENABLED' in app.config
|
|
||||||
|
|
||||||
def test_slow_query_threshold_configurable(self, app):
|
|
||||||
"""Test that slow query threshold is configurable"""
|
|
||||||
assert 'METRICS_SLOW_QUERY_THRESHOLD' in app.config
|
|
||||||
assert isinstance(app.config['METRICS_SLOW_QUERY_THRESHOLD'], float)
|
|
||||||
|
|
||||||
def test_sampling_rate_configurable(self, app):
|
|
||||||
"""Test that sampling rate is configurable"""
|
|
||||||
assert 'METRICS_SAMPLING_RATE' in app.config
|
|
||||||
assert isinstance(app.config['METRICS_SAMPLING_RATE'], float)
|
|
||||||
assert 0.0 <= app.config['METRICS_SAMPLING_RATE'] <= 1.0
|
|
||||||
|
|
||||||
def test_buffer_size_configurable(self, app):
|
|
||||||
"""Test that buffer size is configurable"""
|
|
||||||
assert 'METRICS_BUFFER_SIZE' in app.config
|
|
||||||
assert isinstance(app.config['METRICS_BUFFER_SIZE'], int)
|
|
||||||
assert app.config['METRICS_BUFFER_SIZE'] > 0
|
|
||||||
|
|
||||||
def test_memory_interval_configurable(self, app):
|
|
||||||
"""Test that memory monitor interval is configurable"""
|
|
||||||
assert 'METRICS_MEMORY_INTERVAL' in app.config
|
|
||||||
assert isinstance(app.config['METRICS_MEMORY_INTERVAL'], int)
|
|
||||||
assert app.config['METRICS_MEMORY_INTERVAL'] > 0
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def app():
|
|
||||||
"""Create test Flask app with minimal configuration"""
|
|
||||||
from flask import Flask
|
|
||||||
from pathlib import Path
|
|
||||||
import tempfile
|
|
||||||
|
|
||||||
app = Flask(__name__)
|
|
||||||
|
|
||||||
# Create temp directory for testing
|
|
||||||
temp_dir = tempfile.mkdtemp()
|
|
||||||
temp_path = Path(temp_dir)
|
|
||||||
|
|
||||||
# Minimal configuration to avoid migration issues
|
|
||||||
app.config.update({
|
|
||||||
'TESTING': True,
|
|
||||||
'DATABASE_PATH': temp_path / 'test.db',
|
|
||||||
'DATA_PATH': temp_path,
|
|
||||||
'NOTES_PATH': temp_path / 'notes',
|
|
||||||
'SESSION_SECRET': 'test-secret',
|
|
||||||
'ADMIN_ME': 'https://test.example.com',
|
|
||||||
'METRICS_ENABLED': True,
|
|
||||||
'METRICS_SLOW_QUERY_THRESHOLD': 1.0,
|
|
||||||
'METRICS_SAMPLING_RATE': 1.0,
|
|
||||||
'METRICS_BUFFER_SIZE': 1000,
|
|
||||||
'METRICS_MEMORY_INTERVAL': 30,
|
|
||||||
})
|
|
||||||
|
|
||||||
return app
|
|
||||||
@@ -1,103 +0,0 @@
|
|||||||
"""
|
|
||||||
Tests for feed statistics tracking
|
|
||||||
|
|
||||||
Tests feed statistics aggregation per v1.1.2 Phase 3.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from starpunk.monitoring.business import get_feed_statistics, track_feed_generated
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_feed_statistics_returns_structure():
|
|
||||||
"""Test get_feed_statistics returns expected structure"""
|
|
||||||
stats = get_feed_statistics()
|
|
||||||
|
|
||||||
# Check top-level keys
|
|
||||||
assert "by_format" in stats
|
|
||||||
assert "cache" in stats
|
|
||||||
assert "total_requests" in stats
|
|
||||||
assert "format_percentages" in stats
|
|
||||||
|
|
||||||
# Check by_format structure
|
|
||||||
assert "rss" in stats["by_format"]
|
|
||||||
assert "atom" in stats["by_format"]
|
|
||||||
assert "json" in stats["by_format"]
|
|
||||||
|
|
||||||
# Check format stats structure
|
|
||||||
for format_name in ["rss", "atom", "json"]:
|
|
||||||
fmt_stats = stats["by_format"][format_name]
|
|
||||||
assert "generated" in fmt_stats
|
|
||||||
assert "cached" in fmt_stats
|
|
||||||
assert "total" in fmt_stats
|
|
||||||
assert "avg_duration_ms" in fmt_stats
|
|
||||||
|
|
||||||
# Check cache structure
|
|
||||||
assert "hits" in stats["cache"]
|
|
||||||
assert "misses" in stats["cache"]
|
|
||||||
assert "hit_rate" in stats["cache"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_feed_statistics_empty_metrics():
|
|
||||||
"""Test get_feed_statistics with no metrics returns zeros"""
|
|
||||||
stats = get_feed_statistics()
|
|
||||||
|
|
||||||
# All values should be zero or empty
|
|
||||||
assert stats["total_requests"] >= 0
|
|
||||||
assert stats["cache"]["hit_rate"] >= 0.0
|
|
||||||
assert stats["cache"]["hit_rate"] <= 1.0
|
|
||||||
|
|
||||||
|
|
||||||
def test_feed_statistics_cache_hit_rate_calculation():
|
|
||||||
"""Test cache hit rate is calculated correctly"""
|
|
||||||
stats = get_feed_statistics()
|
|
||||||
|
|
||||||
# Hit rate should be between 0 and 1
|
|
||||||
assert 0.0 <= stats["cache"]["hit_rate"] <= 1.0
|
|
||||||
|
|
||||||
# If there are hits and misses, hit rate should be hits / (hits + misses)
|
|
||||||
if stats["cache"]["hits"] + stats["cache"]["misses"] > 0:
|
|
||||||
expected_rate = stats["cache"]["hits"] / (
|
|
||||||
stats["cache"]["hits"] + stats["cache"]["misses"]
|
|
||||||
)
|
|
||||||
assert abs(stats["cache"]["hit_rate"] - expected_rate) < 0.001
|
|
||||||
|
|
||||||
|
|
||||||
def test_feed_statistics_format_percentages():
|
|
||||||
"""Test format percentages sum to 1.0 when there are requests"""
|
|
||||||
stats = get_feed_statistics()
|
|
||||||
|
|
||||||
if stats["total_requests"] > 0:
|
|
||||||
total_percentage = sum(stats["format_percentages"].values())
|
|
||||||
# Should sum to approximately 1.0 (allowing for floating point errors)
|
|
||||||
assert abs(total_percentage - 1.0) < 0.001
|
|
||||||
|
|
||||||
|
|
||||||
def test_feed_statistics_total_requests_sum():
|
|
||||||
"""Test total_requests equals sum of all format totals"""
|
|
||||||
stats = get_feed_statistics()
|
|
||||||
|
|
||||||
format_total = sum(
|
|
||||||
fmt["total"] for fmt in stats["by_format"].values()
|
|
||||||
)
|
|
||||||
|
|
||||||
assert stats["total_requests"] == format_total
|
|
||||||
|
|
||||||
|
|
||||||
def test_track_feed_generated_records_metrics():
|
|
||||||
"""Test track_feed_generated creates metrics entries"""
|
|
||||||
# Note: This test just verifies the function runs without error.
|
|
||||||
# Actual metrics tracking is tested in integration tests.
|
|
||||||
track_feed_generated(
|
|
||||||
format="rss",
|
|
||||||
item_count=10,
|
|
||||||
duration_ms=50.5,
|
|
||||||
cached=False
|
|
||||||
)
|
|
||||||
|
|
||||||
# Get statistics - may be empty if metrics buffer hasn't persisted yet
|
|
||||||
stats = get_feed_statistics()
|
|
||||||
|
|
||||||
# Verify structure is correct
|
|
||||||
assert "total_requests" in stats
|
|
||||||
assert "by_format" in stats
|
|
||||||
assert "cache" in stats
|
|
||||||
Reference in new issue
Block a user