Compare commits

...
6 Commits
Author SHA1 Message Date
Phil 16925663a3 Merge branch 'feature/v2.2.0-bluesky' for v2.2.0 release
Build Container / build (push) Successful in 7s
2026-10-31 14:19:17 -06:00
PhilandClaude Opus 5.5 e92c27a84c feat: Crosspost notes to Bluesky (v2.2.0)
BlueskyClient behind the ADR-065 reconcile loop: plain text with link and
hashtag facets, images under 1 MB with alt text, link cards, app-password
sessions with refresh. Deletes follow; edits don't (ADR-066).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 14:19:17 -06:00
PhilandClaude Opus 5.5 ed63e70c46 Merge branch 'feature/v2.1.0-gotosocial-syndication' for v2.1.0 release
Build Container / build (push) Successful in 8s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:46:28 -06:00
PhilandClaude Opus 5.5 57b02ad6c7 feat: Crosspost notes to GoToSocial (v2.1.0)
Published notes are copied to a GoToSocial (Mastodon-API) account and
edits, unpublishing and deletes follow them (ADR-065). On by default for
new notes, untickable in the editor and selectable from Micropub via
mp-syndicate-to. A background worker reconciles each (note, target)
row with atomic claims, an idempotency key and a version so edits during
a send are not lost. Synced copies are linked with u-syndication.

Also fixes the Micropub Location header and search result links, which
pointed at /notes/<slug> (404), and the nested delete form on the edit
page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:46:26 -06:00
PhilandClaude Opus 5.5 94e8c672df release: v2.0.2 (CI registry login)
Build Container / build (push) Successful in 18s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:20:01 -06:00
PhilandClaude Opus 5.5 011f7cc87d ci: Log in to the registry with a write:package token
The automatic Actions token is refused by the container registry
(run 107: unauthorized). Use the REGISTRY_TOKEN repository secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:13:43 -06:00
22 changed files with 1685 additions and 26 deletions

No files matched your search

+13
View File
@@ -50,6 +50,19 @@ SESSION_LIFETIME=30
# Lifetime of Micropub access tokens StarPunk issues, in days (default: 90)
INDIEAUTH_TOKEN_LIFETIME_DAYS=90
# =============================================================================
# CROSSPOSTING (optional, v2.1.0)
# =============================================================================
# GoToSocial or Mastodon account to copy new notes to. Create an app token
# with scopes: write:statuses write:media
GOTOSOCIAL_URL=
GOTOSOCIAL_ACCESS_TOKEN=
# Name shown in the editor and Micropub apps (default: GoToSocial)
GOTOSOCIAL_NAME=GoToSocial
# public, unlisted, private (default: public)
GOTOSOCIAL_VISIBILITY=public
# =============================================================================
# DATA STORAGE
# =============================================================================
+4 -1
View File
@@ -23,8 +23,11 @@ jobs:
run: apk add --no-cache nodejs git docker-cli docker-cli-buildx
- name: Checkout
uses: actions/checkout@v4
# The automatic Actions token can't log in to the container registry, so
# this uses a personal access token with only the write:package scope,
# stored as the repository secret REGISTRY_TOKEN.
- name: Log in to the Gitea registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login "$REGISTRY" -u "${{ github.actor }}" --password-stdin
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login "$REGISTRY" -u "${{ github.repository_owner }}" --password-stdin
- name: Use the BuildKit socket
run: docker buildx create --name ci --driver remote "$BUILDKIT" --use
- name: Build and push
+38
View File
@@ -7,6 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.2.0] - 2026-10-31
### Added
- **Crossposting to Bluesky**, ADR-066
- Plain-text note (Markdown stripped) with clickable link and hashtag facets, shortened to 300 characters with the permalink kept
- Images (up to 4, shrunk under Bluesky's 1 MB limit) with alt text; notes without images get a link card
- Unpublishing, unticking and deleting remove the Bluesky post; edits don't follow (Bluesky has no edits)
- Logs in with an app password: `BLUESKY_HANDLE`, `BLUESKY_APP_PASSWORD` (optional `BLUESKY_SERVICE`, default `https://bsky.social`)
- Shown in the editor and Micropub `syndicate-to` next to GoToSocial
## [2.1.0] - 2026-10-31
### Added
- **Crossposting to GoToSocial** (or any Mastodon-API server), ADR-065
- Full note as Markdown with images (alt text from captions), permalink and hashtags
- Edits, unpublishing and deletes follow the note
- On by default for new notes; untick in the editor, or send `mp-syndicate-to` from Micropub apps
- Micropub `q=syndicate-to` and `q=config` list the target
- Editor shows crosspost status, errors and a retry button; note pages link the copy with `u-syndication`
- Configure with `GOTOSOCIAL_URL` and `GOTOSOCIAL_ACCESS_TOKEN` (scopes `write:statuses write:media`)
- Migration 012: `syndications`
### Fixed
- Micropub `Location` header pointed at `/notes/<slug>` (404); now the real permalink
- "Delete Note" on the edit page submitted the edit form (nested forms)
- Search results linked to `/notes/<slug>` (404); now `/note/<slug>`
## [2.0.2] - 2026-10-31
### Fixed
- Container builds in Gitea Actions log in to the registry with the
`REGISTRY_TOKEN` secret (a `write:package` token); the automatic Actions
token is refused by the registry. No application changes.
## [2.0.1] - 2026-10-31
### Fixed
@@ -0,0 +1,45 @@
# ADR-065: Direct Syndication (POSSE) to GoToSocial
## Status
Accepted (v2.1.0, 2026-10-31)
## Context
StarPunk posts were copied to other networks by Echofeed reading the feed.
That gives no control per post, no record of where a post went, and edits
and deletes never follow. The operator runs GoToSocial locally and wants
StarPunk to post there directly, with Bluesky likely next.
## Decision
1. **Publish here, copy out (POSSE).** Each (note, target) has a row in
`syndications` (migration 012). Creating, editing, unpublishing or
deleting a note marks its rows pending; a background worker reconciles
the remote post: create, update (`PUT /api/v1/statuses/:id`) or delete.
2. **Defaults.** New notes go to every configured target unless unticked in
the editor or a Micropub client sends `mp-syndicate-to` (an explicit
list, possibly empty). Existing notes are never backfilled.
3. **Content.** The full note as Markdown (`content_type: text/markdown`),
then the permalink and the note's tags as hashtags. Longer than the
target's limit: shortened at a word boundary with the link kept.
Images are uploaded first (the `large` variant) with their captions as
alt text.
4. **Delivery.** Rows are claimed atomically so the four gunicorn workers'
threads never send twice; an `Idempotency-Key` guards retries; a row
`version` keeps edits made during a send. Network errors and 5xx/429
retry with exponential backoff (8 attempts); other 4xx fail at once.
A remote post deleted by hand is posted again on the next edit.
5. **Backlinks.** Synced URLs are shown on the note page as `u-syndication`
links, and in the editor with status, last error and a retry button.
6. **Targets** are config-driven. One kind exists, a Mastodon-API client
(`GOTOSOCIAL_URL`, `GOTOSOCIAL_ACCESS_TOKEN`); it also works for
Mastodon. A Bluesky target would add a second client behind the same
reconcile loop.
## Consequences
- Echofeed must stop posting to GoToSocial or posts appear twice.
- The token needs only `write:statuses write:media`.
- `syndications.note_id` has no foreign key so a hard-deleted note's
remote post can still be deleted.
@@ -0,0 +1,37 @@
# ADR-066: Crossposting to Bluesky
## Status
Accepted (v2.2.0, 2026-10-31)
## Context
ADR-065 added direct crossposting to GoToSocial. Bluesky was still fed by
Echofeed, with the same problems: no per-post control, no record of the
copy, deletes never follow.
## Decision
1. **Second client, same loop.** `BlueskyClient` sits behind the reconcile
loop and `syndications` table from ADR-065, as target id `bluesky`.
Clients share a small interface: `create`, `update`, `delete`, `can_edit`.
2. **Auth.** An app password (`BLUESKY_HANDLE`, `BLUESKY_APP_PASSWORD`).
`createSession` against `BLUESKY_SERVICE` (default `https://bsky.social`);
writes go to the PDS named in the returned DID document. Expired access
tokens are refreshed, falling back to a new session.
3. **Content.** Plain text (Markdown rendered and stripped), then the
permalink and hashtags (at most a third of the space), shortened to 300
characters at a word boundary. Link and hashtag facets use UTF-8 byte
offsets. Images (up to 4) are re-encoded as JPEG if over 1 MB, with alt
text and aspect ratio; a note without images gets a link card.
4. **Edits don't follow.** Bluesky has no edit; rewriting the record would
break likes and replies, and delete-and-repost would lose them. The post
stays as first sent. Unpublish, untick and delete remove it.
5. **No double posts.** When a create is retried, recent posts are checked
for one already linking to the note (a send whose reply was lost).
## Consequences
- Echofeed is no longer needed; it must stop posting to Bluesky or posts
appear twice.
- A post deleted by hand on Bluesky is not re-created.
+29
View File
@@ -0,0 +1,29 @@
-- Migration 012: Crossposting (POSSE) to syndication targets
-- Version: 2.1.0
-- Per ADR-065: Direct syndication to GoToSocial / Mastodon-API servers
--
-- One row per (note, target). The worker reconciles the remote post with the
-- note: create, update, or delete as needed. note_id has no foreign key on
-- purpose: a row must outlive a hard-deleted note so the remote post can be
-- deleted too.
CREATE TABLE IF NOT EXISTS syndications (
id INTEGER PRIMARY KEY AUTOINCREMENT,
note_id INTEGER NOT NULL,
target TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
remote_id TEXT,
remote_url TEXT,
state TEXT NOT NULL DEFAULT 'pending'
CHECK (state IN ('pending', 'synced', 'failed')),
version INTEGER NOT NULL DEFAULT 1,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
claimed_until TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE (note_id, target)
);
CREATE INDEX IF NOT EXISTS idx_syndications_due ON syndications(state, next_attempt_at);
+11 -2
View File
@@ -336,6 +336,15 @@ def create_app(config=None):
if hasattr(app, 'webmention_worker') and app.webmention_worker.is_alive():
app.webmention_worker.stop()
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
app.syndication_worker = SyndicationWorker(
app, poll_interval=app.config["SYNDICATION_POLL_INTERVAL"]
)
app.syndication_worker.start()
app.logger.info("Syndication worker started")
# Health check endpoint for containers and monitoring
@app.route("/health")
def health_check():
@@ -440,5 +449,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.0.1"
__version_info__ = (2, 0, 1)
__version__ = "2.2.0"
__version_info__ = (2, 1, 0)
+15
View File
@@ -124,6 +124,21 @@ def load_config(app, config_override=None):
app.config["WEBMENTION_RATE_LIMIT_WINDOW"] = int(os.getenv("WEBMENTION_RATE_LIMIT_WINDOW", "60"))
app.config["TRUSTED_PROXY"] = os.getenv("TRUSTED_PROXY", "false").lower() == "true"
# Crossposting to GoToSocial / Mastodon-API (v2.1.0, ADR-065)
# Enabled when both URL and token are set
app.config["GOTOSOCIAL_URL"] = os.getenv("GOTOSOCIAL_URL", "")
app.config["GOTOSOCIAL_ACCESS_TOKEN"] = os.getenv("GOTOSOCIAL_ACCESS_TOKEN", "")
app.config["GOTOSOCIAL_NAME"] = os.getenv("GOTOSOCIAL_NAME", "GoToSocial")
app.config["GOTOSOCIAL_VISIBILITY"] = os.getenv("GOTOSOCIAL_VISIBILITY", "public")
app.config["GOTOSOCIAL_MAX_CHARS"] = int(os.getenv("GOTOSOCIAL_MAX_CHARS", "5000"))
app.config["GOTOSOCIAL_MAX_MEDIA"] = int(os.getenv("GOTOSOCIAL_MAX_MEDIA", "4"))
app.config["BLUESKY_HANDLE"] = os.getenv("BLUESKY_HANDLE", "")
app.config["BLUESKY_APP_PASSWORD"] = os.getenv("BLUESKY_APP_PASSWORD", "")
app.config["BLUESKY_SERVICE"] = os.getenv("BLUESKY_SERVICE", "https://bsky.social")
app.config["BLUESKY_NAME"] = os.getenv("BLUESKY_NAME", "Bluesky")
app.config["BLUESKY_MAX_CHARS"] = int(os.getenv("BLUESKY_MAX_CHARS", "300"))
app.config["SYNDICATION_POLL_INTERVAL"] = int(os.getenv("SYNDICATION_POLL_INTERVAL", "10"))
# Apply overrides if provided
if config_override:
app.config.update(config_override)
+31 -9
View File
@@ -30,6 +30,7 @@ from flask import Request, current_app, jsonify
from starpunk.notes import create_note, get_note, InvalidNoteDataError, NoteNotFoundError
from starpunk.indieauth import check_scope
from starpunk.utils import absolute_url_for
# Custom Exceptions
@@ -398,6 +399,17 @@ def handle_create(data: dict, token_info: dict):
elif isinstance(slug_value, str):
custom_slug = slug_value
# mp-syndicate-to is also a server command, not a property (v2.1.0).
# Absent means "all targets"; present means exactly those.
requested_targets = None
if isinstance(data, dict):
for key in ("mp-syndicate-to", "mp-syndicate-to[]"):
if key in data:
value = data[key]
requested_targets = (requested_targets or []) + (
value if isinstance(value, list) else [value]
)
# Normalize and extract properties
try:
properties = normalize_properties(data)
@@ -427,10 +439,14 @@ def handle_create(data: dict, token_info: dict):
if photos:
_attach_photos_to_note(note.id, photos)
# Build permalink URL
# Note: SITE_URL is normalized to include trailing slash (for IndieAuth spec compliance)
site_url = current_app.config.get("SITE_URL", "http://localhost:5000")
permalink = f"{site_url}notes/{note.slug}"
# Crosspost after photos are attached (v2.1.0)
from starpunk.syndication import get_targets, resolve_targets, select_targets
if requested_targets is None:
select_targets(note.id, list(get_targets()))
else:
select_targets(note.id, resolve_targets(requested_targets))
permalink = absolute_url_for("public.note", slug=note.slug)
# Return 201 Created with Location header
return "", 201, {"Location": permalink}
@@ -444,6 +460,13 @@ def handle_create(data: dict, token_info: dict):
)
def _syndicate_to() -> list:
"""Configured crosspost targets in Micropub syndicate-to form"""
from starpunk.syndication import get_targets
return [{"uid": t.uid, "name": t.name} for t in get_targets().values()]
def handle_query(args: dict, token_info: dict):
"""
Handle Micropub query endpoints
@@ -451,7 +474,7 @@ def handle_query(args: dict, token_info: dict):
Supports:
- q=config: Return server configuration
- q=source: Return post source in Microformats2 JSON
- q=syndicate-to: Return syndication targets (empty for V1)
- q=syndicate-to: Return syndication targets
Args:
args: Query string arguments
@@ -467,7 +490,7 @@ def handle_query(args: dict, token_info: dict):
site_url = current_app.config.get("SITE_URL", "http://localhost:5000").rstrip('/')
config = {
"media-endpoint": f"{site_url}/micropub/media",
"syndicate-to": [], # No syndication targets in V1
"syndicate-to": _syndicate_to(),
"post-types": [
{"type": "note", "name": "Note", "properties": ["content"]},
{"type": "photo", "name": "Photo", "properties": ["photo"]}
@@ -483,7 +506,7 @@ def handle_query(args: dict, token_info: dict):
# Extract slug from URL
try:
# URL format: https://example.com/notes/{slug}
# URL format: https://example.com/note/{slug}
slug = url.rstrip("/").split("/")[-1]
note = get_note(slug)
@@ -520,8 +543,7 @@ def handle_query(args: dict, token_info: dict):
return jsonify(mf2), 200
elif q == "syndicate-to":
# Return syndication targets (none for V1)
return jsonify({"syndicate-to": []}), 200
return jsonify({"syndicate-to": _syndicate_to()}), 200
else:
return error_response("invalid_request", f"Unknown query: {q}")
+14
View File
@@ -851,6 +851,13 @@ def update_note(
# Reply context fetch failure should not prevent note update
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
# Crossposts follow the note (v2.1.0, best-effort)
try:
from starpunk.syndication import note_changed
note_changed(updated_note.id)
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication update: {e}")
# 10. RETURN UPDATED NOTE
return updated_note
@@ -1018,5 +1025,12 @@ def delete_note(
)
# Don't fail - database record already deleted
# Delete crossposts too (v2.1.0, best-effort)
try:
from starpunk.syndication import note_changed
note_changed(existing_note.id)
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
# 6. RETURN (no value)
return None
+33 -2
View File
@@ -97,7 +97,9 @@ def new_note_form():
Decorator: @require_auth
Template: templates/admin/new.html
"""
return render_template("admin/new.html")
from starpunk.syndication import get_targets
return render_template("admin/new.html", syndication_targets=get_targets().values())
@bp.route("/new", methods=["POST"])
@@ -182,6 +184,11 @@ def create_note_submit():
if errors:
flash(f"Note created, but some images failed: {'; '.join(errors)}", "warning")
# Crosspost after media is attached so images go out too (v2.1.0)
if "syndication_form" in request.form:
from starpunk.syndication import select_targets
select_targets(note.id, request.form.getlist("syndicate_to"))
flash(f"Note created: {note.slug}", "success")
return redirect(url_for("admin.dashboard"))
@@ -223,7 +230,17 @@ def edit_note_form(note_id: int):
tags = get_note_tags(note.id)
object.__setattr__(note, '_cached_tags', tags)
return render_template("admin/edit.html", note=note)
from starpunk.syndication import get_note_syndications, get_targets
syndications = get_note_syndications(note.id)
enabled = {s["target"] for s in syndications if s["enabled"]}
return render_template(
"admin/edit.html",
note=note,
syndication_targets=get_targets().values(),
syndications=syndications,
syndication_enabled=enabled,
)
@bp.route("/edit/<int:note_id>", methods=["POST"])
@@ -274,6 +291,9 @@ def update_note_submit(note_id: int):
published=published,
tags=tags if tags else None
)
if "syndication_form" in request.form:
from starpunk.syndication import select_targets
select_targets(note.id, request.form.getlist("syndicate_to"))
flash(f"Note updated: {note.slug}", "success")
return redirect(url_for("admin.dashboard"))
except ValueError as e:
@@ -838,3 +858,14 @@ def revoke_token(token_id):
revoke_token_by_id(token_id)
flash("App access revoked", "success")
return redirect(url_for("admin.tokens"))
@bp.route("/syndication/<int:syndication_id>/retry", methods=["POST"])
@require_auth
def retry_syndication(syndication_id):
"""Retry a crosspost now"""
from starpunk.syndication import retry
retry(syndication_id)
flash("Crosspost queued", "success")
return redirect(request.referrer or url_for("admin.dashboard"))
+5
View File
@@ -331,9 +331,14 @@ def note(slug: str):
logger = logging.getLogger(__name__)
logger.error(f"Error loading webmentions for note {slug}: {e}")
from starpunk.syndication import get_note_syndications
syndications = [s for s in get_note_syndications(note_obj.id) if s["state"] == "synced"]
return render_template(
"note.html",
note=note_obj,
syndications=syndications,
webmentions=webmentions,
reply_context=reply_context,
**grouped if webmentions else {}
+2 -2
View File
@@ -108,7 +108,7 @@ def api_search():
"title": r["title"] or f"Note from {r['created_at'][:10]}",
"excerpt": r["snippet"], # Already has <mark> tags
"published_at": r["created_at"],
"url": f"/notes/{r['slug']}",
"url": f"/note/{r['slug']}",
}
for r in results
],
@@ -176,7 +176,7 @@ def search_page():
"title": r["title"] or f"Note from {r['created_at'][:10]}",
"excerpt": Markup(safe_snippet), # Mark as safe since we've escaped it ourselves
"published_at": r["created_at"],
"url": f"/notes/{r['slug']}",
"url": f"/note/{r['slug']}",
})
results = formatted_results
except Exception as e:
+716
View File
@@ -0,0 +1,716 @@
"""
Crossposting (POSSE) for StarPunk (v2.1.0 ADR-065, v2.2.0 ADR-066)
Published notes are copied to syndication targets, and later edits and
deletes follow them. Two target kinds exist: a GoToSocial (or any
Mastodon-API) account, and a Bluesky account. Bluesky posts can't be edited,
so there only deletes follow.
Each (note, target) pair has a row in `syndications`. Changing a note marks
its rows pending; a background worker then reconciles the remote post with
the note:
note published + target enabled, no remote post -> create
note published + target enabled, remote post -> update
otherwise, remote post exists -> delete
Rows carry a version so an edit made while a send is in flight is not lost.
"""
import io
import re
import threading
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional
import httpx
from flask import current_app
from starpunk.database import get_db
MAX_ATTEMPTS = 8
CLAIM_MINUTES = 5
class SyndicationError(Exception):
"""A send failed. permanent=True means retrying won't help."""
def __init__(self, message: str, permanent: bool = False):
super().__init__(message)
self.permanent = permanent
class RemoteGone(SyndicationError):
"""The remote post no longer exists"""
class MastodonClient:
"""Minimal Mastodon-API client (works with GoToSocial)"""
def __init__(self, base_url: str, token: str, visibility: str = "public",
http: Optional[httpx.Client] = None):
self.base_url = base_url.rstrip("/")
self.visibility = visibility
self.http = http or httpx.Client(timeout=30.0)
self.headers = {"Authorization": f"Bearer {token}"}
def _call(self, method: str, path: str, **kwargs) -> httpx.Response:
headers = {**self.headers, **kwargs.pop("headers", {})}
try:
response = self.http.request(method, self.base_url + path, headers=headers, **kwargs)
except httpx.HTTPError as e:
raise SyndicationError(f"{method} {path}: {e}")
if response.status_code == 404:
raise RemoteGone(f"{method} {path}: not found", permanent=True)
if response.status_code >= 400:
# 4xx other than rate limiting won't fix itself
permanent = 400 <= response.status_code < 500 and response.status_code != 429
raise SyndicationError(
f"{method} {path}: HTTP {response.status_code} {response.text[:200]}",
permanent=permanent,
)
return response
def upload_media(self, path: Path, mime_type: str, description: str) -> str:
with open(path, "rb") as f:
response = self._call(
"POST", "/api/v2/media",
files={"file": (path.name, f, mime_type)},
data={"description": description or ""},
)
return response.json()["id"]
def _status_body(self, text: str, media_ids: List[str]) -> dict:
return {
"status": text,
"media_ids": media_ids,
"visibility": self.visibility,
"content_type": "text/markdown",
}
def create_status(self, text: str, media_ids: List[str], idempotency_key: str) -> dict:
response = self._call(
"POST", "/api/v1/statuses",
json=self._status_body(text, media_ids),
headers={"Idempotency-Key": idempotency_key},
)
data = response.json()
return {"id": data["id"], "url": data.get("url") or data.get("uri")}
def update_status(self, remote_id: str, text: str, media_ids: List[str]) -> dict:
response = self._call("PUT", f"/api/v1/statuses/{remote_id}",
json=self._status_body(text, media_ids))
data = response.json()
return {"id": data["id"], "url": data.get("url") or data.get("uri")}
def delete_status(self, remote_id: str) -> None:
try:
self._call("DELETE", f"/api/v1/statuses/{remote_id}")
except RemoteGone:
pass # already gone
# Interface used by the reconcile loop
can_edit = True
def _prepare(self, post: "Post") -> tuple:
text = compose_text(post.note, post.permalink, post.tags, post.max_chars)
media_ids = [self.upload_media(f["path"], f["mime_type"], f["description"]) for f in post.media]
return text, media_ids
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
text, media_ids = self._prepare(post)
return self.create_status(text, media_ids, idempotency_key=key)
def update(self, remote_id: str, post: "Post") -> dict:
text, media_ids = self._prepare(post)
return self.update_status(remote_id, text, media_ids)
def delete(self, remote_id: str) -> None:
self.delete_status(remote_id)
BLUESKY_BLOB_LIMIT = 1_000_000 # app.bsky.embed.images maxSize
_TID_CHARS = "234567abcdefghijklmnopqrstuvwxyz"
class BlueskyClient:
"""
Minimal AT Protocol client for posting to Bluesky with an app password
Posts are plain text with link and hashtag facets. Images go in an
images embed; a note without images gets a link card instead.
"""
can_edit = False # Bluesky has no edits; only deletes follow the note
POST = "app.bsky.feed.post"
def __init__(self, service: str, identifier: str, password: str,
http: Optional[httpx.Client] = None):
self.service = service.rstrip("/")
self.identifier = identifier
self.password = password
self.http = http or httpx.Client(timeout=30.0)
self.pds = self.service
self.did: Optional[str] = None
self.handle = identifier
self._access: Optional[str] = None
self._refresh: Optional[str] = None
self._lock = threading.Lock()
# --- XRPC plumbing ---
def _raw(self, method: str, nsid: str, base: str, token: Optional[str], **kwargs) -> httpx.Response:
headers = kwargs.pop("headers", {})
if token:
headers["Authorization"] = f"Bearer {token}"
try:
return self.http.request(method, f"{base}/xrpc/{nsid}", headers=headers, **kwargs)
except httpx.HTTPError as e:
raise SyndicationError(f"{nsid}: {e}")
@staticmethod
def _error_name(response: httpx.Response) -> str:
try:
return response.json().get("error") or ""
except ValueError:
return ""
def _raise_for(self, nsid: str, response: httpx.Response) -> None:
if response.status_code < 400:
return
permanent = 400 <= response.status_code < 500 and response.status_code != 429
raise SyndicationError(
f"{nsid}: HTTP {response.status_code} {response.text[:200]}", permanent=permanent
)
def _login(self) -> None:
response = None
if self._refresh:
response = self._raw("POST", "com.atproto.server.refreshSession", self.service, self._refresh)
if response.status_code >= 400:
response = None
if response is None:
response = self._raw("POST", "com.atproto.server.createSession", self.service, None,
json={"identifier": self.identifier, "password": self.password})
self._raise_for("com.atproto.server.createSession", response)
data = response.json()
self._access, self._refresh = data["accessJwt"], data["refreshJwt"]
self.did = data["did"]
self.handle = data.get("handle") or self.handle
for service in (data.get("didDoc") or {}).get("service", []):
if service.get("id", "").endswith("#atproto_pds") and service.get("serviceEndpoint"):
self.pds = service["serviceEndpoint"].rstrip("/")
def _call(self, method: str, nsid: str, **kwargs) -> httpx.Response:
with self._lock:
if not self._access:
self._login()
response = self._raw(method, nsid, self.pds, self._access, **kwargs)
if response.status_code == 401 or self._error_name(response) == "ExpiredToken":
self._access = None
self._login()
response = self._raw(method, nsid, self.pds, self._access, **kwargs)
self._raise_for(nsid, response)
return response
# --- Posts ---
def upload_blob(self, data: bytes, mime_type: str) -> dict:
response = self._call("POST", "com.atproto.repo.uploadBlob", content=data,
headers={"Content-Type": mime_type})
return response.json()["blob"]
def _image(self, media: dict) -> dict:
data, mime_type, width, height = _fit_image(media["path"], media["mime_type"], BLUESKY_BLOB_LIMIT)
image = {"image": self.upload_blob(data, mime_type), "alt": media["description"] or ""}
if width and height:
image["aspectRatio"] = {"width": width, "height": height}
return image
def _record(self, post: "Post") -> dict:
text = compose_text(post.note, post.permalink, post.tags, post.max_chars,
body=plain_text(post.note.content))
record = {
"$type": self.POST,
"text": text,
"createdAt": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z",
}
facets = bluesky_facets(text)
if facets:
record["facets"] = facets
if post.media:
record["embed"] = {"$type": "app.bsky.embed.images",
"images": [self._image(m) for m in post.media]}
else:
record["embed"] = {"$type": "app.bsky.embed.external", "external": {
"uri": post.permalink,
"title": post.note.title[:300],
"description": plain_text(post.note.content)[:300],
}}
return record
def _web_url(self, uri: str) -> str:
rkey = uri.rsplit("/", 1)[1]
return f"https://bsky.app/profile/{self.handle or self.did}/post/{rkey}"
def _find_existing(self, permalink: str) -> Optional[dict]:
"""A recent post that already links to this note (an earlier send whose reply was lost)"""
response = self._call("GET", "com.atproto.repo.listRecords",
params={"repo": self.did, "collection": self.POST, "limit": 25})
for item in response.json().get("records", []):
if permalink in (item.get("value") or {}).get("text", ""):
return {"id": item["uri"], "url": self._web_url(item["uri"])}
return None
def create(self, post: "Post", key: str, retrying: bool = False) -> dict:
if retrying:
if not self.did:
with self._lock:
self._login()
existing = self._find_existing(post.permalink)
if existing:
return existing
record = self._record(post)
response = self._call("POST", "com.atproto.repo.createRecord",
json={"repo": self.did or self.identifier, "collection": self.POST,
"record": record})
uri = response.json()["uri"]
return {"id": uri, "url": self._web_url(uri)}
def update(self, remote_id: str, post: "Post") -> dict:
raise NotImplementedError("Bluesky posts can't be edited")
def delete(self, remote_id: str) -> None:
# at://<did>/app.bsky.feed.post/<rkey>; deleting a missing record succeeds
repo, collection, rkey = remote_id[len("at://"):].split("/", 2)
self._call("POST", "com.atproto.repo.deleteRecord",
json={"repo": repo, "collection": collection, "rkey": rkey})
def _fit_image(path: Path, mime_type: str, limit: int) -> tuple:
"""Image bytes no bigger than `limit`, re-encoded as JPEG if needed. Returns (data, mime, w, h)."""
from PIL import Image
data = Path(path).read_bytes()
with Image.open(io.BytesIO(data)) as img:
width, height = img.size
if len(data) <= limit:
return data, mime_type, width, height
img = img.convert("RGB")
for max_side, quality in ((2000, 85), (2000, 75), (1600, 70), (1200, 65), (1000, 60)):
copy = img.copy()
copy.thumbnail((max_side, max_side))
buf = io.BytesIO()
copy.save(buf, format="JPEG", quality=quality, optimize=True)
if buf.tell() <= limit:
return buf.getvalue(), "image/jpeg", copy.size[0], copy.size[1]
raise SyndicationError(f"{Path(path).name}: image too large for Bluesky", permanent=True)
@dataclass
class Target:
id: str # stored in syndications.target
uid: str # Micropub syndicate-to uid
name: str # shown in the editor and Micropub clients
client: object # MastodonClient or BlueskyClient
max_chars: int = 5000
max_media: int = 4
@dataclass
class Post:
"""What a client needs to publish a note"""
note: object
permalink: str
tags: List[str]
max_chars: int
media: List[dict] = field(default_factory=list)
_client_cache: Dict[str, object] = {}
def targets_configured(config) -> bool:
"""Whether any crossposting target has credentials (used to start the worker)"""
return bool(
(config.get("GOTOSOCIAL_URL") and config.get("GOTOSOCIAL_ACCESS_TOKEN"))
or (config.get("BLUESKY_HANDLE") and config.get("BLUESKY_APP_PASSWORD"))
)
def get_targets() -> Dict[str, Target]:
"""Configured syndication targets, by id"""
config = current_app.config
targets: Dict[str, Target] = {}
url = (config.get("GOTOSOCIAL_URL") or "").rstrip("/")
token = config.get("GOTOSOCIAL_ACCESS_TOKEN") or ""
if url and token:
client = _client_cache.get(url)
if client is None:
client = MastodonClient(url, token, config.get("GOTOSOCIAL_VISIBILITY", "public"))
_client_cache[url] = client
targets["gotosocial"] = Target(
id="gotosocial",
uid=url + "/",
name=config.get("GOTOSOCIAL_NAME", "GoToSocial"),
client=client,
max_chars=int(config.get("GOTOSOCIAL_MAX_CHARS", 5000)),
max_media=int(config.get("GOTOSOCIAL_MAX_MEDIA", 4)),
)
handle = (config.get("BLUESKY_HANDLE") or "").lstrip("@")
password = config.get("BLUESKY_APP_PASSWORD") or ""
if handle and password:
service = (config.get("BLUESKY_SERVICE") or "https://bsky.social").rstrip("/")
key = f"bluesky:{service}:{handle}"
client = _client_cache.get(key)
if client is None:
client = BlueskyClient(service, handle, password)
_client_cache[key] = client
targets["bluesky"] = Target(
id="bluesky",
uid=f"https://bsky.app/profile/{handle}",
name=config.get("BLUESKY_NAME", "Bluesky"),
client=client,
max_chars=int(config.get("BLUESKY_MAX_CHARS", 300)),
max_media=4,
)
return targets
def resolve_targets(requested: List[str]) -> List[str]:
"""Map Micropub uids or target ids to configured target ids"""
targets = get_targets()
by_uid = {t.uid.rstrip("/"): t.id for t in targets.values()}
chosen = []
for value in requested:
value = (value or "").strip()
target_id = value if value in targets else by_uid.get(value.rstrip("/"))
if target_id and target_id not in chosen:
chosen.append(target_id)
return chosen
# --- Queue -----------------------------------------------------------------
def select_targets(note_id: int, target_ids: List[str]) -> None:
"""
Set which targets a note goes to, and queue the reconcile
Targets not listed are disabled (their remote post, if any, is deleted).
Nothing is recorded for unlisted targets the note never went to.
"""
db = get_db(current_app)
for target_id in get_targets():
enabled = 1 if target_id in target_ids else 0
existing = db.execute(
"SELECT id FROM syndications WHERE note_id = ? AND target = ?",
(note_id, target_id),
).fetchone()
if existing:
db.execute(
"""
UPDATE syndications
SET enabled = ?, state = 'pending', version = version + 1, attempts = 0,
last_error = NULL, next_attempt_at = datetime('now'), updated_at = datetime('now')
WHERE id = ?
""",
(enabled, existing["id"]),
)
elif enabled:
db.execute(
"INSERT INTO syndications (note_id, target, enabled) VALUES (?, ?, 1)",
(note_id, target_id),
)
db.commit()
def note_changed(note_id: int) -> None:
"""Queue a reconcile for every target the note was sent to"""
db = get_db(current_app)
db.execute(
"""
UPDATE syndications
SET state = 'pending', version = version + 1, attempts = 0, last_error = NULL,
next_attempt_at = datetime('now'), updated_at = datetime('now')
WHERE note_id = ?
""",
(note_id,),
)
db.commit()
def retry(syndication_id: int) -> None:
"""Retry a failed or pending row now"""
db = get_db(current_app)
db.execute(
"""
UPDATE syndications
SET state = 'pending', attempts = 0, last_error = NULL, next_attempt_at = datetime('now')
WHERE id = ?
""",
(syndication_id,),
)
db.commit()
def get_note_syndications(note_id: int) -> List[dict]:
"""Rows for a note, with the target's display name"""
targets = get_targets()
rows = get_db(current_app).execute(
"SELECT * FROM syndications WHERE note_id = ? ORDER BY target", (note_id,)
).fetchall()
result = []
for row in rows:
item = dict(row)
item["name"] = targets[row["target"]].name if row["target"] in targets else row["target"]
result.append(item)
return result
# --- Composing --------------------------------------------------------------
def _hashtag(display_name: str) -> str:
word = re.sub(r"[^\w]", "", display_name)
return f"#{word}" if word else ""
def compose_text(note, permalink: str, tags: List[str], max_chars: int,
body: Optional[str] = None) -> str:
"""
Full note (Markdown unless `body` is given), then the permalink and
hashtags; shortened at a word boundary if too long. On short targets the
hashtags may use at most a third of the space.
"""
hashtags = ""
for h in (_hashtag(t) for t in tags):
if h and len(hashtags) + len(h) + 1 <= max(max_chars // 3, 1):
hashtags = f"{hashtags} {h}".strip()
tail = f"\n\n{permalink}" + (f"\n\n{hashtags}" if hashtags else "")
body = (note.content if body is None else body).strip()
if len(body) + len(tail) > max_chars:
room = max_chars - len(tail) - 1
body = body[:room].rsplit(" ", 1)[0].rstrip() + "…"
return body + tail
_BLOCKS = ["p", "h1", "h2", "h3", "h4", "h5", "h6", "pre", "blockquote", "ul", "ol", "hr", "table"]
def plain_text(markdown_text: str) -> str:
"""Markdown as readable plain text (for targets without Markdown)"""
import markdown
from bs4 import BeautifulSoup
soup = BeautifulSoup(markdown.markdown(markdown_text, extensions=["fenced_code"]), "html.parser")
for img in soup.find_all("img"):
img.decompose()
for node in soup.find_all(string=True):
if not node.strip() and "\n" in node and node.parent.name != "pre":
node.extract() # layout newlines between tags; we add our own
for br in soup.find_all("br"):
br.replace_with("\n")
for li in soup.find_all("li"):
li.insert(0, "• ")
li.append("\n")
for block in soup.find_all(_BLOCKS):
block.append("\n\n")
text = soup.get_text()
text = re.sub(r"[ \t]+\n", "\n", text)
return re.sub(r"\n{3,}", "\n\n", text).strip()
_URL_RE = re.compile(r"https?://[^\s<>\"]+")
_TAG_RE = re.compile(r"(?<![\w#])#(\w*[^\W\d]\w*)")
def bluesky_facets(text: str) -> List[dict]:
"""Link and hashtag facets (UTF-8 byte ranges) for a Bluesky post"""
def span(start: int, end: int) -> dict:
return {"byteStart": len(text[:start].encode()), "byteEnd": len(text[:end].encode())}
facets = []
for m in _URL_RE.finditer(text):
url = m.group(0).rstrip(".,;:!?)\"'…")
facets.append({"index": span(m.start(), m.start() + len(url)),
"features": [{"$type": "app.bsky.richtext.facet#link", "uri": url}]})
urls = [(f["index"]["byteStart"], f["index"]["byteEnd"]) for f in facets]
for m in _TAG_RE.finditer(text):
index = span(m.start(), m.end())
if any(a <= index["byteStart"] < b for a, b in urls):
continue # a #fragment inside a URL
facets.append({"index": index,
"features": [{"$type": "app.bsky.richtext.facet#tag", "tag": m.group(1)}]})
return facets
def _media_files(note_id: int, limit: int) -> List[dict]:
"""Image files for a note: the 'large' variant when there is one"""
from starpunk.media import get_note_media
media_dir = Path(current_app.config["DATA_PATH"]) / "media"
files = []
for media in get_note_media(note_id)[:limit]:
variant = (media.get("variants") or {}).get("large")
rel = variant["path"] if variant else media["path"]
files.append({
"path": media_dir / rel,
"mime_type": media.get("mime_type") or "image/jpeg",
"description": media.get("caption") or "",
})
return files
# --- Worker -----------------------------------------------------------------
def _claim(db, row_id: int) -> bool:
cursor = db.execute(
f"""
UPDATE syndications SET claimed_until = datetime('now', '+{CLAIM_MINUTES} minutes')
WHERE id = ? AND state = 'pending'
AND (claimed_until IS NULL OR claimed_until < datetime('now'))
""",
(row_id,),
)
db.commit()
return cursor.rowcount == 1
def _finish(db, row, remote_id: Optional[str], remote_url: Optional[str]) -> None:
# Only mark synced if nothing changed while we were sending
db.execute(
"""
UPDATE syndications
SET remote_id = ?, remote_url = ?, claimed_until = NULL, attempts = 0, last_error = NULL,
state = CASE WHEN version = ? THEN 'synced' ELSE 'pending' END,
updated_at = datetime('now')
WHERE id = ?
""",
(remote_id, remote_url, row["version"], row["id"]),
)
db.commit()
def _fail(db, row, error: SyndicationError) -> None:
attempts = row["attempts"] + 1
failed = error.permanent or attempts >= MAX_ATTEMPTS
delay = min(30 * (2 ** attempts), 6 * 3600)
db.execute(
f"""
UPDATE syndications
SET attempts = ?, last_error = ?, claimed_until = NULL,
state = CASE WHEN version = ? THEN ? ELSE 'pending' END,
next_attempt_at = datetime('now', '+{delay} seconds'), updated_at = datetime('now')
WHERE id = ?
""",
(attempts, str(error)[:500], row["version"], "failed" if failed else "pending", row["id"]),
)
db.commit()
current_app.logger.warning(
f"Syndication {row['target']} for note {row['note_id']} failed "
f"(attempt {attempts}{', giving up' if failed else ''}): {error}"
)
def _reconcile(row, target: Target) -> tuple:
"""Bring the remote post in line with the note. Returns (remote_id, remote_url)."""
from starpunk.notes import get_note
from starpunk.tags import get_note_tags
note = get_note(id=row["note_id"], load_content=True)
wanted = bool(row["enabled"]) and note is not None and note.published
if not wanted:
if row["remote_id"]:
target.client.delete(row["remote_id"])
return None, None
if row["remote_id"] and not target.client.can_edit:
return row["remote_id"], row["remote_url"] # posted already; edits don't follow
site = current_app.config["SITE_URL"].rstrip("/")
post = Post(
note=note,
permalink=f"{site}/note/{note.slug}",
tags=[t["display_name"] for t in get_note_tags(note.id)],
max_chars=target.max_chars,
media=_media_files(note.id, target.max_media),
)
if row["remote_id"]:
try:
result = target.client.update(row["remote_id"], post)
return result["id"], result["url"]
except RemoteGone:
pass # deleted on the remote side; post it again
result = target.client.create(
post, key=f"starpunk-{row['id']}-{row['version']}", retrying=row["attempts"] > 0
)
return result["id"], result["url"]
def process_due(limit: int = 5) -> int:
"""Reconcile up to `limit` pending rows. Returns how many were handled."""
targets = get_targets()
if not targets:
return 0
db = get_db(current_app)
rows = db.execute(
"""
SELECT * FROM syndications
WHERE state = 'pending' AND next_attempt_at <= datetime('now')
AND (claimed_until IS NULL OR claimed_until < datetime('now'))
ORDER BY next_attempt_at LIMIT ?
""",
(limit,),
).fetchall()
handled = 0
for row in rows:
if not _claim(db, row["id"]):
continue # another worker took it
handled += 1
target = targets.get(row["target"])
if target is None:
_fail(db, row, SyndicationError(f"target {row['target']} is not configured", permanent=True))
continue
try:
remote_id, remote_url = _reconcile(row, target)
except SyndicationError as e:
_fail(db, row, e)
continue
except Exception as e: # never let one bad note stop the worker
_fail(db, row, SyndicationError(f"unexpected error: {e}"))
continue
_finish(db, row, remote_id, remote_url)
if remote_id:
current_app.logger.info(f"Syndicated note {row['note_id']} to {row['target']}: {remote_url}")
return handled
class SyndicationWorker(threading.Thread):
"""Daemon thread that drains the syndication queue"""
def __init__(self, app, poll_interval: int = 10):
super().__init__(daemon=True, name="syndication-worker")
self.app = app
self.poll_interval = poll_interval
self._stop_event = threading.Event()
def run(self):
while not self._stop_event.is_set():
try:
with self.app.app_context():
handled = process_due()
except Exception as e:
self.app.logger.error(f"Syndication worker error: {e}", exc_info=True)
handled = 0
if handled == 0:
self._stop_event.wait(self.poll_interval)
def stop(self):
self._stop_event.set()
+47 -4
View File
@@ -52,15 +52,58 @@
<label for="published">Published</label>
</div>
{% if syndication_targets %}
<input type="hidden" name="syndication_form" value="1">
<fieldset class="form-group">
<legend>Crosspost</legend>
{% for target in syndication_targets %}
<div class="form-checkbox">
<input type="checkbox" id="syndicate-{{ target.id }}" name="syndicate_to" value="{{ target.id }}"{% if target.id in syndication_enabled %} checked{% endif %}>
<label for="syndicate-{{ target.id }}">{{ target.name }}</label>
</div>
{% endfor %}
</fieldset>
{% endif %}
<div class="form-actions">
<button type="submit" class="button button-primary">Update Note</button>
<a href="{{ url_for('admin.dashboard') }}" class="button button-secondary">Cancel</a>
<form action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" style="display: inline;" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');">
<input type="hidden" name="confirm" value="yes">
<button type="submit" class="button button-danger">Delete Note</button>
</form>
<button type="submit" form="delete-note-form" class="button button-danger">Delete Note</button>
</div>
</form>
{# Outside the edit form: nested forms are not allowed in HTML #}
<form id="delete-note-form" action="{{ url_for('admin.delete_note_submit', note_id=note.id) }}" method="POST" class="delete-form" onsubmit="return confirm('Are you sure you want to delete this note? This cannot be undone.');">
<input type="hidden" name="confirm" value="yes">
</form>
{% if syndications %}
<section class="syndication-status">
<h3>Crossposts</h3>
<ul>
{% for s in syndications %}
<li>
<strong>{{ s.name }}</strong>:
{% if s.state == 'synced' and s.remote_url %}
<a href="{{ s.remote_url }}" rel="noopener">posted</a>
{% elif s.state == 'synced' %}
removed
{% elif s.state == 'pending' %}
pending{% if s.attempts %} (retry {{ s.attempts }}){% endif %}
{% else %}
failed
{% endif %}
{% if s.last_error %}<br><small>{{ s.last_error }}</small>{% endif %}
{% if s.state != 'synced' %}
<form action="{{ url_for('admin.retry_syndication', syndication_id=s.id) }}" method="POST" style="display: inline;">
<button type="submit" class="button button-secondary">Retry now</button>
</form>
{% endif %}
</li>
{% endfor %}
</ul>
</section>
{% endif %}
</div>
{% endblock %}
+13
View File
@@ -65,6 +65,19 @@
<label for="published">Publish immediately</label>
</div>
{% if syndication_targets %}
<input type="hidden" name="syndication_form" value="1">
<fieldset class="form-group">
<legend>Crosspost</legend>
{% for target in syndication_targets %}
<div class="form-checkbox">
<input type="checkbox" id="syndicate-{{ target.id }}" name="syndicate_to" value="{{ target.id }}" checked>
<label for="syndicate-{{ target.id }}">{{ target.name }}</label>
</div>
{% endfor %}
</fieldset>
{% endif %}
<div class="form-actions">
<button type="submit" class="button button-primary">Create Note</button>
<a href="{{ url_for('admin.dashboard') }}" class="button button-secondary">Cancel</a>
+3
View File
@@ -49,6 +49,9 @@
<footer class="note-meta">
{# u-url and u-uid same for notes (per Q23) #}
{% for s in syndications if s.remote_url %}
<a class="u-syndication" href="{{ s.remote_url }}" rel="syndication">Also on {{ s.name }}</a>
{% endfor %}
<a class="u-url u-uid" href="{{ absolute_url_for('public.note', slug=note.slug) }}">
<time class="dt-published" datetime="{{ note.created_at.isoformat() }}">
{{ note.created_at.strftime('%B %d, %Y at %I:%M %p') }}
+2 -2
View File
@@ -206,7 +206,7 @@ def test_micropub_create_with_custom_slug_form(client, app, mock_valid_token):
# Verify the custom slug was used
location = response.headers['Location']
assert location.endswith('/notes/my-custom-slug')
assert location.endswith('/note/my-custom-slug')
# Verify note exists with the custom slug
with app.app_context():
@@ -236,7 +236,7 @@ def test_micropub_create_with_custom_slug_json(client, app, mock_valid_token):
# Verify the custom slug was used
location = response.headers['Location']
assert location.endswith('/notes/json-custom-slug')
assert location.endswith('/note/json-custom-slug')
# Verify note exists with the custom slug
with app.app_context():
+2 -2
View File
@@ -220,8 +220,8 @@ def test_search_api_generates_correct_urls(client, test_notes):
if data["count"] > 0:
result = data["results"][0]
assert result["url"].startswith("/notes/")
assert result["url"] == f"/notes/{result['slug']}"
assert result["url"].startswith("/note/")
assert result["url"] == f"/note/{result['slug']}"
def test_search_api_provides_fallback_title(client, app):
+2 -2
View File
@@ -185,8 +185,8 @@ def test_search_page_links_to_notes(client, test_notes):
response = client.get("/search?q=test")
assert response.status_code == 200
# Should contain links to /notes/
assert b'href="/notes/' in response.data
# Should contain links to /note/
assert b'href="/note/' in response.data
def test_search_form_validation(client):
+349
View File
@@ -0,0 +1,349 @@
"""
Tests for crossposting to GoToSocial (starpunk/syndication.py, v2.1.0)
A fake Mastodon-API server (httpx.MockTransport) records what StarPunk sends.
"""
import io
import json
import httpx
import pytest
from PIL import Image
from starpunk import create_app, syndication
from starpunk.auth import create_session
from starpunk.indieauth import create_access_token
from starpunk.notes import create_note, delete_note, update_note
from starpunk.syndication import (
MastodonClient,
compose_text,
get_note_syndications,
note_changed,
process_due,
select_targets,
)
GTS = "https://social.example.com"
class FakeServer:
def __init__(self):
self.statuses = {}
self.calls = []
self.media = 0
self.fail_next = None # status code to return once
def __call__(self, request: httpx.Request) -> httpx.Response:
self.calls.append((request.method, request.url.path))
assert request.headers["Authorization"] == "Bearer secret-token"
if self.fail_next:
code, self.fail_next = self.fail_next, None
return httpx.Response(code, json={"error": "nope"})
path = request.url.path
if path == "/api/v2/media":
self.media += 1
return httpx.Response(200, json={"id": f"m{self.media}"})
if path == "/api/v1/statuses" and request.method == "POST":
body = json.loads(request.content)
sid = str(len(self.statuses) + 1)
self.statuses[sid] = body
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
if path.startswith("/api/v1/statuses/"):
sid = path.rsplit("/", 1)[1]
if sid not in self.statuses:
return httpx.Response(404, json={"error": "not found"})
if request.method == "PUT":
self.statuses[sid] = json.loads(request.content)
return httpx.Response(200, json={"id": sid, "url": f"{GTS}/@phil/statuses/{sid}"})
if request.method == "DELETE":
del self.statuses[sid]
return httpx.Response(200, json={"id": sid})
return httpx.Response(500)
@pytest.fixture
def server():
return FakeServer()
@pytest.fixture
def app(tmp_path, server):
data = tmp_path / "data"
app = create_app(
{
"TESTING": True,
"SITE_URL": "https://blog.example.com",
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"GOTOSOCIAL_URL": GTS,
"GOTOSOCIAL_ACCESS_TOKEN": "secret-token",
}
)
syndication._client_cache[GTS] = MastodonClient(
GTS, "secret-token", http=httpx.Client(transport=httpx.MockTransport(server))
)
yield app
syndication._client_cache.clear()
def rows(app, note_id):
with app.app_context():
return get_note_syndications(note_id)
def publish(app, content="Hello fediverse", tags=None, targets=("gotosocial",)):
with app.test_request_context():
note = create_note(content=content, published=True, tags=tags)
select_targets(note.id, list(targets))
process_due()
return note
class TestCompose:
def test_full_note_with_link_and_hashtags(self, app):
with app.app_context():
note = create_note(content="# Title\n\nSome *markdown*", published=True)
text = compose_text(note, "https://blog.example.com/note/x", ["IndieWeb", "Self Hosting"], 5000)
assert text == "# Title\n\nSome *markdown*\n\nhttps://blog.example.com/note/x\n\n#IndieWeb #SelfHosting"
def test_long_note_is_shortened_but_keeps_link(self, app):
with app.app_context():
note = create_note(content="word " * 2000, published=True)
text = compose_text(note, "https://blog.example.com/note/x", [], 500)
assert len(text) <= 500
assert text.endswith("https://blog.example.com/note/x")
assert "…" in text
class TestLifecycle:
def test_publish_creates_status(self, app, server):
note = publish(app, tags=["Test"])
(row,) = rows(app, note.id)
assert row["state"] == "synced"
assert row["remote_url"] == f"{GTS}/@phil/statuses/1"
body = server.statuses["1"]
assert body["content_type"] == "text/markdown"
assert body["visibility"] == "public"
assert f"https://blog.example.com/note/{note.slug}" in body["status"]
assert "#Test" in body["status"]
def test_edit_updates_status(self, app, server):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, content="Edited text")
process_due()
assert server.statuses["1"]["status"].startswith("Edited text")
assert ("PUT", "/api/v1/statuses/1") in server.calls
def test_delete_deletes_status(self, app, server):
note = publish(app)
with app.test_request_context():
delete_note(id=note.id)
process_due()
assert server.statuses == {}
(row,) = rows(app, note.id)
assert row["remote_id"] is None and row["state"] == "synced"
def test_hard_delete_deletes_status(self, app, server):
note = publish(app)
with app.test_request_context():
delete_note(id=note.id, soft=False)
process_due()
assert server.statuses == {}
def test_unpublish_then_republish(self, app, server):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, published=False)
process_due()
assert server.statuses == {}
update_note(id=note.id, published=True)
process_due()
assert len(server.statuses) == 1
def test_opt_out_sends_nothing(self, app, server):
note = publish(app, targets=())
assert server.calls == []
assert rows(app, note.id) == []
def test_unticking_later_deletes_status(self, app, server):
note = publish(app)
with app.test_request_context():
select_targets(note.id, [])
process_due()
assert server.statuses == {}
def test_drafts_are_not_sent(self, app, server):
with app.test_request_context():
note = create_note(content="draft", published=False)
select_targets(note.id, ["gotosocial"])
process_due()
assert server.statuses == {}
def test_remote_deleted_by_hand_is_reposted_on_edit(self, app, server):
note = publish(app)
server.statuses.clear()
with app.test_request_context():
note_changed(note.id)
process_due()
assert len(server.statuses) == 1
def test_edit_during_send_is_not_lost(self, app, server):
with app.test_request_context():
note = create_note(content="v1", published=True)
select_targets(note.id, ["gotosocial"])
# Simulate an edit arriving while the first send is in flight
original = syndication._reconcile
def edit_mid_send(row, target):
result = original(row, target)
update_note(id=note.id, content="v2")
return result
syndication._reconcile = edit_mid_send
try:
process_due()
finally:
syndication._reconcile = original
assert rows(app, note.id)[0]["state"] == "pending"
process_due()
assert server.statuses["1"]["status"].startswith("v2")
class TestFailures:
def test_server_error_retries_later(self, app, server):
server.fail_next = 503
note = publish(app)
(row,) = rows(app, note.id)
assert row["state"] == "pending"
assert row["attempts"] == 1
assert "503" in row["last_error"]
def test_client_error_fails_permanently(self, app, server):
server.fail_next = 422
note = publish(app)
(row,) = rows(app, note.id)
assert row["state"] == "failed"
def test_retry_button(self, app, server):
server.fail_next = 422
note = publish(app)
(row,) = rows(app, note.id)
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://example.com"))
client.post(f"/admin/syndication/{row['id']}/retry")
with app.test_request_context():
process_due()
assert rows(app, note.id)[0]["state"] == "synced"
def test_no_targets_configured_does_nothing(self, tmp_path):
data = tmp_path / "d"
app = create_app({
"TESTING": True, "SITE_URL": "https://b.example", "ADMIN_ME": "https://e.example",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "s.db",
})
with app.test_request_context():
note = create_note(content="x", published=True)
select_targets(note.id, ["gotosocial"])
assert process_due() == 0
assert get_note_syndications(note.id) == []
class TestMedia:
def test_images_are_uploaded_with_alt_text(self, app, server):
from starpunk.media import attach_media_to_note, save_media
buf = io.BytesIO()
Image.new("RGB", (800, 600), "red").save(buf, format="JPEG")
with app.test_request_context():
note = create_note(content="With a picture", published=True)
media = save_media(buf.getvalue(), "red.jpg")
attach_media_to_note(note.id, [media["id"]], ["A red square"])
select_targets(note.id, ["gotosocial"])
process_due()
assert ("POST", "/api/v2/media") in server.calls
assert server.statuses["1"]["media_ids"] == ["m1"]
class TestEntryPoints:
@pytest.fixture
def admin(self, app):
client = app.test_client()
with app.test_request_context():
client.set_cookie("starpunk_session", create_session("https://example.com"))
return client
def test_new_note_form_ticks_target_by_default(self, admin):
html = admin.get("/admin/new").data.decode()
assert 'name="syndicate_to" value="gotosocial" checked' in html
def test_admin_create_queues(self, app, admin):
admin.post("/admin/new", data={
"content": "From the editor", "published": "on",
"syndication_form": "1", "syndicate_to": "gotosocial",
})
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 1
def test_admin_create_opt_out(self, app, admin):
admin.post("/admin/new", data={"content": "Not this one", "published": "on", "syndication_form": "1"})
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 0
def _token(self, app):
with app.test_request_context():
return create_access_token("https://quill.example/", "create")["access_token"]
def test_micropub_lists_target(self, app):
token = self._token(app)
data = app.test_client().get(
"/micropub?q=syndicate-to", headers={"Authorization": f"Bearer {token}"}
).get_json()
assert data["syndicate-to"] == [{"uid": f"{GTS}/", "name": "GoToSocial"}]
def test_micropub_default_is_all_targets(self, app):
token = self._token(app)
response = app.test_client().post(
"/micropub", data={"h": "entry", "content": "via quill"},
headers={"Authorization": f"Bearer {token}"},
)
assert response.status_code == 201
assert "/note/" in response.headers["Location"]
assert response.headers["Location"].startswith("https://blog.example.com/note/")
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 1
def test_micropub_explicit_empty_choice(self, app):
token = self._token(app)
app.test_client().post(
"/micropub", json={"type": ["h-entry"], "properties": {"content": ["no crosspost"]},
"mp-syndicate-to": []},
headers={"Authorization": f"Bearer {token}"},
)
with app.app_context():
from starpunk.database import get_db
assert get_db(app).execute("SELECT COUNT(*) FROM syndications").fetchone()[0] == 0
def test_note_page_links_to_crosspost(self, app, server):
note = publish(app)
html = app.test_client().get(f"/note/{note.slug}").data.decode()
assert f'class="u-syndication" href="{GTS}/@phil/statuses/1"' in html
def test_edit_page_shows_status_and_separate_delete_form(self, app, admin, server):
note = publish(app)
html = admin.get(f"/admin/edit/{note.id}").data.decode()
assert 'value="gotosocial" checked' in html
assert f"{GTS}/@phil/statuses/1" in html
# Delete button belongs to its own form, not the edit form
assert 'form="delete-note-form"' in html
edit_form = html.split('class="note-form"', 1)[1].split("</form>", 1)[0]
assert "<form" not in edit_form
+274
View File
@@ -0,0 +1,274 @@
"""
Tests for crossposting to Bluesky (starpunk/syndication.py, v2.2.0)
A fake PDS (httpx.MockTransport) records what StarPunk sends.
"""
import io
import json
import httpx
import pytest
from PIL import Image
from starpunk import create_app, syndication
from starpunk.notes import create_note, delete_note, update_note
from starpunk.syndication import (
BlueskyClient,
bluesky_facets,
get_note_syndications,
plain_text,
process_due,
select_targets,
)
SERVICE = "https://bsky.example"
PDS = "https://pds.example"
DID = "did:plc:abc123"
class FakePDS:
def __init__(self):
self.records = {}
self.blobs = []
self.calls = []
self.logins = 0
self.expire_next = False
self.lose_next_reply = False
self.fail_next = None
def _json(self, code, body):
return httpx.Response(code, json=body)
def __call__(self, request: httpx.Request) -> httpx.Response:
nsid = request.url.path.removeprefix("/xrpc/")
self.calls.append((request.url.host, nsid))
if nsid == "com.atproto.server.createSession":
body = json.loads(request.content)
if body["password"] != "app-pass":
return self._json(401, {"error": "AuthenticationRequired"})
self.logins += 1
return self._json(200, {
"did": DID, "handle": "phil.example", "accessJwt": f"access{self.logins}",
"refreshJwt": "refresh",
"didDoc": {"service": [{"id": "#atproto_pds", "serviceEndpoint": PDS}]},
})
if nsid == "com.atproto.server.refreshSession":
self.logins += 1
return self._json(200, {"did": DID, "handle": "phil.example",
"accessJwt": f"access{self.logins}", "refreshJwt": "refresh"})
assert request.url.host == "pds.example"
if request.headers.get("Authorization") != f"Bearer access{self.logins}" or self.expire_next:
self.expire_next = False
return self._json(400, {"error": "ExpiredToken"})
if self.fail_next:
code, self.fail_next = self.fail_next, None
return self._json(code, {"error": "Nope"})
if nsid == "com.atproto.repo.uploadBlob":
self.blobs.append(len(request.content))
assert len(request.content) <= 1_000_000
return self._json(200, {"blob": {"$type": "blob", "ref": {"$link": f"b{len(self.blobs)}"},
"mimeType": request.headers["Content-Type"],
"size": len(request.content)}})
if nsid == "com.atproto.repo.createRecord":
body = json.loads(request.content)
uri = f"at://{DID}/app.bsky.feed.post/r{len(self.records) + 1}"
self.records[uri] = body["record"]
if self.lose_next_reply:
self.lose_next_reply = False
raise httpx.ReadTimeout("lost reply")
return self._json(200, {"uri": uri, "cid": "c"})
if nsid == "com.atproto.repo.listRecords":
return self._json(200, {"records": [{"uri": u, "value": v} for u, v in self.records.items()]})
if nsid == "com.atproto.repo.deleteRecord":
body = json.loads(request.content)
self.records.pop(f"at://{body['repo']}/{body['collection']}/{body['rkey']}", None)
return self._json(200, {})
return self._json(500, {"error": "unknown"})
@pytest.fixture
def pds():
return FakePDS()
@pytest.fixture
def app(tmp_path, pds):
data = tmp_path / "data"
app = create_app({
"TESTING": True,
"SITE_URL": "https://blog.example.com",
"ADMIN_ME": "https://example.com",
"SESSION_SECRET": "test-secret",
"DATA_PATH": data,
"NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "starpunk.db",
"BLUESKY_HANDLE": "phil.example",
"BLUESKY_APP_PASSWORD": "app-pass",
"BLUESKY_SERVICE": SERVICE,
})
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"] = BlueskyClient(
SERVICE, "phil.example", "app-pass", http=httpx.Client(transport=httpx.MockTransport(pds))
)
yield app
syndication._client_cache.clear()
def publish(app, content="Hello Bluesky", tags=None):
with app.test_request_context():
note = create_note(content=content, published=True, tags=tags)
select_targets(note.id, ["bluesky"])
process_due()
return note
def only_record(pds):
(record,) = pds.records.values()
return record
def row(app, note):
with app.app_context():
(r,) = get_note_syndications(note.id)
return r
class TestText:
def test_markdown_becomes_plain_text(self):
md = "# Title\n\nSome *emphasis* and a [link](https://x.example).\n\n- one\n- two\n\n![pic](/m.jpg)"
assert plain_text(md) == "Title\n\nSome emphasis and a link.\n\n• one\n• two"
def test_facets_use_utf8_byte_offsets(self):
text = "Café ☕ https://blog.example.com/note/x #IndieWeb"
link, tag = bluesky_facets(text)
raw = text.encode()
assert raw[link["index"]["byteStart"]:link["index"]["byteEnd"]] == b"https://blog.example.com/note/x"
assert raw[tag["index"]["byteStart"]:tag["index"]["byteEnd"]] == b"#IndieWeb"
assert tag["features"][0]["tag"] == "IndieWeb"
def test_fragment_in_url_is_not_a_tag(self):
facets = bluesky_facets("see https://x.example/page#section and #42 too")
assert [f["features"][0]["$type"] for f in facets] == ["app.bsky.richtext.facet#link"]
class TestLifecycle:
def test_publish_posts_with_link_card(self, app, pds):
note = publish(app, content="Hello **Bluesky**", tags=["IndieWeb"])
record = only_record(pds)
permalink = f"https://blog.example.com/note/{note.slug}"
assert record["text"] == f"Hello Bluesky\n\n{permalink}\n\n#IndieWeb"
assert record["embed"]["external"]["uri"] == permalink
kinds = {f["features"][0]["$type"] for f in record["facets"]}
assert kinds == {"app.bsky.richtext.facet#link", "app.bsky.richtext.facet#tag"}
r = row(app, note)
assert r["state"] == "synced"
assert r["remote_id"] == f"at://{DID}/app.bsky.feed.post/r1"
assert r["remote_url"] == "https://bsky.app/profile/phil.example/post/r1"
def test_long_note_fits_300(self, app, pds):
note = publish(app, content="word " * 500)
text = only_record(pds)["text"]
assert len(text) <= 300
assert text.endswith(f"https://blog.example.com/note/{note.slug}")
def test_edit_leaves_post_alone(self, app, pds):
note = publish(app, content="first")
with app.test_request_context():
update_note(id=note.id, content="second")
process_due()
assert only_record(pds)["text"].startswith("first")
assert row(app, note)["state"] == "synced"
def test_delete_and_unpublish_remove_post(self, app, pds):
note = publish(app)
with app.test_request_context():
update_note(id=note.id, published=False)
process_due()
assert pds.records == {}
update_note(id=note.id, published=True)
process_due()
assert len(pds.records) == 1
delete_note(id=note.id)
process_due()
assert pds.records == {}
class TestSession:
def test_logs_in_once_and_uses_pds(self, app, pds):
publish(app, content="one")
publish(app, content="two")
assert pds.logins == 1
assert ("bsky.example", "com.atproto.server.createSession") in pds.calls
assert ("pds.example", "com.atproto.repo.createRecord") in pds.calls
def test_expired_token_is_refreshed(self, app, pds):
publish(app, content="one")
pds.expire_next = True
publish(app, content="two")
assert len(pds.records) == 2
assert ("bsky.example", "com.atproto.server.refreshSession") in pds.calls
def test_bad_password_fails_permanently(self, app, pds):
syndication._client_cache[f"bluesky:{SERVICE}:phil.example"].password = "wrong"
note = publish(app)
r = row(app, note)
assert r["state"] == "failed" and "401" in r["last_error"]
def test_lost_reply_does_not_double_post(self, app, pds):
pds.lose_next_reply = True
note = publish(app)
assert row(app, note)["state"] == "pending"
with app.test_request_context():
from starpunk.syndication import retry
retry(row(app, note)["id"])
# retry() resets attempts; simulate the scheduled retry instead
from starpunk.database import get_db
get_db(app).execute("UPDATE syndications SET attempts = 1")
get_db(app).commit()
process_due()
assert len(pds.records) == 1
assert row(app, note)["state"] == "synced"
def test_server_error_retries(self, app, pds):
publish(app, content="warm up") # log in first
pds.fail_next = 502
note = publish(app, content="later")
assert row(app, note)["state"] == "pending"
class TestMedia:
def test_large_image_is_shrunk_with_alt_and_aspect_ratio(self, app, pds):
from starpunk.media import attach_media_to_note, save_media
import os
img = Image.frombytes("RGB", (2400, 1800), os.urandom(2400 * 1800 * 3))
buf = io.BytesIO()
img.save(buf, format="PNG")
assert buf.tell() > 1_000_000
with app.test_request_context():
note = create_note(content="noisy", published=True)
media = save_media(buf.getvalue(), "noise.png")
attach_media_to_note(note.id, [media["id"]], ["Static"])
select_targets(note.id, ["bluesky"])
process_due()
record = only_record(pds)
(image,) = record["embed"]["images"]
assert record["embed"]["$type"] == "app.bsky.embed.images"
assert image["alt"] == "Static"
assert image["aspectRatio"]["width"] > 0
def test_both_targets_listed_for_micropub(tmp_path):
data = tmp_path / "d"
app = create_app({
"TESTING": True, "SITE_URL": "https://b.example", "ADMIN_ME": "https://e.example",
"SESSION_SECRET": "x", "DATA_PATH": data, "NOTES_PATH": data / "notes",
"DATABASE_PATH": data / "s.db",
"GOTOSOCIAL_URL": "https://social.example", "GOTOSOCIAL_ACCESS_TOKEN": "t",
"BLUESKY_HANDLE": "phil.example", "BLUESKY_APP_PASSWORD": "p",
})
with app.app_context():
from starpunk.syndication import get_targets, resolve_targets
assert set(get_targets()) == {"gotosocial", "bluesky"}
assert resolve_targets(["https://bsky.app/profile/phil.example"]) == ["bluesky"]
syndication._client_cache.clear()