StarPunk/docs/migration/fix-hardcoded-endpoints.md

Migration Guide: Fixing Hardcoded IndieAuth Endpoints

Overview

This guide explains how to migrate from the incorrect hardcoded endpoint implementation to the correct dynamic endpoint discovery implementation that actually follows the IndieAuth specification.

The Problem We're Fixing

What's Currently Wrong

# WRONG - auth_external.py (hypothetical incorrect implementation)
class ExternalTokenVerifier:
    def __init__(self):
        # FATAL FLAW: Hardcoded endpoint
        self.token_endpoint = "https://tokens.indieauth.com/token"

    def verify_token(self, token):
        # Uses hardcoded endpoint for ALL users
        response = requests.get(
            self.token_endpoint,
            headers={'Authorization': f'Bearer {token}'}
        )
        return response.json()

Why It's Wrong

1. Not IndieAuth: This completely violates the IndieAuth specification
2. No User Choice: Forces all users to use the same provider
3. Security Risk: Single point of failure for all authentications
4. No Flexibility: Users can't change or choose providers

The Correct Implementation

Step 1: Remove Hardcoded Configuration

Remove from config files:

# DELETE THESE LINES - They are wrong!
TOKEN_ENDPOINT=https://tokens.indieauth.com/token
AUTHORIZATION_ENDPOINT=https://indieauth.com/auth

Keep only:

# CORRECT - Only the admin's identity URL
ADMIN_ME=https://admin.example.com/

Step 2: Implement Endpoint Discovery

Create endpoint_discovery.py:

"""
IndieAuth Endpoint Discovery
Implements: https://www.w3.org/TR/indieauth/#discovery-by-clients
"""

import re
from typing import Dict, Optional
from urllib.parse import urljoin, urlparse
import httpx
from bs4 import BeautifulSoup

class EndpointDiscovery:
    """Discovers IndieAuth endpoints from profile URLs"""

    def __init__(self, timeout: int = 5):
        self.timeout = timeout
        self.client = httpx.Client(
            timeout=timeout,
            follow_redirects=True,
            limits=httpx.Limits(max_redirects=5)
        )

    def discover(self, profile_url: str) -> Dict[str, str]:
        """
        Discover IndieAuth endpoints from a profile URL

        Args:
            profile_url: The user's profile URL (their identity)

        Returns:
            Dictionary with 'authorization_endpoint' and 'token_endpoint'

        Raises:
            DiscoveryError: If discovery fails
        """
        # Ensure HTTPS in production
        if not self._is_development() and not profile_url.startswith('https://'):
            raise DiscoveryError("Profile URL must use HTTPS")

        try:
            response = self.client.get(profile_url)
            response.raise_for_status()
        except Exception as e:
            raise DiscoveryError(f"Failed to fetch profile: {e}")

        endpoints = {}

        # 1. Check HTTP Link headers (highest priority)
        link_header = response.headers.get('Link', '')
        if link_header:
            endpoints.update(self._parse_link_header(link_header, profile_url))

        # 2. Check HTML link elements
        if 'text/html' in response.headers.get('Content-Type', ''):
            endpoints.update(self._extract_from_html(
                response.text,
                profile_url
            ))

        # Validate we found required endpoints
        if 'token_endpoint' not in endpoints:
            raise DiscoveryError("No token endpoint found in profile")

        return endpoints

    def _parse_link_header(self, header: str, base_url: str) -> Dict[str, str]:
        """Parse HTTP Link header for endpoints"""
        endpoints = {}

        # Parse Link: <url>; rel="relation"
        pattern = r'<([^>]+)>;\s*rel="([^"]+)"'
        matches = re.findall(pattern, header)

        for url, rel in matches:
            if rel == 'authorization_endpoint':
                endpoints['authorization_endpoint'] = urljoin(base_url, url)
            elif rel == 'token_endpoint':
                endpoints['token_endpoint'] = urljoin(base_url, url)

        return endpoints

    def _extract_from_html(self, html: str, base_url: str) -> Dict[str, str]:
        """Extract endpoints from HTML link elements"""
        endpoints = {}
        soup = BeautifulSoup(html, 'html.parser')

        # Find <link rel="authorization_endpoint" href="...">
        auth_link = soup.find('link', rel='authorization_endpoint')
        if auth_link and auth_link.get('href'):
            endpoints['authorization_endpoint'] = urljoin(
                base_url,
                auth_link['href']
            )

        # Find <link rel="token_endpoint" href="...">
        token_link = soup.find('link', rel='token_endpoint')
        if token_link and token_link.get('href'):
            endpoints['token_endpoint'] = urljoin(
                base_url,
                token_link['href']
            )

        return endpoints

    def _is_development(self) -> bool:
        """Check if running in development mode"""
        # Implementation depends on your config system
        return False


class DiscoveryError(Exception):
    """Raised when endpoint discovery fails"""
    pass

Step 3: Update Token Verification

Update auth_external.py:

"""
External Token Verification with Dynamic Discovery
"""

import hashlib
import time
from typing import Dict, Optional
import httpx

from .endpoint_discovery import EndpointDiscovery, DiscoveryError


class ExternalTokenVerifier:
    """Verifies tokens using discovered IndieAuth endpoints"""

    def __init__(self, admin_me: str, cache_ttl: int = 300):
        self.admin_me = admin_me
        self.discovery = EndpointDiscovery()
        self.cache = TokenCache(ttl=cache_ttl)

    def verify_token(self, token: str) -> Dict:
        """
        Verify a token using endpoint discovery

        Args:
            token: Bearer token to verify

        Returns:
            Token info dict with 'me', 'scope', 'client_id'

        Raises:
            TokenVerificationError: If verification fails
        """
        # Check cache first
        token_hash = self._hash_token(token)
        cached = self.cache.get(token_hash)
        if cached:
            return cached

        # Discover endpoints for admin
        try:
            endpoints = self.discovery.discover(self.admin_me)
        except DiscoveryError as e:
            raise TokenVerificationError(f"Endpoint discovery failed: {e}")

        # Verify with discovered endpoint
        token_endpoint = endpoints['token_endpoint']

        try:
            response = httpx.get(
                token_endpoint,
                headers={'Authorization': f'Bearer {token}'},
                timeout=5.0
            )
            response.raise_for_status()
        except Exception as e:
            raise TokenVerificationError(f"Token verification failed: {e}")

        token_info = response.json()

        # Validate response
        if 'me' not in token_info:
            raise TokenVerificationError("Invalid token response: missing 'me'")

        # Ensure token is for our admin
        if self._normalize_url(token_info['me']) != self._normalize_url(self.admin_me):
            raise TokenVerificationError(
                f"Token is for {token_info['me']}, expected {self.admin_me}"
            )

        # Check scope
        scopes = token_info.get('scope', '').split()
        if 'create' not in scopes:
            raise TokenVerificationError("Token missing 'create' scope")

        # Cache successful verification
        self.cache.store(token_hash, token_info)

        return token_info

    def _hash_token(self, token: str) -> str:
        """Hash token for secure caching"""
        return hashlib.sha256(token.encode()).hexdigest()

    def _normalize_url(self, url: str) -> str:
        """Normalize URL for comparison"""
        # Add trailing slash if missing
        if not url.endswith('/'):
            url += '/'
        return url.lower()


class TokenCache:
    """Simple in-memory cache for token verifications"""

    def __init__(self, ttl: int = 300):
        self.ttl = ttl
        self.cache = {}

    def get(self, token_hash: str) -> Optional[Dict]:
        """Get cached token info if still valid"""
        if token_hash in self.cache:
            info, expiry = self.cache[token_hash]
            if time.time() < expiry:
                return info
            else:
                del self.cache[token_hash]
        return None

    def store(self, token_hash: str, info: Dict):
        """Cache token info"""
        expiry = time.time() + self.ttl
        self.cache[token_hash] = (info, expiry)


class TokenVerificationError(Exception):
    """Raised when token verification fails"""
    pass

Step 4: Update Micropub Integration

Update Micropub to use discovery-based verification:

# micropub.py
from ..auth.auth_external import ExternalTokenVerifier

class MicropubEndpoint:
    def __init__(self, config):
        self.verifier = ExternalTokenVerifier(
            admin_me=config['ADMIN_ME'],
            cache_ttl=config.get('TOKEN_CACHE_TTL', 300)
        )

    def handle_request(self, request):
        # Extract token
        auth_header = request.headers.get('Authorization', '')
        if not auth_header.startswith('Bearer '):
            return error_response(401, "No bearer token provided")

        token = auth_header[7:]  # Remove 'Bearer ' prefix

        # Verify using discovery
        try:
            token_info = self.verifier.verify_token(token)
        except TokenVerificationError as e:
            return error_response(403, str(e))

        # Process Micropub request
        # ...

Migration Steps

Phase 1: Preparation

1. Review current implementation

   • Identify all hardcoded endpoint references
   • Document current configuration

2. Set up test environment

   • Create test profile with IndieAuth links
   • Set up test IndieAuth provider

3. Write tests for new implementation

   • Unit tests for discovery
   • Integration tests for verification

Phase 2: Implementation

1. Implement discovery module

   • Create endpoint_discovery.py
   • Add comprehensive error handling
   • Include logging for debugging

2. Update token verification

   • Remove hardcoded endpoints
   • Integrate discovery module
   • Add caching layer

3. Update configuration

   • Remove TOKEN_ENDPOINT from config
   • Ensure ADMIN_ME is set correctly

Phase 3: Testing

1. Test discovery with various providers

   • indieauth.com
   • Self-hosted IndieAuth
   • Custom implementations

2. Test error conditions

   • Profile URL unreachable
   • No endpoints in profile
   • Invalid token responses

3. Performance testing

   • Measure discovery latency
   • Verify cache effectiveness
   • Test under load

Phase 4: Deployment

1. Update documentation

   • Explain endpoint discovery
   • Provide setup instructions
   • Include troubleshooting guide

2. Deploy to staging

   • Test with real IndieAuth providers
   • Monitor for issues
   • Verify performance

3. Deploy to production

   • Clear any existing caches
   • Monitor closely for first 24 hours
   • Be ready to roll back if needed

Verification Checklist

After migration, verify:

• No hardcoded endpoints remain in code
• Discovery works with test profiles
• Token verification uses discovered endpoints
• Cache improves performance
• Error messages are clear
• Logs contain useful debugging info
• Documentation is updated
• Tests pass

Troubleshooting

Common Issues

"No token endpoint found"

Cause: Profile URL doesn't have IndieAuth links

Solution:

1. Check profile URL returns HTML
2. Verify link elements are present
3. Check for typos in rel attributes

"Token verification failed"

Cause: Various issues with endpoint or token

Solution:

1. Check endpoint is reachable
2. Verify token hasn't expired
3. Ensure 'me' URL matches expected

"Discovery timeout"

Cause: Profile URL slow or unreachable

Solution:

1. Increase timeout if needed
2. Check network connectivity
3. Verify profile URL is correct

Rollback Plan

If issues arise:

1. Keep old code available

   • Tag release before migration
   • Keep backup of old implementation

2. Quick rollback procedure

   # Revert to previous version
   git checkout tags/pre-discovery-migration
   
   # Restore old configuration
   cp config.ini.backup config.ini
   
   # Restart application
   systemctl restart starpunk
   

3. Document issues for retry

   • What failed?
   • Error messages
   • Affected users

Success Criteria

Migration is successful when:

1. All token verifications use discovered endpoints
2. No hardcoded endpoints remain
3. Performance is acceptable (< 500ms uncached)
4. All tests pass
5. Documentation is complete
6. Users can authenticate successfully

Long-term Benefits

After this migration:

1. True IndieAuth Compliance: Finally following the specification
2. User Freedom: Users control their authentication
3. Better Security: No single point of failure
4. Future Proof: Ready for new IndieAuth providers
5. Maintainable: Cleaner, spec-compliant code

---

Document Version: 1.0 Created: 2024-11-24 Purpose: Fix critical IndieAuth implementation error Priority: CRITICAL - Must be fixed before V1 release