StarPunk/docs/standards/utility-function-patterns.md

# Utility Function Patterns

## Purpose

This document establishes coding patterns and conventions specifically for utility functions in StarPunk. These patterns ensure utilities are consistent, testable, and maintainable.

## Philosophy

Utility functions should be:
- **Pure**: No side effects where possible
- **Focused**: One responsibility per function
- **Predictable**: Same input always produces same output
- **Testable**: Easy to test in isolation
- **Documented**: Clear purpose and usage

## Function Design Patterns

### 1. Pure Functions (Preferred)

**Pattern**: Functions that don't modify state or have side effects.

**Good Example**:
```python
def calculate_content_hash(content: str) -> str:
    """Calculate SHA-256 hash of content."""
    return hashlib.sha256(content.encode('utf-8')).hexdigest()
```

**Why**: Easy to test, no hidden dependencies, predictable behavior.

**When to Use**: Calculations, transformations, validations.

### 2. Functions with I/O Side Effects

**Pattern**: Functions that read/write files or interact with external systems.

**Good Example**:
```python
def write_note_file(file_path: Path, content: str) -> None:
    """Write note content to file atomically."""
    temp_path = file_path.with_suffix(file_path.suffix + '.tmp')
    try:
        temp_path.write_text(content, encoding='utf-8')
        temp_path.replace(file_path)
    except Exception:
        temp_path.unlink(missing_ok=True)
        raise
```

**Why**: Side effects are isolated, error handling is explicit, cleanup is guaranteed.

**When to Use**: File operations, database operations, network calls.

### 3. Validation Functions

**Pattern**: Functions that check validity and return boolean or raise exception.

**Good Example** (Boolean Return):
```python
def validate_slug(slug: str) -> bool:
    """Validate that slug meets requirements."""
    if not slug:
        return False
    if len(slug) > MAX_SLUG_LENGTH:
        return False
    return bool(SLUG_PATTERN.match(slug))
```

**Good Example** (Exception Raising):
```python
def require_valid_slug(slug: str) -> None:
    """Require slug to be valid, raise ValueError if not."""
    if not validate_slug(slug):
        raise ValueError(
            f"Invalid slug '{slug}': must be 1-100 characters, "
            f"lowercase alphanumeric with hyphens only"
        )
```

**When to Use**: Input validation, precondition checking, security checks.

### 4. Generator Functions

**Pattern**: Functions that yield values instead of returning lists.

**Good Example**:
```python
def iter_note_files(notes_dir: Path) -> Iterator[Path]:
    """Iterate over all note files in directory."""
    for year_dir in sorted(notes_dir.iterdir()):
        if not year_dir.is_dir():
            continue
        for month_dir in sorted(year_dir.iterdir()):
            if not month_dir.is_dir():
                continue
            for note_file in sorted(month_dir.glob("*.md")):
                yield note_file
```

**Why**: Memory efficient, lazy evaluation, can be interrupted.

**When to Use**: Processing many items, large datasets, streaming operations.

## Error Handling Patterns

### 1. Specific Exceptions

**Pattern**: Raise specific exception types with descriptive messages.

**Good**:
```python
def generate_slug(content: str) -> str:
    """Generate slug from content."""
    if not content or not content.strip():
        raise ValueError("Content cannot be empty or whitespace-only")

    # ... rest of function
```

**Bad**:
```python
def generate_slug(content: str) -> str:
    if not content:
        raise Exception("Bad input")  # Too generic
```

### 2. Error Message Format

**Pattern**: Include context, expected behavior, and actual problem.

**Good**:
```python
raise ValueError(
    f"Invalid slug '{slug}': must contain only lowercase letters, "
    f"numbers, and hyphens (pattern: {SLUG_PATTERN.pattern})"
)

raise FileNotFoundError(
    f"Note file not found: {file_path}. "
    f"Database may be out of sync with filesystem."
)
```

**Bad**:
```python
raise ValueError("Invalid slug")
raise FileNotFoundError("File missing")
```

### 3. Exception Chaining

**Pattern**: Preserve original exception context when re-raising.

**Good**:
```python
def read_note_file(file_path: Path) -> str:
    """Read note content from file."""
    try:
        return file_path.read_text(encoding='utf-8')
    except UnicodeDecodeError as e:
        raise ValueError(
            f"Failed to read {file_path}: invalid UTF-8 encoding"
        ) from e
```

**Why**: Preserves stack trace, shows root cause.

### 4. Cleanup on Error

**Pattern**: Use try/finally or context managers to ensure cleanup.

**Good**:
```python
def write_note_file(file_path: Path, content: str) -> None:
    """Write note content to file atomically."""
    temp_path = file_path.with_suffix('.tmp')
    try:
        temp_path.write_text(content, encoding='utf-8')
        temp_path.replace(file_path)
    except Exception:
        temp_path.unlink(missing_ok=True)  # Cleanup on error
        raise
```

## Type Hint Patterns

### 1. Basic Types

**Pattern**: Use built-in types where possible.

```python
def generate_slug(content: str, created_at: Optional[datetime] = None) -> str:
    """Generate URL-safe slug from content."""
    pass

def validate_note_path(file_path: Path, data_dir: Path) -> bool:
    """Validate file path is within data directory."""
    pass
```

### 2. Collection Types

**Pattern**: Specify element types for collections.

```python
from typing import List, Dict, Set, Optional

def make_slug_unique(base_slug: str, existing_slugs: Set[str]) -> str:
    """Make slug unique by adding suffix if needed."""
    pass

def get_note_paths(notes_dir: Path) -> List[Path]:
    """Get all note file paths."""
    pass
```

### 3. Optional Types

**Pattern**: Use Optional[T] for nullable parameters and returns.

```python
from typing import Optional

def find_note(slug: str) -> Optional[Path]:
    """Find note file by slug, returns None if not found."""
    pass
```

### 4. Union Types (Use Sparingly)

**Pattern**: Use Union when a parameter truly accepts multiple types.

```python
from typing import Union
from pathlib import Path

def ensure_path(path: Union[str, Path]) -> Path:
    """Convert string or Path to Path object."""
    return Path(path) if isinstance(path, str) else path
```

**Note**: Prefer single types. Only use Union when necessary.

## Documentation Patterns

### 1. Function Docstrings

**Pattern**: Google-style docstrings with sections.

```python
def generate_slug(content: str, created_at: Optional[datetime] = None) -> str:
    """
    Generate URL-safe slug from note content

    Creates a slug by extracting the first few words from the content and
    normalizing them to lowercase with hyphens. If content is insufficient,
    falls back to timestamp-based slug.

    Args:
        content: The note content (markdown text)
        created_at: Optional timestamp for fallback slug (defaults to now)

    Returns:
        URL-safe slug string (lowercase, alphanumeric + hyphens only)

    Raises:
        ValueError: If content is empty or contains only whitespace

    Examples:
        >>> generate_slug("Hello World! This is my first note.")
        'hello-world-this-is-my'

        >>> generate_slug("Testing... with special chars!@#")
        'testing-with-special-chars'

    Notes:
        - This function does NOT check for uniqueness
        - Caller must verify slug doesn't exist in database
    """
```

**Required Sections**:
- Summary (first line)
- Description (paragraph after summary)
- Args (if any parameters)
- Returns (if returns value)
- Raises (if raises exceptions)

**Optional Sections**:
- Examples (highly recommended)
- Notes (for important caveats)
- References (for external specs)

### 2. Inline Comments

**Pattern**: Explain why, not what.

**Good**:
```python
# Use atomic rename to prevent file corruption if interrupted
temp_path.replace(file_path)

# Random suffix prevents enumeration attacks
suffix = secrets.token_urlsafe(4)[:4]
```

**Bad**:
```python
# Rename temp file to final path
temp_path.replace(file_path)

# Generate suffix
suffix = secrets.token_urlsafe(4)[:4]
```

### 3. Module Docstrings

**Pattern**: Describe module purpose and contents.

```python
"""
Core utility functions for StarPunk

This module provides essential utilities for slug generation, file operations,
hashing, and date/time handling. These utilities are used throughout the
application and have no external dependencies beyond standard library and
Flask configuration.

Functions:
    generate_slug: Create URL-safe slug from content
    calculate_content_hash: Calculate SHA-256 hash
    write_note_file: Atomically write note to file
    format_rfc822: Format datetime for RSS feeds

Constants:
    MAX_SLUG_LENGTH: Maximum slug length (100)
    SLUG_PATTERN: Regex for valid slugs
"""
```

## Testing Patterns

### 1. Test Function Naming

**Pattern**: `test_{function_name}_{scenario}`

```python
def test_generate_slug_from_content():
    """Test basic slug generation from content."""
    slug = generate_slug("Hello World This Is My Note")
    assert slug == "hello-world-this-is-my"

def test_generate_slug_empty_content():
    """Test slug generation raises error on empty content."""
    with pytest.raises(ValueError):
        generate_slug("")

def test_generate_slug_special_characters():
    """Test slug generation removes special characters."""
    slug = generate_slug("Testing... with!@# special chars")
    assert slug == "testing-with-special-chars"
```

### 2. Test Organization

**Pattern**: Group related tests in classes.

```python
class TestSlugGeneration:
    """Test slug generation functions"""

    def test_generate_slug_from_content(self):
        """Test basic slug generation."""
        pass

    def test_generate_slug_empty_content(self):
        """Test error on empty content."""
        pass


class TestContentHashing:
    """Test content hashing functions"""

    def test_calculate_hash_consistency(self):
        """Test hash is consistent."""
        pass
```

### 3. Fixtures for Common Setup

**Pattern**: Use pytest fixtures for reusable test data.

```python
import pytest
from pathlib import Path

@pytest.fixture
def temp_note_file(tmp_path):
    """Create temporary note file for testing."""
    file_path = tmp_path / "test.md"
    file_path.write_text("# Test Note")
    return file_path

def test_read_note_file(temp_note_file):
    """Test reading note file."""
    content = read_note_file(temp_note_file)
    assert content == "# Test Note"
```

### 4. Parameterized Tests

**Pattern**: Test multiple cases with one test function.

```python
@pytest.mark.parametrize("content,expected", [
    ("Hello World", "hello-world"),
    ("Testing 123", "testing-123"),
    ("Special!@# Chars", "special-chars"),
])
def test_generate_slug_variations(content, expected):
    """Test slug generation with various inputs."""
    slug = generate_slug(content)
    assert slug == expected
```

## Security Patterns

### 1. Path Validation

**Pattern**: Always validate paths before file operations.

```python
def safe_file_operation(file_path: Path, data_dir: Path) -> None:
    """Perform file operation with path validation."""
    # ALWAYS validate first
    if not validate_note_path(file_path, data_dir):
        raise ValueError(f"Invalid file path: {file_path}")

    # Now safe to operate
    file_path.write_text("content")
```

### 2. Use secrets for Random

**Pattern**: Use `secrets` module, not `random` for security-sensitive operations.

**Good**:
```python
import secrets

def generate_random_suffix(length: int = 4) -> str:
    """Generate cryptographically secure random suffix."""
    chars = 'abcdefghijklmnopqrstuvwxyz0123456789'
    return ''.join(secrets.choice(chars) for _ in range(length))
```

**Bad**:
```python
import random

def generate_random_suffix(length: int = 4) -> str:
    """Generate random suffix."""
    chars = 'abcdefghijklmnopqrstuvwxyz0123456789'
    return ''.join(random.choice(chars) for _ in range(length))  # NOT secure
```

### 3. Input Sanitization

**Pattern**: Validate and sanitize all external input.

```python
def generate_slug(content: str) -> str:
    """Generate slug from content."""
    # Validate input
    if not content or not content.strip():
        raise ValueError("Content cannot be empty")

    # Sanitize by removing dangerous characters
    normalized = normalize_slug_text(content)

    # Additional validation
    if not validate_slug(normalized):
        # Fallback to safe default
        normalized = datetime.utcnow().strftime("%Y%m%d-%H%M%S")

    return normalized
```

## Performance Patterns

### 1. Lazy Evaluation

**Pattern**: Don't compute what you don't need.

**Good**:
```python
def generate_slug(content: str, created_at: Optional[datetime] = None) -> str:
    """Generate slug from content."""
    slug = normalize_slug_text(content)

    # Only generate timestamp if needed
    if not slug:
        created_at = created_at or datetime.utcnow()
        slug = created_at.strftime("%Y%m%d-%H%M%S")

    return slug
```

### 2. Compile Regex Once

**Pattern**: Define regex patterns as module constants.

**Good**:
```python
# Module level - compiled once
SLUG_PATTERN = re.compile(r'^[a-z0-9]+(?:-[a-z0-9]+)*$')
SAFE_SLUG_PATTERN = re.compile(r'[^a-z0-9-]')

def validate_slug(slug: str) -> bool:
    """Validate slug."""
    return bool(SLUG_PATTERN.match(slug))

def normalize_slug_text(text: str) -> str:
    """Normalize text for slug."""
    return SAFE_SLUG_PATTERN.sub('', text)
```

**Bad**:
```python
def validate_slug(slug: str) -> bool:
    """Validate slug."""
    # Compiles regex every call - inefficient
    return bool(re.match(r'^[a-z0-9]+(?:-[a-z0-9]+)*$', slug))
```

### 3. Avoid Premature Optimization

**Pattern**: Write clear code first, optimize if needed.

**Good**:
```python
def extract_first_words(text: str, max_words: int = 5) -> str:
    """Extract first N words from text."""
    words = text.split()
    return ' '.join(words[:max_words])
```

**Don't Do This Unless Profiling Shows It's Necessary**:
```python
def extract_first_words(text: str, max_words: int = 5) -> str:
    """Extract first N words from text."""
    # Premature optimization - more complex, minimal gain
    words = []
    count = 0
    for word in text.split():
        words.append(word)
        count += 1
        if count >= max_words:
            break
    return ' '.join(words)
```

## Constants Pattern

### 1. Module-Level Constants

**Pattern**: Define configuration as constants at module level.

```python
# Slug configuration
MAX_SLUG_LENGTH = 100
MIN_SLUG_LENGTH = 1
SLUG_WORDS_COUNT = 5
RANDOM_SUFFIX_LENGTH = 4

# File operations
TEMP_FILE_SUFFIX = '.tmp'
TRASH_DIR_NAME = '.trash'

# Hashing
CONTENT_HASH_ALGORITHM = 'sha256'
```

**Why**: Easy to modify, clear intent, compile-time resolution.

### 2. Naming Convention

**Pattern**: ALL_CAPS_WITH_UNDERSCORES

```python
MAX_NOTE_LENGTH = 10000        # Good
DEFAULT_ENCODING = 'utf-8'     # Good
maxNoteLength = 10000          # Bad
max_note_length = 10000        # Bad (looks like variable)
```

### 3. Magic Numbers

**Pattern**: Replace magic numbers with named constants.

**Good**:
```python
SLUG_WORDS_COUNT = 5

def generate_slug(content: str) -> str:
    """Generate slug from content."""
    words = content.split()[:SLUG_WORDS_COUNT]
    return normalize_slug_text(' '.join(words))
```

**Bad**:
```python
def generate_slug(content: str) -> str:
    """Generate slug from content."""
    words = content.split()[:5]  # What is 5? Why 5?
    return normalize_slug_text(' '.join(words))
```

## Anti-Patterns to Avoid

### 1. God Functions

**Bad**:
```python
def process_note(content, do_hash=True, do_slug=True, do_path=True, ...):
    """Do everything with a note."""
    # 500 lines of code doing too many things
    pass
```

**Good**:
```python
def generate_slug(content: str) -> str:
    """Generate slug."""
    pass

def calculate_hash(content: str) -> str:
    """Calculate hash."""
    pass

def generate_path(slug: str, timestamp: datetime) -> Path:
    """Generate path."""
    pass
```

### 2. Mutable Default Arguments

**Bad**:
```python
def create_note(content: str, tags: list = []) -> Note:
    tags.append('note')  # Modifies shared default list!
    return Note(content, tags)
```

**Good**:
```python
def create_note(content: str, tags: Optional[List[str]] = None) -> Note:
    if tags is None:
        tags = []
    tags.append('note')
    return Note(content, tags)
```

### 3. Returning Multiple Types

**Bad**:
```python
def find_note(slug: str) -> Union[Note, bool, None]:
    """Find note by slug."""
    if slug_invalid:
        return False
    note = db.query(slug)
    return note if note else None
```

**Good**:
```python
def find_note(slug: str) -> Optional[Note]:
    """Find note by slug, returns None if not found."""
    if not validate_slug(slug):
        raise ValueError(f"Invalid slug: {slug}")
    return db.query(slug)  # Returns Note or None
```

### 4. Silent Failures

**Bad**:
```python
def generate_slug(content: str) -> str:
    """Generate slug."""
    try:
        slug = normalize_slug_text(content)
        return slug if slug else "untitled"  # Silent fallback
    except Exception:
        return "untitled"  # Swallowing errors
```

**Good**:
```python
def generate_slug(content: str) -> str:
    """Generate slug."""
    if not content or not content.strip():
        raise ValueError("Content cannot be empty")

    slug = normalize_slug_text(content)
    if not slug:
        # Explicit fallback with timestamp
        slug = datetime.utcnow().strftime("%Y%m%d-%H%M%S")

    return slug
```

## Summary

**Key Principles**:
1. Pure functions are preferred
2. Specific exceptions with clear messages
3. Type hints on all functions
4. Comprehensive docstrings
5. Security-first validation
6. Test everything thoroughly
7. Constants for configuration
8. Clear over clever

**Remember**: Utility functions are the foundation. Make them rock-solid.

## References

- [Python Coding Standards](/home/phil/Projects/starpunk/docs/standards/python-coding-standards.md)
- [PEP 8 - Style Guide](https://peps.python.org/pep-0008/)
- [PEP 257 - Docstring Conventions](https://peps.python.org/pep-0257/)
- [PEP 484 - Type Hints](https://peps.python.org/pep-0484/)
- [Python secrets Documentation](https://docs.python.org/3/library/secrets.html)
- [Pytest Best Practices](https://docs.pytest.org/en/stable/goodpractices.html)