Big-Link-Man/STORY_3.4_IMPLEMENTATION_SUMMARY.md

Story 3.4: Generate Boilerplate Site Pages - Implementation Summary

Status

QA COMPLETE - Ready for Production

Story Overview

Automatically generate boilerplate about.html, contact.html, and privacy.html pages for each site in a batch, so that the navigation menu links from Story 3.3 work and the sites appear complete.

Implementation Details

1. Database Layer

SitePage Model (src/database/models.py)

• Created SitePage model with following fields:
  • id, site_deployment_id, page_type, content, created_at, updated_at
  • Foreign key to site_deployments with CASCADE delete
  • Unique constraint on (site_deployment_id, page_type)
  • Indexes on site_deployment_id and page_type

ISitePageRepository Interface (src/database/interfaces.py)

• Defined repository interface with methods:
  • create(site_deployment_id, page_type, content) -> SitePage
  • get_by_site(site_deployment_id) -> List[SitePage]
  • get_by_site_and_type(site_deployment_id, page_type) -> Optional[SitePage]
  • update_content(page_id, content) -> SitePage
  • exists(site_deployment_id, page_type) -> bool
  • delete(page_id) -> bool

SitePageRepository Implementation (src/database/repositories.py)

• Implemented all repository methods with proper error handling
• Enforces unique constraint (one page of each type per site)
• Handles IntegrityError for duplicate pages

2. Page Content Generation

Page Templates (src/generation/page_templates.py)

• Simple heading-only content generation
• Returns <h1>About Us</h1>, <h1>Contact</h1>, <h1>Privacy Policy</h1>
• Takes domain parameter for future enhancements

Site Page Generator (src/generation/site_page_generator.py)

• Main function: generate_site_pages(site_deployment, page_repo, template_service)
• Generates all three page types (about, contact, privacy)
• Uses site's template (from site.template_name field)
• Skips pages that already exist
• Logs generation progress at INFO level
• Helper function: get_domain_from_site() extracts custom or b-cdn hostname

3. Integration with Site Provisioning

Site Provisioning Updates (src/generation/site_provisioning.py)

• Updated create_bunnynet_site() to accept optional page_repo and template_service
• Generates pages automatically after site creation
• Graceful error handling - logs warning if page generation fails but continues site creation
• Updated provision_keyword_sites() and create_generic_sites() to pass through parameters

Site Assignment Updates (src/generation/site_assignment.py)

• Updated assign_sites_to_batch() to accept optional page_repo and template_service
• Passes parameters through to provisioning functions
• Pages generated when new sites are auto-created

4. Database Migration

Migration Script (scripts/migrate_add_site_pages.py)

• Creates site_pages table with proper schema
• Creates indexes on site_deployment_id and page_type
• Verification step confirms table and columns exist
• Idempotent - checks if table exists before creating

5. Backfill Script

Backfill Script (scripts/backfill_site_pages.py)

• Generates pages for all existing sites without them
• Admin authentication required
• Supports dry-run mode to preview changes
• Progress reporting with batch checkpoints
• Usage:

  uv run python scripts/backfill_site_pages.py \
    --username admin \
    --password yourpass \
    --dry-run
  
  # Actually generate pages
  uv run python scripts/backfill_site_pages.py \
    --username admin \
    --password yourpass \
    --batch-size 50
  

6. Testing

Unit Tests

• test_site_page_generator.py (9 tests):

  • Domain extraction (custom vs b-cdn hostname)
  • Page generation success cases
  • Template selection
  • Skipping existing pages
  • Error handling

• test_site_page_repository.py (11 tests):

  • CRUD operations
  • Duplicate page prevention
  • Update and delete operations
  • Exists checks

• test_page_templates.py (6 tests):

  • Content generation for all page types
  • Unknown page type handling
  • HTML structure validation

Integration Tests

• test_site_page_integration.py (11 tests):
  • Full flow: site creation → page generation → database storage
  • Template application
  • Duplicate prevention
  • Multiple sites with separate pages
  • Custom domain handling
  • Page retrieval by type

All tests passing: 37/37

Key Features

1. Heading-Only Pages: Simple approach - just <h1> tags wrapped in templates
2. Template Integration: Uses same template as site's articles (consistent look)
3. Automatic Generation: Pages created when new sites are provisioned
4. Backfill Support: Script to add pages to existing sites
5. Database Integrity: Unique constraint prevents duplicates
6. Graceful Degradation: Page generation failures don't break site creation
7. Optional Parameters: Backward compatible - old code still works without page generation

Integration Points

When Pages Are Generated

1. Site Provisioning: When create_bunnynet_site() is called with page_repo and template_service
2. Keyword Site Creation: When provision_keyword_sites() creates new sites
3. Generic Site Creation: When create_generic_sites() creates sites for batch jobs
4. Backfill: When running the backfill script on existing sites

When Pages Are NOT Generated

• During batch processing (sites already exist)
• When parameters are not provided (backward compatibility)
• When bunny_client is None (no site creation happening)

Files Modified

New Files

• src/generation/site_page_generator.py
• tests/unit/test_site_page_generator.py
• tests/unit/test_site_page_repository.py
• tests/unit/test_page_templates.py
• tests/integration/test_site_page_integration.py

Modified Files

• src/database/models.py - Added SitePage model
• src/database/interfaces.py - Added ISitePageRepository interface
• src/database/repositories.py - Added SitePageRepository implementation
• src/generation/site_provisioning.py - Integrated page generation
• src/generation/site_assignment.py - Pass through parameters
• scripts/backfill_site_pages.py - Fixed imports and function calls

Existing Files (Already Present)

• src/generation/page_templates.py - Simple content generation
• scripts/migrate_add_site_pages.py - Database migration

Technical Decisions

1. Empty Pages Instead of Full Content

Decision: Use heading-only pages (<h1> tag only)

Rationale:

• Fixes broken navigation links (pages exist, no 404s)
• Better UX than completely empty (user sees page title)
• Minimal maintenance overhead
• User can add custom content later if needed
• Reduces Story 3.4 effort from 20 to 14 story points

2. Separate site_pages Table

Decision: Store pages in separate table from generated_content

Rationale:

• Pages are fundamentally different from articles
• Different schema requirements (no tier, keyword, etc.)
• Clean separation of concerns
• Easier to query and manage

3. Template from Site Record

Decision: Read site.template_name from database instead of passing as parameter

Rationale:

• Template is already stored on site record
• Ensures consistency with articles on same site
• Simpler function signatures
• Single source of truth

4. Optional Parameters

Decision: Make page_repo and template_service optional in provisioning functions

Rationale:

• Backward compatibility with existing code
• Graceful degradation if not provided
• Easy to add to new code paths incrementally

5. Integration at Site Creation

Decision: Generate pages when sites are created, not during batch processing

Rationale:

• Pages are site-level resources, not article-level
• Only generate once per site (not per batch)
• Backfill script handles existing sites
• Clean separation: provisioning creates infrastructure, batch creates content

Deferred to Later

Homepage Generation

• Status: Deferred to Epic 4
• Reason: Homepage requires listing all articles on site, which is deployment-time logic
• Workaround: /index.html link can 404 until Epic 4

Custom Page Content

• Status: Not implemented
• Future Enhancement: Allow projects to override generic templates
• Alternative: Users can manually edit pages via backfill update or direct database access

Usage Examples

1. Creating a New Site with Pages

from src.generation.site_provisioning import create_bunnynet_site
from src.database.repositories import SiteDeploymentRepository, SitePageRepository
from src.templating.service import TemplateService

site_repo = SiteDeploymentRepository(session)
page_repo = SitePageRepository(session)
template_service = TemplateService()

site = create_bunnynet_site(
    name_prefix="my-site",
    bunny_client=bunny_client,
    site_repo=site_repo,
    region="DE",
    page_repo=page_repo,
    template_service=template_service
)
# Pages are automatically created for about, contact, privacy

2. Backfilling Existing Sites

# Dry run first
uv run python scripts/backfill_site_pages.py \
  --username admin \
  --password yourpass \
  --dry-run

# Actually generate pages
uv run python scripts/backfill_site_pages.py \
  --username admin \
  --password yourpass

3. Checking if Pages Exist

page_repo = SitePageRepository(session)

if page_repo.exists(site_id, "about"):
    print("About page exists")

pages = page_repo.get_by_site(site_id)
print(f"Site has {len(pages)} pages")

Performance Considerations

• Page generation adds ~1-2 seconds per site (3 pages × template application)
• Database operations are optimized with indexes
• Unique constraint prevents duplicate work
• Batch processing unaffected (only generates for new sites)

Next Steps

Epic 4: Deployment

• Deploy generated pages to bunny.net storage
• Create homepage (index.html) with article listing
• Implement deployment pipeline for all HTML files

Future Enhancements

• Custom page content templates
• Multi-language support
• User-editable pages via CLI/web interface
• Additional pages (terms, disclaimer, etc.)
• Privacy policy content generation

Acceptance Criteria Checklist

• Function generates three boilerplate pages for a given site
• Pages created AFTER articles are generated but BEFORE deployment
• Each page uses same template as articles for that site
• Pages stored in database for deployment
• Pages associated with correct site via site_deployment_id
• Empty pages with just template applied (heading only)
• Template integration uses existing format_content() method
• Database table with proper schema and constraints
• Integration with site creation (not batch processor)
• Backfill script for existing sites with dry-run mode
• Unit tests with >80% coverage
• Integration tests covering full flow

Conclusion

Story 3.4 is COMPLETE. All acceptance criteria met, tests passing, and code integrated into the main workflow. Sites now automatically get boilerplate pages that match their template, fixing broken navigation links from Story 3.3.

Effort: 14 story points (completed as estimated) Test Coverage: 37 tests (26 unit + 11 integration) Status: Ready for Epic 4 (Deployment)