`
+- Takes domain parameter for future enhancements
-#### Site Page Generator
-- **Location**: `src/generation/site_page_generator.py`
-- **Main Function**: `generate_site_pages(site_deployment, template_name, page_repo, template_service) -> List[SitePage]`
-- **Features**:
- - Generates all three page types
- - Skips existing pages
- - Wraps content in HTML templates
- - Logs generation progress
- - Handles errors gracefully
+#### Site Page Generator (`src/generation/site_page_generator.py`)
+- Main function: `generate_site_pages(site_deployment, page_repo, template_service)`
+- Generates all three page types (about, contact, privacy)
+- Uses site's template (from `site.template_name` field)
+- Skips pages that already exist
+- Logs generation progress at INFO level
+- Helper function: `get_domain_from_site()` extracts custom or b-cdn hostname
-#### Helper Function
-- `get_domain_from_site(site_deployment) -> str`
-- Extracts domain (custom hostname or bcdn hostname)
+### 3. Integration with Site Provisioning
-### 4. Template Service Updates
+#### Site Provisioning Updates (`src/generation/site_provisioning.py`)
+- Updated `create_bunnynet_site()` to accept optional `page_repo` and `template_service`
+- Generates pages automatically after site creation
+- Graceful error handling - logs warning if page generation fails but continues site creation
+- Updated `provision_keyword_sites()` and `create_generic_sites()` to pass through parameters
-#### New Method: `format_page`
-- **Location**: `src/templating/service.py`
-- Simplified version of `format_content` for pages
-- Uses same templates as articles but with simplified parameters
-- No meta description (reuses page title)
+#### Site Assignment Updates (`src/generation/site_assignment.py`)
+- Updated `assign_sites_to_batch()` to accept optional `page_repo` and `template_service`
+- Passes parameters through to provisioning functions
+- Pages generated when new sites are auto-created
-### 5. Integration with Site Provisioning
+### 4. Database Migration
-#### Updated Functions in `src/generation/site_provisioning.py`
+#### Migration Script (`scripts/migrate_add_site_pages.py`)
+- Creates `site_pages` table with proper schema
+- Creates indexes on `site_deployment_id` and `page_type`
+- Verification step confirms table and columns exist
+- Idempotent - checks if table exists before creating
-##### `create_bunnynet_site`
-- Added optional parameters:
- - `page_repo: Optional[ISitePageRepository] = None`
- - `template_service: Optional[TemplateService] = None`
- - `template_name: str = "basic"`
-- Generates pages after site creation if repos provided
-- Logs page generation results
-- Continues on failure with warning
+### 5. Backfill Script
-##### `provision_keyword_sites`
-- Added same optional parameters
-- Passes to `create_bunnynet_site`
-
-##### `create_generic_sites`
-- Added same optional parameters
-- Passes to `create_bunnynet_site`
-
-#### Updated CLI Command
-- **Location**: `src/cli/commands.py`
-- **Command**: `provision-site`
-- Generates boilerplate pages after site creation
-- Shows success/failure message
-- Continues with site provisioning even if page generation fails
-
-### 6. Backfill Script
-
-#### Script: `scripts/backfill_site_pages.py`
+#### Backfill Script (`scripts/backfill_site_pages.py`)
- Generates pages for all existing sites without them
-- **Features**:
- - Admin authentication required
- - Dry-run mode for preview
- - Batch processing with progress updates
- - Template selection (default: basic)
- - Error handling per site
- - Summary statistics
+- Admin authentication required
+- Supports dry-run mode to preview changes
+- Progress reporting with batch checkpoints
+- Usage:
+ ```bash
+ uv run python scripts/backfill_site_pages.py \
+ --username admin \
+ --password yourpass \
+ --dry-run
+
+ # Actually generate pages
+ uv run python scripts/backfill_site_pages.py \
+ --username admin \
+ --password yourpass \
+ --batch-size 50
+ ```
-#### Usage:
-```bash
-# Dry run
-uv run python scripts/backfill_site_pages.py \
- --username admin \
- --password yourpass \
- --template basic \
- --dry-run
-
-# Actual run
-uv run python scripts/backfill_site_pages.py \
- --username admin \
- --password yourpass \
- --template basic \
- --batch-size 100
-```
-
-### 7. Testing
+### 6. Testing
#### Unit Tests
-- **test_page_templates.py** (5 tests)
- - Tests heading generation for each page type
- - Tests unknown page type handling
- - Tests HTML string output
-
-- **test_site_page_generator.py** (8 tests)
- - Tests domain extraction
- - Tests page generation flow
- - Tests skipping existing pages
- - Tests template usage
- - Tests error handling
-
-- **test_site_page_repository.py** (7 tests)
- - Tests CRUD operations
- - Tests unique constraint
- - Tests exists/delete operations
- - Tests database integration
+- **test_site_page_generator.py** (9 tests):
+ - Domain extraction (custom vs b-cdn hostname)
+ - Page generation success cases
+ - Template selection
+ - Skipping existing pages
+ - Error handling
+
+- **test_site_page_repository.py** (11 tests):
+ - CRUD operations
+ - Duplicate page prevention
+ - Update and delete operations
+ - Exists checks
+
+- **test_page_templates.py** (6 tests):
+ - Content generation for all page types
+ - Unknown page type handling
+ - HTML structure validation
#### Integration Tests
-- **test_site_page_integration.py** (6 tests)
- - Tests full page generation flow
- - Tests template application
- - Tests multiple templates
- - Tests duplicate prevention
- - Tests HTML structure
- - Tests custom vs bcdn hostnames
+- **test_site_page_integration.py** (11 tests):
+ - Full flow: site creation → page generation → database storage
+ - Template application
+ - Duplicate prevention
+ - Multiple sites with separate pages
+ - Custom domain handling
+ - Page retrieval by type
-#### Test Results
-- **20 unit tests passed**
-- **6 integration tests passed**
-- **All tests successful**
+**All tests passing:** 37/37
-## Technical Decisions
+## Key Features
-### 1. Minimal Page Content
-- Pages contain only heading (`
` tag)
-- No body content generated
-- User can add content manually later if needed
-- Simpler implementation, faster generation
-- Reduces maintenance burden
+1. **Heading-Only Pages**: Simple approach - just `
` tags wrapped in templates
+2. **Template Integration**: Uses same template as site's articles (consistent look)
+3. **Automatic Generation**: Pages created when new sites are provisioned
+4. **Backfill Support**: Script to add pages to existing sites
+5. **Database Integrity**: Unique constraint prevents duplicates
+6. **Graceful Degradation**: Page generation failures don't break site creation
+7. **Optional Parameters**: Backward compatible - old code still works without page generation
-### 2. Separate Table for Pages
-- Pages stored in dedicated `site_pages` table
-- Clean separation from article content
-- Different schema needs (no title/outline/word_count)
-- Easier to manage and query
+## Integration Points
-### 3. Optional Integration
-- Page generation is optional in site provisioning
-- Backward compatible with existing code
-- Allows gradual rollout
-- Doesn't break existing workflows
+### When Pages Are Generated
+1. **Site Provisioning**: When `create_bunnynet_site()` is called with `page_repo` and `template_service`
+2. **Keyword Site Creation**: When `provision_keyword_sites()` creates new sites
+3. **Generic Site Creation**: When `create_generic_sites()` creates sites for batch jobs
+4. **Backfill**: When running the backfill script on existing sites
-### 4. CASCADE DELETE
-- Database-level cascade delete
-- Pages automatically deleted when site deleted
-- Maintains referential integrity
-- Simplifies cleanup logic
-
-## Files Created
-
-### Core Implementation
-1. `src/database/models.py` - Added `SitePage` model
-2. `src/database/interfaces.py` - Added `ISitePageRepository` interface
-3. `src/database/repositories.py` - Added `SitePageRepository` class
-4. `src/generation/page_templates.py` - Page content generation
-5. `src/generation/site_page_generator.py` - Page generation logic
-
-### Scripts
-6. `scripts/migrate_add_site_pages.py` - Database migration
-7. `scripts/backfill_site_pages.py` - Backfill script for existing sites
-
-### Tests
-8. `tests/unit/test_page_templates.py`
-9. `tests/unit/test_site_page_generator.py`
-10. `tests/unit/test_site_page_repository.py`
-11. `tests/integration/test_site_page_integration.py`
-
-### Documentation
-12. `STORY_3.4_IMPLEMENTATION_SUMMARY.md` - This file
+### When Pages Are NOT Generated
+- During batch processing (sites already exist)
+- When parameters are not provided (backward compatibility)
+- When bunny_client is None (no site creation happening)
## Files Modified
-1. `src/database/models.py` - Added SitePage model
-2. `src/database/interfaces.py` - Added ISitePageRepository interface
-3. `src/database/repositories.py` - Added SitePageRepository implementation
-4. `src/templating/service.py` - Added format_page method
-5. `src/generation/site_provisioning.py` - Updated all functions to support page generation
-6. `src/cli/commands.py` - Updated provision-site command
+### New Files
+- `src/generation/site_page_generator.py`
+- `tests/unit/test_site_page_generator.py`
+- `tests/unit/test_site_page_repository.py`
+- `tests/unit/test_page_templates.py`
+- `tests/integration/test_site_page_integration.py`
-## Migration Steps
+### Modified Files
+- `src/database/models.py` - Added SitePage model
+- `src/database/interfaces.py` - Added ISitePageRepository interface
+- `src/database/repositories.py` - Added SitePageRepository implementation
+- `src/generation/site_provisioning.py` - Integrated page generation
+- `src/generation/site_assignment.py` - Pass through parameters
+- `scripts/backfill_site_pages.py` - Fixed imports and function calls
-### For Development/Testing
-```bash
-# Run migration
-uv run python scripts/migrate_add_site_pages.py
+### Existing Files (Already Present)
+- `src/generation/page_templates.py` - Simple content generation
+- `scripts/migrate_add_site_pages.py` - Database migration
-# Verify migration
-uv run pytest tests/unit/test_site_page_repository.py -v
+## Technical Decisions
-# Run all tests
-uv run pytest tests/ -v
+### 1. Empty Pages Instead of Full Content
+**Decision**: Use heading-only pages (`
` tag only)
+
+**Rationale**:
+- Fixes broken navigation links (pages exist, no 404s)
+- Better UX than completely empty (user sees page title)
+- Minimal maintenance overhead
+- User can add custom content later if needed
+- Reduces Story 3.4 effort from 20 to 14 story points
+
+### 2. Separate `site_pages` Table
+**Decision**: Store pages in separate table from `generated_content`
+
+**Rationale**:
+- Pages are fundamentally different from articles
+- Different schema requirements (no tier, keyword, etc.)
+- Clean separation of concerns
+- Easier to query and manage
+
+### 3. Template from Site Record
+**Decision**: Read `site.template_name` from database instead of passing as parameter
+
+**Rationale**:
+- Template is already stored on site record
+- Ensures consistency with articles on same site
+- Simpler function signatures
+- Single source of truth
+
+### 4. Optional Parameters
+**Decision**: Make `page_repo` and `template_service` optional in provisioning functions
+
+**Rationale**:
+- Backward compatibility with existing code
+- Graceful degradation if not provided
+- Easy to add to new code paths incrementally
+
+### 5. Integration at Site Creation
+**Decision**: Generate pages when sites are created, not during batch processing
+
+**Rationale**:
+- Pages are site-level resources, not article-level
+- Only generate once per site (not per batch)
+- Backfill script handles existing sites
+- Clean separation: provisioning creates infrastructure, batch creates content
+
+## Deferred to Later
+
+### Homepage Generation
+- **Status**: Deferred to Epic 4
+- **Reason**: Homepage requires listing all articles on site, which is deployment-time logic
+- **Workaround**: `/index.html` link can 404 until Epic 4
+
+### Custom Page Content
+- **Status**: Not implemented
+- **Future Enhancement**: Allow projects to override generic templates
+- **Alternative**: Users can manually edit pages via backfill update or direct database access
+
+## Usage Examples
+
+### 1. Creating a New Site with Pages
+```python
+from src.generation.site_provisioning import create_bunnynet_site
+from src.database.repositories import SiteDeploymentRepository, SitePageRepository
+from src.templating.service import TemplateService
+
+site_repo = SiteDeploymentRepository(session)
+page_repo = SitePageRepository(session)
+template_service = TemplateService()
+
+site = create_bunnynet_site(
+ name_prefix="my-site",
+ bunny_client=bunny_client,
+ site_repo=site_repo,
+ region="DE",
+ page_repo=page_repo,
+ template_service=template_service
+)
+# Pages are automatically created for about, contact, privacy
```
-### For Existing Sites
+### 2. Backfilling Existing Sites
```bash
-# Preview changes
+# Dry run first
uv run python scripts/backfill_site_pages.py \
--username admin \
--password yourpass \
--dry-run
-# Generate pages
+# Actually generate pages
uv run python scripts/backfill_site_pages.py \
--username admin \
- --password yourpass \
- --template basic
+ --password yourpass
```
-## Integration with Existing Stories
+### 3. Checking if Pages Exist
+```python
+page_repo = SitePageRepository(session)
-### Story 3.3: Content Interlinking
-- Pages fulfill navigation menu links
-- No more broken links (about.html, contact.html, privacy.html)
-- Pages use same template as articles
+if page_repo.exists(site_id, "about"):
+ print("About page exists")
-### Story 3.1: Site Assignment
-- Pages generated when sites are created
-- Each site gets its own set of pages
-- Site deletion cascades to pages
+pages = page_repo.get_by_site(site_id)
+print(f"Site has {len(pages)} pages")
+```
-### Story 2.4: Template Service
-- Pages use existing template system
-- Same visual consistency as articles
-- Supports all template types (basic, modern, classic, minimal)
+## Performance Considerations
-## Future Enhancements
+- Page generation adds ~1-2 seconds per site (3 pages × template application)
+- Database operations are optimized with indexes
+- Unique constraint prevents duplicate work
+- Batch processing unaffected (only generates for new sites)
-### Short Term
-1. Homepage (index.html) generation with article listings
-2. Additional page types (terms, disclaimer)
-3. CLI command to update page content
-4. Custom content per project
+## Next Steps
-### Long Term
-1. Rich privacy policy content
-2. Contact form integration
-3. About page with site description
-4. Multi-language support
-5. Page templates with variables
+### Epic 4: Deployment
+- Deploy generated pages to bunny.net storage
+- Create homepage (`index.html`) with article listing
+- Implement deployment pipeline for all HTML files
-## Known Limitations
+### Future Enhancements
+- Custom page content templates
+- Multi-language support
+- User-editable pages via CLI/web interface
+- Additional pages (terms, disclaimer, etc.)
+- Privacy policy content generation
-1. **CASCADE DELETE Testing**: SQLAlchemy's ORM struggles with CASCADE DELETE in test environments due to foreign key handling. The CASCADE DELETE works correctly at the database level in production.
+## Acceptance Criteria Checklist
-2. **Minimal Content**: Pages contain only headings. Users must add content manually if needed.
-
-3. **Single Template**: All pages on a site use the same template (can't mix templates within a site).
-
-4. **No Content Management**: No UI for editing page content (CLI only via backfill script).
-
-## Performance Notes
-
-- Page generation adds ~1-2 seconds per site
-- Backfill script processes ~100 sites per minute
-- Database indexes ensure fast queries
-- No significant performance impact on batch generation
-
-## Deployment Checklist
-
-- [x] Database migration created
-- [x] Migration tested on development database
-- [x] Unit tests written and passing
-- [x] Integration tests written and passing
-- [x] Backfill script created and tested
-- [x] Documentation updated
-- [x] Code integrated with existing modules
-- [x] No breaking changes to existing functionality
-
-## Success Criteria - All Met
-
-- [x] Pages generated for new sites automatically
-- [x] Pages use same template as articles
-- [x] Pages stored in database
-- [x] Navigation menu links work (no 404s)
-- [x] Backfill script for existing sites
-- [x] Tests passing (>80% coverage)
-- [x] Integration with site provisioning
-- [x] Minimal content (heading only)
-
-## Implementation Time
-
-- Total Effort: ~3 hours
-- Database Schema: 30 minutes
-- Core Logic: 1 hour
-- Integration: 45 minutes
-- Testing: 45 minutes
-- Documentation: 30 minutes
+- [x] Function generates three boilerplate pages for a given site
+- [x] Pages created AFTER articles are generated but BEFORE deployment
+- [x] Each page uses same template as articles for that site
+- [x] Pages stored in database for deployment
+- [x] Pages associated with correct site via `site_deployment_id`
+- [x] Empty pages with just template applied (heading only)
+- [x] Template integration uses existing `format_content()` method
+- [x] Database table with proper schema and constraints
+- [x] Integration with site creation (not batch processor)
+- [x] Backfill script for existing sites with dry-run mode
+- [x] Unit tests with >80% coverage
+- [x] Integration tests covering full flow
## Conclusion
-Story 3.4 successfully implements boilerplate page generation for all sites. The implementation is clean, well-tested, and integrates seamlessly with existing code. Navigation menu links now work correctly, and sites appear more complete.
-
-The heading-only approach keeps implementation simple while providing the essential functionality. Users can add custom content to specific pages as needed through future enhancements.
-
-All acceptance criteria have been met, and the system is ready for production deployment.
+Story 3.4 is **COMPLETE**. All acceptance criteria met, tests passing, and code integrated into the main workflow. Sites now automatically get boilerplate pages that match their template, fixing broken navigation links from Story 3.3.
+**Effort**: 14 story points (completed as estimated)
+**Test Coverage**: 37 tests (26 unit + 11 integration)
+**Status**: Ready for Epic 4 (Deployment)
diff --git a/STORY_3.4_QA_SUMMARY.md b/STORY_3.4_QA_SUMMARY.md
new file mode 100644
index 0000000..ed5ce96
--- /dev/null
+++ b/STORY_3.4_QA_SUMMARY.md
@@ -0,0 +1,49 @@
+# Story 3.4 QA Summary
+
+## Status: PASSED - Ready for Production
+
+## Test Results
+- **37/37 tests passing** (26 unit + 11 integration)
+- **0 failures**
+- **0 linter errors** in new code
+- **Database migration verified**
+
+## Key Findings
+
+### Strengths
+1. **Complete test coverage** - All functionality tested at unit and integration level
+2. **Clean implementation** - No code quality issues, proper error handling
+3. **Backward compatible** - Optional parameters don't break existing code
+4. **Well documented** - Clear docstrings and comprehensive documentation
+5. **Database integrity** - Proper indexes, constraints, and foreign keys
+
+### Acceptance Criteria
+All acceptance criteria verified:
+- Generates 3 pages (about, contact, privacy) per site
+- Uses same template as site articles
+- Stores in database with proper associations
+- Integrates with site provisioning
+- Backfill script available with dry-run mode
+- Graceful error handling
+
+### Implementation Quality
+- **Design patterns:** Repository pattern, dependency injection
+- **Code organization:** Modular with clear separation of concerns
+- **Performance:** ~1-2 seconds per site (acceptable)
+- **Scalability:** Can handle hundreds of sites via backfill
+
+## Minor Notes
+- SQLAlchemy deprecation warnings (96) about `datetime.utcnow()` - not related to Story 3.4
+- Markdown linter warnings - pre-existing style issues, not functional problems
+
+## Recommendation
+**APPROVED for production**. Story 3.4 is complete, tested, and ready for deployment.
+
+## Next Steps
+1. Story status updated to "QA COMPLETE"
+2. Ready for Epic 4 (Deployment)
+3. Consider addressing SQLAlchemy deprecation warnings in future sprint
+
+---
+QA completed: October 22, 2025
+
diff --git a/docs/stories/story-3.4-boilerplate-site-pages.md b/docs/stories/story-3.4-boilerplate-site-pages.md
index 9e31cec..6e84c08 100644
--- a/docs/stories/story-3.4-boilerplate-site-pages.md
+++ b/docs/stories/story-3.4-boilerplate-site-pages.md
@@ -1,7 +1,7 @@
# Story 3.4: Generate Boilerplate Site Pages
## Status
-Not Started
+**QA COMPLETE** - Ready for Production
## Story
**As a developer**, I want to automatically generate boilerplate `about.html`, `contact.html`, and `privacy.html` pages for each site in my batch, so that the navigation menu links from Story 3.3 work and the sites appear complete.
diff --git a/scripts/backfill_site_pages.py b/scripts/backfill_site_pages.py
index 9b86fa9..abab72c 100644
--- a/scripts/backfill_site_pages.py
+++ b/scripts/backfill_site_pages.py
@@ -10,11 +10,11 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
-from src.database.connection import DatabaseConnection
-from src.database.repositories import SiteDeploymentRepository, SitePageRepository
+from src.database.session import db_manager
+from src.database.repositories import SiteDeploymentRepository, SitePageRepository, UserRepository
from src.templating.service import TemplateService
from src.generation.site_page_generator import generate_site_pages
-from src.auth.auth_service import AuthService
+from src.auth.password import verify_password
logging.basicConfig(
level=logging.INFO,
@@ -26,7 +26,6 @@ logger = logging.getLogger(__name__)
def backfill_site_pages(
username: str,
password: str,
- template: str = "basic",
dry_run: bool = False,
batch_size: int = 100
):
@@ -36,18 +35,23 @@ def backfill_site_pages(
Args:
username: Admin username for authentication
password: Admin password
- template: Template to use (default: basic)
dry_run: If True, only preview changes without applying
batch_size: Number of sites to process between progress updates
"""
- db = DatabaseConnection()
- session = db.get_session()
+ db_manager.initialize()
+ session = db_manager.get_session()
- auth_service = AuthService(session)
- user = auth_service.authenticate(username, password)
+ user_repo = UserRepository(session)
+ user = user_repo.get_by_username(username)
- if not user or not user.is_admin():
- logger.error("Authentication failed or insufficient permissions")
+ if not user or not verify_password(password, user.hashed_password):
+ logger.error("Authentication failed")
+ session.close()
+ sys.exit(1)
+
+ if not user.is_admin():
+ logger.error("Insufficient permissions - admin required")
+ session.close()
sys.exit(1)
logger.info("Authenticated as admin user")
@@ -89,7 +93,7 @@ def backfill_site_pages(
if missing_types:
logger.info(f"[{idx}/{len(sites_needing_pages)}] Generating pages for site {site.id} ({domain})")
- generate_site_pages(site, template, page_repo, template_service)
+ generate_site_pages(site, page_repo, template_service)
successful += 1
else:
logger.info(f"[{idx}/{len(sites_needing_pages)}] Site {site.id} already has all pages, skipping")
@@ -108,6 +112,7 @@ def backfill_site_pages(
raise
finally:
session.close()
+ db_manager.close()
def main():
@@ -124,12 +129,6 @@ def main():
required=True,
help="Admin password"
)
- parser.add_argument(
- "--template",
- default="basic",
- choices=["basic", "modern", "classic", "minimal"],
- help="Template to use for pages (default: basic)"
- )
parser.add_argument(
"--dry-run",
action="store_true",
@@ -147,7 +146,6 @@ def main():
backfill_site_pages(
username=args.username,
password=args.password,
- template=args.template,
dry_run=args.dry_run,
batch_size=args.batch_size
)
diff --git a/src/database/interfaces.py b/src/database/interfaces.py
index e41baf1..800cf76 100644
--- a/src/database/interfaces.py
+++ b/src/database/interfaces.py
@@ -4,7 +4,7 @@ Abstract repository interfaces for data access layer
from abc import ABC, abstractmethod
from typing import Optional, List, Dict, Any
-from src.database.models import User, SiteDeployment, Project, GeneratedContent, ArticleLink
+from src.database.models import User, SiteDeployment, Project, GeneratedContent, ArticleLink, SitePage
class IUserRepository(ABC):
@@ -211,3 +211,37 @@ class IArticleLinkRepository(ABC):
def delete(self, link_id: int) -> bool:
"""Delete an article link by ID"""
pass
+
+
+class ISitePageRepository(ABC):
+ """Interface for SitePage data access"""
+
+ @abstractmethod
+ def create(self, site_deployment_id: int, page_type: str, content: str) -> SitePage:
+ """Create a new site page"""
+ pass
+
+ @abstractmethod
+ def get_by_site(self, site_deployment_id: int) -> List[SitePage]:
+ """Get all pages for a site"""
+ pass
+
+ @abstractmethod
+ def get_by_site_and_type(self, site_deployment_id: int, page_type: str) -> Optional[SitePage]:
+ """Get a specific page for a site"""
+ pass
+
+ @abstractmethod
+ def update_content(self, page_id: int, content: str) -> SitePage:
+ """Update page content"""
+ pass
+
+ @abstractmethod
+ def exists(self, site_deployment_id: int, page_type: str) -> bool:
+ """Check if a page exists for a site"""
+ pass
+
+ @abstractmethod
+ def delete(self, page_id: int) -> bool:
+ """Delete a site page by ID"""
+ pass
diff --git a/src/database/models.py b/src/database/models.py
index 40035ee..dd2ec58 100644
--- a/src/database/models.py
+++ b/src/database/models.py
@@ -4,8 +4,8 @@ SQLAlchemy database models
from datetime import datetime, timezone
from typing import Optional
-from sqlalchemy import String, Integer, DateTime, Float, ForeignKey, JSON, Text
-from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
+from sqlalchemy import String, Integer, DateTime, Float, ForeignKey, JSON, Text, UniqueConstraint
+from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
class Base(DeclarativeBase):
@@ -174,3 +174,32 @@ class ArticleLink(Base):
def __repr__(self) -> str:
target = f"content_id={self.to_content_id}" if self.to_content_id else f"url={self.to_url}"
return f""
+
+
+class SitePage(Base):
+ """Boilerplate pages for sites (about, contact, privacy)"""
+ __tablename__ = "site_pages"
+
+ id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
+ site_deployment_id: Mapped[int] = mapped_column(
+ Integer,
+ ForeignKey('site_deployments.id', ondelete='CASCADE'),
+ nullable=False,
+ index=True
+ )
+ page_type: Mapped[str] = mapped_column(String(20), nullable=False, index=True)
+ content: Mapped[str] = mapped_column(Text, nullable=False)
+ created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, nullable=False)
+ updated_at: Mapped[datetime] = mapped_column(
+ DateTime,
+ default=datetime.utcnow,
+ onupdate=datetime.utcnow,
+ nullable=False
+ )
+
+ __table_args__ = (
+ UniqueConstraint('site_deployment_id', 'page_type', name='uq_site_page_type'),
+ )
+
+ def __repr__(self) -> str:
+ return f""
diff --git a/src/database/repositories.py b/src/database/repositories.py
index 9bcbe1b..87e5e73 100644
--- a/src/database/repositories.py
+++ b/src/database/repositories.py
@@ -6,8 +6,8 @@ from typing import Optional, List, Dict, Any
from sqlalchemy.orm import Session
from sqlalchemy.exc import IntegrityError
from src.core.config import get_config
-from src.database.interfaces import IUserRepository, ISiteDeploymentRepository, IProjectRepository, IArticleLinkRepository
-from src.database.models import User, SiteDeployment, Project, GeneratedContent, ArticleLink
+from src.database.interfaces import IUserRepository, ISiteDeploymentRepository, IProjectRepository, IArticleLinkRepository, ISitePageRepository
+from src.database.models import User, SiteDeployment, Project, GeneratedContent, ArticleLink, SitePage
class UserRepository(IUserRepository):
@@ -574,3 +574,80 @@ class ArticleLinkRepository(IArticleLinkRepository):
self.session.commit()
return True
return False
+
+
+class SitePageRepository(ISitePageRepository):
+ """Repository for SitePage data access"""
+
+ def __init__(self, session: Session):
+ self.session = session
+
+ def create(self, site_deployment_id: int, page_type: str, content: str) -> SitePage:
+ """
+ Create a new site page
+
+ Args:
+ site_deployment_id: Site deployment ID
+ page_type: Type of page (about, contact, privacy)
+ content: Full HTML content for the page
+
+ Returns:
+ The created SitePage object
+
+ Raises:
+ ValueError: If page already exists for this site and type
+ """
+ page = SitePage(
+ site_deployment_id=site_deployment_id,
+ page_type=page_type,
+ content=content
+ )
+
+ try:
+ self.session.add(page)
+ self.session.commit()
+ self.session.refresh(page)
+ return page
+ except IntegrityError:
+ self.session.rollback()
+ raise ValueError(f"Page '{page_type}' already exists for site {site_deployment_id}")
+
+ def get_by_site(self, site_deployment_id: int) -> List[SitePage]:
+ """Get all pages for a site"""
+ return self.session.query(SitePage).filter(
+ SitePage.site_deployment_id == site_deployment_id
+ ).all()
+
+ def get_by_site_and_type(self, site_deployment_id: int, page_type: str) -> Optional[SitePage]:
+ """Get a specific page for a site"""
+ return self.session.query(SitePage).filter(
+ SitePage.site_deployment_id == site_deployment_id,
+ SitePage.page_type == page_type
+ ).first()
+
+ def update_content(self, page_id: int, content: str) -> SitePage:
+ """Update page content"""
+ page = self.session.query(SitePage).filter(SitePage.id == page_id).first()
+ if not page:
+ raise ValueError(f"Page with ID {page_id} not found")
+
+ page.content = content
+ self.session.commit()
+ self.session.refresh(page)
+ return page
+
+ def exists(self, site_deployment_id: int, page_type: str) -> bool:
+ """Check if a page exists for a site"""
+ return self.session.query(SitePage).filter(
+ SitePage.site_deployment_id == site_deployment_id,
+ SitePage.page_type == page_type
+ ).first() is not None
+
+ def delete(self, page_id: int) -> bool:
+ """Delete a site page by ID"""
+ page = self.session.query(SitePage).filter(SitePage.id == page_id).first()
+ if page:
+ self.session.delete(page)
+ self.session.commit()
+ return True
+ return False
diff --git a/src/generation/site_assignment.py b/src/generation/site_assignment.py
index d254041..2126c28 100644
--- a/src/generation/site_assignment.py
+++ b/src/generation/site_assignment.py
@@ -6,8 +6,9 @@ import logging
import random
from typing import List, Set, Optional
from src.database.models import GeneratedContent, SiteDeployment
-from src.database.repositories import SiteDeploymentRepository
+from src.database.repositories import SiteDeploymentRepository, SitePageRepository
from src.deployment.bunnynet import BunnyNetClient
+from src.templating.service import TemplateService
from src.generation.job_config import Job
from src.generation.site_provisioning import (
provision_keyword_sites,
@@ -49,7 +50,9 @@ def assign_sites_to_batch(
site_repo: SiteDeploymentRepository,
bunny_client: BunnyNetClient,
project_keyword: str,
- region: str = "DE"
+ region: str = "DE",
+ page_repo: Optional[SitePageRepository] = None,
+ template_service: Optional[TemplateService] = None
) -> None:
"""
Assign sites to all articles in a batch based on job config and priority rules
@@ -65,6 +68,8 @@ def assign_sites_to_batch(
bunny_client: BunnyNetClient for creating sites if needed
project_keyword: Main keyword from project (for generic site names)
region: Storage region for new sites (default: DE)
+ page_repo: Optional SitePageRepository for generating boilerplate pages
+ template_service: Optional TemplateService for generating pages
Raises:
ValueError: If insufficient sites and auto_create_sites is False
@@ -79,7 +84,9 @@ def assign_sites_to_batch(
keywords=job.create_sites_for_keywords,
bunny_client=bunny_client,
site_repo=site_repo,
- region=region
+ region=region,
+ page_repo=page_repo,
+ template_service=template_service
)
# Step 2: Query all available sites
@@ -170,7 +177,9 @@ def assign_sites_to_batch(
project_keyword=project_keyword,
bunny_client=bunny_client,
site_repo=site_repo,
- region=region
+ region=region,
+ page_repo=page_repo,
+ template_service=template_service
)
for content, site in zip(unassigned, new_sites):
diff --git a/src/generation/site_page_generator.py b/src/generation/site_page_generator.py
index fc4b38d..47399c2 100644
--- a/src/generation/site_page_generator.py
+++ b/src/generation/site_page_generator.py
@@ -1,92 +1,96 @@
"""
-Site page generator for boilerplate pages (about, contact, privacy)
+Site page generator for creating boilerplate pages (about, contact, privacy)
"""
import logging
from typing import List
from src.database.models import SiteDeployment, SitePage
-from src.database.interfaces import ISitePageRepository
+from src.database.repositories import SitePageRepository
from src.templating.service import TemplateService
from src.generation.page_templates import get_page_content
logger = logging.getLogger(__name__)
-PAGE_TYPES = ["about", "contact", "privacy"]
-
-
def get_domain_from_site(site_deployment: SiteDeployment) -> str:
"""
- Extract domain from site deployment
+ Extract domain from site deployment for use in page content
Args:
- site_deployment: Site deployment object
-
+ site_deployment: SiteDeployment record
+
Returns:
- Domain string (custom hostname or bcdn hostname)
+ Domain string (custom hostname or b-cdn hostname)
"""
- return site_deployment.custom_hostname or site_deployment.pull_zone_bcdn_hostname
+ if site_deployment.custom_hostname:
+ return site_deployment.custom_hostname
+ else:
+ return site_deployment.pull_zone_bcdn_hostname
def generate_site_pages(
site_deployment: SiteDeployment,
- template_name: str,
- page_repo: ISitePageRepository,
+ page_repo: SitePageRepository,
template_service: TemplateService
) -> List[SitePage]:
"""
- Generate all boilerplate pages for a site
+ Generate boilerplate pages for a site (about, contact, privacy)
Args:
- site_deployment: Site deployment to generate pages for
- template_name: Template to use (basic, modern, classic, minimal)
- page_repo: Repository for storing pages
- template_service: Service for applying templates
-
+ site_deployment: SiteDeployment record
+ page_repo: SitePageRepository for database operations
+ template_service: TemplateService for applying HTML templates
+
Returns:
- List of created SitePage objects
-
+ List of created SitePage records
+
Raises:
- ValueError: If page generation fails
+ ValueError: If pages already exist for this site
+ Exception: If page generation fails
"""
domain = get_domain_from_site(site_deployment)
+ template_name = site_deployment.template_name or "basic"
+ page_types = ["about", "contact", "privacy"]
+
+ logger.info(f"Generating boilerplate pages for site {site_deployment.id} ({domain}) with template '{template_name}'")
+
created_pages = []
- logger.info(f"Generating boilerplate pages for site {site_deployment.id} ({domain})")
-
- for page_type in PAGE_TYPES:
+ for page_type in page_types:
+ if page_repo.exists(site_deployment.id, page_type):
+ logger.warning(f"Page '{page_type}' already exists for site {site_deployment.id}, skipping")
+ continue
+
try:
- if page_repo.exists(site_deployment.id, page_type):
- logger.info(f"Page {page_type} already exists for site {site_deployment.id}, skipping")
- continue
+ page_content = get_page_content(page_type, domain)
- raw_content = get_page_content(page_type, domain)
-
- page_title = {
+ page_title_map = {
"about": "About Us",
"contact": "Contact",
"privacy": "Privacy Policy"
- }.get(page_type, page_type.title())
+ }
+ page_title = page_title_map.get(page_type, page_type.title())
- formatted_html = template_service.format_page(
- content=raw_content,
- page_title=page_title,
+ full_html = template_service.format_content(
+ content=page_content,
+ title=page_title,
+ meta_description=f"{page_title} - {domain}",
template_name=template_name
)
page = page_repo.create(
site_deployment_id=site_deployment.id,
page_type=page_type,
- content=formatted_html
+ content=full_html
)
created_pages.append(page)
- logger.info(f"Created {page_type} page for site {site_deployment.id}")
+ logger.info(f" Created {page_type}.html (page_id: {page.id})")
except Exception as e:
- logger.error(f"Failed to create {page_type} page for site {site_deployment.id}: {e}")
- raise ValueError(f"Page generation failed for {page_type}: {e}")
+ logger.error(f"Failed to generate {page_type} page for site {site_deployment.id}: {e}")
+ raise
logger.info(f"Successfully created {len(created_pages)} pages for site {site_deployment.id}")
+
return created_pages
-
diff --git a/src/generation/site_provisioning.py b/src/generation/site_provisioning.py
index f0fd680..4bbc665 100644
--- a/src/generation/site_provisioning.py
+++ b/src/generation/site_provisioning.py
@@ -8,8 +8,10 @@ import string
import re
from typing import List, Dict, Optional
from src.deployment.bunnynet import BunnyNetClient, BunnyNetAPIError
-from src.database.repositories import SiteDeploymentRepository
+from src.database.repositories import SiteDeploymentRepository, SitePageRepository
from src.database.models import SiteDeployment
+from src.templating.service import TemplateService
+from src.generation.site_page_generator import generate_site_pages
logger = logging.getLogger(__name__)
@@ -32,7 +34,9 @@ def create_bunnynet_site(
name_prefix: str,
bunny_client: BunnyNetClient,
site_repo: SiteDeploymentRepository,
- region: str = "DE"
+ region: str = "DE",
+ page_repo: Optional[SitePageRepository] = None,
+ template_service: Optional[TemplateService] = None
) -> SiteDeployment:
"""
Create a bunny.net site (Storage Zone + Pull Zone) without custom domain
@@ -42,6 +46,8 @@ def create_bunnynet_site(
bunny_client: Initialized BunnyNetClient
site_repo: SiteDeploymentRepository for saving to database
region: Storage region code (default: DE)
+ page_repo: Optional SitePageRepository for generating boilerplate pages
+ template_service: Optional TemplateService for generating pages
Returns:
Created SiteDeployment record
@@ -76,6 +82,14 @@ def create_bunnynet_site(
logger.info(f" Saved to database (site_id: {site.id})")
+ if page_repo and template_service:
+ logger.info(f" Generating boilerplate pages for site {site.id}...")
+ try:
+ generate_site_pages(site, page_repo, template_service)
+ logger.info(f" Successfully created about, contact, privacy pages for site {site.id}")
+ except Exception as e:
+ logger.warning(f" Failed to generate pages for site {site.id}: {e}")
+
return site
@@ -83,7 +97,9 @@ def provision_keyword_sites(
keywords: List[Dict[str, any]],
bunny_client: BunnyNetClient,
site_repo: SiteDeploymentRepository,
- region: str = "DE"
+ region: str = "DE",
+ page_repo: Optional[SitePageRepository] = None,
+ template_service: Optional[TemplateService] = None
) -> List[SiteDeployment]:
"""
Pre-create sites for specific keywords/entities
@@ -93,6 +109,8 @@ def provision_keyword_sites(
bunny_client: Initialized BunnyNetClient
site_repo: SiteDeploymentRepository for saving to database
region: Storage region code (default: DE)
+ page_repo: Optional SitePageRepository for generating boilerplate pages
+ template_service: Optional TemplateService for generating pages
Returns:
List of created SiteDeployment records
@@ -123,7 +141,9 @@ def provision_keyword_sites(
name_prefix=slug_prefix,
bunny_client=bunny_client,
site_repo=site_repo,
- region=region
+ region=region,
+ page_repo=page_repo,
+ template_service=template_service
)
created_sites.append(site)
@@ -141,9 +161,11 @@ def create_generic_sites(
project_keyword: str,
bunny_client: BunnyNetClient,
site_repo: SiteDeploymentRepository,
- region: str = "DE"
+ region: str = "DE",
+ page_repo: Optional[SitePageRepository] = None,
+ template_service: Optional[TemplateService] = None
) -> List[SiteDeployment]:
- """
+ """
Create generic sites for a project (used when auto_create_sites is enabled)
Args:
@@ -152,6 +174,8 @@ def create_generic_sites(
bunny_client: Initialized BunnyNetClient
site_repo: SiteDeploymentRepository for saving to database
region: Storage region code (default: DE)
+ page_repo: Optional SitePageRepository for generating boilerplate pages
+ template_service: Optional TemplateService for generating pages
Returns:
List of created SiteDeployment records
@@ -167,7 +191,9 @@ def create_generic_sites(
name_prefix=slug_prefix,
bunny_client=bunny_client,
site_repo=site_repo,
- region=region
+ region=region,
+ page_repo=page_repo,
+ template_service=template_service
)
created_sites.append(site)
diff --git a/tests/integration/test_site_page_integration.py b/tests/integration/test_site_page_integration.py
index ae3856e..961aeef 100644
--- a/tests/integration/test_site_page_integration.py
+++ b/tests/integration/test_site_page_integration.py
@@ -12,24 +12,28 @@ from src.generation.site_page_generator import generate_site_pages
@pytest.fixture
-def test_db():
- """Create a test database"""
- engine = create_engine("sqlite:///:memory:")
+def test_engine():
+ engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)
- Session = sessionmaker(bind=engine)
+ return engine
+
+
+@pytest.fixture
+def test_session(test_engine):
+ Session = sessionmaker(bind=test_engine)
session = Session()
yield session
session.close()
@pytest.fixture
-def site_repo(test_db):
- return SiteDeploymentRepository(test_db)
+def site_repo(test_session):
+ return SiteDeploymentRepository(test_session)
@pytest.fixture
-def page_repo(test_db):
- return SitePageRepository(test_db)
+def page_repo(test_session):
+ return SitePageRepository(test_session)
@pytest.fixture
@@ -38,119 +42,180 @@ def template_service():
@pytest.fixture
-def sample_site(site_repo):
- """Create a sample site for testing"""
- return site_repo.create(
- site_name="integration-test-site",
- storage_zone_id=999,
+def test_site(site_repo):
+ site = site_repo.create(
+ site_name="test-site",
+ storage_zone_id=12345,
storage_zone_name="test-storage",
- storage_zone_password="test-password",
+ storage_zone_password="password123",
storage_zone_region="DE",
- pull_zone_id=888,
- pull_zone_bcdn_hostname="integration-test.b-cdn.net",
+ pull_zone_id=67890,
+ pull_zone_bcdn_hostname="test-site.b-cdn.net",
custom_hostname=None
)
+ return site
-class TestSitePageIntegration:
- """Integration tests for site page generation flow"""
+def test_generate_pages_for_site(test_site, page_repo, template_service):
+ pages = generate_site_pages(test_site, page_repo, template_service)
- def test_full_page_generation_flow(self, sample_site, page_repo, template_service):
- """Test complete flow of generating pages for a site"""
- pages = generate_site_pages(sample_site, "basic", page_repo, template_service)
-
- assert len(pages) == 3
-
- stored_pages = page_repo.get_by_site(sample_site.id)
- assert len(stored_pages) == 3
-
- page_types = {p.page_type for p in stored_pages}
- assert page_types == {"about", "contact", "privacy"}
+ assert len(pages) == 3
- def test_pages_use_correct_template(self, sample_site, page_repo, template_service):
- """Test that pages are formatted with correct template"""
- generate_site_pages(sample_site, "modern", page_repo, template_service)
-
- about_page = page_repo.get_by_site_and_type(sample_site.id, "about")
- assert about_page is not None
- assert "" in content or "" in content.lower()
- assert "" in content.lower()
- assert "