bryanb
/
Big-Link-Man
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Big-Link-Man/docs/stories/story-3.4-boilerplate-site-...

518 lines
20 KiB
Markdown
Raw Blame History

# Story 3.4: Generate Boilerplate Site Pages
## Status
**QA COMPLETE** - Ready for Production
## Story
**As a developer**, I want to automatically generate boilerplate `about.html`, `contact.html`, and `privacy.html` pages for each site in my batch, so that the navigation menu links from Story 3.3 work and the sites appear complete.
## Context
- Story 3.3 added navigation menus to all HTML templates with links to:
- `/index.html` (homepage)
- `about.html` (about page)
- `privacy.html` (privacy policy)
- `contact.html` (contact page)
- Currently, these pages don't exist, creating broken links
- Each site needs its own set of these pages
- Pages should use the same template as the articles (basic/modern/classic/minimal)
- Content should be generic but professional enough for a real site
- Privacy policy needs to be comprehensive and legally sound (generic template)
## Acceptance Criteria
### Core Functionality
- A function generates the three boilerplate pages for a given site
- Pages are created AFTER articles are generated but BEFORE deployment
- Each page uses the same template as the articles for that site
- Pages are stored in the database for deployment
- Pages are associated with the correct site (via `site_deployment_id`)
### Page Content Requirements
#### About Page (`about.html`)
- Empty page with just the template applied
- No content text required (just template navigation/structure)
- User can add content later if needed
#### Contact Page (`contact.html`)
- Empty page with just the template applied
- No content text required (just template navigation/structure)
- User can add content later if needed
#### Privacy Policy (`privacy.html`)
- **Option 1 (Minimal):** Empty page like about/contact
- No content text required (just template navigation/structure)
- User can add content later if needed
**Decision:** Start with Option 1 (empty pages) for all three pages. Privacy policy content can be added later via backfill update or manual edit if needed.
### Template Integration
- Use same template engine as article content (`src/templating/service.py`)
- Read template from `site.template_name` field in database
- Pages use same template as articles on the same site (consistent look)
- Include navigation menu (which will link to these same pages)
### Database Storage
- Create new `site_pages` table (clean separation from articles):
- `id`, `site_deployment_id`, `page_type`, `content`, `created_at`, `updated_at`
- Foreign key to `site_deployments` with CASCADE delete
- Unique constraint on (site_deployment_id, page_type)
- Indexes on site_deployment_id and page_type
- Each site can have one of each page type (about, contact, privacy)
- Pages are fundamentally different from articles, deserve own table
### URL Generation
- Pages use simple filenames: `about.html`, `contact.html`, `privacy.html`
- Full URLs: `https://{hostname}/about.html`
- No slug generation needed (fixed filenames)
- Pages tracked separately from article URLs
### Integration Point
- Hook into batch generation workflow in `src/generation/batch_processor.py`
- After site assignment (Story 3.1) and before deployment (Epic 4)
- Generate pages ONLY for newly created sites (not existing sites)
- One-time backfill script to add pages to all existing imported sites
### Two Use Cases
1. **One-time backfill**: Script to generate pages for all existing sites in database (hundreds of sites)
2. **Ongoing generation**: Automatically generate pages only when new sites are created (provision-site, auto_create_sites, etc.)
## Tasks / Subtasks
### 1. Create SitePage Database Table
**Effort:** 2 story points
- [ ] Create new `site_pages` table with schema:
- `id`, `site_deployment_id`, `page_type`, `content`, `created_at`, `updated_at`
- [ ] Add `SitePage` model to `src/database/models.py`
- [ ] Create migration script `scripts/migrate_add_site_pages.py`
- [ ] Add unique constraint on (site_deployment_id, page_type)
- [ ] Add indexes on site_deployment_id and page_type
- [ ] Add CASCADE delete (if site deleted, pages deleted)
- [ ] Test migration on development database
### 2. Create SitePage Repository
**Effort:** 2 story points
- [ ] Create `ISitePageRepository` interface in `src/database/interfaces.py`:
- `create(site_deployment_id, page_type, content) -> SitePage`
- `get_by_site(site_deployment_id) -> List[SitePage]`
- `get_by_site_and_type(site_deployment_id, page_type) -> Optional[SitePage]`
- `update_content(page_id, content) -> SitePage`
- `exists(site_deployment_id, page_type) -> bool`
- `delete(page_id) -> bool`
- [ ] Implement `SitePageRepository` in `src/database/repositories.py`
- [ ] Add to repository factory/dependency injection
### 3. Create Page Content Templates (SIMPLIFIED)
**Effort:** 1 story point (reduced from 3)
- [ ] Create `src/generation/page_templates.py` module
- [ ] Implement `get_page_content(page_type: str, domain: str) -> str`:
- Returns just a heading: `<h1>About Us</h1>`, `<h1>Contact</h1>`, `<h1>Privacy Policy</h1>`
- All three pages use same heading-only approach
- No other content text
- [ ] No need for extensive content generation
- [ ] Pages are just placeholders until user adds content manually
### 4. Implement Page Generation Logic (SIMPLIFIED)
**Effort:** 2 story points (reduced from 3)
- [ ] Create `src/generation/site_page_generator.py` module
- [ ] Implement `generate_site_pages(site_deployment: SiteDeployment, template_name: str, page_repo, template_service) -> List[SitePage]`:
- Get domain from site (custom_hostname or bcdn_hostname)
- For each page type (about, contact, privacy):
- Get heading-only content from `page_templates.py`
- Wrap heading in HTML template using `template_service`
- Store page in database
- Return list of created pages
- [ ] Pages have just heading (e.g., `<h1>About Us</h1>`) wrapped in template
- [ ] Log page generation at INFO level
### 5. Integrate with Site Creation (Not Batch Processor)
**Effort:** 2 story points
- [ ] Update `src/generation/site_provisioning.py`:
- After creating new site via bunny.net API, generate boilerplate pages
- Call `generate_site_pages()` immediately after site creation
- Log page generation results
- [ ] Update `provision-site` CLI command:
- Generate pages after site is provisioned
- [ ] Handle errors gracefully (log warning if page generation fails, continue with site creation)
- [ ] **DO NOT generate pages in batch processor** (only for new sites, not existing sites)
### 6. Update Template Service (No Changes Needed)
**Effort:** 0 story points
- [x] Template service already handles simple content
- [x] Just pass heading HTML through existing `format_content()` method
- [x] No changes needed to template service
### 7. Create Backfill Script for Existing Sites
**Effort:** 2 story points
- [ ] Create `scripts/backfill_site_pages.py`:
- Query all sites in database that don't have pages
- For each site: generate about, contact, privacy pages
- Use default template (or infer from site name if possible)
- Progress reporting (e.g., "Generating pages for site 50/400...")
- Dry-run mode to preview changes
- CLI arguments: `--dry-run`, `--template`, `--batch-size`
- [ ] Add error handling for individual site failures (continue with next site)
- [ ] Log results: successful, failed, skipped counts
### 8. Homepage Generation (Optional - Deferred)
**Effort:** 2 story points (if implemented)
- [ ] **DEFER to Epic 4 or later**
- [ ] Homepage (`index.html`) requires knowing all articles on the site
- [ ] Not needed for Story 3.4 (navigation menu links to `/index.html` can 404 for now)
- [ ] Document in technical notes
### 9. Unit Tests (SIMPLIFIED)
**Effort:** 2 story points (reduced from 3)
- [ ] Test heading-only page content generation
- [ ] Test domain extraction from SiteDeployment (custom vs bcdn hostname)
- [ ] Test page HTML wrapping with each template type
- [ ] Test SitePage repository CRUD operations
- [ ] Test duplicate page prevention (unique constraint)
- [ ] Test page generation for single site
- [ ] Test backfill script logic
- [ ] Mock template service and repositories
- [ ] Achieve >80% code coverage for new modules
### 10. Integration Tests (SIMPLIFIED)
**Effort:** 1 story point (reduced from 2)
- [ ] Test site creation triggers page generation
- [ ] Test with different template types (basic, modern, classic, minimal)
- [ ] Test with custom domain sites vs bunny.net-only sites
- [ ] Test pages stored correctly in database
- [ ] Test backfill script on real database
- [ ] Verify navigation menu links work (pages exist at expected paths)
## Technical Notes
### SitePage Model
```python
class SitePage(Base):
__tablename__ = "site_pages"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
site_deployment_id: Mapped[int] = mapped_column(
Integer,
ForeignKey('site_deployments.id', ondelete='CASCADE'),
nullable=False
)
page_type: Mapped[str] = mapped_column(String(20), nullable=False) # about, contact, privacy, homepage
content: Mapped[str] = mapped_column(Text, nullable=False) # Full HTML
created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, nullable=False)
updated_at: Mapped[datetime] = mapped_column(
DateTime,
default=datetime.utcnow,
onupdate=datetime.utcnow,
nullable=False
)
# Relationships
site_deployment: Mapped["SiteDeployment"] = relationship("SiteDeployment", back_populates="pages")
# Unique constraint
__table_args__ = (
UniqueConstraint('site_deployment_id', 'page_type', name='uq_site_page_type'),
)
```
### Database Migration
```sql
CREATE TABLE site_pages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site_deployment_id INTEGER NOT NULL,
page_type VARCHAR(20) NOT NULL,
content TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (site_deployment_id) REFERENCES site_deployments(id) ON DELETE CASCADE,
UNIQUE (site_deployment_id, page_type)
);
CREATE INDEX idx_site_pages_site ON site_pages(site_deployment_id);
CREATE INDEX idx_site_pages_type ON site_pages(page_type);
```
### Page Content Template Examples (SIMPLIFIED)
#### Implementation - Heading Only
```python
# src/generation/page_templates.py
def get_page_content(page_type: str, domain: str) -> str:
"""
Generate minimal content for boilerplate pages.
Just a heading - no other content text.
"""
page_titles = {
"about": "About Us",
"contact": "Contact",
"privacy": "Privacy Policy"
}
return f"<h1>{page_titles.get(page_type, page_type.title())}</h1>"
```
#### Result - Heading Only Example
```html
<!DOCTYPE html>
<html>
<head>
<title>About Us</title>
<!-- Same template/styling as articles -->
</head>
<body>
<nav>
<ul>
<li><a href="/index.html">Home</a></li>
<li><a href="about.html">About</a></li>
<li><a href="privacy.html">Privacy</a></li>
<li><a href="contact.html">Contact</a></li>
</ul>
</nav>
<main>
<h1>About Us</h1>
<!-- No other content - user can add later if needed -->
</main>
</body>
</html>
```
#### Why Heading-Only Pages Work
1. **Fixes broken nav links** - Pages exist, no 404 errors
2. **Better UX than completely empty** - User sees something when they click the link
3. **User can customize** - Add content manually later for specific sites
4. **Minimal effort** - No need to generate/maintain generic content
5. **Deployment ready** - Pages can be deployed as-is
6. **Future enhancement** - Can add content generation later if needed
### Integration with Site Creation
```python
# In src/generation/site_provisioning.py
def create_bunnynet_site(name_prefix: str, region: str = "DE", template: str = "basic"):
# Step 1: Create Storage Zone
storage = bunny_client.create_storage_zone(...)
# Step 2: Create Pull Zone
pull = bunny_client.create_pull_zone(...)
# Step 3: Save to database
site = site_repo.create(...)
# Step 4: Generate boilerplate pages (NEW - Story 3.4)
logger.info(f"Generating boilerplate pages for new site {site.id}...")
try:
generate_site_pages(site, template, page_repo, template_service)
logger.info(f"Successfully created about, contact, privacy pages for site {site.id}")
except Exception as e:
logger.warning(f"Failed to generate pages for site {site.id}: {e}")
# Don't fail site creation if page generation fails
return site
```
### Backfill Script Usage
```bash
# One-time backfill for all existing sites (dry-run first)
uv run python scripts/backfill_site_pages.py \
--username admin \
--password yourpass \
--template basic \
--dry-run
# Output:
# Found 423 sites without boilerplate pages
# [DRY RUN] Would generate pages for site 1 (www.example.com)
# [DRY RUN] Would generate pages for site 2 (site123.b-cdn.net)
# ...
# [DRY RUN] Total: 423 sites would be updated
# Actually generate pages
uv run python scripts/backfill_site_pages.py \
--username admin \
--password yourpass \
--template basic
# Output:
# Generating pages for site 1/423 (www.example.com)... ✓
# Generating pages for site 2/423 (site123.b-cdn.net)... ✓
# ...
# Complete: 423 successful, 0 failed, 0 skipped
# Use different template per site (default: basic)
uv run python scripts/backfill_site_pages.py \
--username admin \
--password yourpass \
--template modern \
--batch-size 50 # Process 50 sites at a time
```
### Page URL Structure
```
Homepage: https://example.com/index.html
About: https://example.com/about.html
Contact: https://example.com/contact.html
Privacy: https://example.com/privacy.html
Article 1: https://example.com/how-to-fix-engines.html
Article 2: https://example.com/engine-maintenance-tips.html
```
### Template Application Example
```python
# For articles (existing)
template_service.apply_template(
content=article.content,
template_name="modern",
title=article.title,
meta_description=article.meta_description,
url=article_url
)
# For pages (new)
template_service.apply_template_to_page(
content=page_content, # Markdown or HTML from page_templates.py
template_name="modern",
page_title="About Us", # Static title
domain=site.custom_hostname or site.pull_zone_bcdn_hostname
)
```
### Backfill Script Implementation
```python
# scripts/backfill_site_pages.py
def backfill_site_pages(
page_repo,
site_repo,
template_service,
template: str = "basic",
dry_run: bool = False,
batch_size: int = 100
):
"""Generate boilerplate pages for all sites that don't have them"""
# Get all sites
all_sites = site_repo.get_all()
logger.info(f"Found {len(all_sites)} total sites in database")
# Filter to sites without pages
sites_needing_pages = []
for site in all_sites:
existing_pages = page_repo.get_by_site(site.id)
if len(existing_pages) < 3: # Should have about, contact, privacy
sites_needing_pages.append(site)
logger.info(f"Found {len(sites_needing_pages)} sites without boilerplate pages")
if dry_run:
for site in sites_needing_pages:
domain = site.custom_hostname or site.pull_zone_bcdn_hostname
logger.info(f"[DRY RUN] Would generate pages for site {site.id} ({domain})")
logger.info(f"[DRY RUN] Total: {len(sites_needing_pages)} sites would be updated")
return
# Generate pages for each site
successful = 0
failed = 0
for idx, site in enumerate(sites_needing_pages, 1):
domain = site.custom_hostname or site.pull_zone_bcdn_hostname
logger.info(f"Generating pages for site {idx}/{len(sites_needing_pages)} ({domain})...")
try:
generate_site_pages(site, template, page_repo, template_service)
successful += 1
except Exception as e:
logger.error(f"Failed to generate pages for site {site.id}: {e}")
failed += 1
# Progress checkpoint every batch_size sites
if idx % batch_size == 0:
logger.info(f"Progress: {idx}/{len(sites_needing_pages)} sites processed")
logger.info(f"Complete: {successful} successful, {failed} failed")
```
### Domain Extraction
```python
def get_domain_from_site(site_deployment: SiteDeployment) -> str:
"""Extract domain for use in page content (email addresses, etc.)"""
if site_deployment.custom_hostname:
return site_deployment.custom_hostname
else:
return site_deployment.pull_zone_bcdn_hostname
```
### Privacy Policy Legal Note
The privacy policy template should be:
- Generic enough to apply to blog/content sites
- Comprehensive enough to cover common scenarios (cookies, analytics, third-party links)
- NOT legal advice - users should consult a lawyer for specific requirements
- Include standard disclaimers
- Regularly reviewed and updated (document version/date)
Recommended approach: Use a well-tested generic template from a reputable source (e.g., Privacy Policy Generator) and adapt it to fit our template structure.
## Dependencies
- Story 3.1: Site assignment must be complete (need to know which sites are in use)
- Story 3.3: Navigation menu is already in templates (pages fulfill those links)
- Story 2.4: Template service exists and can apply HTML templates
- Story 1.6: SiteDeployment table exists
## Future Considerations
- Story 4.1 will deploy these pages along with articles
- Future: Custom page content per project (override generic templates)
- Future: Homepage generation with dynamic article listing
- Future: Allow users to edit boilerplate page content via CLI or web interface
- Future: Additional pages (terms of service, disclaimer, etc.)
- Future: Page templates with more customization options (site name, tagline, etc.)
## Deferred to Later
- **Homepage (`index.html`) generation** - Could be part of this story or deferred to Epic 4
- If generated here: Simple page listing all articles on the site
- If deferred: Epic 4 deployment could create a basic redirect or placeholder
- **Custom page content per project** - Allow projects to override default templates
- **Multi-language support** - Generate pages in different languages based on project settings
## Total Effort
14 story points (reduced from 20 due to heading-only simplification and no template service changes)
### Effort Breakdown
1. Database Schema (2 points) - site_pages table only
2. Repository Layer (2 points) - SitePageRepository
3. Page Content Templates (1 point) - heading-only
4. Generation Logic (2 points) - reads site.template_name from DB
5. Site Creation Integration (2 points)
6. Template Service Updates (0 points) - no changes needed
7. Backfill Script (2 points)
8. Homepage Generation (deferred)
9. Unit Tests (2 points)
10. Integration Tests (1 point)
**Total: 14 story points**
### Effort Reduction
Original estimate: 20 story points (with full page content)
Simplified (heading-only pages): 15 story points
Savings: 5 story points (no complex content generation needed)
## Notes
- Pages should be visually consistent with articles (same template)
- **Pages have heading only** - just `<h1>` tag, no body content
- Better UX than completely empty (user sees page title when they click nav link)
- User can manually add content later for specific sites if desired
- Pages are generated once per site at creation time
- Future enhancement: Add content generation for privacy policy if legally required
- Future enhancement: CLI command to update page content for specific sites
Reference in New Issue View Git Blame Copy Permalink