Story 4.1 coded - pre real testing
parent
a17ec02deb
commit
8a382a1db4
|
|
@ -0,0 +1,188 @@
|
|||
# Story 4.1 Implementation Summary
|
||||
|
||||
## Status: COMPLETE
|
||||
|
||||
## Overview
|
||||
Successfully implemented deployment of generated content to Bunny.net cloud storage with tier-segregated URL logging and automatic deployment after batch generation.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Bunny.net Storage Client (`src/deployment/bunny_storage.py`)
|
||||
- `BunnyStorageClient` class for uploading files to Bunny.net storage zones
|
||||
- Uses per-zone `storage_zone_password` from database for authentication
|
||||
- Implements retry logic with exponential backoff (3 attempts)
|
||||
- Methods:
|
||||
- `upload_file()`: Upload HTML content to storage zone
|
||||
- `file_exists()`: Check if file exists in storage
|
||||
- `list_files()`: List files in storage zone
|
||||
|
||||
### 2. Database Updates
|
||||
- Added `deployed_url` (TEXT, nullable) to `generated_content` table
|
||||
- Added `deployed_at` (TIMESTAMP, nullable, indexed) to `generated_content` table
|
||||
- Created migration script: `scripts/migrate_add_deployment_fields.py`
|
||||
- Added repository methods:
|
||||
- `GeneratedContentRepository.mark_as_deployed()`: Update deployment status
|
||||
- `GeneratedContentRepository.get_deployed_content()`: Query deployed articles
|
||||
|
||||
### 3. URL Logger (`src/deployment/url_logger.py`)
|
||||
- `URLLogger` class for tier-segregated URL logging
|
||||
- Creates daily log files in `deployment_logs/` directory:
|
||||
- `YYYY-MM-DD_tier1_urls.txt` for Tier 1 articles
|
||||
- `YYYY-MM-DD_other_tiers_urls.txt` for Tier 2+ articles
|
||||
- Automatic duplicate prevention by reading existing URLs before appending
|
||||
- Boilerplate pages (about, contact, privacy) are NOT logged
|
||||
|
||||
### 4. URL Generation (`src/generation/url_generator.py`)
|
||||
Extended with new functions:
|
||||
- `generate_public_url()`: Create full HTTPS URL from site + file path
|
||||
- `generate_file_path()`: Generate storage path for articles (slug-based)
|
||||
- `generate_page_file_path()`: Generate storage path for boilerplate pages
|
||||
|
||||
### 5. Deployment Service (`src/deployment/deployment_service.py`)
|
||||
- `DeploymentService` class orchestrates deployment workflow
|
||||
- `deploy_batch()`: Deploy all content for a project
|
||||
- Uploads articles with `formatted_html`
|
||||
- Uploads boilerplate pages (about, contact, privacy)
|
||||
- Logs article URLs to tier-segregated files
|
||||
- Updates database with deployment status and URLs
|
||||
- Returns detailed statistics
|
||||
- `deploy_article()`: Deploy single article
|
||||
- `deploy_boilerplate_page()`: Deploy single boilerplate page
|
||||
- Continues on error by default (configurable)
|
||||
|
||||
### 6. CLI Command (`src/cli/commands.py`)
|
||||
Added `deploy-batch` command:
|
||||
```bash
|
||||
uv run python -m src.cli deploy-batch \
|
||||
--batch-id 123 \
|
||||
--admin-user admin \
|
||||
--admin-password mypass
|
||||
```
|
||||
|
||||
Options:
|
||||
- `--batch-id` (required): Project/batch ID to deploy
|
||||
- `--admin-user` / `--admin-password`: Authentication
|
||||
- `--continue-on-error`: Continue if file fails (default: True)
|
||||
- `--dry-run`: Preview what would be deployed
|
||||
|
||||
### 7. Automatic Deployment Integration (`src/generation/batch_processor.py`)
|
||||
- Added `auto_deploy` parameter to `process_job()` (default: True)
|
||||
- Deployment triggers automatically after all tiers complete
|
||||
- Uses same `DeploymentService` as manual CLI command
|
||||
- Graceful error handling (logs warning, continues batch processing)
|
||||
- Can be disabled via `auto_deploy=False` for testing
|
||||
|
||||
### 8. Configuration (`src/core/config.py`)
|
||||
- Added `get_bunny_storage_api_key()` validation function
|
||||
- Checks for `BUNNY_API_KEY` in `.env` file
|
||||
- Clear error messages if keys are missing
|
||||
|
||||
### 9. Testing (`tests/integration/test_deployment.py`)
|
||||
Comprehensive integration tests covering:
|
||||
- URL generation and slug creation
|
||||
- Tier-segregated URL logging with duplicate prevention
|
||||
- Bunny.net storage client uploads
|
||||
- Deployment service (articles, pages, batches)
|
||||
- All 13 tests passing
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
deployment_logs/
|
||||
YYYY-MM-DD_tier1_urls.txt # Tier 1 article URLs
|
||||
YYYY-MM-DD_other_tiers_urls.txt # Tier 2+ article URLs
|
||||
|
||||
src/deployment/
|
||||
bunny_storage.py # Storage upload client
|
||||
deployment_service.py # Main deployment orchestration
|
||||
url_logger.py # Tier-segregated URL logging
|
||||
|
||||
scripts/
|
||||
migrate_add_deployment_fields.py # Database migration
|
||||
|
||||
tests/integration/
|
||||
test_deployment.py # Integration tests
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
ALTER TABLE generated_content ADD COLUMN deployed_url TEXT NULL;
|
||||
ALTER TABLE generated_content ADD COLUMN deployed_at TIMESTAMP NULL;
|
||||
CREATE INDEX idx_generated_content_deployed ON generated_content(deployed_at);
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Manual Deployment
|
||||
```bash
|
||||
# Deploy a specific batch
|
||||
uv run python -m src.cli deploy-batch --batch-id 123 --admin-user admin --admin-password pass
|
||||
|
||||
# Dry run to preview
|
||||
uv run python -m src.cli deploy-batch --batch-id 123 --dry-run
|
||||
```
|
||||
|
||||
### Automatic Deployment
|
||||
```bash
|
||||
# Generate batch with auto-deployment (default)
|
||||
uv run python -m src.cli generate-batch --job-file jobs/my_job.json
|
||||
|
||||
# Generate without auto-deployment
|
||||
# (Add --auto-deploy flag to generate-batch command if needed)
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Authentication**: Uses per-zone `storage_zone_password` from database for uploads. No API key from `.env` needed for storage operations. The `BUNNY_ACCOUNT_API_KEY` is only for zone creation/management.
|
||||
2. **File Locking**: Skipped for simplicity - duplicate prevention via file reading is sufficient
|
||||
3. **Auto-deploy Default**: ON by default for convenience, can be disabled for testing
|
||||
4. **Continue on Error**: Enabled by default to ensure partial deployments complete
|
||||
5. **URL Logging**: Simple text files (one URL per line) for easy parsing by Story 4.2
|
||||
6. **Boilerplate Pages**: Deploy stored HTML from `site_pages.content` (from Story 3.4)
|
||||
|
||||
## Dependencies Met
|
||||
|
||||
- Story 3.1: Site assignment (articles have `site_deployment_id`)
|
||||
- Story 3.3: Content interlinking (HTML is finalized)
|
||||
- Story 3.4: Boilerplate pages (`SitePage` table exists)
|
||||
|
||||
## Environment Variables Required
|
||||
|
||||
```bash
|
||||
BUNNY_ACCOUNT_API_KEY=your_account_api_key_here # For zone creation (already existed)
|
||||
```
|
||||
|
||||
**Important**: File uploads do NOT use an API key from `.env`. They use the per-zone `storage_zone_password` stored in the database (in the `site_deployments` table). This password is set automatically when zones are created via `provision-site` or `sync-sites` commands.
|
||||
|
||||
## Testing Results
|
||||
|
||||
All 13 integration tests passing:
|
||||
- URL generation (4 tests)
|
||||
- URL logging (4 tests)
|
||||
- Storage client (2 tests)
|
||||
- Deployment service (3 tests)
|
||||
|
||||
## Known Limitations / Technical Debt
|
||||
|
||||
1. Only supports Bunny.net (multi-cloud deferred to future stories)
|
||||
2. No CDN cache purging after deployment (Story 4.x)
|
||||
3. No deployment verification/validation (Story 4.4)
|
||||
4. URL logging is file-based (no database tracking)
|
||||
5. Boilerplate pages stored as full HTML in DB (inefficient, works for now)
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Story 4.2: URL logging enhancements (partially implemented here)
|
||||
- Story 4.3: Database status updates (partially implemented here)
|
||||
- Story 4.4: Post-deployment verification
|
||||
- Future: Multi-cloud support, CDN cache purging, parallel uploads
|
||||
|
||||
## Notes
|
||||
|
||||
- Simple and reliable implementation prioritized over complex features
|
||||
- Auto-deployment is the default happy path
|
||||
- Manual CLI command available for re-deployment or troubleshooting
|
||||
- Comprehensive error reporting for debugging
|
||||
- All API keys managed via `.env` only (not `master.config.json`)
|
||||
|
||||
|
|
@ -0,0 +1,172 @@
|
|||
# Story 4.1: Deploy Content to Cloud - Quick Start Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Ensure `BUNNY_ACCOUNT_API_KEY` is in your `.env` file (for creating zones):
|
||||
```bash
|
||||
BUNNY_ACCOUNT_API_KEY=your_account_api_key_here
|
||||
```
|
||||
|
||||
**Note**: File uploads use per-zone `storage_zone_password` from the database, NOT an API key from `.env`. These passwords are set automatically when sites are created via `provision-site` or `sync-sites` commands.
|
||||
|
||||
2. Run the database migration:
|
||||
```bash
|
||||
uv run python scripts/migrate_add_deployment_fields.py
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Automatic Deployment (Recommended)
|
||||
|
||||
Content deploys automatically after batch generation completes:
|
||||
|
||||
```bash
|
||||
uv run python -m src.cli generate-batch \
|
||||
--job-file jobs/my_job.json \
|
||||
--username admin \
|
||||
--password mypass
|
||||
```
|
||||
|
||||
Output will show deployment progress after all tiers complete:
|
||||
```
|
||||
Deployment: Starting automatic deployment for project 123...
|
||||
Deployment: 48 articles, 6 pages deployed
|
||||
Deployment: Complete in 45.2s
|
||||
```
|
||||
|
||||
### Manual Deployment
|
||||
|
||||
Deploy (or re-deploy) a batch manually:
|
||||
|
||||
```bash
|
||||
uv run python -m src.cli deploy-batch \
|
||||
--batch-id 123 \
|
||||
--admin-user admin \
|
||||
--admin-password mypass
|
||||
```
|
||||
|
||||
### Dry Run Mode
|
||||
|
||||
Preview what would be deployed without actually uploading:
|
||||
|
||||
```bash
|
||||
uv run python -m src.cli deploy-batch \
|
||||
--batch-id 123 \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
## What Gets Deployed
|
||||
|
||||
1. **Articles**: All generated articles with `formatted_html`
|
||||
- Uploaded to: `{slug}.html` (e.g., `how-to-fix-engines.html`)
|
||||
- URL logged to: `deployment_logs/YYYY-MM-DD_tier1_urls.txt` (Tier 1)
|
||||
- URL logged to: `deployment_logs/YYYY-MM-DD_other_tiers_urls.txt` (Tier 2+)
|
||||
|
||||
2. **Boilerplate Pages**: About, contact, privacy (if they exist)
|
||||
- Uploaded to: `about.html`, `contact.html`, `privacy.html`
|
||||
- NOT logged to URL files
|
||||
|
||||
## URL Logging
|
||||
|
||||
Deployed article URLs are automatically logged to tier-segregated files:
|
||||
|
||||
```
|
||||
deployment_logs/
|
||||
2025-10-22_tier1_urls.txt
|
||||
2025-10-22_other_tiers_urls.txt
|
||||
```
|
||||
|
||||
Each file contains one URL per line:
|
||||
```
|
||||
https://example.com/article-1.html
|
||||
https://example.com/article-2.html
|
||||
https://example.com/article-3.html
|
||||
```
|
||||
|
||||
Duplicate URLs are automatically prevented (safe to re-run deployments).
|
||||
|
||||
## Database Updates
|
||||
|
||||
After successful deployment, each article is updated with:
|
||||
- `deployed_url`: Public URL where content is live
|
||||
- `deployed_at`: Timestamp of deployment
|
||||
- `status`: Changed to 'deployed'
|
||||
|
||||
Query deployed content:
|
||||
```python
|
||||
from src.database.session import db_manager
|
||||
from src.database.repositories import GeneratedContentRepository
|
||||
|
||||
session = db_manager.get_session()
|
||||
repo = GeneratedContentRepository(session)
|
||||
|
||||
deployed = repo.get_deployed_content(project_id=123)
|
||||
for article in deployed:
|
||||
print(f"{article.title}: {article.deployed_url}")
|
||||
```
|
||||
|
||||
## Deployment Summary
|
||||
|
||||
After deployment completes, you'll see a summary:
|
||||
|
||||
```
|
||||
======================================================================
|
||||
Deployment Summary
|
||||
======================================================================
|
||||
Articles deployed: 48
|
||||
Articles failed: 2
|
||||
Pages deployed: 6
|
||||
Pages failed: 0
|
||||
Total time: 45.2s
|
||||
|
||||
Errors:
|
||||
Article 15 (Engine Maintenance Tips): Connection timeout
|
||||
Article 32 (Common Problems): Invalid HTML content
|
||||
======================================================================
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
By default, deployment continues even if individual files fail. This ensures partial deployments complete successfully.
|
||||
|
||||
Failed files are:
|
||||
- Logged to console with error details
|
||||
- Listed in deployment summary
|
||||
- NOT marked as deployed in database
|
||||
|
||||
To stop on first error:
|
||||
```bash
|
||||
uv run python -m src.cli deploy-batch \
|
||||
--batch-id 123 \
|
||||
--continue-on-error false
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Authentication failed for zone"
|
||||
Check that the `storage_zone_password` in your database is correct. This is set when sites are created via `provision-site` or `sync-sites` commands.
|
||||
|
||||
### "Article has no formatted_html to deploy"
|
||||
Ensure articles have templates applied. This happens automatically during batch processing in `_post_process_tier()`.
|
||||
|
||||
### "Site not found"
|
||||
Ensure articles are assigned to sites. This happens automatically during batch processing via site assignment logic.
|
||||
|
||||
## Manual Re-deployment
|
||||
|
||||
To re-deploy content after fixes:
|
||||
|
||||
1. Fix the issue (update HTML, fix credentials, etc.)
|
||||
2. Run manual deployment:
|
||||
```bash
|
||||
uv run python -m src.cli deploy-batch --batch-id 123
|
||||
```
|
||||
3. Duplicate URLs are automatically prevented in log files
|
||||
|
||||
## Integration with Other Stories
|
||||
|
||||
- **Story 3.1**: Articles must be assigned to sites before deployment
|
||||
- **Story 3.4**: Boilerplate pages are deployed if they exist in `site_pages` table
|
||||
- **Story 4.2**: URL log files are consumed by post-deployment processes
|
||||
- **Story 4.3**: Database status updates enable deployment tracking
|
||||
|
||||
|
|
@ -1,23 +1,106 @@
|
|||
# Epic 4: Cloud Deployment & Handoff
|
||||
|
||||
## Epic Goal
|
||||
To deploy the finalized, interlinked HTML files to their cloud targets, log the results, and hand off the data to the link-building machine.
|
||||
To deploy all finalized HTML content (articles and boilerplate pages) for a batch to the correct cloud storage targets, purge the CDN cache, and verify the successful deployment.
|
||||
|
||||
## Status
|
||||
- **Story 4.1**: ✓ Documented (Ready for implementation)
|
||||
- **Story 4.2**: Partially implemented in Story 4.1 (URL logging)
|
||||
- **Story 4.3**: Partially implemented in Story 4.1 (Database status updates)
|
||||
- **Story 4.4**: Not started
|
||||
- **Story 4.5**: Not started
|
||||
|
||||
## Stories
|
||||
|
||||
### Story 4.1: Deploy Finalized HTML Content
|
||||
**As a User**, I want to deploy the fully interlinked HTML files to their pre-validated cloud storage locations, so that the content goes live.
|
||||
### Story 4.1: Deploy Content to Cloud Storage
|
||||
**Status:** ✓ Documented - Ready for Implementation (22 story points)
|
||||
**Document:** [story-4.1-deploy-content-to-cloud.md](../stories/story-4.1-deploy-content-to-cloud.md)
|
||||
|
||||
**As a developer**, I want to upload all generated HTML files for a batch to their designated cloud storage buckets so that the content is hosted and ready to be served.
|
||||
|
||||
**Acceptance Criteria**
|
||||
- A script takes the final, interlinked HTML content for an article and its corresponding final URL.
|
||||
- It uses the unified cloud deployment module to upload the content to the correct location.
|
||||
- The deployment is verified as successful.
|
||||
* CLI command (`deploy-batch --batch_id <id>`) and auto-deploy after batch generation
|
||||
* Bunny.net Storage API integration (multi-cloud is technical debt)
|
||||
* Uploads articles and boilerplate pages (about, contact, privacy) if they exist
|
||||
* Authentication via `BUNNY_API_KEY` from `.env` and `storage_zone_password` from database
|
||||
* Continue on error, report detailed summary (successful, failed, total time)
|
||||
* **Includes Story 4.2 functionality**: Log URLs to tier-segregated daily text files
|
||||
* **Includes Story 4.3 functionality**: Update database status to 'deployed', store public URLs
|
||||
|
||||
### Story 4.2: URL Logging & API Handoff
|
||||
**As a developer**, I want to log all the pre-determined URLs to the database and transmit the job data via the internal API, so that the workflow is completed and tracked.
|
||||
**Implementation Notes**
|
||||
* Auto-deploy is ON by default
|
||||
* Duplicate URL prevention in text files (critical for manual re-runs)
|
||||
* All API keys from `.env` only (not master.config.json)
|
||||
* Storage API authentication details TBD during implementation
|
||||
|
||||
**Acceptance Criteria**
|
||||
- After successful deployment, the list of pre-determined public URLs from the "link map" is saved to the database.
|
||||
- The content Tier is correctly recorded for each URL.
|
||||
- The URLs are appended to the local .txt file.
|
||||
- The necessary job data, including the list of new URLs, is successfully transmitted to the link-building machine's API endpoint.
|
||||
### Story 4.2: Log Deployed URLs to Tiered Text Files
|
||||
**Status:** ✓ Implemented in Story 4.1
|
||||
|
||||
**As a developer**, I want to save the URLs of all deployed articles into daily, tier-segregated text files, so that I have a clean list for indexing services and other external tools.
|
||||
|
||||
**Acceptance Criteria** (Implemented in Story 4.1)
|
||||
* After an article is successfully deployed, its public URL is logged to a text file.
|
||||
* A `deployment_logs/` folder will be used to store the output files.
|
||||
* Two separate files are created for each day's deployments, using a `YYYY-MM-DD` timestamp.
|
||||
* `deployment_logs/YYYY-MM-DD_tier1_urls.txt`
|
||||
* `deployment_logs/YYYY-MM-DD_other_tiers_urls.txt`
|
||||
* URLs for Tier 1 articles are appended to the `_tier1_urls.txt` file.
|
||||
* URLs for all other tiers (T2, T3, etc.) are appended to the `_other_tiers_urls.txt` file.
|
||||
* The script automatically creates new files when the date changes.
|
||||
* URLs for boilerplate pages (about, contact, privacy) are explicitly excluded from these files.
|
||||
* **Duplicate prevention**: Check file before appending to avoid duplicate URLs
|
||||
|
||||
### Story 4.3: Update Deployment Status
|
||||
**Status:** ✓ Implemented in Story 4.1
|
||||
|
||||
**As a developer**, I want to update the status of each article and site in the database to 'deployed' and record the final public URL, so that the system has an accurate record of what content is live.
|
||||
|
||||
**Acceptance Criteria** (Implemented in Story 4.1)
|
||||
* Upon successful upload of an article, its status in the `generated_content` table is updated to 'deployed'.
|
||||
* The final, verified public URL for the article is stored in `deployed_url` field.
|
||||
* New fields added: `deployed_url` and `deployed_at` to `generated_content` table.
|
||||
* Database updates are transactional to ensure data integrity.
|
||||
|
||||
**Note:** The `last_deployed_at` timestamp for `site_deployments` could be added as enhancement if needed.
|
||||
|
||||
### Story 4.4: Post-Deployment Verification
|
||||
**Status:** Not Started
|
||||
|
||||
**As a user**, I want a simple way to verify that a batch of articles has been deployed successfully, by checking a sample of URLs for a `200 OK` status, so I can have confidence the deployment worked.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
* A post-deployment script or CLI command is available (e.g., `verify-deployment --batch_id <id>`).
|
||||
* The script takes a batch ID as input.
|
||||
* It retrieves the URLs for all articles in that batch from database.
|
||||
* It makes an HTTP GET request to a random sample of (or all based on command flags) URLs.
|
||||
* It reports which URLs return a `200 OK` status and which do not.
|
||||
* The output is clear and easy to read (e.g., a list of successful and failed URLs).
|
||||
* Can be run manually after deployment or integrated into auto-deploy workflow.
|
||||
|
||||
### Story 4.5: Create URL and Link Reporting Script
|
||||
**Status:** Not Started
|
||||
|
||||
**As a user**, I want a script to generate custom lists of URLs based on project and tier, with optional link details, so that I can easily export data for analysis or external tools.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
* A new CLI script is created (e.g., `scripts/get_links.py` or CLI command `get-links`).
|
||||
* The script accepts a mandatory `project_id`.
|
||||
* The script accepts a `tier` specifier that supports:
|
||||
* A single tier (e.g., `--tier 1`).
|
||||
* An open-ended range for a tier and above (e.g., `--tier 2+`).
|
||||
* An optional flag (`--with-anchor-text`) includes the anchor text used for each link in the output.
|
||||
* An optional flag (`--with-destination-url`) includes the destination URL of the tiered link placed on that page.
|
||||
* The script queries the database to retrieve the required link and content information.
|
||||
* The output is a well-formatted list (e.g., CSV or plain text) printed to the console.
|
||||
|
||||
## Technical Debt
|
||||
- Multi-cloud support (AWS S3, Azure, DigitalOcean, etc.) - deferred from Story 4.1
|
||||
- CDN cache purging after deployment
|
||||
- Boilerplate page storage optimization (regenerate on-the-fly vs storing HTML)
|
||||
- Homepage (`index.html`) generation for sites
|
||||
|
||||
## Notes
|
||||
- Story 4.1 is the primary deployment story and includes core functionality from Stories 4.2 and 4.3
|
||||
- Auto-deploy is enabled by default to streamline workflow
|
||||
- All cloud provider credentials come from `.env` file only
|
||||
- Story 4.4 and 4.5 are independent utilities that can be implemented as needed
|
||||
|
|
|
|||
|
|
@ -0,0 +1,650 @@
|
|||
# Story 4.1: Deploy Content to Cloud Storage
|
||||
|
||||
## Status
|
||||
**DRAFT** - Needs Review
|
||||
|
||||
## Story
|
||||
**As a developer**, I want to upload all generated HTML files for a batch to their designated Bunny.net storage buckets so that the content is hosted and ready to be served.
|
||||
|
||||
## Context
|
||||
- Epic 4 is about deploying finalized content to cloud storage
|
||||
- Story 3.4 implemented boilerplate site pages (about, contact, privacy)
|
||||
- Articles have URLs and are assigned to sites (Story 3.1)
|
||||
- Interlinking is complete (Story 3.3)
|
||||
- Content is ready to deploy after batch processing completes
|
||||
- Bunny.net is the only cloud provider for now (multi-cloud is technical debt)
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Core Deployment Functionality
|
||||
- CLI command `deploy-batch --batch_id <id>` deploys all content in a batch
|
||||
- Deployment is also triggered automatically after batch generation completes
|
||||
- Deployment uploads both articles and boilerplate pages (about, contact, privacy)
|
||||
- For boilerplate pages: Check `site_pages` table, deploy pages that exist
|
||||
- Read HTML content directly from `site_pages.content` field (stored in Story 3.4)
|
||||
- Authentication uses:
|
||||
- `BUNNY_API_KEY` from `.env` (storage API operations)
|
||||
- `storage_zone_password` from SiteDeployment model (per-zone)
|
||||
- `BUNNY_ACCOUNT_API_KEY` from `.env` (only for creating zones, not uploads)
|
||||
- For each piece of content, identify correct destination storage bucket/path
|
||||
- Upload final HTML to target path (e.g., `about.html`, `my-article-slug.html`)
|
||||
|
||||
### Error Handling
|
||||
- Continue on error (don't halt entire deployment if one file fails)
|
||||
- Log errors for individual file failures
|
||||
- Report summary at end: successful uploads, failed uploads, total time
|
||||
- Both screen output and log file
|
||||
|
||||
### URL Tracking (Story 4.2 Preview)
|
||||
- After article is successfully deployed, log its public URL to tier-segregated text file
|
||||
- Create `deployment_logs/` folder if it doesn't exist
|
||||
- Two files per day: `YYYY-MM-DD_tier1_urls.txt` and `YYYY-MM-DD_other_tiers_urls.txt`
|
||||
- URLs for Tier 1 articles → `_tier1_urls.txt`
|
||||
- URLs for Tier 2+ articles → `_other_tiers_urls.txt`
|
||||
- Boilerplate pages (about, contact, privacy) are NOT logged to these files
|
||||
- **Must avoid duplicate URLs**: Read file, check if URL exists, only append if new
|
||||
- Prevents duplicates from manual re-runs after automatic deployment
|
||||
|
||||
### Database Updates (Story 4.3 Preview)
|
||||
- Update article status to 'deployed' after successful upload
|
||||
- Store final public URL in database
|
||||
- Transactional updates to ensure data integrity
|
||||
|
||||
## Tasks / Subtasks
|
||||
|
||||
### 1. Create Bunny.net Storage Upload Client
|
||||
**Effort:** 3 story points
|
||||
|
||||
- [ ] Create `src/deployment/bunny_storage.py` module
|
||||
- [ ] Implement `BunnyStorageClient` class for uploading files
|
||||
- [ ] Use Bunny.net Storage API (different from Account API)
|
||||
- [ ] Authentication using:
|
||||
- `BUNNY_API_KEY` from `.env` (account-level storage API key)
|
||||
- `storage_zone_password` from SiteDeployment model (per-zone password)
|
||||
- Determine correct authentication method during implementation
|
||||
- [ ] Methods:
|
||||
- `upload_file(zone_name, zone_password, file_path, content, content_type='text/html')`
|
||||
- `file_exists(zone_name, zone_password, file_path) -> bool`
|
||||
- `list_files(zone_name, zone_password, prefix='') -> List[str]`
|
||||
- [ ] Handle HTTP errors, timeouts, retries (3 retries with exponential backoff)
|
||||
- [ ] Logging at INFO level for uploads, ERROR for failures
|
||||
|
||||
### 2. Create Deployment Service
|
||||
**Effort:** 3 story points
|
||||
|
||||
- [ ] Create `src/deployment/deployment_service.py` module
|
||||
- [ ] Implement `DeploymentService` class with:
|
||||
- `deploy_batch(batch_id, project_id, continue_on_error=True)`
|
||||
- `deploy_article(content_id, site_deployment)`
|
||||
- `deploy_boilerplate_page(site_page, site_deployment)`
|
||||
- [ ] Query all `GeneratedContent` records for project_id
|
||||
- [ ] Query all `SitePage` records for sites in batch
|
||||
- [ ] For each article:
|
||||
- Get site deployment info (storage zone, region, hostname)
|
||||
- Generate file path (slug-based, e.g., `my-article-slug.html`)
|
||||
- Upload HTML content to Bunny.net storage
|
||||
- Log success/failure
|
||||
- [ ] For each boilerplate page (if exists):
|
||||
- Get site deployment info
|
||||
- Generate file path (e.g., `about.html`, `contact.html`, `privacy.html`)
|
||||
- Upload HTML content
|
||||
- Log success/failure
|
||||
- [ ] Track deployment results (successful, failed, skipped)
|
||||
- [ ] Return deployment summary
|
||||
|
||||
### 3. Implement URL Generation for Deployment
|
||||
**Effort:** 2 story points
|
||||
|
||||
- [ ] Extend `src/generation/url_generator.py` module
|
||||
- [ ] Add `generate_public_url(site_deployment, file_path) -> str`:
|
||||
- Use custom_hostname if available, else pull_zone_bcdn_hostname
|
||||
- Return full URL: `https://{hostname}/{file_path}`
|
||||
- [ ] Add `generate_file_path(content) -> str`:
|
||||
- For articles: Use slug from title or keyword (lowercase, hyphens, .html extension)
|
||||
- For boilerplate pages: Fixed names (about.html, contact.html, privacy.html)
|
||||
- [ ] Handle edge cases (special characters, long slugs, conflicts)
|
||||
|
||||
### 4. Implement URL Logging to Text Files
|
||||
**Effort:** 2 story points
|
||||
|
||||
- [ ] Create `src/deployment/url_logger.py` module
|
||||
- [ ] Implement `URLLogger` class with:
|
||||
- `log_article_url(url, tier, date=None)`
|
||||
- `get_existing_urls(tier, date=None) -> Set[str]`
|
||||
- [ ] Create `deployment_logs/` directory if doesn't exist
|
||||
- [ ] Determine file based on tier and date:
|
||||
- Tier 1: `deployment_logs/YYYY-MM-DD_tier1_urls.txt`
|
||||
- Tier 2+: `deployment_logs/YYYY-MM-DD_other_tiers_urls.txt`
|
||||
- [ ] Check if URL already exists in file before appending
|
||||
- [ ] Append URL to file (one per line)
|
||||
- [ ] Thread-safe file writing (use file locks)
|
||||
|
||||
### 5. Implement Database Status Updates
|
||||
**Effort:** 2 story points
|
||||
|
||||
- [ ] Update `src/database/models.py`:
|
||||
- Add `deployed_url` field to `GeneratedContent` (nullable string)
|
||||
- Add `deployed_at` field to `GeneratedContent` (nullable datetime)
|
||||
- [ ] Create migration script `scripts/migrate_add_deployment_fields.py`
|
||||
- [ ] Update `GeneratedContentRepository` with:
|
||||
- `mark_as_deployed(content_id, url, timestamp=None)`
|
||||
- `get_deployed_content(project_id) -> List[GeneratedContent]`
|
||||
- [ ] Use transactions to ensure atomicity
|
||||
- [ ] Log status updates at INFO level
|
||||
|
||||
### 6. Create CLI Command: deploy-batch
|
||||
**Effort:** 2 story points
|
||||
|
||||
- [ ] Add `deploy-batch` command to `src/cli/commands.py`
|
||||
- [ ] Arguments:
|
||||
- `--batch_id` (required): Batch/project ID to deploy
|
||||
- `--admin-user` (optional): Admin username for authentication
|
||||
- `--admin-password` (optional): Admin password
|
||||
- `--continue-on-error` (default: True): Continue if file fails
|
||||
- `--dry-run` (default: False): Preview what would be deployed
|
||||
- [ ] Authenticate admin user
|
||||
- [ ] Load Bunny.net credentials from `.env`
|
||||
- [ ] Call `DeploymentService.deploy_batch()`
|
||||
- [ ] Display progress (articles uploaded, pages uploaded, errors)
|
||||
- [ ] Show final summary with statistics
|
||||
- [ ] Exit code 0 if all succeeded, 1 if any failures
|
||||
|
||||
### 7. Integrate Deployment into Batch Processing
|
||||
**Effort:** 2 story points
|
||||
|
||||
- [ ] Update `src/generation/batch_processor.py`
|
||||
- [ ] Add optional `auto_deploy` parameter to `process_job()`
|
||||
- [ ] After interlinking completes, trigger deployment if `auto_deploy=True`
|
||||
- [ ] Use same deployment service as CLI command
|
||||
- [ ] Log deployment results
|
||||
- [ ] Handle deployment errors gracefully (don't fail batch if deployment fails)
|
||||
- [ ] Make `auto_deploy=True` by default (deploy immediately after generation)
|
||||
- [ ] Allow `auto_deploy=False` flag for testing/debugging scenarios
|
||||
|
||||
### 8. Environment Variable Validation
|
||||
**Effort:** 1 story point
|
||||
|
||||
- [ ] Confirm `src/core/config.py` loads Bunny.net keys from `.env` only
|
||||
- [ ] Add validation in deployment service to check required env vars:
|
||||
- `BUNNY_API_KEY` (for storage uploads)
|
||||
- `BUNNY_ACCOUNT_API_KEY` (for account operations, if needed)
|
||||
- [ ] Raise clear error if keys are missing
|
||||
- [ ] Document in technical notes which keys are required
|
||||
- [ ] Do NOT reference `master.config.json` for any API keys
|
||||
|
||||
### 9. Unit Tests
|
||||
**Effort:** 3 story points
|
||||
|
||||
- [ ] Test `BunnyStorageClient` upload functionality (mock HTTP calls)
|
||||
- [ ] Test URL generation for various content types
|
||||
- [ ] Test file path generation (slug creation, special characters)
|
||||
- [ ] Test URL logger (file creation, duplicate prevention)
|
||||
- [ ] Test deployment service (successful upload, failed upload, mixed results)
|
||||
- [ ] Test database status updates
|
||||
- [ ] Mock Bunny.net API responses
|
||||
- [ ] Achieve >80% code coverage for new modules
|
||||
|
||||
### 10. Integration Tests
|
||||
**Effort:** 2 story points
|
||||
|
||||
- [ ] Test end-to-end deployment of small batch (2-3 articles)
|
||||
- [ ] Test deployment with boilerplate pages
|
||||
- [ ] Test deployment without boilerplate pages
|
||||
- [ ] Test URL logging (multiple deployments, different days)
|
||||
- [ ] Test database updates (status changes, URLs stored)
|
||||
- [ ] Test CLI command with dry-run mode
|
||||
- [ ] Test continue-on-error behavior
|
||||
- [ ] Verify no duplicate URLs in log files
|
||||
|
||||
## Technical Notes
|
||||
|
||||
### Bunny.net Storage API
|
||||
|
||||
Bunny.net has two separate APIs:
|
||||
1. **Account API** (existing `BunnyNetClient`): For creating storage zones, pull zones
|
||||
- Uses `BUNNY_ACCOUNT_API_KEY` from `.env`
|
||||
2. **Storage API** (new `BunnyStorageClient`): For uploading/managing files
|
||||
- Uses `BUNNY_API_KEY` from `.env` (account-level storage access)
|
||||
- Uses `storage_zone_password` from `SiteDeployment` model (per-zone password)
|
||||
- Requires BOTH credentials for authentication
|
||||
|
||||
Storage API authentication:
|
||||
- Base URL: `https://storage.bunnycdn.com/{zone_name}/{file_path}`
|
||||
- Authentication method to be determined during implementation:
|
||||
- `BUNNY_API_KEY` from `.env` (account-level)
|
||||
- `storage_zone_password` from database (per-zone, returned in JSON when zone is created)
|
||||
- May require one or both keys depending on Bunny.net's API requirements
|
||||
- Storage API key can be extracted from Bunny.net JSON response during zone creation
|
||||
- If implementation issues arise, reference code/examples can be provided
|
||||
|
||||
Upload example:
|
||||
```python
|
||||
# Get site from database
|
||||
site = site_repo.get_by_id(site_deployment_id)
|
||||
|
||||
# Get API key from .env
|
||||
bunny_api_key = os.getenv("BUNNY_API_KEY")
|
||||
|
||||
# Upload (authentication method TBD during implementation)
|
||||
PUT https://storage.bunnycdn.com/{site.storage_zone_name}/my-article.html
|
||||
Headers:
|
||||
AccessKey: {bunny_api_key OR site.storage_zone_password} # TBD
|
||||
Content-Type: text/html
|
||||
Body:
|
||||
<html>...</html>
|
||||
```
|
||||
|
||||
### File Path Structure
|
||||
|
||||
```
|
||||
Storage Zone: my-zone
|
||||
Region: DE (Germany)
|
||||
|
||||
Articles:
|
||||
/my-article-slug.html
|
||||
/another-article.html
|
||||
/third-article-title.html
|
||||
|
||||
Boilerplate pages:
|
||||
/about.html
|
||||
/contact.html
|
||||
/privacy.html
|
||||
|
||||
Not using subdirectories for simplicity
|
||||
Future: Could organize by date or category
|
||||
```
|
||||
|
||||
### URL Logger Implementation
|
||||
|
||||
```python
|
||||
# src/deployment/url_logger.py
|
||||
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Set
|
||||
from pathlib import Path
|
||||
import fcntl # For file locking on Unix
|
||||
|
||||
class URLLogger:
|
||||
def __init__(self, logs_dir: str = "deployment_logs"):
|
||||
self.logs_dir = Path(logs_dir)
|
||||
self.logs_dir.mkdir(exist_ok=True)
|
||||
|
||||
def log_article_url(self, url: str, tier: str, date: datetime = None):
|
||||
if date is None:
|
||||
date = datetime.utcnow()
|
||||
|
||||
# Determine file
|
||||
tier_num = self._extract_tier_number(tier)
|
||||
if tier_num == 1:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_tier1_urls.txt"
|
||||
else:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_other_tiers_urls.txt"
|
||||
|
||||
filepath = self.logs_dir / filename
|
||||
|
||||
# Check for duplicates
|
||||
existing = self.get_existing_urls(tier, date)
|
||||
if url in existing:
|
||||
return # Skip duplicate
|
||||
|
||||
# Append to file (with lock)
|
||||
with open(filepath, 'a') as f:
|
||||
fcntl.flock(f, fcntl.LOCK_EX)
|
||||
f.write(f"{url}\n")
|
||||
fcntl.flock(f, fcntl.LOCK_UN)
|
||||
|
||||
def get_existing_urls(self, tier: str, date: datetime = None) -> Set[str]:
|
||||
"""
|
||||
Get existing URLs from log file to prevent duplicates
|
||||
|
||||
This is critical for preventing duplicate entries when:
|
||||
- Auto-deployment runs, then manual re-run happens
|
||||
- Deployment fails partway and is restarted
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.utcnow()
|
||||
|
||||
tier_num = self._extract_tier_number(tier)
|
||||
if tier_num == 1:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_tier1_urls.txt"
|
||||
else:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_other_tiers_urls.txt"
|
||||
|
||||
filepath = self.logs_dir / filename
|
||||
|
||||
if not filepath.exists():
|
||||
return set()
|
||||
|
||||
with open(filepath, 'r') as f:
|
||||
return set(line.strip() for line in f if line.strip())
|
||||
|
||||
def _extract_tier_number(self, tier: str) -> int:
|
||||
# Extract number from "tier1", "tier2", etc.
|
||||
return int(''.join(c for c in tier if c.isdigit()))
|
||||
```
|
||||
|
||||
### Deployment Service Implementation
|
||||
|
||||
```python
|
||||
# src/deployment/deployment_service.py
|
||||
|
||||
from typing import List, Dict, Any
|
||||
from src.deployment.bunny_storage import BunnyStorageClient
|
||||
from src.deployment.url_logger import URLLogger
|
||||
from src.database.repositories import GeneratedContentRepository, SitePageRepository, SiteDeploymentRepository
|
||||
from src.generation.url_generator import generate_public_url, generate_file_path
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class DeploymentService:
|
||||
def __init__(
|
||||
self,
|
||||
storage_client: BunnyStorageClient,
|
||||
content_repo: GeneratedContentRepository,
|
||||
site_repo: SiteDeploymentRepository,
|
||||
page_repo: SitePageRepository,
|
||||
url_logger: URLLogger
|
||||
):
|
||||
self.storage = storage_client
|
||||
self.content_repo = content_repo
|
||||
self.site_repo = site_repo
|
||||
self.page_repo = page_repo
|
||||
self.url_logger = url_logger
|
||||
|
||||
def deploy_batch(self, project_id: int, continue_on_error: bool = True) -> Dict[str, Any]:
|
||||
"""
|
||||
Deploy all content for a project/batch
|
||||
|
||||
Returns:
|
||||
Dict with deployment statistics:
|
||||
{
|
||||
'articles_deployed': 10,
|
||||
'articles_failed': 1,
|
||||
'pages_deployed': 6,
|
||||
'pages_failed': 0,
|
||||
'total_time': 45.2
|
||||
}
|
||||
"""
|
||||
results = {
|
||||
'articles_deployed': 0,
|
||||
'articles_failed': 0,
|
||||
'pages_deployed': 0,
|
||||
'pages_failed': 0,
|
||||
'errors': []
|
||||
}
|
||||
|
||||
# Get all articles for project
|
||||
articles = self.content_repo.get_by_project_id(project_id)
|
||||
logger.info(f"Found {len(articles)} articles to deploy for project {project_id}")
|
||||
|
||||
# Deploy articles
|
||||
for article in articles:
|
||||
if not article.site_deployment_id:
|
||||
logger.warning(f"Article {article.id} has no site assigned, skipping")
|
||||
continue
|
||||
|
||||
try:
|
||||
site = self.site_repo.get_by_id(article.site_deployment_id)
|
||||
if not site:
|
||||
raise ValueError(f"Site {article.site_deployment_id} not found")
|
||||
|
||||
# Deploy article
|
||||
url = self.deploy_article(article, site)
|
||||
|
||||
# Log URL to text file
|
||||
self.url_logger.log_article_url(url, article.tier)
|
||||
|
||||
# Update database
|
||||
self.content_repo.mark_as_deployed(article.id, url)
|
||||
|
||||
results['articles_deployed'] += 1
|
||||
logger.info(f"Deployed article {article.id} to {url}")
|
||||
|
||||
except Exception as e:
|
||||
results['articles_failed'] += 1
|
||||
results['errors'].append({
|
||||
'type': 'article',
|
||||
'id': article.id,
|
||||
'error': str(e)
|
||||
})
|
||||
logger.error(f"Failed to deploy article {article.id}: {e}")
|
||||
|
||||
if not continue_on_error:
|
||||
raise
|
||||
|
||||
# Get unique sites from articles
|
||||
site_ids = set(a.site_deployment_id for a in articles if a.site_deployment_id)
|
||||
|
||||
# Deploy boilerplate pages for each site
|
||||
for site_id in site_ids:
|
||||
site = self.site_repo.get_by_id(site_id)
|
||||
pages = self.page_repo.get_by_site(site_id)
|
||||
|
||||
if not pages:
|
||||
logger.debug(f"Site {site_id} has no boilerplate pages, skipping")
|
||||
continue
|
||||
|
||||
logger.info(f"Found {len(pages)} boilerplate pages for site {site_id}")
|
||||
|
||||
for page in pages:
|
||||
try:
|
||||
# Read HTML from database (stored in page.content from Story 3.4)
|
||||
url = self.deploy_boilerplate_page(page, site)
|
||||
results['pages_deployed'] += 1
|
||||
logger.info(f"Deployed page {page.page_type} to {url}")
|
||||
|
||||
except Exception as e:
|
||||
results['pages_failed'] += 1
|
||||
results['errors'].append({
|
||||
'type': 'page',
|
||||
'site_id': site_id,
|
||||
'page_type': page.page_type,
|
||||
'error': str(e)
|
||||
})
|
||||
logger.error(f"Failed to deploy page {page.page_type} for site {site_id}: {e}")
|
||||
|
||||
if not continue_on_error:
|
||||
raise
|
||||
|
||||
return results
|
||||
|
||||
def deploy_article(self, article, site) -> str:
|
||||
"""Deploy a single article, return public URL"""
|
||||
file_path = generate_file_path(article)
|
||||
url = generate_public_url(site, file_path)
|
||||
|
||||
# Upload using both BUNNY_API_KEY and zone password
|
||||
# BunnyStorageClient determines which auth method to use
|
||||
self.storage.upload_file(
|
||||
zone_name=site.storage_zone_name,
|
||||
zone_password=site.storage_zone_password, # Per-zone password from DB
|
||||
file_path=file_path,
|
||||
content=article.formatted_html,
|
||||
content_type='text/html'
|
||||
)
|
||||
|
||||
return url
|
||||
|
||||
def deploy_boilerplate_page(self, page, site) -> str:
|
||||
"""
|
||||
Deploy a boilerplate page, return public URL
|
||||
|
||||
Note: Uses stored HTML from page.content (from Story 3.4)
|
||||
Technical debt: Could regenerate on-the-fly instead of storing
|
||||
"""
|
||||
file_path = f"{page.page_type}.html"
|
||||
url = generate_public_url(site, file_path)
|
||||
|
||||
# Upload using both BUNNY_API_KEY and zone password
|
||||
self.storage.upload_file(
|
||||
zone_name=site.storage_zone_name,
|
||||
zone_password=site.storage_zone_password,
|
||||
file_path=file_path,
|
||||
content=page.content, # Full HTML stored in DB
|
||||
content_type='text/html'
|
||||
)
|
||||
|
||||
return url
|
||||
```
|
||||
|
||||
### CLI Command Example
|
||||
|
||||
```bash
|
||||
# Deploy a batch manually
|
||||
uv run python -m src.cli deploy-batch \
|
||||
--batch_id 123 \
|
||||
--admin-user admin \
|
||||
--admin-password mypass
|
||||
|
||||
# Output:
|
||||
# Authenticating...
|
||||
# Loading Bunny.net credentials...
|
||||
# Deploying batch 123...
|
||||
# [1/50] Deploying article "How to Fix Engines"... ✓
|
||||
# [2/50] Deploying article "Engine Maintenance Tips"... ✓
|
||||
# ...
|
||||
# [50/50] Deploying article "Common Engine Problems"... ✓
|
||||
# Deploying boilerplate pages...
|
||||
# [1/6] Deploying about.html for site1.b-cdn.net... ✓
|
||||
# [2/6] Deploying contact.html for site1.b-cdn.net... ✓
|
||||
# ...
|
||||
#
|
||||
# Deployment Summary:
|
||||
# ==================
|
||||
# Articles deployed: 48
|
||||
# Articles failed: 2
|
||||
# Pages deployed: 6
|
||||
# Pages failed: 0
|
||||
# Total time: 2m 34s
|
||||
#
|
||||
# Failed articles:
|
||||
# - Article 15: Connection timeout
|
||||
# - Article 32: Invalid HTML content
|
||||
|
||||
# Dry-run mode
|
||||
uv run python -m src.cli deploy-batch \
|
||||
--batch_id 123 \
|
||||
--dry-run
|
||||
|
||||
# Output shows what would be deployed without actually uploading
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Required in `.env` file:
|
||||
```bash
|
||||
# Bunny.net Account API (for creating/managing storage zones and pull zones)
|
||||
BUNNY_ACCOUNT_API_KEY=your_account_api_key_here
|
||||
|
||||
# Bunny.net Storage API (for uploading files to storage)
|
||||
BUNNY_API_KEY=your_storage_api_key_here
|
||||
|
||||
# Note: storage_zone_password is per-zone and stored in database
|
||||
# Both BUNNY_API_KEY and storage_zone_password may be needed for uploads
|
||||
# API keys should ONLY be in .env file, NOT in master.config.json
|
||||
```
|
||||
|
||||
### Database Schema Updates
|
||||
|
||||
```sql
|
||||
-- Add deployment tracking fields to generated_content
|
||||
ALTER TABLE generated_content ADD COLUMN deployed_url TEXT NULL;
|
||||
ALTER TABLE generated_content ADD COLUMN deployed_at TIMESTAMP NULL;
|
||||
|
||||
CREATE INDEX idx_generated_content_deployed ON generated_content(deployed_at);
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
- Story 3.1: Site assignment (need site_deployment_id on articles)
|
||||
- Story 3.3: Content interlinking (HTML must be finalized)
|
||||
- Story 3.4: Boilerplate pages (need SitePage table)
|
||||
- Bunny.net Storage API access
|
||||
- Environment variables configured in `.env`
|
||||
|
||||
## Future Considerations
|
||||
- Story 4.2: URL logging (partially implemented here)
|
||||
- Story 4.3: Database status updates (partially implemented here)
|
||||
- Story 4.4: Post-deployment verification
|
||||
- Multi-cloud support (AWS S3, Azure, DigitalOcean, etc.)
|
||||
- CDN cache purging after deployment
|
||||
- Parallel uploads for faster deployment
|
||||
- Resumable uploads for large files
|
||||
- Deployment rollback mechanism
|
||||
|
||||
## Technical Debt Created
|
||||
- Multi-cloud support deferred (only Bunny.net for now)
|
||||
- No CDN cache purging yet (Story 4.x)
|
||||
- No deployment verification yet (Story 4.4)
|
||||
- URL logging is simple (no database tracking of logged URLs)
|
||||
- Boilerplate pages stored as full HTML in database (inefficient)
|
||||
- Better approach: Store just page_type marker, regenerate HTML on-the-fly at deployment
|
||||
- Reduces storage, ensures consistency with current templates
|
||||
- Defer optimization to later story
|
||||
|
||||
## Total Effort
|
||||
22 story points
|
||||
|
||||
### Effort Breakdown
|
||||
1. Bunny Storage Client (3 points)
|
||||
2. Deployment Service (3 points)
|
||||
3. URL Generation (2 points)
|
||||
4. URL Logging (2 points)
|
||||
5. Database Updates (2 points)
|
||||
6. CLI Command (2 points)
|
||||
7. Batch Integration (2 points)
|
||||
8. Environment Audit (1 point)
|
||||
9. Unit Tests (3 points)
|
||||
10. Integration Tests (2 points)
|
||||
|
||||
## Questions & Clarifications
|
||||
|
||||
### Question 1: Boilerplate Page Deployment Strategy
|
||||
**Status:** ✓ RESOLVED
|
||||
|
||||
The approach:
|
||||
- Check `site_pages` table in database
|
||||
- Only deploy boilerplate pages if they exist in DB
|
||||
- Read HTML content from `site_pages.content` field
|
||||
- Most sites won't have them (only newly created sites from Story 3.4+)
|
||||
- Don't check remote buckets (database is source of truth)
|
||||
|
||||
### Question 2: URL Duplicate Prevention
|
||||
**Status:** ✓ RESOLVED
|
||||
|
||||
Approach:
|
||||
- Read entire file before appending
|
||||
- Check if URL exists in memory (set), skip if duplicate
|
||||
- File locking for thread-safety
|
||||
- This prevents duplicate URLs from manual re-runs after automatic deployment
|
||||
- No database tracking needed (file is source of truth)
|
||||
|
||||
### Question 3: Auto-deploy Default Behavior
|
||||
**Status:** ✓ RESOLVED
|
||||
|
||||
Decision: **ON by default**
|
||||
- Auto-deploy after batch generation completes
|
||||
- No reason to delay deployment in normal workflow
|
||||
- CLI command still available for manual re-deployment if auto-deploy fails
|
||||
- Can be disabled for testing via flag if needed
|
||||
|
||||
### Question 4: API Keys in master.config.json
|
||||
**Status:** ✓ RESOLVED
|
||||
|
||||
Decision: **Ignore master.config.json for API keys**
|
||||
- All API keys come from `.env` file only
|
||||
- Even if keys exist in master.config.json now, they'll be removed in future epics
|
||||
- Don't reference master.config.json for any authentication
|
||||
- Only use .env for credentials
|
||||
|
||||
## Notes
|
||||
- Keep deployment simple for first iteration
|
||||
- Focus on reliability over speed
|
||||
- Auto-deploy is ON by default (deploy immediately after batch generation)
|
||||
- Manual CLI command available for re-deployment or testing
|
||||
- Comprehensive error reporting is critical
|
||||
- URL logging format is simple (one URL per line)
|
||||
- All API keys come from `.env` file, NOT master.config.json
|
||||
- Storage API authentication details will be determined during implementation
|
||||
|
||||
|
|
@ -16,9 +16,11 @@ AZURE_STORAGE_ACCOUNT_NAME=your_azure_account_name_here
|
|||
AZURE_STORAGE_ACCOUNT_KEY=your_azure_account_key_here
|
||||
|
||||
# Bunny.net Configuration
|
||||
# Account API key for creating/managing zones (required)
|
||||
BUNNY_ACCOUNT_API_KEY=your_bunny_account_api_key_here
|
||||
BUNNY_API_KEY=your_bunny_api_key_here
|
||||
BUNNY_STORAGE_ZONE=your_bunny_zone_here
|
||||
|
||||
# Note: For file uploads, the system uses per-zone storage_zone_password from the database
|
||||
# (set automatically when zones are created). No additional API key needed for uploads.
|
||||
|
||||
# Digital Ocean Spaces Configuration
|
||||
DO_SPACES_ACCESS_KEY=your_do_spaces_key_here
|
||||
|
|
|
|||
|
|
@ -0,0 +1,99 @@
|
|||
"""
|
||||
Migration script to add deployment tracking fields to generated_content table
|
||||
Story 4.1: Deploy Content to Cloud Storage
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.database.session import db_manager
|
||||
from sqlalchemy import text
|
||||
|
||||
|
||||
def migrate():
|
||||
"""Add deployment tracking fields to generated_content table"""
|
||||
|
||||
session = db_manager.get_session()
|
||||
|
||||
try:
|
||||
print("Starting migration: Add deployment tracking fields...")
|
||||
|
||||
print(" Adding deployed_url column...")
|
||||
session.execute(text("""
|
||||
ALTER TABLE generated_content
|
||||
ADD COLUMN deployed_url TEXT NULL
|
||||
"""))
|
||||
|
||||
print(" Adding deployed_at column...")
|
||||
session.execute(text("""
|
||||
ALTER TABLE generated_content
|
||||
ADD COLUMN deployed_at TIMESTAMP NULL
|
||||
"""))
|
||||
|
||||
print(" Creating index on deployed_at...")
|
||||
session.execute(text("""
|
||||
CREATE INDEX idx_generated_content_deployed
|
||||
ON generated_content(deployed_at)
|
||||
"""))
|
||||
|
||||
session.commit()
|
||||
|
||||
print("Migration completed successfully!")
|
||||
print("\nNew fields added:")
|
||||
print(" - deployed_url (TEXT, nullable)")
|
||||
print(" - deployed_at (TIMESTAMP, nullable, indexed)")
|
||||
|
||||
except Exception as e:
|
||||
session.rollback()
|
||||
print(f"Migration failed: {e}")
|
||||
raise
|
||||
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def rollback():
|
||||
"""Rollback migration (remove deployment fields)"""
|
||||
|
||||
session = db_manager.get_session()
|
||||
|
||||
try:
|
||||
print("Rolling back migration: Remove deployment tracking fields...")
|
||||
|
||||
print(" Dropping index...")
|
||||
session.execute(text("""
|
||||
DROP INDEX IF EXISTS idx_generated_content_deployed
|
||||
"""))
|
||||
|
||||
print(" Removing deployed_at column...")
|
||||
session.execute(text("""
|
||||
ALTER TABLE generated_content
|
||||
DROP COLUMN deployed_at
|
||||
"""))
|
||||
|
||||
print(" Removing deployed_url column...")
|
||||
session.execute(text("""
|
||||
ALTER TABLE generated_content
|
||||
DROP COLUMN deployed_url
|
||||
"""))
|
||||
|
||||
session.commit()
|
||||
|
||||
print("Rollback completed successfully!")
|
||||
|
||||
except Exception as e:
|
||||
session.rollback()
|
||||
print(f"Rollback failed: {e}")
|
||||
raise
|
||||
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "rollback":
|
||||
rollback()
|
||||
else:
|
||||
migrate()
|
||||
|
||||
|
|
@ -19,7 +19,10 @@ from src.ingestion.parser import CORAParser, CORAParseError
|
|||
from src.generation.ai_client import AIClient, PromptManager
|
||||
from src.generation.service import ContentGenerator
|
||||
from src.generation.batch_processor import BatchProcessor
|
||||
from src.database.repositories import GeneratedContentRepository
|
||||
from src.database.repositories import GeneratedContentRepository, SitePageRepository
|
||||
from src.deployment.bunny_storage import BunnyStorageClient, BunnyStorageError
|
||||
from src.deployment.deployment_service import DeploymentService
|
||||
from src.deployment.url_logger import URLLogger
|
||||
import os
|
||||
|
||||
|
||||
|
|
@ -967,5 +970,133 @@ def generate_batch(
|
|||
raise click.Abort()
|
||||
|
||||
|
||||
@app.command("deploy-batch")
|
||||
@click.option('--batch-id', '-b', required=True, type=int, help='Project/batch ID to deploy')
|
||||
@click.option('--admin-user', help='Admin username for authentication')
|
||||
@click.option('--admin-password', help='Admin password for authentication')
|
||||
@click.option('--continue-on-error', is_flag=True, default=True,
|
||||
help='Continue if file fails (default: True)')
|
||||
@click.option('--dry-run', is_flag=True, help='Preview what would be deployed')
|
||||
def deploy_batch(
|
||||
batch_id: int,
|
||||
admin_user: Optional[str],
|
||||
admin_password: Optional[str],
|
||||
continue_on_error: bool,
|
||||
dry_run: bool
|
||||
):
|
||||
"""Deploy all content in a batch to cloud storage"""
|
||||
try:
|
||||
if not admin_user or not admin_password:
|
||||
admin_user, admin_password = prompt_admin_credentials()
|
||||
|
||||
admin = authenticate_admin(admin_user, admin_password)
|
||||
if not admin:
|
||||
click.echo("Error: Authentication failed or insufficient permissions", err=True)
|
||||
raise click.Abort()
|
||||
|
||||
click.echo(f"Authenticated as: {admin.username} ({admin.role})")
|
||||
|
||||
session = db_manager.get_session()
|
||||
|
||||
try:
|
||||
project_repo = ProjectRepository(session)
|
||||
content_repo = GeneratedContentRepository(session)
|
||||
site_repo = SiteDeploymentRepository(session)
|
||||
page_repo = SitePageRepository(session)
|
||||
|
||||
project = project_repo.get_by_id(batch_id)
|
||||
if not project:
|
||||
click.echo(f"Error: Project/batch {batch_id} not found", err=True)
|
||||
raise click.Abort()
|
||||
|
||||
click.echo(f"\nDeploying batch: {project.name} (ID: {batch_id})")
|
||||
click.echo(f"Keyword: {project.main_keyword}")
|
||||
|
||||
articles = content_repo.get_by_project_id(batch_id)
|
||||
click.echo(f"Found {len(articles)} articles")
|
||||
|
||||
if dry_run:
|
||||
click.echo("\nDRY RUN MODE - No files will be uploaded\n")
|
||||
|
||||
for article in articles:
|
||||
if not article.site_deployment_id:
|
||||
click.echo(f"SKIP: Article {article.id} - No site assigned")
|
||||
continue
|
||||
|
||||
site = site_repo.get_by_id(article.site_deployment_id)
|
||||
if not site:
|
||||
click.echo(f"SKIP: Article {article.id} - Site not found")
|
||||
continue
|
||||
|
||||
from src.generation.url_generator import generate_file_path, generate_public_url
|
||||
file_path = generate_file_path(article)
|
||||
url = generate_public_url(site, file_path)
|
||||
|
||||
click.echo(f"WOULD DEPLOY: {article.title[:50]}")
|
||||
click.echo(f" File: {file_path}")
|
||||
click.echo(f" URL: {url}")
|
||||
|
||||
site_ids = set(a.site_deployment_id for a in articles if a.site_deployment_id)
|
||||
for site_id in site_ids:
|
||||
pages = page_repo.get_by_site(site_id)
|
||||
for page in pages:
|
||||
click.echo(f"WOULD DEPLOY: {page.page_type}.html")
|
||||
|
||||
click.echo("\nDry run complete. Use without --dry-run to actually deploy.")
|
||||
return
|
||||
|
||||
storage_client = BunnyStorageClient(max_retries=3)
|
||||
url_logger = URLLogger(logs_dir="deployment_logs")
|
||||
|
||||
deployment_service = DeploymentService(
|
||||
storage_client=storage_client,
|
||||
content_repo=content_repo,
|
||||
site_repo=site_repo,
|
||||
page_repo=page_repo,
|
||||
url_logger=url_logger
|
||||
)
|
||||
|
||||
click.echo(f"\nStarting deployment...")
|
||||
click.echo(f"Continue on error: {continue_on_error}")
|
||||
click.echo("")
|
||||
|
||||
results = deployment_service.deploy_batch(
|
||||
project_id=batch_id,
|
||||
continue_on_error=continue_on_error
|
||||
)
|
||||
|
||||
click.echo("\n" + "=" * 70)
|
||||
click.echo("Deployment Summary")
|
||||
click.echo("=" * 70)
|
||||
click.echo(f"Articles deployed: {results['articles_deployed']}")
|
||||
click.echo(f"Articles failed: {results['articles_failed']}")
|
||||
click.echo(f"Pages deployed: {results['pages_deployed']}")
|
||||
click.echo(f"Pages failed: {results['pages_failed']}")
|
||||
click.echo(f"Total time: {results['total_time']:.1f}s")
|
||||
|
||||
if results['errors']:
|
||||
click.echo("\nErrors:")
|
||||
for error in results['errors']:
|
||||
if error['type'] == 'article':
|
||||
click.echo(f" Article {error['id']} ({error.get('title', 'N/A')[:40]}): {error['error']}")
|
||||
else:
|
||||
click.echo(f" Page {error.get('page_type', 'N/A')} (Site {error.get('site_id')}): {error['error']}")
|
||||
|
||||
click.echo("=" * 70)
|
||||
|
||||
if results['articles_failed'] > 0 or results['pages_failed'] > 0:
|
||||
raise click.Abort()
|
||||
|
||||
except BunnyStorageError as e:
|
||||
click.echo(f"\nError: Storage upload failed - {e}", err=True)
|
||||
raise click.Abort()
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
except Exception as e:
|
||||
click.echo(f"Error deploying batch: {e}", err=True)
|
||||
raise click.Abort()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app()
|
||||
|
|
|
|||
|
|
@ -220,7 +220,14 @@ def get_ai_api_key() -> str:
|
|||
|
||||
|
||||
def get_bunny_account_api_key() -> str:
|
||||
"""Get the bunny.net Account API key from environment variables"""
|
||||
"""
|
||||
Get the bunny.net Account API key from environment variables
|
||||
|
||||
This key is used for account-level operations like creating storage zones
|
||||
and pull zones. It is NOT used for uploading files to storage.
|
||||
|
||||
For file uploads, use the per-zone storage_zone_password from the database.
|
||||
"""
|
||||
api_key = os.getenv("BUNNY_ACCOUNT_API_KEY")
|
||||
if not api_key:
|
||||
raise ValueError("BUNNY_ACCOUNT_API_KEY environment variable is required")
|
||||
|
|
|
|||
|
|
@ -138,6 +138,8 @@ class GeneratedContent(Base):
|
|||
formatted_html: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||
template_used: Mapped[Optional[str]] = mapped_column(String(50), nullable=True)
|
||||
site_deployment_id: Mapped[Optional[int]] = mapped_column(Integer, ForeignKey('site_deployments.id'), nullable=True, index=True)
|
||||
deployed_url: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||
deployed_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True, index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow, nullable=False)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime,
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ Concrete repository implementations
|
|||
"""
|
||||
|
||||
from typing import Optional, List, Dict, Any
|
||||
from datetime import datetime
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from src.core.config import get_config
|
||||
|
|
@ -491,6 +492,53 @@ class GeneratedContentRepository:
|
|||
self.session.refresh(content)
|
||||
return content
|
||||
|
||||
def mark_as_deployed(
|
||||
self,
|
||||
content_id: int,
|
||||
url: str,
|
||||
timestamp: Optional[datetime] = None
|
||||
) -> GeneratedContent:
|
||||
"""
|
||||
Mark content as deployed and store the public URL
|
||||
|
||||
Args:
|
||||
content_id: Content ID to mark as deployed
|
||||
url: Public URL where content is deployed
|
||||
timestamp: Deployment timestamp (defaults to now)
|
||||
|
||||
Returns:
|
||||
Updated GeneratedContent object
|
||||
|
||||
Raises:
|
||||
ValueError: If content not found
|
||||
"""
|
||||
content = self.get_by_id(content_id)
|
||||
if not content:
|
||||
raise ValueError(f"Content ID {content_id} not found")
|
||||
|
||||
content.deployed_url = url
|
||||
content.deployed_at = timestamp or datetime.now()
|
||||
content.status = 'deployed'
|
||||
|
||||
self.session.commit()
|
||||
self.session.refresh(content)
|
||||
return content
|
||||
|
||||
def get_deployed_content(self, project_id: int) -> List[GeneratedContent]:
|
||||
"""
|
||||
Get all deployed content for a project
|
||||
|
||||
Args:
|
||||
project_id: Project ID to filter by
|
||||
|
||||
Returns:
|
||||
List of deployed GeneratedContent records
|
||||
"""
|
||||
return self.session.query(GeneratedContent).filter(
|
||||
GeneratedContent.project_id == project_id,
|
||||
GeneratedContent.deployed_at.isnot(None)
|
||||
).all()
|
||||
|
||||
def delete(self, content_id: int) -> bool:
|
||||
"""Delete content by ID"""
|
||||
content = self.get_by_id(content_id)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,221 @@
|
|||
"""
|
||||
Bunny.net Storage API client for uploading files to storage zones
|
||||
"""
|
||||
|
||||
import requests
|
||||
import time
|
||||
import logging
|
||||
from typing import List, Optional
|
||||
from dataclasses import dataclass
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BunnyStorageError(Exception):
|
||||
"""Base exception for Bunny.net Storage API errors"""
|
||||
pass
|
||||
|
||||
|
||||
class BunnyStorageAuthError(BunnyStorageError):
|
||||
"""Authentication error with Bunny.net Storage API"""
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class UploadResult:
|
||||
"""Result from file upload"""
|
||||
success: bool
|
||||
file_path: str
|
||||
message: str
|
||||
|
||||
|
||||
class BunnyStorageClient:
|
||||
"""Client for uploading files to Bunny.net Storage Zones"""
|
||||
|
||||
BASE_URL = "https://storage.bunnycdn.com"
|
||||
|
||||
def __init__(self, max_retries: int = 3):
|
||||
"""
|
||||
Initialize Bunny.net Storage client
|
||||
|
||||
Args:
|
||||
max_retries: Maximum number of retry attempts for failed uploads
|
||||
"""
|
||||
self.max_retries = max_retries
|
||||
self.session = requests.Session()
|
||||
|
||||
def upload_file(
|
||||
self,
|
||||
zone_name: str,
|
||||
zone_password: str,
|
||||
file_path: str,
|
||||
content: str,
|
||||
content_type: str = 'text/html'
|
||||
) -> UploadResult:
|
||||
"""
|
||||
Upload a file to Bunny.net storage zone
|
||||
|
||||
Args:
|
||||
zone_name: Storage zone name
|
||||
zone_password: Storage zone password (from database)
|
||||
file_path: Path within storage zone (e.g., 'my-article.html')
|
||||
content: File content to upload
|
||||
content_type: MIME type (default: text/html)
|
||||
|
||||
Returns:
|
||||
UploadResult with success status and message
|
||||
|
||||
Raises:
|
||||
BunnyStorageAuthError: If authentication fails
|
||||
BunnyStorageError: For other API errors
|
||||
"""
|
||||
url = f"{self.BASE_URL}/{zone_name}/{file_path}"
|
||||
headers = {
|
||||
"AccessKey": zone_password,
|
||||
"Content-Type": content_type
|
||||
}
|
||||
|
||||
for attempt in range(self.max_retries):
|
||||
try:
|
||||
response = self.session.put(
|
||||
url,
|
||||
data=content.encode('utf-8'),
|
||||
headers=headers,
|
||||
timeout=60
|
||||
)
|
||||
|
||||
if response.status_code == 401:
|
||||
raise BunnyStorageAuthError(
|
||||
f"Authentication failed for zone '{zone_name}'. Check storage_zone_password."
|
||||
)
|
||||
|
||||
if response.status_code == 201:
|
||||
logger.info(f"Uploaded {file_path} to {zone_name}")
|
||||
return UploadResult(
|
||||
success=True,
|
||||
file_path=file_path,
|
||||
message="Upload successful"
|
||||
)
|
||||
|
||||
if response.status_code >= 400:
|
||||
error_msg = f"Upload failed with status {response.status_code}"
|
||||
if response.text:
|
||||
error_msg += f": {response.text}"
|
||||
|
||||
if attempt < self.max_retries - 1:
|
||||
wait_time = 2 ** attempt
|
||||
logger.warning(f"{error_msg}. Retrying in {wait_time}s (attempt {attempt + 1}/{self.max_retries})")
|
||||
time.sleep(wait_time)
|
||||
continue
|
||||
|
||||
raise BunnyStorageError(error_msg)
|
||||
|
||||
return UploadResult(
|
||||
success=True,
|
||||
file_path=file_path,
|
||||
message=f"Upload completed with status {response.status_code}"
|
||||
)
|
||||
|
||||
except requests.exceptions.Timeout:
|
||||
if attempt < self.max_retries - 1:
|
||||
wait_time = 2 ** attempt
|
||||
logger.warning(f"Upload timeout. Retrying in {wait_time}s (attempt {attempt + 1}/{self.max_retries})")
|
||||
time.sleep(wait_time)
|
||||
continue
|
||||
raise BunnyStorageError(f"Upload timeout after {self.max_retries} attempts")
|
||||
|
||||
except requests.exceptions.ConnectionError as e:
|
||||
if attempt < self.max_retries - 1:
|
||||
wait_time = 2 ** attempt
|
||||
logger.warning(f"Connection error: {e}. Retrying in {wait_time}s (attempt {attempt + 1}/{self.max_retries})")
|
||||
time.sleep(wait_time)
|
||||
continue
|
||||
raise BunnyStorageError(f"Connection error after {self.max_retries} attempts: {e}")
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
raise BunnyStorageError(f"Upload failed: {str(e)}")
|
||||
|
||||
raise BunnyStorageError(f"Upload failed after {self.max_retries} attempts")
|
||||
|
||||
def file_exists(
|
||||
self,
|
||||
zone_name: str,
|
||||
zone_password: str,
|
||||
file_path: str
|
||||
) -> bool:
|
||||
"""
|
||||
Check if a file exists in storage zone
|
||||
|
||||
Args:
|
||||
zone_name: Storage zone name
|
||||
zone_password: Storage zone password
|
||||
file_path: Path within storage zone
|
||||
|
||||
Returns:
|
||||
True if file exists, False otherwise
|
||||
|
||||
Raises:
|
||||
BunnyStorageError: For API errors (excluding 404)
|
||||
"""
|
||||
url = f"{self.BASE_URL}/{zone_name}/{file_path}"
|
||||
headers = {"AccessKey": zone_password}
|
||||
|
||||
try:
|
||||
response = self.session.head(url, headers=headers, timeout=30)
|
||||
|
||||
if response.status_code == 401:
|
||||
raise BunnyStorageAuthError(f"Authentication failed for zone '{zone_name}'")
|
||||
|
||||
if response.status_code == 404:
|
||||
return False
|
||||
|
||||
if response.status_code == 200:
|
||||
return True
|
||||
|
||||
raise BunnyStorageError(f"Unexpected status code {response.status_code} when checking file existence")
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
raise BunnyStorageError(f"Failed to check file existence: {str(e)}")
|
||||
|
||||
def list_files(
|
||||
self,
|
||||
zone_name: str,
|
||||
zone_password: str,
|
||||
prefix: str = ''
|
||||
) -> List[str]:
|
||||
"""
|
||||
List files in storage zone
|
||||
|
||||
Args:
|
||||
zone_name: Storage zone name
|
||||
zone_password: Storage zone password
|
||||
prefix: Optional path prefix to filter results
|
||||
|
||||
Returns:
|
||||
List of file paths
|
||||
|
||||
Raises:
|
||||
BunnyStorageError: For API errors
|
||||
"""
|
||||
url = f"{self.BASE_URL}/{zone_name}/{prefix}"
|
||||
headers = {"AccessKey": zone_password}
|
||||
|
||||
try:
|
||||
response = self.session.get(url, headers=headers, timeout=30)
|
||||
|
||||
if response.status_code == 401:
|
||||
raise BunnyStorageAuthError(f"Authentication failed for zone '{zone_name}'")
|
||||
|
||||
if response.status_code != 200:
|
||||
raise BunnyStorageError(f"List files failed with status {response.status_code}")
|
||||
|
||||
data = response.json()
|
||||
|
||||
if isinstance(data, list):
|
||||
return [item.get('ObjectName', '') for item in data if item.get('ObjectName')]
|
||||
|
||||
return []
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
raise BunnyStorageError(f"Failed to list files: {str(e)}")
|
||||
|
||||
|
|
@ -0,0 +1,229 @@
|
|||
"""
|
||||
Deployment service for uploading content to Bunny.net storage
|
||||
Story 4.1: Deploy Content to Cloud Storage
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Dict, Any, List
|
||||
from datetime import datetime
|
||||
from src.deployment.bunny_storage import BunnyStorageClient, BunnyStorageError
|
||||
from src.deployment.url_logger import URLLogger
|
||||
from src.database.repositories import (
|
||||
GeneratedContentRepository,
|
||||
SitePageRepository,
|
||||
SiteDeploymentRepository
|
||||
)
|
||||
from src.database.models import GeneratedContent, SitePage, SiteDeployment
|
||||
from src.generation.url_generator import (
|
||||
generate_public_url,
|
||||
generate_file_path,
|
||||
generate_page_file_path
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DeploymentService:
|
||||
"""Service for deploying content to cloud storage"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_client: BunnyStorageClient,
|
||||
content_repo: GeneratedContentRepository,
|
||||
site_repo: SiteDeploymentRepository,
|
||||
page_repo: SitePageRepository,
|
||||
url_logger: URLLogger
|
||||
):
|
||||
"""
|
||||
Initialize deployment service
|
||||
|
||||
Args:
|
||||
storage_client: BunnyStorageClient for uploads
|
||||
content_repo: Repository for content records
|
||||
site_repo: Repository for site deployments
|
||||
page_repo: Repository for boilerplate pages
|
||||
url_logger: URLLogger for tracking deployed URLs
|
||||
"""
|
||||
self.storage = storage_client
|
||||
self.content_repo = content_repo
|
||||
self.site_repo = site_repo
|
||||
self.page_repo = page_repo
|
||||
self.url_logger = url_logger
|
||||
|
||||
def deploy_batch(
|
||||
self,
|
||||
project_id: int,
|
||||
continue_on_error: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Deploy all content for a project/batch
|
||||
|
||||
Args:
|
||||
project_id: Project ID to deploy
|
||||
continue_on_error: If True, continue on individual file failures
|
||||
|
||||
Returns:
|
||||
Dict with deployment statistics:
|
||||
{
|
||||
'articles_deployed': 10,
|
||||
'articles_failed': 1,
|
||||
'pages_deployed': 6,
|
||||
'pages_failed': 0,
|
||||
'total_time': 45.2,
|
||||
'errors': [...]
|
||||
}
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
results = {
|
||||
'articles_deployed': 0,
|
||||
'articles_failed': 0,
|
||||
'pages_deployed': 0,
|
||||
'pages_failed': 0,
|
||||
'errors': []
|
||||
}
|
||||
|
||||
articles = self.content_repo.get_by_project_id(project_id)
|
||||
logger.info(f"Found {len(articles)} articles to deploy for project {project_id}")
|
||||
|
||||
for article in articles:
|
||||
if not article.site_deployment_id:
|
||||
logger.warning(f"Article {article.id} has no site assigned, skipping")
|
||||
continue
|
||||
|
||||
try:
|
||||
site = self.site_repo.get_by_id(article.site_deployment_id)
|
||||
if not site:
|
||||
raise ValueError(f"Site {article.site_deployment_id} not found")
|
||||
|
||||
url = self.deploy_article(article, site)
|
||||
|
||||
self.url_logger.log_article_url(url, article.tier)
|
||||
|
||||
self.content_repo.mark_as_deployed(article.id, url)
|
||||
|
||||
results['articles_deployed'] += 1
|
||||
logger.info(f"Deployed article {article.id} to {url}")
|
||||
|
||||
except Exception as e:
|
||||
results['articles_failed'] += 1
|
||||
results['errors'].append({
|
||||
'type': 'article',
|
||||
'id': article.id,
|
||||
'title': article.title,
|
||||
'error': str(e)
|
||||
})
|
||||
logger.error(f"Failed to deploy article {article.id}: {e}")
|
||||
|
||||
if not continue_on_error:
|
||||
raise
|
||||
|
||||
site_ids = set(a.site_deployment_id for a in articles if a.site_deployment_id)
|
||||
|
||||
for site_id in site_ids:
|
||||
site = self.site_repo.get_by_id(site_id)
|
||||
pages = self.page_repo.get_by_site(site_id)
|
||||
|
||||
if not pages:
|
||||
logger.debug(f"Site {site_id} has no boilerplate pages, skipping")
|
||||
continue
|
||||
|
||||
logger.info(f"Found {len(pages)} boilerplate pages for site {site_id}")
|
||||
|
||||
for page in pages:
|
||||
try:
|
||||
url = self.deploy_boilerplate_page(page, site)
|
||||
results['pages_deployed'] += 1
|
||||
logger.info(f"Deployed page {page.page_type} to {url}")
|
||||
|
||||
except Exception as e:
|
||||
results['pages_failed'] += 1
|
||||
results['errors'].append({
|
||||
'type': 'page',
|
||||
'site_id': site_id,
|
||||
'page_type': page.page_type,
|
||||
'error': str(e)
|
||||
})
|
||||
logger.error(f"Failed to deploy page {page.page_type} for site {site_id}: {e}")
|
||||
|
||||
if not continue_on_error:
|
||||
raise
|
||||
|
||||
results['total_time'] = time.time() - start_time
|
||||
|
||||
logger.info(
|
||||
f"Deployment complete: {results['articles_deployed']} articles, "
|
||||
f"{results['pages_deployed']} pages, "
|
||||
f"{results['articles_failed']} article failures, "
|
||||
f"{results['pages_failed']} page failures, "
|
||||
f"{results['total_time']:.1f}s"
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
def deploy_article(self, article: GeneratedContent, site: SiteDeployment) -> str:
|
||||
"""
|
||||
Deploy a single article, return public URL
|
||||
|
||||
Args:
|
||||
article: GeneratedContent record with HTML
|
||||
site: SiteDeployment record with storage credentials
|
||||
|
||||
Returns:
|
||||
Public URL of deployed article
|
||||
|
||||
Raises:
|
||||
ValueError: If article has no formatted_html
|
||||
BunnyStorageError: If upload fails
|
||||
"""
|
||||
if not article.formatted_html:
|
||||
raise ValueError(f"Article {article.id} has no formatted_html to deploy")
|
||||
|
||||
file_path = generate_file_path(article)
|
||||
url = generate_public_url(site, file_path)
|
||||
|
||||
self.storage.upload_file(
|
||||
zone_name=site.storage_zone_name,
|
||||
zone_password=site.storage_zone_password,
|
||||
file_path=file_path,
|
||||
content=article.formatted_html,
|
||||
content_type='text/html'
|
||||
)
|
||||
|
||||
return url
|
||||
|
||||
def deploy_boilerplate_page(self, page: SitePage, site: SiteDeployment) -> str:
|
||||
"""
|
||||
Deploy a boilerplate page, return public URL
|
||||
|
||||
Args:
|
||||
page: SitePage record with HTML content
|
||||
site: SiteDeployment record with storage credentials
|
||||
|
||||
Returns:
|
||||
Public URL of deployed page
|
||||
|
||||
Raises:
|
||||
ValueError: If page has no content
|
||||
BunnyStorageError: If upload fails
|
||||
|
||||
Note:
|
||||
Uses stored HTML from page.content (from Story 3.4)
|
||||
"""
|
||||
if not page.content:
|
||||
raise ValueError(f"Page {page.page_type} for site {site.id} has no content to deploy")
|
||||
|
||||
file_path = generate_page_file_path(page)
|
||||
url = generate_public_url(site, file_path)
|
||||
|
||||
self.storage.upload_file(
|
||||
zone_name=site.storage_zone_name,
|
||||
zone_password=site.storage_zone_password,
|
||||
file_path=file_path,
|
||||
content=page.content,
|
||||
content_type='text/html'
|
||||
)
|
||||
|
||||
return url
|
||||
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
"""
|
||||
URL logger for tier-segregated deployment URL tracking
|
||||
Story 4.1: Deploy Content to Cloud Storage
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Set
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class URLLogger:
|
||||
"""Logs deployed article URLs to tier-segregated daily text files"""
|
||||
|
||||
def __init__(self, logs_dir: str = "deployment_logs"):
|
||||
"""
|
||||
Initialize URL logger
|
||||
|
||||
Args:
|
||||
logs_dir: Directory for log files (created if doesn't exist)
|
||||
"""
|
||||
self.logs_dir = Path(logs_dir)
|
||||
self.logs_dir.mkdir(exist_ok=True)
|
||||
logger.info(f"URL logger initialized with logs_dir: {self.logs_dir}")
|
||||
|
||||
def log_article_url(self, url: str, tier: str, date: datetime = None):
|
||||
"""
|
||||
Log an article URL to the appropriate tier-segregated file
|
||||
|
||||
Args:
|
||||
url: Public URL of deployed article
|
||||
tier: Tier string (e.g., 'tier1', 'tier2', 'tier3')
|
||||
date: Deployment date (defaults to now)
|
||||
|
||||
Note:
|
||||
- Tier 1 articles go to YYYY-MM-DD_tier1_urls.txt
|
||||
- Tier 2+ articles go to YYYY-MM-DD_other_tiers_urls.txt
|
||||
- Duplicate URLs are automatically skipped
|
||||
- Boilerplate pages should NOT be logged here
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.now()
|
||||
|
||||
tier_num = self._extract_tier_number(tier)
|
||||
if tier_num == 1:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_tier1_urls.txt"
|
||||
else:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_other_tiers_urls.txt"
|
||||
|
||||
filepath = self.logs_dir / filename
|
||||
|
||||
existing = self.get_existing_urls(tier, date)
|
||||
if url in existing:
|
||||
logger.debug(f"URL already logged, skipping: {url}")
|
||||
return
|
||||
|
||||
with open(filepath, 'a', encoding='utf-8') as f:
|
||||
f.write(f"{url}\n")
|
||||
|
||||
logger.info(f"Logged URL to {filename}: {url}")
|
||||
|
||||
def get_existing_urls(self, tier: str, date: datetime = None) -> Set[str]:
|
||||
"""
|
||||
Get existing URLs from log file to prevent duplicates
|
||||
|
||||
This is critical for preventing duplicate entries when:
|
||||
- Auto-deployment runs, then manual re-run happens
|
||||
- Deployment fails partway and is restarted
|
||||
|
||||
Args:
|
||||
tier: Tier string (e.g., 'tier1', 'tier2')
|
||||
date: Date to check (defaults to now)
|
||||
|
||||
Returns:
|
||||
Set of URLs already logged in the file
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.now()
|
||||
|
||||
tier_num = self._extract_tier_number(tier)
|
||||
if tier_num == 1:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_tier1_urls.txt"
|
||||
else:
|
||||
filename = f"{date.strftime('%Y-%m-%d')}_other_tiers_urls.txt"
|
||||
|
||||
filepath = self.logs_dir / filename
|
||||
|
||||
if not filepath.exists():
|
||||
return set()
|
||||
|
||||
with open(filepath, 'r', encoding='utf-8') as f:
|
||||
return set(line.strip() for line in f if line.strip())
|
||||
|
||||
def _extract_tier_number(self, tier: str) -> int:
|
||||
"""
|
||||
Extract tier number from tier string
|
||||
|
||||
Args:
|
||||
tier: Tier string like 'tier1', 'tier2', 'tier3'
|
||||
|
||||
Returns:
|
||||
Integer tier number
|
||||
|
||||
Examples:
|
||||
'tier1' -> 1
|
||||
'tier2' -> 2
|
||||
'tier3' -> 3
|
||||
"""
|
||||
digits = ''.join(c for c in tier if c.isdigit())
|
||||
return int(digits) if digits else 1
|
||||
|
||||
|
|
@ -4,14 +4,18 @@ Batch processor for content generation jobs
|
|||
|
||||
from typing import Dict, Any, Optional
|
||||
import click
|
||||
import os
|
||||
from src.generation.service import ContentGenerator
|
||||
from src.generation.job_config import JobConfig, Job, TierConfig
|
||||
from src.generation.deployment_assignment import validate_and_resolve_targets, assign_site_for_article
|
||||
from src.database.repositories import GeneratedContentRepository, ProjectRepository, SiteDeploymentRepository, ArticleLinkRepository
|
||||
from src.database.repositories import GeneratedContentRepository, ProjectRepository, SiteDeploymentRepository, ArticleLinkRepository, SitePageRepository
|
||||
from src.generation.url_generator import generate_urls_for_batch
|
||||
from src.interlinking.tiered_links import find_tiered_links
|
||||
from src.interlinking.content_injection import inject_interlinks
|
||||
from src.generation.site_assignment import assign_sites_to_batch
|
||||
from src.deployment.bunny_storage import BunnyStorageClient
|
||||
from src.deployment.deployment_service import DeploymentService
|
||||
from src.deployment.url_logger import URLLogger
|
||||
|
||||
|
||||
class BatchProcessor:
|
||||
|
|
@ -41,7 +45,8 @@ class BatchProcessor:
|
|||
self,
|
||||
job_file_path: str,
|
||||
debug: bool = False,
|
||||
continue_on_error: bool = False
|
||||
continue_on_error: bool = False,
|
||||
auto_deploy: bool = True
|
||||
):
|
||||
"""
|
||||
Process all jobs in job file
|
||||
|
|
@ -50,6 +55,7 @@ class BatchProcessor:
|
|||
job_file_path: Path to job JSON file
|
||||
debug: If True, save AI responses to debug_output/
|
||||
continue_on_error: If True, continue on article generation failure
|
||||
auto_deploy: If True, deploy to cloud storage after generation (default: True)
|
||||
"""
|
||||
job_config = JobConfig(job_file_path)
|
||||
jobs = job_config.get_jobs()
|
||||
|
|
@ -58,7 +64,7 @@ class BatchProcessor:
|
|||
|
||||
for job_idx, job in enumerate(jobs, 1):
|
||||
try:
|
||||
self._process_single_job(job, job_idx, debug, continue_on_error)
|
||||
self._process_single_job(job, job_idx, debug, continue_on_error, auto_deploy)
|
||||
self.stats["processed_jobs"] += 1
|
||||
except Exception as e:
|
||||
click.echo(f"Error processing job {job_idx}: {e}")
|
||||
|
|
@ -72,7 +78,8 @@ class BatchProcessor:
|
|||
job: Job,
|
||||
job_idx: int,
|
||||
debug: bool,
|
||||
continue_on_error: bool
|
||||
continue_on_error: bool,
|
||||
auto_deploy: bool = True
|
||||
):
|
||||
"""Process a single job"""
|
||||
project = self.project_repo.get_by_id(job.project_id)
|
||||
|
|
@ -107,6 +114,15 @@ class BatchProcessor:
|
|||
debug,
|
||||
continue_on_error
|
||||
)
|
||||
|
||||
if auto_deploy:
|
||||
try:
|
||||
self._deploy_job(job.project_id, continue_on_error)
|
||||
except Exception as e:
|
||||
click.echo(f" Warning: Auto-deployment failed: {e}")
|
||||
if debug:
|
||||
import traceback
|
||||
click.echo(f" Traceback: {traceback.format_exc()}")
|
||||
|
||||
def _process_tier(
|
||||
self,
|
||||
|
|
@ -370,6 +386,44 @@ class BatchProcessor:
|
|||
click.echo(f" Applied templates to {template_count}/{len(content_records)} articles")
|
||||
click.echo(f" {tier_name}: Post-processing complete")
|
||||
|
||||
def _deploy_job(self, project_id: int, continue_on_error: bool):
|
||||
"""
|
||||
Deploy all content for a project to cloud storage
|
||||
|
||||
Args:
|
||||
project_id: Project ID to deploy
|
||||
continue_on_error: If True, continue on individual file failures
|
||||
|
||||
Note:
|
||||
Uses per-zone storage_zone_password from database for authentication.
|
||||
No API key from .env is required for uploads.
|
||||
"""
|
||||
click.echo(f"\n Deployment: Starting automatic deployment for project {project_id}...")
|
||||
|
||||
storage_client = BunnyStorageClient(max_retries=3)
|
||||
url_logger = URLLogger(logs_dir="deployment_logs")
|
||||
page_repo = SitePageRepository(self.content_repo.session)
|
||||
|
||||
deployment_service = DeploymentService(
|
||||
storage_client=storage_client,
|
||||
content_repo=self.content_repo,
|
||||
site_repo=self.site_deployment_repo,
|
||||
page_repo=page_repo,
|
||||
url_logger=url_logger
|
||||
)
|
||||
|
||||
results = deployment_service.deploy_batch(
|
||||
project_id=project_id,
|
||||
continue_on_error=continue_on_error
|
||||
)
|
||||
|
||||
click.echo(f" Deployment: {results['articles_deployed']} articles, {results['pages_deployed']} pages deployed")
|
||||
|
||||
if results['articles_failed'] > 0 or results['pages_failed'] > 0:
|
||||
click.echo(f" Deployment: {results['articles_failed']} article failures, {results['pages_failed']} page failures")
|
||||
|
||||
click.echo(f" Deployment: Complete in {results['total_time']:.1f}s")
|
||||
|
||||
def _print_summary(self):
|
||||
"""Print job processing summary"""
|
||||
click.echo("\n" + "="*60)
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ URL generation logic for generated content
|
|||
import re
|
||||
import logging
|
||||
from typing import List, Dict
|
||||
from src.database.models import GeneratedContent
|
||||
from src.database.models import GeneratedContent, SiteDeployment, SitePage
|
||||
from src.database.repositories import SiteDeploymentRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -91,3 +91,60 @@ def generate_urls_for_batch(
|
|||
|
||||
return url_mappings
|
||||
|
||||
|
||||
def generate_public_url(site: SiteDeployment, file_path: str) -> str:
|
||||
"""
|
||||
Generate full public URL for a file path on a site
|
||||
|
||||
Args:
|
||||
site: SiteDeployment record with hostname information
|
||||
file_path: File path within storage zone (e.g., 'my-article.html')
|
||||
|
||||
Returns:
|
||||
Full HTTPS URL (e.g., 'https://example.com/my-article.html')
|
||||
|
||||
Examples:
|
||||
site with custom_hostname='www.example.com', file_path='about.html'
|
||||
-> 'https://www.example.com/about.html'
|
||||
|
||||
site with pull_zone_bcdn_hostname='mysite.b-cdn.net', file_path='article.html'
|
||||
-> 'https://mysite.b-cdn.net/article.html'
|
||||
"""
|
||||
hostname = site.custom_hostname or site.pull_zone_bcdn_hostname
|
||||
return f"https://{hostname}/{file_path}"
|
||||
|
||||
|
||||
def generate_file_path(content: GeneratedContent) -> str:
|
||||
"""
|
||||
Generate storage file path for content
|
||||
|
||||
Args:
|
||||
content: GeneratedContent record
|
||||
|
||||
Returns:
|
||||
File path with .html extension (e.g., 'my-article-slug.html')
|
||||
|
||||
Note:
|
||||
Uses title-based slug generation with fallback to content_id
|
||||
"""
|
||||
slug = generate_slug(content.title)
|
||||
|
||||
if not slug or slug == "article":
|
||||
slug = f"article-{content.id}"
|
||||
logger.warning(f"Empty slug for content {content.id}, using fallback: {slug}")
|
||||
|
||||
return f"{slug}.html"
|
||||
|
||||
|
||||
def generate_page_file_path(page: SitePage) -> str:
|
||||
"""
|
||||
Generate storage file path for boilerplate page
|
||||
|
||||
Args:
|
||||
page: SitePage record
|
||||
|
||||
Returns:
|
||||
File path with .html extension (e.g., 'about.html', 'contact.html')
|
||||
"""
|
||||
return f"{page.page_type}.html"
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,317 @@
|
|||
"""
|
||||
Integration tests for Story 4.1: Deploy Content to Cloud Storage
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
from datetime import datetime
|
||||
from src.deployment.bunny_storage import BunnyStorageClient, UploadResult, BunnyStorageError
|
||||
from src.deployment.url_logger import URLLogger
|
||||
from src.deployment.deployment_service import DeploymentService
|
||||
from src.database.models import GeneratedContent, SiteDeployment, SitePage
|
||||
from src.generation.url_generator import (
|
||||
generate_public_url,
|
||||
generate_file_path,
|
||||
generate_page_file_path,
|
||||
generate_slug
|
||||
)
|
||||
|
||||
|
||||
class TestURLGenerator:
|
||||
"""Test URL generation functions"""
|
||||
|
||||
def test_generate_slug(self):
|
||||
"""Test slug generation from titles"""
|
||||
assert generate_slug("How to Fix Your Engine") == "how-to-fix-your-engine"
|
||||
assert generate_slug("10 Best SEO Tips for 2024!") == "10-best-seo-tips-for-2024"
|
||||
assert generate_slug("C++ Programming Guide") == "c-programming-guide"
|
||||
assert generate_slug("Multiple Spaces") == "multiple-spaces"
|
||||
|
||||
def test_generate_public_url(self):
|
||||
"""Test public URL generation"""
|
||||
site = Mock(spec=SiteDeployment)
|
||||
site.custom_hostname = "www.example.com"
|
||||
site.pull_zone_bcdn_hostname = "example.b-cdn.net"
|
||||
|
||||
url = generate_public_url(site, "my-article.html")
|
||||
assert url == "https://www.example.com/my-article.html"
|
||||
|
||||
site.custom_hostname = None
|
||||
url = generate_public_url(site, "about.html")
|
||||
assert url == "https://example.b-cdn.net/about.html"
|
||||
|
||||
def test_generate_file_path(self):
|
||||
"""Test file path generation for articles"""
|
||||
content = Mock(spec=GeneratedContent)
|
||||
content.id = 42
|
||||
content.title = "How to Fix Your Engine"
|
||||
|
||||
path = generate_file_path(content)
|
||||
assert path == "how-to-fix-your-engine.html"
|
||||
|
||||
def test_generate_page_file_path(self):
|
||||
"""Test file path generation for boilerplate pages"""
|
||||
page = Mock(spec=SitePage)
|
||||
page.page_type = "about"
|
||||
|
||||
path = generate_page_file_path(page)
|
||||
assert path == "about.html"
|
||||
|
||||
|
||||
class TestURLLogger:
|
||||
"""Test URL logging functionality"""
|
||||
|
||||
def test_tier_number_extraction(self, tmp_path):
|
||||
"""Test extracting tier numbers from tier strings"""
|
||||
logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
assert logger._extract_tier_number("tier1") == 1
|
||||
assert logger._extract_tier_number("tier2") == 2
|
||||
assert logger._extract_tier_number("tier3") == 3
|
||||
|
||||
def test_log_article_url(self, tmp_path):
|
||||
"""Test logging URLs to tier-segregated files"""
|
||||
logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
test_date = datetime(2025, 10, 22)
|
||||
|
||||
logger.log_article_url("https://example.com/article1.html", "tier1", test_date)
|
||||
logger.log_article_url("https://example.com/article2.html", "tier2", test_date)
|
||||
|
||||
tier1_file = tmp_path / "2025-10-22_tier1_urls.txt"
|
||||
tier2_file = tmp_path / "2025-10-22_other_tiers_urls.txt"
|
||||
|
||||
assert tier1_file.exists()
|
||||
assert tier2_file.exists()
|
||||
|
||||
with open(tier1_file) as f:
|
||||
urls = f.read().strip().split('\n')
|
||||
assert "https://example.com/article1.html" in urls
|
||||
|
||||
with open(tier2_file) as f:
|
||||
urls = f.read().strip().split('\n')
|
||||
assert "https://example.com/article2.html" in urls
|
||||
|
||||
def test_duplicate_prevention(self, tmp_path):
|
||||
"""Test that duplicate URLs are not logged twice"""
|
||||
logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
test_date = datetime(2025, 10, 22)
|
||||
url = "https://example.com/article1.html"
|
||||
|
||||
logger.log_article_url(url, "tier1", test_date)
|
||||
logger.log_article_url(url, "tier1", test_date)
|
||||
|
||||
tier1_file = tmp_path / "2025-10-22_tier1_urls.txt"
|
||||
|
||||
with open(tier1_file) as f:
|
||||
urls = [line.strip() for line in f if line.strip()]
|
||||
assert urls.count(url) == 1
|
||||
|
||||
def test_get_existing_urls(self, tmp_path):
|
||||
"""Test retrieving existing URLs from log files"""
|
||||
logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
test_date = datetime(2025, 10, 22)
|
||||
|
||||
logger.log_article_url("https://example.com/article1.html", "tier1", test_date)
|
||||
logger.log_article_url("https://example.com/article2.html", "tier1", test_date)
|
||||
|
||||
existing = logger.get_existing_urls("tier1", test_date)
|
||||
|
||||
assert len(existing) == 2
|
||||
assert "https://example.com/article1.html" in existing
|
||||
assert "https://example.com/article2.html" in existing
|
||||
|
||||
|
||||
class TestBunnyStorageClient:
|
||||
"""Test Bunny Storage client"""
|
||||
|
||||
@patch('src.deployment.bunny_storage.requests.Session')
|
||||
def test_upload_file_success(self, mock_session_class):
|
||||
"""Test successful file upload"""
|
||||
mock_session = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 201
|
||||
mock_session.put.return_value = mock_response
|
||||
mock_session_class.return_value = mock_session
|
||||
|
||||
client = BunnyStorageClient(max_retries=3)
|
||||
|
||||
result = client.upload_file(
|
||||
zone_name="test-zone",
|
||||
zone_password="test-password",
|
||||
file_path="test.html",
|
||||
content="<html>Test</html>"
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.file_path == "test.html"
|
||||
|
||||
mock_session.put.assert_called_once()
|
||||
call_args = mock_session.put.call_args
|
||||
assert call_args[0][0] == "https://storage.bunnycdn.com/test-zone/test.html"
|
||||
assert call_args[1]['headers']['AccessKey'] == "test-password"
|
||||
|
||||
@patch('src.deployment.bunny_storage.requests.Session')
|
||||
def test_upload_file_auth_error(self, mock_session_class):
|
||||
"""Test authentication error handling"""
|
||||
mock_session = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 401
|
||||
mock_session.put.return_value = mock_response
|
||||
mock_session_class.return_value = mock_session
|
||||
|
||||
client = BunnyStorageClient(max_retries=3)
|
||||
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
client.upload_file(
|
||||
zone_name="test-zone",
|
||||
zone_password="bad-password",
|
||||
file_path="test.html",
|
||||
content="<html>Test</html>"
|
||||
)
|
||||
|
||||
assert "Authentication failed" in str(exc_info.value)
|
||||
|
||||
|
||||
class TestDeploymentService:
|
||||
"""Test deployment service integration"""
|
||||
|
||||
def test_deploy_article(self, tmp_path):
|
||||
"""Test deploying a single article"""
|
||||
mock_storage = Mock(spec=BunnyStorageClient)
|
||||
mock_storage.upload_file.return_value = UploadResult(
|
||||
success=True,
|
||||
file_path="test-article.html",
|
||||
message="Success"
|
||||
)
|
||||
|
||||
mock_content_repo = Mock()
|
||||
mock_site_repo = Mock()
|
||||
mock_page_repo = Mock()
|
||||
|
||||
url_logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
service = DeploymentService(
|
||||
storage_client=mock_storage,
|
||||
content_repo=mock_content_repo,
|
||||
site_repo=mock_site_repo,
|
||||
page_repo=mock_page_repo,
|
||||
url_logger=url_logger
|
||||
)
|
||||
|
||||
article = Mock(spec=GeneratedContent)
|
||||
article.id = 1
|
||||
article.title = "Test Article"
|
||||
article.formatted_html = "<html>Test Content</html>"
|
||||
|
||||
site = Mock(spec=SiteDeployment)
|
||||
site.id = 1
|
||||
site.custom_hostname = "www.example.com"
|
||||
site.storage_zone_name = "test-zone"
|
||||
site.storage_zone_password = "test-password"
|
||||
|
||||
url = service.deploy_article(article, site)
|
||||
|
||||
assert url == "https://www.example.com/test-article.html"
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
def test_deploy_boilerplate_page(self, tmp_path):
|
||||
"""Test deploying a boilerplate page"""
|
||||
mock_storage = Mock(spec=BunnyStorageClient)
|
||||
mock_storage.upload_file.return_value = UploadResult(
|
||||
success=True,
|
||||
file_path="about.html",
|
||||
message="Success"
|
||||
)
|
||||
|
||||
mock_content_repo = Mock()
|
||||
mock_site_repo = Mock()
|
||||
mock_page_repo = Mock()
|
||||
|
||||
url_logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
service = DeploymentService(
|
||||
storage_client=mock_storage,
|
||||
content_repo=mock_content_repo,
|
||||
site_repo=mock_site_repo,
|
||||
page_repo=mock_page_repo,
|
||||
url_logger=url_logger
|
||||
)
|
||||
|
||||
page = Mock(spec=SitePage)
|
||||
page.page_type = "about"
|
||||
page.content = "<html>About Page</html>"
|
||||
|
||||
site = Mock(spec=SiteDeployment)
|
||||
site.id = 1
|
||||
site.custom_hostname = "www.example.com"
|
||||
site.storage_zone_name = "test-zone"
|
||||
site.storage_zone_password = "test-password"
|
||||
|
||||
url = service.deploy_boilerplate_page(page, site)
|
||||
|
||||
assert url == "https://www.example.com/about.html"
|
||||
mock_storage.upload_file.assert_called_once()
|
||||
|
||||
def test_deploy_batch(self, tmp_path):
|
||||
"""Test deploying an entire batch"""
|
||||
mock_storage = Mock(spec=BunnyStorageClient)
|
||||
mock_storage.upload_file.return_value = UploadResult(
|
||||
success=True,
|
||||
file_path="test.html",
|
||||
message="Success"
|
||||
)
|
||||
|
||||
mock_content_repo = Mock()
|
||||
mock_site_repo = Mock()
|
||||
mock_page_repo = Mock()
|
||||
|
||||
article1 = Mock(spec=GeneratedContent)
|
||||
article1.id = 1
|
||||
article1.title = "Article 1"
|
||||
article1.formatted_html = "<html>Content 1</html>"
|
||||
article1.site_deployment_id = 1
|
||||
article1.tier = "tier1"
|
||||
|
||||
article2 = Mock(spec=GeneratedContent)
|
||||
article2.id = 2
|
||||
article2.title = "Article 2"
|
||||
article2.formatted_html = "<html>Content 2</html>"
|
||||
article2.site_deployment_id = 1
|
||||
article2.tier = "tier2"
|
||||
|
||||
mock_content_repo.get_by_project_id.return_value = [article1, article2]
|
||||
|
||||
site = Mock(spec=SiteDeployment)
|
||||
site.id = 1
|
||||
site.custom_hostname = "www.example.com"
|
||||
site.storage_zone_name = "test-zone"
|
||||
site.storage_zone_password = "test-password"
|
||||
|
||||
mock_site_repo.get_by_id.return_value = site
|
||||
mock_page_repo.get_by_site.return_value = []
|
||||
|
||||
url_logger = URLLogger(logs_dir=str(tmp_path))
|
||||
|
||||
service = DeploymentService(
|
||||
storage_client=mock_storage,
|
||||
content_repo=mock_content_repo,
|
||||
site_repo=mock_site_repo,
|
||||
page_repo=mock_page_repo,
|
||||
url_logger=url_logger
|
||||
)
|
||||
|
||||
results = service.deploy_batch(project_id=1, continue_on_error=True)
|
||||
|
||||
assert results['articles_deployed'] == 2
|
||||
assert results['articles_failed'] == 0
|
||||
assert results['pages_deployed'] == 0
|
||||
assert mock_storage.upload_file.call_count == 2
|
||||
assert mock_content_repo.mark_as_deployed.call_count == 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
Loading…
Reference in New Issue