bryanb
/
Big-Link-Man
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Story 4.1-3 coded and real-world tested. Working

main
PeninsulaInd 2025-10-22 13:04:03 -05:00
parent 8a382a1db4
commit b3e35b0b4d
11 changed files with 197 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
STORY_4.1_IMPLEMENTATION_SUMMARY.md
View File

@ -10,11 +10,16 @@ Successfully implemented deployment of generated content to Bunny.net cloud stor
### 1. Bunny.net Storage Client (`src/deployment/bunny_storage.py`)
- `BunnyStorageClient` class for uploading files to Bunny.net storage zones
- Uses per-zone `storage_zone_password` from database for authentication
- Region-aware URL generation:
- Frankfurt (DE): `storage.bunnycdn.com` (no prefix)
- Other regions: `{region}.storage.bunnycdn.com` (e.g., `la.storage.bunnycdn.com`)
- Uses `application/octet-stream` content-type per Bunny.net API requirements
- Implements retry logic with exponential backoff (3 attempts)
- Methods:
- `upload_file()`: Upload HTML content to storage zone
- `file_exists()`: Check if file exists in storage
- `list_files()`: List files in storage zone
- **Tested with real Bunny.net storage** - successful upload verified
### 2. Database Updates
- Added `deployed_url` (TEXT, nullable) to `generated_content` table
@ -157,11 +162,14 @@ BUNNY_ACCOUNT_API_KEY=your_account_api_key_here # For zone creation (already ex
## Testing Results
All 13 integration tests passing:
All 18 tests passing:
- URL generation (4 tests)
- URL logging (4 tests)
- Storage client (2 tests)
- Deployment service (3 tests)
- Storage URL generation (5 tests - including DE region special case)
**Real-world validation:** Successfully uploaded test file to Bunny.net storage and verified HTTP 201 response.
## Known Limitations / Technical Debt

65
STORY_4.1_REAL_UPLOAD_VALIDATION.md 100644
View File

@ -0,0 +1,65 @@
# Story 4.1: Real Upload Validation
## Summary
Successfully validated real file uploads to Bunny.net storage on October 22, 2025.
## Test Details
**Storage Zone:** 5axislaser925
**Region:** DE (Frankfurt)
**File:** story-4.1-test.html
**Result:** HTTP 201 (Success)
**Public URL:** https://5axislaser925.b-cdn.net/story-4.1-test.html
## Key Learnings
### 1. Region-Specific URLs
Frankfurt (DE) is the default region and uses a different URL pattern:
- **DE:** `https://storage.bunnycdn.com/{zone}/{file}`
- **Other regions:** `https://{region}.storage.bunnycdn.com/{zone}/{file}`
This was implemented in `BunnyStorageClient._get_storage_url()`.
### 2. Content-Type Requirements
Per Bunny.net API documentation:
- **Required:** `application/octet-stream`
- **NOT** `text/html` or other MIME types
- File content must be raw binary (we use `.encode('utf-8')`)
### 3. Success Response Code
- Bunny.net returns **HTTP 201** for successful uploads (not 200)
- This is documented in their API reference
### 4. Authentication
- Uses per-zone `storage_zone_password` via `AccessKey` header
- Password is stored in database (`site_deployments.storage_zone_password`)
- Set automatically when zones are created via `provision-site` or `sync-sites`
- NO API key from `.env` needed for uploads
## Implementation Changes Made
1. **Fixed region URL logic** - DE uses no prefix
2. **Changed default Content-Type** - Now uses `application/octet-stream`
3. **Updated success detection** - Looks for HTTP 201
4. **Added region parameter** - All upload methods now require `zone_region`
## Test Coverage
**Unit Tests (5):**
- DE region URL generation (with/without case)
- LA, NY, SG region URL generation
**Integration Tests (13):**
- Full upload workflow mocking
- Deployment service orchestration
- URL generation and logging
- Error handling
**Real-World Test:**
- Actual upload to Bunny.net storage
- File accessible via CDN URL
- HTTP 201 response confirmed
## Status
✅ **VALIDATED** - Ready for production use

12
docs/prd/epic-4-deployment.md
View File

@ -4,16 +4,16 @@
To deploy all finalized HTML content (articles and boilerplate pages) for a batch to the correct cloud storage targets, purge the CDN cache, and verify the successful deployment.
## Status
- **Story 4.1**: ✓ Documented (Ready for implementation)
- **Story 4.2**: Partially implemented in Story 4.1 (URL logging)
- **Story 4.3**: Partially implemented in Story 4.1 (Database status updates)
- **Story 4.1**: ✅ COMPLETE (22 story points, real-world validated)
- **Story 4.2**: ✅ COMPLETE (implemented in Story 4.1)
- **Story 4.3**: ✅ COMPLETE (implemented in Story 4.1)
- **Story 4.4**: Not started
- **Story 4.5**: Not started
## Stories
### Story 4.1: Deploy Content to Cloud Storage
**Status:** ✓ Documented - Ready for Implementation (22 story points)
**Status:** ✅ COMPLETE (22 story points, real-world validated)
**Document:** [story-4.1-deploy-content-to-cloud.md](../stories/story-4.1-deploy-content-to-cloud.md)
**As a developer**, I want to upload all generated HTML files for a batch to their designated cloud storage buckets so that the content is hosted and ready to be served.
@ -34,7 +34,7 @@ To deploy all finalized HTML content (articles and boilerplate pages) for a batc
* Storage API authentication details TBD during implementation
### Story 4.2: Log Deployed URLs to Tiered Text Files
**Status:** ✓ Implemented in Story 4.1
**Status:** ✅ COMPLETE (implemented in Story 4.1)
**As a developer**, I want to save the URLs of all deployed articles into daily, tier-segregated text files, so that I have a clean list for indexing services and other external tools.
@ -51,7 +51,7 @@ To deploy all finalized HTML content (articles and boilerplate pages) for a batc
* **Duplicate prevention**: Check file before appending to avoid duplicate URLs
### Story 4.3: Update Deployment Status
**Status:** ✓ Implemented in Story 4.1
**Status:** ✅ COMPLETE (implemented in Story 4.1)
**As a developer**, I want to update the status of each article and site in the database to 'deployed' and record the final public URL, so that the system has an accurate record of what content is live.

3
env.example
View File

@ -22,6 +22,9 @@ BUNNY_ACCOUNT_API_KEY=your_bunny_account_api_key_here
# Note: For file uploads, the system uses per-zone storage_zone_password from the database
# (set automatically when zones are created). No additional API key needed for uploads.
# Deployment Logs Directory (optional, defaults to 'deployment_logs')
# DEPLOYMENT_LOGS_DIR=deployment_logs
# Digital Ocean Spaces Configuration
DO_SPACES_ACCESS_KEY=your_do_spaces_key_here
DO_SPACES_SECRET_KEY=your_do_secret_key_here

2
src/cli/commands.py
View File

@ -1046,7 +1046,7 @@ def deploy_batch(
return
storage_client = BunnyStorageClient(max_retries=3)
url_logger = URLLogger(logs_dir="deployment_logs")
url_logger = URLLogger()
deployment_service = DeploymentService(
storage_client=storage_client,

61
src/deployment/bunny_storage.py
View File

@ -32,8 +32,6 @@ class UploadResult:
class BunnyStorageClient:
"""Client for uploading files to Bunny.net Storage Zones"""
BASE_URL = "https://storage.bunnycdn.com"
def __init__(self, max_retries: int = 3):
"""
Initialize Bunny.net Storage client
@ -44,13 +42,35 @@ class BunnyStorageClient:
self.max_retries = max_retries
self.session = requests.Session()
def _get_storage_url(self, region: str) -> str:
"""
Get region-specific storage URL
Args:
region: Region code (e.g., 'LA', 'NY', 'DE', 'SG', 'SYD')
Returns:
Base URL for storage API in that region
Note:
Frankfurt (DE) is the default region and uses storage.bunnycdn.com
All other regions use {region}.storage.bunnycdn.com
"""
region_upper = region.upper()
if region_upper == 'DE':
return "https://storage.bunnycdn.com"
else:
region_lower = region.lower()
return f"https://{region_lower}.storage.bunnycdn.com"
def upload_file(
self,
zone_name: str,
zone_password: str,
zone_region: str,
file_path: str,
content: str,
content_type: str = 'text/html'
content_type: str = 'application/octet-stream'
) -> UploadResult:
"""
Upload a file to Bunny.net storage zone
@ -58,9 +78,10 @@ class BunnyStorageClient:
Args:
zone_name: Storage zone name
zone_password: Storage zone password (from database)
zone_region: Storage zone region (e.g., 'LA', 'NY', 'DE')
file_path: Path within storage zone (e.g., 'my-article.html')
content: File content to upload
content_type: MIME type (default: text/html)
content_type: MIME type (default: application/octet-stream per Bunny.net docs)
Returns:
UploadResult with success status and message
@ -68,11 +89,17 @@ class BunnyStorageClient:
Raises:
BunnyStorageAuthError: If authentication fails
BunnyStorageError: For other API errors
Note:
Per Bunny.net docs, content must be raw binary and content-type should be
application/octet-stream. Success response is HTTP 201.
"""
url = f"{self.BASE_URL}/{zone_name}/{file_path}"
base_url = self._get_storage_url(zone_region)
url = f"{base_url}/{zone_name}/{file_path}"
headers = {
"AccessKey": zone_password,
"Content-Type": content_type
"Content-Type": content_type,
"accept": "application/json"
}
for attempt in range(self.max_retries):
@ -86,7 +113,9 @@ class BunnyStorageClient:
if response.status_code == 401:
raise BunnyStorageAuthError(
f"Authentication failed for zone '{zone_name}'. Check storage_zone_password."
f"Authentication failed for zone '{zone_name}'. "
f"Check storage_zone_password, region hostname, or data format. "
f"Bunny.net requires raw binary data with application/octet-stream."
)
if response.status_code == 201:
@ -94,7 +123,13 @@ class BunnyStorageClient:
return UploadResult(
success=True,
file_path=file_path,
message="Upload successful"
message="Upload successful (HTTP 201)"
)
if response.status_code == 400:
raise BunnyStorageError(
f"Bad request (HTTP 400): File upload unsuccessful. "
f"Ensure data is raw binary format."
)
if response.status_code >= 400:
@ -141,6 +176,7 @@ class BunnyStorageClient:
self,
zone_name: str,
zone_password: str,
zone_region: str,
file_path: str
) -> bool:
"""
@ -149,6 +185,7 @@ class BunnyStorageClient:
Args:
zone_name: Storage zone name
zone_password: Storage zone password
zone_region: Storage zone region (e.g., 'LA', 'NY', 'DE')
file_path: Path within storage zone
Returns:
@ -157,7 +194,8 @@ class BunnyStorageClient:
Raises:
BunnyStorageError: For API errors (excluding 404)
"""
url = f"{self.BASE_URL}/{zone_name}/{file_path}"
base_url = self._get_storage_url(zone_region)
url = f"{base_url}/{zone_name}/{file_path}"
headers = {"AccessKey": zone_password}
try:
@ -181,6 +219,7 @@ class BunnyStorageClient:
self,
zone_name: str,
zone_password: str,
zone_region: str,
prefix: str = ''
) -> List[str]:
"""
@ -189,6 +228,7 @@ class BunnyStorageClient:
Args:
zone_name: Storage zone name
zone_password: Storage zone password
zone_region: Storage zone region (e.g., 'LA', 'NY', 'DE')
prefix: Optional path prefix to filter results
Returns:
@ -197,7 +237,8 @@ class BunnyStorageClient:
Raises:
BunnyStorageError: For API errors
"""
url = f"{self.BASE_URL}/{zone_name}/{prefix}"
base_url = self._get_storage_url(zone_region)
url = f"{base_url}/{zone_name}/{prefix}"
headers = {"AccessKey": zone_password}
try:

8
src/deployment/deployment_service.py
View File

@ -186,9 +186,9 @@ class DeploymentService:
self.storage.upload_file(
zone_name=site.storage_zone_name,
zone_password=site.storage_zone_password,
zone_region=site.storage_zone_region,
file_path=file_path,
content=article.formatted_html,
content_type='text/html'
content=article.formatted_html
)
return url
@ -220,9 +220,9 @@ class DeploymentService:
self.storage.upload_file(
zone_name=site.storage_zone_name,
zone_password=site.storage_zone_password,
zone_region=site.storage_zone_region,
file_path=file_path,
content=page.content,
content_type='text/html'
content=page.content
)
return url

8
src/deployment/url_logger.py
View File

@ -6,7 +6,7 @@ Story 4.1: Deploy Content to Cloud Storage
import os
import logging
from datetime import datetime
from typing import Set
from typing import Set, Optional
from pathlib import Path
logger = logging.getLogger(__name__)
@ -15,13 +15,17 @@ logger = logging.getLogger(__name__)
class URLLogger:
"""Logs deployed article URLs to tier-segregated daily text files"""
def __init__(self, logs_dir: str = "deployment_logs"):
def __init__(self, logs_dir: Optional[str] = None):
"""
Initialize URL logger
Args:
logs_dir: Directory for log files (created if doesn't exist)
Defaults to DEPLOYMENT_LOGS_DIR env var or 'deployment_logs'
"""
if logs_dir is None:
logs_dir = os.getenv("DEPLOYMENT_LOGS_DIR", "deployment_logs")
self.logs_dir = Path(logs_dir)
self.logs_dir.mkdir(exist_ok=True)
logger.info(f"URL logger initialized with logs_dir: {self.logs_dir}")

2
src/generation/batch_processor.py
View File

@ -401,7 +401,7 @@ class BatchProcessor:
click.echo(f"\n Deployment: Starting automatic deployment for project {project_id}...")
storage_client = BunnyStorageClient(max_retries=3)
url_logger = URLLogger(logs_dir="deployment_logs")
url_logger = URLLogger()
page_repo = SitePageRepository(self.content_repo.session)
deployment_service = DeploymentService(

6
tests/integration/test_deployment.py
View File

@ -141,6 +141,7 @@ class TestBunnyStorageClient:
result = client.upload_file(
zone_name="test-zone",
zone_password="test-password",
zone_region="DE",
file_path="test.html",
content="<html>Test</html>"
)
@ -150,6 +151,7 @@ class TestBunnyStorageClient:
mock_session.put.assert_called_once()
call_args = mock_session.put.call_args
# DE region uses storage.bunnycdn.com without prefix
assert call_args[0][0] == "https://storage.bunnycdn.com/test-zone/test.html"
assert call_args[1]['headers']['AccessKey'] == "test-password"
@ -168,6 +170,7 @@ class TestBunnyStorageClient:
client.upload_file(
zone_name="test-zone",
zone_password="bad-password",
zone_region="DE",
file_path="test.html",
content="<html>Test</html>"
)
@ -211,6 +214,7 @@ class TestDeploymentService:
site.custom_hostname = "www.example.com"
site.storage_zone_name = "test-zone"
site.storage_zone_password = "test-password"
site.storage_zone_region = "DE"
url = service.deploy_article(article, site)
@ -249,6 +253,7 @@ class TestDeploymentService:
site.custom_hostname = "www.example.com"
site.storage_zone_name = "test-zone"
site.storage_zone_password = "test-password"
site.storage_zone_region = "DE"
url = service.deploy_boilerplate_page(page, site)
@ -289,6 +294,7 @@ class TestDeploymentService:
site.custom_hostname = "www.example.com"
site.storage_zone_name = "test-zone"
site.storage_zone_password = "test-password"
site.storage_zone_region = "DE"
mock_site_repo.get_by_id.return_value = site
mock_page_repo.get_by_site.return_value = []

45
tests/unit/test_bunny_storage_url.py 100644
View File

@ -0,0 +1,45 @@
"""
Unit tests for Bunny.net storage URL generation
"""
import pytest
from src.deployment.bunny_storage import BunnyStorageClient
class TestBunnyStorageURLGeneration:
"""Test region-specific URL generation"""
def test_de_region_no_prefix(self):
"""Test that DE region uses storage.bunnycdn.com without prefix"""
client = BunnyStorageClient()
url = client._get_storage_url("DE")
assert url == "https://storage.bunnycdn.com"
def test_de_lowercase_no_prefix(self):
"""Test that lowercase 'de' also works"""
client = BunnyStorageClient()
url = client._get_storage_url("de")
assert url == "https://storage.bunnycdn.com"
def test_la_region_with_prefix(self):
"""Test that LA region uses la.storage.bunnycdn.com"""
client = BunnyStorageClient()
url = client._get_storage_url("LA")
assert url == "https://la.storage.bunnycdn.com"
def test_ny_region_with_prefix(self):
"""Test that NY region uses ny.storage.bunnycdn.com"""
client = BunnyStorageClient()
url = client._get_storage_url("NY")
assert url == "https://ny.storage.bunnycdn.com"
def test_sg_region_with_prefix(self):
"""Test that SG region uses sg.storage.bunnycdn.com"""
client = BunnyStorageClient()
url = client._get_storage_url("SG")
assert url == "https://sg.storage.bunnycdn.com"
if __name__ == "__main__":
pytest.main([__file__, "-v"])