Big-Link-Man/README.md

# Big Link Man - Content Automation & Syndication Platform

AI-powered content generation and multi-tier link building system with cloud deployment.

## Quick Start

```bash
# Install dependencies
uv pip install -r requirements.txt

# Setup environment
cp env.example .env
# Edit .env with your credentials

# Initialize database
uv run python scripts/init_db.py

# Create first admin user
uv run python scripts/create_first_admin.py

# Run CLI
uv run python main.py --help
```

## Environment Configuration

Required environment variables in `.env`:

```bash
DATABASE_URL=sqlite:///./content_automation.db
OPENROUTER_API_KEY=your_key_here
BUNNY_ACCOUNT_API_KEY=your_bunny_key_here
```

See `env.example` for full configuration options.

## Database Management

### Initialize Database
```bash
uv run python scripts/init_db.py
```

### Reset Database (drops all data)
```bash
uv run python scripts/init_db.py reset
```

### Create First Admin
```bash
uv run python scripts/create_first_admin.py
```

### Database Migrations
```bash
# Story 3.1 - Site deployments
uv run python scripts/migrate_story_3.1_sqlite.py

# Story 3.2 - Anchor text
uv run python scripts/migrate_add_anchor_text.py

# Story 3.3 - Template fields
uv run python scripts/migrate_add_template_fields.py

# Story 3.4 - Site pages
uv run python scripts/migrate_add_site_pages.py

# Story 4.1 - Deployment fields
uv run python scripts/migrate_add_deployment_fields.py

# Backfill site pages after migration
uv run python scripts/backfill_site_pages.py
```

## User Management

### Add User
```bash
uv run python main.py add-user \
  --username newuser \
  --password password123 \
  --role Admin \
  --admin-user admin \
  --admin-password adminpass
```

### List Users
```bash
uv run python main.py list-users \
  --admin-user admin \
  --admin-password adminpass
```

### Delete User
```bash
uv run python main.py delete-user \
  --username olduser \
  --admin-user admin \
  --admin-password adminpass
```

## Site Management

### Provision New Site
```bash
uv run python main.py provision-site \
  --name "My Site" \
  --domain www.example.com \
  --storage-name my-storage-zone \
  --region DE \
  --admin-user admin \
  --admin-password adminpass
```

Regions: `DE`, `NY`, `LA`, `SG`, `SYD`

### Attach Domain to Existing Storage
```bash
uv run python main.py attach-domain \
  --name "Another Site" \
  --domain www.another.com \
  --storage-name my-storage-zone \
  --admin-user admin \
  --admin-password adminpass
```

### Sync Existing Bunny.net Sites
```bash
# Dry run
uv run python main.py sync-sites \
  --admin-user admin \
  --dry-run

# Actually import
uv run python main.py sync-sites \
  --admin-user admin
```

### List Sites
```bash
uv run python main.py list-sites \
  --admin-user admin \
  --admin-password adminpass
```

### Get Site Details
```bash
uv run python main.py get-site \
  --domain www.example.com \
  --admin-user admin \
  --admin-password adminpass
```

### Remove Site
```bash
uv run python main.py remove-site \
  --domain www.example.com \
  --admin-user admin \
  --admin-password adminpass
```

## Project Management

### Ingest CORA Report
```bash
uv run python main.py ingest-cora \
  --file shaft_machining.xlsx \
  --name "Shaft Machining Project" \
  --custom-anchors "shaft repair,engine parts" \
  --username admin \
  --password adminpass
```

### List Projects
```bash
uv run python main.py list-projects \
  --username admin \
  --password adminpass
```

## Content Generation

### Create Job Configuration
```bash
# Tier 1 only
uv run python create_job_config.py 1 tier1 15

# Multi-tier
uv run python create_job_config.py 1 multi 15 50 100
```

### Generate Content Batch
```bash
uv run python main.py generate-batch \
  --job-file jobs/project_1_tier1_15articles.json \
  --username admin \
  --password adminpass
```

With options:
```bash
uv run python main.py generate-batch \
  --job-file jobs/my_job.json \
  --username admin \
  --password adminpass \
  --debug \
  --continue-on-error \
  --model gpt-4o-mini
```

Available models: `gpt-4o-mini`, `claude-sonnet-4.5`, - anything at openrouter

**Note:** If your job file contains a `models` config, it will override the `--model` flag and use different models for title, outline, and content generation stages.

## Deployment

### Deploy Batch
```bash
# Automatic deployment (runs after generation)
uv run python main.py generate-batch \
  --job-file jobs/my_job.json \
  --username admin \
  --password adminpass

# Manual deployment
uv run python main.py deploy-batch \
  --batch-id 123 \
  --admin-user admin \
  --admin-password adminpass
```

### Dry Run Deployment
```bash
uv run python main.py deploy-batch \
  --batch-id 123 \
  --dry-run
```

### Verify Deployment
```bash
# Check all URLs
uv run python main.py verify-deployment --batch-id 123

# Check random sample
uv run python main.py verify-deployment \
  --batch-id 123 \
  --sample 10 \
  --timeout 10
```

## Link Export

### Export Article URLs
```bash
# Tier 1 only
uv run python main.py get-links \
  --project-id 123 \
  --tier 1

# Tier 2 and above
uv run python main.py get-links \
  --project-id 123 \
  --tier 2+

# With anchor text and destinations
uv run python main.py get-links \
  --project-id 123 \
  --tier 2+ \
  --with-anchor-text \
  --with-destination-url
```

Output is CSV format to stdout. Redirect to save:
```bash
uv run python main.py get-links \
  --project-id 123 \
  --tier 1 > tier1_urls.csv
```

## Utility Scripts

### Check Last Generated Content
```bash
uv run python check_last_gen.py
```

### List All Users (Direct DB Access)
```bash
uv run python scripts/list_users.py
```

### Add Admin (Direct DB Access)
```bash
uv run python scripts/add_admin_direct.py
```

### Check Migration Status
```bash
uv run python scripts/check_migration.py
```

### Add Tier to Projects
```bash
uv run python scripts/add_tier_to_projects.py
```

## Testing

### Run All Tests
```bash
uv run pytest
```

### Run Unit Tests
```bash
uv run pytest tests/unit/ -v
```

### Run Integration Tests
```bash
uv run pytest tests/integration/ -v
```

### Run Specific Test File
```bash
uv run pytest tests/unit/test_url_generator.py -v
```

### Run Story 3.1 Tests
```bash
uv run pytest tests/unit/test_url_generator.py \
               tests/unit/test_site_provisioning.py \
               tests/unit/test_site_assignment.py \
               tests/unit/test_job_config_extensions.py \
               tests/integration/test_story_3_1_integration.py \
               -v
```

### Run with Coverage
```bash
uv run pytest --cov=src --cov-report=html
```

## System Information

### Show Configuration
```bash
uv run python main.py config
```

### Health Check
```bash
uv run python main.py health
```

### List Available Models
```bash
uv run python main.py models
```

## Directory Structure

```
Big-Link-Man/
├── main.py                 # CLI entry point
├── src/                    # Source code
│   ├── api/               # FastAPI endpoints
│   ├── auth/              # Authentication
│   ├── cli/               # CLI commands
│   ├── core/              # Configuration
│   ├── database/          # Models, repositories
│   ├── deployment/        # Cloud deployment
│   ├── generation/        # Content generation
│   ├── ingestion/         # CORA parsing
│   ├── interlinking/      # Link injection
│   └── templating/        # HTML templates
├── scripts/               # Database & utility scripts
├── tests/                 # Test suite
│   ├── unit/
│   └── integration/
├── jobs/                  # Job configuration files
├── docs/                  # Documentation
└── deployment_logs/       # Deployed URL logs
```

## Job Configuration Format

Example job config (`jobs/example.json`):

```json
{
  "job_name": "Multi-Tier Launch",
  "project_id": 1,
  "description": "Site build with 165 articles",
  "models": {
    "title": "openai/gpt-4o-mini",
    "outline": "anthropic/claude-3.5-sonnet",
    "content": "anthropic/claude-3.5-sonnet"
  },
  "tiers": [
    {
      "tier": 1,
      "article_count": 15,
      "validation_attempts": 3
    },
    {
      "tier": 2,
      "article_count": 50,
      "validation_attempts": 2
    }
  ],
  "failure_config": {
    "max_consecutive_failures": 10,
    "skip_on_failure": true
  },
  "interlinking": {
    "links_per_article_min": 2,
    "links_per_article_max": 4,
    "include_home_link": true
  },
  "deployment_targets": ["www.primary.com"],
  "tier1_preferred_sites": ["www.premium.com"],
  "auto_create_sites": true
}
```

### Per-Stage Model Configuration

You can specify different AI models for each generation stage (title, outline, content):

```json
{
  "models": {
    "title": "openai/gpt-4o-mini",
    "outline": "anthropic/claude-3.5-sonnet",
    "content": "openai/gpt-4o"
  }
}
```

**Available models:**
- `openai/gpt-4o-mini` - Fast and cost-effective
- `openai/gpt-4o` - Higher quality, more expensive
- `anthropic/claude-3.5-sonnet` - Excellent for long-form content

If `models` is not specified in the job file, all stages use the model from the `--model` CLI flag (default: `gpt-4o-mini`).

## Common Workflows

### Initial Setup
```bash
uv pip install -r requirements.txt
cp env.example .env
# Edit .env
uv run python scripts/init_db.py
uv run python scripts/create_first_admin.py
uv run python main.py sync-sites --admin-user admin
```

### New Project Workflow
```bash
# 1. Ingest CORA report
uv run python main.py ingest-cora \
  --file project.xlsx \
  --name "My Project" \
  --username admin \
  --password adminpass

# 2. Create job config
uv run python create_job_config.py 1 multi 15 50 100

# 3. Generate content (auto-deploys)
uv run python main.py generate-batch \
  --job-file jobs/project_1_multi_3tiers_165articles.json \
  --username admin \
  --password adminpass

# 4. Verify deployment
uv run python main.py verify-deployment --batch-id 1

# 5. Export URLs for link building
uv run python main.py get-links \
  --project-id 1 \
  --tier 1 > tier1_urls.csv
```

### Re-deploy After Changes
```bash
uv run python main.py deploy-batch \
  --batch-id 123 \
  --admin-user admin \
  --admin-password adminpass
```

## Troubleshooting

### Database locked
```bash
# Stop any running processes, then:
uv run python scripts/init_db.py reset
```

### Missing dependencies
```bash
uv pip install -r requirements.txt --force-reinstall
```

### AI API errors
Check `OPENROUTER_API_KEY` in `.env`

### Bunny.net authentication failed
Check `BUNNY_ACCOUNT_API_KEY` in `.env`

### Storage upload failed
Verify `storage_zone_password` in database (set during site provisioning)

## Documentation

- Product Requirements: `docs/prd.md`
- Architecture: `docs/architecture/`
- Implementation Summaries: `STORY_*.md` files
- Quick Start Guides: `*_QUICKSTART.md` files

## License

All rights reserved.