Big-Link-Man/docs/gui-planning.md

# GUI Planning Document

Comprehensive mapping of all CLI commands, scripts, and their parameters to guide front-end GUI development.

## Main Workflows

### 1. Project Creation Workflow
- Ingest CORA file → Create project → Create job file → Generate content → Deploy

### 2. Site Management Workflow
- Discover/Provision sites → Sync sites → Manage domains → Deploy content

### 3. Content Generation Workflow
- Create/Edit job file → Generate batch → Verify → Deploy → Export links

### 4. Maintenance Workflow
- Update index pages → Add robots.txt → Verify deployments → Check status

---

## CLI Commands Reference

### System Commands

#### `config`
Show current configuration

**Parameters:** None

**Example:**
```bash
uv run python main.py config
```

---

#### `health`
Check system health

**Parameters:** None

**Example:**
```bash
uv run python main.py health
```

---

#### `models`
List available AI models

**Parameters:** None

**Example:**
```bash
uv run python main.py models
```

---

### User Management

#### `add-user`
Create a new user (requires admin authentication)

**Parameters:**
- `--username` (required) - Username for the new user
- `--password` (required) - Password for the new user
- `--role` (required) - Role: "Admin" or "User"
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication

**Example:**
```bash
uv run python main.py add-user --username newuser --password pass123 --role Admin
```

---

#### `delete-user`
Delete a user by username (requires admin authentication)

**Parameters:**
- `--username` (required) - Username to delete
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication
- `--yes` (flag) - Skip confirmation prompt

**Example:**
```bash
uv run python main.py delete-user --username olduser --yes
```

---

#### `list-users`
List all users (requires admin authentication)

**Parameters:**
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication

**Example:**
```bash
uv run python main.py list-users
```

---

### Site Management

#### `provision-site`
Provision a new site with Storage Zone and Pull Zone (requires admin)

**Parameters:**
- `--name` (required) - Site name
- `--domain` (required) - Custom domain (FQDN, e.g., www.example.com)
- `--storage-name` (required) - Storage Zone name (must be globally unique)
- `--region` (required) - Storage region: "DE", "NY", "LA", "SG", "SYD"
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication

**Example:**
```bash
uv run python main.py provision-site --name "My Site" --domain www.example.com --storage-name mysite-storage --region NY
```

---

#### `attach-domain`
Attach a domain to an existing Storage Zone (requires admin)

**Parameters:**
- `--name` (required) - Site name
- `--domain` (required) - Custom domain (FQDN, e.g., www.example.com)
- `--storage-name` (required) - Existing Storage Zone name
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication

**Example:**
```bash
uv run python main.py attach-domain --name "New Site" --domain new.example.com --storage-name existing-storage
```

---

#### `list-sites`
List all site deployments (requires admin)

**Parameters:**
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication

**Example:**
```bash
uv run python main.py list-sites
```

---

#### `get-site`
Get detailed information about a site deployment (requires admin)

**Parameters:**
- `--domain` (required) - Custom domain to lookup
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication

**Example:**
```bash
uv run python main.py get-site --domain www.example.com
```

---

#### `remove-site`
Remove a site deployment record (requires admin)

**Parameters:**
- `--domain` (required) - Custom domain to remove
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication
- `--yes` (flag) - Skip confirmation prompt

**Example:**
```bash
uv run python main.py remove-site --domain www.example.com --yes
```

---

#### `sync-sites`
Sync existing bunny.net sites with custom domains to database (requires admin)

**Parameters:**
- `--admin-user` (optional) - Admin username for authentication
- `--admin-password` (optional) - Admin password for authentication
- `--dry-run` (flag) - Show what would be imported without making changes

**Example:**
```bash
uv run python main.py sync-sites --dry-run
```

---

#### `discover-s3-buckets`
Discover and register AWS S3 buckets as site deployments

**Parameters:** None (runs the discovery script)

**Example:**
```bash
uv run python main.py discover-s3-buckets
```

---

### Project Management

#### `ingest-cora`
Ingest a CORA .xlsx report and create a new project

**Parameters:**
- `--file` / `-f` (required) - Path to CORA .xlsx file
- `--name` / `-n` (required) - Project name
- `--money-site-url` / `-m` (optional) - Money site URL (e.g., https://example.com)
- `--custom-anchors` / `-a` (optional) - Comma-separated list of custom anchor text
- `--tier1-branded-ratio` (optional) - Ratio of branded anchor text for tier1 (default: 0.75). When specified, prompts for branded anchor text and configures tier1 job with explicit anchor text terms achieving the specified ratio.
- `--username` / `-u` (optional) - Username for authentication
- `--password` / `-p` (optional) - Password for authentication

**Example:**
```bash
uv run python main.py ingest-cora --file report.xlsx --name "My Project" --money-site-url https://example.com
```

---

#### `ingest-simple`
Ingest a simple spreadsheet and create a new project

**Expected spreadsheet format:**
- First row: Headers (main_keyword, project_name, related_searches, entities)
- Second row: Data values

**Required columns:** main_keyword, project_name, related_searches, entities

**Optional columns:** word_count (default: 1500), term_frequency (default: 3)

**Parameters:**
- `--file` / `-f` (required) - Path to simple .xlsx spreadsheet file
- `--name` / `-n` (optional) - Project name (overrides project_name from spreadsheet)
- `--money-site-url` / `-m` (optional) - Money site URL (e.g., https://example.com)
- `--username` / `-u` (optional) - Username for authentication
- `--password` / `-p` (optional) - Password for authentication

**Example:**
```bash
uv run python main.py ingest-simple --file simple.xlsx
```

---

#### `list-projects`
List all projects for the authenticated user

**Parameters:**
- `--username` / `-u` (optional) - Username for authentication
- `--password` / `-p` (optional) - Password for authentication

**Example:**
```bash
uv run python main.py list-projects
```

---

#### `create-job`
Create a job file from an existing project ID

**Parameters:**
- `--project-id` / `-p` (required) - Project ID to create job file for
- `--deployment-targets` / `-d` (multiple, optional) - Deployment target hostnames (can specify multiple times)
- `--tier1-count` (optional, default: 10) - Number of tier1 articles
- `--tier2-count` (optional, default: 30) - Number of tier2 articles
- `--output` / `-o` (optional) - Output file path (default: jobs/{project_name}.json)
- `--username` / `-u` (optional) - Username for authentication
- `--password` / `-pwd` (optional) - Password for authentication

**Example:**
```bash
uv run python main.py create-job --project-id 1 --tier1-count 15 --tier2-count 50 --deployment-targets www.site1.com --deployment-targets www.site2.com
```

---

### Content Generation

#### `generate-batch`
Generate content batch from job file

**Parameters:**
- `--job-file` / `-j` (required) - Path to job JSON file
- `--username` / `-u` (optional) - Username for authentication
- `--password` / `-p` (optional) - Password for authentication
- `--debug` (flag) - Save AI responses to debug_output/
- `--continue-on-error` (flag) - Continue processing if article generation fails
- `--model` / `-m` (optional, default: "gpt-4o-mini") - AI model to use (gpt-4o-mini, x-ai/grok-4-fast)

**Example:**
```bash
uv run python main.py generate-batch --job-file jobs/my-project.json --debug --continue-on-error
```

---

### Deployment

#### `deploy-batch`
Deploy all content in a batch to cloud storage

**Parameters:**
- `--batch-id` / `-b` (required) - Project/batch ID to deploy
- `--username` / `-u` (optional) - Username for authentication
- `--password` / `-p` (optional) - Password for authentication
- `--continue-on-error` (flag, default: True) - Continue if file fails
- `--dry-run` (flag) - Preview what would be deployed

**Example:**
```bash
uv run python main.py deploy-batch --batch-id 1 --dry-run
```

---

#### `verify-deployment`
Verify deployed URLs return 200 OK status

**Parameters:**
- `--batch-id` / `-b` (required) - Project/batch ID to verify
- `--sample` / `-s` (optional) - Number of random URLs to check (default: check all)
- `--timeout` / `-t` (optional, default: 10) - Request timeout in seconds

**Example:**
```bash
uv run python main.py verify-deployment --batch-id 1 --sample 10 --timeout 15
```

---

### Link Export

#### `get-links`
Export article URLs with optional link details for a project and tier

**Parameters:**
- `--project-id` / `-p` (required) - Project ID to get links for
- `--tier` / `-t` (required) - Tier to filter (e.g., "1" or "2+" for tier 2 and above)
- `--with-anchor-text` (flag) - Include anchor text used for tiered links
- `--with-destination-url` (flag) - Include destination URL that the article links to

**Example:**
```bash
uv run python main.py get-links --project-id 1 --tier 1 --with-anchor-text --with-destination-url
```

---

## Scripts Reference

### Site Management Scripts

#### `discover_s3_buckets.py`
Discover and register AWS S3 buckets as site deployments

**Parameters:**
- `--auto-import-all` (flag) - Automatically import all unregistered buckets

**Usage:**
```bash
# Interactive mode (select buckets manually)
uv run python scripts/discover_s3_buckets.py

# Auto-import mode (import all unregistered buckets)
uv run python scripts/discover_s3_buckets.py --auto-import-all
```

**Features:**
- Discovers all S3 buckets in AWS account
- Skips buckets already registered in database
- Respects exclusion list in `s3_bucket_exclusions.txt`
- Automatically detects bucket region

---

#### `list_s3_fqdn_sites.py`
List all S3 sites with custom domains (FQDNs)

**Parameters:** None

**Usage:**
```bash
uv run python scripts/list_s3_fqdn_sites.py
```

---

#### `list_sites_by_id.py`
List site deployments by specific IDs

**Parameters:** None (edit script to set target_ids)

**Usage:**
```bash
# Edit the script to set target_ids, then:
uv run python scripts/list_sites_by_id.py
```

---

#### `delete_sites.py`
Delete site deployments by ID

**Parameters:** None (edit script to set site_ids)

**Usage:**
```bash
# Edit the script to set site_ids, then:
uv run python scripts/delete_sites.py
```

**Warning:** This permanently deletes site records from the database.

---

#### `check_sites.py`
Check site status

**Parameters:** None

**Usage:**
```bash
uv run python scripts/check_sites.py
```

---

#### `check_specific_sites.py`
Check specific sites

**Parameters:** None

**Usage:**
```bash
uv run python scripts/check_specific_sites.py
```

---

### Deployment Scripts

#### `add_robots_txt_to_buckets.py`
Add standardized `robots.txt` files to all storage buckets (both S3 and Bunny)

**Parameters:**
- `--dry-run` (flag) - Preview what would be done without actually uploading
- `--provider` (optional) - Only process specific provider: "s3" or "bunny"

**Usage:**
```bash
# Preview what would be done (recommended first)
uv run python scripts/add_robots_txt_to_buckets.py --dry-run

# Upload to all buckets
uv run python scripts/add_robots_txt_to_buckets.py

# Only process S3 buckets
uv run python scripts/add_robots_txt_to_buckets.py --provider s3

# Only process Bunny storage zones
uv run python scripts/add_robots_txt_to_buckets.py --provider bunny
```

**robots.txt behavior:**
- ✅ Allows: Google, Bing, Yahoo, DuckDuckGo, Baidu, Yandex
- ✅ Allows: GPTBot, Claude, Common Crawl, Perplexity, ByteDance AI
- ❌ Blocks: Ahrefs, Semrush, Moz, and other SEO tools
- ❌ Blocks: HTTrack, Wget, and other scrapers/bad bots

---

#### `update_index_pages.py`
Automatically generate or update `index.html` and `sitemap.xml` files for all storage buckets

**Parameters:**
- `--dry-run` (flag) - Preview what would be updated without actually uploading
- `--provider` (optional) - Only process specific provider: "s3" or "bunny"
- `--force` (flag) - Force update even if no changes detected
- `--limit` (optional, integer) - Limit number of sites to process (useful for testing)
- `--hostname` (optional, string) - Process only specific hostname (exact match)

**Usage:**
```bash
# Preview what would be updated (recommended first)
uv run python scripts/update_index_pages.py --dry-run

# Update all buckets
uv run python scripts/update_index_pages.py

# Only process S3 buckets
uv run python scripts/update_index_pages.py --provider s3

# Only process Bunny storage zones
uv run python scripts/update_index_pages.py --provider bunny

# Force update even if no changes detected
uv run python scripts/update_index_pages.py --force

# Test on specific site
uv run python scripts/update_index_pages.py --hostname example.com

# Limit number of sites (useful for testing)
uv run python scripts/update_index_pages.py --limit 10
```

**Features:**
- Lists all HTML files in each bucket's root directory
- Extracts titles from `<title>` tags (or formats filenames as fallback)
- Generates article listings sorted by most recent modification date
- Creates or updates `index.html` with article links
- Generates `sitemap.xml` with industry-standard settings
- Tracks last run timestamps to avoid unnecessary updates
- Excludes boilerplate pages: `index.html`, `about.html`, `privacy.html`, `contact.html`

---

### Database Scripts

#### `init_db.py`
Initialize or reset the database

**Parameters:** None (or "reset" argument)

**Usage:**
```bash
# Initialize database
uv run python scripts/init_db.py

# Reset database (drops all data)
uv run python scripts/init_db.py reset
```

---

#### `create_first_admin.py`
Create the first admin user in the system

**Parameters:** None (interactive)

**Usage:**
```bash
uv run python scripts/create_first_admin.py
```

---

#### `add_admin_direct.py`
Add an admin user directly to the database (bypasses authentication)

**Parameters:** None (interactive)

**Usage:**
```bash
uv run python scripts/add_admin_direct.py
```

**Use case:** When you're locked out or need to create an admin without existing admin credentials.

---

#### `list_users.py`
List all users in the database (direct database access)

**Parameters:** None

**Usage:**
```bash
uv run python scripts/list_users.py
```

---

#### Migration Scripts
Database schema migrations for various features

**Scripts:**
- `migrate_story_3.1_sqlite.py` - Site deployments
- `migrate_add_anchor_text.py` - Anchor text
- `migrate_add_template_fields.py` - Template fields
- `migrate_add_site_pages.py` - Site pages
- `migrate_add_deployment_fields.py` - Deployment fields
- `migrate_add_multi_cloud_storage_fields.py` - Multi-cloud storage fields
- `migrate_add_image_fields.py` - Image fields

**Parameters:** None

**Usage:**
```bash
uv run python scripts/migrate_story_3.1_sqlite.py
```

---

#### `backfill_site_pages.py`
Backfill site pages after running the site pages migration

**Parameters:** None

**Usage:**
```bash
uv run python scripts/backfill_site_pages.py
```

---

#### `check_migration.py`
Check the status of database migrations

**Parameters:** None

**Usage:**
```bash
uv run python scripts/check_migration.py
```

---

#### `add_tier_to_projects.py`
Add tier field to existing projects

**Parameters:** None

**Usage:**
```bash
uv run python scripts/add_tier_to_projects.py
```

---

### Content Management Scripts

#### `check_last_gen.py`
Check the last generated content batch

**Parameters:** None

**Usage:**
```bash
uv run python check_last_gen.py
```

---

#### `check_project_url.py`
Check project URL

**Parameters:** None (edit script for project_id)

**Usage:**
```bash
# Edit the script to set project_id, then:
uv run python scripts/check_project_url.py
```

---

#### `check_job_last_run.py`
Check job last run

**Parameters:** None (edit script for project_id)

**Usage:**
```bash
# Edit the script to set project_id, then:
uv run python scripts/check_job_last_run.py
```

---

#### `fix_tier1_money_links.py`
Fix tier1 money links

**Parameters:** None (edit script for project_id)

**Usage:**
```bash
# Edit the script to set project_id, then:
uv run python scripts/fix_tier1_money_links.py
```

---

#### `investigate_tier1_links.py`
Investigate tier1 links

**Parameters:** None (edit script for project_id)

**Usage:**
```bash
# Edit the script to set project_id, then:
uv run python scripts/investigate_tier1_links.py
```

---

#### `list_t1_articles.py`
List tier1 articles

**Parameters:** None

**Usage:**
```bash
uv run python scripts/list_t1_articles.py
```

---

#### `check_orphaned_articles.py`
Check for orphaned articles

**Parameters:** None

**Usage:**
```bash
uv run python scripts/check_orphaned_articles.py
```

---

#### `fix_orphaned_articles.py`
Fix orphaned articles

**Parameters:** None

**Usage:**
```bash
uv run python scripts/fix_orphaned_articles.py
```

---

### Utility Scripts

#### `generate_cli_docs.py`
Regenerate the CLI command reference documentation

**Parameters:** None

**Usage:**
```bash
uv run python scripts/generate_cli_docs.py
```

Generates `docs/CLI_COMMAND_REFERENCE.md` from Click command definitions.

---

#### Test Scripts
Various testing and debugging scripts

**Scripts:**
- `test_image_generation.py` - Test image generation
- `test_image_reinsertion.py` - Test image reinsertion
- `test_s3_upload.py` - Test S3 upload
- `test_s3_uploads_localtris.py` - Test S3 uploads for localtris
- `test_s3_real.py` - Test real S3 upload
- `test_story_6_3.py` - Test story 6.3

**Parameters:** Various (see individual scripts)

---

## Suggested GUI Structure

### Main Navigation

#### 1. Dashboard
- System health status
- Recent activity feed
- Quick stats:
  - Total projects
  - Total sites
  - Total content generated
  - Recent deployments
- Quick actions:
  - Create new project
  - Provision new site
  - Generate content

---

#### 2. Projects
**List View:**
- Table of all projects
- Columns: ID, Name, Keyword, Created, Status
- Filters: By user, by date, by status
- Search: By name, keyword
- Actions: View, Edit, Delete, Create Job, Generate

**Create Project:**
- Tab 1: CORA Ingestion
  - File upload (drag & drop or file picker)
  - Project name input
  - Money site URL input
  - Custom anchors (comma-separated)
  - Authentication fields
- Tab 2: Simple Ingestion
  - File upload
  - Project name (optional, can override spreadsheet)
  - Money site URL
  - Authentication fields

**Project Details:**
- Project information display
- Related content list
- Job files list
- Actions:
  - Create job file
  - Generate content
  - Export links
  - Edit project

**Create Job File:**
- Project selector (dropdown)
- Deployment targets (multi-select with search)
- Tier 1 count (number input)
- Tier 2 count (number input)
- Model selection (dropdown)
- Output path (optional, file picker)
- Authentication fields

---

#### 3. Sites
**List View:**
- Table of all sites
- Columns: ID, Site Name, Domain, Storage Zone, Region, Provider
- Filters: By provider, by region, by status
- Search: By name, domain
- Actions: View, Edit, Delete, Sync

**Provision Site:**
- Site name input
- Domain input (FQDN)
- Storage name input (with uniqueness check)
- Region selector (dropdown: DE, NY, LA, SG, SYD)
- Authentication fields
- DNS configuration instructions display

**Attach Domain:**
- Site name input
- Domain input (FQDN)
- Storage name selector (dropdown of existing)
- Authentication fields

**Discover S3 Buckets:**
- Auto-import toggle
- Bucket list display
- Status indicators (Available, Registered, Excluded)
- Bulk selection
- Import button

**Site Details:**
- Site information display
- Storage zone details
- Pull zone details
- DNS configuration
- Deployed content count
- Actions: Edit, Remove, View Content

---

#### 4. Content
**List View:**
- Table of all generated content
- Columns: ID, Title, Project, Tier, Site, Status, Created
- Filters: By project, by tier, by site, by status
- Search: By title
- Actions: View, Deploy, Delete

**Deploy Batch:**
- Batch/Project selector (dropdown)
- Continue on error (checkbox)
- Dry run (checkbox)
- Authentication fields
- Progress bar
- Results display

**Verify Deployment:**
- Batch/Project selector (dropdown)
- Sample size (number input, optional)
- Timeout (number input, default: 10)
- Results display:
  - Total checked
  - Successful
  - Failed (with details)

**Content Details:**
- Content display (HTML preview)
- Metadata display
- Deployment status
- Actions: Edit, Redeploy, Delete

---

#### 5. Maintenance
**Update Index Pages:**
- Provider filter (dropdown: All, S3, Bunny)
- Force update (checkbox)
- Limit (number input, optional)
- Hostname filter (text input, optional)
- Dry run (checkbox)
- Progress bar
- Results display

**Add Robots.txt:**
- Provider filter (dropdown: All, S3, Bunny)
- Dry run (checkbox)
- Progress bar
- Results display

**Verify Deployment:**
- Same as Content → Verify Deployment

**Database Tools:**
- Initialize database (with confirmation)
- Reset database (with strong confirmation)
- Run migrations (list with status)
- Check migration status

---

#### 6. Settings
**User Management:**
- List users (table)
- Add user form
- Edit user form
- Delete user (with confirmation)
- Role management

**System Configuration:**
- Display current config
- API keys management (view/edit)
- Model selection
- Environment variables display

**Authentication:**
- Login/logout
- Session management
- Remember credentials option

---

## Key Forms/Components

### Project Creation Form
**Fields:**
- File upload (CORA or Simple)
  - Drag & drop zone
  - File picker button
  - File validation (must be .xlsx)
- Project name (text input, required)
- Money site URL (text input, URL validation)
- Custom anchors (text input, comma-separated, CORA only)
- Authentication (username/password, optional if session exists)

**Validation:**
- File format check
- URL format validation
- Required field validation

---

### Job Creation Form
**Fields:**
- Project selector (dropdown, required)
- Deployment targets (multi-select with search, optional)
- Tier 1 count (number input, default: 10)
- Tier 2 count (number input, default: 30)
- Model selection (dropdown, optional)
- Output path (file picker, optional)
- Authentication (username/password, optional if session exists)

**Features:**
- Real-time validation
- Preview of job file structure
- Save as template option

---

### Generate Batch Form
**Fields:**
- Job file selector (file picker, required)
- Model override (dropdown, optional)
- Debug mode (checkbox)
- Continue on error (checkbox)
- Authentication (username/password, optional if session exists)

**Features:**
- Progress bar with percentage
- Live log streaming
- Cancel button
- Results summary

---

### Deploy Batch Form
**Fields:**
- Batch/Project selector (dropdown, required)
- Continue on error (checkbox, default: checked)
- Dry run (checkbox)
- Authentication (username/password, optional if session exists)

**Features:**
- Progress bar
- File-by-file status
- Results summary
- Error details

---

### Site Provision Form
**Fields:**
- Site name (text input, required)
- Domain (text input, FQDN validation, required)
- Storage name (text input, required, uniqueness check)
- Region selector (dropdown, required)
- Authentication (username/password, optional if session exists)

**Features:**
- Real-time domain validation
- Storage name uniqueness check
- DNS configuration instructions display
- Copy DNS settings button

---

### Maintenance Tools

#### Update Index Pages
**Fields:**
- Provider filter (dropdown: All, S3, Bunny)
- Force update (checkbox)
- Limit (number input, optional)
- Hostname filter (text input, optional)
- Dry run (checkbox)

**Features:**
- Progress bar
- Site-by-site status
- Results summary

#### Add Robots.txt
**Fields:**
- Provider filter (dropdown: All, S3, Bunny)
- Dry run (checkbox)

**Features:**
- Progress bar
- Results summary

---

## Implementation Notes

### 1. Authentication
Many commands require `--username`/`--password`. Consider:
- **Session-based authentication** in GUI
  - Login page with remember option
  - JWT or session tokens
  - Auto-refresh tokens
- **Remember credentials** option
  - Secure storage (encrypted local storage)
  - Optional "remember me" checkbox
- **Admin vs User role handling**
  - Role-based UI visibility
  - Permission checks before actions
  - Clear error messages for unauthorized actions

---

### 2. File Operations
- **File upload** for CORA/Simple ingestion
  - Drag & drop interface
  - File picker fallback
  - Progress indicator
  - File size validation
  - Format validation (.xlsx only)
- **File picker** for job files
  - Recent files list
  - File browser
  - File validation (JSON format)
- **File download** for link exports
  - CSV export
  - JSON export options
  - Download button with progress

---

### 3. Real-time Updates
- **Progress bars** for generation/deployment
  - Percentage complete
  - Current item being processed
  - Estimated time remaining
  - Cancel button
- **Live status updates**
  - WebSocket or polling for status
  - Real-time log streaming
  - Status badges (Pending, Processing, Complete, Failed)
- **Log streaming**
  - Scrollable log viewer
  - Auto-scroll to bottom
  - Filter by log level
  - Export logs option

---

### 4. Validation
- **File format validation** before upload
  - Check file extension
  - Validate file structure (for spreadsheets)
  - Show clear error messages
- **URL/domain validation**
  - Real-time validation as user types
  - Format checking (http://, https://)
  - Domain format validation
- **Job file JSON validation**
  - Schema validation
  - Syntax checking
  - Required field checking
  - Clear error messages with line numbers

---

### 5. Error Handling
- **Display error messages clearly**
  - User-friendly error messages
  - Technical details in expandable section
  - Error codes for support
- **Retry mechanisms**
  - Retry button for failed operations
  - Automatic retry with exponential backoff
  - Max retry limit
- **Continue-on-error options**
  - Checkbox for batch operations
  - Summary of successes and failures
  - Detailed error log

---

### 6. Data Display
- **Tables** for projects, sites, content
  - Sortable columns
  - Filterable rows
  - Pagination for large datasets
  - Row selection (multi-select)
  - Bulk actions
- **Filters and search**
  - Text search with highlighting
  - Advanced filters (date range, status, etc.)
  - Saved filter presets
- **Pagination**
  - Page size selector
  - Page navigation
  - Total count display
- **Export to CSV/JSON**
  - Export current view (with filters applied)
  - Export all data
  - Custom field selection

---

### 7. User Experience Enhancements
- **Confirmation dialogs** for destructive actions
  - Delete operations
  - Reset database
  - Remove sites
- **Loading states**
  - Skeleton loaders
  - Spinners for buttons
  - Disable forms during submission
- **Success notifications**
  - Toast notifications
  - Success banners
  - Confirmation messages
- **Keyboard shortcuts**
  - Quick navigation
  - Common actions (Ctrl+S to save, etc.)
- **Responsive design**
  - Mobile-friendly layouts
  - Tablet optimization
  - Desktop-first with mobile fallbacks

---

### 8. Performance Considerations
- **Lazy loading** for large lists
  - Virtual scrolling
  - Infinite scroll option
  - Load on demand
- **Caching**
  - Cache project/site lists
  - Cache user session
  - Cache configuration
- **Optimistic updates**
  - Update UI immediately
  - Rollback on error
- **Debouncing** for search/filter inputs
  - Reduce API calls
  - Improve performance

---

### 9. Accessibility
- **Keyboard navigation**
  - Tab order
  - Focus management
  - Keyboard shortcuts
- **Screen reader support**
  - ARIA labels
  - Semantic HTML
  - Alt text for images
- **Color contrast**
  - WCAG AA compliance
  - High contrast mode
- **Error announcements**
  - Screen reader announcements
  - Error message association

---

## Technology Recommendations

### Frontend Framework Options
1. **React + TypeScript**
   - Component library: Material-UI or Ant Design
   - State management: Redux or Zustand
   - Forms: React Hook Form
   - HTTP client: Axios

2. **Vue.js + TypeScript**
   - Component library: Vuetify or Element Plus
   - State management: Pinia
   - Forms: Vee-Validate
   - HTTP client: Axios

3. **Next.js (React)**
   - Full-stack option
   - Server-side rendering
   - API routes

### Backend Integration
- **REST API** (if creating new API endpoints)
  - FastAPI (Python) to match existing codebase
  - OpenAPI/Swagger documentation
- **Direct CLI execution** (simpler initial approach)
  - Execute CLI commands from backend
  - Parse output
  - Return structured data

### Additional Tools
- **File upload**: react-dropzone or similar
- **Tables**: react-table or ag-grid
- **Charts**: Chart.js or Recharts
- **Notifications**: react-toastify or similar
- **Logs**: react-terminal or xterm.js

---

## Next Steps

1. **Choose technology stack**
2. **Set up project structure**
3. **Create API layer** (if needed) or CLI wrapper
4. **Build authentication system**
5. **Implement core pages** (Dashboard, Projects, Sites)
6. **Add forms and validation**
7. **Implement real-time updates**
8. **Add error handling and notifications**
9. **Testing and refinement**
10. **Deployment**

---

## Related Documentation

- [CLI Command Reference](./CLI_COMMAND_REFERENCE.md) - Detailed CLI documentation
- [Job Schema](./job-schema.md) - Job file structure and configuration
- [Architecture](./architecture.md) - System architecture overview
- [Scripts README](../scripts/README.md) - Scripts documentation