# 09 — Deployment & Operations

## 1. Environments

| Env | Purpose | Infra | Provider apps |
|-----|---------|-------|---------------|
| local | Development | Laravel Sail (Docker): MySQL 8.4, Redis 7, MinIO, Mailpit, ClamAV | Fakes + optional personal sandbox apps |
| testing / CI | Automated tests | GitHub Actions services: MySQL 8.4, Redis | Fakes + `Http::fake` fixtures |
| staging | Release validation, provider app-review screencasts | Same shape as prod, smaller | Separate staging apps for every provider |
| production | Live | See below | Production apps (approved) |

## 2. Production topology (AWS `ap-south-1`, cloud-portable)

```
Route 53 / Cloudflare DNS (one record: app.example.com)
  → ALB (TLS, WAF) ─┬─ web ASG / ECS service: Nginx + PHP-FPM 8.3 (stateless)
  worker services (ECS/EC2, separate scaling):
     horizon-publishing   (queues: publishing)          — scale on queue depth
     horizon-media        (media; FFmpeg + ClamAV sidecar) — CPU heavy
     horizon-sync         (sync, webhooks)
     horizon-general      (default, notifications, reports, ai)
     provisioner          (provisioning; ONLY service with DB_PROVISIONER creds)
     scheduler            (single instance: schedule:run each minute, onOneServer)
  RDS / Aurora MySQL 8.x (Multi-AZ, private subnets) — master DB + tenant DBs (cluster pool)
  ElastiCache Redis 7 (TLS, AUTH, Multi-AZ) — queues, cache, locks, sessions
  S3 (private, SSE-KMS, versioning, lifecycle) + CloudFront signed URLs for media delivery
  Secrets Manager (app secrets, tenant DB credentials) · KMS
  SES / Postmark (mail) · Sentry · CloudWatch / Grafana
```

- Images are built once per commit (`php:8.3-fpm` + extensions: `pdo_mysql`, `redis`, `intl`, `gd`/`imagick`, `pcntl`, `bcmath`, `zip`). The same image runs every role with a different entrypoint.
- Config caches are built at boot: `config:cache route:cache view:cache event:cache`.
- Deploys are zero-downtime (rolling): `php artisan migrate --database=master --force` → `tenants:migrate --force` (batched, with progress and failure report) → `horizon:terminate` for graceful worker restart.
- Feature flags (Pennant) gate progressive rollout per tenant.

## 3. Scheduled tasks (Laravel Scheduler)

| Command | Frequency | Notes |
|---------|-----------|-------|
| `outbox:relay` | every minute | Per-tenant, bounded batch, `withoutOverlapping` |
| `publishing:dispatch-due` | every minute | `onOneServer`, SKIP LOCKED |
| `publishing:sweep-stale` | every 5 min | Re-queue orphaned `enqueued` rows, reconcile stuck `in_flight` |
| `tokens:refresh-due` | hourly | Tokens expiring < 72 h |
| `accounts:health-check` | daily | |
| `metrics:sync-due` | every 15 min (staggered per account) | M5 |
| `usage:rollup` | hourly | Tenant events → master `usage_summaries` |
| `reports:send-scheduled` | every 5 min | M5 |
| `approvals:remind-and-escalate` | every 15 min | M4 |
| `billing:dunning` | daily | M6 |
| `backups:tenants` | nightly | Per-tenant logical dumps |
| `backups:verify-restore` | weekly | Restores a random tenant into a scratch DB and runs checks |
| `oauth-states:prune`, `webhook-events:prune`, `logs:prune` | daily | Retention |
| `horizon:snapshot`, `pulse:check` | per docs | |

## 4. Backup & restore

**Layers**
1. **Cluster level:** RDS automated backups + binlog PITR (35-day retention), daily snapshots copied cross-region.
2. **Per-tenant logical backups:** nightly `mysqldump --single-transaction --routines --triggers social_tenant_xxxx`, compressed, encrypted (KMS envelope), stored at `s3://backups/tenants/{tenant_id}/{date}.sql.zst.enc`, with checksum, recorded in master `backup_runs`. Retention: 30 daily, 12 monthly.
3. **Master DB:** the same, hourly.
4. **Media:** S3 versioning + replication. Tenant prefix makes per-tenant export/restore possible.

**Runbook — restore a single tenant** (`docs/runbooks/restore-tenant.md`, written in M1, drilled in M6)
1. Put the tenant in `suspended` (reason `maintenance_restore`). This pauses dispatch for the tenant.
2. `php artisan tenants:restore {tenant} --backup={id} --into=social_tenant_xxxx_restore` restores to a side DB.
3. Run `tenants:verify --database=…_restore` (schema version, row counts, smoke queries).
4. Swap: update `tenant_database_connections.database_name` (master TX), rotate credentials, and grant on the new schema.
5. Reconcile publishing: cancel dispatch rows newer than the backup point. Targets in `publishing/unknown_outcome` go through reconciliation.
6. Reactivate the tenant. Keep the old DB for 7 days, then drop. All steps are audited in `admin_audit_logs`.

**Runbook — rotate tenant DB credential:** `tenants:rotate-credentials {tenant}` → create a new password → `ALTER USER` → store a new secret version → verify → bump `secret_version` → workers pick it up on the next `initialize()` (no caching beyond the request/job).

## 5. Observability

| Signal | Tooling | Alerts |
|--------|---------|--------|
| Errors | Sentry (tags: tenant_id, provider, queue; tokens scrubbed) | New issue in publishing/provisioning |
| Queues | Horizon metrics, queue depth/wait time | Publishing wait > 2 min; failed jobs spike |
| App perf | Laravel Pulse, slow queries | p95 > 800 ms on composer/calendar |
| Publishing health | Custom metrics: success rate per provider, unknown_outcome count | Provider success < 95 % for 15 min |
| Tokens | Count of `needs_reconnect`, tokens expiring < 72 h | Daily digest to agencies + admin |
| Quota | Per-provider usage % | > 80 % |
| Backups | `backup_runs` status | Any failure; weekly restore test failure |
| Infra | CloudWatch (RDS CPU/connections, Redis memory) | Standard thresholds |

**MySQL connection budget:** tenant connections are opened lazily per request/job and purged. Use RDS Proxy or limit FPM/Horizon process counts. Watch `Threads_connected`.

## 6. Security operations

- TLS everywhere (HSTS), security headers (CSP with Livewire nonce, X-Frame-Options DENY except portal embeds if required), private subnets for DB/Redis.
- Least-privilege DB users: `app_web` (DML on master), `t_{n}` per tenant (DML + DDL limited to its schema, used for tenant migrations), `app_provisioner` (provisioning only), `app_backup` (read-only `SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, EVENT`).
- Dependency scanning (`composer audit`, Dependabot), container image scanning.
- Annual pen test plus pre-launch review against OWASP ASVS L2.
- Incident response runbook and status page.