# Agencio Predict - Infrastructure & Disaster Recovery

**Last Updated**: 2026-04-24
**Region**: ap-southeast-1 (Singapore)

---

## Quick Connection Reference

### SSH to EC2
```bash
ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122
```

### PostgreSQL (from your machine)
```bash
# CLI via SSH tunnel
ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122 \
  "docker exec -it predict-postgres psql -U postgres -d predict_db"

# Or direct connection (requires psql installed locally)
psql "postgresql://postgres:postgres@54.255.100.122:5432/predict_db"

# GUI Tools (TablePlus, DBeaver, pgAdmin):
#   Host: 54.255.100.122
#   Port: 5432
#   Database: predict_db
#   User: postgres
#   Password: postgres
```

### Redis (from your machine)
```bash
# CLI via SSH tunnel
ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122 \
  "docker exec -it predict-redis redis-cli"

# Or direct connection (requires redis-cli installed locally)
redis-cli -h 54.255.100.122 -p 6379

# GUI Tools (RedisInsight, Medis):
#   Host: 54.255.100.122
#   Port: 6379
#   Password: (none)
```

---

## What's Running on EC2

| Container | Image | Port | Purpose |
|-----------|-------|------|---------|
| `predict-postgres` | `postgres:16-alpine` | 5432 | Main database (186 tables across 9 schemas) |
| `predict-redis` | `redis:7-alpine` | 6379 | Cache, sessions, rate limiting, pub/sub |

### Database Schemas
| Schema | Tables | Purpose |
|--------|--------|---------|
| `predict` | 116 | Events, predictions, algorithms, AI insights |
| `trading` | 32 | Positions, orders, executions, broker configs |
| `derivatives` | 10 | VIX, funding rates, OI, liquidations |
| `omniscient` | 10 | All-Seeing Eye AI orchestration |
| `billing` | 7 | Subscriptions, plans, usage metering |
| `auth` | 4 | Users, sessions, MFA, API keys |
| `marketing` | 4 | Attribution, campaigns, analytics |
| `assistant` | 2 | AI assistant conversations |

### Server Resources
| Resource | Value |
|----------|-------|
| Instance Type | t3.small (2 vCPU, 2GB RAM) |
| Disk | 30GB (38% used) |
| Memory | 1.9GB (225MB used) |
| OS | Amazon Linux 2023 |

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                              PRODUCTION                                      │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   ┌─────────────┐     ┌──────────────────────────────────────────────────┐ │
│   │   Users     │────▶│  Vercel Edge Network (Global CDN)                │ │
│   └─────────────┘     │  predict.agencio.cloud                           │ │
│                       │  - Next.js Frontend (SSR)                        │ │
│                       │  - API Routes (Serverless Functions)             │ │
│                       │  - Auto-scaling (0 to ∞)                         │ │
│                       └──────────────────┬───────────────────────────────┘ │
│                                          │                                  │
│                                          ▼                                  │
│   ┌──────────────────────────────────────────────────────────────────────┐ │
│   │  AWS ap-southeast-1 (Singapore)                                      │ │
│   │                                                                      │ │
│   │  ┌────────────────┐  ┌────────────────┐  ┌────────────────────────┐ │ │
│   │  │ EC2 t3.small   │  │ S3 Buckets     │  │ Cognito User Pool      │ │ │
│   │  │ 54.255.100.122 │  │                │  │ ap-southeast-1_z4jaYFNJX│ │
│   │  │                │  │ agencio-       │  │                        │ │ │
│   │  │ ┌────────────┐ │  │ predict-       │  │ - MFA: OPTIONAL        │ │ │
│   │  │ │ PostgreSQL │ │  │ uploads        │  │ - TOTP enabled         │ │ │
│   │  │ │ :5432      │ │  │                │  │ - Password: 12+ chars  │ │ │
│   │  │ └────────────┘ │  │ agencio-       │  │   + upper/lower/num/sym│ │ │
│   │  │ ┌────────────┐ │  │ predict-       │  │                        │ │ │
│   │  │ │ Redis      │ │  │ storage        │  │ App Clients:           │ │ │
│   │  │ │ :6379      │ │  │                │  │ - web (with secret)    │ │ │
│   │  │ └────────────┘ │  └────────────────┘  │ - web-public (no secret)│ │
│   │  └────────────────┘                      └────────────────────────┘ │ │
│   └──────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## 1. AWS Resources

### 1.1 EC2 Instance

| Property | Value |
|----------|-------|
| Instance ID | `i-030715bbe93611746` |
| Name | `agencio-predict-backend` |
| Type | `t3.small` (2 vCPU, 2GB RAM) |
| Public IP | `54.255.100.122` |
| Region | `ap-southeast-1` |
| SSH Key | `~/.ssh/agencio-predict-sg.pem` |

**SSH Access:**
```bash
ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122
```

**Docker Containers Running:**
| Container | Image | Ports |
|-----------|-------|-------|
| `predict-postgres` | `postgres:16-alpine` | `5432:5432` |
| `predict-redis` | `redis:7-alpine` | `6379:6379` |

### 1.2 Security Group

| Group ID | `sg-00c0ecc701b7533dc` |
|----------|------------------------|
| Name | `agencio-predict-SecurityGroup-uSwA21UNyTeY` |

**Ingress Rules:**
| Port | Protocol | Source | Description |
|------|----------|--------|-------------|
| 22 | TCP | 0.0.0.0/0 | SSH |
| 80 | TCP | 0.0.0.0/0 | HTTP |
| 443 | TCP | 0.0.0.0/0 | HTTPS |
| 3000 | TCP | 0.0.0.0/0 | Next.js API |
| 5432 | TCP | 0.0.0.0/0 | PostgreSQL |
| 6379 | TCP | 0.0.0.0/0 | Redis |

> ⚠️ **Security Note**: Ports 5432/6379 are open to 0.0.0.0/0. For production hardening, restrict to Vercel IP ranges or use VPC peering.

### 1.3 S3 Buckets

| Bucket | Purpose |
|--------|---------|
| `agencio-predict-uploads` | User avatars, file uploads |
| `agencio-predict-storage` | General storage |

**CORS Configuration (agencio-predict-uploads):**
```json
{
  "CORSRules": [{
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "DELETE", "HEAD"],
    "AllowedOrigins": [
      "https://predict.agencio.cloud",
      "https://*.vercel.app",
      "http://localhost:3333"
    ],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3600
  }]
}
```

### 1.4 Cognito User Pool

| Property | Value |
|----------|-------|
| Pool ID | `ap-southeast-1_z4jaYFNJX` |
| Pool Name | `agencio-predict-users` |
| MFA | `OPTIONAL` (TOTP enabled) |

**Password Policy:**
- Minimum length: 12 characters
- Requires: uppercase, lowercase, numbers, symbols
- Temporary password validity: 7 days

**App Clients:**
| Client Name | Client ID | Has Secret |
|-------------|-----------|------------|
| `agencio-predict-web` | `4uo6tnpi35k5t56f2n1isn6rba` | Yes |
| `agencio-predict-web-public` | `4ha765t83ga216obbnsgiov42g` | No |

---

## 2. Vercel Configuration

### 2.1 Project Info

| Property | Value |
|----------|-------|
| Team | `thejustinjames-projects` |
| Project | `agencio-predict-web` |
| Domain | `predict.agencio.cloud` |
| Framework | Next.js |
| Region | Auto (Edge) |

### 2.2 Environment Variables (Production)

```
# Auth
AUTH_PROVIDER=cognito
COGNITO_USER_POOL_ID=ap-southeast-1_z4jaYFNJX
COGNITO_CLIENT_ID=4uo6tnpi35k5t56f2n1isn6rba
COGNITO_CLIENT_SECRET=<encrypted>
COGNITO_REGION=ap-southeast-1
COGNITO_DOMAIN=<encrypted>
JWT_SECRET=<encrypted>

# Database
DATABASE_URL=postgresql://postgres:<password>@54.255.100.122:5432/predict_db
DATABASE_POOL_MIN=2
DATABASE_POOL_MAX=10

# Cache
REDIS_URL=redis://54.255.100.122:6379

# Storage
STORAGE_PROVIDER=s3
S3_BUCKET_PREFIX=agencio-
AWS_REGION=ap-southeast-1
AWS_ACCESS_KEY_ID=<encrypted>
AWS_SECRET_ACCESS_KEY=<encrypted>

# API Keys
ANTHROPIC_API_KEY=<encrypted>
CLAUDE_API_KEY=<encrypted>
FINNHUB_API_KEY=<encrypted>
NEWSAPI_KEY=<encrypted>
FRED_API_KEY=<encrypted>
GITHUB_TOKEN=<encrypted>

# Notifications
VAPID_PUBLIC_KEY=<encrypted>
VAPID_PRIVATE_KEY=<encrypted>
VAPID_SUBJECT=<encrypted>
TWILIO_ACCOUNT_SID=<encrypted>
TWILIO_AUTH_TOKEN=<encrypted>
TWILIO_PHONE_NUMBER=<encrypted>

# Email (Azure Graph)
AZURE_CLIENT_ID=<encrypted>
AZURE_CLIENT_SECRET=<encrypted>
AZURE_TENANT_ID=<encrypted>
AZURE_GRAPH_SENDER_EMAIL=<encrypted>

# Captcha
NEXT_PUBLIC_TURNSTILE_SITE_KEY=<encrypted>
TURNSTILE_SECRET_KEY=<encrypted>

# App Config
NODE_ENV=production
API_VERSION=v1
SCHEDULER_ENABLED=true
NEXT_PUBLIC_APP_URL=https://predict.agencio.cloud
NEXT_PUBLIC_API_URL=https://predict.agencio.cloud/api
```

---

## 3. Scaling Architecture

### Current State
- **Frontend/API**: Vercel Serverless (auto-scales 0 → ∞)
- **Database**: Single EC2 Docker container (PostgreSQL 16)
- **Cache**: Single EC2 Docker container (Redis 7)

### Scaling Options

#### Option A: Vertical Scaling (Quick)
```bash
# Upgrade EC2 instance
aws ec2 modify-instance-attribute --instance-id i-030715bbe93611746 \
  --instance-type t3.medium  # or t3.large, t3.xlarge
```

#### Option B: Horizontal Scaling with ECS Fargate (Recommended)
The repo includes a CloudFormation template at `deploy/aws/cloudformation.yml`:

```bash
# Deploy ECS cluster with auto-scaling
cd deploy/aws
./deploy.sh
```

This creates:
- ECS Fargate cluster
- Application Load Balancer
- Auto-scaling (2-10 tasks based on CPU/memory)
- CloudWatch logging
- Secrets Manager integration

#### Option C: Managed Services (Production-Grade)

| Current | Upgrade To | Benefits |
|---------|------------|----------|
| EC2 PostgreSQL | **RDS PostgreSQL** | Auto-backups, Multi-AZ, read replicas |
| EC2 Redis | **ElastiCache Redis** | Clustering, auto-failover |
| EC2 containers | **ECS Fargate** | No server management, auto-scaling |

**Migration Commands:**
```bash
# Create RDS instance
aws rds create-db-instance \
  --db-instance-identifier agencio-predict-db \
  --db-instance-class db.t3.medium \
  --engine postgres \
  --engine-version 16 \
  --master-username postgres \
  --master-user-password <password> \
  --allocated-storage 100 \
  --multi-az \
  --region ap-southeast-1

# Create ElastiCache cluster
aws elasticache create-cache-cluster \
  --cache-cluster-id agencio-predict-cache \
  --cache-node-type cache.t3.micro \
  --engine redis \
  --num-cache-nodes 1 \
  --region ap-southeast-1
```

---

## 4. Disaster Recovery Procedures

### 4.1 Database Backup

**Manual Backup:**
```bash
# SSH to EC2 and dump database
ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122 \
  "docker exec predict-postgres pg_dump -U postgres predict_db" > backup_$(date +%Y%m%d).sql

# Upload to S3
aws s3 cp backup_$(date +%Y%m%d).sql s3://agencio-predict-storage/backups/
```

**Restore from Backup:**
```bash
# Download backup
aws s3 cp s3://agencio-predict-storage/backups/backup_YYYYMMDD.sql ./

# Restore to database
cat backup_YYYYMMDD.sql | ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122 \
  "docker exec -i predict-postgres psql -U postgres -d predict_db"
```

### 4.2 EC2 Recovery

**Create AMI Snapshot:**
```bash
aws ec2 create-image \
  --instance-id i-030715bbe93611746 \
  --name "agencio-predict-backup-$(date +%Y%m%d)" \
  --no-reboot \
  --region ap-southeast-1
```

**Launch from AMI:**
```bash
aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.small \
  --key-name agencio-predict-sg \
  --security-group-ids sg-00c0ecc701b7533dc \
  --region ap-southeast-1
```

### 4.3 Vercel Recovery

**Rollback Deployment:**
```bash
# List recent deployments
npx vercel ls

# Promote previous deployment
npx vercel rollback <deployment-url>
```

**Pull Environment Variables:**
```bash
npx vercel env pull .env.production.local
```

### 4.4 Cognito Recovery

**Export Users:**
```bash
aws cognito-idp list-users \
  --user-pool-id ap-southeast-1_z4jaYFNJX \
  --region ap-southeast-1 > cognito_users_backup.json
```

---

## 5. Monitoring & Alerts

### Health Check Endpoints
- **App Health**: `https://predict.agencio.cloud/api/predict/v1/health`
- **Vercel Status**: `https://vercel-status.com/`
- **AWS Status**: `https://health.aws.amazon.com/health/status`

### Key Metrics to Monitor
| Metric | Warning | Critical |
|--------|---------|----------|
| API Response Time | >500ms | >2000ms |
| Error Rate | >1% | >5% |
| Database Connections | >80% pool | >95% pool |
| Redis Memory | >70% | >90% |
| EC2 CPU | >70% | >90% |

---

## 6. Quick Reference Commands

```bash
# SSH to backend
ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122

# Check containers
docker ps
docker logs predict-postgres --tail 100
docker logs predict-redis --tail 100

# Database CLI
docker exec -it predict-postgres psql -U postgres -d predict_db

# Redis CLI
docker exec -it predict-redis redis-cli

# Deploy to Vercel
npx vercel --prod --yes

# Add Vercel env var
echo "value" | npx vercel env add VAR_NAME production --yes

# Run database migration
cat db/migrations/NNN.sql | ssh -i ~/.ssh/agencio-predict-sg.pem ec2-user@54.255.100.122 \
  "docker exec -i predict-postgres psql -U postgres -d predict_db"
```

---

## 7. Stripe Billing Configuration

### API Keys (Live Mode)

| Key | Environment Variable | Location |
|-----|---------------------|----------|
| Publishable Key | `NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY` | Vercel Production |
| Secret Key | `STRIPE_SECRET_KEY` | Vercel Production |
| Webhook Secret | `STRIPE_WEBHOOK_SECRET` | Vercel Production |

### Webhook Endpoint

| Property | Value |
|----------|-------|
| Destination ID | `we_1TPY1tFVPivSwdo96Yq5w18i` |
| Name | `elegant-voyage` |
| URL | `https://predict.agencio.cloud/api/predict/v1/billing/webhooks/stripe` |
| API Version | `2026-02-25.clover` |
| Status | Active |

### Events Listened

| Event | Purpose |
|-------|---------|
| `checkout.session.completed` | New subscription created via checkout |
| `customer.subscription.created` | Subscription created |
| `customer.subscription.updated` | Plan change, renewal, etc. |
| `customer.subscription.deleted` | Subscription cancelled |
| `invoice.paid` | Successful payment |
| `invoice.payment_failed` | Failed payment (trigger dunning) |

### Stripe Dashboard Links

- **Dashboard**: https://dashboard.stripe.com
- **Webhooks**: https://dashboard.stripe.com/webhooks
- **API Keys**: https://dashboard.stripe.com/apikeys
- **Customers**: https://dashboard.stripe.com/customers
- **Subscriptions**: https://dashboard.stripe.com/subscriptions

### Testing Webhooks

```bash
# Send test event from Stripe CLI
stripe trigger checkout.session.completed

# Or from Dashboard:
# Webhooks → elegant-voyage → Send test webhook
```

### Rotating Keys

If keys are compromised:

1. **Stripe Dashboard** → API Keys → Roll key
2. Update Vercel:
```bash
npx vercel env rm STRIPE_SECRET_KEY production --yes
echo "sk_live_NEW_KEY" | npx vercel env add STRIPE_SECRET_KEY production --yes
npx vercel --prod --yes
```

3. For webhook secret:
```bash
# In Stripe Dashboard: Webhooks → endpoint → Roll secret
npx vercel env rm STRIPE_WEBHOOK_SECRET production --yes
echo "whsec_NEW_SECRET" | npx vercel env add STRIPE_WEBHOOK_SECRET production --yes
npx vercel --prod --yes
```

---

## 8. Contacts & Escalation

| Role | Contact |
|------|---------|
| Platform Owner | justin@agencio.cloud |
| AWS Account | thejustinjames |
| Vercel Team | thejustinjames-projects |

---

**Document Version**: 1.0
**Next Review**: 2026-05-24