LLM API Architecture & Billing System

System Statistics

1. System Overview

The LLM API system routes requests through a unified billing layer with three payment options and four provider backends.

Key Components

2. Provider Routing Chain

The system uses a priority-based fallback chain to select the LLM provider.

Provider Configuration

Model Tier Mapping

3. Three Billing Modes

Users can choose how they want to pay for LLM usage. The billing mode is orthogonal to subscription tier (features).

4. Billing Resolution Flow

Every LLM call goes through this flow to determine billing mode and key source.

Error Handling

5. BYOK Architecture

Bring Your Own Key allows users to use their own Anthropic/OpenAI API keys with full encryption.

Key Storage Schema

billing.user_byok_keys
├── id UUID PRIMARY KEY
├── user_id UUID NOT NULL
├── provider VARCHAR(20)         -- 'anthropic' | 'openai'
├── encrypted_key TEXT           -- AES-256-GCM ciphertext (base64)
├── key_iv TEXT                  -- Initialization vector (hex)
├── key_tag TEXT                 -- Authentication tag (hex)
├── key_last_four VARCHAR(4)     -- Display only ("...wxyz")
├── is_valid BOOLEAN             -- Validation status
├── last_validated_at TIMESTAMPTZ
├── validation_error TEXT
├── total_calls INTEGER          -- Usage counter
├── label VARCHAR(100)           -- User-provided name
└── UNIQUE(user_id, provider)    -- One key per provider per user
    

6. Key Encryption (AES-256-GCM)

BYOK keys are encrypted with per-user key derivation for defense in depth.

Security Properties

Decryption Flow

function decryptKey(encrypted, iv, tag, userId) {
  // 1. Derive same key using scrypt
  const derivedKey = scrypt(BYOK_ENCRYPTION_SECRET, `byok:${userId}`, 32);

  // 2. Create decipher with IV
  const decipher = createDecipheriv('aes-256-gcm', derivedKey, Buffer.from(iv, 'hex'));

  // 3. Set auth tag (validates integrity)
  decipher.setAuthTag(Buffer.from(tag, 'hex'));

  // 4. Decrypt base64 ciphertext
  let plaintext = decipher.update(encrypted, 'base64', 'utf8');
  plaintext += decipher.final('utf8');

  return plaintext; // User's API key
}
    

7. BYOK User Interface

Users manage their API keys at /settings/ai-billing/byok.

API Endpoints

8. Key Validation

Keys are validated against provider APIs before storage and can be re-validated on demand.

9. Credit System Flow

Credits use a reserve-finalize pattern to prevent overdrafts during concurrent calls.

Credit Transaction Types

10. Model Pricing

Pricing is stored in billing.ai_model_pricing and admin-configurable.

Cost Calculation Formula

inputCost = (inputTokens / 1,000,000) × inputCostPer1M
outputCost = (outputTokens / 1,000,000) × outputCostPer1M
baseCost = inputCost + outputCost
totalCost = baseCost × (1 + markupPercent / 100)
finalCents = Math.max(1, Math.ceil(totalCost × 100))
    

11. Usage Tracking

All LLM calls are logged to predict.ai_usage_log regardless of billing mode.

Usage Query Examples

-- User's daily usage (last 30 days)
SELECT DATE(created_at) as date,
       COUNT(*) as total_calls,
       SUM(input_tokens + output_tokens) as total_tokens,
       SUM(estimated_cost_cents) as total_cost_cents
FROM predict.ai_usage_log
WHERE user_id = $1 AND created_at >= NOW() - INTERVAL '30 days'
GROUP BY DATE(created_at)
ORDER BY date DESC;

-- Usage breakdown by billing mode
SELECT billing_mode, COUNT(*) as calls,
       SUM(input_tokens + output_tokens) as tokens
FROM predict.ai_usage_log
WHERE created_at >= NOW() - INTERVAL '7 days'
GROUP BY billing_mode;

-- Model popularity
SELECT model, COUNT(*) as call_count
FROM predict.ai_usage_log
GROUP BY model ORDER BY call_count DESC;
    

12. Organization Spending Limits

Organization owners can set daily spending limits for members in credits mode.

Limit Configuration

13. LLM-Powered Features

Complete inventory of features that consume LLM API calls.

User-Initiated Features

Automated Features (Platform Cost)

14. Cost Breakdown

Estimated annual costs by feature category.

Cost Control Mechanisms

15. Platform vs BYOK Gaps

Critical differences and limitations between billing modes.

Feature Availability Matrix

Mode-Specific Limitations

Identified Gaps

16. Database Schema

Key tables for LLM billing and usage tracking.

Key Files

Environment Variables

Provider Configuration

Encryption & Billing