# Market Paradox Analysis

> CNN-Style Market Paradox Detection + AI Narrative Synthesis

This document describes the Market Paradox Analysis feature, which provides sophisticated market analysis similar to CNN's "We're all reading the markets wrong" article by detecting when market behavior contradicts expected fundamentals and synthesizing AI-powered narratives.

## Overview

The feature consists of four integrated components:

1. **Economic Surprise Index** - Composite measure of economic data vs expectations
2. **Geopolitical Event Tagging** - Pattern-based detection of geopolitical events in news
3. **Market Paradox Detection** - Rule-based detection of market contradictions
4. **LLM Narrative Synthesis** - Goldman-quality market analysis prose generation
5. **DSL Primitives** - Integration with AI Algorithm Builder for trading strategies

## Architecture

```
┌─────────────────────────────────────────────────────────────────────────┐
│                        Market Paradox Analysis                           │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │   Economic   │  │ Geopolitical │  │    Market    │  │   Headline   │ │
│  │   Surprise   │  │    Event     │  │    Prices    │  │  Sentiment   │ │
│  │    Index     │  │   Tagging    │  │  + Yields    │  │   + News     │ │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘ │
│         │                 │                 │                 │          │
│         └────────────────┴─────────────────┴─────────────────┘          │
│                                    │                                     │
│                                    ▼                                     │
│                    ┌───────────────────────────────┐                    │
│                    │      Paradox Detector         │                    │
│                    │    (10 Rule-based checks)     │                    │
│                    └───────────────┬───────────────┘                    │
│                                    │                                     │
│                                    ▼                                     │
│                    ┌───────────────────────────────┐                    │
│                    │    Narrative Synthesizer      │                    │
│                    │  (LLM Goldman-quality prose)  │                    │
│                    └───────────────────────────────┘                    │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

## Database Migrations

| Migration | Purpose |
|-----------|---------|
| 149 | `predict.economic_surprise_snapshots` - Daily ESI snapshots |
| 150 | `predict.news_archive.geopolitical_event_ids` + `predict.geopolitical_event_aggregates` |
| 151 | `predict.detected_paradoxes` + `predict.market_narratives` |
| 152 | `predict.user_algorithms.macro_overlay_settings` + `predict.algorithm_trades.macro_overlay_influence` |
| 161 | `predict.paradox_triggers` + `predict.crisis_validation` + `predict.regime_thresholds` - Calibration system |

## Component Details

### 1. Economic Surprise Index

**File:** `packages/be/src/overlays/surprise-index.ts`

Builds a composite economic surprise index (similar to Citi ESI) from FRED data:

| Component | FRED Series | Weight | Category |
|-----------|-------------|--------|----------|
| NFP | PAYEMS | 25% | Labor |
| CPI | CPIAUCSL | 20% | Inflation |
| GDP | A191RL1Q225SBEA | 20% | GDP |
| Retail Sales | RSAFS | 15% | Consumer |
| ISM Manufacturing | MANEMP | 10% | Manufacturing |
| Housing Starts | HOUST | 10% | Housing |

**Scoring:** Calculates MoM changes, compares to expectations, normalizes to -100 to +100 scale.

**API Endpoints:**
- `GET /api/predict/v1/overlays/surprise-index` - Current composite
- `GET /api/predict/v1/overlays/surprise-index/history?days=90` - Historical

### 2. Geopolitical Event Tagging

**Files:**
- `packages/be/src/news/geopolitical-events.ts` - Event registry
- `packages/be/src/scheduler/news-collector.ts` - Integration

**Tracked Events:**

| Event ID | Description | Key Assets Affected |
|----------|-------------|---------------------|
| `iran-conflict` | Iran / Strait of Hormuz | CL, XLE, GLD |
| `china-taiwan` | China-Taiwan Tensions | TSM, SOXL, LMT |
| `russia-ukraine` | Russia-Ukraine War | WEAT, UNG, LMT |
| `middle-east-escalation` | Israel/Hezbollah/Houthis | CL, GLD |
| `trade-war` | Trade War / Tariffs | FXI, EEM |
| `north-korea` | North Korea Tensions | EWY, LMT |
| `energy-crisis` | Energy Supply Disruption | CL, UNG, XLE |
| `cyber-attack` | Major Cyber Attack | HACK, PANW, CRWD |

**API Endpoints:**
- `GET /api/predict/v1/news/geopolitical/active` - Active events with intensity
- `GET /api/predict/v1/news/geopolitical/:eventId/articles` - Tagged articles

### 3. Market Paradox Detection

**Files:**
- `packages/be/src/all-seeing-eye/insights/paradox-rules.ts` - Rule definitions
- `packages/be/src/all-seeing-eye/insights/paradox-detector.ts` - Detection engine
- `packages/be/src/all-seeing-eye/calibration/` - Calibration module (see below)

**Paradox Rules (18 total):**

| Rule ID | Name | Severity | Detection Logic |
|---------|------|----------|-----------------|
| `vix-complacency` | VIX Complacency | High | VIX < regime threshold AND geopolitical > 60 |
| `bad-news-rally` | Bad News Is Good News | Medium | ESI < -25 AND SPY 1d > 0.8% |
| `yield-equity-divergence` | Yield-Equity Divergence | Medium | 10Y up >15bps AND SPY 1w > 1.5% |
| `fear-greed-mismatch` | Sentiment-Price Mismatch | Low | Sentiment < -0.25 AND SPY 1w > 1.2% |
| `prediction-engine` | Markets As Prediction Engines | Medium | Strong sentiment opposite to price direction |
| `crisis-calm` | Crisis Without Panic | Medium | 2+ geo events, VIX < 18, gold flat |
| `inversion-rally` | Inverted Curve Rally | Medium | 2Y10Y < -0.2% AND SPY 1w > 1.5% |
| `crypto-equity-decouple` | Crypto-Equity Decoupling | Low | BTC/SPY opposite directions |
| `vix-spike-no-selloff` | VIX Spike Without Selloff | Low | VIX > 22 AND SPY flat/up |
| `surprise-regime-mismatch` | Economic vs Regime | Low | ESI > 30 AND regime BEAR/CRISIS |
| `credit-vix-mispricing` | Credit-VIX Mispricing | High | HY spread > regime threshold AND VIX < 18 |
| `move-vix-bond-panic` | MOVE-VIX Bond Panic | High | MOVE > regime threshold AND VIX < 18 |
| `curve-steepen-equity-down` | Steepening + Equities Down | Medium | Curve steepening AND SPY 1w < -2% |
| `breakeven-collapse-equity-rally` | Deflation Fear + Rally | Medium | Breakeven < 1.8% AND SPY 1w > 1.5% |
| `real-yield-spike-equity-calm` | Real Yield Spike | Medium | Real yield > regime threshold AND SPY stable |
| `sofr-stress-no-panic` | Funding Stress No Panic | High | SOFR-FF spread > regime threshold AND VIX < 20 |
| `peripheral-stress-no-contagion` | Eurozone Stress Contained | Low | BTP-Bund > 200bps AND SPY stable |
| `long-end-selloff-short-calm` | Long-End Selloff | Low | 5s30s steepening, 3m10y stable |

**API Endpoints:**
- `GET /api/predict/v1/all-seeing-eye/paradoxes` - Active paradoxes
- `GET /api/predict/v1/all-seeing-eye/paradoxes/history?days=7` - Historical
- `GET /api/predict/v1/all-seeing-eye/calibration/stats` - False positive rates
- `GET /api/predict/v1/all-seeing-eye/calibration/report` - Full calibration report

### 3a. Historical Crisis Calibration

**File:** `packages/be/src/all-seeing-eye/calibration/historical-crisis-data.ts`

Documented signal values for 6 major historical crises used to validate and calibrate paradox rule thresholds:

| Crisis | Dates | VIX Peak | MOVE Peak | HY Spread | Key Lessons |
|--------|-------|----------|-----------|-----------|-------------|
| 2008 GFC | Sep 2008 - Mar 2009 | 89.53 | 264 | 2182bps | MOVE led VIX by 1-2 weeks |
| 2011 Euro Crisis | Jul 2011 - Sep 2012 | 34.43 | 117 | 875bps | BTP-Bund > 500bps = systemic |
| 2015-16 China/Oil | Aug 2015 - Feb 2016 | 28.14 | 98 | 887bps | VIX < 12 before crash = complacency |
| 2018 Q4 Taper | Oct 2018 - Dec 2018 | 36.07 | 85 | 533bps | Curve flattening preceded selloff |
| 2019 Repo Crisis | Sep 2019 | 15.52 | 62 | 375bps | SOFR spiked 300bps, VIX missed it |
| 2020 COVID | Feb 2020 - Mar 2020 | 82.69 | 163 | 1100bps | VIX 14→82 in 3 weeks |

**Functions:**
- `getCrisisSignals(crisisId)` - Get all signals for a crisis
- `validateThreshold(signal, threshold, operator, crisisId)` - Test if threshold catches crisis
- `validateParadoxRule(ruleId, thresholds)` - Validate rule accuracy across all crises

### 3b. Regime-Aware Thresholds

**File:** `packages/be/src/all-seeing-eye/calibration/historical-crisis-data.ts`

Thresholds adjust based on market regime because "normal" varies by era:

| Regime | Dates | SOFR Stress | MOVE Panic | HY Elevated | VIX Complacent | Real Yield Spike |
|--------|-------|-------------|------------|-------------|----------------|------------------|
| Pre-2008 | < 2008 | 25bps | 150 | 400bps | 12 | 2.5% |
| ZIRP 2009-15 | 2009-2015 | 25bps | 130 | 500bps | 12 | 1.0% |
| Normalization | 2015-2019 | 25bps | 120 | 450bps | 12 | 1.5% |
| Pandemic | 2020-2021 | 15bps | 160 | 550bps | 18 | 0.5% |
| Inflation | 2022-2023 | 30bps | 170 | 500bps | 15 | 2.5% |
| **Post-2023** | 2023+ | **40bps** | 150 | 550bps | 12 | 2.5% |

**Key Insight:** Post-2023, SOFR structurally trades 10-25bps above Fed Funds due to new Fed framework. The 2019 repo crisis threshold (25bps) would false-alarm constantly in 2024+.

**Functions:**
- `detectRegime(date)` - Determine market regime from date
- `getRegimeThreshold(date)` - Get regime-appropriate thresholds

### 3c. False Positive Rate Tracking

**File:** `packages/be/src/all-seeing-eye/calibration/false-positive-tracker.ts`

Tracks paradox rule triggers and compares to actual market outcomes to measure accuracy:

**Expected Outcomes by Rule Type:**

| Outcome Type | SPY Threshold | VIX Threshold | Window |
|--------------|---------------|---------------|--------|
| `equity_selloff` | -5% OR | +50% | 7-30 days |
| `credit_crisis` | -3% OR | +30% | 14 days |
| `volatility_spike` | -2% OR | +25% | 7-14 days |
| `regime_change` | -7% OR | +75% | 30-60 days |

**Tracking Flow:**
1. When a paradox rule triggers, `recordParadoxTrigger()` logs signal values
2. After the outcome window expires, `checkPendingOutcomes()` fetches actual SPY/VIX data
3. `recordParadoxOutcome()` marks trigger as true/false positive
4. `generateCalibrationReport()` produces accuracy stats for all rules

**Database Tables (Migration 161):**
- `paradox_triggers` - Each rule trigger with signal snapshot
- `crisis_validation` - Historical crisis backtests
- `regime_thresholds` - Admin-adjustable regime thresholds (DB > code defaults)

**API Endpoints:**
- `GET /api/predict/v1/all-seeing-eye/calibration/stats` - Per-rule FP rates
- `GET /api/predict/v1/all-seeing-eye/calibration/report` - Full calibration report
- `POST /api/predict/v1/all-seeing-eye/calibration/check-outcomes` - Trigger outcome check (admin)

### 4. LLM Narrative Synthesis

**Files:**
- `packages/be/src/all-seeing-eye/insights/narrative-synthesizer.ts` - Main synthesizer
- `packages/be/src/all-seeing-eye/insights/narrative-prompts.ts` - Prompt templates
- `packages/be/src/all-seeing-eye/orchestrator/index.ts` - Orchestrator integration

**Orchestrator Integration (Wired 2026-05-06):**
The narrative synthesizer is now called as part of the All-Seeing Eye orchestrator cycle:
1. After aggregation, insight generation, and black swan detection
2. `runNarrativeCycle()` builds paradox context from current market data
3. Detects paradoxes using the rule-based engine
4. If paradoxes are detected, synthesizes narrative via LLM (detailed style)
5. Stores both detected paradoxes and narrative to database

This runs every 5 minutes as part of the All-Seeing Eye scheduler job.

**Output Structure:**
```typescript
interface SynthesizedNarrative {
  headline: string;           // "Markets Are Prediction Engines, Not News Reactors"
  summary: string;            // 2-3 sentence executive summary
  fullAnalysis: string;       // 4-6 paragraph Goldman-quality prose
  keyInsights: string[];      // 2-6 bullet points
  tradingImplications: string;// Positioning implications
  dataCitations: Array<{      // Data supporting claims
    claim: string;
    dataPoint: string;
    source: string;
  }>;
  confidence: number;         // 0-1
}
```

**API Endpoints:**
- `GET /api/predict/v1/all-seeing-eye/narratives/latest` - Latest narrative
- `GET /api/predict/v1/all-seeing-eye/narratives/history?limit=10` - Historical
- `POST /api/predict/v1/all-seeing-eye/narratives/generate` - Force regenerate (admin)

### 5. DSL Primitives for AI Algorithm Builder

**Files:**
- `packages/be/src/algorithms/dsl/types.ts` — Primitive definitions
- `packages/be/src/algorithms/dsl/evaluator.ts` — Dispatch logic
- `packages/be/src/algorithms/paper/executor.ts` — Live context wiring
- `packages/be/src/algorithms/backtest/engine.ts` — Backtest stubs
- `packages/be/src/algorithms/macro-overlay-provider.ts` — Data provider

**Macro Overlay Primitives (7 total):**

| Primitive | Returns | Description | Executor | Backtest |
|-----------|---------|-------------|----------|----------|
| `paradox_severity()` | number (0-100) | Current paradox intensity | ✅ Live | Stub: 0 |
| `paradox_active(type)` | boolean | Check if specific paradox is active | ✅ Live | Stub: false |
| `narrative_confidence()` | number (0-1) | LLM confidence in narrative | ✅ Live | Stub: 0.5 |
| `geopolitical_intensity()` | number (0-100) | Active geopolitical risk | ✅ Live | Stub: 20 |
| `surprise_index()` | number (-100 to +100) | Economic surprise index | ✅ Live | Stub: 0 |
| `regime_mismatch()` | boolean | True if regime contradicts sentiment | ✅ Live | Stub: false |
| `narrative_theme(keyword)` | boolean | Check if narrative matches theme | ✅ Live | Stub: false |

**Wiring Status (Updated 2026-05-06):**

✅ **Fully wired** — All 7 primitives are now available in paper/live trading runs:

```
┌─────────────────────────────────────────────────────────────────────┐
│                        All-Seeing Eye                                │
│  ┌─────────────────┐  ┌──────────────────┐  ┌───────────────────┐   │
│  │ Paradox Detector│  │Narrative Synth.  │  │ Geopolitical      │   │
│  │ (18 rules)      │  │(LLM confidence)  │  │ (news aggregation)│   │
│  └────────┬────────┘  └────────┬─────────┘  └─────────┬─────────┘   │
└───────────┼────────────────────┼──────────────────────┼─────────────┘
            │                    │                      │
            ▼                    ▼                      ▼
┌─────────────────────────────────────────────────────────────────────┐
│               getMacroOverlayContext() [cached 60s]                  │
│  Returns: paradoxSeverity, activeParadoxes[], narrativeConfidence,   │
│           narrativeTheme, geopoliticalIntensity, surpriseIndex,      │
│           regime, regimeMismatch                                     │
└───────────────────────────────┬─────────────────────────────────────┘
                                │
            ┌───────────────────┼───────────────────┐
            ▼                   ▼                   ▼
┌───────────────────┐  ┌────────────────┐  ┌────────────────────┐
│  Paper/Live Run   │  │   Backtest     │  │   Live Status UI   │
│  (executor.ts)    │  │   (engine.ts)  │  │ (LiveStatusPanel)  │
│  Real ASE data    │  │   Stub values  │  │ Shows indicators   │
└───────────────────┘  └────────────────┘  └────────────────────┘
```

**Live Status Indicators:**
The following are visible in `/admin/algorithms` LiveStatusPanel:
- `paradoxSeverity` (0-100)
- `narrativeConfidence` (0-1)
- `geoIntensity` (0-100)
- `surpriseIndex` (-100 to +100)

**Backtest Behavior:**
Backtests use deterministic stub values to ensure reproducibility. Historical macro overlay data is not replayed during backtests — strategies should not over-fit to paradox signals that can't be backtested.

**User-Tuneable Settings:**

```typescript
interface MacroOverlaySettings {
  enabled: boolean;
  paradoxWeight: number;        // 0-2x
  narrativeWeight: number;      // 0-2x
  geopoliticalWeight: number;   // 0-2x
  surpriseIndexWeight: number;  // 0-2x
  autoReduceSizeThreshold: number | null;  // Paradox severity threshold
  autoPauseThreshold: number | null;       // Geopolitical intensity threshold
  contrarianOnMismatch: boolean;
  contrarianConfidenceMin: number;
}
```

## Frontend Components

### MarketNarrativePanel

**File:** `packages/fe/src/components/all-seeing-eye/MarketNarrativePanel.tsx`

Displays:
- AI-generated headline and summary
- Key insights as bullet points
- Trading implications (if available)
- Expandable full analysis
- Expandable data citations
- Expandable raw paradoxes
- Context summary (VIX, SPY, yields, sentiment)
- Confidence meter

**Features:**
- Mobile responsive (stacks on small screens)
- Light/dark mode support
- Auto-refresh every 60 seconds
- Manual regenerate button (admin)

**Integration:**
Added to AllSeeingEyeDashboard as "Narrative" tab.

## Data Flow

1. **Collection Phase** (every 15 minutes via scheduler)
   - News collector fetches headlines
   - Geopolitical patterns detected and tagged
   - News stored with `geopolitical_event_ids[]`

2. **Aggregation Phase** (on demand via API)
   - Economic surprise index computed from FRED
   - Geopolitical intensity aggregated from tagged news
   - Market data fetched (VIX, yields, SPY returns)

3. **Detection Phase**
   - Paradox context built from all data sources
   - 10 rules evaluated against context
   - Active paradoxes stored with supporting data

4. **Synthesis Phase** (every 4 hours or on-demand)
   - Paradoxes + context + headlines fed to LLM
   - Goldman-quality narrative generated
   - Stored with data citations

5. **Consumption Phase**
   - Frontend fetches latest narrative
   - DSL primitives read cached context
   - Algorithms can use macro overlay signals

## Example Usage

### Algorithm DSL Example

```yaml
strategy: "Contrarian Fed Watcher"
universe: ["SPY", "QQQ"]
signals:
  paradox_present:
    - paradox_severity() > 50
  go_contrarian:
    - regime_mismatch()
    - narrative_confidence() > 0.7
  geo_risk:
    - geopolitical_intensity() > 70
entry:
  when: all(paradox_present, go_contrarian, !geo_risk)
  size: kelly(0.6, 2.0, 50000)
exit:
  when: paradox_severity() < 20 || geopolitical_intensity() > 85
risk:
  max_position_usd: 50000
  stop_loss_pct: 2
```

### API Response Example

```json
{
  "narrative": {
    "id": "nar_abc123",
    "headline": "Markets Are Prediction Engines, Not News Reactors",
    "summary": "Despite escalating tensions in the Strait of Hormuz, the VIX remains suppressed at 12.3...",
    "fullAnalysis": "The current market behavior reveals a sophisticated understanding...",
    "keyInsights": [
      "VIX at 12.3 despite geopolitical intensity at 67/100",
      "Economic surprise index +15 supports bullish positioning",
      "Yield-equity divergence suggests Fed pause pricing"
    ],
    "tradingImplications": "Consider reduced position sizes given elevated paradox severity...",
    "dataCitations": [
      {"claim": "VIX complacency", "dataPoint": "12.3", "source": "VIX"},
      {"claim": "Geopolitical elevation", "dataPoint": "67/100", "source": "News Aggregates"}
    ],
    "confidence": 0.82,
    "llmModel": "claude-sonnet-4-20250514",
    "generatedAt": "2026-05-01T10:30:00Z"
  },
  "hasNarrative": true
}
```

## Dependencies

| Component | Depends On |
|-----------|------------|
| Economic Surprise Index | FRED API (`FRED_API_KEY`) |
| Geopolitical Tagging | News collector (Finnhub) |
| Paradox Detector | ESI, Geopolitical, Black Swan detector |
| Narrative Synthesizer | Anthropic LLM (`ANTHROPIC_API_KEY`) |
| DSL Primitives | Macro overlay provider |

## Performance

- Economic surprise index: Cached 1 hour
- Macro overlay context: Cached 1 minute
- Paradox detection: ~50ms (rule-based)
- Narrative synthesis: ~5-10s (LLM call)
- Frontend refresh: 60 seconds

## Testing

1. Verify FRED data fetches correctly
2. Test geopolitical pattern matching with sample headlines
3. Validate paradox rules with mock contexts
4. Test narrative generation with sample paradoxes
5. Verify DSL primitives return correct values
6. End-to-end: headline → tag → paradox → narrative → API → UI