Integrations & Data Flow Architecture

Comprehensive view of all external integrations and how data enriches the platform

← Back to Document Index
65+
External Services
16
Categories
5
Brokers
500+
DSL Primitives
188+
Overlay Series

🔀 Platform Data Flow

High-level view of how external data flows through the platform layers.

flowchart TB subgraph External["External Data Sources"] direction TB MP[("Market Prices
Yahoo, Polygon, Finnhub")] PM[("Prediction Markets
Polymarket, Kalshi")] SS[("Social Sentiment
Reddit, Twitter, Telegram")] ME[("Macro Economic
FRED, World Bank, SEC")] DV[("Derivatives
VIX, Funding, OI")] NW[("News
NewsAPI, GDELT")] end subgraph Ingestion["Ingestion Layer"] direction TB SC["Scheduler Jobs
(15min cadence)"] PC["Price Collector"] NI["News Ingestion"] SF["Social Follows"] DS["Derivatives Sync"] end subgraph Processing["Processing Layer"] direction TB DE["Divergence Engine
(Human vs Bot)"] TC["Tick Classifier
(Whale/Bot)"] PD["Pattern Detector
(FVG, Candles)"] BS["Black Swan Detector"] MD["Manipulation Detector"] end subgraph Intelligence["Intelligence Layer"] direction TB ASE["All-Seeing Eye
(Signal Aggregation)"] SH["AI Stock Hunter
(Claude Analysis)"] FM["AI Fund Manager"] AB["AI Algorithm Builder"] end subgraph Execution["Execution Layer"] direction TB PE["Paper Executor"] LE["Live Executor"] BA["Broker Adapters
(Alpaca, IBKR, Binance)"] end MP --> PC PM --> SC SS --> SF ME --> SC DV --> DS NW --> NI PC --> DE PC --> TC PC --> PD SF --> DE DS --> BS DS --> MD DE --> ASE TC --> ASE PD --> ASE BS --> ASE ASE --> SH ASE --> AB SH --> FM AB --> FM AB --> PE AB --> LE LE --> BA

📊 Integration Categories

Provider Asset Classes Key Features Auth Status
Yahoo Finance Stocks, ETFs, Crypto, Forex, Indices OHLCV, company profiles, unlimited history None Live
Polygon.io / Massive Stocks, Options, Forex, Crypto Trades, quotes, aggregates, technicals API Key Live BYOK
Finnhub Stocks, Forex Real-time quotes, fundamentals, calendar API Key Live
CoinGecko Crypto 8000+ coins, market cap, trending Optional Live
Binance Crypto aggTrades, OHLCV, futures, spot None (public) Live
Alpaca Stocks, Crypto Real-time, historical, minute bars OAuth Live
Frankfurter Forex ECB reference rates, 20+ currencies None Live
Provider Markets Coverage Status
Polymarket 200+ Crypto, politics, economy, sports Live
Kalshi 100+ US elections, economic events Live
Metaculus 1000+ Long-term forecasts, science, AI Live
PredictIt 50+ US politics Blocked
Platform Access Data Status
Reddit Public API Posts, comments, upvotes Live
Twitter/X Bearer Token Tweets, engagement metrics Live
Telegram User Bot Token Channel messages (public/private) Live
Discord User Bot Token Server messages Live
Bluesky Public API Posts, reposts Live
Truth Social Scraper Posts Fragile
Provider Data Series Update Frequency Status
FRED Treasury yields, CPI, SOFR, employment Daily/Monthly Live
World Bank International GDP, CPI Quarterly Live
FXStreet Economic calendar events Real-time Live
SEC EDGAR 13F holdings, 13D/13G activist filings As filed Live
Provider Model Use Case Status
Anthropic Claude claude-sonnet-4, claude-haiku-4 Analysis, DSL translation, reports Live BYOK
Agencio LLMRouter Routed Primary when configured Live
Voyage AI voyage-3-lite Text embeddings for RAG Live
OpenAI text-embedding-3-small Embedding fallback Live BYOK
Broker Assets Auth Paper/Live Status
Alpaca Stocks, ETFs, Crypto API Key Both Live
Interactive Brokers Multi-asset Session Both Live
Binance Crypto HMAC Both Live
Pepperstone Forex, CFDs OAuth 2.0 Both Live
Charles Schwab US Equities, Options OAuth 2.0 Both Live

🔐 Polygon.io BYOK & Tier Detection

Users can bring their own Polygon API key. The system automatically detects tier capabilities.

flowchart LR subgraph User["User Request"] REQ["API Call
(with userId)"] end subgraph Config["Key Resolution"] BYOK{"User has
BYOK key?"} PK["Platform Key"] UK["User BYOK Key"] end subgraph Tier["Tier Detection"] PROBE["Probe 9+ endpoints"] CACHE["Cache result
(24h memory, 7d DB)"] end subgraph Caps["Capabilities"] BASIC["Basic: Aggregates only"] STARTER["Starter: +Snapshots, Technicals"] DEV["Developer: +Trades, Short Interest"] ADV["Advanced: +Quotes, Financials"] end REQ --> BYOK BYOK -->|Yes| UK BYOK -->|No| PK UK --> PROBE PK --> PROBE PROBE --> CACHE CACHE --> BASIC CACHE --> STARTER CACHE --> DEV CACHE --> ADV
Tier Price Rate Limit Key Features
Basic $0/month 5 calls/min EOD aggregates, 2-year history
Starter $29-49/month Higher limits Snapshots, technical indicators, Options Greeks
Developer $79/month Higher limits Trades access, extended hours, short interest
Advanced $199/month Highest limits Real-time quotes, financials, insider transactions
Business Bundle ~$8.50/month Enterprise All features (enterprise pricing)

🔧 DSL Primitive to Data Source Mapping

500+ DSL primitives powered by external integrations.

Price Primitives

  • price() - Yahoo, Polygon, Finnhub
  • sma(), ema() - Computed
  • rsi(), macd() - Polygon Technicals
  • bollinger_bands() - Computed
  • atr(), adx() - Computed

Sentiment Primitives

  • social_sentiment() - Reddit, Twitter
  • news_sentiment() - NewsAPI, GDELT
  • reddit_mentions() - Reddit API
  • twitter_volume() - Twitter API
  • telegram_activity() - User bot

Prediction Primitives

  • prediction_market() - Aggregated
  • polymarket_probability() - Polymarket
  • kalshi_probability() - Kalshi
  • polymarket_smart_money() - Wallet analysis
  • polymarket_manipulation_risk() - Cluster detection

Macro Primitives

  • yield_spread() - FRED
  • fed_funds() - FRED
  • inflation() - FRED, World Bank
  • treasury_yield() - FRED
  • economic_surprise() - Computed

Institutional Primitives

  • institutional_ownership_pct() - SEC 13F
  • institutional_change_qoq() - SEC 13F
  • smart_money_accumulating() - SEC 13F
  • has_activist_position() - SEC 13D/13G
  • activist_count() - SEC 13D/13G

Derivatives Primitives

  • vix() - Yahoo VIX
  • funding_rate() - Binance Futures
  • fear_greed() - Alternative.me
  • open_interest() - Binance, Deribit
  • implied_volatility() - Deribit DVOL

Pattern Primitives

  • is_fvg() - Computed from prices
  • is_hammer(), is_doji() - Candlestick
  • dtw_similarity() - DTW templates
  • pattern_warming() - Pattern detector
  • support_resistance() - Computed

Manipulation Primitives

  • is_stop_hunt() - Binance L2
  • manipulation_risk_score() - Computed
  • is_pump_dump_active() - Volume analysis
  • spoofing_detected() - Order book
  • whale_activity() - Tick classifier

🔄 Provider Fallback Chains

flowchart LR subgraph Price["Price Data"] P1["Alpaca"] --> P2["Yahoo"] P2 --> P3["Finnhub"] end subgraph Crypto["Crypto Data"] C1["Alpaca"] --> C2["Yahoo"] C2 --> C3["CoinGecko"] C3 --> C4["Binance"] end subgraph Forex["Forex Data"] F1["Polygon"] --> F2["Yahoo"] F2 --> F3["Frankfurter"] end subgraph Embed["Embeddings"] E1["Voyage"] --> E2["OpenAI"] end subgraph LLM["LLM Calls"] L1["LLMRouter"] --> L2["Anthropic Direct"] end