# Social Follows & Telegram Integration

**Status:** Shipped 2026-05-06
**Migration:** 174 (credentials column)
**Related:** `docs/24-api-routes.md` §4c-1a

---

## Overview

Social Follows is the platform's social intelligence layer. It monitors 7 social platforms for market-relevant content, extracts entities (tickers, sectors, themes), scores sentiment, and feeds the data into the divergence engine for human-vs-automation classification.

**Key capabilities:**
- Monitor public and private social sources
- User-provided credentials for private Telegram/Discord groups
- LLM-powered entity extraction with regex fallback
- Real-time sentiment scoring and aggregation
- Per-user personal follows (private) + admin-configured global follows (platform-wide)

---

## Supported Platforms

| Platform | Follow Types | Auth Required | Private Access |
|----------|--------------|---------------|----------------|
| Reddit | `subreddit`, `user` | None (public API) | N/A |
| Twitter/X | `user` | `TWITTER_BEARER_TOKEN` or syndication fallback | N/A |
| Telegram | `channel` | None for public | User-provided bot token |
| Discord | `channel` | `DISCORD_BOT_TOKEN` | User-provided bot token |
| Bluesky | `user` | None (public API) | N/A |
| Truth Social | `user` | None (HTML scrape) | N/A |
| RSS | `feed` | None | N/A |

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                         SOCIAL FOLLOWS PIPELINE                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐           │
│  │   Adapters   │    │   Service    │    │  Scheduler   │           │
│  │              │    │              │    │              │           │
│  │ reddit       │◄───│ pollFollow() │◄───│ pollDueFollows│          │
│  │ twitter      │    │              │    │ (15 min cron) │          │
│  │ telegram ────┼────│ credentials  │    └──────────────┘           │
│  │ discord      │    │ decrypt      │                                │
│  │ bluesky      │    │              │                                │
│  │ truth_social │    └──────┬───────┘                                │
│  │ rss          │           │                                        │
│  └──────────────┘           ▼                                        │
│                    ┌──────────────┐                                  │
│                    │   Entity     │                                  │
│                    │  Extractor   │                                  │
│                    │              │                                  │
│                    │ LLM (Haiku)  │                                  │
│                    │ + regex      │                                  │
│                    └──────┬───────┘                                  │
│                           │                                          │
│                           ▼                                          │
│                    ┌──────────────┐    ┌──────────────┐              │
│                    │   Sentiment  │    │  Sentiment   │              │
│                    │   Scoring    │───▶│   Rollup     │              │
│                    │              │    │  (hourly)    │              │
│                    │ lexical +    │    └──────┬───────┘              │
│                    │ LLM blend    │           │                      │
│                    └──────────────┘           ▼                      │
│                                       ┌──────────────┐               │
│                                       │  Divergence  │               │
│                                       │   Engine     │               │
│                                       │              │               │
│                                       │ Human vs Bot │               │
│                                       │ classification│              │
│                                       └──────────────┘               │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Database Schema

### `predict.social_follows`

| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key |
| `owner_type` | VARCHAR(10) | `global` (admin) or `user` (personal) |
| `user_id` | UUID | Owner for user-scoped follows |
| `platform` | VARCHAR(20) | reddit, twitter, telegram, discord, bluesky, truth_social, rss |
| `follow_type` | VARCHAR(20) | subreddit, user, keyword, channel, feed |
| `handle` | VARCHAR(120) | Channel name, username, or feed URL |
| `display_name` | VARCHAR(200) | Optional friendly name |
| `category` | VARCHAR(50) | crypto, stocks, forex, ai, tech, politics, economy, news |
| `enabled` | BOOLEAN | Whether polling is active |
| `poll_interval_minutes` | INTEGER | How often to poll (default 60) |
| `credentials_encrypted` | TEXT | AES-256-GCM encrypted JSON (Migration 174) |
| `last_checked_at` | TIMESTAMPTZ | Last successful poll |
| `last_error` | TEXT | Last error message |
| `post_count` | INTEGER | Total posts ingested |

### `predict.social_posts`

| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key |
| `follow_id` | UUID | FK to social_follows |
| `platform` | VARCHAR(20) | Source platform |
| `external_id` | VARCHAR(200) | Platform-specific post ID |
| `author` | VARCHAR(120) | Post author |
| `title` | TEXT | Post title (if any) |
| `content` | TEXT | Post body |
| `url` | TEXT | Link to original post |
| `posted_at` | TIMESTAMPTZ | When posted |
| `upvotes` | INTEGER | Engagement metric |
| `comments_count` | INTEGER | Engagement metric |
| `sentiment_score` | NUMERIC(4,3) | -1.0 to +1.0 |
| `sentiment_label` | VARCHAR(20) | positive, neutral, negative |
| `keywords` | JSONB | Extracted tickers, sectors, themes |
| `category` | VARCHAR(50) | Inherited from follow |

### `predict.sentiment_hourly`

Aggregated sentiment by topic (ticker or category) and platform, recomputed every 15 minutes over a 48-hour window.

---

## Telegram Integration

### Public Channels (No Token Required)

For public Telegram channels, the adapter scrapes the web preview at `t.me/s/<channel>`:

```typescript
// No credentials needed
await createFollow({
  platform: 'telegram',
  followType: 'channel',
  handle: 'duaborsa',  // @duaborsa
});
```

### Private Groups (Bot Token Required)

For private groups/channels, users must provide their own Telegram bot token:

#### Step 1: Create a Bot

1. Open Telegram and search for [@BotFather](https://t.me/BotFather)
2. Send `/newbot` and follow the prompts
3. Copy the bot token (format: `123456789:ABCdefGHIjklMNOpqrsTUVwxyz`)

#### Step 2: Add Bot to Group

1. Add your bot to the private group/channel
2. Make the bot an **administrator** with at least "Read Messages" permission
3. For groups: the bot needs to be in the group to receive messages
4. For channels: the bot needs to be an admin to access channel posts

#### Step 3: Provide Token to Agencio

**Option A: When creating the follow**

```typescript
POST /api/predict/v1/user/social-follows
{
  "platform": "telegram",
  "followType": "channel",
  "handle": "-1001234567890",  // Channel ID (numeric) or @username
  "botToken": "123456789:ABCdefGHIjklMNOpqrsTUVwxyz"
}
```

**Option B: Update existing follow**

```typescript
PATCH /api/predict/v1/user/social-follows/:id/credentials
{
  "botToken": "123456789:ABCdefGHIjklMNOpqrsTUVwxyz"
}
```

### Token Security

- Tokens are encrypted with **AES-256-GCM** using `CREDENTIALS_ENCRYPTION_KEY`
- Same encryption key used for broker credentials
- Tokens never returned in API responses (only `hasCredentials: true/false`)
- Decrypted only at poll time, held in memory briefly

### Getting the Channel ID

For private channels/groups, you'll need the numeric chat ID:

1. Add [@userinfobot](https://t.me/userinfobot) to your group temporarily
2. It will reply with the chat ID (negative number like `-1001234567890`)
3. Remove the bot after getting the ID

Or use the Bot API directly:
```bash
curl "https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates"
# Look for "chat": {"id": -1001234567890, ...}
```

---

## Sentiment Analysis Pipeline

### 1. Lexicon-Based Scoring

Fast, deterministic scoring using keyword lists:

**Bullish keywords:** bullish, buy, long, rally, surge, moon, pump, breakout, ath, upgrade, beat, strong, growth, record

**Bearish keywords:** bearish, sell, short, crash, dump, dip, plunge, downgrade, miss, weak, recession, loss, bankruptcy, fraud

Score = (positive_count - negative_count) / total_count

### 2. LLM Entity Extraction

Claude Haiku extracts structured entities:

```typescript
{
  tickers: ["AAPL", "NVDA"],      // Mentioned or implied
  sectors: ["technology", "ai"],  // Market sectors
  themes: ["earnings", "tariffs"], // Macro themes
  stance: "bullish",              // Overall sentiment
  confidence: 0.85                // Extraction confidence
}
```

### 3. Blended Score

When LLM confidence > 0.5:
```
finalScore = 0.7 * lexicalScore + 0.3 * llmStanceScore
```

### 4. Noise Filtering

The pipeline filters noise through multiple layers:

1. **Entity extraction:** Only extracts market-relevant content
2. **Stopwords:** Filters common English words that look like tickers (A, I, US, AI, etc.)
3. **Universe validation:** Rollup only aggregates tickers from `price_collection_universe`
4. **Engagement weighting:** Posts with more upvotes/comments have more influence
5. **Category bucketing:** Generic posts go to category buckets, not ticker-specific

---

## API Reference

### Create Follow

```http
POST /api/predict/v1/user/social-follows
Content-Type: application/json
Authorization: Bearer <token>

{
  "platform": "telegram",
  "followType": "channel",
  "handle": "@cryptosignals",
  "category": "crypto",
  "pollIntervalMinutes": 30,
  "botToken": "optional-for-private-groups"
}
```

**Response:**
```json
{
  "follow": {
    "id": "uuid",
    "platform": "telegram",
    "followType": "channel",
    "handle": "@cryptosignals",
    "enabled": true,
    "hasCredentials": true,
    "postCount": 0
  }
}
```

### Update Credentials

```http
PATCH /api/predict/v1/user/social-follows/:id/credentials
Content-Type: application/json
Authorization: Bearer <token>

{
  "botToken": "new-token-here"
}
```

To remove credentials:
```json
{
  "botToken": null
}
```

### List Posts

```http
GET /api/predict/v1/user/social-follows/:id/posts
Authorization: Bearer <token>
```

**Response:**
```json
{
  "posts": [
    {
      "id": "uuid",
      "platform": "telegram",
      "author": "cryptosignals",
      "content": "BTC looking bullish, breakout imminent",
      "postedAt": "2026-05-06T12:00:00Z",
      "sentimentScore": 0.72,
      "sentimentLabel": "positive",
      "keywords": ["BTC"]
    }
  ]
}
```

---

## UI Components

### Settings Page

`/settings/social-follows` — User's personal follows

- Platform selector with availability indicators
- Bot token input for Telegram/Discord
- Inline instructions for BotFather setup
- Key icon shows credential status (green = configured, amber = not set)
- Manual poll button for testing

### Admin Page

`/admin/flow` — Global follows that drive platform-wide sentiment

---

## File Locations

| Purpose | Path |
|---------|------|
| Types | `packages/be/src/social/types.ts` |
| Service | `packages/be/src/social/service.ts` |
| Telegram adapter | `packages/be/src/social/telegram-adapter.ts` |
| Entity extractor | `packages/be/src/social/entity-extractor.ts` |
| API handlers | `packages/be/src/social/api/handlers.ts` |
| Sentiment rollup | `packages/be/src/scheduler/sentiment-rollup.ts` |
| Divergence engine | `packages/be/src/insights/divergence-engine.ts` |
| UI component | `apps/web/src/components/social/SocialFollowsPanel.tsx` |
| Settings page | `apps/web/src/app/settings/social-follows/page.tsx` |
| User API routes | `apps/web/src/app/api/predict/v1/user/social-follows/` |
| Admin API routes | `apps/web/src/app/api/predict/v1/admin/social-follows/` |
| Migration | `db/migrations/174_social_follow_credentials.sql` |

---

## Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `CREDENTIALS_ENCRYPTION_KEY` | Yes (prod) | 32-byte key for AES-256-GCM encryption |
| `TWITTER_BEARER_TOKEN` | No | Twitter API v2 bearer token |
| `DISCORD_BOT_TOKEN` | No | Discord bot token (platform default) |

In development without `CREDENTIALS_ENCRYPTION_KEY`, a warning is logged and a dev fallback key is used.

---

## Troubleshooting

### "Bot cannot access this channel"

- Ensure the bot is an **admin** of the channel/group
- For groups, the bot must be a member
- Check the channel ID is correct (negative number for groups)

### Posts not appearing

1. Check `lastError` on the follow record
2. Try manual poll: `POST /api/predict/v1/user/social-follows/:id/poll`
3. Verify bot has "Read Messages" permission

### Sentiment all neutral

- The lexical scorer is conservative; many posts score neutral
- Enable LLM extraction (set `ANTHROPIC_API_KEY` or `LLM_ROUTER_URL`)
- Check if posts contain market-relevant keywords

### Rate limited

- Telegram: 30 requests/second per bot
- Reduce `pollIntervalMinutes` or stagger polls
- Check `lastError` for specific rate limit info

---

## Future Enhancements

- **WhatsApp:** Not planned due to ToS restrictions. Users can export chat history and upload as a file for batch analysis.
- **Webhook mode:** Receive Telegram updates via webhook instead of polling
- **Relevance scoring:** LLM-based "is this market relevant?" pre-filter
- **Multi-language:** Sentiment analysis for non-English content