Pearl Social Media & LinkedIn - The Smithy Knowledge Sanctuary

Overview¶

Pearl's Social Media feature manages the full content lifecycle for LinkedIn and X (Twitter) posts — from drafting and scheduling through human approval to publishing and engagement tracking. It consists of two core services:

SocialMediaService — Post CRUD, content calendar, approval workflow, metrics recording, milestone notifications, and content intelligence analytics.
LinkedInCollectorService — Automated polling of the LinkedIn API for engagement metrics on published posts, with deduplication and milestone detection.

Both services integrate with Pearl's Slack bot for operator interaction and are backed by PostgreSQL via Pearl's memory_manager.

Pearl also supports X (Twitter) integration via an OAuth 2.0 PKCE flow and dedicated API callback endpoint, enabling monitoring of investor tweets, engagement with content, and publishing posts on X.

Pearl additionally includes a TikTok Creator Analytics collector (TikTokCollectorService) that polls the TikTok Content Posting API v2 for follower counts and video engagement metrics, recording them into Pearl's revenue analytics with deduplication.

Product Context¶

Pearl is Glass Umbrella's AI-powered Customer Lifecycle Architect — a Slack-native coworker whose north-star metric is topline revenue growth. The Social Media feature is Pearl's public-facing content engine, serving the founding team and marketing operators who need to:

Build brand presence on LinkedIn, X, and TikTok with a consistent voice (OKR 4: brand voice audit score ≥ 85 %, social engagement rate ≥ 3 %).
Attract investors by monitoring target investors' X activity and surfacing EdTech market conversations (investor intel, not vanity metrics).
Convert awareness into revenue by tying content performance back to the customer funnel — top posts, best posting days, and hashtag effectiveness all feed Pearl's LLM-powered suggest command so every piece of content is data-informed.
Protect trust through a mandatory human-approval workflow (draft → queued → approved → published) — Pearl never sends external communications without operator sign-off (Constitution P1–P5).

The target product is The Smithy, an EdTech game-based learning platform for children. Content must be accurate, transparent to parents, and free of dark patterns or hype (Constitution P1, P4).

X Collector Service¶

XCollectorService (src/services/x_collector_service.py) monitors target investors' X timelines and searches EdTech conversations, storing results as investor intel — not post-engagement metrics. It is command-driven: an operator runs /pearl social x-monitor run in Slack to trigger a collection cycle. There is no background scheduler for the X collector.

How `collect()` works¶

Pre-flight checks — Verifies the XClient is configured, the monthly spending cap has not been exceeded, and the OAuth token is valid (proactively refreshing if it expires within 24 hours).
Phase 1 — Poll investor timelines (_poll_investor_timelines) — Fetches investors (up to PEARL_X_MAX_INVESTORS_PER_POLL, default 10) from InvestorService, filters to those with a twitter_handle, resolves each handle to an X user ID (caching it on the investor record), and retrieves new tweets since the last poll via XClient.fetch_user_tweets(). Each tweet is stored as an intel record (intel_type="tweet", source="x_api") through InvestorService.add_intel(). Poll state (last tweet ID, total fetched) is persisted in the x_poll_state table so subsequent runs only fetch new activity. Polling stops early if the spending cap is reached.
Phase 2 — Search EdTech conversations (_search_edtech) — Runs up to three search queries against the X recent-search endpoint (XClient.search_recent_tweets()). Default queries target #EdTech game-based learning, educational games children STEAM, and #edtech angel investment seed; these can be overridden via PEARL_X_SEARCH_QUERIES (comma-separated). Search poll state is tracked per query in the same x_poll_state table.
Cost recording — Final spend is read from XClient.get_spending_summary() and included in the result dict.

collect() returns:

{
    "investors_polled": int,
    "tweets_stored": int,
    "searches_run": int,
    "cost_usd": float,
    "errors": [str]
}

get_poll_status() returns all rows from x_poll_state (ordered by last_polled_at descending) for operator visibility via /pearl social x-monitor status.

X Post Lifecycle¶

Publishing a post to X follows the same approval workflow as LinkedIn but uses a separate command:

Create a draft: /pearl social draft <content> --platform x
Schedule and approve the post through the standard workflow (schedule, approve).
Publish to X: /pearl social x-post <id> — calls XClient.publish_tweet(), logs the tweet URL, and updates the post status to published.

Only approved posts can be published. The tweet URL is logged before the status update so it is never lost on a post-publish failure.

X Engagement Commands¶

Command	Description
`/pearl social x-search <query>`	Search recent tweets matching a query
`/pearl social x-engage like <url_or_id>`	Like a tweet
`/pearl social x-engage retweet <url_or_id>`	Retweet
`/pearl social x-engage reply <url_or_id> <text>`	Reply to a tweet
`/pearl social x-engage draft-reply <url_or_id> <tweet text>`	Draft a value-only reply (review before sending)
`/pearl social x-reach`	Daily reach pass — search lanes + draft replies in one batch
`/pearl social x-spend`	X API cost dashboard

X Cost Model¶

The XClient tracks API costs on a pay-per-use model (reads: $0.005, writes: $0.01 per operation). A configurable monthly spending cap (X_SPENDING_CAP_USD, default $10.00) is enforced before every API call; if exceeded, the request is skipped and a warning is logged. Spend is persisted in the x_oauth_tokens table and resets monthly.

Database Tables (X-specific)¶

Table	Purpose
`x_oauth_tokens`	OAuth 2.0 token storage, spending tracking, user identity (one row per `account_label`)
`x_poll_state`	Per-target poll cursor — tracks `last_tweet_id`, `last_polled_at`, and cumulative `tweets_fetched_total` (unique on `poll_type` + `poll_key`)

Architecture¶

Services¶

Service	File	Responsibility
`SocialMediaService`	`src/services/social_media_service.py`	Post lifecycle, calendar, metrics storage, milestones, content intelligence
`LinkedInCollectorService`	`src/services/linkedin_collector_service.py`	Polls LinkedIn API, records metrics, triggers milestones
`TikTokCollectorService`	`src/services/tiktok_collector_service.py`	Polls TikTok API v2 for creator analytics (followers, video metrics), records into revenue analytics
`SocialMediaCommandHandler`	`src/handlers/social_media_command_handler.py`	Slack command routing, analytics formatting, LLM-powered suggestions
`XCollectorService`	`src/services/x_collector_service.py`	Command-driven investor timeline polling and EdTech search on X
`XOAuthRouter`	`src/api/x_oauth_router.py`	Handles X OAuth 2.0 PKCE callback, exchanges authorization codes for tokens
`XClient`	`src/integrations/x_client.py`	X API client — OAuth token management, API interactions, cost tracking

API Server¶

The ApiServer (src/services/api_server.py) hosts several sub-applications:

Mount Path	Sub-App	Purpose
`/api`	`api_subapp`	Internal API endpoints
`/oauth`	`public_subapp`	Public OAuth flows (optional)
`/webhook`	`webhook_subapp`	Incoming webhook handlers (optional)
`/x`	`x_oauth_subapp`	X OAuth callback endpoint (optional, requires `x_client`)

The /x sub-app is only mounted when pearl.x_client is configured. It provides:

GET /x/callback — Receives the OAuth 2.0 PKCE redirect from X, validates the state parameter, and exchanges the authorization code for access tokens.

Database Tables¶

The feature creates and manages five PostgreSQL tables:

social_media_posts — Stores post content, status, platform, scheduling, approval metadata, and LinkedIn URN.
social_media_calendar — Themed content calendar entries by day-of-week (e.g., "Monday: Thought Leadership").
social_media_metrics — Time-series engagement data (impressions, reactions, comments, shares, clicks) per post.
linkedin_oauth_tokens — OAuth token storage for LinkedIn API access (access token, refresh token, expiry).
social_media_milestones — Records when engagement milestones are crossed, preventing duplicate notifications.

TikTok metrics are stored in the revenue_metrics table (managed by RevenueAnalyticsService), not in social_media_metrics.

Post Workflow¶

Posts follow a linear status workflow:

draft → queued (with scheduled_for date) → approved → published → metrics polling
                                         ↘ rejected

Only draft posts can be edited.
Only draft posts can be queued.
Only queued posts can be approved or rejected.
Only approved posts can be published (setting linkedin_post_urn and published_at).

Milestone Notifications¶

When engagement metrics cross predefined thresholds, milestone notifications are sent to Slack. Thresholds are:

Metric	Thresholds
Impressions	100, 500, 1,000, 5,000, 10,000
Reactions	10, 50, 100, 500
Comments	10, 50, 100
Shares	10, 50, 100

Each milestone is recorded once per post (via a unique constraint on post_id + metric_type + threshold_value), so notifications are never duplicated.

Architecture Diagram¶

graph TD
    subgraph Slack
        OP[Operator]
    end

    OP -->|/pearl social ...| SMCH[SocialMediaCommandHandler]

    subgraph Pearl Slack Bot
        SMCH
        SMS[SocialMediaService]
        LCS[LinkedInCollectorService]
        XCS[XCollectorService]
        TCS[TikTokCollectorService]
        XC[XClient]
        LC[LinkedInClient]
        API[ApiServer :8000]
    end

    SMCH --> SMS
    SMCH --> XCS
    SMCH --> XC
    SMCH --> TCS

    LCS -->|fetch_post_metrics| LC
    LCS --> SMS
    XCS -->|fetch_user_tweets / search_recent_tweets| XC
    XCS -->|add_intel| IS[InvestorService]

    API -->|/x/callback| XOR[XOAuthRouter]
    XOR --> XC

    subgraph PostgreSQL
        SMP[(social_media_posts)]
        SMC[(social_media_calendar)]
        SMM[(social_media_metrics)]
        LOT[(linkedin_oauth_tokens)]
        SMI[(social_media_milestones)]
        XOT[(x_oauth_tokens)]
        XPS[(x_poll_state)]
    end

    SMS --> SMP
    SMS --> SMC
    SMS --> SMM
    SMS --> SMI
    LCS --> LOT
    XC --> XOT
    XCS --> XPS

    LC ---|LinkedIn API| LI[LinkedIn]
    XC ---|X API v2| XI[X / Twitter]
    TCS ---|TikTok API v2| TI[TikTok]

    subgraph Schedulers
        LSCHED[linkedin_collector_scheduler]
        TSCHED[tiktok_collector_scheduler]
    end

    LSCHED -->|interval| LCS
    TSCHED -->|interval| TCS

Note: The LinkedIn and TikTok collectors run on background schedulers (_linkedin_collector_scheduler, _tiktok_collector_scheduler in src/mvp_slack_bot.py). The X collector is command-driven only — there is no background scheduler; operators trigger collection via /pearl social x-monitor run.

Usage¶

Slack Commands¶

The feature is accessed via Slack slash commands:

/pearl social draft <content>       # Create a new draft post
/pearl social view <id>             # View post details
/pearl social approve <id>          # Approve a queued post
/pearl social calendar add <day> <theme>  # Add a content calendar entry
/pearl social suggest               # Get next post suggestion based on performance
/pearl social analytics [sub]       # Content performance analytics
/pearl social x-oauth url           # Generate X OAuth authorization URL

X (Twitter) OAuth Setup¶

X integration uses OAuth 2.0 with PKCE. The setup flow:

Run /pearl social x-oauth url in Slack to generate an authorization URL.
Open the URL in a browser and authorize Pearl on X.
X redirects to the configured X_REDIRECT_URI callback with a code and state.
The XOAuthRouter validates the state, exchanges the code for tokens, and stores them.
On success, the browser displays: "Pearl connected to X as @{username}".

Error cases for the /x/callback endpoint:

Missing code or state → HTTP 400: "Missing code or state parameter"
X client not configured → HTTP 500: "X client not configured"
Invalid or expired state → HTTP 400: "Invalid or expired OAuth state" — the user should re-run /pearl social x-oauth url to generate a fresh link.
Token exchange failure (e.g., expired authorization code, X API error) → HTTP 500: "Failed to connect to X" — the XClient logs the HTTP status, reason, and error body, and returns None rather than raising.

Prerequisite: The X_REDIRECT_URI environment variable must match the callback URL registered with the X developer application.

Content Calendar¶

Calendar entries map days of the week to content themes (e.g., "Monday: Industry Insights"). The get_today_theme() method returns the active theme for the current weekday, which can be used for content generation context.

Content Intelligence (Phase 3)¶

Phase 3 adds data-driven content intelligence on top of the existing post lifecycle: an LLM-powered suggest command and a set of analytics views.

Post Suggestions¶

The suggest command uses SocialMediaService.get_content_intelligence_summary() to gather performance data and feeds it to the configured LLM, which returns a single post suggestion with topic, style, hashtags, and a rationale tied to actual performance metrics.

/pearl social suggest

The intelligence summary includes:

Top posts — up to 5 published posts ranked by total engagement (reactions + comments + shares).
Theme performance — average engagement per content calendar theme, matched by day-of-week of publication.
Day analysis — average engagement by day of week across all published posts.
Hashtag performance — average engagement per hashtag across published posts.
Recent excerpts — the last 5 published/queued post excerpts (to avoid repetition).
Today's theme — the active calendar theme for the current weekday.

The suggestion is returned in a structured format:

**Topic:** ...
**Style:** (thought-leadership / story / tip / question / announcement)
**Hashtags:** #... #...
**Rationale:** ...

Prerequisite: An LLM provider must be configured (llm_manager). If unavailable, the command returns: "Post suggestions require an LLM provider — not configured."

Analytics¶

The analytics command provides content performance breakdowns without requiring an LLM. Sub-commands:

Command	Description
`/pearl social analytics`	Overview — top post, best theme, best day, with drill-down links
`/pearl social analytics themes`	Average impressions and engagement per calendar theme
`/pearl social analytics timing`	Average impressions and engagement by day of week
`/pearl social analytics hashtags`	Average impressions and engagement per hashtag (top 15)
`/pearl social analytics top`	Top 10 posts ranked by total engagement

Engagement is defined as the sum of reactions, comments, and shares. All analytics queries use the most recent metrics snapshot per post (via LATERAL JOIN on social_media_metrics ORDER BY recorded_at DESC LIMIT 1).

If no published posts with metrics exist, each sub-command returns a descriptive empty-state message (e.g., "No theme performance data yet — publish posts on calendar days.").

Service Methods¶

The following methods on SocialMediaService power the content intelligence features:

Method	Returns
`get_top_posts(limit=10)`	Published posts ranked by engagement (reactions + comments + shares)
`get_theme_performance()`	Average engagement per calendar theme, sorted descending
`get_posting_day_analysis()`	Average engagement by day of week, sorted descending
`get_hashtag_performance()`	Average engagement per hashtag, sorted descending
`get_content_intelligence_summary()`	Aggregated dict of all above, plus recent excerpts and today's theme

LinkedIn Metrics Collection¶

LinkedInCollectorService.collect() runs on a scheduled loop via _linkedin_collector_scheduler in mvp_slack_bot.py. It:

Proactively refreshes the LinkedIn OAuth token if nearing expiry.
Fetches all published posts with a linkedin_post_urn.
Calls linkedin_client.fetch_post_metrics(post_urn) for each.
Deduplicates — skips recording if the latest metrics are identical and were recorded today.
Records new metrics via SocialMediaService.record_metrics().
Checks for milestone crossings and posts notifications to Slack.

Collection results are returned as:

{
    "recorded": [<post_ids>],
    "skipped": [<post_ids>],
    "errors": [<post_ids>],
    "milestones": [<milestone_dicts>]
}

TikTok Creator Analytics Collection¶

TikTokCollectorService.collect() runs on a scheduled loop via _tiktok_collector_scheduler in mvp_slack_bot.py. It polls the TikTok Content Posting API v2 for creator-level analytics and records them into RevenueAnalyticsService (not the social media metrics tables). The TikTok OAuth access token is auto-refreshed every 3 hours by Taryn.

The collector gathers two tiers of metrics:

Tier 1 — User info (GET /v2/user/info/): follower count, total likes, video count.
Tier 2 — Video averages (POST /v2/video/list/): average views, likes, comments, and shares across recent videos.

Metrics stored in the revenue_metrics table:

Metric Name	Description
`tiktok_followers`	Follower Count
`tiktok_total_likes`	Total Likes
`tiktok_video_count`	Video Count
`tiktok_avg_views`	Avg Views per Video
`tiktok_avg_likes`	Avg Likes per Video
`tiktok_avg_comments`	Avg Comments per Video
`tiktok_avg_shares`	Avg Shares per Video

Deduplication: Each metric is skipped if the latest recorded value is identical and was recorded on the same day.

Scheduler timing: The TikTok collector starts with a 90-minute stagger (5,430 seconds) after the Matomo collector, then repeats at the configured interval (default 24 hours).

Collection results follow the same pattern as other collectors:

{
    "recorded": [<metric_names>],
    "skipped": [<metric_names>],
    "errors": [<metric_names>]
}

Prerequisites: - TIKTOK_ACCESS_TOKEN — A valid OAuth 2.0 access token from TikTok Login Kit. - PEARL_TIKTOK_COLLECTOR_ENABLED must be set to "true".

Configuration¶

LinkedIn¶

Variable	Purpose	Default
`PEARL_STATUS_CHANNEL`	Slack channel for milestone notifications	`C05SL0R2THT`
`PEARL_LINKEDIN_COLLECTOR_ENABLED`	Enable/disable the LinkedIn collector scheduler	`false`
`PEARL_LINKEDIN_COLLECTOR_INTERVAL`	Collection interval in minutes	`1440` (24 hours)

TikTok¶

Variable	Purpose	Default
`PEARL_TIKTOK_COLLECTOR_ENABLED`	Enable/disable the TikTok collector scheduler	`false`
`PEARL_TIKTOK_COLLECTOR_INTERVAL`	Collection interval in minutes	`1440` (24 hours)
`TIKTOK_ACCESS_TOKEN`	OAuth 2.0 access token from TikTok Login Kit (auto-refreshed by Taryn)	—

X (Twitter)¶

Variable	Purpose	Default
`X_REDIRECT_URI`	OAuth 2.0 PKCE callback URL for X	`https://<pearl-host>/x/callback`
`X_SPENDING_CAP_USD`	Spending cap for X API usage (USD)	`10.00`
`PEARL_X_COLLECTOR_ENABLED`	Enable/disable the X collector scheduler	`false`
`PEARL_X_COLLECTOR_INTERVAL`	X collection interval in minutes	`720` (12 hours)
`PEARL_X_MAX_INVESTORS_PER_POLL`	Maximum number of investors to poll per collection cycle	`10`

LinkedIn OAuth¶

LinkedIn API access requires OAuth tokens stored in the linkedin_oauth_tokens table. The linkedin_client handles token refresh automatically when needs_refresh() returns True.

X OAuth¶

X API access uses OAuth 2.0 with PKCE. Tokens are obtained via the /x/callback endpoint (see X (Twitter) OAuth Setup). The XClient manages token storage, state validation, and code exchange.

Error Handling¶

Per-post errors during LinkedIn collection are isolated — one post failing does not block collection for others.
Token refresh failures are logged as warnings but do not halt the collection loop.
Milestone check failures do not affect metrics recording — posts are still recorded successfully.
Deduplication prevents redundant writes when metrics haven't changed within the same day.
Slack API errors in milestone/alert posting are caught and logged without propagating.
LLM failures in suggest are caught and surfaced to the user (e.g., "Failed to generate suggestion: …") without affecting other commands.
Missing LLM provider — the suggest command returns an informational message rather than erroring.
X token exchange failures — the XClient logs the HTTP status, reason, and full error body, then returns None instead of raising. The XOAuthRouter translates this into an HTML error page advising the user to retry.
X OAuth state validation failures — expired or invalid state parameters return an HTML error page directing the user to generate a new authorization URL via Slack.
TikTok auth failures — if the access token is invalid or expired (HTTP 401/403, or API-level access_token_invalid / token_expired / scope_not_authorized), a TikTokAuthError is raised, auth is marked invalid, and all metrics are reported as errors. Subsequent collection cycles are skipped until auth is revalidated.
TikTok per-tier isolation — Tier 2 (video averages) failures are non-fatal; Tier 1 (user info) metrics are still recorded successfully.
TikTok scheduler exceptions — unhandled exceptions in the collector are caught and logged without crashing the background loop.

MatomoCollectorService — follows the same collector pattern for web analytics.
Pearl Slack bot (src/mvp_slack_bot.py) — hosts the scheduler loops and Slack command handlers.

Feature Repositories¶

pearl → project_id: the-smithy1/agents/pearl
chisel → project_id: the-smithy1/agents/chisel

Code Paths to Explore¶

tests/test_social_media_service.py in pearl
tests/test_scheduler_functions.py in pearl
tests/test_linkedin_collector_service.py in pearl
src/services/social_media_service.py in pearl
src/services/linkedin_collector_service.py in pearl
src/services/tiktok_collector_service.py in pearl
src/mvp_slack_bot.py in chisel
src/api/x_oauth_router.py in pearl
src/integrations/x_client.py in pearl
src/services/api_server.py in pearl