Social scan skill

social-scan

doctor once per machine, then one scan per account: one report.

Script: scripts/social_scan.py in this skill's folder (written SCAN below; use its full path). Python 3.8+, standard library only. doctor says what else is installed and what each missing piece turns off: yt-dlp for YouTube and TikTok; ffmpeg, whisper-cli (whisper.cpp) and a model file (default ~/.local/share/whisper-cpp/ggml-base.en.bin, override with WHISPER_MODEL) for hooks; an Apify account for Instagram. Outlier tiers, hooks, comment mining and the brief are adapted from ScrapeCreators' social-media-research-skills (MIT); method, attribution and gaps in references/research-methods.md.

Rules

• Logged-out data only. Never pass --cookies or --cookies-from-browser, never log in to anyone's account to scrape, never use Instaloader with a login. (Platform terms and account-ban risk.)
• The Apify token comes only from the APIFY_TOKEN environment variable the user sets in their own terminal. Never read it out of a keychain or config file yourself, never print it, never put it in a URL.
• Cap every Apify run (--limit, plus maxItems for pay-per-result actors or maxTotalChargeUsd for pay-per-event ones; the script sends both, at $0.30 per Apify call). An Instagram scan costs about $0.11 per 40 reels and can't pass $0.60 (two calls); YouTube, TikTok, Snapchat and podcasts are free. The optional add-ons below bill separately.
• One folder per account and run: <project>/social-scan/<account>/<YYYY-MM-DD>/. scan reports everything already in its --out folder, and compares it with the newest earlier dated folder beside it. Keep the raw files; they're the receipts.

Steps

1. Check the machine (once): python3 SCAN doctor. Exit 0 means YouTube and TikTok can run; everything marked optional only adds to the report. For a MISSING or BROKEN tool, show the user the install line it prints and run it only on their OK. The whisper model (148 MB) downloads only with doctor --get-model, and only when the user says yes. Never run the APIFY_TOKEN line, never ask for the token, and never run --instagram yourself even if doctor shows the token as set: the direct Instagram path is paid, so the user runs it in their own terminal (step 3).
2. Get handles. YouTube @handle, TikTok handle, Instagram username, Snapchat username, and the podcast's name or Apple Podcasts ID (the digits after id in its podcasts.apple.com link; the link works too). The account's website footer usually links them. TikTok followers come from the public profile page automatically. A mistyped TikTok or Snapchat handle is named in the end-of-run Notes ("check the handle") and that platform is skipped; check the Notes for "failed:" lines before reading the report, because summary.md doesn't list a skipped platform.
3. Instagram, if it's in scope (reels tab, newest N in order; about $0.11 for 40). Do it before step 4. Pick whichever path works:
   • Direct (preferred, scriptable). Ask the user to run the whole scan in their own terminal, so their token never passes through the chat:

     APIFY_TOKEN=YOUR_TOKEN python3 SCAN scan --out <DIR> --instagram username --youtube @Handle --tiktok handle --snapchat handle --name "Account Name"
     

     Ask them to paste back the Notes it ends with, tell them what was skipped, then go to step 5.
   • Apify MCP. Call apify/instagram-reel-scraper with {"username":["<user>"],"resultsLimit":40} (the direct path pulls 50) and callOptions.maxTotalChargeUsd: 0.30 (it bills per event, so maxItems doesn't cap it), and apify/instagram-profile-scraper with {"usernames":["<user>"]}. Use waitSecs: 0, then poll get-actor-run. Save each get-dataset-items result as a JSON array in <DIR>/raw/, then:

     python3 SCAN ingest-ig --out <DIR> --reels <DIR>/raw/instagram_reels.json --profile <DIR>/raw/instagram_profile.json
     

     Step 4 picks the ingested posts up from <DIR> (with --days, it trims them to the window too; ingest-ig --days N does the same on its own). If an MCP call stalls, stop retrying and switch to the direct path. A large get-dataset-items result (about 430 KB for 40 reels) may come back as a saved tool-result file: extract items from that file with Python into <DIR>/raw/instagram_reels.json. Never retype the JSON by hand; the signed video links must stay byte-exact. The Apify dataset API refuses token-free reads.
   • Fallback. The logged-out Instagram page shows the follower count only. Record it with meta --platform instagram.
   • Why the reels tab, not the posts feed. Logged out, the posts feed (apify/instagram-scraper with resultsType: posts, and apify/instagram-post-scraper) returns the grid's first 12 (pinned plus newest), then a scattered set of older posts. On one test account it skipped 29 live posts across ten weeks and inflated the reel median about fifteenfold. apify/instagram-reel-scraper returned the newest reels with nothing missing. Photos and carousels aren't sampled. ingest-ig --posts still reads old posts-feed files and marks them Unreliable sample. If a report says Check the profile, a gap of 7+ days broke a regular cadence: confirm on the profile before quoting that group's median. On each new account, check once that the newest 9 non-pinned reels match the logged-out grid (the /reel/ links on the profile page). If the grid is mostly photos or carousels, say in the brief that the reels median doesn't represent the account.
4. Scan (about a minute; YouTube, TikTok, Snapchat and podcasts are free):

   python3 SCAN scan --out <DIR> --youtube @Handle --tiktok handle --snapchat handle --podcast "Show name" --name "Account Name"
   

   Add --days 30 to collect the last 30 days instead of a post count (up to 300 per tab; YouTube enriches every post in the window to date it; Instagram and Snapchat are trimmed to it): use it whenever accounts will be compared. It collects (--limit 60 posts per tab; --yt-enrich 45 fetches likes, comments, exact dates and posting times for the newest 45 uploads, since Shorts have no dates without it, plus the top 10 of each kind by views so the top-posts tables carry dates), then runs hooks, comments and report on everything in <DIR>, Instagram posts from step 3 included. Run it straight after an Instagram ingest: the video links expire within hours. A step whose tool or token is missing is skipped and the report is still written; the Notes at the end say what was skipped or failed, and when posts or a follower count came from an earlier run. Tell the user both. After a step-3 ingest, the note that the ingested Instagram posts go into the report is expected; it isn't an earlier run. The last line on stdout is the path to summary.md. --tiktok-followers N overrides the count read off the profile page. --html also writes report.html, a self-contained one-page version for sending to a client (charts, the scorecard, top posts, format bars, hooks and each group's caveats from summary.md, plus a short read-before-quoting list; the bottom-5 tables and the full caveat list stay in summary.md; about 10-120 KB, each embedded frame about 25 KB more); add --own-account only when the account is the client's own, which embeds its hook frames (a competitor's footage stays in hooks/). --tz America/New_York (or an offset like UTC-05:00) sets the zone for the weekday and posting-time tables; the default is this computer's zone, so pass the audience's zone when it differs.
   • Hooks: top 3 and bottom 2 posts per group: the caption opening, the first 10 seconds of speech (whisper-cli, approximate; Snapchat's own transcript for Spotlights) and the frame at 0.5 s (a TikTok carousel's first slide), into hooks/ and hooks.json. Sung lyrics (♪) never count as a spoken hook. Without whisper they keep the caption and frame (Spotlights keep Snap's transcript); without ffmpeg (or, for YouTube and TikTok, yt-dlp), the caption only, except that Spotlights keep Snap's transcript and photo posts keep their image. An earlier row for the same post is kept when this run can't do better. The report leaves out columns nothing filled, and leaves out a hooks.json built from an earlier collect.
   • Podcast (--podcast, also on its own): Apple's US chart rank in each of the show's categories and overall (top 200), Spotify's US rank and move in Top Podcasts, Trending (top 200) and the show's category (top 50), the US Apple rating and ratings count with Apple's own split of those ratings by stars, Apple reviews into reviews_apple.json (rating, title, text, date; no names), and episodes per week and median length over the 8 weeks before the newest episode, from the show's RSS feed (a feed covering less than 8 weeks over the weeks it covers, pinned old trailers aside; under two weeks of episodes, no rate). A name without an exact match takes Apple's first search result, and a name several shows share takes the first of them; the Notes and the report give the Apple IDs. Confirm it's the right show (the report names the publisher), and pass the ID to pin it. Writes podcast.json.
   • Comments: YouTube top comments via yt-dlp; Instagram's newest ~15 per post come with the reels scrape; Snapchat's top ~25 per Spotlight come with the profile page; TikTok's (up to 100 each on the top 3 per group) only with --tiktok-comments, through TikTok's unofficial web comment list, which can stop answering (the Notes say so). Writes comments.json, handles dropped and mentions masked, the creator's own comments flagged (by_creator) on YouTube, Snapchat and TikTok, and for TikTok whether the creator liked a comment (creator_liked). TikTok gives top-level comments only: reply threads aren't read (replies is their count), so a missing creator reply there isn't evidence the creator doesn't reply. A failure after the first page keeps what was read and says where it stopped; a 429 stops TikTok requests for the rest of the run.
   • Report: summary.md (accounts; the Podcast block; scorecard with outlier tiers and confidence; per group the top posts with lift, tier and engagement lift, the bottom 5, and for TikTok the median share and save rates; What formats work: per video group, length, weekday and 3-hour posting-block tables with posts, median views, median lift and 2x+ share, thin rows marked, the strongest row named and a length check; for Snapchat, Spotlight checks: the 30 s earn floor, length buckets, upload bursts, descriptions and hashtags, Story counts, Snap's own read of the top clips; the Hooks section), summary.json (with per-post lift) and posts.csv; with --html, report.html too.
5. Write the brief. Read the report and its caveats, look at the hook frames, then write brief.md beside it with the template and the why-it-worked, comment-mining and hook-category methods in references/research-methods.md. Check any spoken hook against the video before it goes in client copy. To compare accounts, run each into its own folder and follow the competitor-teardown section there.

Single steps, for re-running one part: collect, ingest-ig, meta, hooks [--top 3 --bottom 2 --seconds 20], comments [--top 3 --max 100] [--tiktok-comments], report [--name N] [--since YYYY-MM-DD] [--tz ZONE] [--html [--own-account]] [--prev DIR | --no-prev], diff OLD NEW (prints the Since last scan section for any two scan folders), compare DIR DIR ... [--out FILE] (one table across accounts). scan --no-hooks or --no-comments leaves a step out; scan --out <DIR> with no handles re-runs hooks, comments and report on what's there (TikTok comments read earlier are kept; add --tiktok-comments to read them again). For a date window, run scan, then report --out <DIR> --name "Account Name" --since YYYY-MM-DD (add --html to rebuild report.html for the window, with --own-account again if it's the client's own account) (hook rows outside the window drop out; comments.json still covers the whole scan).

Optional add-ons

• More comments than comments pulls: apify/instagram-comment-scraper (the free plan returns only about 15 newest per post). More TikTok comments than --tiktok-comments pulls need an Apify actor (untested).
• Ad check: Meta Ad Library via apify/facebook-ads-scraper. Search by Page ID, not name; in testing a name search found 1 of 3 real ads. Only active ads and EU-delivered ads stay visible, so "none found" is weak evidence. Google Ads Transparency Center for YouTube ads.
• Guest and partner reach: apify/instagram-profile-scraper with a list of usernames gives follower counts and verified badges in one run.

Reading the numbers

• Views are logged-out plays. Instagram plays include replays. None are unique viewers.
• Followers per 1K views = today's followers ÷ (views on that platform's posts in the window ÷ 1,000). It's a proxy for how well reach turns into follows. Compare platforms and runs over time; don't treat it as follows-from-posts.
• Posts under 3 days old are still climbing; the report leaves them out of the bottom 5, the baseline and the tier counts.
• Lift = a post's views ÷ its own group's median (reels with reels, Shorts with Shorts). Tiers: huge 5x+, strong 2-5x, mild 1.5-2x. Confidence: low under 10 posts, medium 10-19, high 20+. Never compare lift or raw views across accounts without the teardown method.
• Pinned posts sit in the Instagram sample whatever their age (the reels tab has its own pins, separate from the grid's). The report lists them but keeps them out of the dates, cadence, median, tier counts and followers per 1K.
• Un-enriched YouTube uploads (past the newest --yt-enrich) only carry a "3 months ago"-style date, so their dates bunch onto a few days and can be months late. The report marks them (approx) and leaves them out of per week and the gap check; date_approx in posts.csv and summary.json says which. Raise --yt-enrich to date more. The top 10 of each kind by views are looked up too, so the tables show their exact dates, but view counts, cadence, rates, --since and the format tables read them as listed (top_lookup in posts.csv; --since leaves the looked-up Shorts out, since their listing has no date).
• What formats work slices one account's own video posts (TikTok videos, YouTube Shorts and long-form, Instagram reels; not photo carousels, whose length is the song's, and not Snapchat, whose hidden counts would tilt every row). Settled, unpinned posts only; a group needs 10 to be sliced, and so does each kind of table. Rows under 5 posts, and weekday and time rows from fewer than 3 dates, are marked thin; a table of only thin rows, or of one row, becomes one line and is never ranked, and thin rows of 3+ posts at 1.5x+ (from any table of 2+ rows) are listed as Thin leads, anecdotes, each with the length note when the leading length explains it. Weekday and time use exact posting times only (TikTok, Instagram and Snapchat listings; YouTube only for uploads looked up in the newest run), in --tz or this computer's zone. A table that covers only some of the group (a YouTube Short's length and every YouTube posting time exist only for uploads looked up in the newest run; long-form lengths come from the listing) measures lift against those posts' own median views, and the lead line says so. Strongest is the highest median lift among rows that aren't thin, named only when it reaches 1.5x and random splits of the same posts give some row that isn't thin a lead as unusual for its size under 5% of the time (each row is judged against shuffles of its own size, so a big row's clear lead isn't drowned by noisy 5-post rows; for weekday and time, whole dates trade places only with dates that have as many posts, so every row keeps its dates and posts); a lead that random splits can't move (no other row has dates of its size) reads "can't be tested against chance"; a lead that fails the second test reads Not a finding yet, with that share. The bar is per table, so a group with three tables names a chance lead about one time in eight. Check the length fires when the leading length is over-represented in the strongest day or time (against that table's own posts with a length) and the row's median lift drops by more than a quarter without those posts, or falls at all when fewer than 5 are left; no line means no length leads, the leading length isn't over-represented in that row, or the row's lift held within a quarter without those posts; it never means the row still beats every other row. These are correlations: use a strong row as a content test, quote its n, and never set a schedule from thin rows. top_lookup in posts.csv and summary.json marks the top-post lookups; drop them before cutting either yourself.
• TikTok photo carousels are their own group (tiktok photos): the listing reports the song's length as their duration, so they'd skew any length or median read if mixed with videos.
• Snapchat lists only the newest Spotlights logged out (3 to 26 on accounts tested), with no paging, so a creator who posts daily loses yesterday's from view fast; scan often, or keep a capture running. Snapchat hides view counts on its weakest Spotlights ("views hidden"): the report ranks them below every visible count, and with half or more hidden it gives the median as "under" the lowest visible count, leaves lift and tiers out and ranks by likes. Snap's title and keywords come from its SEO pass; the content tags come from a separate classifier and often disagree. No followers per 1K for Snapchat. Image Spotlights can't earn and sit outside the length checks. Snapchat pays only on Spotlights of 30 s or more (Snap help 14669003687444; re-check after 2026-12-31).
• Podcast: downloads and listens aren't public, so the block shows standing, not audience size. Charts are US and move daily; "not in the top N" is weak evidence. Spotify's category charts list only 50, and shows are matched to Spotify by name (ignoring a leading "The"), so a show listed there under another name reads as not charting. Apple's reviewCount is the ratings count; the star split comes from the show page's own data and is used only when it adds up to that count. Ratings and reviews are the US store's: Apple keeps them per country, so a show based elsewhere can have far more at home. A show nobody has rated yet reads "no ratings yet", not a failed read. Most ratings come without a written review, so the review sample isn't the rating. Apple's review feed serves some pages empty, sometimes all of them; the script reads past a stray empty page and the report names the missing pages (when page 1, the newest 50, is missing, the show page's reviews are added), and when none answer it uses the handful Apple shows on the show page and says so. A publisher's Apple ID is refused with its shows listed: pass the show's own ID. When the public RSS feed carries far fewer episodes than Apple lists (a trimmed or paywalled feed), the report flags the cadence; feed items dated in the future stay out of it. A report rendered after the pull day says how old its podcast numbers are.
• --days N makes every account cover the same dates: the N calendar days ending on the collect day, with per week counted over the window itself. YouTube posts it can't date (its per-video lookups refused) are left out and counted in the Short window line. A platform that can't reach back that far (Snapchat lists only its newest Spotlights, Instagram's paid scrape stops at --limit, any tab past 300 posts) is flagged "Short window" in the report; its numbers cover less. window.json and summary.json record what each platform covered.
• compare leaves out an account's median views where its window is more than 1.5 times off the one the others share or no two accounts share one, and shows followers per 1K (which scales with the window's length) only for accounts scanned with the same --days that reached back the whole window; outlier rate (the share of settled posts at 2x+ their own median, n/a under 5) and cadence still compare across accounts of any size. Top-3 share depends on how many posts the window holds, so compare it only between rows with similar post counts. Raw views never compare. Short-window and all-fresh rows are marked, and scans collected more than two days apart are flagged.
• Since last scan rebuilds the earlier folder's numbers from its own posts files as of the day it was collected (collected.json records that; a later re-render of that folder doesn't change it), and every report is measured as of the day its data was collected. A group whose date window is more than 1.5 times longer in one scan is flagged: part of its change is the window, so quote it with that caveat or rescan with the same --days as last time (by post count, a --limit that covers a similar range; for Snapchat, which lists only its newest Spotlights, compare scans taken at a similar interval). A window that's unknown in one scan (Shorts past --yt-enrich have no date) is compared on its dated posts and says so. Platforms collected in only one of the two scans are named under "Not compared" and left out of the post counts. Posts are split into posted since the last scan, older posts newly in the sample, undated (Shorts past --yt-enrich), in both, and only in the earlier scan (aged out, or deleted: check before saying deleted). Gains on rounded counts (TikTok shows views in 100s from 10K and in 100Ks from 1M; YouTube rounds counts outside --yt-enrich) are ranked by what they are at least, and marked ~ when one rounding step is a tenth of the gain or more; YouTube and Snapchat round follower counts too. A post counts as a new outlier only when its own views crossed 2x the earlier median and it's 2x+ in this report too; a group whose earlier median wasn't measurable gets no outlier comparison, with the reason. A --since report leaves the section out (with a note when --prev named a folder), and an earlier folder that is damaged or not actually earlier leaves it out with a note.
• Snapchat history (posts_snapchat_history.json): every Snapchat collect (scan or collect) merges the Spotlights it saw, with first and last seen and the counts at each pull, carried forward from the newest earlier dated folder that has one. Scan often to keep the Spotlights that scroll off the profile.
• A following count far above the follower count suggests follow-for-follow, which inflates followers.
• Top 3 share = the three biggest posts' share of all views. A high share means the account lives on a few hits, which is a finding in itself.

Status

• Live-tested (1.8.0): --html on a creator's TikTok and Duolingo's YouTube and TikTok, light and dark, phone width, tap tooltips.
• Live-tested (1.7.0): the format tables, posting times and top-post lookups on a creator's TikTok and on Duolingo's YouTube and TikTok.
• Live-tested (1.6.0): --tiktok-comments on a creator's six top TikTok posts (all six loaded, 395 comments in 16 s).
• Live-tested (1.5.0): --days 30 on a creator's TikTok and Snapchat and on Duolingo's YouTube and TikTok; compare across them and a scan by post count.
• Live-tested (1.4.0): Since last scan and the Snapchat history on a creator's Snapchat plus podcast, and on Duolingo's YouTube and TikTok, each against an earlier scan folder; diff.
• Live-tested (1.3.0): the Podcast block on four shows, by name, Apple ID and pasted link, on its own and with Snapchat, plus a name with no match.
• Live-tested (1.2.0): Snapchat Spotlights, TikTok photo carousels and TikTok follower counts on public accounts, logged out, plus degraded runs without ffmpeg, whisper or yt-dlp.
• Live-tested: doctor (also with ffmpeg, whisper-cli or yt-dlp off PATH, and --get-model, including simulated failed downloads), scan on YouTube and TikTok (also with tools off PATH, a mistyped handle and a reused folder), meta, hooks, comments, report, and Instagram through the Apify MCP plus ingest-ig.
• The direct --instagram call is tested against a mocked Apify response (actor, input, caps, files), not yet against a live run. The Linux install lines in doctor follow each project's docs but haven't been run on Linux.
• Not built yet: same-age cohort comparisons, caption length vs views, matching the same clip across platforms.

Access checklist (native analytics)

Public data can't show reach, follows per post, retention, traffic sources or ad spend. Ask the account owner for:

1. Instagram: Partner access in Meta Business Suite with Insights. Never take a password.
2. Ads: Analyst access in Ads Manager (the only way to see boosts and spend).
3. TikTok: Business Center access or a CSV export.
4. Snapchat: a Collaborator or Admin role on the Public Profile for Insights (saves, watch time, completion, earnings). Never take a password.
5. YouTube: Studio Viewer, or Viewer (Limited) to hide revenue.
6. Podcast host dashboard (downloads per episode) and link-in-bio click data.