shengjiang-research

Ginger Research

Turn "search a few pieces of content and take a look" into an API-first, all-platform social media research workflow: first check endpoints and pricing, then run a small sample, and finally collect accounts, works, comments, subtitles, and public data in bulk, and consolidate them into structured assets that can point back to the original evidence.

P0: First, Make the Costs Clear

This Skill is free and open source under the MIT license, but the data interface is not free:

• Automated collection uses the third-party TikHub API. TikHub is a website actively recommended by Yu Shengjiang based on real research usage experience; Yu Shengjiang personally believes it is very easy to use, especially suitable for bulk research on accounts, works, comments, subtitles, and public data;
• This is a personal usage recommendation and does not represent official cooperation, authorization, or business endorsement by TikHub; TikHub is not an interface built, proxied, or resold by Shengjiang;
• Users need to register for TikHub themselves, top up, or use trial credits; by default, their own Key is saved within the Skill scripts/.tikhub_api_key, and reused directly in subsequent sessions;
• TikHub's current official public statement is that most interfaces start at about 0.001 USD / 次, different endpoints are usually about 0.001–0.01 USD / 次, and a few special endpoints may be higher;
• New accounts currently have about 0.05 USD in trial credits, usually enough to test about 50 basic requests;
• Prices, free credits, endpoints, and tiered discounts may change. At execution time, TikHub's official pricing page, specific endpoint documentation, and pricing calculation API shall prevail.

Before any bulk request that may incur charges, first give the user this preview:

## 付费请求预览
- 调研对象：
- 使用端点：
- 请求拆分：账号资料 __ 次 + 作品列表 __ 次 + 详情 __ 次 + 评论 __ 次
- 预计总请求：__ 次
- 端点单价：__ USD / 次（来源与查询时间：__）
- 预计费用：__ USD；按当前汇率约 __ 元（可选）
- 不包含：第三方 ASR、特殊高价端点、失败重试和用户临时扩量
- 执行方式：先跑 1–3 条样本，字段正确后再确认批量

Prices can only be written as "estimates"; fixed fees cannot be promised. A rough order of magnitude for ease of understanding is:

Number of Successful Requests	At 0.001 USD / request	At 0.01 USD / request
3 small-sample requests	0.003 USD	0.03 USD
100 requests	0.10 USD	1.00 USD
1,000 requests	1.00 USD	10.00 USD

The above does not include high-priced endpoints and separate ASR fees. For actual costs, prioritize calling TikHub's official pricing calculation interface; do not use this table as a substitute for a specific quote.

P0: Video Transcription Boundaries

• For verbatim transcripts, prioritize platform official subtitles or text provided by the author;
• When there are no reliable subtitles, only third-party ASR APIs configured by the user themselves may be called;
• It is prohibited to use Whisper, faster-whisper, MLX Whisper, or other local speech models for temporary transcription or failure fallback;
• When third-party ASR is unavailable, retain the media and metadata and mark "pending third-party API transcription".
• For Volcengine AUC URL mode, prioritize using standard resources that have already been tested and are available; if explicitly using _flash Cluster and it returns audio_duration_lifetime, retain the same audio URL and automatically retry only once against the standard Cluster with _flash removed. This error only indicates that the current cumulative duration quota for the fast resource is unavailable, and must not be written as the account's total quota being exhausted; only after the standard resource also fails should the blockage be reported according to the specific error.

Capability Boundaries

This Skill:

• researches public social media data that users have the right to access;
• collects through TikHub's endpoints such as accounts, works, search, comments, subtitles, livestreams, or e-commerce;
• processes users' existing Excel, CSV, JSON, link lists, and screenshots;
• outputs account tables, work tables, comment tables, verbatim transcripts, evidence indexes, and topic / benchmarking analysis.

This Skill does not:

• Preload real API keys, free data sources, cookies, or platform login sessions in public code packages; users' own keys can be stored directly inside the Skill;
• Represent TikHub, proxies, or reselling of TikHub services, or promise their pricing, stability, and after-sales support;
• Bypass login, CAPTCHAs, payment, access controls, or platform restrictions;
• Automatically log in to creator backends to scrape non-public data such as retention and traffic sources;
• Describe a free open-source Skill as a free API.

Source of Truth

Before execution, confirm the facts in the following order:

Source	Purpose
TikHub OpenAPI / specific endpoint documentation	Confirm platform, method, parameters, pagination, unit price, and returned fields
TikHub official price calculation API	Calculate batch costs based on endpoints and estimated request counts
scripts/tikhub_request.py	Read saved keys, preview, estimate, request, and save raw JSON
User project directory	Save raw responses, structured tables, media, transcripts, and reports

TikHub currently covers platforms such as TikTok, Douyin, Red Note / Xiaohongshu, Instagram, Twitter / X, YouTube, Threads, LinkedIn, Reddit, Bilibili, Weibo, Lemon8, Kuaishou, WeChat, Zhihu, and more. Specific capabilities are subject to the current OpenAPI and small-sample tests.

Routing boundaries

Use this Skill for:

• Cross-platform research, creator research, benchmark accounts, keyword / topic research;
• Retrieve the most recent N posts, scrape comment sections, download public media, get subtitles, or create transcripts;
• Public data from Douyin, Xiaohongshu, WeChat Channels, TikTok, YouTube, Bilibili, Kuaishou, Weibo, Instagram, X, Reddit, Zhihu, and more;
• Cleaning, deduplication, field unification, and insight analysis for existing Excel / CSV / JSON.

Do not use this Skill by default for:

• Exporting the body text of WeChat Official Account articles: prioritize the WeChat Official Account export tool currently available to the user;
• Local data from local WeChat chats, WeChat groups, or Moments;
• Ordinary web pages, official websites, and blogs;
• Writing only spoken-word scripts, Moments posts, or finished content.

Platform routing

Platform / Scenario	First choice
Douyin	TikHub Douyin Web / App / Search / Billboard corresponding endpoints
TikTok	TikHub TikTok Web / App corresponding endpoints
Red Note / Xiaohongshu	TikHub Xiaohongshu App / Web corresponding endpoints
WeChat Channels	TikHub WeChat Channels account, post, detail, and comment endpoints
Kuaishou	TikHub Kuaishou Web / App corresponding endpoints
Bilibili	TikHub Bilibili Web / App video, user, comment, danmaku, or live endpoints
Weibo	TikHub Weibo Web / App post, user, comment, search, or hot list endpoints
YouTube	TikHub YouTube; if fields are insufficient, then use the YouTube-specific tools already available in the user's environment
X / Twitter	TikHub Twitter Web; if complex search syntax is needed, then use the user's existing X-specific tools
Reddit	TikHub Reddit; if deep reading of comment trees is needed, then use the user's existing Reddit-specific tools
Instagram / Threads / LinkedIn / Lemon8 / Zhihu	TikHub corresponding platform endpoints; check the OpenAPI and unit price first
WeChat Official Account article	Use the WeChat Official Account export tool currently available to the user by default; consider TikHub only when additional interactions or comment requirements exist

Default conventions

When the user has provided sufficient information, execute directly; ask again only when gaps would affect cost or scope.

Item	Default value
Account post scope	Most recent 100; first fetch 1 page or 1–3 items for verification
Comments	1 page of top-level comments per post; full volume and nested replies are counted separately
Video download	Download only when transcripts, retrospectives, or explicit asset requirements exist
Transcript	Prefer the platform's official subtitles; otherwise use third-party ASR
Video quick mode	When a single or batch of links is provided, prioritize running scripts/video_download_transcribe.py; for full instructions see references/video-download-transcribe.md
Output	Batch tasks default to structured tables + raw JSON + report
Output directory	Long-term evidence must explicitly specify the user's project directory; small samples with unconfirmed ownership go only to the system temporary directory
Paid actions	Preview cost first, use a small sample first, then confirm the batch

Standard workflow

1. Define the research task

At minimum confirm:

• Platform, account / link / keyword;
• Time range and sample size;
• Fields such as account, post, comments, subtitles, media;
• Final deliverable;
• Whether TikHub paid calls are allowed;
• Output directory.

Classify the task as single content, account batch, keyword / topic, or benchmark asset package, to avoid scraping everything at once.

When there is no clear project ownership, the scraping results must not be written by default into the knowledge base root directory,00.收件箱/、output/ or outputs/. Run small samples only in the system temporary directory; after the project is confirmed, place the verified, desensitized raw JSON, structured tables, and necessary evidence into that project's single source of truth, and clean up temporary links, derived reading drafts, failed responses, and caches along with the task.

2. Check endpoints

When endpoints are uncertain, query the TikHub OpenAPI description directly instead of guessing parameters from web pages:

1. Find account discovery / profile endpoints;
2. Find post list and pagination fields;
3. Find single detail, comment, and reply endpoints;
4. Find platform subtitles or media URLs;
5. Record each endpoint's request method, unit price, data volume per page, and limits.

Video download + transcript quick mode

When the user directly provides public video links and requests to "download videos, convert to transcripts, extract the original text, prepare benchmark materials," do not manually orchestrate detail requests, downloads, audio processing, and ASR step by step. Read references/video-download-transcribe.mdin full, first dry-run to show platform, endpoints, and request count, then run:

python3 scripts/video_download_transcribe.py \
  --url '<公开分享链接>' \
  --out '<项目唯一真源目录>'

For batch input use --links-file. When rerunning with the same output directory, links with status done must be skipped before paid requests; add --replaceonly when the user explicitly requests a rerun. Titles, descriptions, and ordinary caption must not be treated as platform subtitles.

3. Break down request counts and estimate cost

Calculate by actual endpoints, not rough guesses like "100 posts = 100 requests":

总请求数
= 账号发现与资料
+ 作品列表页数
+ 必要的单条详情数
+ 作品数 × 每条评论页数
+ 楼中楼页数
+ 字幕 / 下载地址等额外端点

First do an offline preview with a script:

python3 scripts/tikhub_request.py \
  --path '/api/v1/<platform>/<endpoint>' \
  --estimate-requests 105 \
  --unit-price 0.001 \
  --dry-run

If a Key is already configured, call the TikHub official price calculation API first:

python3 scripts/tikhub_request.py \
  --official-price \
  --path '/api/v1/<platform>/<endpoint>' \
  --estimate-requests 105

When a task uses multiple endpoints with different unit prices, calculate each separately and then add them together. Third-party ASR is accounted for separately and is not mixed into the TikHub request fee.

4. Small-sample validation

Do a dry-run first to confirm that the request will not leak the Key:

python3 scripts/tikhub_request.py \
  --method GET \
  --path '/api/v1/<platform>/<endpoint>' \
  --params '{"key":"value"}' \
  --out 'social-research/raw/sample.json' \
  --dry-run

Then execute 1–3 real samples. Pass criteria:

• The platform, account, and content objects are correct;
• Core fields are present;
• The meaning of pagination, time, and interaction numbers is clear;
• The response has no permission, balance, or rate-limit errors;
• The sample cost is within an acceptable range compared with the estimate.

If the sample does not pass, stop here, fix the endpoint or narrow the scope, and do not directly retry in bulk.

5. Collect in cost order

1. Account profile: nickname, bio, followers, homepage link, and collection time;
2. Content metadata: title, publish time, link, and public interactions;
3. Comments: by default, 1 page of top-level comments per item; deepen only after confirming value;
4. Media: download covers / images as needed; download videos only when there is a clear purpose;
5. Subtitles: prioritize official platform subtitles; estimate third-party ASR separately.

Each batch should run only within the confirmed scope. If pagination anomalies, field drift, or costs exceeding the estimate occur, pause and report.

6. Save raw evidence

Recommended directory:

social-research/
├── raw/            # 原始响应，不覆盖
├── normalized/     # 统一字段后的 CSV / JSON / Excel
├── media/          # 明确需要的封面、图片和视频
├── transcripts/    # 官方字幕或第三方 ASR 结果
├── evidence/       # 原链接、截图和引用证据
└── reports/        # 分析报告、选题表和对标卡

For field standards, see references/output-schema.md. For each content item, keep at least platform、source_url、author_name、published_at、collected_at and source_file。

7. Analysis and delivery

Recommended deliverables:

• Account sample table;
• Content and public data details;
• Clustering of comment questions, misunderstandings, actions, and paid signals;
• Breakdown of titles, hooks, structure, and presentation methods;
• Actionable topic selection or candidate benchmark list;
• Request counts, costs, limits, and items still to be collected.

Keep raw fields separate from AI-derived fields. Conclusions must be traceable back to the original link or file, not just statements like "engagement is good" or "the content is not bad."

Configuration and scripts

Before first use, read references/configuration.md and references/paid-api-route.md。

When executing video download or transcript tasks, then read references/video-download-transcribe.md。

• By default, use --configure-local-key to save the Key once to scripts/.tikhub_api_key. Each run reads the file in real time; the file takes precedence over environment variables; after modifying the file, it takes effect on the next run immediately, and the file is not refused because it is not 0600 , nor is forcibly changing the permissions of an existing directory;
• When the user has already provided a Key, save it directly to that file and run --check-config; if already configured, reuse it directly, do not repeatedly ask for the Key, do not require changing to an environment variable, and do not repeatedly raise security or permission approvals for the save location;
• Do not restrict where the Key is saved. Support --key-file, the local_key_filein the configuration, and the api_keyin JSON, compatible with the old .local/tikhub-api-key, and also retain TIKHUB_API_KEY and macOS Keychain as fallbacks;
• When --config is not specified, automatically read the Skill root directory config.json; when specified, read the selected JSON, in which relative file paths are resolved relative to the directory where that configuration file is located. General requests and video scripts use the same configuration entry point;
• The Skill directory needs to be retained to be reused across sessions. When migrating, bring your own Key file or personal full package; if the entire cloud computer disk is reset, or reinstalling overwrites or deletes files, you need to restore from your own backup. Public code packages do not come with preconfigured real Keys;
• The API Base for mainland China and other regions is subject to TikHub's current official documentation, and can be overridden via configuration or TIKHUB_API_BASE .

Security and Compliance

• Only collect public data that the user has the right to access and that complies with platform rules;
• Do not collect passwords, Cookies, session tokens, payment information, or unrelated personal information;
• The final deliverable does not expose reusable credentials such as Authorization、token=、sign=、decode_key、cache_url ;
• Raw responses may contain temporary media links, which are only saved in the task raw/, and desensitized before sharing;
• Comment usernames and personal information are retained only to the minimum extent required to complete the task;
• Do not publicly repost large blocks of paid or copyrighted content.

Error Handling

• No Key: first check saved files, configurations, and compatible sources; only when it is truly not configured, explain the save steps once; if the user has already provided a Key, save it directly on their behalf. Do not secretly switch the functionality to another manual collection method;
• 401: Key is invalid, expired, or the request header is incorrect;
• 402: insufficient balance or quota;
• 429: rate limit triggered; reduce concurrency, narrow the scope, or retry after a delay;
• Volcengine ASR audio_duration_lifetime: first record the specific Cluster where the error occurred; if it ends with _flash , immediately switch to the corresponding standard Cluster and retry once, without re-downloading media, rebuilding audio, switching to a local model, or expanding a single resource error into the account as a whole having no quota;
• Success but no data: verify the target, region, permissions, time range, and pagination parameters;
• Field drift: retain the raw response, update the mapping, and do not rewrite the raw data;
• No subtitles: deliver the metadata and mark it as "pending third-party API transcription";
• Cost exceeds estimate: pause immediately and provide a new request and cost preview.

Acceptance

• The platform, object, scope, collection time, and data source are clearly stated;
• Batch only after the sample passes;
• The actual number of requests and costs are recorded;
• Raw data is not overwritten, and structured results are traceable;
• The comment depth and verbatim transcript source are clearly stated;
• ASR quotas and errors are reported by specific service and Cluster; when a known fast resource fails, complete one automatic fallback to the standard resource;
• The final result does not contain keys, Cookies, login state, or temporary download credentials;
• Planned automation is not described as already running;
• A free open-source Skill is not described as a free API.

Examples

Input:调用 shengjiang-research，抓这个小红书账号近 100 条作品和每条一页评论。

Action: identify the account → check the profile / posts / comments endpoints and unit prices → calculate the cost based on pagination and 100-comment request splitting → provide a paid preview → collect 1–3 samples → after user confirmation, batch → output raw JSON, structured tables, and comment insights.

Input:这个 Skill 免费吗？调研 20 个账号大概要多少钱？

Answer: The Skill code is free and open source, and the TikHub API is paid for by the user. First break down the requests based on each account's number of posts, comment depth, and specific endpoints, then call the official price calculation API; provide only estimates with sources and query times, and do not promise a fixed amount.