Overview
Tracks what speaker "Pu Boat" (Thai name: โบ๊ต, also rendered PhuBoat) does on screen and in his webcam throughout an Anibon Official live stream. Solves the Speech vs. Action Divergence problem where the speaker discusses real-world topics while casually playing a game or browsing the web.
When to Use
- Live streams featuring Pu Boat where he plays games, browses websites/reviews, reacts to videos, or pulls gacha.
- When
anibon-timestamperneeds visual ground truth to prevent hallucinated in-game lore, resolve deictic pronouns ("Look at this", "This game"), and ground AFK breaks. - In multi-hour streams where speaker leaves the desk (AFK empty chair detection).
Storyboard Resolution & Download
Always download the highest-resolution storyboard tier available to avoid pixelation and blur when inspecting UI text, character names, or facial expressions:
# Downloads highest resolution storyboard (sb3 -> sb2 -> sb1 -> sb0)
yt-dlp -f "sb3/sb2/sb1/sb0" "https://www.youtube.com/watch?v=VIDEO_ID" -o "<workspace>/frames/storyboard.%(ext)s"
Pipeline & Helper Scripts
All helper scripts reside in skills/anibon-stream-activity/scripts/:
1. Unpack Storyboard Slides (unpack_storyboard.py)
Extracts clean native JPEG slide sheets from MHTML and provides frame cropping across arbitrary grid sizes (3x3, 5x5, 10x10):
python3 scripts/unpack_storyboard.py \
--mhtml "<workspace>/frames/storyboard*" \
--out-dir "<workspace>/frames/slides"
2. Extract Activity & Webcam Timeline (extract_activity_timeline.py)
Samples storyboard frames every 60–90 seconds + scene change shifts, runs Gemini Vision / agy classification, and merges into continuous time intervals:
python3 scripts/extract_activity_timeline.py \
--slides-dir "<workspace>/frames/slides" \
--duration <DURATION_SECONDS> \
--step-sec 60 \
-o "<workspace>/activity_timeline.json"
Output Schema (activity_timeline.json):
[
{
"start": "00:15:00",
"end": "00:45:00",
"start_sec": 900,
"end_sec": 2700,
"app_or_game": "Monster Hunter Wilds",
"category": "Gameplay",
"details": "Hunting Dosshaguma in Windward Plains",
"webcam": {
"speaker_present": true,
"expression": "focused",
"layout": "corner_cam"
}
}
]
3. Align to Transcript Chunks (align_activity_timeline.py)
Maps activity intervals across 5-minute transcript chunks:
python3 scripts/align_activity_timeline.py \
--timeline "<workspace>/activity_timeline.json" \
--chunks "<workspace>/chunks/" \
-o "<workspace>/activity/"
Emits <workspace>/activity/activity_chunk_NN.txt ready for subagent prompt injection.
Disambiguation & Subagent Reasoning Rules
When injecting ON_SCREEN_ACTIVITY into anibon-chunk-timestamper, subagents MUST apply these rules:
- Divergence Rule (Speech ≠ Screen Action):
- When dialogue covers general topics (life stories, food, news, anime lore) while casually playing or farming:
- Use speech-primary tag:
[Talk],[News], or[Question]. - Suffix the background game in parentheses:
[Talk] เม้าท์มอยเรื่อง... (ระหว่างเล่น Monster Hunter). - NEVER fabricate or hallucinate real-life stories as in-game lore or quest text.
- Use speech-primary tag:
- When dialogue covers general topics (life stories, food, news, anime lore) while casually playing or farming:
- Convergence Rule (Speech = Screen Action):
- When dialogue actively reacts to in-game combat, boss mechanics, or gacha:
- Use action tag:
[Gameplay],[Gacha], or[Reaction].
- Use action tag:
- When dialogue actively reacts to in-game combat, boss mechanics, or gacha:
- Webcam & AFK Grounding:
- When
speaker_present: false(empty chair / away from desk) and audio is silent or ambient:- Tag as
[AFK] ปู่โบ๊ตลุกไปพักเบรก / เข้าห้องน้ำif > 2 minutes. - Explains silent gaps cleanly and prevents false topic switching.
- Tag as
- When
- Physical Emotion Grounding:
- Cross-reference webcam expression (
laughing,shocked_facepalm) with LiveChat555bursts to accurately calibrate tone in descriptions.
- Cross-reference webcam expression (
- Deictic Pronoun Grounding:
- Resolve "ดูนี่ดิ", "This game", "This one" using the
app_or_gamename and visible UI text inON_SCREEN_ACTIVITY.
- Resolve "ดูนี่ดิ", "This game", "This one" using the
- Visual Ground Truth Priority Over Spoken Misspeaks:
- Streamers frequently misspeak numbers, campaign titles, or dates when tired. On-screen graphics, web browser tabs, and game banners take strict precedence over spoken words in ASR.
- Japanese Number Unit Conversion Trap (
万vsล้าน):- Japanese games and wikis use
万(10,000 / man) and億(100,000,000 / oku). - Streamers commonly misread
3500万as "3,500 million" or "More than 30,000 million". - Always convert mathematically:
3,500 × 10,000 = 35,000,000= 35 million (35M DL).
- Japanese games and wikis use
- LiveChat as Real-Time Correction Canary:
- When spoken numbers sound absurd or contradictory, look at the LiveChat comments around that timestamp. Chat listeners are often the first to notice and correct the streamer (e.g. "35 million is enough, Grandpa. 30,000 million is more than the world's population already.").