SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsedu-math-video

edu-math-video

Writing & research

View repository

Updated 10-16 04:01

About this skill

Math Problem Explainer Video (edu-math-video)

Turn a math problem into a 16:9, 1920×1080 MP4 with Chinese narration + Chinese-English bilingual subtitles + hand-drawn notebook-style animation.

Pipeline (one folder per video):

script.json ──build_audio.py──► build/timeline.json + build/mix.wav + ../<name>.srt
     (旁白)     (GLM-TTS + 合成音乐)          │
anim.js + engine.js ◄─────────────────────────┘ (按旁白时间点驱动动画)
     └──node render.mjs video──► ../<name>.mp4

Core principle: the visuals follow the narration, and the diagrams "explain the problem" too. For every narration sentence, there is an action on the diagram that performs this reasoning (a segment slides over to coincide, a triangle is overlaid, the camera turns to a top-down view, a moving point moves…), not just an extra line or label appearing. Every animation is timed using "the moment when the k-th narration sentence begins"S.at(k, f) , never with hardcoded seconds.

This skill includes all the code itself, do not copy files from other projects:

  • template/: a complete, directly runnable example project (right-triangle median-to-hypotenuse problem).engine.js is the general engine (do not modify),anim.js / script.json / episode.json / problem.png must be rewritten for every problem.
  • shared/pron.py + shared/pron.json: pronunciation control (GLM-TTS has no SSML/pinyin input).
  • examples/cone-parallel/: a complete solid geometry example (anim.js + storyboard.md + script.json): camera oblique view ↔ top-down view, a side face unfolds into a sector, alternate interior angles rotate, a plane translates. Read it first when doing solid geometry, circles, or multi-question problems.
  • scripts/：new_video.sh(create project),setup_check.sh(check environment),make_problem_png.py(word problem → image + highlight boxes),problem_boxes.py(screenshot → highlight box coordinates),contact_sheet.py(screenshot montage review).

Below, SKILL = the directory containing this file,WS = the user's current directory (your working directory,$PWD）: the video folder is created here, and the mp4/srt are output here too.Do not switch WS over just because another directory (including the project this skill is in) already has .env、node_modules、pron.json Just switch WS over, and whatever is missing will be filled in by new_video.sh and step 2.PROJ = this video folder,PY = setup_check.sh the python reported by /usr/bin/python3; on macOS this is usually python3 in PATH may not have numpy).

Rules that must be followed

If the exit code is not 0, it is not done. --check exit code 1 (even if only one ⚠ polyphonic character remains),motion exit code 1, or a problem in a screenshot — none of these can be written off as "just a warning, no impact" and then continue. Fix it before moving on.

  1. Strictly follow the 10 steps below in order, and only move to the next step once that step's "pass criteria" are met. Do not skip steps, and do not "check everything together at the end".
  2. First explain the problem clearly. The first scene of the video must show the original problem image (a screenshot or an image generated by make_problem_png.py ), and the narration reads the conditions one by one, framing that condition on the image each time it is read.
  3. tts The field can only be pronounceable Chinese. It cannot contain Arabic numerals,= + − × ÷ / ^ √ ∠ △ ∥ ° ( ) or symbols such as:AC=3 is written as AC等于三，x² is written as x的平方. The subtitle zh field keeps normal mathematical notation.--check will report an error and block it.
  4. Pronunciation must be fixed. Before calling TTS, --check must display 未固定读音的生僻字/多音字: 0 and script errors: 0. The annotation for a polyphonic character is written after that characterafter：长[cháng]、要[yào]求。must notwrite it as 中点[zhōng](that would read "点" as zhōng).
  5. The API Key can only be provided by the user. When missing GLM_API_KEY , first ask the user: configure a key (best audio quality, recommended), or use the fallback engine (edge-tts / macOS say, see reference/glm-tts-setup.md). Do not make up a key, model name, or API endpoint address; do not print the key or write it into the project folder.
  6. All on-screen content within y ≤ 860（y≈9141044 is the subtitle box). Put graphics in the left half x 60900; write the derivation text on the ruled board on the right board() .
  7. First write the storyboard, then write the animation. storyboard.md For each line of narration, plan "point/move/keep", and follow reference/visual-design.md 's action table to choose actions;node render.mjs motion must pass.
  8. Preview first, then spend money. Use --preview + stills auto to check the layout, and only run the real TTS once it's fine.
  9. Verify by looking at the images. After every rendered screenshot, open build/sheet.png and check it with your own eyes (using an image-reading tool); do not just look at whether the command reported an error. When reporting, write what you actually saw; do not check everything off from impression.
  10. Do not modify engine.js, unless the user asks to change the style; if you need a new graphic, write a function in anim.js .
  11. Do not copy node_modules、package*.json into the video folder; install the dependencies once in the WS root directory, and all videos share them.

11 steps

Step 1: Understand the problem, and first solve it correctly yourself

  • Get the problem text (if there is a screenshot, read the image first).First solve it completely yourself and verify the calculation, and write down: the list of known conditions, what is being asked, the key ideas (1–2), each step of the calculation, the final answer, and the verification method.
  • When the answer is uncertain or the problem is unclear, ask the user; do not guess.
  • Pass criteria: You can explain it in 3–6 "scenes", with each scene's purpose clear in one sentence; the answer has been verified.

Step 2: Create the project, check the environment

bash SKILL/scripts/new_video.sh "$PWD" <ascii_folder_name>     # WS 就是当前目录
bash SKILL/scripts/setup_check.sh WS/<ascii_folder_name>
  • new_video.sh will link pron.py/pron.json to the skill's built-in shared word list, and when it can find it, link node_modules to the existing installation (without copying); only if it cannot find it do you need to WS under npm install. For Python packages, use PY -m pip install --user numpy requests pypinyin pillow。
  • new_video.sh reports ERROR: ... is where the skill is installed , it means you set WS to the project where the skill is located; switch to the current directory.
  • The default voice is chuichui(Chuichui); the user can change it in .env using GLM_VOICE .
  • Missing GLM_API_KEY：stop, and follow reference/glm-tts-setup.md to guide the userto configure (recommended: put it in ~/.config/math-problem-video/.env, works in all directories), then continue.Do not go looking for .env in other projects, and do not switch the working directory because of this. If the user has no key or does not want to configure one:TTS_ENGINE=auto(default) will automatically switch to the fallback engine,setup_check.sh will show which one is being used; get the user's consent before installing edge-tts.
  • Pass criteria: setup_check.sh outputs ALL OK。

Step 3: Test TTS (verify the key and voice)

cd PROJ && PY build_audio.py --say "你好，我们来看一道数学题。"
  • on success it will print WROTE build/say_xxxx.wav and voice: ...（edge: / say: Start = fallback engine). 401/403 = wrong key or no balance; tell the user, do not retry 10 times.
  • The first time you make a video for this user, tell the user this wav path to preview and confirm the voice (see glm-tts-setup.md for the voice list).
  • Pass criteria: wav generated.

Step 4: Prepare the problem image problem.png and highlight boxes

choose one of the two (details in reference/animation.md）：

  • text only：PY SKILL/scripts/make_problem_png.py --text "完整题目" --mark "条件1" "条件2" ... "所求" --out PROJ/problem.png, it directly outputs srcW/srcH/boxes。
  • has screenshot: copy to PROJ/, if necessary problem_boxes.py crop crop out irrelevant parts and save as problem.png; then problem_boxes.py grid generate a coordinate grid image and read out the pixel box for each condition; then problem_boxes.py check draw boxes to confirm.
  • Fill the dimensions and boxes into anim.js the top of PROBLEM。
  • Pass criteria: you opened *.preview.png or *.check.png, the text is complete (no ☒/□ boxes), and each box exactly covers the corresponding text.

Step 5: Write episode.json and script.json

Write according to reference/script-writing.md . Key points:

  • episode.json：title(video title),output_name(output filename, e.g. 斜边中线_Median_to_Hypotenuse）、pop_scenes(the act that needs the "bo" sound effect).
  • script.json: array of acts, each act {"scene": "英文id", "lines": [{"zh","tts","en"}, ...]}. First act intro reads the problem, last act outro recap + report the answer.
  • Each narration sentence ≤ 36 Chinese characters wide (otherwise subtitles wrap to two lines), and each sentence covers only one thing.
  • Pass criteria: JSON is valid; every act id has a planned visual.

Step 6: Write the storyboard storyboard.md

First read reference/visual-design.md, then follow the format of template/storyboard.md to write one line for each narration sentence:| 幕id 句号 | 旁白要点 | 指 | 动 | 留 | 板书 |。

  • "Point": how the element mentioned in this sentence lights up. "Move": on the imageareasoning action, choose from the action table (equal→copy slides over to coincide, congruent→triangle overlays, 3D↔2D→camera rotates, find length→number counting...). "Leave": the mark left after the action.
  • Except for the first and last acts, "Move" cannot be empty, and cannot just say "appear/show". Each act must have at least one continuous motion.
  • Think first: if the viewer only looks at the image and not the board, can they understand why this step holds? If not, change the action.
  • First write at the top of storyboard.mdfigure list: all points, line segments, parallel/perpendicular/midpoint relationships, and known lengths in the problem must all be drawn, with positions calculated from the data.
  • Pass criteria: every sentence has a line; the figure list is complete; you can state the action in each act that is "most understandable".

Step 7: Pronunciation and script check, fix to 0

cd PROJ && PY build_audio.py --check ; cat build/pron_report.txt
  • ERROR lines: follow the prompt to change script.json(usually there are numbers/symbols in tts, or anim.js is missing SC.<id>; after step 8 writes anim.js, the latter will report an error, which is normal).
  • ⚠ 多音/生僻 lines: handle according to reference/pronunciation.md : add after the character [拼音], or add recurring words to pron.json 's words, and add words confirmed to have correct default pronunciation to ok。
  • Pass criteria: 未固定读音的生僻字/多音字: 0 and script errors: 0, exit code 0.

Step 8: Write anim.js

Implement sentence by sentence according to storyboard.md, API see reference/animation.md. First read through template/anim.js(example) before starting, keep the structure consistent:

  • PROBLEM(data from step 4),TAGS(label at the top-left of each act), graphics functions (such as fig(st)）、SC.<id> = (lt, S) => {...}(one per act).
  • The appearance time of each element = the moment the narration mentions it:S.P(S.at(k, f), 时长)。
  • Each sentence:glow(..., bump(lt, at(k))) light up the mentioned element → the action in the storyboard (slideSeg、movePoly、camTween, moving point,countTo…) → leave a mark.
  • On the right, board() + bl(行号, 文字, S.at(k, f), lt) write the derivation line by line (same color as the diagram), finally stamp() stamp to give the answer.
  • Pass criteria: PY build_audio.py --check no ERROR.

Step 9: Free preview: action check + layout check

cd PROJ && PY build_audio.py --preview && node render.mjs motion && node render.mjs stills auto && PY SKILL/scripts/contact_sheet.py .
  • motion Take 6 frames in each sentence, compare the changes in the left-half graphics area:STATIC(no change) must be fixed; each act has at least one sentence MOVE. If it does not pass, go back to the storyboard and replace it with a better action; do not just add a meaningless shake to make up the numbers.
  • stills auto Take a frame near the end of each narration sentence (at this point everything that should appear in this sentence should be on screen). Command output FAILED: page error indicates anim.js has a JS error, fix it first.
  • Open build/sheet.png(when more than 12, it is sheet_1.png、sheet_2.png…) check them one by one, against the review checklist in reference/animation.md: no overlap, no out of bounds, no blocking the subtitles, everything mentioned in each sentence is drawn, and the values match the solution.
  • If there is a problem → modify anim.js → rerun this step.
  • Pass criteria: motion check passed; all screenshot checks pass (explain act by act in the reply what you saw).

Step 10: Actually generate the voiceover, automatically transcribe and verify, then check the visuals once more

cd PROJ && PY build_audio.py && PY build_audio.py --asr && node render.mjs stills auto && PY SKILL/scripts/contact_sheet.py .
  • TTS is cached by text in build/tts/, changing a sentence will only re-synthesize that sentence.
  • --asr Use Zhipu speech recognition to convert each voiceover sentence back to text, write to build/asr_report.txt(results are cached).Sentences whose letter sequence does not match are marked ✗, exit code 1: modify that sentence (see the "letters" section of pronunciation.md) and rerun these two commands. Also scan the Chinese characters in the report; if you find obviously mispronounced characters (not homophones, and not cases like ASR writing "三" as "3"), handle them according to step 7.
  • The actual duration differs from the preview, review the sheet again.
  • Pass criteria: prints mix + srt written；--asr shows 字母读错的句子: 0(when there is no GLM key, shows ASR SKIPPED: when delivering, tell the user that the letter pronunciations have not been machine-verified, and list the sentences containing names for the user to listen to carefully); screenshot checks pass.

Step 11: Render the video and deliver

cd PROJ && node render.mjs video 6      # 6 = 并行浏览器页数（不是帧率！帧率固定 30）
  • Output WS/<output_name>.mp4 and WS/<output_name>.srt(WS = the user's current directory). A 2~3 minute video takes about a few minutes.
  • When delivering, tell the user: mp4/srt paths, duration, content of each act;explain that you cannot listen to audio, ask the user to listen to the pronunciation, and especially list the characters whose pronunciation you annotated. When the user reports a mispronunciation: add to pron.json or inline annotation → then run steps 10 and 11 again (only the changed sentences will be re-synthesized).

Common mistakes (that actually happened)

Wrong approach Correct approach
The current directory does not have .env/node_modulesthen build the project in another directory that has these things (for example, the project where the skill is located) WS is always the user's current directory; missing dependencies are linked by new_video.sh, and if the key is missing, ask the user to configure it ~/.config/math-problem-video/.env
copy code from another video folder (chem, etc.) Use new_video.sh from this skill's template/ to build the project
fabricate GLM_MODEL=glm-4-voice、api.zhipuai.cn and other configurations such as The model is only glm-tts, and the address is already written in build_audio.py; the user only needs to provide GLM_API_KEY, optional GLM_VOICE
node render.mjs video 24 thought it was 24fps the parameter is the parallelism count; use stills autofor previewing the layout, and do not repeatedly render the entire video
中点[zhōng] The annotation only applies to the immediately preceding character:中[zhōng]点(actually "midpoint" does not need to be annotated)
write in tts AC=3、60 − 7t、x² AC等于三、六十减七t、x的平方
the problem screenshot can be placed arbitrarily at (50,50) 400×400 Use PROBLEM + problemCard() to automatically fit into the card below the title
Use absolute seconds to time if (lt > 12.5) S.at(k, f): at proportion f after the k-th sentence starts
deliver after rendering without looking at the visuals look at it every time build/sheet.png
manually write in tts C、O or C,O to separate letters No need to worry about it: pron.py automatically splits adjacent letters into C O(in practice, commas and enumeration commas cause extra pauses)
--check If there is still ⚠, adjust TTS fix it to 0; otherwise you will spend money synthesizing incorrect pronunciations
add all common words to pron.json or inline annotations for the sake of "safety" Only annotate characters that will be mispronounced; replacing them with homophones will break sentence segmentation (直搅三搅形），pron.py already only replaces when mispronounced in context
a clause of more than twenty characters without punctuation add commas at breath points, clauses ≤ 18 characters (--check will warn), otherwise TTS pauses in the middle of a word by itself
In the problem image, x² is displayed as x☒ but it was not noticed zoom in to check the text during review; make_problem_png.py already switches fonts automatically
after modifying JSON with an editing tool, curly quotes appear “, parsing fails JSON syntax quotes must be English "; see troubleshooting.md
draw the diagram once and for all, then each sentence only adds one more line/label, and all explanation is on the board every sentence has "point→move→keep"; if equal, slide over and coincide, and use camTween；render.mjs motion for 3D↔top view must pass
when a top view is needed, draw another diagram beside it the same 3D model camTween rotates to pitch = π/2
cram three or four knowledge points into one narration sentence one point per sentence, and one element appears synchronously on the screen

Reference files

  • reference/glm-tts-setup.md: Zhipu GLM-TTS registration, API Key,.env, voice, speed, error handling.Read this when guiding the user through configuration.
  • reference/script-writing.md：script.json format, act design, and a comparison table of spoken ways to write mathematical expressions.
  • reference/pronunciation.md: pronunciation control principles, annotation syntax,pron.json, common polyphonic characters in mathematics.
  • reference/visual-design.md：how to design explanatory animations: Finger→Move→Stay→Connect, reasoning→animation action table, 3D to top-down view, cone unfolding, storyboard format. Must-read before writing storyboards.
  • reference/animation.md: engine.js API, scene syntax, layout coordinates, geometry/function/3D/distance problems drawing methods, review checklist.
  • reference/troubleshooting.md: Errors and handling.