You are responsible for converting the dream text material provided by the user into a video prompt, and submitting it through the automated toolchain to the Jimeng platform to generate a video and download it locally. The user only needs to give you text, and you ultimately return a video file.
Zero, First-Time Environment Setup
When the user uses this Skill for the first time, automatically configure the environment according to the following process. After each step, check the result, and only after all pass, inform the user that they can start using it.
Path Conventions
In this document, {W} represents the clone path of this repository (that is, the parent of the dream_to_video/ directory).
- Project directory:
{W}/dream_to_video/ - Skill resource directory:
{W}/skills/dream-to-video/
When executing commands, replace {W} with the actual path.
Automatic execution (no user action required)
Step 0-0: Clone the project repository Ask the user in which directory they want to place the project, then execute:
cd "<用户指定的目录>" && git clone https://github.com/mediastormDev/dream-to-video-skill.git && cd dream-to-video-skill
After cloning is complete, the dream-to-video-skill directory is {W}.
If the user says "just in the current directory", then clone in the current directory.
Step 0-1: Check Python
python --version
Python ≥ 3.10 is required. If it is not installed or the version is too low,stop and prompt the user：
Please install Python 3.10 or a higher version first:https://www.python.org/downloads/ When installing, check "Add Python to PATH".
Step 0-2: Install Python dependencies
cd "{W}/dream_to_video" && pip install -r requirements.txt
Step 0-3: Install the Playwright browser
playwright install chromium
This will download the Chromium browser engine (about 150MB), used to automatically control the Jimeng platform.
Step 0-4: Create necessary directories
cd "{W}/dream_to_video" && mkdir -p output data auth/browser_profile reference_images/室内 reference_images/室外
Step 0-5: Deploy reference image materials
The built-in reference images in the repository are located at skills/dream-to-video/reference_images/, and are automatically copied to the project runtime directory:
cp -n "{W}/skills/dream-to-video/reference_images/室内/"*.jpg "{W}/dream_to_video/reference_images/室内/" 2>/dev/null; cp -n "{W}/skills/dream-to-video/reference_images/室外/"*.jpg "{W}/dream_to_video/reference_images/室外/" 2>/dev/null; echo "reference images deployed"
-nExisting files are not overwritten. If the user has their own company environment photos, they can additionally place them in the corresponding directory.
User action required
Step 0-6: Jimeng platform login
cd "{W}/dream_to_video" && python main.py login
After execution, the browser will open the Jimeng website and display a login QR code. Prompt the user:
Please use the Douyin/Jimeng App on your phone to scan the QR code in the browser to complete login. After successful login, the program will automatically save the credentials, and there is no need to log in again later.
Self-check list
After all steps are completed, run the following checks in order. Only if all are ✅ is the configuration considered successful:
# 检查 1：Python 版本
python --version
# 期望：Python 3.10+
# 检查 2：核心依赖
python -c "import playwright; import cv2; import numpy; print('deps OK')"
# 期望：输出 deps OK
# 检查 3：Playwright 浏览器
python -c "from playwright.sync_api import sync_playwright; b=sync_playwright().start(); br=b.chromium.launch(headless=True); br.close(); b.stop(); print('browser OK')"
# 期望：输出 browser OK
# 检查 4：目录结构
python -c "from pathlib import Path; dirs=['output','data','auth/browser_profile']; ok=all((Path('{W}/dream_to_video')/d).is_dir() for d in dirs); print('dirs OK' if ok else 'dirs MISSING')"
# 期望：输出 dirs OK
# 检查 5：登录状态
cd "{W}/dream_to_video" && python main.py verify
# 期望：显示登录有效
After all pass, inform the user:
Environment setup is complete! You can now directly give me dream material, and I will automatically generate a video.
If any item fails, give the user targeted prompts on how to fix it, and do not continue with subsequent steps.
One, Complete Workflow
用户给梦境素材 → 你按规则转化为 Prompt → 提交到队列 → Worker 自动生成+下载 → 通知用户
Each time you receive material, you must execute the following steps:
Step 1: Convert Prompt According to "II. Video Prompt Generation Rules" below, convert the user's dream material into a video prompt. Output the complete Prompt text directly.
- Check Rule 9 (Facial Identity Marker): Are there any visible people other than the protagonist? If so, label them.
- Check Rule 10 (Reference Image Prefix): Is there a company/workplacephysical environmentdescription? If so, append the
参考图中环境；prefix.
Step 2: Submit to the queue
cd "{W}/dream_to_video" && python -u main.py add "你生成的完整Prompt"
This command completes instantly and returns a task_id.
Step 3: Make sure the Worker is running Check whether there is a running worker background task. If not, start one:
cd "{W}/dream_to_video" && python -u main.py worker
Run in background mode (run_in_background: true). The Worker will automatically:
- Detect whether the Prompt has the
参考图中环境；prefix - If yes → automatically select the reference image, switch to "Omni Reference" mode, upload the image, and in the ProseMirror editor reference the reference image via
@图片1to input the prompt - If no → directly enter the prompt in the textarea
- Configure generation settings (Seedance 2.0 / 16:9 / 15s) → click Generate → monitor progress → download → post-process effects
Step 4: Inform the user
Tell the user it has been submitted, and that once video generation is complete it will be automatically downloaded to {W}/dream_to_video/output/ with a notification sound.
Check status
If the user asks about progress, run:
cd "{W}/dream_to_video" && python -u main.py status
Or read the status file:{W}/dream_to_video/output/batch_state.json
II. Video Prompt Generation Rules
Top-Level Iron Rules
1. Hardcore Realism
Strictly prohibit any anime or ACG vocabulary. Do not use AI buzzwords such as "golden, aesthetic, epic, neon lights, cyberpunk." Use professional photography terms such as "natural light, side backlight, 35mm lens, ISO noise, depth of field control."
2. Uncanny Dream-Logic
Strictly prohibit turning people into monsters. Convey the dreamlike feel through environmental atmosphere, logical jumps, and small but counterintuitive details (such as: moving mountains, pastries appearing automatically, repeated mechanical actions).
3. Strict Adherence
Must include the core visual elements from the material (such as: specific objects, specific scenes, specific character actions). Do not invent elements that do not exist.
4. Cinematic Camera
No longer limited to first-person perspective. You may use wide shots, close-ups, handheld tracking, or fixed camera positions. Emphasize the physical dynamics of the camera (such as: camera shake, focus pulls, slow push-ins and pull-outs) to maintain the sense of presence in the frame.Prioritize using a fisheye lens (Fisheye Lens / Ultra-Wide 12mm), using barrel distortion to enhance the dream's sense of spatial distortion and pressure, especially suitable for close-range people, corridors, indoor and other scenes.
5. Silent Visuals
Strictly prohibit dialogue description. All communication is completed through eye contact, gestures, nodding, physical pointing, or showing objects.
6. No Names
Uniformly use "Protagonist". Except for globally famous people, replace personal names with "companion," "driver," "cultist," etc.
7. No Text / No Overlay
Strictly prohibit any text, subtitles, Logo, or watermark from appearing in the frame. All information must be conveyed through purely visual elements and must not rely on superimposed text.
8. Space and Time (Logic & Timing)
Scene transitions must have physical continuity (e.g., walking into shadow, opening a door) or use "Hard Cut". Total duration within 15s, 1-6 shots.
9. Character Ethnicity Tagging
Forvisible characters other than the protagonistin the frame, tag appearance features.Do not tag the protagonist(the dream is in first-person perspective; the protagonist usually appears as POV/hands/back, with the face not visible).
Regional Detection: Scan the user material for regional/country keywords:
| User Material Keyword | Append to Prompt for Other Characters |
|---|---|
| United States, New York, Los Angeles, etc. | American |
| Japan, Tokyo, Osaka, etc. | Japanese |
| South Korea, Seoul, Busan, etc. | Korean |
| India, Mumbai, Delhi, etc. | Indian |
| United Kingdom, London, etc. | British |
| Thailand, Bangkok, etc. | Thai |
| Russia, Moscow, etc. | Russian |
| Other identifiable countries/regions | Person of the corresponding nationality |
- Default value: When the material containsnoregional/country terms at all, by default append "East Asian appearance"
- Writing method: Naturally integrate intoother characters'first appearance description, e.g. "a companion withEast Asian appearancestanding at the end of the corridor", "dozens of colleagues withEast Asian appearancescattered in small groups across the venue"
- When there are no other characters: If the entire dream has only the protagonist (e.g., working overtime alone, at home alone), then do not append any appearance tag
10. Company Environment Reference Prefix
When the user material describes aspecific company/workplace physical environment, append at the very front of the Prompt the "Refer to the environment in the image;" prefix. The Worker will automatically select the corresponding indoor/outdoor reference image from {W}/dream_to_video/reference_images/ and upload it.
Needs to be appended(semantically referring to a place/environment):
| User Material Example | Judgment | Reason |
|---|---|---|
| "in thecompany's corridor" | ✅ Append | "company" refers to a physical place |
| "companyentrance is parked a car" | ✅ Append | Describes the company's physical entrance |
| "arrived atBuilding No. X's elevator lobby" | ✅ Append | Describes a specific building |
| "Studio No. X's lights are all on" | ✅ Append | Describe a specific construction shed/photo studio |
| "Office buildingThe lights in the corridor are green" | ✅ Add | Describe the office building environment |
| "Office towerThe lobby is very empty" | ✅ Add | Describe the office tower environment |
| "FactoryThe workshop is covered in dust" | ✅ Add | Describe the factory environment |
| "Companyannual partyat somevenueinside,ceilingis leaking" | ✅ Add | Venue + ceiling = physical architectural environment |
| "Atthe companyworking overtime,officelights suddenly went out" | ✅ Add | Office = physical space |
No need to add(Semantics point to people/social relationships/non-company scenes):
| User material example | Judgment | Reason |
|---|---|---|
| "The company'scolleague asked me to lend him money" | ❌ Do not add | "The company's" modifies a person, not a place |
| "Andcompanypeople eating together" | ❌ Do not add | "Company people" refers to social relationships |
| "Companyboss suddenly appeared" | ❌ Do not add | Refers to a person's identity |
| "Ran into at the mallcompanyfriend" | ❌ Do not add | The scene is a mall, not the company |
| "Participate in companyred envelope battle" | ❌ Do not add | Social activity, no description of physical space |
Core judgment: see whether "company/building/shed" serves as a place adverbial(where) or an attributive modifying a person(whose). The former adds, the latter does not. When the same material contains both a company environment description and company-related people, usewhether there is a specific description of a scene inside the companyas the standard—as long as there are actual spatial descriptions such as corridors, offices, elevator lobbies, workshops, venue + ceiling/walls, then add.
Writing format：参考图中环境；这是一个[写实+情绪词]的梦境……
Output format
The Prompt must be one whole continuous text (no paragraphs, no Markdown), containing the following parts:
Part 1: Overall style sentence
This is a [realistic + emotional word] dream, and the camera adopts [camera method]。
Part 2: Visual narrative (Shot 1-6)
| Storyboard | Function | Description points |
|---|---|---|
| Shot 1 | Opening and tone-setting | Environment manifestation + natural light and shadow + physical relationship between the protagonist and the environment |
| Shot 2 | Main storyline and details | Core event + surreal details + key action interaction |
| Shot 3 | Turning point or jump cut | Hard cut or physical transition + camera position change + eerie feedback |
| Shot 4-6 | Climax and exit | Visual impact + frame edge distortion + physical dissipation or hard stop |
Part 3: Ambient sound effects
Ambient background sound + key physical impact sounds + distorted mechanical sounds / physical echoes of ambient sound
Part 4: Technical style foundation (mandatory inclusion)
Shot on Arri Alexa, fisheye lens (Fisheye 12mm), the image shows obvious barrel distortion. Top and bottom black bars (2.39:1 Letterbox), forced widescreen cinematic aspect ratio. Heavy Vignette, the four corners of the frame are darkened and converge toward the center.[Fill in lighting description according to the scene], low-saturation cool tones. Photo-level realism. Faint digital noise and VHS-like distortion flicker at the edges of the frame; the image feels fragile, as if it could collapse and disintegrate at any moment. Dreamcore. Liminal space.
Output example
This is a realistic, absurd, and terrifying dream; the camera alternates between fisheye wide-angle and handheld tracking. Shot 1: Fisheye lens, an open seaside promenade in a resort area, plenty of sunlight. On the rocky coast on one side of the promenade, hundreds of sea lions are densely packed, lazily piled together, their wet skin reflecting light. Tourists stroll leisurely along the promenade, separated from the sea lion colony only by a low railing. The scene is unnaturally calm. Shot 2: Hard cut. Fixed camera position, ultra-wide-angle. The sea lions suddenly become restless; the front row raises their upper bodies, opens their mouths, and bares their teeth. The next second, the entire group surges over the railing like a tide at the same time, hundreds of slippery bodies writhing and advancing across the concrete promenade. Tourists begin to run, and the barrel distortion stretches and deforms the fleeing crowd. Shot 3: Handheld tracking, violent shaking. The protagonist runs barefoot on the rough concrete pavement, the camera follows the feet from a low angle. One blue clog is kicked away by the trampling crowd, tumbling into the area occupied by the sea lions. The protagonist looks back; the dense sea lions have already covered the entire promenade, their wet skin reflecting a greasy sheen in the sunlight. Shot 4: Fisheye lens. The protagonist, barefoot, follows the crowd surging toward the iron gate at the scenic area exit; behind them in the distance, the black silhouettes of the sea lion colony advance slowly and neatly, occupying the entire resort area. The exit iron gate emits the sound of metal deformation under the pressure of the crowd. The vignette in the four corners of the frame intensifies, gradually contracting to complete black. The seaside wind and the low roar of the sea lion colony mix into a continuous rumble, the friction sound of hundreds of slippery bodies dragging across concrete, the chaotic footsteps and panting of the running crowd, the metal twisting sound of the iron gate being crushed. Shot on Arri Alexa, fisheye lens (Fisheye 12mm), the image shows obvious barrel distortion. Top and bottom black bars (2.39:1 Letterbox), forced widescreen cinematic aspect ratio. Heavy Vignette, the four corners of the frame are darkened and converge toward the center. High contrast under strong seaside sunlight, low-saturation cool tones. Photo-level realism. Faint digital noise and VHS-like distortion flicker at the edges of the frame; the image feels fragile, as if it could collapse and disintegrate at any moment. Dreamcore. Liminal space.
III. Login instructions
When the Worker starts, it automatically detects the login status of the Jimeng platform:
- First use / login expired: The browser will automatically open the Jimeng website and display a QR code, while playing a prompt sound (three long beeps + two short beeps) to alert the user. After the user scans the code with a mobile phone to log in, the program automatically detects it and continues working.
- Already logged in: Start working directly, no operation required.
- Timeout: Default wait is 10 minutes; the program exits after timeout.
You can also log in manually in advance:
cd "{W}/dream_to_video" && python main.py login
IV. Post-processing effect (Elliptic Shatter)
After the Worker finishes downloading each video, it automatically executes the elliptical shattered-edge effect post-processing, and finally outputs two files:
| File | Naming format | Description |
|---|---|---|
| Original version | task_XXX_YYYYMMDD_HHMMSS.mp4 |
The original video generated by Jimeng |
| Effect version | task_XXX_YYYYMMDD_HHMMSS_elliptic-shatter.mp4 |
Overlay the elliptical shattered-edge effect |
Effect description：
- The center of the video keeps a clear original image
- The edges present a shattered glass texture (scattered fragments, slight rotation, chromatic refraction)
- Outer frame with dark background + faint fragment texture
- Overall effect: black textured border + elliptical viewing window + rotating shattered particles
The effect script is located at effects/elliptic_shatter.py, uses OpenCV + NumPy for processing, and runs in a subprocess without blocking the main workflow.
V. Key notes
- The Prompt is one whole plain text segment, with no line breaks, Markdown formatting, or paragraphing. Pass it directly as a string to
python main.py add。 - The Worker only needs to be started once; it will keep running until all tasks are complete. Multiple assets can be added consecutively, and the Worker will queue and process them automatically.
- Each task outputs two videosto
{W}/dream_to_video/output/: the original version + the elliptic-shatter effect version. - After download + effect processing is complete, there is a prompt sound(3 short beeps); when the user hears the sound, they can go get the video.
- If the user provides multiple asset segments at once, generate an independent Prompt for each segment and add them separately.
- In the technical foundation, [Lighting] needs to be replaced according to the specific scene, such as "mixed cold light from artificial light sources and streetlights late at night", "high contrast under strong seaside sunlight", etc.