hellomedia

HelloMedia

Standalone, complete multimodal Skill:Image recognition / image generation / image editing · video reading / video generation / video editing · audio reading / audio generation。
The script defaults to pure standard library, zero hard dependencies(Pillow optional, used before image recognition forgentlelarge-image compression, defaults to favoring clarity).

Relationship with Grok Build built-in tools

In a Grok Build session, image_to_video / reference_to_video is a host built-in tool(injected at runtime).
Its underlying product is xAI Imagine REST. This skill provides a cross-host usable REST equivalent implementation:

Grok Build host tool	This skill's CLI
`image_to_video`	`python scripts/video.py --mode image_to_video --image ...`
`reference_to_video`	`python scripts/video.py --mode reference_to_video --reference ...`
(no pure T2V host tool)	`python scripts/video.py --mode text_to_video` or first `generate.py` then I2V

by default always go through this skill(assuming the host main model has no vision). Only when the user explicitly requests to use Grok Build built-in image_to_video / reference_to_video / image_gen and other host tools, or when host tools are clearly more suitable, may you switch to host.
If the relay returns 403 not enabled for this group = the account has not enabled Imagine, not a script contract error.

Sub2API / relay(e.g. base_url not api.x.ai but api_format: xai）：

image generation goes through POST /v1/images/generations, payload uses aspect_ratio + resolution(1k/2k), rather than OpenAI's size/quality prioritized
image-to-video image object field defaults to image_url(official api.x.ai still uses url); can be overridden with the channel field video_image_url_field override
will notbecause the local machine cannot access api.x.ai block relay video generation (official CDN precheck only applies to api.x.ai）
media downloads use a browser-style User-Agent(can be overridden with HELLOMEDIA_USER_AGENT override), to avoid imgen/vidgen CDN connection refusals
Reference implementation alignment:happy-loki/grok-media-skill

Routing

Default premise: the host main model cannot view images / cannot reliably handle multimodal.
When containing media or generation-type requests, directly use this skill, do not first try the main model and then fail, and do not merely say "cannot see / cannot hear".

消息含 图片/截图/照片（含粘贴、路径、附件）
  → 直接本技能识图（勿先试主模型）
  → 有磁盘路径：
       python scripts/vision.py --image <path> --prompt "..."
  → 用户粘贴进 CLI 但消息里只有 [Image #N] / 无路径：
       1) 请用户「重新复制」该图片到系统剪贴板（截图/Copy Image，不是只复制路径文字）
       2) python scripts/vision.py --from-clipboard --prompt "..."
       3) 若返回 clipboard_empty / text_only：引导重拷或保存为文件再 --image
  → 需要无损像素：加 --no-compress
  → 也可：scripts/understand.py --image / --from-clipboard

消息含 视频 / 音频
  → scripts/understand.py --video / --audio（或 audio.py stt）

要求 生图 / 改图     → scripts/generate.py
要求 单图生视频      → scripts/video.py --mode image_to_video
要求 多参考图生视频  → scripts/video.py --mode reference_to_video
要求 纯文生视频      → scripts/video.py --mode text_to_video
                     或：generate.py 出首帧 → image_to_video
要求 改视频 / 延长   → scripts/video.py --mode edit|extend
要求 TTS / STT       → scripts/audio.py
配置排障             → scripts/doctor.py

Capability	Script
Image understanding	`scripts/vision.py` or `scripts/understand.py --image`
Video/audio understanding	`scripts/understand.py --video` / `--audio`
Image generation and editing	`scripts/generate.py`
Video generation/editing/extension	`scripts/video.py`
Speech synthesis / transcription	`scripts/audio.py tts` / `stt`
Connectivity diagnostics	`scripts/doctor.py`
Runtime credential discovery	`scripts/_auth_discovery.py`(invoked by generate)

Config (config.json)

{
  "channels": [
    {
      "name": "xAI Grok (full multimodal)",
      "base_url": "https://api.x.ai",
      "api_key": "xai-your-key",
      "model": "grok-4.5",
      "image_model": "grok-imagine-image",
      "video_model": "grok-imagine-video",
      "tts_voice": "eve",
      "api_format": "xai",
      "vision": true,
      "generate": true,
      "video": true,
      "video_edit": true,
      "video_extend": true,
      "audio": true,
      "priority": 1
    }
  ],
  "defaults": {
    "max_tokens": 4096,
    "timeout_seconds": 300,
    "retry_count": 2,
    "video_poll_timeout": 600,
    "max_resolution": "2k",
    "cooldown_seconds": 2.5
  }
}

Field	Description
`model`	Vision/understanding model
`image_model`	Image generation model (falls back if empty `model`）
`video_model`	Video model
`image_api_key` / `image_base_url`	Image-generation dedicated credentials (optional)
`video_api_key` / `video_base_url`	Video dedicated credentials (optional)
`audio_api_key` / `audio_base_url`	Speech dedicated credentials (optional)
`tts_voice`	Default TTS voice
`api_format`	`openai` / `anthropic` / `xai` / `sd-webui` / `fal`(when generating images `sd-webui`/`fal` will automatically map the endpoint)
`wire_api`	Optional:`responses` image generation preferentially goes through `/v1/responses`
`requires_openai_auth`	Optional: set true when using a local OpenAI-auth proxy
`vision` / `generate` / `video` / `audio`	Enable/disable capabilities by channel
`video_edit` / `video_extend`	Video edit/extend (default true; set false to disable)
`priority`	Smaller is higher priority
`defaults.max_resolution`	Image generation resolution ceiling (e.g. `2k`）
`defaults.cooldown_seconds`	Image generation request interval
`defaults.video_poll_timeout`	Maximum video polling seconds (CLI `--poll-timeout` takes effect when unspecified)

See template at config.example.json。

Image generation credential resolution order

CLI：--base-url / --api-key / --model / --provider …
Environment variables:OPENAI_API_KEY、OPENAI_BASE_URL、OPENAI_IMAGE_MODEL …
Skill config.json in generate: true channels
Runtime discovery:~/.codex, Hermes, OpenClaw (can use --no-runtime-auth to disable)

When there is no generate channel at all, image generation is still possible as long as there are runtime credentials such as CLI/env/Codex.

Workflow

Windows paths: backslashes are escaped in bash. Use forward slashes or single quotes:'C:/Users/xxx/img.png'。

Image understanding

python scripts/vision.py --image ./screenshot.png --prompt "描述图片内容"
python scripts/vision.py --image ./a.png --image ./b.png --prompt "对比两图"
python scripts/vision.py --image-dir ./pages/ --prompt "批量分析"
python scripts/vision.py --from-clipboard --prompt "描述剪贴板中的截图"
python scripts/vision.py --image ./big.png --prompt "..." --no-compress
python scripts/understand.py --image ./shot.png --prompt "提取 UI 文案"
python scripts/understand.py --from-clipboard --prompt "提取 UI 文案"

Pasted image instructions: after the host pastes an image into the input box, the skillcannotdirectly read [Image #N] attachment token.
The closed-loop method is --from-clipboard(read current OS clipboard) or --image <磁盘路径>。
If the host has cleared the clipboard when pasting, the user needs to copy the image again. Pasting into the CLI in Windows terminals commonly uses Alt+V（Claude/Grok）。

Video / audio understanding

python scripts/understand.py --video ./clip.mp4 --prompt "总结镜头与台词"
python scripts/understand.py --audio ./meeting.mp3 --prompt "提炼待办事项"
python scripts/audio.py stt --audio ./meeting.mp3 --format-text

Image generation / editing

# config.json 渠道
python scripts/generate.py --prompt "施工质量评分雷达图" --output ./output/chart.png
python scripts/generate.py --prompt "oil painting" --image ./sketch.png --output ./output/painting.png
python scripts/generate.py --prompt "fantasy concept" --count 3 --output ./output/monster.png
python scripts/generate.py --prompt-file ./prompts/heroine.txt --output ./output/heroine.png
python scripts/generate.py --thinking medium --prompt "complex composite" --output ./output/cmp.png

# Codex provider / 显式凭据 / 本地代理
python scripts/generate.py --provider fluxcode --prompt "..." --output ./output/image.png
python scripts/generate.py --base-url https://api.openai.com --api-key sk-... --prompt "..." --output ./output/oai.png
python scripts/generate.py --endpoint-mode responses --prompt "..." --output ./output/r.png

# SD WebUI (A1111)
python scripts/generate.py --endpoint-mode sd-webui --base-url http://localhost:7860 --prompt "anime girl" --output ./output/sd.png

python scripts/generate.py --prompt "test" --dry-run

Video generation

Default aligns with Build:duration=6、resolution=480p。
--mode auto: with --image → I2V; with --reference → reference; prompt only → T2V.

python scripts/video.py --mode image_to_video --image ./still.png \
  --prompt "水面落下，镜头缓缓拉远" --duration 6 --resolution 480p --output ./output/water.mp4

python scripts/video.py --mode reference_to_video \
  --reference ./char.png --reference ./outfit.png \
  --prompt "模特从 <IMAGE_1> 走上秀场，穿着 <IMAGE_2>" --duration 10 --aspect-ratio 16:9 --output ./output/run.mp4

python scripts/video.py --mode text_to_video --prompt "火星升起的水晶火箭" --duration 6 --output ./output/rocket.mp4

python scripts/generate.py --prompt "静帧：水晶火箭在火星沙丘" --output ./output/frame.png
python scripts/video.py --mode image_to_video --image ./output/frame.png --prompt "缓缓升空" --output ./output/launch.mp4

python scripts/video.py --mode edit --video ./src.mp4 --prompt "给人物加上红色外套" --output ./output/edit.mp4
python scripts/video.py --mode extend --video ./src.mp4 --prompt "镜头转向群山" --output ./output/ext.mp4
python scripts/video.py --prompt "test" --dry-run

Audio (TTS / STT)

python scripts/audio.py tts --text "你好，欢迎使用 HelloMedia" --language zh --voice eve --output ./output/hello.mp3
python scripts/audio.py stt --audio ./voice.mp3 --format-text --language en
python scripts/audio.py voices
python scripts/audio.py tts --text "test" --dry-run

Diagnostics and capability table

python scripts/doctor.py --dry-run
python scripts/doctor.py --capabilities
python scripts/doctor.py --xai-network
python scripts/doctor.py --vision-only
python scripts/doctor.py --video-only
python scripts/doctor.py --audio-only
python scripts/doctor.py --clipboard
python scripts/doctor.py --clipboard --clipboard-capture

Video download resume (without regenerating)

python scripts/video.py --recover-url "https://.../video.mp4" --output ./output/recovered.mp4

Understanding type defaults to stdout. Generation type writes to ./output/, writing to user directories such as the desktop is prohibited.

Rules

Skill first (default): assume the host's main model has no vision; image recognition/video reading/audio reading/image generation/video generation/TTS/STT go directly through this skill, do not try the main model's image recognition first
Paste/clipboard：vision.py --from-clipboard / understand.py --from-clipboard via scripts/_clipboard.py is written to disk at .runtime/clipboard/ then image recognition; an empty clipboard returns a recoverable error code (clipboard_empty etc.).cannotparse the host's internal [Image #N] token
vision.py: try channels by priority; on 429/5xx/timeout follow retry_count retry; permanent 4xx switches to the next channel
vision.py large image compression (enabled by default, favoring clarity): optional Pillow; only when the file is large and (exceeds long edge or is extremely large) does it resize/re-encode; by default the longest edge is 2048, JPEG q=90, and files smaller than 256KB are sent as-is;--no-compress disable
generate.py：
- Endpoints:responses / images / images/edits / chat / sd-webui / fal；api_format can be automatically mapped
- Multi-channel cascade: after the current generate channel fails, follow priority to try the next channel
- Semantic canvas ratio, adaptive timeout, cooldown, thinking/seed
- Codex/Hermes/OpenClaw credential discovery;wire_api with local auth proxy affects endpoint order and request headers
- Secure disk write:./output/ or skill runtime; reject Desktop/Downloads etc.
video.py：I2V / reference / text / edit / extend（video_edit/video_extend channel flag can be disabled); async polling; validate aspect ratio/duration/resolution before POST
Download recovery: when generation succeeded but local download failed, the result contains video_url/urls + download_error; use video.py --recover-url <URL> --output ... GET only to re-download; re-POST generation is forbidden
audio.py: xAI TTS/STT, can fall back to OpenAI-compatible /v1/audio/*
proxy: automatically read HELLOMEDIA_PROXY and standard HTTP(S)_PROXY / system proxy; must not write proxy URL/password into result JSON
Diagnostics：doctor.py --capabilities keyless capability table;doctor.py --xai-network probe xAI CDN
This skill does not do online search; use the host web tool or a separate search skill

Creation narrowing (optional, not mandatory)

Only when simultaneously are satisfied, enable round-by-round clarification (1–3 questions per round, can use 1A 2B quick answers):

The task is generation-type image/video; and
the description is vague (at least two of subject/style/purpose missing); and
the user has not provided a complete usable prompt,--prompt-file, or has not said "generate directly / use defaults"

Form a brief and confirm it before calling generate.py / video.py。
Skip the questionnaire: complete prompt, batch script, user explicitly requests direct generation.

Finished-product display contract

When the command succeeds and there is a local file:

Confirm the file exists and size > 0
Use the absolute pathreturned by the script (forward slashes, e.g. C:/proj/output/a.png）
For each finished product output a Markdown media tag:![desc](C:/abs/path.png), do not just print the filename
Display multiple results one by one; when the path contains spaces use ![desc](<C:/My Project/a.png>)
If there is only a URL and download_error: display the URL, and indicate that --recover-url can be used to recover

Environment variables (optional)

Variable	Default	Meaning
`HELLOMEDIA_COMPRESS_MIN_BYTES`	262144 (256KB)	if not exceeded, send as-is and do not attempt compression
`HELLOMEDIA_COMPRESS_MAX_SIDE`	2048	only scale down the long edge (LANCZOS) if it exceeds
`HELLOMEDIA_COMPRESS_JPEG_QUALITY`	90	JPEG quality when re-encoding is needed (leaning clear)
`HELLOMEDIA_COMPRESS_REENCODE_MIN_BYTES`	2097152 (2MB)	only allow re-encoding when the long edge is not exceeded but the size is ≥ this value
`HELLOMEDIA_CLIPBOARD_MAX_BYTES`	41943040 (40MB)	Clipboard capture size limit
`HELLOMEDIA_PROXY`	—	HTTP(S) proxy (used for both http/https)
`HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY`	—	Standard proxy environment variables
`OPENAI_API_KEY` / `GPT_API_KEY`	—	Image-generation API Key
`OPENAI_BASE_URL` / `GPT_BASE_URL`	—	Image-generation Base URL
`OPENAI_IMAGE_MODEL`	—	Image-generation model
`HELLOMEDIA_CLIENT_VERSION` / `HELLOMEDIA_ORIGINATOR`	—	Codex account flow attribution header