byted-las-long-video-understand

LAS Fine-Grained Video Understanding (las_long_video_understand）

Provides multi-dimensional, fine-grained structured video understanding based on large models. Supports global understanding of hour-level (up to 3h) videos, event and behavior recognition, video question answering, efficient summarization, and structured output.

Design Pattern

This skill mainly adopts:

Tool Wrapper: Encapsulates lasutil CLI calls
Pipeline: Contains a sequential workflow from Step 0 → Step N

Core API and Configuration

Operator ID: las_long_video_understand
API: Asynchronous (submit → poll）
Environment Variable: LAS_API_KEY (Required)
Supported Duration: Supports videos up to 3h (10G).

For detailed parameters and interface definitions, see references/api.md。

Gotchas

Cannot be precisely estimated: Billed by Token, greatly affected by video duration and complexity.
Key Security: If the chat box blocks keys, have the user create env.sh in the current directory and write export LAS_API_KEY="...", and the SDK will automatically read it.
Disclaimer: The final reply result must include: "The billing for this method is all estimated billing, which differs from the actual cost. The actual cost is subject to the bill generated by Volcano after running. For billing details, please refer to Volcengine LAS Pricing.", and it is prohibited to use the words "actual cost" to describe the estimated price.

Workflow (Execute Strictly by Steps)

Copy this checklist and track progress:

执行进度：
- [ ] Step 0: 前置检查
- [ ] Step 1: 初始化与准备
- [ ] Step 2: 预估价格
- [ ] Step 3: 提交异步任务
- [ ] Step 4: 轮询任务状态
- [ ] Step 5: 结果呈现

Step 0: Pre-checks (⚠️ Must be completed in the first round of dialogue)

After accepting the user's task,do not start executing immediately, and you must first perform the following environment checks:

Check LAS_API_KEY and LAS_REGION: Confirm whether the environment variable or .env is configured.
- If not, you must immediately ask the user for it (hint:LAS_REGION is commonly cn-beijing）。
- Note：LAS_REGION must be completely consistent with the region where your API Key and TOS Bucket are located. If the user switches Region midway, you must remind the user that their TOS Bucket also needs to be changed accordingly, otherwise it will cause permission exceptions or upload failures.
Check Input Path：
- If what the user requests to process is alocal file, it needs to first be uploaded to TOS via the File API (only LAS_API_KEY, no additional TOS credentials are needed).
- If the operator'soutput resultis stored on TOS and the user needs to download it back locally, then VOLCENGINE_ACCESS_KEY and VOLCENGINE_SECRET_KEYare needed. Forscenarios where only the input file needs to be uploaded, TOS credentialsare no longer required。
After confirmation is correct: only then can you proceed to the next step.

Step 1: Initialization and Preparation

Environment Initialization (Mandatory for Agent)：

# 执行统一的环境初始化与更新脚本（会自动创建/激活虚拟环境，并检查更新）
source "$(dirname "$0")/scripts/env_init.sh" las_long_video_understand
workdir=$LAS_WORKDIR

If a network issue causes the update to fail, the script will skip the check and continue executing using the locally installed SDK.

When processing local files: first check the format and duration locally, estimate the price, and upload only after the user confirms:

# 提前检查视频格式（避免参数错误）
./scripts/check_format.sh <local_path>
# 本地使用 ffprobe 获取时长（无需上传即可预估价格）
duration_sec=$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:noprint_section=1 <local_path>)

Calculate the estimated price and wait for user confirmation, then perform the upload:

# 用户确认后，上传到 TOS
lasutil file-upload <local_path>

After a successful upload, JSON is returned; take the tos_uri(format tos://bucket/key) is passed to the operator as the input path.

Step 2: Estimate the price (⚠️ user confirmation is required)

This skill is billed by tokens. Because the video is long and the analysis is complex, the cost cannot be accurately estimated before submission.

Refer to the description of references/prices.md .
Note that the number of tokens generated by this task may be very large (especially for long videos).
Inform the user of the billing unit price and forcibly pause execution, explicitly waiting for the user to reply with confirmation. Before the user explicitly replies with an approval instruction such as "continue" or "confirm",it is absolutely forbiddento proceed to the next step (execute/submit the task). Tip: The estimate is for reference only; the actual amount is subject to the Volcengine bill. For billing instructions, please refer to Volcengine LAS pricing。

Step 3: Submit the asynchronous task (Submit)

Construct data.json：

{
  "video_url": "<url>",
  "query": "请总结这个视频的主要内容",
  "fps": 1.0,
  "model_name": "doubao-seed-2-0-lite-260215"
}

Execute the command:

data=$(cat "$workdir/data.json")
lasutil submit las_long_video_understand "$data" > "$workdir/submit.json"
task_id=$(cat "$workdir/submit.json" | jq -r '.metadata.task_id')
echo "Task ID: $task_id"

Step 4: Poll the task status (Poll)

# 获取任务状态
lasutil poll las_long_video_understand "$task_id" > "$workdir/poll.json"
cat "$workdir/poll.json" | jq -r '.metadata.task_status'

If the status is PENDING or RUNNING, wait for a period of time and then execute the above command again. Because this is long-video understanding, it may take a long time.
If the status is COMPLETED, continue to Step 5.
If the status is FAILED, report the error to the user error_msg。

Step 4: Asynchronous query (Poll)

⚠️ Constraints for asynchronous tasks and background polling：

If the environment supports background tasks, you can use the optimized background polling script to automatically poll until completion:
```
mkdir -p "./output/${task_id}"
./scripts/poll_background.sh ${task_id} "./output/${task_id}" & disown
```
Script features:
- Dynamic interval: the first 5 times are 30s, times 5-10 are 60s, and after 10 times it is 120s
- Completion marker: generate the COMPLETED marker file
- Suitable for tasks that take a long time, such as long-video understanding
If the environment does not support background tasks, poll manually:
```
# 获取任务状态
lasutil poll las_long_video_understand "$task_id" > "$workdir/poll.json"
cat "$workdir/poll.json" | jq -r '.metadata.task_status'
```
- If the status is PENDING or RUNNING, wait for a period of time and then execute the above command again. Because this is long-video understanding, it may take a long time.
- If the status is COMPLETED, continue to Step 5.
- If the status is FAILED, report the error to the user error_msg。

Step 5: Present the results

Process the results：

Use the script to automatically generate the result presentation (automatically includes the billing statement):

./scripts/generate_result.md.sh ${task_id} "./output/${task_id}" <estimated_price>

Manual processing：

# 解析最终摘要
cat "$workdir/poll.json" | jq -r '.data.final_summary'

# 可选：保存 clips 详情到本地
cat "$workdir/poll.json" | jq '.data.clips' > "./output/${task_id}/clips_detail.json"

Show the user：

Use the generated markdown to present
Present the video's final_summary(final summary)
If there are highlight Q&As or answers to specific queries, combine with clips to extract and present them
Automatically includes the billing statement ✅