ssh-dev-suite/long-task

ssh-long-task

Overview

Manage remote tasks expected to take >2 minutes, supporting agent rest and cross-session resume through the checkpoint mechanism.

Applicable scenarios

Model training (>10 minutes)
Large-scale data processing
Compilation and build tasks
Database migration

Process steps

1. Start the task and write a checkpoint

Usageconnect/scripts/ssh-job.sh startStart the task, and write the checkpoint immediately after obtaining the job_id:

bash connect/scripts/ssh-job.sh start <profile> "<command>"
# 输出: 任务已启动: job-20260306-143022-1234

bash long-task/scripts/checkpoint.sh write <profile> <job_id> \
  --task "模型训练 epoch 50" \
  --duration "~2h" \
  --next "检查训练完成,下载模型权重,运行验证" \
  --context "lr=0.001,batch=32"

Output: checkpoint file ~/.ssh-jobs/<job_id>/checkpoint.md

2. Inform the user and rest

Agent output:

✓ 任务已启动: job-20260306-143022-1234
✓ Checkpoint已保存，预计耗时 ~2h

你可以：
1. 稍后输入"检查任务 <job_id>"查看状态
2. 关闭会话，任务继续运行
3. 新会话中我会自动读取checkpoint恢复上下文

3. Read the checkpoint on resume

In a new session or when the user requests a check:

bash long-task/scripts/checkpoint.sh read <profile> <job_id>

Output: parse the checkpoint content (task, started, duration, next_steps, context)

4. Intelligent status checking

Determine the checking strategy based on the expected duration:

bash connect/scripts/ssh-job.sh status <profile> <job_id>

<5 minutes: check every 30 seconds
5-30 minutes: check every 2 minutes
>30 minutes: check every 10 minutes, or manually triggered by the user

5. Output checking (Token optimization)

Check the size first:

bash connect/scripts/ssh-job.sh status <profile> <job_id>
# 输出: stdout: 245KB, stderr: 1.2KB

Decide based on size:

<10KB: read the whole content bash connect/scripts/ssh-job.sh output <profile> <job_id>
10KB-100KB: read only the tail bash connect/scripts/ssh-job.sh output <profile> <job_id> --tail 50
>100KB: search for errors first bash connect/scripts/ssh-job.sh output <profile> <job_id> --grep 'error|fail|exception'

6. Execute next_steps

After the task is complete, continue according to the next_steps in the checkpoint: Execute step by step, and after each step is completed, update or delete the checkpoint.

Checkpoint format

Stored in ~/.ssh-jobs/<job_id>/checkpoint.md:

job_id: job-20260306-143022-1234
profile: dev-server
task: "模型训练 epoch 50"
started: 2026-03-06T14:30:22
expected_duration: ~2h
next_steps:
  - 检查训练完成状态
  - 下载模型权重到本地
  - 运行验证脚本
context: "learning_rate=0.001, batch_size=32"

Token constraints

Output checking: check the size first, and read in tiers by size (whole/tail/grep)
Status polling: adjust the frequency based on the expected duration to avoid frequent checks
Checkpoint content: streamline key information, do not store complete logs