volcengine-tosutil

Volcano Engine tosutil Skill

This Skill targets the command-line tool for Volcano Engine Object Storage TOS, tosutil and is responsible for converting user intent into safe, executable, and verifiable tosutil operation processes.

Goal in one sentence

Convert the requirement of "what operation to perform on TOS" intopreview-only by defaultof tosutil commands, and when execution is needed, outputstructured JSON results + reproducible evidence + diagnostic suggestions。

Input and output

• Input: target command (such as ls/cp/rm/du/setmeta) + minimum necessary parameters (such as tos:// address, local path, recursive switch) + optional common parameters (endpoint/region/credentials/conf）。
• Output: unified JSON protocol (ok/code/message/data/ts), including preview.shell(masked reproducible command), execution summary, and on failure advice.code and next_actions。

Default behavior (reduce user cost + safety)

• By default, only a command preview is generated, without execution (explicit --run is required for execution).
• Destructive commands (such as rm) are not executed by default: explicit --yes(or in legacy mode --assume-yes) is required for actual execution.
• Output is masked by default: AK/SK/Token will not be echoed.

When to use

Invoke this Skill when the user requests any of the following scenarios:

• Use tosutil to initialize or update TOS configuration
• Create buckets, list buckets/objects, query object attributes
• Upload, download, copy, batch delete objects
• Calculate object capacity, set object metadata
• Analyze Http status [403], connectivity failures, command parameter errors, and other issues
• Generate execution plans for advanced commands such as share、set-acl、mount、probe、netdig and other advanced commands to generate an execution plan

Documentation fact baseline

Based on the tosutil documentation system, the following facts can be confirmed:

• tosutil is a command-line tool for accessing and managing Volcano Engine Object Storage TOS, suitable for batch data processing between local and TOS, script integration, and small-to-medium data migration.
• Core commands include ls、mkdir、du、mb、cp、setmeta、stat、rm、share、set-acl、mount。
• Auxiliary commands include config、help、probe、netdig、hash、fcp、clear、version、ping、connect、traceroute、curl。
• Initialization configuration supports three methods: permanent keys, STS temporary keys, and anonymous access.
• During initialization, use the TOS protocol domain name, not the S3 protocol domain name.
• rm has secondary confirmation by default; recursive and batch deletions must explicitly assess risks.
• du may take a long time with millions of objects; prioritize splitting calculations by directory.

Download and installation

tosutil supports Windows, Linux, and macOS. Before using this Skill, it is recommended to first download the corresponding version according to the current operating system and chip architecture, and complete execution permission settings.

Official download recommendations

• Linux amd64: supports directly downloading the binary and executing it
• macOS amd64 (Intel): supports directly downloading the binary and executing it
• macOS arm64 (Apple M series chips): supports directly downloading the binary and executing it
• Windows 64bit: download tosutil.exe
• The official website also provides the corresponding sha256 checksum file; it is recommended to perform an integrity check after downloading
• Current tosutil latest version is mainly applicable to Windows, macOS, and Linux amd systems

Installation commands

Linux：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/linux/amd64/tosutil
chmod a+x tosutil
sudo mv tosutil /usr/local/bin

macOS Intel：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/darwin/amd64/tosutil
chmod a+x tosutil
sudo mv tosutil /usr/local/bin

macOS Apple Silicon：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/darwin/arm64/tosutil
chmod a+x tosutil
sudo mv tosutil /usr/local/bin

Windows：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/windows/tosutil -O tosutil.exe

Installation notes

• macOS may by default block apps from unidentified developers; if on first execution tosutil is blocked by the system, you need to allow it in the system security settings
• After downloading on Linux / macOS, you usually need to run chmod a+x tosutil
• If you want to execute directly in any directory tosutil, it is recommended to move the binary to a directory already included in PATH , for example /usr/local/bin
• If the binary is not added to PATH, please call it using an absolute path, for example /absolute/path/to/tosutil version
• When this Skill is not added to PATH , it is recommended to explicitly specify the binary location via --tosutil-binary <absolute-path> to avoid the tool not being found

Post-installation verification

If already added to PATH：

tosutil version
tosutil config
tosutil ls

If not added to PATH：

/absolute/path/to/tosutil version
/absolute/path/to/tosutil config
/absolute/path/to/tosutil ls

Combined with this Skill's recommendations:

• First use version to verify whether the binary is executable
• Then use config to confirm the configuration file path
• Finally use ls to verify credentials, region, and network connectivity

If tosutil has been added to PATH, you can call this Skill like this (path relative to the skill root directory):

python3 scripts/main.py \
  ls \
  --preflight

If tosutil has not been added to PATH, you can call this Skill like this:

python3 scripts/main.py \
  ls \
  --tosutil-binary /absolute/path/to/tosutil \
  --preflight

Working principles

• First confirm whether the goal is a "read operation" or a "write operation/delete operation".
• Prefer generating the minimum viable command, avoiding concatenating too many dangerous parameters at once.
• For batch upload, download, and copy tasks, confirm the concurrency and part size thresholds before executing.
• For deletion-type tasks, provide pre-check steps and rollback hints by default; do not skip confirmation directly.
• When encountering advanced commands and the parameters are not fully confirmed, first use tosutil help <command> to verify; do not invent parameters.
• For non-interactive deletion use -f, not -y. When cleaning up deployment artifacts, prefer deleting the restricted prefix, for example tosutil rm tos://bucket/prefix/ -r -f, to avoid missing test artifacts or mistakenly deleting unrelated objects.

Standard process

1. Identify the scenario

Classify the user request into one of the following scenarios:

• Initialization configuration:config
• Bucket operations:ls、mb、stat
• Object transfer:cp
• Object deletion:rm
• Object metadata:setmeta
• Capacity statistics:du
• Fault diagnosis:version、ls、help、probe、netdig、ping、connect、traceroute

2. Pre-check

Perform or recommend the following checks:

tosutil version
tosutil config
tosutil ls

Check focus:

• Whether the tool is installed and executable
• Endpoint Whether it is a TOS protocol domain
• Region Whether it is consistent with the target bucket region
• AK/SK or STS Token Whether it exists and has sufficient permissions
• Whether Bucket number is:、Http status [403]、A connection attempt failed

3. Parameter normalization

Before generating the command, uniformly organize the following parameters:

• Resource address: local path,tos://bucket、tos://bucket/prefix
• Credential mode: permanent key, STS, anonymous access
• Common parameters:-e、-re、-i、-k、-t、-conf
• Bucket type:-bt=fns|hns
• Batch task concurrency:-j
• Part concurrency or part task control:-p、-threshold、-ps
• Result output directory:-o

4. Command generation

Automatically select the command mode according to the resource type:

• Local -> TOS: upload
• TOS -> Local: Download
• TOS -> TOS: Object Copy
• Single object: single-task mode
• Directory or prefix: recursive mode -r
• Large files: switch to multipart tasks based on the threshold

5. Output Validation

Parse the following signals in the execution result:

• Success indicator:successfully、Bucket number is:、Succeed count is:、Task id is:
• Permission issue:Http status [403]
• Network issue:A connection attempt failed
• Parameter issue: command help output, missing required parameters, invalid path format
• Cleanup suggestion: when resumable transfer fails, consider clear

Command Mapping

Initial Configuration

Permanent key (recommended):

tosutil config -i <ak> -k <sk> -e <endpoint> -re <region>

For endpoint and region, refer to "Appendix: Regions and Access Domains". Prefer the internal endpoint; if the internal endpoint is unavailable, use the public endpoint

STS：

tosutil config -i <ak> -k <sk> -t <token> -e <endpoint> -re <region>

Anonymous access:

tosutil config -i= -k= -t= -e <endpoint> -re <region>

Common Commands for Buckets and Objects

tosutil ls
tosutil mb tos://bucketname
tosutil cp /local/file.txt tos://bucketname/file.txt
tosutil cp tos://bucketname/file.txt /local/file.txt
tosutil rm tos://bucketname/file.txt
tosutil du tos://bucketname
tosutil setmeta tos://bucketname/object.png -meta aaa:bbb#ccc:ddd

Security Policy

• Before deleting an object, first determine whether it is a single object, directory prefix, or bucket-level deletion.
• For rm -r、rm -f, batch metadata updates, batch copies, and similar operations, first output a description of the impact scope.
• If the user only requests "generate a command", do not execute it directly by default.
• If a high-risk command needs to be executed, first suggest listing the target objects or performing a dryRun -style validation; if the command itself does not support dryRun, first perform a read-only check.

Integration Implementation Recommendations

This Skill is implemented primarily as a "local CLI wrapper layer" rather than "directly calling the TOS HTTP API", because the main documentation provides the tosutil command interface.

Script entry point (recommended subcommand mode)

Command preview (do not execute):

python3 scripts/main.py ls --cloud-url tos://bucketname

Execute and return structured results:

python3 scripts/main.py ls --cloud-url tos://bucketname --preflight --run

High-risk deletion (explicit confirmation required):

python3 scripts/main.py rm --cloud-url tos://bucketname/prefix/ --recursive --run --yes

Compatible with legacy entry point (legacy):

python3 scripts/main.py --command ls --cloud-url tos://bucketname

Recommended directory structure:

volcengine-tosutil/
├── SKILL.md
├── references/
│   └── doc-survey.md
└── scripts/
    ├── main.py
    ├── models.py
    ├── result_handler.py
    └── tosutil_service.py

Recommended core classes:

• CommonOptions / CredentialMode: encapsulates permanent keys, STS, anonymous mode, and common parameters
• TosResource: encapsulates the parsing results of local paths and tos:// resources
• CommandSpec: encapsulates the command name, parameter list, risk level, and whether it is a destructive operation
• TosutilRunner: uniformly executes the tosutil command and collects the exit code, standard output, and standard error
• result_handler: uniformly handles result parsing, error mapping, and structured output field organization
• tosutil_service: uniformly handles command construction, pre-checks, execution, masking, and common validation logic

Key Rules

Configuration Rules

• config writes to the local configuration file, which is located by default under the user directory at .tosutilconfig
• -conf can only point to an existing configuration file path and is not responsible for automatically creating a new file
• Prefer to recommend using an explicit region + endpoint, to avoid cross-region misoperations

Transfer Rules

• cp needs to automatically infer whether it is an upload, download, or cloud copy based on the source and destination address types
• The actual concurrency of batch tasks needs to be evaluated comprehensively with -j and the multipart concurrency parameters, to avoid blindly increasing them
• For large files and batch tasks, explicitly set the threshold, concurrency, and output directory first to facilitate troubleshooting

Statistics Rules

• du In massive object scenarios, do not perform a full-bucket recursive scan by default
• If bucket versioning is enabled, version-related parameters can be added as needed and the statistics scope can be split

Metadata rules

• setmeta Supports single-object and batch prefix modes
• Batch mode must specify -r, and limit the maximum concurrency -j

Troubleshooting tips

Permission issues

• If the result shows Http status [403], first check whether the AK/SK and STS have expired, as well as the target bucket/object permissions

Network issues

• If the result shows A connection attempt failed, first check the network, proxy, firewall, and Endpoint

Parameter issues

• If you are unsure about advanced command parameters, first run tosutil help <command>

Cleanup issues

• After resumable transfer or an abnormal interruption, consider clear cleaning up the record file and making a best effort to clean up the cloud

Best practices

• When taking over an environment for the first time, first run version、config、ls the three-step check
• Initialization must use the TOS protocol domain name
• Anonymous access is only applicable to public read or public write scenarios
• Split statistics for millions of objects by directory
• For batch tasks, set concurrency conservatively first, then tune it gradually
• Keep secondary confirmation and impact scope explanations for high-risk commands

Interaction templates

The user wants to upload a file

1. Confirm the local path, target bucket, and target object name
2. Determine whether recursive upload is needed
3. Check configuration and connectivity
4. Generate the cp command
5. Return success criteria and failure troubleshooting points

The user wants to delete a directory

1. First confirm the target prefix
2. First recommend ls to verify the impact scope
3. Then generate rm -r
4. If the user confirms forced deletion, then consider -f

The user wants to troubleshoot inability to access

1. Check version
2. Check config
3. Run ls
4. Based on 403, connection failures, and parameter errors, perform classified diagnosis

Testing requirements

• Use documentation examples as command construction test samples
• Use example output as parser snapshot samples
• For 403, network failures, empty configuration, anonymous access, recursive deletion, and other scenarios, perform unit tests
• For cp , perform integration tests separately for the three directions: upload/download/cloud-to-cloud copy
• For rm、setmeta、du , perform boundary tests on the batch parameters

Limitations

• This Skill prioritizes covering the most core parts of the documentation, such as config、ls、mb、cp、rm、du、setmeta、stat、help、version
• share、set-acl、mount、probe、netdig、curl and other advanced commands use an extended adaptation mode
• If the parameters of an advanced command are not fully expanded in the current reference materials, you must first confirm through help or supplementary documentation before executing

**Appendix: Regions and access domain names **

• Region: indicates the physical location of the TOS data center.
• Endpoint: indicates the access domain name for TOS external services.

Region Chinese name	Region ID	Endpoint (internal/external)	S3 Endpoint (internal/external)
North China 2 (Beijing)	cn-beijing	Internal: tos-cn-beijing.ivolces.comExternal: tos-cn-beijing.volces.com	Internal: tos-s3-cn-beijing.ivolces.comExternal: tos-s3-cn-beijing.volces.com
South China 1 (Guangzhou)	cn-guangzhou	Internal: tos-cn-guangzhou.ivolces.comExternal: tos-cn-guangzhou.volces.com	Internal: tos-s3-cn-guangzhou.ivolces.comExternal: tos-s3-cn-guangzhou.volces.com
East China 2 (Shanghai)	cn-shanghai	Internal: tos-cn-shanghai.ivolces.comExternal: tos-cn-shanghai.volces.com	Internal: tos-s3-cn-shanghai.ivolces.comExternal: tos-s3-cn-shanghai.volces.com
China Hong Kong	cn-hongkong	Internal: tos-cn-hongkong.ivolces.comExternal: tos-cn-hongkong.volces.com	Internal: tos-s3-cn-hongkong.ivolces.comExternal: tos-s3-cn-hongkong.volces.com
Asia Pacific Southeast (Johor)	ap-southeast-1	Internal: tos-ap-southeast-1.ivolces.comExternal: tos-ap-southeast-1.volces.com	Internal: tos-s3-ap-southeast-1.ivolces.comExternal: tos-s3-ap-southeast-1.volces.com
Asia Pacific Southeast (Jakarta)	ap-southeast-3	Internal: tos-ap-southeast-3.ivolces.comExternal: tos-ap-southeast-3.volces.com	Internal: tos-s3-ap-southeast-3.ivolces.comExternal: tos-s3-ap-southeast-3.volces.com