SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsvolcengine-tosutil

volcengine-tosutil

Deployment & operations

View repository

Updated 10-19 16:18

About this skill

Volcengine tosutil Skill

This Skill targets the command-line tool of Volcengine Object Storage TOS tosutil and is responsible for converting user intent into a safe, executable, and verifiable tosutil operation workflow.

Goal in one sentence

Convert the requirement of "what operation to perform on TOS" intopreview by defaultof tosutil commands, and when execution is needed, outputstructured JSON results + reproducible evidence + diagnosable suggestions。

Inputs and outputs

  • Input: target command (e.g., ls/cp/rm/du/setmeta) + minimum necessary parameters (e.g., tos:// address, local path, recursive switch) + optional common parameters (endpoint/region/credentials/conf）。
  • Output: unified JSON protocol (ok/code/message/data/ts), including preview.shell(masked reproducible command), execution summary, and on failure the advice.code and next_actions。

Default behavior (reduce user cost + safety)

  • By default, only generate a command preview, do not execute (explicit --run is required to execute).
  • Destructive commands (e.g., rm) are not executed by default: explicit --yesis required (or --assume-yesin legacy mode) for them to actually run.
  • Output is masked by default: AK/SK/Token will not be echoed.

When to use

Invoke this Skill when the user requests any of the following scenarios:

  • Use tosutil to initialize or update the TOS configuration
  • Create buckets, list buckets/objects, query object attributes
  • Upload, download, copy, batch delete objects
  • Calculate object capacity, set object metadata
  • Analyze Http status [403], connectivity failures, command parameter errors, and other issues
  • For share、set-acl、mount、probe、netdig and other advanced commands, generate execution plans

Documentation fact baseline

Based on the tosutil documentation system, the following facts can be confirmed:

  • tosutil is a command-line tool for accessing and managing Volcengine Object Storage TOS, suitable for batch data processing between local and TOS, script integration, and small-to-medium data migration.
  • Core commands include ls、mkdir、du、mb、cp、setmeta、stat、rm、share、set-acl、mount。
  • Auxiliary commands include config、help、probe、netdig、hash、fcp、clear、version、ping、connect、traceroute、curl。
  • Initialization configuration supports three methods: permanent keys, STS temporary keys, and anonymous access.
  • During initialization, the TOS protocol domain should be used, not the S3 protocol domain.
  • rm has a secondary confirmation by default; recursive and batch deletion must be explicitly assessed for risk.
  • du may take a long time with millions of objects; prioritize splitting calculations by directory.

Download and installation

tosutil supports Windows, Linux, and macOS. Before using this Skill, it is recommended to first download the corresponding version according to the current operating system and chip architecture, and complete the execution permission settings.

Official download recommendations

  • Linux amd64: supports direct download of the binary and execution
  • macOS amd64 (Intel): supports direct download of the binary and execution
  • macOS arm64 (Apple M series chips): supports direct download of the binary and execution
  • Windows 64bit: download tosutil.exe
  • The official site also provides the corresponding sha256 checksum file; it is recommended to perform an integrity check after download
  • Currently, the latest version of tosutil is mainly applicable to Windows, macOS, and Linux amd systems

Installation commands

Linux：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/linux/amd64/tosutil
chmod a+x tosutil
sudo mv tosutil /usr/local/bin

macOS Intel：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/darwin/amd64/tosutil
chmod a+x tosutil
sudo mv tosutil /usr/local/bin

macOS Apple Silicon：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/darwin/arm64/tosutil
chmod a+x tosutil
sudo mv tosutil /usr/local/bin

Windows：

wget https://m645b3e1bb36e-mrap.mrap.accesspoint.tos-global.volces.com/windows/tosutil -O tosutil.exe

Installation notes

  • macOS may by default block apps from unidentified developers; if on first execution tosutil is blocked by the system, you need to allow it in the system security settings
  • After downloading on Linux / macOS, you usually need to run chmod a+x tosutil
  • If you want to execute tosutildirectly from any directory, it is recommended to move the binary to a directory already included in PATH , for example /usr/local/bin
  • If the binary has not been added to PATH, please invoke it using an absolute path, for example /absolute/path/to/tosutil version
  • In scenarios where this Skill is not added to PATH , it is recommended to explicitly specify the binary location via --tosutil-binary <absolute-path> to avoid the tool not being found

Post-installation verification

If added to PATH：

tosutil version
tosutil config
tosutil ls

If not added to PATH：

/absolute/path/to/tosutil version
/absolute/path/to/tosutil config
/absolute/path/to/tosutil ls

In line with this Skill's recommendations:

  • First use version to verify whether the binary is executable
  • Then use config to confirm the configuration file path
  • Finally use ls to verify credentials, region, and network connectivity

If tosutil has been added to PATH, you can invoke this Skill as follows (path relative to the skill root directory):

python3 scripts/main.py \
  ls \
  --preflight

If tosutil has not been added to PATH, you can invoke this Skill as follows:

python3 scripts/main.py \
  ls \
  --tosutil-binary /absolute/path/to/tosutil \
  --preflight

Working principles

  • First confirm whether the target is a "read operation" or a "write operation/delete operation".
  • Prefer generating the minimal viable command, and avoid concatenating too many dangerous parameters at once.
  • For batch upload, download, and copy tasks, confirm concurrency and multipart thresholds before executing.
  • For deletion-type tasks, provide pre-check steps and rollback hints by default; do not skip confirmation directly.
  • When encountering advanced commands with parameters not fully confirmed, prefer validating with tosutil help <command> rather than fabricating parameters.
  • Use -ffor non-interactive deletion, not -y. When cleaning up deployment artifacts, prefer deleting a restricted prefix, for example tosutil rm tos://bucket/prefix/ -r -f, to avoid missing test artifacts or mistakenly deleting unrelated objects.

Standard workflow

1. Identify the scenario

Classify the user request into one of the following scenarios:

  • Initial configuration:config
  • Bucket operations:ls、mb、stat
  • Object transfer:cp
  • Object deletion:rm
  • Object metadata:setmeta
  • Capacity statistics:du
  • Troubleshooting:version、ls、help、probe、netdig、ping、connect、traceroute

2. Pre-check

Perform or recommend the following checks:

tosutil version
tosutil config
tosutil ls

Key checks:

  • Whether the tool is installed and executable
  • Endpoint whether it is a TOS protocol domain
  • Region whether it is consistent with the target bucket region
  • AK/SK or STS Token whether it exists and has sufficient permissions
  • Whether Bucket number is:、Http status [403]、A connection attempt failed

3. Parameter normalization

Before generating commands, uniformly organize the following parameters:

  • Resource addresses: local paths,tos://bucket、tos://bucket/prefix
  • Credential mode: permanent key, STS, anonymous access
  • Common parameters:-e、-re、-i、-k、-t、-conf
  • Bucket type:-bt=fns|hns
  • Batch task concurrency:-j
  • Multipart concurrency or multipart task control:-p、-threshold、-ps
  • Result output directory:-o

4. Command generation

Automatically select the command mode based on the resource type:

  • Local -> TOS: upload
  • TOS -> Local: download
  • TOS -> TOS: object copy
  • Single object: single-task mode
  • Directory or prefix: recursive mode -r
  • Large files: switch to multipart tasks based on threshold

5. Output validation

Parse the following signals in the execution result:

  • Success indicator:successfully、Bucket number is:、Succeed count is:、Task id is:
  • Permission issue:Http status [403]
  • Network issue:A connection attempt failed
  • Parameter issue: command help output, missing required parameters, invalid path format
  • Cleanup suggestion: when resumable transfer fails, consider clear

Command mapping

Initialization configuration

Permanent key (recommended):

tosutil config -i <ak> -k <sk> -e <endpoint> -re <region>

For endpoint and region, refer to "Appendix: Regions and Access Domains"; prefer the internal endpoint; if the internal endpoint is unavailable, use the public endpoint

STS：

tosutil config -i <ak> -k <sk> -t <token> -e <endpoint> -re <region>

Anonymous access:

tosutil config -i= -k= -t= -e <endpoint> -re <region>

Common bucket and object commands

tosutil ls
tosutil mb tos://bucketname
tosutil cp /local/file.txt tos://bucketname/file.txt
tosutil cp tos://bucketname/file.txt /local/file.txt
tosutil rm tos://bucketname/file.txt
tosutil du tos://bucketname
tosutil setmeta tos://bucketname/object.png -meta aaa:bbb#ccc:ddd

Security policy

  • Before deleting an object, first determine whether it is a single object, directory prefix, or bucket-level deletion.
  • For rm -r、rm -f, batch metadata updates, batch copies, and similar operations, first output a description of the impact scope.
  • If the user only asks to "generate a command", by default do not execute it directly.
  • If a high-risk command needs to be executed, first suggest listing the target objects or performing a dryRun -style check; if the command itself does not support dryRun, perform a read-only check first.

Integration implementation suggestions

This Skill is implemented primarily as a "local CLI wrapper layer" rather than by "directly calling the TOS HTTP API", because the main body of the documentation provides the tosutil command interface.

Script entry point (recommended subcommand mode)

Command preview (do not execute):

python3 scripts/main.py ls --cloud-url tos://bucketname

Execute and return structured results:

python3 scripts/main.py ls --cloud-url tos://bucketname --preflight --run

High-risk deletion (explicit confirmation required):

python3 scripts/main.py rm --cloud-url tos://bucketname/prefix/ --recursive --run --yes

Legacy entry point compatibility (legacy):

python3 scripts/main.py --command ls --cloud-url tos://bucketname

Recommended directory structure:

volcengine-tosutil/
├── SKILL.md
├── references/
│   └── doc-survey.md
└── scripts/
    ├── main.py
    ├── models.py
    ├── result_handler.py
    └── tosutil_service.py

Recommended core classes:

  • CommonOptions / CredentialMode: encapsulates permanent keys, STS, anonymous mode, and common parameters
  • TosResource: encapsulates the parsing results of local paths and tos:// resources
  • CommandSpec: encapsulates the command name, parameter list, risk level, and whether it is a destructive operation
  • TosutilRunner: uniformly executes tosutil commands and collects exit codes, standard output, and standard error
  • result_handler: uniformly responsible for result parsing, error mapping, and organizing structured output fields
  • tosutil_service: uniformly responsible for command construction, pre-checks, execution, masking, and common validation logic

Key rules

Configuration rules

  • config writes to the local configuration file, which by default is located in the user directory under .tosutilconfig
  • -conf can only point to an existing configuration file path; it is not responsible for automatically creating new files
  • Prefer explicitly using region + endpointto avoid cross-region misoperations

Transfer rules

  • cp needs to automatically infer whether it is an upload, download, or cloud-side copy based on the source and destination address types
  • The actual concurrency of batch tasks must be evaluated comprehensively with -j and the multipart concurrency parameters to avoid blindly increasing them
  • For large files and batch tasks, explicitly set the threshold, concurrency, and output directory first to facilitate troubleshooting

Statistics rules

  • du In scenarios with massive numbers of objects, do not default to a full-bucket recursive scan
  • If the bucket has versioning enabled, add version-related parameters as needed and split the statistics scope

Metadata rules

  • setmeta Supports single-object and batch prefix modes
  • Batch mode must explicitly specify -r, and limit the maximum concurrency -j

Troubleshooting tips

Permission issues

  • If the result shows Http status [403], first check whether the AK/SK or STS has expired, and the permissions on the target bucket/object

Network issues

  • If the result shows A connection attempt failed, first check the network, proxy, firewall and Endpoint

Parameter issues

  • If unsure about advanced command parameters, first run tosutil help <command>

Cleanup issues

  • After a resumable transfer or abnormal interruption, consider clear to clean up the record files and make a best-effort cloud cleanup

Best practices

  • When taking over an environment for the first time, first run the version、config、ls triple check
  • Initialization must use a TOS protocol domain
  • Anonymous access applies only to public-read or public-write scenarios
  • Split million-level object statistics by directory
  • For batch tasks, set concurrency conservatively first, then tune gradually
  • Keep secondary confirmation and impact descriptions for high-risk commands

Interaction templates

The user wants to upload a file

  1. Confirm the local path, target bucket, and target object name
  2. Determine whether recursive upload is needed
  3. Check the configuration and connectivity
  4. Generate the cp command
  5. Return the success criteria and failure troubleshooting points

The user wants to delete a directory

  1. First confirm the target prefix
  2. First recommend ls to verify the scope of impact
  3. Then generate rm -r
  4. If the user confirms forced deletion, then consider -f

The user wants to troubleshoot inaccessible access

  1. Check version
  2. Check config
  3. Run ls
  4. Perform classified diagnosis based on 403, connection failures, and parameter errors

Testing requirements

  • Use documentation examples as test samples for command construction
  • Use example output as snapshot samples for the parser
  • Write unit tests for 403, network failures, empty configuration, anonymous access, recursive deletion and other scenarios
  • Write integration tests for cp in the three directions of upload/download/cloud copy
  • Write boundary tests for the batch parameters of rm、setmeta、du Perform boundary testing on the batch parameters of

Limitations

  • This Skill prioritizes coverage of the most core config、ls、mb、cp、rm、du、setmeta、stat、help、version
  • share、set-acl、mount、probe、netdig、curl and other advanced commands use an extended adaptation mode
  • If the parameters of an advanced command are not fully expanded in the current reference materials, you must first confirm via help or supplementary documentation before executing

**Appendix: Regions and access domains **

  • Region: indicates the physical location of the TOS data center.
  • Endpoint: indicates the access domain of the TOS external service.
Region Chinese name Region ID Endpoint (internal/external) S3 Endpoint (internal/external)
North China 2 (Beijing) cn-beijing Internal: tos-cn-beijing.ivolces.comExternal: tos-cn-beijing.volces.com Internal: tos-s3-cn-beijing.ivolces.comExternal: tos-s3-cn-beijing.volces.com
South China 1 (Guangzhou) cn-guangzhou Internal network: tos-cn-guangzhou.ivolces.comExternal network: tos-cn-guangzhou.volces.com Internal network: tos-s3-cn-guangzhou.ivolces.comExternal network: tos-s3-cn-guangzhou.volces.com
East China 2 (Shanghai) cn-shanghai Internal network: tos-cn-shanghai.ivolces.comExternal network: tos-cn-shanghai.volces.com Internal network: tos-s3-cn-shanghai.ivolces.comExternal network: tos-s3-cn-shanghai.volces.com
Hong Kong, China cn-hongkong Internal network: tos-cn-hongkong.ivolces.comExternal network: tos-cn-hongkong.volces.com Internal network: tos-s3-cn-hongkong.ivolces.comExternal network: tos-s3-cn-hongkong.volces.com
Asia Pacific Southeast (Johor) ap-southeast-1 Internal network: tos-ap-southeast-1.ivolces.comExternal network: tos-ap-southeast-1.volces.com Internal network: tos-s3-ap-southeast-1.ivolces.comExternal network: tos-s3-ap-southeast-1.volces.com
Asia Pacific Southeast (Jakarta) ap-southeast-3 Internal network: tos-ap-southeast-3.ivolces.comExternal network: tos-ap-southeast-3.volces.com Internal network: tos-s3-ap-southeast-3.ivolces.comExternal network: tos-s3-ap-southeast-3.volces.com