SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmodel-datasets

model-datasets

Data & analytics

View repository

Updated 10-15 20:08

About this skill

Unified storage and loading conventions for models / datasets

In one sentence

The download platform is just a means; the files themselves are the essence. All models and datasets, whether from HF or ModelScope, land in the same shape:

/data/models/<org>__<name>/      模型实体
/data/datasets/<org>__<name>/    数据集实体

The directory directly contains the file bodies (config.json、*.safetensors, tokenizer… or dataset data files),with no snapshots/, commit-hash intermediate layers of that kind. Loadinguses only absolute paths, no id indexing, no platform distinction.

Naming rules

Directory name = repo_id in / replaced with __：

repo_id directory
meta-llama/Llama-3.1-8B-Instruct /data/models/meta-llama__Llama-3.1-8B-Instruct
ZhipuAI/GLM-4.7-Flash /data/models/ZhipuAI__GLM-4.7-Flash
HuggingFaceH4/ultrachat_200k /data/datasets/HuggingFaceH4__ultrachat_200k

The org name occasionally differs slightly between the two platforms (e.g. ZhipuAI vs zai-org) doesn't matter, as long as you can tell from the name whose model it is. The id name doesn't matter,being able to load with an absolute path is the goal。

Install tools (uv)

Use uv to install these two CLIs (isolated environment, executables shimmed to ~/.local/bin, make sure it's in PATH).

# 1) uv 本身（没有才装）
curl -LsSf https://astral.sh/uv/install.sh | sh

# 国内加速：给 uv 配清华 PyPI 源
mkdir -p ~/.config/uv
printf 'index-url = "https://pypi.tuna.tsinghua.edu.cn/simple"\n' > ~/.config/uv/uv.toml

# 2) HF CLI（提供 hf 命令）
uv tool install huggingface_hub        # 老版本写 huggingface_hub[cli]

# 3) ModelScope CLI —— 包名是 modelscope-hub，不是 modelscope！
uv tool install modelscope-hub
# 注意：uv tool install modelscope 会报 "No executables are provided"，
#       因为新版把 CLI 拆到了 modelscope-hub 这个包，装它的同时会带上 modelscope 库。

Verify after installation:hf --version、modelscope --help. Available commands:hf、modelscope、ms. Upgrade:uv tool upgrade --all。

Download

Uniformly use the dl tool (source code at the end of the document, put it in ~/.local/bin/dl and chmod +x）：

dl <hf|ms> <repo_id> [--type model|dataset] [--name NAME]

dl hf meta-llama/Llama-3.1-8B-Instruct
dl ms ZhipuAI/GLM-4.7-Flash
dl hf HuggingFaceH4/ultrachat_200k --type dataset

Equivalent to the original command (dl is just assembling the path + forcing --local-dir）：

hf download <repo_id> --repo-type model|dataset --local-dir /data/models/<org>__<name>
modelscope download --repo-type model|dataset <repo_id> --local-dir /data/models/<org>__<name>

Platform choice only depends on which platform has the repo: for domestic models try ModelScope first, if not available go with hf(configure a mirror domestically). The on-disk result is exactly the same.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
AutoModelForCausalLM.from_pretrained("/data/models/ZhipuAI__GLM-4.7-Flash")
AutoTokenizer.from_pretrained("/data/models/ZhipuAI__GLM-4.7-Flash")

from datasets import load_dataset
load_dataset("/data/datasets/<org>__<name>")

What ModelScope downloads is standard HF format (config.json + safetensors + tokenizer), so the absolute path usage is the same for both sources.

Environment variables

export HF_ENDPOINT=https://hf-mirror.com   # HF 走镜像（国内）
export HF_HOME=/data/hub/hf                # HF 工具缓存
export MODELSCOPE_CACHE=/data/hub/ms       # ModelScope 缓存/中转
export HF_HUB_DISABLE_XET=1                # 必须：HF 2.0 的 Xet 协议经镜像会 401，禁用走普通 HTTP

Writing to ~/.bashrc / ~/.zshrc。/data/hub is only a tool cache, can be cleared at any time;/data/models、/data/datasets is the actual entity, don't delete it carelessly.

Preparing /data on a new machine (decide based on the current situation)

Principles:Models/datasets are shared immutable files and must be placed on a large disk, don't put them on the root disk,~、~/.cache. Users customarily use /data。

Current situation Approach
/data already exists and is the large disk use it directly
no /data, but there is already a mounted large-disk directory (e.g. /datahdd、/datanfs） ln -s /datahdd /data（/data can only be created as a symlink when it doesn't exist; if /data is already an empty directory, first rmdir then link)
no /data, but there is an uninitialized disk Initialize and mount as /data: partition/format (prefer LVM if available, for easier expansion later) → write to /etc/fstab(using UUID + nofail）→ mount -a
multiple disks available can combine into an LVM volume to form one big /data; models and datasets are shared and immutable, merging into one volume is the least worrisome
not sure first lsblk -f、df -h to see clearly whether there is a large disk and where it's mounted, then decide; analyze specific cases individually

Initializing a disk will cause the existing data to be lost, so before you start, first confirm that the disk is empty. After initialization, remember to create /data/models、/data/datasets、/data/hub。

Verification checklist

ls /data/models /data/datasets         # 目录在
df -h /data                            # 指向大盘，且不是根盘
dl hf hf-internal-testing/tiny-random-gpt2   # 下个小模型试通
python -c "import json;print(json.load(open('/data/models/hf-internal-testing__tiny-random-gpt2/config.json'))['model_type'])"
rm -rf /data/models/hf-internal-testing__tiny-random-gpt2   # 清掉测试

Appendix:dl Script (self-contained, copy directly)

#!/usr/bin/env bash
# dl - 统一的模型/数据集下载（HF / ModelScope 只是下载手段，落盘组织完全一致）
#   dl <hf|ms> <repo_id> [--type model|dataset] [--name NAME]
# 目录：/data/models/<org>__<name>  或  /data/datasets/<org>__<name>
set -euo pipefail
export PATH="$HOME/.local/bin:$PATH"
ROOT=/data
export HF_HOME="${HF_HOME:-$ROOT/hub/hf}"
export MODELSCOPE_CACHE="${MODELSCOPE_CACHE:-$ROOT/hub/ms}"
export HF_HUB_DISABLE_XET=1   # HF 2.0 的 Xet 经 hf-mirror 会 401，禁用改走普通 HTTP

usage() {
  cat >&2 <<'EOF'
用法:
  dl <hf|ms> <repo_id> [--type model|dataset] [--name NAME]
说明:
  --type   默认 model；dataset 落到 /data/datasets/<org>__<name>
  --name   自定义目录名，默认 repo_id 的 "/" 换成 "__"
示例:
  dl hf meta-llama/Llama-3.1-8B-Instruct
  dl ms ZhipuAI/GLM-4.7-Flash
  dl hf HuggingFaceH4/ultrachat_200k --type dataset
EOF
  exit 1
}
die() { echo "错误: $*" >&2; exit 1; }

[ $# -ge 2 ] || usage
src="$1"; repo="$2"; shift 2
[ "$src" = hf ] || [ "$src" = ms ] || die "平台只能是 hf 或 ms"
type=model; name=""
while [ $# -gt 0 ]; do
  case "$1" in
    --type) type="$2"; shift 2;;
    --name) name="$2"; shift 2;;
    *) die "未知参数: $1";;
  esac
done
[ "$type" = model ] || [ "$type" = dataset ] || die "--type 只能是 model 或 dataset"
[ -n "$name" ] || name="${repo//\//__}"
if [ "$type" = dataset ]; then dest="$ROOT/datasets/$name"; else dest="$ROOT/models/$name"; fi
mkdir -p "$dest"

if [ "$src" = hf ]; then
  echo ">> hf  $repo -> $dest"
  hf download "$repo" --repo-type "$type" --local-dir "$dest"
else
  echo ">> ms  $repo -> $dest"
  modelscope download --repo-type "$type" "$repo" --local-dir "$dest"
fi

echo "完成: $dest"
echo "加载: from_pretrained(\"$dest\")"

Depends on two CLIs:hf、modelscope(for installation, see “Install tools (uv)” above), both under ~/.local/bin。