zh-encoding

Chinese/Unicode encoding handling (Windows)

When handling Chinese text runs into garbled characters, detection/counting results look suspicious, or the command to be written contains Unicode code points, use the verified practices in this skill.

Which tool to use

Goal	Practice
Detect/count Chinese	python: Unicode semantics, independent of shell locale, preferred
Match/scan	rg(Unicode semantics are reliable, no locale pitfalls); do not use GNU grep -P to handle Unicode
Pipe/file output of Chinese	python (on this machine PYTHONUTF8 is already in effect) or explicitly encoding=

Counting/detecting Chinese: python one-liner

python - <<'EOF'
import glob, re
for f in sorted(glob.glob('src/*.rs')):
    n = sum(1 for l in open(f, encoding='utf-8')
            if re.search(r'[\u4e00-\u9fff]', l))
    print(f, n)
EOF

Completion criterion: Chinese output is readable, and the line count matches the result of manually counting one or two files.

Environment verification and fix (PYTHONUTF8)

It is already persistently in effect on this machine; the following is kept for reference in a new environment or after a reset.

python -c "import sys; print(sys.stdout.encoding)"   # 期望 utf-8

Persistent modification (setx writes the registry,config.toml writes the config) is a disk-write operation; before acting:

• First read ~/.config/mise/config.toml, confirm there is no sensitive content such as keys/tokens before modifying;
• Prefer temporary in-session effect—only add before the current command PYTHONUTF8=1, or for this session export PYTHONUTF8=1, without writing to disk;
• If persistence is truly needed, first show the user the [env] section to be appended and the setx command, and only write after obtaining explicit consent; see the end of this section for how to undo.

If it is not utf-8 (a typical symptom of locale falling back to GBK/cp936):

setx PYTHONUTF8 1

and in ~/.config/mise/config.toml append:

[env]
PYTHONUTF8 = "1"

Takes effect after restarting pi. Undo:

reg delete "HKCU\Environment" /v PYTHONUTF8 /f

and delete config.toml from [env] section.

Pitfalls and mechanisms

• grep -P large code points \x{} in an old environment (C locale) will report too large errors line by line and pollute grep -q statistics; when UTF-8 but PCRE has no UTF support, there are silent false negatives—avoid both, use rg。
• line-by-line while read ... | grep loop counting: each line starts a subprocess, and thousands of lines take tens of seconds (1255 lines ≈ 56s).
• When python pipe output is not encoded as UTF-8, Chinese is garbled—pipe encoding and interactive terminal are two different things; when troubleshooting garbled text, first align "the output side's byte encoding vs the consumer side's decoding assumption".