alibabacloud-safety-checker

Content Security & AI Guardrails Tester

Alibaba Cloud Content Moderation + AI Security Guardrail automated testing Skill. Covers the full workflow from scenario selection, console configuration, batch testing to annotation analysis, and also supports AI Security Guardrail effectiveness testing.

Important: Execution Rules

1. Working Directory: All python3 commands must be executed using absolute paths. Scripts are located in two places:
   • Skill directory: ~/.qoderwork/skills/safety-checker/(default)
   • User project directory: if the user has a selected folder, use that directory first
2. Command Format: Always use the python3 <绝对路径>/script.py format, for example: python3 ~/.qoderwork/skills/safety-checker/scenario_manager.py list
3. Do not execute the same command in a loop: If a command has already been executed successfully and returned output, present the result directly to the user; do not execute it repeatedly
4. Respond after a single execution: Each step only needs to execute the command once; after obtaining the output, immediately report the result to the user or proceed to the next step

Quick Start

# 设置工作目录变量
SAFETY_DIR=~/.qoderwork/skills/safety-checker

Install Dependencies

pip install -r $SAFETY_DIR/scripts/requirements.txt

Verify connectivity (relies on the default credential chain; no need to set credentials manually)

python3 $SAFETY_DIR/verify_auth.py

## 工作流

### Step 1: 确认业务场景

通过对话了解用户业务类型，从预定义场景中选择最匹配的：

| 场景ID | 名称 | 适用场景 |
|--------|------|---------|
| `ai_companion_chat` | AI陪聊/虚拟伴侣 | AI角色扮演、情感陪伴 |
| `game_chat` | 游戏聊天室 | 游戏内公聊、组队频道 |
| `general_chat` | 通用Chat/社交 | IM私聊群聊、论坛 |
| `ecommerce` | 零售电商 | 商品评价、描述、客服 |
| `news_content` | 资讯/新闻 | 新闻平台、自媒体 |

```bash
# 查看所有场景（使用绝对路径）
python3 ~/.qoderwork/skills/safety-checker/scenario_manager.py list

# 查看某个场景的配置详情（执行一次后直接将输出呈现给用户）
python3 ~/.qoderwork/skills/safety-checker/scenario_manager.py show <场景ID>

Completion Condition: After the command outputs the scenario information, directly show the result to the user and ask whether adjustments are needed. Do not execute the same command repeatedly.

Step 2: Configure console label switches

Method A: Browser automation (recommended)

python3 ~/.qoderwork/skills/safety-checker/console_automator.py --scenario <场景ID>

The script automatically completes: open the console → guide the user to log in → copy the Service → configure label switches one by one → save.

Method B: Generate an operation guide and let the user configure manually

python3 ~/.qoderwork/skills/safety-checker/scenario_manager.py guide <场景ID>

Generate a detailed Markdown operation guide. After the configuration is saved, wait 2-5 minutes for it to take effect.

Step 3: Prepare test samples

Samples support JSON / CSV / Excel formats. For novels/long texts, automatically split:

python3 ~/.qoderwork/skills/safety-checker/prepare_novel.py /path/to/novel.txt --max-samples 100

Sample JSON format:

{
  "samples": [{
    "sample_id": "text_001",
    "modality": "text",
    "content": "待审核文本",
    "data_id": "自定义ID",
    "category": "分类标签",
    "expected_risk_level": "期望风险等级(可选)",
    "expected_labels": ["期望标签(可选)"]
  }]
}

Step 4: Run moderation test

python3 ~/.qoderwork/skills/safety-checker/main.py run --samples <样本文件> \
  --text-services <自定义Service名> \
  --concurrent 10 --format xlsx

Step 5: Manual annotation

python3 ~/.qoderwork/skills/safety-checker/main.py annotate --file results/results_batch_xxx.json --annotator "姓名"

Interactive annotation options:c=correct,fp=false positive,fn=false negative,u=uncertain,s=skip,q=exit and save.

Step 6: Generate alignment analysis report

python3 ~/.qoderwork/skills/safety-checker/main.py report --file results/annotated_xxx.xlsx

The report includes: accuracy/false positive rate/false negative rate by service, analysis by modality, analysis by category.

Input/Output Examples

Text moderation - input

{
  "samples": [
    {"sample_id": "t001", "modality": "text", "content": "今天天气真不错，适合出去玩", "category": "normal"}
  ]
}

Text moderation - output

{
  "sample_id": "t001",
  "risk_level": "none",
  "labels": [],
  "request_id": "A216D7F5-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "latency_ms": 120
}

Exception Handling

• Credentials not configured：verify_auth.py Return a clear error message, guiding the user to check the default credential chain configuration
• Region unavailable: When the SDK returns an InvalidRegionId error, the script prompts to switch to cn-shanghai or cn-beijing
• Timeout: default read_timeout=30s; for large text/image scenarios, it can be adjusted via the --timeout parameter
• API rate limiting: on a 429 status code, automatically retry with exponential backoff (up to 3 times), and mark the affected samples in the results

AI Security Guardrail Testing

Feature Overview

AI Security Guardrail is a security protection solution for large model applications and AI Agents, supporting three major detection dimensions:

Protection Dimension	Returned Field	Description	Default Status
Content Compliance	Result / RiskLevel	Detects pornographic, violent, political, and other harmful content	Enabled by default
Sensitive content detection	SensitiveResult / SensitiveLevel	Detects PII data such as ID card numbers, credit card numbers, and mobile phone numbers	Must be manually enabled, billed separately
Prompt attack detection	AttackResult / AttackLevel	Detects malicious attacks such as prompt injection and jailbreaking	Must be manually enabled, billed separately

Step G1: Enable the AI Security Guardrail service

1. Go to AI Security Guardrail product activation page to enable the service (pay-as-you-go)
2. Grant the RAM user the AliyunYundunGreenWebFullAccess permission
3. Log in to the AI Security Guardrail console, in Protection Configuration → Detection Item Configuration , confirm the switch status of the three detection dimensions

Important: Sensitive content detection and prompt attack detection are disabled by default, must be manually enabled in the console, and are billed separately. If not enabled, the corresponding returned fields (SensitiveResult/AttackResult) will be empty.

Step G2: Run the guardrail effectiveness test

# 运行默认测试套件（20个样本：10个输入检测+10个生成检测）
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py

# 仅测试AI输入检测
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --service query

# 仅测试AI生成检测
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --service response

# 指定输出文件
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --output results/my_guardrail_test.json

# 输出Excel三Sheet报告（详细结果 + 服务汇总 + 三维度拆解）
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --format xlsx

# 同时输出JSON和XLSX
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --format both

# 输出暗色主题 HTML 可视化报告（三维度卡片 + 逐样本表格 + 自动洞察）
python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --format html

# 通过一站式入口调用
python3 ~/.qoderwork/skills/safety-checker/cst.py guardrail
python3 ~/.qoderwork/skills/safety-checker/cst.py guardrail --service query --format xlsx

Step G3: Test with custom samples

{
  "query": [
    {"name": "测试名称", "content": "待检测文本", "expected_risk_level": "high"}
  ],
  "response": [
    {"name": "测试名称", "content": "待检测文本", "expected_risk_level": "none"}
  ]
}

python3 ~/.qoderwork/skills/safety-checker/ai_guardrail.py --samples assets/my_guardrail_samples.json

Step G4: Interpret the test results

AI Security Guardrail returns detection results for three dimensions:

RiskLevel (content compliance risk level)：

• none: No risk detected
• low: Low risk (recommended to handle as no risk in daily operations)
• medium: Medium risk (manual review recommended)
• high: High risk (direct handling recommended)

SensitiveLevel (sensitive information level)：

• S0: No sensitive content detected
• S1~S4: The higher the number, the higher the sensitivity level

AttackLevel (attack level)：

• none: No attack detected
• low/medium/high: Attack risk level

Step G5: Root cause analysis of missed detections

When missed detections occur in the test results, troubleshoot according to the following logic:

Missed detection scenario	Possible root cause	Troubleshooting method	Remediation plan
SensitiveResult is entirely empty	The sensitive content detection dimension is not enabled	Check the detection item configuration in the console	Console → Protection Configuration → Detection Item Configuration → Enable "Sensitive content detection"
AttackResult is entirely empty	The prompt attack detection dimension is not enabled	Check the detection item configuration in the console	Console → Protection Configuration → Detection Item Configuration → Enable "Prompt attack detection"
Pornographic intent is missed	The model focuses on content description rather than intent	Compare "contains pornographic descriptions" vs "requests pornographic resources"	Configure a custom dictionary to blacklist keywords such as "pornographic images"
Prompt injection is classified as prohibited behavior	Attack detection is not enabled, only content compliance takes effect	Check whether AttackLevel is N/A	Enable prompt attack detection
Discrimination/metaphor is missed	The corresponding tag is not enabled or the model sensitivity is insufficient	Check the status of the inappropriate_discrimination tag	Enable the tag in the console + supplement the custom dictionary

Default test sample coverage

AI Input Detection (query_security_check) 10 samples: normal greeting, politically sensitive, pornographic content, abusive attack, gambling promotion, prompt injection (ignore instruction type), prompt injection (role-playing type), jailbreak inducement, sensitive information (ID card number), discrimination and bias

AI Generation Detection (response_security_check) 10 samples: normal answer, generate sensitive information (credit card number), generate sensitive information (phone number), generate sensitive information (email), generate violating content (drug production), generate violating content (violence teaching), generate discriminatory content, generate false information, generate pornographic content, generate violent terrorist content

AI-Mode and SDK Configuration Declaration

This Skill uses the Alibaba Cloud Python SDK to call content security services.references/console-guide.md The aliyun CLI example included is only for manual debugging and is not part of the automated call chain.

aliyun CLI AI-Mode Configuration

This Skill is mainly called through the Python SDK and does not depend on the AI-Mode plugin.references/console-guide.md The aliyun CLI commands in are only for manual debugging reference. The AI-Mode lifecycle management commands are as follows:

# 启用 AI-Mode
aliyun configure ai-mode enable

# 设置 User-Agent
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-safety-checker"

# 禁用 AI-Mode
aliyun configure ai-mode disable

# 更新插件
aliyun plugin update

Current status: AI-Mode is disabled by default, and this Skill does not call the CLI through the AI-Mode plugin. System commands (such as --help、configure) do not set user-agent.

SDK Configuration Specification

• SDK Config has set user_agent="AlibabaCloud-Agent-Skills/alibabacloud-safety-checker"
• SDK Config has set read_timeout=30 and connect_timeout=10
• All API calls obtain credentials through the default credential chain (environment variables → credentials file → ECS RAM role), and credentials are not hardcoded in the code

Security Specification

• Credentials are automatically obtained through the default credential chain, with no explicit configuration required
• The browser login step guides the user to complete it manually, and does not automatically enter the password
• Confirm with the user before each console configuration

Reference Resources

For the complete tag system, audit service list, and console operation guide, see the references/ directory:

• Complete tag configuration matrix: references/label-matrix.md
• Detailed scenario description: references/scenarios.md
• Console automation guide: references/console-guide.md

File Description

File	Description
main.py	CLI main entry (run/annotate/report)
ai_guardrail.py	AI security guardrail effectiveness test (input detection + generation detection, supports JSON/XLSX output)
cst.py	One-stop entry (init/configure/test/annotate/report/guardrail/status/full)
scenarios.py	Scenario templates and tag configuration matrix
console_automator.py	Console browser automation
scenario_manager.py	Scenario management tool
prepare_novel.py	Long text preprocessing
verify_auth.py	Credential chain connectivity verification