SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsazure-ai-content-safety

azure-ai-content-safety

Security & auditing

View repository

Updated 10-16 02:29

About this skill

Azure AI Content Safety

Azure AI Content Safety is the platform for real-time content moderation and generative AI safety. It analyzes text, images, and multimodal content for harm categories, blocks prompt-injection / jailbreak attempts (Prompt Shields), detects ungrounded responses (Groundedness), and flags protected text/code reproduction.

It pairs with Azure OpenAI as built-in content filters (default safety net) and as configurable guardrails for any AI/LLM application (including non-Microsoft models).

When to use

Building or hardening any generative AI application — chat assistant, agent, RAG app, multimodal product — that needs guardrails on inputs and outputs.

Do not use this skill for SOC threat detection on AI workloads (defender-for-cloud-ai), end-user AI usage governance (purview-ai-hub), or generic AI application architecture.

Capability map

Capability What it does Where to apply
Text moderation Classifies text into hate, violence, sexual, self-harm (severity 0–7) Input + output
Image moderation Same categories on images User uploads, generated images
Multimodal moderation Text+image combined context Multimodal chat
Prompt Shields — User Prompt Detects jailbreak attempts in the user prompt Input
Prompt Shields — Indirect Detects injection in untrusted grounding content (RAG, tool results, web pages) Pre-grounding step
Groundedness detection Checks LLM response against grounding sources for hallucination Output
Protected material — text Detects verbatim copyrighted text Output
Protected material — code Detects reproduction of public-repo code Code-generation output
Custom categories Customer-defined harm categories (e.g., off-brand topics) Input + output
Blocklists Exact-match + regex blocklists Input + output

Approach

  1. Start with default Azure OpenAI content filters. Every Azure OpenAI deployment has a default Content Safety filter at medium severity across all four harm categories. Don't deploy without it.

  2. Tune severity thresholds per use case.

    • Children's product: low threshold (block at severity 2).
    • Adult creative app: high threshold (block only at severity 6). Author a custom content filter in Azure OpenAI Studio; assign to deployment.

  3. Enable Prompt Shields for any user-facing chat or agent.

    • User Prompt Shield on the incoming prompt.
    • Indirect Prompt Shield on grounding content (RAG document chunks, tool outputs, fetched web pages) before they hit the model. Indirect injection is the higher-impact, lower-visibility risk.

  4. Groundedness detection for RAG apps. After the model produces a response, call groundedness detection with the source chunks. Flag/reject ungrounded claims. Adds latency (one extra model call) but is the main defense against hallucination in high-stakes apps (medical, legal, financial).

  5. Protected material detection for products that generate text or code at scale:

    • Text: SaaS copywriting tools.
    • Code: developer assistants (Copilot-style). Flag and either suppress or annotate the output.

  6. Custom categories + blocklists for brand- or domain-specific risks (competitor mentions, regulated disclosure language, internal codenames). Train custom categories via Content Safety Studio with ~100 examples per category.

  7. Architecture pattern (sidecar / middleware).

    • Request → Prompt Shield → (LLM call) → Output classifiers (harm, groundedness, protected material) → Response.
    • Fail-closed: if Content Safety call errors, do not return uncontrolled LLM output.
    • Log every decision (allowed/blocked/flagged) for audit and tuning.

  8. Performance. Each guardrail adds 50–300ms. Parallelize input and output checks; short-circuit on first hard block. Budget for the latency in user expectation.

  9. Operate. Weekly review of blocked-content samples (with privacy controls). Tune thresholds, custom categories, and blocklists. Track false positive / false negative rates per category.

Guardrails

  • Default filters are baseline, not a complete safety strategy. Prompt Shields, groundedness, and custom categories close the real gaps.
  • Don't expose harm-category scores to end users. A "you said X violence score 6" message helps attackers tune jailbreaks.
  • Fail-closed on Content Safety errors. A 500 from the service should not result in unmoderated LLM output reaching users.
  • Indirect prompt injection is the under-appreciated risk. A user's prompt is visible; a malicious string in a fetched web page or document is not. Always shield grounding content.
  • Groundedness detection requires you to send the grounding sources. RAG apps natively have them; pure chat apps don't.
  • Custom categories are not a substitute for product policy. They detect; they don't prevent the user from generating harmful content with another tool.
  • Protected material detection is best-effort. Copyright/IP review is a legal process, not an API result.
  • Content Safety is per-call billed. High-traffic apps need cost modeling; not all features run on every call (groundedness is expensive — gate on use case).
  • Pair with Defender for Cloud AI for SOC-side detection of attempts and patterns (defender-for-cloud-ai).

Common anti-patterns

  • "Default filter is enough" — no jailbreak/injection protection.
  • "Indirect prompt shield skipped because 'all our sources are trusted'" — they're trusted to be there; not trusted to be uncompromised. RAG corpora get poisoned.
  • "Groundedness on every chat call regardless of grounding" — burns cost on pure chat without grounding sources to check against.
  • "Logged user prompts in plaintext for tuning" — privacy/PII exposure. Pseudonymize.
  • "Returned the Content Safety severity score in error messages" — gives attacker a tuning oracle.
  • "Custom categories trained on 5 examples" — no model lift. Need ~100 minimum.
  • "Blocklists used as the main strategy" — misses paraphrases. Combine with classifier categories.
  • "Content Safety only on output" — input attacks (jailbreaks) reach the model.

Example prompts

  • Architect Content Safety guardrails for a RAG-based customer support assistant on Azure OpenAI.
  • Configure Prompt Shields for a Foundry agent with internet-grounded tools.
  • Tune content filter severity thresholds for a teen-audience product.
  • Add groundedness detection to flag hallucinations in our medical Q&A assistant.
  • Train a custom category for "competitor mention" and integrate as output blocklist.
  • Cost and latency model: Content Safety calls per chat turn on a 10K-user app.
  • IR runbook: how SOC handles "prompt shield blocked attempted jailbreak" pattern (paired with Defender for Cloud AI alert).

Microsoft Learn