SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsai-data-security

ai-data-security

Data & analytics

View repository

Updated 10-16 07:22

About this skill

AI data security testing methodology

Attack surface overview

Phase Attack type Target
Application layer System Prompt leakage, API information leakage, PII theft Configuration/interface/user data
Model layer Training data inference, membership inference, model inversion Training data/model parameters
Deployment layer RAG poisoning, vector store privilege escalation, cache leakage Knowledge base/infrastructure

Boundary with prompt-leak

prompt-leak Focuses on 6 categories of techniques for System Prompt leakage (direct request, formatting, indirect inducement, reflection, side channel, multi-turn extraction). This skill treats Prompt leakage as one of the application-layer attack surfaces, with emphasis on broader data security issues such as training data, the model layer, and the RAG data pipeline. When in-depth Prompt leakage is needed, directly call prompt-leak。

Phase 0: Data asset identification

Before testing, first clarify the data asset boundaries of the AI system:

  • Training data — pre-training corpus, fine-tuning datasets, RLHF annotation data
  • RAG knowledge base — document repository, vector database, retrieval index
  • API endpoints — inference interfaces, management interfaces, internal debugging endpoints
  • Caches and logs — conversation history, intermediate reasoning results, audit logs

Phase 1: Application-layer data leakage

  • System Prompt leakage — see prompt-leak skill for details, not expanded here
  • API information leakage — craft prompts to induce the model to output API endpoints, model versions, parameter configurations; probe /v1/models、/debug、/health and other common endpoints for exposure of internal information
  • Private data theft — use conversation context to extract PII (name, phone, email); make the model repeat other users' interaction content by assuming business scenarios; pay attention to context isolation flaws in multi-tenant scenarios

Phase 2: Model-layer data attacks

  • Training data inference — prefix completion: provide a possible prefix of training data and let the model autocomplete it ("My social security number is..."); Divergence attack: ask the model to repeat a word indefinitely, and when the model deviates from the normal distribution it may emit training samples
  • Membership inference attack — train a Shadow Model to imitate the target model's behavior; compare the difference in output probabilities of the target model for data "inside the training set" versus "outside the training set"; high-confidence outputs often mean the data appeared in the training set
  • Model inversion attack — in white-box scenarios, infer input features from model outputs through gradient optimization; in black-box scenarios, reconstruct the input distribution through a large number of API queries (such as face reconstruction attacks)

Phase 3: Data poisoning and tampering

  • RAG knowledge base poisoning — Inject malicious documents into public data sources, wait for the RAG system to index them and thereby affect retrieval results; manipulate embeddings so that malicious content is semantically close to high-frequency queries
  • Dialogue corpus poisoning — If the system uses user feedback for fine-tuning, inject specific dialogue patterns to contaminate the training set
  • Cascading hallucination attack — Induce the model to generate erroneous information, and when downstream systems rely on that output, trigger data-link-level errors

Phase 4: Deployment-layer data security

  • Unauthorized access to vector databases — Detect the authentication configuration and network exposure surface of vector databases such as Milvus / Pinecone / Weaviate
  • Backup/log/cache leakage — Dialogue history stored in plaintext, inference logs containing sensitive prompts, model snapshots stored unencrypted

Detection checklist

  • Whether the System Prompt can be leaked through any method
  • Whether API endpoints expose model version/parameters/internal configuration
  • Whether the model leaks fragments of training data in its output
  • Whether dialogue context is isolated in multi-tenant scenarios
  • Whether RAG data sources can be poisoned by external parties
  • Whether the vector database has authentication and access control
  • Whether logs/caches contain sensitive data and are unencrypted

In-depth reference