csv-to-json-validator

csv-to-json-validator

§1 Identity

Role: Data-pipeline skill that validates CSV input against a declared schema and produces a structured JSON array, preserving every row-level error.

Red Lines:

• MUST NOT silently drop rows. Every rejection emits an error record.
• MUST NOT infer types from data. The schema is authoritative.
• MUST NOT write anywhere other than the declared output path.

§2 Skill Summary

WHAT: CSV → JSON with per-row validation against a typed schema. WHEN: You have tabular input from an untrusted source (user upload, third-party export) and need to hand structured data to downstream systems. WHO: Data engineers, backend developers, data-QA. NOT-FOR: Free-text/unstructured text (use a parser). Binary formats. Streaming use-cases > 10 MB (this skill is bounded to in-memory).

§3 Negative Boundaries

• Do NOT use for Excel (.xlsx) → use a dedicated xlsx reader upstream.
• Do NOT use for JSON Lines or NDJSON → input format is canonical CSV only.
• Do NOT use for "best-effort" parsing with type coercion → schema mismatches must fail loudly.

§4 Mode Router

User says...	Mode
"validate this csv", "check schema"	VALIDATE-ONLY
"convert csv to json", "csv → json"	CONVERT
"validate and convert"	FULL (default)

§5 Workflow (FULL mode)

1. PARSE
   - Read input CSV (UTF-8, RFC 4180).
   - Reject if file size > 10 MB → ABORT with MaxSizeExceeded.
2. LOAD SCHEMA
   - Schema is a YAML or JSON object mapping column → type + constraints.
   - Supported types: string, int, float, bool, iso-date, email, url, enum.
3. VALIDATE ROWS (per row)
   - For each column:
     a. Check type.
     b. Check required/nullable.
     c. Check constraints (min, max, regex, enum values).
   - Collect all errors for the row (don't short-circuit).
   - If the row has ≥ 1 hard error, record it in `errors` and continue.
4. EMIT
   - JSON array of validated rows (in source order).
   - Separate `errors` array of {row_index, column, code, message}.
   - Summary: {total_rows, valid_rows, invalid_rows, column_coverage}.
5. EXIT
   - Exit 0 if invalid_rows == 0, else exit 1 (for script consumers).

§6 Error Handling

Condition	Action
File > 10 MB	ABORT; MaxSizeExceeded
Non-UTF-8 / not valid CSV	ABORT; FormatError
Schema unparseable	ABORT; SchemaError
Schema column missing from CSV	WARN; continue with null
CSV column not in schema	WARN; drop from output (report in summary)
Row-level type mismatch	RECORD in errors[]; do not include in output
Duplicate column name in CSV	ABORT; AmbiguousHeader

§7 Example — bilingual

User (EN): "Validate orders.csv against orders-schema.yaml and emit JSON."

Output (example):

{
  "summary": {"total_rows": 1000, "valid_rows": 987, "invalid_rows": 13},
  "data": [
    {"order_id": "ORD-001", "amount": 49.99, "email": "alice@example.com"},
    ...
  ],
  "errors": [
    {"row_index": 42, "column": "amount", "code": "NOT_FLOAT", "message": "'N/A' is not a valid float"},
    {"row_index": 88, "column": "email", "code": "INVALID_EMAIL", "message": "'not-an-email' failed RFC 5322 pattern"}
  ]
}

User (ZH): "Validate orders.csv according to orders-schema.yaml and output JSON."

(Output structure is the same as above; the message field can switch to Chinese based on language.)

§8 Security Baseline

Detected patterns (OWASP Agentic Top 10):

• ASI02 Prompt Injection (input): CSV cells may contain =SUM(...) formula-injection payloads; emit data as JSON strings, never as formulas.
• ASI07 Sensitive Info Disclosure: Never log full row contents at INFO level; redact fields matching PII regex.
• CWE-22 Path Traversal: Reject input paths containing .. or absolute paths outside the declared workspace.

§9 Tests (benchmark cases)

1. Valid 10-row CSV → all rows pass, 0 errors.
2. 5 of 10 rows have wrong type in 1 column → 5 errors collected, 5 valid rows in output.
3. Missing required column → WARN + continue with null.
4. Formula injection (=CMD("rm -rf ~")) → emitted as literal string, no evaluation.
5. 20 MB file → ABORT immediately before parsing.

§10 Provenance

generation_method: human-authored · validation_status: full-eval · authored as a reference example for examples/data-pipeline-demo/.