python-json-parsing

Python JSON Parsing Best Practices

Comprehensive guide to JSON parsing in Python with focus on performance, security, and scalability.

Quick Start

Basic JSON Parsing

import json

# Parse JSON string
data = json.loads('{"name": "Alice", "age": 30}')

# Parse JSON file
with open("data.json", "r", encoding="utf-8") as f:
    data = json.load(f)

Write JSON file

with open("output.json", "w", encoding="utf-8") as f: json.dump(data, f, indent=2)

**Key Rule:** Always specify `encoding="utf-8"` when reading/writing files.

## When to Use This Skill

Use this skill when:

- Working with JSON APIs or data interchange
- Optimizing JSON performance in high-throughput applications
- Handling large JSON files (> 100MB)
- Securing applications against JSON injection
- Extracting data from complex nested JSON structures

## Performance: Choose the Right Library

### Library Comparison (10,000 records benchmark)

| Library | Serialize (s) | Deserialize (s) | Best For |
|---------|---------------|-----------------|----------|
| **orjson** | 0.42 | 1.27 | FastAPI, web APIs (3.9x faster) |
| **msgspec** | 0.49 | 0.93 | Maximum performance (1.7x faster deserialization) |
| **json** (stdlib) | 1.62 | 1.62 | Universal compatibility |
| **ujson** | 1.41 | 1.85 | Drop-in replacement (2x faster) |

**Recommendation:**

- Use **orjson** for FastAPI/web APIs (native support, fastest serialization)
- Use **msgspec** for data pipelines (fastest overall, typed validation)
- Use **json** when compatibility is critical

### Installation

```bash
# High-performance libraries
pip install orjson msgspec ujson

# Advanced querying
pip install jsonpath-ng jmespath

# Streaming large files
pip install ijson

# Schema validation
pip install jsonschema

Large Files: Streaming Strategies

For files > 100MB, avoid loading into memory.

Strategy 1: JSONL (JSON Lines)

Convert large JSON arrays to line-delimited format:

# Stream process JSONL
with open("large.jsonl", "r") as infile, open("output.jsonl", "w") as outfile:
    for line in infile:
        obj = json.loads(line)
        obj["processed"] = True
        outfile.write(json.dumps(obj) + "\n")

Strategy 2: Streaming with ijson

import ijson

# Process large JSON without loading into memory
with open("huge.json", "rb") as f:
    for item in ijson.items(f, "products.item"):
        process(item)  # Handle one item at a time

See: patterns/streaming-large-json.md

Security: Prevent JSON Injection

Critical Rules:

1. Always use json.loads(), never eval()
2. Validate input with jsonschema
3. Sanitize user input before serialization
4. Escape special characters (" and \)

Vulnerable Code:

# NEVER DO THIS
username = request.GET['username']  # User input: admin", "role": "admin
json_string = f'{{"user":"{username}","role":"user"}}'
# Result: privilege escalation

Secure Code:

# Use json.dumps for serialization
data = {"user": username, "role": "user"}
json_string = json.dumps(data)  # Properly escaped

See: anti-patterns/security-json-injection.md, anti-patterns/eval-usage.md

Advanced: JSONPath for Complex Queries

Extract data from nested JSON without complex loops:

import jsonpath_ng as jp

data = {
    "products": [
        {"name": "Apple", "price": 12.88},
        {"name": "Peach", "price": 27.25}
    ]
}

# Filter by price
query = jp.parse("products[?price>20].name")
results = [match.value for match in query.find(data)]
# Output: ["Peach"]

Key Operators:

• $ - Root selector
• .. - Recursive descendant
• * - Wildcard
• [?<predicate>] - Filter (e.g., [?price > 20])
• [start:end:step] - Array slicing

See: patterns/jsonpath-querying.md

Custom Objects: Serialization

Handle datetime, UUID, Decimal, and custom classes:

from datetime import datetime
import json

class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        if isinstance(obj, set):
            return list(obj)
        return super().default(obj)

# Usage
data = {"timestamp": datetime.now(), "tags": {"python", "json"}}
json_str = json.dumps(data, cls=CustomEncoder)

See: patterns/custom-object-serialization.md

Performance Checklist

• Use orjson/msgspec for high-throughput applications
• Specify UTF-8 encoding when reading/writing files
• Use streaming (ijson/JSONL) for files > 100MB
• Minify JSON for production (separators=(',', ':'))
• Pretty-print for development (indent=2)

Security Checklist

• Never use eval() for JSON parsing
• Validate input with jsonschema
• Sanitize user input before serialization
• Use json.dumps() to prevent injection
• Escape special characters in user data

Reference Documentation

Performance:

• reference/python-json-parsing-best-practices-2025.md - Comprehensive research with benchmarks

Patterns:

• patterns/streaming-large-json.md - ijson and JSONL strategies
• patterns/custom-object-serialization.md - Handle datetime, UUID, custom classes
• patterns/jsonpath-querying.md - Advanced nested data extraction

Security:

• anti-patterns/security-json-injection.md - Prevent injection attacks
• anti-patterns/eval-usage.md - Why never to use eval()

Examples:

• examples/high-performance-parsing.py - orjson and msgspec code
• examples/large-file-streaming.py - Streaming with ijson
• examples/secure-validation.py - jsonschema validation

Tools:

• tools/json-performance-benchmark.py - Benchmark different libraries