django-storages

Django Storages

Django package for cloud storage backends. Provides a unified API for storing files across different cloud providers.

Quick Start

pip install django-storages[boto3]  # AWS S3 example

# settings.py
import os

STORAGES = { "default": { "BACKEND": "storages.backends.s3.S3Storage", "OPTIONS": { "bucket_name": "my-app-media", "region_name": "us-east-1", }, }, "staticfiles": { "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage", }, }

Credentials from environment (never in settings)

AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"] AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]

---

## Critical Pitfall: Filename Encoding

**The single most common failure** with user-uploaded files is non-ASCII characters, spaces, and special characters in filenames.

### What Breaks

- Unicode filenames (`报告.pdf`, `vacation été.jpg`) may not survive URL encoding
- Spaces become `%20` or `+` inconsistently across backends
- Duplicate names overwrite existing files silently
- Some backends reject certain characters entirely

### Solution: Sanitize in `upload_to`

```python
import os
import unicodedata
import uuid
from django.utils.text import slugify

def sanitize_upload_path(instance, filename):
    """
    Sanitize filename and guarantee uniqueness.
    Never trust client-supplied filenames.
    """
    # Extract extension
    ext = os.path.splitext(filename)[1]
    
    # Normalize unicode (NFKD decomposes, then slugify strips diacritics)
    sanitized = unicodedata.normalize("NFKD", filename)
    sanitized = slugify(sanitized)
    
    # Fallback if slugify produces empty string
    if not sanitized:
        sanitized = "file"
    
    # Append random suffix to guarantee uniqueness
    unique_id = uuid.uuid4().hex[:8]
    new_filename = f"{sanitized}_{unique_id}{ext}"
    
    return f"uploads/{new_filename}"

class Document(models.Model):
    title = models.CharField(max_length=255)
    file = models.FileField(upload_to=sanitize_upload_path)

Why Not Just Use slugify?

• Two users upload report.pdf → second overwrites first
• Filename contains only special characters → slugify produces empty string
• You need to reference the original name for download → store it separately

class Document(models.Model):
    title = models.CharField(max_length=255)
    original_name = models.CharField(max_length=255)  # For download
    file = models.FileField(upload_to=sanitize_upload_path)

---

Configuration (Django 6.0+)

STORAGES Setting

The STORAGES dict replaces DEFAULT_FILE_STORAGE and STATICFILES_STORAGE (removed in Django 6.0).

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "my-app-media",
            "region_name": "us-east-1",
        },
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
    },
}

Why Backend Selection Goes in Settings, Not Code

Problem: Hardcoding the backend in code forces environment-specific branching.

# ❌ DON'T: Environment branching in code
if settings.DEBUG:
    from django.core.files.storage import FileSystemStorage
    storage = FileSystemStorage()
else:
    from storages.backends.s3 import S3Storage
    storage = S3Storage()

Solution: Swap backends via settings per environment.

# settings/dev.py
STORAGES = {
    "default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
}

# settings/prod.py
STORAGES = {
    "default": {"BACKEND": "storages.backends.s3.S3Storage"},
}

# Code stays the same:
from django.core.files.storage import default_storage
default_storage.save("file.txt", content)

Credentials from Environment Variables

Problem: A leaked settings file = leaked cloud credentials.

# ❌ DON'T: Hardcoded credentials
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "access_key": "AKIAIOSFODNN7EXAMPLE",  # Leaked = compromised
            "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        },
    },
}

Solution: Environment variables or IAM roles.

# ✅ DO: Environment variables
import os
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "access_key": os.environ["AWS_ACCESS_KEY_ID"],
            "secret_key": os.environ["AWS_SECRET_ACCESS_KEY"],
        },
    },
}

# ✅ BETTER: IAM role (EC2/ECS) - no credentials needed
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "my-app-media",
            # No access_key/secret_key → uses IAM role
        },
    },
}

---

ACL and Authorization Decisions

default_acl: When to Make Files Public

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "default_acl": "public-read",  # or None for private
        },
    },
}

Set public-read only for genuinely public assets:

• Static website assets (images, CSS, JS)
• Publicly accessible user avatars
• Marketing materials

Keep None (private) for:

• User documents (PDFs, spreadsheets)
• Financial records
• Any data requiring authorization

Why it matters: Public buckets are a common security incident vector. Obscurity is not security.

querystring_auth: Signed URLs

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "querystring_auth": True,   # Require signatures (private)
            "querystring_expire": 3600, # 1 hour expiry
        },
    },
}

When querystring_auth = False:

• Bucket has a public ACL policy
• CDN frontends the bucket (CloudFront with OAI)
• Static assets that never expire

Critical: If querystring_auth = False, signed URLs must carry their own expiry via url() parameters:

# Private bucket with querystring_auth=False (e.g., CDN-protected)
# You MUST pass expiry explicitly:
url = storage.url("file.pdf", expire=3600)  # Expires in 1 hour

When you need signed URLs:

# Download as attachment (forces download with original filename)
url = storage.url(
    "documents/report.pdf",
    parameters={"ResponseContentDisposition": "attachment; filename=report.pdf"}
)

# Temporary access (e.g., password-protected page)
url = storage.url("private/secret.pdf", expire=1800)  # 30 minutes

---

CDN Integration Pattern

Private Bucket + CDN Origin

Architecture:

1. S3 bucket is private (querystring_auth = True)
2. CloudFront (or other CDN) fronts the bucket
3. Public users access via CDN URL
4. Signed URLs bypass CDN for direct S3 access (or use CloudFront signed URLs)

# settings.py
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "my-app-private",
            "region_name": "us-east-1",
            "querystring_auth": True,  # Private bucket
            "custom_domain": "d1234abcdef.cloudfront.net",  # CDN domain
        },
    },
}

Cache-Control Headers

Differentiate cacheable public assets from private content:

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "object_parameters": {
                # Public static assets: cache aggressively
                "CacheControl": "public, max-age=31536000, immutable",
            },
        },
    },
}

# Per-file cache control (overrides default)
from django.core.files.storage import default_storage

# Public image - cache for a year
default_storage.save("images/logo.png", content)
# S3 object params can be set via custom storage subclass or boto3 directly

# Temporary report - don't cache
default_storage.save("reports/daily.pdf", content)

CDN Invalidation

CloudFront:

import boto3

def invalidate_cloudfront(paths):
    """
    Invalidate CloudFront cache for given paths.
    Cost: $0.005 per path, 1000 paths/month free.
    """
    cloudfront = boto3.client("cloudfront")
    distribution_id = "E1234ABCDEF"
    
    cloudfront.create_invalidation(
        DistributionId=distribution_id,
        InvalidationBatch={
            "Id": f"invalidation-{uuid.uuid4().hex[:8]}",
            "Paths": {
                "Quantity": len(paths),
                "Items": paths,
            },
        },
    )

# Invalidate specific files
invalidate_cloudfront(["/images/logo.png", "/reports/daily.pdf"])

# Invalidate by prefix (expensive - use sparingly)
invalidate_cloudfront(["/reports/*"])  # Wildcards supported in CloudFront

Azure CDN:

# Azure CDN Classic invalidation
from azure.cdn.management import CdnManagementClient

client = CdnManagementClient(credential, subscription_id)
client.profiles.begin_create_or_update(...)

# Invalidate by path prefix
client.endpoints.invalidate(
    resource_group_name="my-rg",
    profile_name="my-profile",
    endpoint_name="my-endpoint",
    content_paths=["/images/*", "/css/*"],
)

Source: AWS CloudFront Invalidations | Azure CDN Invalidation

---

When Not to Use django-storages

Local Development

Use Django's FileSystemStorage instead:

# settings/dev.py
STORAGES = {
    "default": {
        "BACKEND": "django.core.files.storage.FileSystemStorage",
        "OPTIONS": {
            "location": "/tmp/media",  # or MEDIA_ROOT
        },
    },
}

Why: No network calls, no credentials, instant file access for debugging.

Static Assets Pipeline

Use a build pipeline instead of storage backend for assets that ship with code:

• Webpack/Vite/Rollup output
• Compiled CSS/JS
• Optimized images from source control

Why: These are versioned at deploy time, not user-uploaded. Use collectstatic with ManifestStaticFilesStorage.

STORAGES = {
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
    },
}

Single Bucket, No Django Integration Needed

Use the SDK directly:

import boto3

s3 = boto3.client("s3")
s3.upload_file("local.txt", "my-bucket", "remote.txt")

Why: If you only use one bucket and never swap backends, django-storages adds indirection without benefit.

Presigned URL Uploads from Browser

Bypass Django entirely:

# View that returns upload credentials
def get_upload_url(request):
    s3 = boto3.client("s3")
    url = s3.generate_presigned_url(
        "put_object",
        Params={
            "Bucket": "my-bucket",
            "Key": f"uploads/{uuid.uuid4()}.pdf",
            "ContentType": "application/pdf",
        },
        ExpiresIn=3600,
    )
    return JsonResponse({"url": url, "key": key})

# Browser uploads directly to S3
# POST { url, key } → client PUTs file to S3

Why: Large file uploads shouldn't proxy through Django. Use presigned URLs for direct browser-to-storage uploads.

---

Configuration Settings (When You Need Them)

S3-Specific Options

Most settings have sensible defaults. Use these only when you need specific behavior:

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            # Only set if you need non-default storage class
            "object_parameters": {
                "StorageClass": "STANDARD_IA",  # Infrequent access (cheaper)
            },
            
            # Only set if you want to prevent overwrites
            "file_overwrite": False,  # Default: True (overwrites same name)
            
            # Only set if using KMS encryption
            "object_parameters": {
                "SSEKMSKeyId": "arn:aws:kms:...",
            },
            
            # Only set if bucket is in different region
            "region_name": "eu-west-1",  # Default: us-east-1
            
            # Only set if using custom domain/CDN
            "custom_domain": "cdn.example.com",
            
            # Only set if you need non-default ACL
            "default_acl": "public-read",  # Default: None (private)
            
            # Only set if you need URL prefix in bucket
            "location": "media/",  # All files stored under media/
        },
    },
}

Source: django-storages S3 Backend Docs

---

Development and Testing

Environment Switching

# settings.py
from pathlib import Path

env = os.environ.get("DJANGO_ENV", "development")

if env == "development":
    STORAGES = {
        "default": {
            "BACKEND": "django.core.files.storage.FileSystemStorage",
        },
    }
else:
    STORAGES = {
        "default": {
            "BACKEND": "storages.backends.s3.S3Storage",
            "OPTIONS": {
                "bucket_name": os.environ["AWS_STORAGE_BUCKET_NAME"],
            },
        },
    }

Testing with Mock

from unittest.mock import patch
from django.core.files.uploadedfile import SimpleUploadedFile
from django.test import TestCase

class DocumentUploadTest(TestCase):
    @patch("django.core.files.storage.default_storage")
    def test_upload(self, mock_storage):
        mock_storage.save.return_value = "uploads/file.txt"
        
        file = SimpleUploadedFile("file.txt", b"content")
        path = default_storage.save("file.txt", file)
        
        self.assertEqual(path, "uploads/file.txt")
        mock_storage.save.assert_called_once()

---

Deep Dives

Load these reference files for detailed backend-specific guidance:

• Provider Configuration — references/providers.md — Amazon S3, Google Cloud Storage, Azure Blob Storage detailed setup
• Models and Code — references/models-code-urls.md — FileField usage, direct backend access, uploads, signed URLs
• Advanced Cloud Features — references/cloud-features.md — Encryption, CDN, caching, multipart uploads (load when implementing provider-specific features)

---

References

• Documentation: https://django-storages.readthedocs.io/
• GitHub: https://github.com/jschneier/django-storages
• PyPI: https://pypi.org/project/django-storages/
• S3 Boto3 Docs: https://boto3.readthedocs.io/
• GCS Python Client: https://google-cloud-python.readthedocs.io/