polars

Polars Skill

Polars DataFrame library for high-performance data manipulation in Python. Covers lazy/eager execution, expressions, I/O (CSV, Parquet, JSON, database), aggregations, joins, string/datetime operations, pandas/NumPy interop, and performance optimization. Use when working with Polars DataFrames, migrating from pandas, reading Parquet files, or optimizing data pipeline performance.

Comprehensive skill for high-performance data manipulation with Polars. Use decision trees below to find the right guidance, then load detailed references.

What is Polars?

Polars is a fast DataFrame library for Python (and Rust):

• Fast: Written in Rust, optimized for modern CPUs with SIMD and parallelism
• Lazy Evaluation: Build query plans that get optimized before execution
• Expressive: Powerful expression API for complex transformations
• Memory Efficient: Columnar format, streaming for larger-than-memory data
• No Dependencies: Pure Rust core, no NumPy/Pandas required

Version Notes

This skill targets Polars 1.x (tested with 1.37.1). Key changes from 0.x:

• apply renamed to map_elements (0.19+)
• groupby renamed to group_by (0.19+)
• melt renamed to unpivot (1.0+)
• Streaming engine improvements in 1.x
• pl.Utf8 is now pl.String (1.0+, Utf8 still works as alias)

How to Use This Skill

Reference File Structure

Each topic in ./references/ contains focused documentation:

File	Purpose	When to Read
quickstart.md	Installation, concepts, first DataFrame	Starting with Polars
dataframes-series.md	Creation, selection, filtering, modification	Basic data manipulation
io-data.md	CSV, Parquet, JSON, database I/O	Loading/saving data
expressions.md	Expression system, contexts, chaining	Understanding Polars idioms
aggregations-grouping.md	GroupBy, window functions, statistics	Summarizing data
joins-concat.md	Joins, concatenation, pivot/unpivot	Combining DataFrames
strings-datetime-categorical.md	String ops, datetime, categoricals	Type-specific operations
performance.md	Lazy execution, optimization, anti-patterns	Making code faster
interop.md	Pandas, NumPy, PyArrow, DuckDB	Working with other tools
gotchas.md	Common errors, anti-patterns, migration	Debugging issues

Reading Order

1. New to Polars? Start with quickstart.md then expressions.md
2. Coming from Pandas? Read quickstart.md, expressions.md, then interop.md
3. Performance issues? Check performance.md first

Quick Decision Trees

"I need to get started"

Getting started?
├─ Install Polars → ./references/quickstart.md
├─ Create first DataFrame → ./references/quickstart.md
├─ Understand lazy vs eager → ./references/quickstart.md
├─ Learn expression syntax → ./references/expressions.md
└─ Coming from Pandas → ./references/interop.md

"I need to load or save data"

Loading/saving data?
├─ Read CSV file → ./references/io-data.md
├─ Read Parquet (recommended) → ./references/io-data.md
├─ Read JSON/NDJSON → ./references/io-data.md
├─ Read from database → ./references/io-data.md
├─ Read multiple files (glob) → ./references/io-data.md
├─ Write to file → ./references/io-data.md
└─ Larger-than-memory data → ./references/performance.md

"I need to filter or select data"

Filtering/selecting?
├─ Select columns by name → ./references/dataframes-series.md
├─ Select by pattern/regex → ./references/dataframes-series.md
├─ Select by data type → ./references/dataframes-series.md
├─ Filter rows by condition → ./references/dataframes-series.md
├─ Filter with multiple conditions → ./references/dataframes-series.md
├─ Handle null values → ./references/dataframes-series.md
└─ Add/modify columns → ./references/dataframes-series.md

"I need to aggregate or group data"

Aggregating data?
├─ Basic statistics (sum, mean, etc.) → ./references/aggregations-grouping.md
├─ Group by columns → ./references/aggregations-grouping.md
├─ Multiple aggregations → ./references/aggregations-grouping.md
├─ Window functions (over) → ./references/aggregations-grouping.md
├─ Rolling/moving averages → ./references/aggregations-grouping.md
├─ Cumulative operations → ./references/aggregations-grouping.md
└─ Ranking within groups → ./references/aggregations-grouping.md

"I need to combine DataFrames"

Combining data?
├─ Join two DataFrames → ./references/joins-concat.md
├─ Left/right/outer join → ./references/joins-concat.md
├─ Anti-join (not in) → ./references/joins-concat.md
├─ Concatenate vertically → ./references/joins-concat.md
├─ Pivot (long to wide) → ./references/joins-concat.md
└─ Unpivot/melt (wide to long) → ./references/joins-concat.md

"I need better performance"

Performance issues?
├─ Use lazy evaluation → ./references/performance.md
├─ Avoid row iteration → ./references/performance.md
├─ Reduce memory usage → ./references/performance.md
├─ Process large files → ./references/performance.md
├─ Optimize query plan → ./references/performance.md
└─ Common anti-patterns → ./references/performance.md

"Something isn't working"

Having issues?
├─ Type errors → ./references/gotchas.md
├─ Null handling → ./references/gotchas.md
├─ Expression context errors → ./references/gotchas.md
├─ String operations → ./references/strings-datetime-categorical.md
├─ Date parsing issues → ./references/strings-datetime-categorical.md
├─ Performance problems → ./references/gotchas.md
├─ Pandas migration issues → ./references/gotchas.md
├─ Memory errors → ./references/gotchas.md
└─ General troubleshooting → ./references/gotchas.md

File-First Execution in Research Workflows

Important: In data research pipelines (see CLAUDE.md), Polars transformations are executed through script files, not interactively. This ensures auditability and reproducibility.

The pattern:

1. Write transformation code to scripts/stage{N}_{type}/{step}_{task-name}.py
2. Execute via Bash with automatic output capture wrapper script
3. Validation results get automatically embedded in scripts as comments
4. If failed, create versioned copy for fixes

Closely read agent_reference/SCRIPT_EXECUTION_REFERENCE.md for the mandatory file-first execution protocol covering complete code file writing, output capture, and file versioning rules.

See:

• agent_reference/SCRIPT_EXECUTION_REFERENCE.md — Script execution protocol and format with validation

The examples below show Polars syntax. In research workflows, wrap them in scripts following the file-first pattern.

---

Quick Reference

Essential Import

import polars as pl
import polars.selectors as cs  # For column selection by type

Lazy vs Eager (One-Liner)

# Eager: immediate execution
df = pl.read_csv("data.csv")

# Lazy: deferred, optimized execution (preferred for large data)
lf = pl.scan_csv("data.csv")
df = lf.collect()  # Execute when ready

Core Expression Patterns

# Select columns
df.select("a", "b")
df.select(pl.col("a"), pl.col("b"))
df.select(pl.all().exclude("id"))

# Filter rows
df.filter(pl.col("a") > 10)
df.filter((pl.col("a") > 10) & (pl.col("b") == "x"))

# Add/modify columns
df.with_columns(
    (pl.col("a") * 2).alias("a_doubled"),
    pl.col("b").str.to_uppercase().alias("b_upper")
)

# Conditional column
df.with_columns(
    pl.when(pl.col("a") > 10)
      .then(pl.lit("high"))
      .otherwise(pl.lit("low"))
      .alias("category")
)

# Group and aggregate
df.group_by("category").agg(
    pl.col("value").sum().alias("total"),
    pl.col("value").mean().alias("average"),
    pl.len().alias("count")
)

Essential Functions

Function	Purpose
pl.col("name")	Reference a column
pl.lit(value)	Literal value
pl.all()	All columns
pl.exclude("col")	All except specified
pl.len()	Row count
pl.when().then().otherwise()	Conditional logic
.alias("name")	Rename result
.cast(pl.Int64)	Convert type

Common Data Types

Type	Description
pl.Int64, pl.Int32	Integers
pl.Float64, pl.Float32	Floats
pl.String (or pl.Utf8)	Strings
pl.Boolean	True/False
pl.Date, pl.Datetime	Dates and timestamps
pl.Duration	Time differences
pl.Categorical	Categorical strings
pl.List	List of values
pl.Struct	Named fields

Quick Cheatsheet

# I/O
df = pl.read_csv/parquet/json("file")
lf = pl.scan_csv/parquet/ndjson("file")  # Lazy
df.write_csv/parquet/json("file")

# Selection
df.select("a", "b")
df.select(cs.numeric())  # By type

# Filtering
df.filter(pl.col("a") > 1)

# Aggregation
df.group_by("key").agg(pl.col("val").sum())

# Joining
df1.join(df2, on="key", how="left")

# Sorting
df.sort("col", descending=True)

# Lazy execution
lf.collect()  # Run query
lf.explain()  # Show plan

Topic Index

Topic	Reference File
Installation	./references/quickstart.md
DataFrame Creation	./references/quickstart.md
Lazy vs Eager	./references/quickstart.md
Column Selection	./references/dataframes-series.md
Row Filtering	./references/dataframes-series.md
Adding Columns	./references/dataframes-series.md
CSV Files	./references/io-data.md
Parquet Files	./references/io-data.md
Database Connections	./references/io-data.md
Expressions	./references/expressions.md
Method Chaining	./references/expressions.md
Contexts	./references/expressions.md
GroupBy	./references/aggregations-grouping.md
Window Functions	./references/aggregations-grouping.md
Rolling Windows	./references/aggregations-grouping.md
Joins	./references/joins-concat.md
Concatenation	./references/joins-concat.md
Pivot/Unpivot	./references/joins-concat.md
String Operations	./references/strings-datetime-categorical.md
Datetime Handling	./references/strings-datetime-categorical.md
Categorical Data	./references/strings-datetime-categorical.md
Query Optimization	./references/performance.md
Memory Management	./references/performance.md
Anti-Patterns	./references/performance.md
Pandas Conversion	./references/interop.md
NumPy Integration	./references/interop.md
DuckDB Integration	./references/interop.md
Type Errors	./references/gotchas.md
qcut Label Gotcha	./references/gotchas.md
Null Handling Issues	./references/gotchas.md
Expression Context Errors	./references/gotchas.md
Performance Anti-Patterns	./references/gotchas.md
Migration from Pandas	./references/gotchas.md
Memory Issues	./references/gotchas.md

Citation

When this library is used as a primary analytical tool, include in the report's Software & Tools references:

Vink, R. et al. Polars: Blazingly fast DataFrames [Computer software]. https://pola.rs/

Cite when: Polars is the core data processing engine for the analysis (typically always true in DAAF pipelines). Do not cite when: Only used for trivial file I/O in a script primarily using another tool.