Polars

Polars Skill

Polars DataFrame library for high-performance data manipulation in Python. Covers lazy/eager execution, expressions, I/O (CSV, Parquet, JSON, database), aggregations, joins, string/datetime operations, pandas/NumPy interop, and performance optimization. Use when working with Polars DataFrames, migrating from pandas, reading Parquet files, or optimizing data pipeline performance.

Comprehensive skill for high-performance data manipulation with Polars. Use decision trees below to find the right guidance, then load detailed references.

What is Polars?

Polars is a fast DataFrame library for Python (and Rust):

• Fast: Written in Rust, optimized for modern CPUs with SIMD and parallelism
• Lazy Evaluation: Build query plans that get optimized before execution
• Expressive: Powerful expression API for complex transformations
• Memory Efficient: Columnar format, streaming for larger-than-memory data
• No Dependencies: Pure Rust core, no NumPy/Pandas required

Version Notes

This skill targets Polars 1.x (tested with 1.37.1). Key changes from 0.x:

• apply renamed to map_elements (0.19+)
• groupby renamed to group_by (0.19+)
• melt renamed to unpivot (1.0+)
• Streaming engine improvements in 1.x
• pl.Utf8 is now pl.String (1.0+, Utf8 still works as alias)

How to Use This Skill

Reference File Structure

Each topic in ./references/ contains focused documentation:

Reading Order

1. New to Polars? Start with quickstart.md then expressions.md 2. Coming from Pandas? Read quickstart.md, expressions.md, then interop.md 3. Performance issues? Check performance.md first

Quick Decision Trees

"I need to get started"

Getting started?
├─ Install Polars → ./references/quickstart.md
├─ Create first DataFrame → ./references/quickstart.md
├─ Understand lazy vs eager → ./references/quickstart.md
├─ Learn expression syntax → ./references/expressions.md
└─ Coming from Pandas → ./references/interop.md

"I need to load or save data"

Loading/saving data?
├─ Read CSV file → ./references/io-data.md
├─ Read Parquet (recommended) → ./references/io-data.md
├─ Read JSON/NDJSON → ./references/io-data.md
├─ Read from database → ./references/io-data.md
├─ Read multiple files (glob) → ./references/io-data.md
├─ Write to file → ./references/io-data.md
└─ Larger-than-memory data → ./references/performance.md

"I need to filter or select data"

Filtering/selecting?
├─ Select columns by name → ./references/dataframes-series.md
├─ Select by pattern/regex → ./references/dataframes-series.md
├─ Select by data type → ./references/dataframes-series.md
├─ Filter rows by condition → ./references/dataframes-series.md
├─ Filter with multiple conditions → ./references/dataframes-series.md
├─ Handle null values → ./references/dataframes-series.md
└─ Add/modify columns → ./references/dataframes-series.md

"I need to aggregate or group data"

Aggregating data?
├─ Basic statistics (sum, mean, etc.) → ./references/aggregations-grouping.md
├─ Group by columns → ./references/aggregations-grouping.md
├─ Multiple aggregations → ./references/aggregations-grouping.md
├─ Window functions (over) → ./references/aggregations-grouping.md
├─ Rolling/moving averages → ./references/aggregations-grouping.md
├─ Cumulative operations → ./references/aggregations-grouping.md
└─ Ranking within groups → ./references/aggregations-grouping.md

"I need to combine DataFrames"

Combining data?
├─ Join two DataFrames → ./references/joins-concat.md
├─ Left/right/outer join → ./references/joins-concat.md
├─ Anti-join (not in) → ./references/joins-concat.md
├─ Concatenate vertically → ./references/joins-concat.md
├─ Pivot (long to wide) → ./references/joins-concat.md
└─ Unpivot/melt (wide to long) → ./references/joins-concat.md

"I need better performance"

Performance issues?
├─ Use lazy evaluation → ./references/performance.md
├─ Avoid row iteration → ./references/performance.md
├─ Reduce memory usage → ./references/performance.md
├─ Process large files → ./references/performance.md
├─ Optimize query plan → ./references/performance.md
└─ Common anti-patterns → ./references/performance.md

"Something isn't working"

Having issues?
├─ Type errors → ./references/gotchas.md
├─ Null handling → ./references/gotchas.md
├─ Expression context errors → ./references/gotchas.md
├─ String operations → ./references/strings-datetime-categorical.md
├─ Date parsing issues → ./references/strings-datetime-categorical.md
├─ Performance problems → ./references/gotchas.md
├─ Pandas migration issues → ./references/gotchas.md
├─ Memory errors → ./references/gotchas.md
└─ General troubleshooting → ./references/gotchas.md

File-First Execution in Research Workflows

Important: In data research pipelines (see CLAUDE.md), Polars transformations are executed through script files, not interactively. This ensures auditability and reproducibility.

The pattern: 1. Write transformation code to scripts/stage{N}_{type}/{step}_{task-name}.py 2. Execute via Bash with automatic output capture wrapper script 3. Validation results get automatically embedded in scripts as comments 4. If failed, create versioned copy for fixes

Closely read agent_reference/SCRIPT_EXECUTION_REFERENCE.md for the mandatory file-first execution protocol covering complete code file writing, output capture, and file versioning rules.

See:

• agent_reference/SCRIPT_EXECUTION_REFERENCE.md — Script execution protocol and format with validation

The examples below show Polars syntax. In research workflows, wrap them in scripts following the file-first pattern.

---

Quick Reference

Essential Import

import polars as pl
import polars.selectors as cs  # For column selection by type

Lazy vs Eager (One-Liner)

# Eager: immediate execution
df = pl.read_csv("data.csv")

# Lazy: deferred, optimized execution (preferred for large data)
lf = pl.scan_csv("data.csv")
df = lf.collect()  # Execute when ready

Core Expression Patterns

# Select columns
df.select("a", "b")
df.select(pl.col("a"), pl.col("b"))
df.select(pl.all().exclude("id"))

# Filter rows
df.filter(pl.col("a") > 10)
df.filter((pl.col("a") > 10) & (pl.col("b") == "x"))

# Add/modify columns
df.with_columns(
    (pl.col("a") * 2).alias("a_doubled"),
    pl.col("b").str.to_uppercase().alias("b_upper")
)

# Conditional column
df.with_columns(
    pl.when(pl.col("a") > 10)
      .then(pl.lit("high"))
      .otherwise(pl.lit("low"))
      .alias("category")
)

# Group and aggregate
df.group_by("category").agg(
    pl.col("value").sum().alias("total"),
    pl.col("value").mean().alias("average"),
    pl.len().alias("count")
)

Essential Functions

Common Data Types

Quick Cheatsheet

# I/O
df = pl.read_csv/parquet/json("file")
lf = pl.scan_csv/parquet/ndjson("file")  # Lazy
df.write_csv/parquet/json("file")

# Selection
df.select("a", "b")
df.select(cs.numeric())  # By type

# Filtering
df.filter(pl.col("a") > 1)

# Aggregation
df.group_by("key").agg(pl.col("val").sum())

# Joining
df1.join(df2, on="key", how="left")

# Sorting
df.sort("col", descending=True)

# Lazy execution
lf.collect()  # Run query
lf.explain()  # Show plan

Topic Index

Citation

When this library is used as a primary analytical tool, include in the report's Software & Tools references:

Vink, R. et al. Polars: Blazingly fast DataFrames [Computer software]. https://pola.rs/

Cite when: Polars is the core data processing engine for the analysis (typically always true in DAAF pipelines). Do not cite when: Only used for trivial file I/O in a script primarily using another tool.

Aggregations & Grouping

Basic Aggregations

On Entire DataFrame

# Single aggregation
df.select(pl.col("value").sum())
df.select(pl.col("value").mean())

# Multiple aggregations
df.select(
    pl.col("value").sum().alias("total"),
    pl.col("value").mean().alias("average"),
    pl.col("value").std().alias("std_dev"),
    pl.col("value").min().alias("minimum"),
    pl.col("value").max().alias("maximum"),
    pl.col("value").count().alias("non_null_count"),
    pl.len().alias("row_count")
)

Available Aggregation Functions

GroupBy Operations

Basic GroupBy

df.group_by("category").agg(
    pl.col("value").sum().alias("total"),
    pl.col("value").mean().alias("average"),
    pl.len().alias("count")
)

Multiple Grouping Columns

df.group_by("region", "category").agg(
    pl.col("sales").sum().alias("total_sales"),
    pl.col("quantity").sum().alias("total_quantity")
)

# Or as list
df.group_by(["region", "category"]).agg(...)

Maintain Order

# By default, group_by doesn't preserve order
# Use maintain_order=True to keep original order
df.group_by("category", maintain_order=True).agg(
    pl.col("value").sum()
)

Multiple Aggregations on Same Column

df.group_by("category").agg(
    pl.col("value").sum().alias("sum"),
    pl.col("value").mean().alias("mean"),
    pl.col("value").std().alias("std"),
    pl.col("value").min().alias("min"),
    pl.col("value").max().alias("max"),
    pl.col("value").count().alias("count"),
    pl.col("value").quantile(0.25).alias("q25"),
    pl.col("value").quantile(0.75).alias("q75"),
)

Aggregating Different Columns

df.group_by("category").agg(
    pl.col("revenue").sum().alias("total_revenue"),
    pl.col("quantity").sum().alias("total_quantity"),
    pl.col("customer_id").n_unique().alias("unique_customers"),
    pl.col("discount").mean().alias("avg_discount")
)

Collecting Values into Lists

df.group_by("category").agg(
    pl.col("product_id"),           # Collect all values into list
    pl.col("product_id").alias("products"),  # Same with alias
    pl.col("value").sort().alias("sorted_values")
)

First/Last per Group

df.group_by("category").agg(
    pl.col("timestamp").first().alias("first_ts"),
    pl.col("timestamp").last().alias("last_ts"),
    pl.col("value").first().alias("first_value"),
    pl.col("value").last().alias("last_value")
)

# First/last with sorting
df.sort("timestamp").group_by("category").agg(
    pl.col("value").first().alias("earliest_value"),
    pl.col("value").last().alias("latest_value")
)

Window Functions (over)

Window functions compute values within groups without reducing rows.

Basic Window

# Sum per category (added as column, keeps all rows)
df.with_columns(
    pl.col("value").sum().over("category").alias("category_total")
)

# Mean per category
df.with_columns(
    pl.col("value").mean().over("category").alias("category_avg")
)

Multiple Grouping Columns

df.with_columns(
    pl.col("value").sum().over(["region", "category"]).alias("group_total")
)

Common Window Operations

df.with_columns(
    # Aggregations over group
    pl.col("value").sum().over("category").alias("group_sum"),
    pl.col("value").mean().over("category").alias("group_mean"),
    pl.col("value").min().over("category").alias("group_min"),
    pl.col("value").max().over("category").alias("group_max"),
    pl.col("value").count().over("category").alias("group_count"),
    
    # Percentage of group
    (pl.col("value") / pl.col("value").sum().over("category") * 100)
        .alias("pct_of_category"),
    
    # Deviation from group mean
    (pl.col("value") - pl.col("value").mean().over("category"))
        .alias("deviation_from_mean")
)

Ranking

df.with_columns(
    # Rank within group (1, 2, 3, ...)
    pl.col("value").rank().over("category").alias("rank"),
    
    # Dense rank (no gaps)
    pl.col("value").rank(method="dense").over("category").alias("dense_rank"),
    
    # Ordinal rank (unique values)
    pl.col("value").rank(method="ordinal").over("category").alias("ordinal_rank"),
    
    # Descending rank
    pl.col("value").rank(descending=True).over("category").alias("rank_desc")
)

Row Numbers

df.with_columns(
    # Row number within group
    pl.col("id").cum_count().over("category").alias("row_num"),
    
    # Using arange
    pl.int_range(1, pl.len() + 1).over("category").alias("row_num2")  # pl.arange deprecated; use pl.int_range
)

Shift/Lead/Lag

df.with_columns(
    # Previous value (lag)
    pl.col("value").shift(1).over("category").alias("prev_value"),
    
    # Next value (lead)
    pl.col("value").shift(-1).over("category").alias("next_value"),
    
    # Difference from previous
    (pl.col("value") - pl.col("value").shift(1).over("category"))
        .alias("diff_from_prev")
)

Cumulative Operations

df.with_columns(
    pl.col("value").cum_sum().over("category").alias("running_total"),
    pl.col("value").cum_max().over("category").alias("running_max"),
    pl.col("value").cum_min().over("category").alias("running_min"),
    pl.col("value").cum_count().over("category").alias("running_count"),
    pl.col("value").cum_prod().over("category").alias("running_product"),
)

# Without grouping
df.with_columns(
    pl.col("value").cum_sum().alias("cumulative_sum")
)

Rolling Windows

By Row Count

df.with_columns(
    pl.col("value").rolling_mean(window_size=7).alias("rolling_avg_7"),
    pl.col("value").rolling_sum(window_size=7).alias("rolling_sum_7"),
    pl.col("value").rolling_std(window_size=7).alias("rolling_std_7"),
    pl.col("value").rolling_min(window_size=7).alias("rolling_min_7"),
    pl.col("value").rolling_max(window_size=7).alias("rolling_max_7"),
)

# With minimum periods
df.with_columns(
    pl.col("value").rolling_mean(window_size=7, min_periods=3).alias("rolling_avg")
)

# Center window
df.with_columns(
    pl.col("value").rolling_mean(window_size=7, center=True).alias("centered_avg")
)

Within Groups

df.with_columns(
    pl.col("value").rolling_mean(window_size=7).over("category").alias("rolling_avg")
)

By Time Period

# DataFrame must be sorted by the time column
df.sort("date").rolling(
    index_column="date",
    period="7d"           # 7-day window
).agg(
    pl.col("value").mean().alias("weekly_avg"),
    pl.col("value").sum().alias("weekly_sum")
)

# Group by category within rolling window
df.sort("date").rolling(
    index_column="date",
    period="7d",
    group_by="category"
).agg(
    pl.col("value").mean().alias("weekly_avg")
)

Time Periods

Group By Dynamic (Time-Based Grouping)

# Resample by time period
df.sort("timestamp").group_by_dynamic(
    "timestamp",
    every="1d"              # Daily
).agg(
    pl.col("value").sum().alias("daily_total"),
    pl.col("value").mean().alias("daily_avg")
)

# Weekly aggregation
df.sort("timestamp").group_by_dynamic(
    "timestamp",
    every="1w",
    start_by="monday"       # Week starts Monday
).agg(...)

# With additional grouping
df.sort("timestamp").group_by_dynamic(
    "timestamp",
    every="1d",
    group_by="category"
).agg(...)

# Offset the start
df.sort("timestamp").group_by_dynamic(
    "timestamp",
    every="1d",
    offset="-6h"            # Offset by 6 hours
).agg(...)

Common Patterns

Top N per Group

# Top 3 per category by value
df.sort("value", descending=True).group_by("category").head(3)

# Or using over + filter
df.with_columns(
    pl.col("value").rank(descending=True).over("category").alias("rank")
).filter(pl.col("rank") <= 3)

Percentage of Total

df.with_columns(
    (pl.col("value") / pl.col("value").sum() * 100).alias("pct_of_total"),
    (pl.col("value") / pl.col("value").sum().over("category") * 100)
        .alias("pct_of_category")
)

Year-over-Year Comparison

df.with_columns(
    pl.col("timestamp").dt.year().alias("year")
).group_by(["category", "year"]).agg(
    pl.col("value").sum().alias("annual_total")
).sort("category", "year").with_columns(
    (pl.col("annual_total") - pl.col("annual_total").shift(1).over("category"))
        .alias("yoy_change")
)

Moving Average with Fill

df.with_columns(
    pl.col("value")
      .rolling_mean(window_size=7, min_periods=1)
      .alias("rolling_avg")
)

Summary

DataFrames & Series

DataFrame Creation

From Dictionary

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "active": [True, False, True]
})

With Schema Control

df = pl.DataFrame(
    {"id": [1, 2], "value": [10.5, 20.3]},
    schema={"id": pl.Int32, "value": pl.Float32}
)

# Override inferred types
df = pl.DataFrame(
    {"a": [1, 2, 3]},
    schema_overrides={"a": pl.Int16}
)

From Rows (List of Dicts)

df = pl.DataFrame([
    {"name": "Alice", "age": 25},
    {"name": "Bob", "age": 30}
])

From NumPy/Sequences

import numpy as np

df = pl.DataFrame({
    "a": np.array([1, 2, 3]),
    "b": range(3),
    "c": [1.0, 2.0, 3.0]
})

Series Creation

# From list
s = pl.Series("name", [1, 2, 3])

# With explicit type
s = pl.Series("name", [1, 2, 3], dtype=pl.Float64)

# From range
s = pl.Series("nums", range(10))

Column Selection

By Name

# Single column (returns Series)
df["name"]
df.get_column("name")

# Multiple columns (returns DataFrame)
df.select("name", "age")
df.select(["name", "age"])
df.select(pl.col("name"), pl.col("age"))

All Columns / Exclude

df.select(pl.all())                    # All columns
df.select(pl.all().exclude("id"))      # All except "id"
df.select(pl.exclude("id", "temp"))    # Exclude multiple

By Pattern (Regex)

# Columns starting with "col_"
df.select(pl.col("^col_.*$"))

# Columns ending with "_id"
df.select(pl.col("^.*_id$"))

# Columns containing "price"
df.select(pl.col("^.*price.*$"))

By Data Type (Selectors)

import polars.selectors as cs

df.select(cs.numeric())          # All numeric columns
df.select(cs.string())           # All string columns
df.select(cs.temporal())         # Date/datetime/time/duration
df.select(cs.boolean())          # Boolean columns
df.select(cs.categorical())      # Categorical columns

# Combine selectors
df.select(cs.numeric() | cs.string())    # OR
df.select(cs.numeric() & ~cs.float())    # AND NOT
df.select(cs.numeric() - cs.integer())   # Difference

# Exclude from selector
df.select(cs.numeric().exclude("id"))

First/Last Columns

df.select(pl.first())           # First column
df.select(pl.last())            # Last column
df.select(pl.nth(0, 2, 4))      # By index positions

Row Filtering

Single Condition

df.filter(pl.col("age") > 25)
df.filter(pl.col("name") == "Alice")
df.filter(pl.col("active"))              # Boolean column
df.filter(~pl.col("active"))             # NOT (negation)

Multiple Conditions

# AND: both conditions must be true
df.filter((pl.col("age") > 25) & (pl.col("active")))

# OR: either condition true
df.filter((pl.col("age") > 30) | (pl.col("name") == "Alice"))

# Complex combinations
df.filter(
    ((pl.col("age") > 25) & (pl.col("active"))) |
    (pl.col("name") == "Admin")
)

Membership (IN)

df.filter(pl.col("name").is_in(["Alice", "Bob"]))
df.filter(~pl.col("name").is_in(["Alice", "Bob"]))  # NOT IN

String Conditions

df.filter(pl.col("name").str.contains("li"))
df.filter(pl.col("name").str.starts_with("A"))
df.filter(pl.col("name").str.ends_with("e"))

Null Handling

df.filter(pl.col("value").is_null())
df.filter(pl.col("value").is_not_null())
df.filter(pl.col("value").is_nan())        # For floats
df.filter(pl.col("value").is_not_nan())

Between

df.filter(pl.col("age").is_between(25, 35))
df.filter(pl.col("age").is_between(25, 35, closed="left"))   # [25, 35)
df.filter(pl.col("age").is_between(25, 35, closed="right"))  # (25, 35]
df.filter(pl.col("age").is_between(25, 35, closed="none"))   # (25, 35)

Adding and Modifying Columns

with_columns

# Add new columns
df.with_columns(
    (pl.col("age") + 1).alias("age_next_year"),
    (pl.col("score") * 1.1).alias("boosted_score"),
    pl.lit("active").alias("status")
)

# Modify existing column (same name)
df.with_columns(
    pl.col("name").str.to_uppercase()  # No alias = replaces "name"
)

Conditional Columns (when/then/otherwise)

# Simple if-else
df.with_columns(
    pl.when(pl.col("age") >= 30)
      .then(pl.lit("senior"))
      .otherwise(pl.lit("junior"))
      .alias("level")
)

# Multiple conditions (if-elif-else)
df.with_columns(
    pl.when(pl.col("score") >= 90)
      .then(pl.lit("A"))
      .when(pl.col("score") >= 80)
      .then(pl.lit("B"))
      .when(pl.col("score") >= 70)
      .then(pl.lit("C"))
      .otherwise(pl.lit("F"))
      .alias("grade")
)

# Conditional with column values
df.with_columns(
    pl.when(pl.col("active"))
      .then(pl.col("salary") * 1.1)
      .otherwise(pl.col("salary"))
      .alias("adjusted_salary")
)

Rename Columns

df.rename({"old_name": "new_name"})
df.rename({"col1": "a", "col2": "b"})

Drop Columns

df.drop("column_name")
df.drop(["col1", "col2"])
df.drop(cs.temporal())  # Drop by selector

Reorder Columns

df.select("name", "id", pl.all().exclude("name", "id"))

Null Values

Check for Nulls

df.null_count()                  # Null count per column
df.select(pl.all().is_null())    # Boolean mask per column

Fill Nulls

df.with_columns(
    pl.col("value").fill_null(0),                    # With constant
    pl.col("value").fill_null(strategy="forward"),   # Forward fill
    pl.col("value").fill_null(strategy="backward"),  # Backward fill
    pl.col("value").fill_null(pl.col("value").mean()), # No strategy="mean"; compute mean explicitly
    pl.col("value").fill_null(pl.col("default"))     # From another column
)

Drop Nulls

df.drop_nulls()                      # Rows with any null
df.drop_nulls(subset=["a", "b"])     # Rows with null in a or b

Replace Values

df.with_columns(
    pl.col("status").replace({"old": "new", "unknown": "pending"})
)

# Replace with default for unmatched
df.with_columns(
    pl.col("status").replace(
        {"A": 1, "B": 2},
        default=0
    )
)

Sorting

Basic Sort

df.sort("age")
df.sort("age", descending=True)

Multiple Columns

df.sort(["category", "date"], descending=[False, True])

Null Handling in Sort

df.sort("value", nulls_last=True)   # Nulls at end
df.sort("value", nulls_last=False)  # Nulls at start (default)

Sort by Expression

df.sort(pl.col("name").str.len_chars())
df.sort(pl.col("date").dt.month())

Unique and Duplicates

Unique Values

df.unique()                          # Unique rows
df.unique(subset=["name"])           # Unique by column
df.unique(subset=["a", "b"])         # Unique by multiple columns
df.unique(keep="first")              # Keep first occurrence
df.unique(keep="last")               # Keep last occurrence
df.unique(keep="none")               # Drop all duplicates

Count Unique

df.select(pl.col("name").n_unique())
df.n_unique()  # Unique row count

Find Duplicates

df.filter(pl.struct("a", "b").is_duplicated())
df.filter(~pl.struct("a", "b").is_duplicated())  # Non-duplicates

Sampling

df.sample(n=10)                 # N random rows
df.sample(fraction=0.1)         # 10% of rows
df.sample(n=10, seed=42)        # Reproducible
df.sample(n=10, with_replacement=True)

Row Indexing

By Position

df.head(5)                # First 5
df.tail(5)                # Last 5
df.slice(10, 20)          # Rows 10-29 (offset, length)
df.gather([0, 5, 10])     # Specific row indices

Row Index Column

df.with_row_index()                    # Add "index" column
df.with_row_index(name="row_num")      # Custom name
df.with_row_index(offset=1)            # Start from 1

Column Operations

Get Column Info

df.columns              # List of column names
df.schema               # Dict of name: dtype
df.dtypes               # List of dtypes
df.estimated_size()     # Memory estimate in bytes

Apply to All Columns

# Same operation on all columns
df.select(pl.all() * 2)
df.select(pl.all().cast(pl.String))
df.select(pl.all().fill_null(0))

Summary

Expressions

Expressions are the core of Polars. They describe transformations on columns.

Expression Basics

What is an Expression?

An expression is a function that transforms Series → Series:

pl.col("a")              # Reference column "a"
pl.col("a") + 1          # Add 1 to each value
pl.col("a").sum()        # Aggregate to single value
pl.col("a").alias("b")   # Rename result to "b"

Expressions are lazy: they describe transformations but don't execute until placed in a context.

Column References

pl.col("name")           # Single column
pl.col("a", "b", "c")    # Multiple columns
pl.col("^prefix_.*$")    # Regex pattern
pl.all()                 # All columns
pl.exclude("id")         # All except "id"
pl.first()               # First column
pl.last()                # Last column

Literal Values

pl.lit(5)                # Integer literal
pl.lit("hello")          # String literal
pl.lit(None)             # Null value
pl.lit([1, 2, 3])        # List literal

Contexts

Expressions are evaluated in specific contexts that determine their behavior.

select - Choose/Transform Columns

df.select(
    pl.col("a"),
    pl.col("b") * 2,
    (pl.col("a") + pl.col("b")).alias("sum")
)
# Returns DataFrame with only these columns

with_columns - Add New Columns

df.with_columns(
    (pl.col("a") * 2).alias("a_doubled"),
    pl.col("b").str.to_uppercase().alias("b_upper")
)
# Returns DataFrame with original columns PLUS new ones

filter - Filter Rows

df.filter(pl.col("a") > 10)
df.filter((pl.col("a") > 10) & (pl.col("b") == "x"))
# Expression must return boolean

group_by().agg() - Aggregation Context

df.group_by("category").agg(
    pl.col("value").sum(),       # Aggregates
    pl.col("value").mean(),
    pl.len()                     # Row count per group
)

Arithmetic Operations

pl.col("a") + pl.col("b")    # Addition
pl.col("a") - pl.col("b")    # Subtraction
pl.col("a") * pl.col("b")    # Multiplication
pl.col("a") / pl.col("b")    # Division (float result)
pl.col("a") // pl.col("b")   # Floor division
pl.col("a") % pl.col("b")    # Modulo
pl.col("a") ** 2             # Power
-pl.col("a")                 # Negation

# With literals
pl.col("a") + 5
pl.col("a") * 1.1

Comparison Operations

pl.col("a") == pl.col("b")   # Equal
pl.col("a") != pl.col("b")   # Not equal
pl.col("a") > pl.col("b")    # Greater than
pl.col("a") >= pl.col("b")   # Greater or equal
pl.col("a") < pl.col("b")    # Less than
pl.col("a") <= pl.col("b")   # Less or equal

# With literals
pl.col("a") > 10
pl.col("name") == "Alice"

Boolean Operations

# AND
(pl.col("a") > 10) & (pl.col("b") < 20)

# OR
(pl.col("a") > 10) | (pl.col("b") < 20)

# NOT
~(pl.col("a") > 10)

# XOR
(pl.col("a") > 10) ^ (pl.col("b") < 20)

Important: Always wrap conditions in parentheses due to Python operator precedence.

Common Expression Methods

Numeric

pl.col("a").abs()            # Absolute value
pl.col("a").sqrt()           # Square root
pl.col("a").log()            # Natural log
pl.col("a").log10()          # Log base 10
pl.col("a").exp()            # e^x
pl.col("a").pow(2)           # Power
pl.col("a").round(2)         # Round to 2 decimals
pl.col("a").floor()          # Floor
pl.col("a").ceil()           # Ceiling
pl.col("a").clip(0, 100)     # Clip to range
pl.col("a").sign()           # Sign (-1, 0, 1)

Aggregation

pl.col("a").sum()            # Sum
pl.col("a").mean()           # Mean
pl.col("a").median()         # Median
pl.col("a").min()            # Minimum
pl.col("a").max()            # Maximum
pl.col("a").std()            # Standard deviation
pl.col("a").var()            # Variance
pl.col("a").count()          # Non-null count
pl.col("a").n_unique()       # Unique count
pl.col("a").first()          # First value
pl.col("a").last()           # Last value
pl.col("a").quantile(0.95)   # Percentile
pl.len()                     # Total row count

Null Handling

pl.col("a").is_null()        # Boolean: is null
pl.col("a").is_not_null()    # Boolean: is not null
pl.col("a").fill_null(0)     # Fill nulls with value
pl.col("a").drop_nulls()     # Remove nulls
pl.col("a").null_count()     # Count nulls

Type Conversion

pl.col("a").cast(pl.Float64)
pl.col("a").cast(pl.String)
pl.col("a").cast(pl.Int32)
pl.col("a").cast(pl.Datetime)

# Strict vs lenient
pl.col("a").cast(pl.Int64, strict=True)   # Error on failure
pl.col("a").cast(pl.Int64, strict=False)  # Null on failure

Renaming

pl.col("a").alias("new_name")
(pl.col("a") + pl.col("b")).alias("sum")

Conditional Logic

when/then/otherwise

# If-else
pl.when(pl.col("a") > 10)
  .then(pl.lit("high"))
  .otherwise(pl.lit("low"))

# If-elif-else
pl.when(pl.col("score") >= 90)
  .then(pl.lit("A"))
  .when(pl.col("score") >= 80)
  .then(pl.lit("B"))
  .when(pl.col("score") >= 70)
  .then(pl.lit("C"))
  .otherwise(pl.lit("F"))

# With column values
pl.when(pl.col("discount"))
  .then(pl.col("price") * 0.9)
  .otherwise(pl.col("price"))

Coalesce

# First non-null value
pl.coalesce("a", "b", "c")  # First non-null from a, b, or c
pl.coalesce(pl.col("primary"), pl.col("fallback"), pl.lit("default"))

Horizontal Operations

Operations across multiple columns in the same row:

# Sum across columns
pl.sum_horizontal("a", "b", "c")
pl.sum_horizontal(pl.col("^value_.*$"))  # Regex

# Mean across columns
pl.mean_horizontal("a", "b", "c")

# Min/Max across columns
pl.min_horizontal("a", "b", "c")
pl.max_horizontal("a", "b", "c")

# Any/All (boolean)
pl.any_horizontal("flag1", "flag2", "flag3")
pl.all_horizontal("check1", "check2")

# Concatenate strings
pl.concat_str(["first", "last"], separator=" ")

List Operations

# Create list column
pl.col("a").implode()                    # Column to single-row list

# List element access
pl.col("list_col").list.first()          # First element
pl.col("list_col").list.last()           # Last element
pl.col("list_col").list.get(0)           # By index
pl.col("list_col").list.len()            # List length

# List aggregations
pl.col("list_col").list.sum()
pl.col("list_col").list.mean()
pl.col("list_col").list.min()
pl.col("list_col").list.max()

# List transformations
pl.col("list_col").list.unique()
pl.col("list_col").list.sort()
pl.col("list_col").list.reverse()
pl.col("list_col").list.contains(5)

Struct Operations

# Create struct
pl.struct("a", "b", "c")
pl.struct(pl.col("a"), pl.col("b").alias("renamed"))

# Access fields
pl.col("struct_col").struct.field("name")
pl.col("struct_col").struct["name"]

# Unnest struct to columns
df.unnest("struct_col")

Method Chaining

Chain expressions for readable transformations:

(
    pl.col("price")
    .fill_null(0)
    .clip(0, 1000)
    .round(2)
    .alias("cleaned_price")
)

(
    pl.col("text")
    .str.strip_chars()
    .str.to_lowercase()
    .str.replace_all(r"\s+", " ")
    .alias("normalized")
)

Apply Custom Functions (Use Sparingly)

# map_elements - Python function per element (SLOW)
# Avoid when possible - prefer native expressions
pl.col("a").map_elements(lambda x: x * 2, return_dtype=pl.Int64)

# map_batches - Function on entire Series (faster)
pl.col("a").map_batches(lambda s: s * 2)

# Note: "apply" was renamed to "map_elements" in Polars 0.19+

Warning: map_elements is slow because it uses Python. Always prefer native Polars expressions.

Expression Patterns

Multiple Columns Same Operation

# Apply same operation to multiple columns
df.with_columns(
    pl.col("a", "b", "c").fill_null(0)
)

df.with_columns(
    pl.all().exclude("id").round(2)
)

df.with_columns(
    cs.numeric().fill_null(0)
)

Rename Pattern

# Add suffix/prefix to column names
df.select(pl.all().name.suffix("_new"))
df.select(pl.all().name.prefix("col_"))

# Map names
df.select(pl.all().name.map(str.upper))

Over (Window Context)

# Expression within groups (see aggregations-grouping.md)
pl.col("value").sum().over("category")  # Sum per category

Summary

Gotchas & Common Issues

Common mistakes, error patterns, and troubleshooting for Polars.

Contents

• Type Errors
• Null Handling
• Expression Context Errors
• Performance Anti-Patterns
• Migration from Pandas
• Memory Issues

See also: qcut() label format surprise under Type Errors.

Type Errors

"Expected X, got Y" in Expressions

Problem: Type mismatch in operations.

# Error: cannot compare String to Int
df.filter(pl.col("id") == "123")  # id is Int64

Fix: Cast to correct type or use correct literal:

df.filter(pl.col("id") == 123)           # Use int literal
df.filter(pl.col("id") == pl.lit("123").cast(pl.Int64))  # Or cast

Arithmetic on Wrong Types

Problem: Operations between incompatible types.

# Error: cannot add String and Int
df.with_columns(pl.col("a") + pl.col("b"))  # a is String

Fix: Cast before operations:

df.with_columns(
    (pl.col("a").cast(pl.Int64) + pl.col("b")).alias("sum")
)

Null Handling

Nulls in Comparisons

Problem: Nulls don't equal anything, including themselves.

# This does NOT find nulls
df.filter(pl.col("x") == None)  # Wrong

Fix: Use is_null() or is_not_null():

df.filter(pl.col("x").is_null())
df.filter(pl.col("x").is_not_null())

Nulls in Aggregations

Problem: Aggregations skip nulls by default.

# sum() ignores nulls, may give unexpected count
df.select(pl.col("x").sum())

Fix: Handle nulls explicitly if needed:

df.select(
    pl.col("x").fill_null(0).sum(),        # Replace nulls
    pl.col("x").drop_nulls().count(),       # Count non-nulls
    pl.col("x").null_count()                # Count nulls
)

Nulls in String Operations

Problem: String operations propagate nulls.

# If "name" has nulls, result has nulls
df.with_columns(pl.col("name").str.to_uppercase())

Fix: Fill nulls before or after:

df.with_columns(
    pl.col("name").fill_null("").str.to_uppercase()
)

Expression Context Errors

Using Expressions Outside Context

Problem: Expressions need a context (select, filter, with_columns).

# Error: expressions must be used in a context
result = pl.col("a") + pl.col("b")  # Wrong

Fix: Use within a DataFrame context:

result = df.select(pl.col("a") + pl.col("b"))

Column Not Found

Problem: Referencing non-existent column.

# Error: column "foo" not found
df.select("foo")  # Column doesn't exist

Fix: Check column names:

print(df.columns)  # List columns
df.select(pl.col("^foo.*$"))  # Regex if unsure of exact name

Aliasing Required in Aggregations

Problem: Multiple aggregations on same column need aliases.

# Error: duplicate column name
df.group_by("cat").agg(
    pl.col("val").sum(),
    pl.col("val").mean()  # Same output name "val"
)

Fix: Use .alias():

df.group_by("cat").agg(
    pl.col("val").sum().alias("total"),
    pl.col("val").mean().alias("average")
)

qcut() Labels Get Unexpected Suffixes

Problem: pl.Series.qcut() and pl.Expr.qcut() with custom labels append descriptive suffixes to the first and last labels when include_breaks=False (the default).

# Requesting labels ["Q1", "Q2", "Q3", "Q4", "Q5"] with 5 quantiles
# produces: "Q1 (Lowest)", "Q2", "Q3", "Q4", "Q5 (Highest)"
s = pl.Series("val", range(100))
result = s.qcut(5, labels=["Q1", "Q2", "Q3", "Q4", "Q5"])
print(result.unique())  # Shows "Q1 (Lowest)" and "Q5 (Highest)"

This causes failures when downstream code uses exact string matching:

# WRONG: no rows match because actual label is "Q1 (Lowest)"
df.filter(pl.col("quintile") == "Q1")

# CORRECT: use starts_with or contains
df.filter(pl.col("quintile").str.starts_with("Q1"))
df.filter(pl.col("quintile").str.contains("Q1"))

Best practice: Always inspect value_counts() on qcut output before building downstream logic on label values:

df = df.with_columns(
    pl.col("score").qcut(5, labels=["Q1", "Q2", "Q3", "Q4", "Q5"]).alias("quintile")
)
print(df["quintile"].value_counts())  # Inspect actual label values

Performance Anti-Patterns

Row-by-Row Iteration

Problem: Using Python loops destroys performance.

# Very slow - don't do this
results = []
for row in df.iter_rows():
    results.append(process(row))

Fix: Use expressions or map_elements as last resort:

# Preferred: use expressions
df.with_columns(
    (pl.col("a") * 2 + pl.col("b")).alias("result")
)

# Last resort: map_elements (still slow)
df.with_columns(
    pl.col("a").map_elements(process, return_dtype=pl.Int64)
)

Using .apply() (Deprecated)

Problem: .apply() was renamed in Polars 0.19+.

# Deprecated/removed
df.select(pl.col("a").apply(lambda x: x * 2))

Fix: Use map_elements or preferably native expressions:

# Preferred: native expression
df.select(pl.col("a") * 2)

# If custom function needed
df.select(pl.col("a").map_elements(func, return_dtype=pl.Int64))

Collecting Too Early

Problem: Calling .collect() breaks optimization.

# Bad: collects early, loses optimization
df1 = lf.filter(pl.col("a") > 10).collect()
df2 = df1.lazy().filter(pl.col("b") < 5).collect()

Fix: Chain lazy operations, collect once:

# Good: single optimized query
df = (
    lf.filter(pl.col("a") > 10)
      .filter(pl.col("b") < 5)
      .collect()
)

Not Using Lazy Mode for Large Data

Problem: Eager mode loads everything into memory.

# May run out of memory on large files
df = pl.read_csv("huge_file.csv")

Fix: Use lazy/scan for large data:

# Streaming, memory efficient
lf = pl.scan_csv("huge_file.csv")
result = lf.filter(...).select(...).collect()

Migration from Pandas

groupby → group_by

# Pandas style (wrong in Polars 0.19+)
df.groupby("col")

# Polars style
df.group_by("col")

melt → unpivot

# Old (Polars <1.0)
df.melt(id_vars=["id"], value_vars=["a", "b"])

# New (Polars 1.0+)
df.unpivot(on=["a", "b"], index=["id"])

Index-Based Access

Problem: Polars has no index like Pandas.

# Pandas style (doesn't work)
df.loc[0]
df.iloc[0:5]

Fix: Use Polars row access methods:

df.head(5)           # First 5 rows
df.tail(5)           # Last 5 rows
df.slice(0, 5)       # Rows 0-4
df.row(0)            # Single row as tuple
df.row(0, named=True)  # First row as dict

Chained Assignment

Problem: Pandas-style chained assignment doesn't work.

# Pandas style (doesn't work)
df["new_col"] = df["a"] * 2

Fix: Use with_columns:

df = df.with_columns(
    (pl.col("a") * 2).alias("new_col")
)

Memory Issues

Out of Memory on Large Files

Problem: File too large for available RAM.

Fix: Use streaming/lazy mode:

# Streaming read
lf = pl.scan_csv("huge.csv")

# Process in batches
result = (
    lf.filter(...)
      .group_by(...)
      .agg(...)
      .collect(streaming=True)  # Enable streaming engine
)

Memory Not Released

Problem: DataFrames not garbage collected.

Fix: Delete references explicitly:

del df
import gc
gc.collect()

String Columns Using Too Much Memory

Problem: Repeated strings waste memory.

Fix: Use Categorical for low-cardinality strings:

df = df.with_columns(
    pl.col("category").cast(pl.Categorical)
)

Quick Fixes

Interoperability

Pandas Integration

Polars to Pandas

# Basic conversion
pandas_df = polars_df.to_pandas()

# With PyArrow backend (recommended - preserves types better)
pandas_df = polars_df.to_pandas(use_pyarrow_extension_array=True)

# Series to pandas Series
pandas_series = polars_series.to_pandas()

Pandas to Polars

import pandas as pd
import polars as pl

# From pandas DataFrame
polars_df = pl.from_pandas(pandas_df)

# From pandas Series
polars_series = pl.from_pandas(pandas_series)

# With schema override
polars_df = pl.from_pandas(pandas_df, schema_overrides={"id": pl.Int32})

Type Mapping

Common Gotchas

# Pandas uses NaN for missing values in numeric columns
# Polars uses null - conversion handles this automatically

# String columns with None in pandas become pl.String with null
# Make sure to handle null values appropriately

# Index is NOT converted - use reset_index() first if needed
pandas_df = pandas_df.reset_index()
polars_df = pl.from_pandas(pandas_df)

NumPy Integration

Polars to NumPy

import numpy as np

# Series to numpy array
arr = polars_series.to_numpy()

# DataFrame to 2D numpy array
arr = polars_df.to_numpy()

# Specific column to numpy
arr = polars_df["column_name"].to_numpy()

# Allow copy (default) or zero-copy if possible
arr = polars_series.to_numpy(allow_copy=False)  # Raises if copy needed

NumPy to Polars

# From numpy array
arr = np.array([1, 2, 3, 4, 5])
series = pl.Series("values", arr)

# 2D array to DataFrame
arr = np.array([[1, 2], [3, 4], [5, 6]])
df = pl.DataFrame(arr, schema=["a", "b"])

# From dict of numpy arrays
df = pl.DataFrame({
    "a": np.array([1, 2, 3]),
    "b": np.array([4.0, 5.0, 6.0])
})

Using NumPy Functions

# NumPy ufuncs work on Polars expressions
df.with_columns(
    np.log(pl.col("value")).alias("log_value"),
    np.sqrt(pl.col("value")).alias("sqrt_value"),
    np.exp(pl.col("value")).alias("exp_value"),
)

# Note: This goes through Python/NumPy, prefer native Polars when available
# Polars native:
df.with_columns(
    pl.col("value").log().alias("log_value"),
    pl.col("value").sqrt().alias("sqrt_value"),
    pl.col("value").exp().alias("exp_value"),
)

PyArrow Integration

PyArrow provides zero-copy conversion when possible.

Polars to Arrow

import pyarrow as pa

# DataFrame to Arrow Table
arrow_table = polars_df.to_arrow()

# Series to Arrow Array
arrow_array = polars_series.to_arrow()

# Chunked (for large data)
arrow_table = polars_df.to_arrow()  # Already chunked internally

Arrow to Polars

# From Arrow Table
arrow_table = pa.table({"a": [1, 2, 3], "b": ["x", "y", "z"]})
polars_df = pl.from_arrow(arrow_table)

# From Arrow Array
arrow_array = pa.array([1, 2, 3])
polars_series = pl.from_arrow(arrow_array, schema={"values": pl.Int64})

# From RecordBatch
polars_df = pl.from_arrow(record_batch)

Zero-Copy Benefits

# Arrow is Polars' native memory format
# Conversion is often zero-copy (no data copying)
arrow_table = polars_df.to_arrow()  # Fast, shares memory
polars_df = pl.from_arrow(arrow_table)  # Fast, shares memory

DuckDB Integration

DuckDB and Polars can share data efficiently via Arrow.

Query Polars DataFrame with DuckDB

import duckdb
import polars as pl

df = pl.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "value": [100, 200, 300]
})

# Method 1: Register and query
con = duckdb.connect()
con.register("my_table", df.to_arrow())
result = con.execute("SELECT * FROM my_table WHERE value > 100").arrow()
result_df = pl.from_arrow(result)

# Method 2: Direct query (DuckDB 0.8+)
result = duckdb.query("SELECT * FROM df WHERE value > 100").pl()

Use DuckDB for Complex SQL

# For complex SQL that's easier in SQL syntax
result = duckdb.query("""
    WITH ranked AS (
        SELECT *,
               ROW_NUMBER() OVER (PARTITION BY category ORDER BY value DESC) as rn
        FROM df
    )
    SELECT * FROM ranked WHERE rn <= 3
""").pl()

Polars SQL Interface

Polars has built-in SQL support:

# Query single DataFrame
result = df.sql("SELECT name, SUM(value) as total FROM self GROUP BY name")

# SQL Context for multiple tables
ctx = pl.SQLContext({"users": users_df, "orders": orders_df})
result = ctx.execute("""
    SELECT u.name, COUNT(*) as order_count
    FROM users u
    JOIN orders o ON u.id = o.user_id
    GROUP BY u.name
""").collect()

Database Connections

SQLAlchemy

from sqlalchemy import create_engine

# Create engine
engine = create_engine("postgresql://user:pass@host:5432/database")

# Read with SQL
df = pl.read_database("SELECT * FROM users", engine)

# Write (via pandas for now)
df.to_pandas().to_sql("table_name", engine, if_exists="replace")

ConnectorX (Recommended for Speed)

# Requires: pip install connectorx

# Direct connection string
df = pl.read_database_uri(
    "SELECT * FROM users WHERE active = true",
    "postgresql://user:pass@host:5432/database"
)

# Supported databases:
# PostgreSQL, MySQL, SQLite, SQL Server, Oracle, Redshift, ClickHouse

ADBC (Arrow Database Connectivity)

# Modern, high-performance option
# pip install adbc-driver-postgresql

import adbc_driver_postgresql.dbapi as pg_dbapi

with pg_dbapi.connect("postgresql://user:pass@host/db") as conn:
    df = pl.read_database("SELECT * FROM users", conn)

Scikit-learn Integration

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

# Polars for data prep
df = (
    pl.scan_parquet("data.parquet")
    .select(["feature1", "feature2", "feature3", "target"])
    .drop_nulls()
    .collect()
)

# Convert to numpy for sklearn
X = df.select(pl.exclude("target")).to_numpy()
y = df["target"].to_numpy()

# Train model
X_train, X_test, y_train, y_test = train_test_split(X, y)
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Predictions back to Polars
predictions = model.predict(X_test)
result_df = pl.DataFrame({
    "prediction": predictions,
    "actual": y_test
})

Visualization Libraries

Matplotlib

import matplotlib.pyplot as plt

# Convert column to numpy for plotting
plt.plot(df["x"].to_numpy(), df["y"].to_numpy())
plt.show()

# Or convert to pandas
df.to_pandas().plot(x="date", y="value")

Altair

import altair as alt

# Altair works with pandas
chart = alt.Chart(df.to_pandas()).mark_point().encode(
    x="x:Q",
    y="y:Q",
    color="category:N"
)

Plotly

import plotly.express as px

# Plotly works with pandas
fig = px.scatter(df.to_pandas(), x="x", y="y", color="category")
fig.show()

Common Patterns

Mixed Pipeline

import polars as pl
import pandas as pd

# Start with Polars for efficient data loading and transformation
df = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("date") > "2024-01-01")
    .group_by("category")
    .agg(pl.col("value").sum())
    .collect()
)

# Convert to pandas for specific library that requires it
pandas_df = df.to_pandas()

# Use pandas-specific functionality
result = some_pandas_only_function(pandas_df)

# Convert back to Polars if needed
polars_result = pl.from_pandas(result)

Efficient Data Pipeline

# Best practice: Use Polars as much as possible
# Only convert at boundaries (I/O, specific libraries)

# Read with Polars (fast)
df = pl.scan_parquet("input.parquet")

# Transform with Polars (fast, parallel)
df = df.filter(...).with_columns(...).group_by(...).agg(...)

# Only convert when necessary
if need_sklearn:
    X = df.select(features).to_numpy()
    
if need_specific_pandas_function:
    pandas_df = df.to_pandas()
    
# Write with Polars (fast)
df.collect().write_parquet("output.parquet")

Summary

Data I/O

CSV Files

Read CSV (Eager)

df = pl.read_csv("data.csv")

# With options
df = pl.read_csv(
    "data.csv",
    separator=",",              # Delimiter (default: ",")
    has_header=True,            # First row is header (default: True)
    skip_rows=1,                # Skip first N rows
    n_rows=1000,                # Only read first N rows
    columns=["a", "b", "c"],    # Only these columns
    schema_overrides={"id": pl.Int32},  # Override types ("dtypes" renamed to "schema_overrides" in 0.20.31)
    null_values=["NA", ""],     # Treat as null
    ignore_errors=True,         # Skip bad rows
    encoding="utf8",            # Encoding (default: utf8)
)

Scan CSV (Lazy - Recommended)

lf = pl.scan_csv("data.csv")

# With options
lf = pl.scan_csv(
    "data.csv",
    has_header=True,
    separator=",",
    skip_rows=0,
    n_rows=None,                # None = all rows
    schema_overrides={"id": pl.Int32},
    null_values=["NA"],
    infer_schema_length=10000,  # Rows to infer types (default: 100)
    low_memory=False,           # Reduce memory at cost of speed
)

df = lf.collect()

Write CSV

df.write_csv("output.csv")

# With options
df.write_csv(
    "output.csv",
    separator=",",
    include_header=True,
    null_value="",              # How to write nulls
    datetime_format="%Y-%m-%d", # Format for datetime
    float_precision=6,          # Decimal places
)

Streaming Write (Large Data)

lf.sink_csv("output.csv")

Parquet Files (Recommended)

Parquet is the preferred format: compressed, fast, type-preserving.

Read Parquet (Eager)

df = pl.read_parquet("data.parquet")

# Specific columns only
df = pl.read_parquet("data.parquet", columns=["a", "b"])

# Row groups
df = pl.read_parquet("data.parquet", n_rows=1000)

Scan Parquet (Lazy - Best for Large Files)

lf = pl.scan_parquet("data.parquet")

# Benefits: predicate/projection pushdown
result = (
    pl.scan_parquet("huge.parquet")
    .filter(pl.col("date") > "2024-01-01")  # Only reads matching rows
    .select("id", "value")                   # Only reads these columns
    .collect()
)

Write Parquet

df.write_parquet("output.parquet")

# With compression
df.write_parquet(
    "output.parquet",
    compression="zstd",         # "zstd", "snappy", "gzip", "lz4", "uncompressed"
    compression_level=3,        # 1-22 for zstd (higher = smaller, slower)
    statistics=True,            # Include column statistics
    row_group_size=100_000,     # Rows per group
)

Streaming Write

lf.sink_parquet("output.parquet", compression="zstd")

JSON Files

Read JSON

# Standard JSON (array of objects)
df = pl.read_json("data.json")

Read NDJSON (Newline-Delimited JSON)

# Better for large files, streaming
df = pl.read_ndjson("data.ndjson")

# Lazy scan
lf = pl.scan_ndjson("data.ndjson")

Write JSON/NDJSON

df.write_json("output.json")
df.write_ndjson("output.ndjson")

# Streaming
lf.sink_ndjson("output.ndjson")

Multiple Files (Glob Patterns)

Read Multiple Files

# All CSVs in directory
lf = pl.scan_csv("data/*.csv")

# Recursive glob
lf = pl.scan_parquet("data/**/*.parquet")

# With pattern matching
lf = pl.scan_csv("data/sales_202[34]*.csv")

Add Source File Column

lf = pl.scan_csv("data/*.csv", include_file_paths="source_file")
# Adds column with the file path for each row

Hive Partitioned Data

# For directory structure: data/year=2024/month=01/data.parquet
lf = pl.scan_parquet("data/**/*.parquet", hive_partitioning=True)
# Automatically creates year and month columns from path

Excel Files

# Requires: pip install xlsx2csv or pip install openpyxl

df = pl.read_excel("data.xlsx")

# Specific sheet
df = pl.read_excel("data.xlsx", sheet_name="Sheet2")
df = pl.read_excel("data.xlsx", sheet_id=1)  # 1-indexed

# Read all sheets
sheets = pl.read_excel("data.xlsx", sheet_id=0)  # Returns dict

# Write Excel
df.write_excel("output.xlsx")

Database Connections

SQLite

# Read
df = pl.read_database(
    "SELECT * FROM users WHERE active = 1",
    "sqlite:///database.db"
)

# Or with connection object
import sqlite3
conn = sqlite3.connect("database.db")
df = pl.read_database("SELECT * FROM users", conn)

PostgreSQL

# Requires: pip install connectorx

df = pl.read_database(
    "SELECT * FROM users",
    "postgresql://user:pass@host:5432/database"
)

# With query parameters
df = pl.read_database_uri(
    "SELECT * FROM users WHERE id = $1",
    "postgresql://user:pass@host/db",
    execute_options={"parameters": [42]}
)

MySQL

df = pl.read_database(
    "SELECT * FROM users",
    "mysql://user:pass@host:3306/database"
)

Generic ADBC (Arrow Database Connectivity)

# Modern, high-performance option
import adbc_driver_postgresql.dbapi

conn = adbc_driver_postgresql.dbapi.connect("postgresql://...")
df = pl.read_database("SELECT * FROM users", conn)

Cloud Storage

AWS S3

# Requires: pip install polars[fsspec] and s3fs

# Read directly
df = pl.read_parquet("s3://bucket/path/data.parquet")
lf = pl.scan_parquet("s3://bucket/path/*.parquet")

# With credentials (via environment or explicit)
import s3fs
fs = s3fs.S3FileSystem(
    key="ACCESS_KEY",
    secret="SECRET_KEY",
    endpoint_url="https://s3.region.amazonaws.com"
)
df = pl.read_parquet("s3://bucket/data.parquet", storage_options={"fs": fs})

Google Cloud Storage

# Requires: pip install gcsfs

df = pl.read_parquet("gs://bucket/data.parquet")
lf = pl.scan_parquet("gs://bucket/path/*.parquet")

Azure Blob Storage

# Requires: pip install adlfs

df = pl.read_parquet("az://container/data.parquet")
lf = pl.scan_parquet("abfss://container@account.dfs.core.windows.net/path/*.parquet")

HTTP/URLs

# Read directly from URL
df = pl.read_csv("https://example.com/data.csv")
df = pl.read_parquet("https://example.com/data.parquet")

# Large files - download first for better performance
import urllib.request
urllib.request.urlretrieve("https://example.com/large.parquet", "local.parquet")
df = pl.read_parquet("local.parquet")

IPC (Arrow/Feather)

# Fast binary format, good for inter-process communication

# Read
df = pl.read_ipc("data.arrow")
df = pl.read_ipc("data.feather")

# Lazy scan
lf = pl.scan_ipc("data.arrow")

# Write
df.write_ipc("output.arrow")
df.write_ipc("output.feather", compression="zstd")

Avro Files

# Avro support is built into polars (no extra install needed)

df = pl.read_avro("data.avro")
df.write_avro("output.avro", compression="snappy")

Delta Lake

# Requires: pip install deltalake

df = pl.read_delta("delta_table_path/")
lf = pl.scan_delta("delta_table_path/")

# With version
# Version is passed directly: pl.read_delta("path/", version=5)
df = pl.read_delta("delta_table/", version=5)

Common Patterns

Type Inference Control

# More rows for type inference
lf = pl.scan_csv("data.csv", infer_schema_length=100000)

# Disable inference (use explicit schema)
df = pl.read_csv("data.csv", infer_schema=False)  # All strings

# Explicit schema
df = pl.read_csv(
    "data.csv",
    schema={
        "id": pl.Int64,
        "name": pl.String,
        "date": pl.Date,
        "value": pl.Float64
    }
)

Handling Large Files

# Use lazy + streaming
result = (
    pl.scan_csv("huge_file.csv")
    .filter(pl.col("category") == "A")
    .group_by("region")
    .agg(pl.col("sales").sum())
    .collect(streaming=True)
)

# Or sink directly to file
(
    pl.scan_csv("input.csv")
    .filter(pl.col("active"))
    .sink_parquet("output.parquet")
)

Reading Specific Rows/Columns

# Only specific columns
df = pl.read_parquet("data.parquet", columns=["id", "name", "value"])

# Only first N rows
df = pl.read_csv("data.csv", n_rows=1000)

# Skip rows
df = pl.read_csv("data.csv", skip_rows=10, skip_rows_after_header=5)

Summary

Key recommendation: Use Parquet with lazy scanning (scan_parquet) for large datasets.

Joins & Concatenation

Join Types

Inner Join (Default)

Returns only rows with matches in both DataFrames:

df1.join(df2, on="key")
df1.join(df2, on="key", how="inner")

Left Join

Returns all rows from left, matching rows from right (null if no match):

df1.join(df2, on="key", how="left")

Right Join

Returns all rows from right, matching rows from left:

df1.join(df2, on="key", how="right")

Full Outer Join

Returns all rows from both DataFrames:

df1.join(df2, on="key", how="full")
# Note: how="outer" is NOT valid in Polars 1.x; use how="full"

Cross Join

Cartesian product (every row paired with every row):

df1.join(df2, how="cross")

Anti Join

Rows in left that have NO match in right:

df1.join(df2, on="key", how="anti")

Semi Join

Rows in left that HAVE a match in right (but doesn't include right columns):

df1.join(df2, on="key", how="semi")

Join Keys

Single Key

df1.join(df2, on="id")

Multiple Keys

df1.join(df2, on=["id", "date"])

Different Column Names

df1.join(df2, left_on="user_id", right_on="id")
df1.join(df2, left_on=["user_id", "date"], right_on=["id", "timestamp"])

Join on Expression

# Join on transformed keys
df1.join(
    df2,
    left_on=pl.col("date").dt.date(),
    right_on=pl.col("timestamp").dt.date()
)

Handling Duplicate Columns

When both DataFrames have columns with the same name (other than join keys):

# Default: adds "_right" suffix
df1.join(df2, on="id")
# If both have "value" column, result has "value" and "value_right"

# Custom suffix
df1.join(df2, on="id", suffix="_from_df2")

# Coalesce (keep left value, fill with right if null)
df1.join(df2, on="id", how="full", coalesce=True)

Join Strategies

For performance optimization with large datasets:

# Let Polars decide (default)
df1.join(df2, on="key")

# Force hash join (good for unique keys)
df1.join(df2, on="key", join_nulls=False)

# Include null keys in join
df1.join(df2, on="key", join_nulls=True)

Concatenation

Vertical Concatenation (Stack Rows)

# Two DataFrames
pl.concat([df1, df2])

# Multiple DataFrames
pl.concat([df1, df2, df3, df4])

# Different column order (aligns by name)
pl.concat([df1, df2], how="align")

# Diagonal (union of all columns, nulls for missing)
pl.concat([df1, df2], how="diagonal")

# Diagonal with relaxed types
pl.concat([df1, df2], how="diagonal_relaxed")

Horizontal Concatenation (Stack Columns)

pl.concat([df1, df2], how="horizontal")

# Requires same number of rows

From List of DataFrames

dfs = [pl.read_csv(f"data_{i}.csv") for i in range(10)]
combined = pl.concat(dfs)

Pivot (Long to Wide)

Convert long format to wide format:

# Long format:
# | date       | product | sales |
# | 2024-01-01 | A       | 100   |
# | 2024-01-01 | B       | 150   |
# | 2024-01-02 | A       | 120   |

df.pivot(
    on="product",           # Column to spread
    index="date",           # Keep as rows
    values="sales"          # Values to fill
)

# Result:
# | date       | A   | B   |
# | 2024-01-01 | 100 | 150 |
# | 2024-01-02 | 120 | null|

Pivot with Aggregation

df.pivot(
    on="product",
    index="date",
    values="sales",
    aggregate_function="sum"     # If duplicates exist
)

# Multiple aggregations
df.pivot(
    on="product",
    index="date",
    values="sales",
    aggregate_function="first"   # first, last, sum, mean, count, etc.
)

Multiple Index Columns

df.pivot(
    on="product",
    index=["date", "region"],
    values="sales"
)

Multiple Value Columns

df.pivot(
    on="product",
    index="date",
    values=["sales", "quantity"]
)

Unpivot / Melt (Wide to Long)

Convert wide format to long format:

# Wide format:
# | date       | product_A | product_B |
# | 2024-01-01 | 100       | 150       |

df.unpivot(
    on=["product_A", "product_B"],   # Columns to melt
    index="date",                     # Keep as identifier
    variable_name="product",          # Name for column names
    value_name="sales"                # Name for values
)

# Result:
# | date       | product   | sales |
# | 2024-01-01 | product_A | 100   |
# | 2024-01-01 | product_B | 150   |

Note: melt was renamed to unpivot in Polars 1.0. Both still work.

Unpivot All Except Index

df.unpivot(
    index=["id", "date"],    # Keep these
    # All other columns are melted
)

Select Columns to Melt

# By name
df.unpivot(on=["col1", "col2", "col3"], index="id")

# By pattern
import polars.selectors as cs
df.unpivot(on=cs.starts_with("value_"), index="id")

Common Patterns

Self Join

# Join DataFrame to itself (e.g., find pairs)
df.join(df, on="category", suffix="_other").filter(
    pl.col("id") < pl.col("id_other")
)

Lookup Table

# Main data
orders = pl.DataFrame({
    "order_id": [1, 2, 3],
    "product_code": ["A", "B", "A"]
})

# Lookup table
products = pl.DataFrame({
    "code": ["A", "B"],
    "name": ["Apple", "Banana"]
})

# Add product names
orders.join(products, left_on="product_code", right_on="code", how="left")

Update Values from Another DataFrame

# Use coalesce with full join
df1.join(df2, on="id", how="left", suffix="_new").with_columns(
    pl.coalesce(pl.col("value_new"), pl.col("value")).alias("value")
).drop("value_new")

Conditional Join (Inequality)

# Polars doesn't have direct inequality joins
# Use cross join + filter
df1.join(df2, how="cross").filter(
    (pl.col("start") <= pl.col("date")) &
    (pl.col("date") < pl.col("end"))
)

# Or use join_asof for time-based inequality

As-Of Join (Time-Based)

Join on nearest time match:

# df1 has events at various times
# df2 has reference values at specific times
# Match each event to the most recent reference

df1.sort("timestamp").join_asof(
    df2.sort("timestamp"),
    on="timestamp",
    strategy="backward"    # Use most recent value <= event time
)

# Strategies:
# "backward" - most recent value <= key
# "forward" - next value >= key
# "nearest" - closest value

# With tolerance
df1.join_asof(
    df2,
    on="timestamp",
    tolerance="1h"         # Match only within 1 hour
)

# With grouping
df1.join_asof(
    df2,
    on="timestamp",
    by="symbol",           # Match within same symbol
    strategy="backward"
)

Multiple DataFrames Join

# Chain joins
result = (
    df1
    .join(df2, on="key1", how="left")
    .join(df3, on="key2", how="left")
    .join(df4, on="key3", how="left")
)

# Or reduce over list
from functools import reduce
dfs = [df1, df2, df3, df4]
result = reduce(lambda a, b: a.join(b, on="id", how="left"), dfs)

Summary

Performance Optimization

Lazy Evaluation

Why Use Lazy Mode

Lazy evaluation is Polars' killer feature for performance:

1. Query Optimization: Polars rewrites your query for efficiency 2. Predicate Pushdown: Filters are pushed to file read operations 3. Projection Pushdown: Only required columns are loaded 4. Common Subexpression Elimination: Avoids redundant computation 5. Streaming: Enables processing larger-than-memory datasets

Lazy vs Eager Comparison

# Eager (reads entire file, then filters)
df = pl.read_csv("huge.csv")           # Load all data
df = df.filter(pl.col("status") == "active")  # Then filter

# Lazy (optimized - reads only what's needed)
df = (
    pl.scan_csv("huge.csv")            # Just creates plan
    .filter(pl.col("status") == "active")  # Adds filter to plan
    .collect()                         # Executes optimized plan
)

Inspecting Query Plans

lf = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("date") > "2024-01-01")
    .select("id", "name", "value")
    .group_by("name")
    .agg(pl.col("value").sum())
)

# View logical plan (what you wrote)
print(lf.explain())

# View optimized plan (what Polars will do)
print(lf.explain(optimized=True))

Best Practices

Use scan_ Instead of read_

# Prefer this
lf = pl.scan_csv("data.csv")
lf = pl.scan_parquet("data.parquet")
lf = pl.scan_ndjson("data.ndjson")

# Over this
df = pl.read_csv("data.csv")
df = pl.read_parquet("data.parquet")

Filter Early

# Good - filter pushed to scan
result = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("year") == 2024)      # First
    .select("id", "name", "value")
    .group_by("name")
    .agg(pl.col("value").sum())
    .collect()
)

# Less efficient - processes more data than needed
result = (
    pl.scan_parquet("data.parquet")
    .group_by("name")
    .agg(pl.col("value").sum())
    .filter(pl.col("year") == 2024)      # Too late!
    .collect()
)

Select Only Needed Columns

# Good - only reads 3 columns from file
result = (
    pl.scan_parquet("wide_table.parquet")
    .select("id", "name", "value")
    .collect()
)

# Bad - reads all columns then discards
df = pl.read_parquet("wide_table.parquet")
result = df.select("id", "name", "value")

Avoid Multiple collect() Calls

# Bad - scans file twice
lf = pl.scan_parquet("data.parquet")
sum_result = lf.select(pl.col("value").sum()).collect()
mean_result = lf.select(pl.col("value").mean()).collect()

# Good - single scan
lf = pl.scan_parquet("data.parquet")
results = lf.select(
    pl.col("value").sum().alias("sum"),
    pl.col("value").mean().alias("mean")
).collect()

# Or use collect_all for separate LazyFrames
lf = pl.scan_parquet("data.parquet")
lf1 = lf.filter(pl.col("type") == "A").select(pl.col("value").sum())
lf2 = lf.filter(pl.col("type") == "B").select(pl.col("value").sum())
results = pl.collect_all([lf1, lf2])  # Optimized together

Anti-Patterns to Avoid

Row-by-Row Iteration

# BAD - Extremely slow
result = []
for row in df.iter_rows(named=True):
    result.append(row["value"] * 2)

# GOOD - Vectorized
df.with_columns(
    (pl.col("value") * 2).alias("doubled")
)

Using map_elements (Python UDFs)

# BAD - Python overhead for each element
df.with_columns(
    pl.col("value").map_elements(lambda x: x * 2, return_dtype=pl.Int64)
)

# GOOD - Native Polars expression
df.with_columns(
    (pl.col("value") * 2).alias("result")
)

# If you must use Python functions, prefer map_batches
df.with_columns(
    pl.col("value").map_batches(lambda s: s * 2)  # Works on entire Series
)

String Concatenation in Loop

# BAD
result = df.select(pl.col("a"))
for col in ["b", "c", "d"]:
    result = result.with_columns(pl.col(col))

# GOOD
result = df.select("a", "b", "c", "d")

Creating DataFrames in Loops

# BAD - Creates many small DataFrames
dfs = []
for file in files:
    dfs.append(pl.read_parquet(file))
combined = pl.concat(dfs)

# GOOD - Single scan with glob
combined = pl.scan_parquet("data/*.parquet").collect()

Memory Optimization

Use Appropriate Data Types

# Check current memory usage
print(df.estimated_size())  # Bytes

# Downcast numeric types
df = df.with_columns(
    pl.col("small_int").cast(pl.Int16),      # Instead of Int64
    pl.col("tiny_int").cast(pl.Int8),
    pl.col("unsigned").cast(pl.UInt32),
    pl.col("small_float").cast(pl.Float32),  # Instead of Float64
)

Use Categorical for Repeated Strings

# High cardinality strings = lots of memory
df = pl.DataFrame({"category": ["A", "B", "A", "C", "B"] * 1000000})
print(df.estimated_size())  # Large

# Categorical = much smaller
df = df.with_columns(pl.col("category").cast(pl.Categorical))
print(df.estimated_size())  # Much smaller

Streaming for Large Files

# Process larger-than-memory files
result = (
    pl.scan_parquet("huge_file.parquet")
    .filter(pl.col("status") == "active")
    .group_by("category")
    .agg(pl.col("value").sum())
    .collect(streaming=True)  # Processes in chunks
)

# Write directly without collecting
(
    pl.scan_parquet("input.parquet")
    .filter(pl.col("date") > "2024-01-01")
    .sink_parquet("output.parquet")  # Streams to file
)

Release Memory

# Delete when done
del large_df

# Or use context manager for temporary data
def process():
    df = pl.read_parquet("data.parquet")
    result = df.group_by("key").agg(pl.col("value").sum())
    return result  # df goes out of scope

summary = process()  # Large df is released

Parallel Processing

Polars automatically parallelizes operations across CPU cores.

Check Thread Count

import polars as pl
print(pl.threadpool_size())

Control Parallelism

# Set at runtime (before any operations)
import os
os.environ["POLARS_MAX_THREADS"] = "4"

# Or in code
pl.Config.set_streaming_chunk_size(100_000)

Operations That Parallelize Well

• Aggregations (sum, mean, etc.)
• Filtering
• Joins
• Group by operations
• Window functions

Benchmarking

Profile Your Code

import time

start = time.perf_counter()
result = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("x") > 0)
    .group_by("category")
    .agg(pl.col("value").sum())
    .collect()
)
elapsed = time.perf_counter() - start
print(f"Elapsed: {elapsed:.3f}s")

Compare Approaches

import timeit

# Approach 1: Eager
def eager_approach():
    df = pl.read_parquet("data.parquet")
    return df.filter(pl.col("x") > 0).group_by("y").agg(pl.col("z").sum())

# Approach 2: Lazy
def lazy_approach():
    return (
        pl.scan_parquet("data.parquet")
        .filter(pl.col("x") > 0)
        .group_by("y")
        .agg(pl.col("z").sum())
        .collect()
    )

print("Eager:", timeit.timeit(eager_approach, number=10))
print("Lazy:", timeit.timeit(lazy_approach, number=10))

File Format Performance

Use Parquet

Parquet is significantly faster than CSV:

# CSV - slow, no type preservation
df = pl.read_csv("data.csv")           # Slow
df = pl.scan_csv("data.csv").collect() # Better but still slow

# Parquet - fast, typed, compressed
df = pl.scan_parquet("data.parquet").collect()  # Fast!

Parquet Compression

# Write with compression
df.write_parquet("data.parquet", compression="zstd")  # Good balance
df.write_parquet("data.parquet", compression="snappy")  # Faster decompress
df.write_parquet("data.parquet", compression="lz4")     # Fastest

# Compression levels (zstd)
df.write_parquet("data.parquet", compression="zstd", compression_level=3)  # Default
df.write_parquet("data.parquet", compression="zstd", compression_level=19) # High compression (max is 22)

Common Performance Issues

Checklist for Optimal Performance

1. Use scan_* instead of read_* 2. Use Parquet format when possible 3. Filter early in the pipeline 4. Select only needed columns 5. Avoid map_elements/Python UDFs 6. Use appropriate data types (Int32 vs Int64, Categorical) 7. Use single collect() or collect_all() 8. Enable streaming for large data: collect(streaming=True) 9. Use native expressions instead of iteration 10. Profile and benchmark your code

Quickstart

Installation

Basic Install

pip install polars
# or
uv add polars
# or
conda install -c conda-forge polars

With Optional Dependencies

# All optional features
pip install "polars[all]"

# Specific extras
pip install "polars[numpy,pandas,pyarrow]"  # Interop
pip install "polars[timezone]"               # Timezone support
pip install "polars[connectorx]"             # Database connections
pip install "polars[fsspec]"                 # Cloud storage (S3, GCS, Azure)

Verify Installation

import polars as pl
print(pl.__version__)  # Should be 1.x

Core Concepts

Lazy vs Eager Execution

Polars has two execution modes:

Eager - Executes immediately (like Pandas):

df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
result = df.filter(pl.col("a") > 1)  # Runs now

Lazy - Builds query plan, optimizes, then executes:

lf = pl.LazyFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
# Or from file scan
lf = pl.scan_csv("data.csv")

result = (
    lf.filter(pl.col("a") > 1)
    .select("a", "b")
    .collect()  # Executes optimized plan here
)

When to use which:

• Eager: Small data, interactive exploration, quick tests
• Lazy: Large data, complex pipelines, production code (recommended)

Why Lazy is Better for Large Data

1. Query Optimization: Polars rewrites your query for efficiency 2. Predicate Pushdown: Filters applied at read time (skip unneeded rows) 3. Projection Pushdown: Only reads columns you actually use 4. Streaming: Process larger-than-memory datasets

# This reads ONLY the needed columns and rows
result = (
    pl.scan_parquet("huge_file.parquet")
    .filter(pl.col("date") > "2024-01-01")  # Pushed down to file read
    .select("id", "value")                   # Only these columns loaded
    .collect()
)

Expressions

Expressions are the heart of Polars. They describe transformations:

# Expressions are functions: Series → Series
pl.col("a")              # Reference column "a"
pl.col("a") + 1          # Add 1 to each value
pl.col("a").sum()        # Sum all values
pl.col("a").alias("b")   # Rename to "b"

Expressions are evaluated in contexts:

• select() - Choose/transform columns
• with_columns() - Add new columns
• filter() - Filter rows
• group_by().agg() - Aggregate by groups

Creating DataFrames

From Dictionary

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, 30, 35],
    "score": [85.5, 90.0, 78.5]
})

With Explicit Schema

df = pl.DataFrame(
    {
        "id": [1, 2, 3],
        "value": ["a", "b", "c"]
    },
    schema={
        "id": pl.Int32,
        "value": pl.String
    }
)

From Series

s1 = pl.Series("a", [1, 2, 3])
s2 = pl.Series("b", [4, 5, 6])
df = pl.DataFrame([s1, s2])

Empty DataFrame with Schema

df = pl.DataFrame(
    schema={
        "id": pl.Int64,
        "name": pl.String,
        "created": pl.Datetime
    }
)

LazyFrame (Lazy Mode)

# From data
lf = pl.LazyFrame({"a": [1, 2, 3]})

# From files (preferred for large data)
lf = pl.scan_csv("data.csv")
lf = pl.scan_parquet("data.parquet")
lf = pl.scan_ndjson("data.ndjson")

Basic Operations

View Data

df.head(5)          # First 5 rows
df.tail(5)          # Last 5 rows
df.sample(10)       # Random 10 rows
df.glimpse()        # Transposed view
df.describe()       # Statistics summary
df.schema           # Column names and types
df.columns          # List of column names
df.shape            # (rows, cols) tuple
len(df)             # Row count

Select Columns

df.select("name", "age")
df.select(pl.col("name"), pl.col("age"))
df.select(pl.all())                      # All columns
df.select(pl.exclude("id"))              # All except "id"

Filter Rows

df.filter(pl.col("age") > 25)
df.filter((pl.col("age") > 25) & (pl.col("score") > 80))

Add Columns

df.with_columns(
    (pl.col("age") + 1).alias("age_next_year"),
    pl.lit("active").alias("status")
)

Sort

df.sort("age")
df.sort("age", descending=True)
df.sort(["name", "age"], descending=[False, True])

LazyFrame Operations

Building a Query

query = (
    pl.scan_csv("data.csv")
    .filter(pl.col("status") == "active")
    .select("id", "name", "value")
    .with_columns(
        (pl.col("value") * 1.1).alias("adjusted")
    )
    .sort("value", descending=True)
)

Executing

# Execute and get DataFrame
df = query.collect()

# Execute with streaming (for large data)
df = query.collect(streaming=True)

# Write directly to file (streaming)
query.sink_parquet("output.parquet")
query.sink_csv("output.csv")

Inspecting the Plan

# Logical plan (what you wrote)
print(query.explain())

# Optimized plan (what Polars will do)
print(query.explain(optimized=True))

Type System

Common Types

Casting Types

df.with_columns(
    pl.col("id").cast(pl.Int32),
    pl.col("price").cast(pl.Float64),
    pl.col("category").cast(pl.Categorical)
)

Next Steps

• Learn the expression system - the core of Polars
• Master data I/O for loading files
• Understand performance patterns for efficient code

Strings, Datetime & Categorical

String Operations

All string operations are accessed via the .str namespace.

Basic String Operations

df.with_columns(
    pl.col("text").str.to_uppercase().alias("upper"),
    pl.col("text").str.to_lowercase().alias("lower"),
    pl.col("text").str.to_titlecase().alias("title"),
    pl.col("text").str.len_chars().alias("char_count"),      # Character count
    pl.col("text").str.len_bytes().alias("byte_count"),      # Byte count (UTF-8)
)

Whitespace Handling

df.with_columns(
    pl.col("text").str.strip_chars(),               # Strip both ends
    pl.col("text").str.strip_chars_start(),         # Left strip
    pl.col("text").str.strip_chars_end(),           # Right strip
    pl.col("text").str.strip_chars(" \t\n"),        # Strip specific chars
)

Slicing and Substrings

df.with_columns(
    pl.col("text").str.slice(0, 5),        # First 5 characters
    pl.col("text").str.slice(-3),          # Last 3 characters
    pl.col("text").str.head(5),            # First 5 characters
    pl.col("text").str.tail(3),            # Last 3 characters
)

Search and Match

df.with_columns(
    pl.col("text").str.contains("pattern"),              # Contains substring
    pl.col("text").str.contains("pat.*n", literal=False),# Regex match
    pl.col("text").str.starts_with("prefix"),            # Starts with
    pl.col("text").str.ends_with("suffix"),              # Ends with
    pl.col("text").str.find("needle"),                   # Index of first match (null if not found)
    pl.col("text").str.count_matches("a"),               # Count occurrences
)

# Filter by string content
df.filter(pl.col("name").str.contains("Smith"))
df.filter(pl.col("email").str.ends_with("@gmail.com"))

Replace

df.with_columns(
    # Replace first occurrence
    pl.col("text").str.replace("old", "new"),
    
    # Replace all occurrences
    pl.col("text").str.replace_all("old", "new"),
    
    # Regex replace
    pl.col("text").str.replace_all(r"\d+", "NUM"),
    
    # Replace with captured groups
    pl.col("text").str.replace_all(r"(\w+)@(\w+)", r"\2:\1"),
)

Split and Join

df.with_columns(
    # Split to list
    pl.col("tags").str.split(","),                  # Returns List[String]
    pl.col("path").str.split("/"),
    
    # Split and extract
    pl.col("tags").str.split(",").list.first(),     # First element
    pl.col("tags").str.split(",").list.get(1),      # Second element
    pl.col("tags").str.split(",").list.len(),       # Count elements
)

# Join list to string
df.with_columns(
    pl.col("list_col").list.join(", "),            # Join with separator
)

Extract with Regex

df.with_columns(
    # Extract first match of pattern
    pl.col("text").str.extract(r"(\d+)", group_index=1),
    
    # Extract all matches
    pl.col("text").str.extract_all(r"\d+"),         # Returns list
    
    # Named groups
    pl.col("text").str.extract_groups(r"(?<name>\w+):(?<value>\d+)"),
)

Padding and Alignment

df.with_columns(
    pl.col("code").str.pad_start(5, "0"),       # "42" -> "00042"
    pl.col("code").str.pad_end(10, " "),        # Right pad with spaces
    pl.col("code").str.zfill(5),                # Zero-fill left
)

Concatenation

# Concatenate columns
df.with_columns(
    pl.concat_str([
        pl.col("first_name"),
        pl.lit(" "),
        pl.col("last_name")
    ]).alias("full_name")
)

# With separator
df.with_columns(
    pl.concat_str(["city", "state", "country"], separator=", ").alias("location")
)

Datetime Operations

All datetime operations are accessed via the .dt namespace.

Parsing Strings to Datetime

df.with_columns(
    # Auto-detect format
    pl.col("date_str").str.to_datetime(),
    
    # Explicit format
    pl.col("date_str").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S"),
    pl.col("date_str").str.strptime(pl.Date, "%Y-%m-%d"),
    pl.col("date_str").str.strptime(pl.Time, "%H:%M:%S"),
    
    # With timezone
    pl.col("date_str").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S", time_zone="UTC"),
)

Common Format Codes

Extracting Components

df.with_columns(
    pl.col("datetime").dt.year().alias("year"),
    pl.col("datetime").dt.month().alias("month"),
    pl.col("datetime").dt.day().alias("day"),
    pl.col("datetime").dt.hour().alias("hour"),
    pl.col("datetime").dt.minute().alias("minute"),
    pl.col("datetime").dt.second().alias("second"),
    pl.col("datetime").dt.microsecond().alias("microsecond"),
    pl.col("datetime").dt.nanosecond().alias("nanosecond"),
    
    # Day of week (1=Monday, 7=Sunday)
    pl.col("datetime").dt.weekday().alias("weekday"),
    
    # Week of year
    pl.col("datetime").dt.week().alias("week"),
    
    # Day of year (1-366)
    pl.col("datetime").dt.ordinal_day().alias("day_of_year"),
    
    # Quarter (1-4)
    pl.col("datetime").dt.quarter().alias("quarter"),
    
    # ISO year and week
    pl.col("datetime").dt.iso_year().alias("iso_year"),
)

Formatting to String

df.with_columns(
    pl.col("datetime").dt.strftime("%Y-%m-%d").alias("date_str"),
    pl.col("datetime").dt.strftime("%B %d, %Y").alias("formatted"),
    pl.col("datetime").dt.to_string("%Y-%m-%d %H:%M").alias("str"),
)

Truncation (Rounding Down)

df.with_columns(
    pl.col("datetime").dt.truncate("1h").alias("hour_start"),
    pl.col("datetime").dt.truncate("1d").alias("day_start"),
    pl.col("datetime").dt.truncate("1w").alias("week_start"),
    pl.col("datetime").dt.truncate("1mo").alias("month_start"),
)

Date/Time Arithmetic

df.with_columns(
    # Add duration
    (pl.col("date") + pl.duration(days=7)).alias("next_week"),
    (pl.col("datetime") + pl.duration(hours=2)).alias("plus_2h"),
    
    # Subtract dates (returns Duration)
    (pl.col("end_date") - pl.col("start_date")).alias("duration"),
    
    # Duration to days/hours
    (pl.col("end_date") - pl.col("start_date")).dt.total_days().alias("days"),
    (pl.col("end_date") - pl.col("start_date")).dt.total_hours().alias("hours"),
)

# Duration literals
pl.duration(days=1)
pl.duration(hours=2, minutes=30)
pl.duration(weeks=1)

Timezone Handling

df.with_columns(
    # Set timezone (localize)
    pl.col("datetime").dt.replace_time_zone("UTC"),
    
    # Convert between timezones
    pl.col("datetime").dt.convert_time_zone("America/New_York"),
    
    # Remove timezone
    pl.col("datetime").dt.replace_time_zone(None),
)

Filtering by Date

# Date comparisons
df.filter(pl.col("date") > pl.date(2024, 1, 1))
df.filter(pl.col("date").is_between(pl.date(2024, 1, 1), pl.date(2024, 12, 31)))

# Filter by component
df.filter(pl.col("datetime").dt.year() == 2024)
df.filter(pl.col("datetime").dt.month() == 7)
df.filter(pl.col("datetime").dt.weekday() < 5)  # Weekdays only

Creating Dates/Datetimes

# Date literals
pl.date(2024, 7, 15)

# Datetime literals
pl.datetime(2024, 7, 15, 14, 30, 0)

# From columns
df.with_columns(
    pl.datetime(
        pl.col("year"),
        pl.col("month"),
        pl.col("day")
    ).alias("date")
)

# Date range
pl.date_range(
    start=pl.date(2024, 1, 1),
    end=pl.date(2024, 12, 31),
    interval="1d",
    eager=True
)

Categorical Data

Categorical type stores strings as integers for memory efficiency.

Creating Categorical

df.with_columns(
    pl.col("category").cast(pl.Categorical)
)

# Or on read
df = pl.read_csv("data.csv", dtypes={"category": pl.Categorical})

Enum Type (Fixed Categories)

For known, fixed set of categories:

# Define enum type
status_type = pl.Enum(["pending", "active", "completed", "cancelled"])

df.with_columns(
    pl.col("status").cast(status_type)
)

# Enum is stricter - errors on unknown values

StringCache (Consistent Categories)

When working with multiple DataFrames with same categorical column:

# Without StringCache, categories may not match between DataFrames
with pl.StringCache():
    df1 = pl.DataFrame({"cat": ["a", "b"]}).with_columns(
        pl.col("cat").cast(pl.Categorical)
    )
    df2 = pl.DataFrame({"cat": ["b", "c"]}).with_columns(
        pl.col("cat").cast(pl.Categorical)
    )
    
    # Now can safely concatenate or join
    combined = pl.concat([df1, df2])

Categorical Operations

df.with_columns(
    # Get physical (integer) representation
    pl.col("category").to_physical().alias("cat_code"),
    
    # Get categories
    pl.col("category").cat.get_categories().alias("categories"),
)

# Sort by categorical order (not alphabetically)
df.sort("category")

Converting Back to String

df.with_columns(
    pl.col("category").cast(pl.String)
)

Summary

String Functions

Datetime Functions

Categorical