Skip to main content

Polars Lazy DataFrames

Polars offers eager DataFrame operations and lazy LazyFrame query plans. A lazy scan describes work without immediately materializing the full result. When collect() runs, the optimizer can move supported filters and projections toward the source and the execution engine evaluates the plan.

Polars 2.0 makes the streaming engine the default for lazy collection. Operations that do not semantically require row order may therefore return rows in a different incidental order. Sort when order is part of the output contract.

Prerequisites​

Complete Schemas, Types, and Evolution. Familiarity with GROUP BY will help connect expressions to SQL.

Learning goals​

  • Distinguish an eager result from a lazy query plan.
  • Supply a complete schema to a CSV scan.
  • Inspect an optimized plan and materialize only at an explicit boundary.
  • Make output ordering deterministic instead of relying on engine behavior.

Worked example​

The query scans the repository fixture, keeps shipped rows, aggregates by customer, and sorts the final result. Nothing is executed until collect().

from pathlib import Path
import polars as pl

source = Path("fixtures/data-engineering/files-tools/orders.csv")
schema = {
    "order_id": pl.Int64,
    "ordered_at": pl.Date,
    "customer_id": pl.Int64,
    "amount": pl.Float64,
    "status": pl.String,
}
query = (
    pl.scan_csv(source, schema=schema)
    .filter(pl.col("status") == "shipped")
    .group_by("customer_id")
    .agg(pl.col("amount").sum().alias("known_amount"))
    .sort("customer_id")
)
plan = query.explain()
result = query.collect()
print(result.to_dicts())

The result is [{'customer_id': 1, 'known_amount': 100.0}, {'customer_id': 3, 'known_amount': 80.0}]. explain() is useful for seeing whether a filter and column selection reached the scan, but plan text is version-sensitive and should not be treated as a stable serialized API.

The schema is known before execution, which lets Polars type-check many expressions while resolving the plan. Some transformations, such as a pivot whose output column names depend on data values, cannot stay fully lazy because the resulting schema is not knowable in advance.

Materialization boundaries​

collect() returns an eager DataFrame. Collect at the point where the result is genuinely needed, not after every expression. For outputs larger than memory or for file-to-file transformations, investigate streaming-compatible sink_* operations and verify that the whole plan supports the intended engine. “Lazy” does not mean “uses no memory,” and “streaming” does not mean that every operator can process an unlimited dataset with constant memory.

Avoid Python row functions when a native expression expresses the same transformation. Native expressions give the engine more information for type checking and optimization. If a Python callback is necessary, measure its impact and document its input/output contract.

Limitations and common mistakes​

  • Lazy execution postpones errors until schema resolution or collection; test the materialization boundary.
  • Optimizations are conditional on source and expression support. Inspect the plan rather than assuming pushdown.
  • Grouping and joins do not promise incidental row order; sort explicitly for deterministic presentation and tests.
  • A declared physical type does not validate domain rules such as positive amounts or allowed statuses.
  • The five-row fixture demonstrates semantics, not a performance advantage over another tool.

Practice​

Change the filter to keep all non-cancelled orders, add a row count, and inspect the plan. Explain which columns the scan needs and why the final sort remains part of the contract.

Next steps​

Compare execution and interchange boundaries in pandas, Polars, Arrow, and DuckDB.

Primary sources​

Checked 2026-10-15: Polars lazy API, Polars lazy schemas, Polars query plans, and Polars 2.0 upgrade guide.