pandas, Polars, Arrow, and DuckDB

pandas, Polars, Apache Arrow, and DuckDB overlap, but they are not four interchangeable brands of the same abstraction.

• pandas provides an eager, mutable-feeling Python DataFrame API with a broad analysis ecosystem.
• Polars provides expression-oriented eager and lazy DataFrames backed by its query engines.
• Arrow specifies columnar data structures and interchange/IPC interfaces; PyArrow also supplies compute and I/O libraries.
• DuckDB is an embedded analytical SQL database and execution engine with file and Arrow integration.

Choose a boundary by the operations, data size, ecosystem, language, deployment, and operational constraints—not a universal speed ranking.

Prerequisites

Complete Apache Arrow Memory Model, DuckDB for Local Analytics, and Polars Lazy DataFrames.

Learning goals

• Name the primary abstraction each tool provides.
• Implement one result with explicit schemas in all four tools.
• Identify where interchange may share buffers and where conversion can occur.
• Design a representative benchmark instead of generalizing from a toy input.

Worked example

Each path computes the same customer-level total. The final normalization exists only to compare observable results; it does not claim identical internal execution.

from pathlib import Path
import duckdb
import pandas as pd
import polars as pl
import pyarrow as pa
import pyarrow.csv as pacsv

source = Path("fixtures/data-engineering/files-tools/orders.csv")
expected = [(1, 100.0), (2, 25.0), (3, 80.0)]

pandas_frame = pd.read_csv(
    source,
    dtype={"order_id": "int64", "customer_id": "int64", "amount": "Float64", "status": "string"},
    parse_dates=["ordered_at"],
)
pandas_result = list(
    pandas_frame.groupby("customer_id", sort=True, dropna=False)["amount"].sum().items()
)

polars_result = [
    tuple(row)
    for row in (
        pl.scan_csv(source, schema={
            "order_id": pl.Int64, "ordered_at": pl.Date,
            "customer_id": pl.Int64, "amount": pl.Float64, "status": pl.String,
        })
        .group_by("customer_id")
        .agg(pl.col("amount").sum())
        .sort("customer_id")
        .collect()
        .rows()
    )
]

arrow_table = pacsv.read_csv(
    source,
    convert_options=pacsv.ConvertOptions(column_types={
        "order_id": pa.int64(), "ordered_at": pa.date32(),
        "customer_id": pa.int64(), "amount": pa.float64(), "status": pa.string(),
    }),
)
arrow_result = [
    tuple(row.values())
    for row in arrow_table.group_by("customer_id").aggregate([("amount", "sum")])
        .sort_by("customer_id").to_pylist()
]

duckdb_result = duckdb.sql("""
    SELECT customer_id, CAST(SUM(amount) AS DOUBLE)
    FROM read_csv(
        'fixtures/data-engineering/files-tools/orders.csv', header=true,
        columns={'order_id':'BIGINT','ordered_at':'DATE','customer_id':'BIGINT','amount':'DOUBLE','status':'VARCHAR'}
    )
    GROUP BY customer_id ORDER BY customer_id
""").fetchall()

assert pandas_result == polars_result == arrow_result == duckdb_result == expected
print(expected)

The assertion verifies semantic agreement on this fixture. It is intentionally not timed. A useful benchmark would preserve identical types and null semantics, include warm and cold reads where relevant, use representative scale and selectivity, cap threads consistently, measure peak memory and total I/O, repeat runs, and report hardware and versions.

Interchange boundaries

Arrow-aware APIs can sometimes expose existing buffers to another library. Whether that path is zero-copy depends on the exact source type, target type, chunking, null representation, mutability, ownership, and API version. Converting an Arrow string column to a consumer's incompatible string representation, combining chunks, or materializing Python objects can copy.

Keep transformations in one engine when that makes the plan simpler, and cross a boundary intentionally when another ecosystem provides needed functionality. Verify schema, nulls, ordering, categorical metadata, timestamp timezone, decimal precision, and buffer ownership at every boundary.

Limitations and common mistakes

• Syntax similarity does not guarantee identical null, grouping, ordering, or type behavior.
• “Uses Arrow” does not guarantee that every conversion shares buffers.
• A fastest result on one laptop, dataset, query, or version is not a universal ranking.
• pandas ecosystem compatibility, Polars lazy plans, Arrow interoperability, and DuckDB SQL each solve different constraints.
• Data larger than memory still requires a supported streaming, pushdown, batching, or out-of-core path; a library name alone is not a memory plan.

Practice

Design a benchmark for a 50-column Parquet dataset where a query selects three columns and one percent of rows. Specify correctness checks, cache policy, thread limits, repetitions, memory measurement, and which result would change your tool choice.

Next steps

Use Choosing a Data Processing Tool to turn these distinctions into a documented project decision.

Primary sources

Checked 2026-10-15: pandas scaling guidance, Polars pandas migration guide, Polars Arrow producer/consumer guide, Arrow C Data Interface, and DuckDB Python data ingestion.