Skip to main content

Columnar Storage and Parquet

A logical table says which fields and rows exist. A physical file layout says how their bytes are organized. CSV interleaves all fields of each record. Apache Parquet groups rows, then stores each column separately within those row groups, together with schema and metadata. That organization can support column selection, compression, encoding, and row-group skipping.

Prerequisites​

Complete CSV, JSON, and Schema Inference. You should recognize explicit Arrow types and missing values.

Learning goals​

  • Separate a logical schema from Parquet's physical row groups and column chunks.
  • Inspect schema, row-group count, and statistics instead of guessing a file's layout.
  • Explain projection and predicate pushdown without promising that every query skips data.
  • Choose row-group and file sizes as workload-dependent tradeoffs.

Worked example​

This example converts the typed order fixture to a temporary Parquet file. A row-group size of two is deliberately tiny so its structure is visible; it is not a production recommendation.

from pathlib import Path
from tempfile import TemporaryDirectory
import pyarrow as pa
import pyarrow.csv as pacsv
import pyarrow.parquet as pq

source = Path("fixtures/data-engineering/files-tools/orders.csv")
types = {
    "order_id": pa.int64(),
    "ordered_at": pa.date32(),
    "customer_id": pa.int64(),
    "amount": pa.decimal128(10, 2),
    "status": pa.string(),
}
table = pacsv.read_csv(source, convert_options=pacsv.ConvertOptions(column_types=types))

with TemporaryDirectory() as directory:
    parquet_path = Path(directory) / "orders.parquet"
    pq.write_table(table, parquet_path, compression="zstd", row_group_size=2)
    metadata = pq.ParquetFile(parquet_path).metadata
    selected = pq.read_table(parquet_path, columns=["order_id", "status"])
    print(metadata.num_rows, metadata.num_row_groups)
    print(selected.column_names, selected.num_rows)

The file has five rows and three row groups; the selective read returns only two columns. Parquet metadata may also contain per-column statistics. Engines can use relevant statistics to exclude row groups, but only when the writer produced usable metadata and the predicate can exploit it.

Parquet is a persistent file format. It is not a query language, a DataFrame library, or an in-memory object model. DuckDB, pandas, Polars, Spark, and Arrow libraries can all read Parquet, but their execution strategies and supported logical types differ.

How selective reads work​

Projection pushdown asks a reader for only required columns. Predicate pushdown moves a filter toward the scan. Row-group pruning uses metadata such as minimum and maximum values to avoid reading groups that cannot match. These are related optimizations, not synonyms, and none guarantees a particular speedup.

Performance depends on selected columns, predicate selectivity, row-group layout, codec, storage latency, concurrency, data distribution, reader implementation, and cache state. Measure the actual workload with representative data.

Limitations and common mistakes​

  • One tiny file per event creates metadata and scheduling overhead; one enormous file reduces parallelism and makes rewrites expensive.
  • Tiny row groups add metadata and may reduce compression; huge row groups can weaken pruning and increase memory pressure.
  • A Parquet schema does not by itself enforce business constraints such as unique order IDs or nonnegative amounts.
  • Statistics can reveal value ranges. Treat metadata as part of the data's security and privacy surface.
  • Do not infer a universal CSV-versus-Parquet performance ratio from this five-row fixture.

Practice​

Write the same table with row-group sizes two and five. Inspect the metadata, predict which layout could skip more work for order_id >= 105, and state why file size measurements on five rows would not generalize.

Next steps​

Next, study the Apache Arrow Memory Model. Later, Compression, Encoding, and Partitioning connects Parquet's internal choices to dataset layout.

Primary sources​

Checked 2026-10-15: Apache Parquet overview, Apache Parquet file format, PyArrow Parquet API, and DuckDB Parquet overview.