Compression, Encoding, and Partitioning

Compression, encoding, and partitioning act at different layers. A compression codec reduces byte size. A column encoding represents repeated or structured values efficiently before or alongside compression. Dataset partitioning places records into separate files or directories, often by a field such as date or region.

These choices can reduce I/O, but they also add CPU work, metadata, and operational complexity. There is no universally best codec, row-group size, or partition key.

Prerequisites

Complete Columnar Storage and Parquet and Apache Arrow Memory Model.

Learning goals

• Distinguish compression codecs from Parquet encodings.
• Explain how dictionary encoding can represent repeated values.
• Use partition pruning without confusing directory names with row-level indexes.
• Identify small-file, skew, cardinality, and privacy risks in a partition design.

Worked example

The code writes the typed fixture as a Hive-style directory dataset partitioned by status. The directory names are deterministic; the individual Parquet filenames are generated, so the example compares directories rather than unstable filenames.

from pathlib import Path
from tempfile import TemporaryDirectory
import pyarrow as pa
import pyarrow.csv as pacsv
import pyarrow.parquet as pq

source = Path("fixtures/data-engineering/files-tools/orders.csv")
types = {
    "order_id": pa.int64(), "ordered_at": pa.date32(),
    "customer_id": pa.int64(), "amount": pa.decimal128(10, 2),
    "status": pa.string(),
}
table = pacsv.read_csv(source, convert_options=pacsv.ConvertOptions(column_types=types))

with TemporaryDirectory() as directory:
    root = Path(directory) / "orders"
    pq.write_to_dataset(
        table, root_path=root, partition_cols=["status"],
        compression="zstd", use_dictionary=["status"],
    )
    partitions = sorted(path.name for path in root.iterdir() if path.is_dir())
    shipped = pq.read_table(root, filters=[[('status', '=', 'shipped')]])
    print(partitions)
    print(shipped.num_rows, shipped.column("status").to_pylist())

The three directories are status=cancelled, status=pending, and status=shipped; the filtered read returns three shipped rows. A dataset reader can derive the partition value from the path and avoid opening irrelevant partitions. That is partition pruning, not a claim that every byte inside the selected files is skipped.

Dictionary encoding is useful when many rows repeat a small set of values: indices refer to a dictionary instead of repeating each string. It may be ineffective for high-cardinality values. Compression effectiveness then depends on the encoded bytes and codec. Inspect actual metadata and measure representative reads and writes before standardizing.

Choosing a partition key

A useful key appears in common selective predicates, has bounded cardinality, distributes data reasonably, and produces files large enough to process efficiently. Dates are often partitioned at day or month granularity, but the right choice depends on arrival rate and query horizon. Partitioning by a near-unique identifier creates a directory explosion; partitioning by a sensitive field can expose it through paths and listings.

Partition paths are metadata, not a substitute for values and validation. Writers and readers must agree on how path text maps to typed fields. Updating or deleting records across many partition files is also more involved than updating a database table.

Limitations and common mistakes

• More compression usually trades additional CPU for fewer stored and transferred bytes; the balance changes by codec level, hardware, and storage.
• Dictionary encoding has overhead and can fall back or perform poorly when cardinality is high.
• Excess partitions and tiny files increase listing, planning, opening, and metadata costs.
• Skew can leave one partition enormous while others are nearly empty.
• A filtered read can still scan all data when metadata is absent, predicates are unsupported, or values do not align with partitions and row groups.

Practice

Propose a partition layout for five years of hourly sensor readings. State the expected query filters, arrival volume, retention policy, and why partitioning by individual sensor ID might or might not be acceptable.

Next steps

Continue with Schemas, Types, and Evolution to make compatibility decisions explicit across dataset versions.

Primary sources

Checked 2026-10-15: Apache Parquet encodings, Apache Parquet compression, PyArrow partitioned datasets, and DuckDB partitioned writes.