Skip to main content

Apache Arrow Memory Model

Apache Arrow specifies a language-independent columnar memory format. An Arrow array combines a logical type with one or more buffers and metadata. A primitive numeric array commonly uses a validity bitmap plus a contiguous values buffer; variable-length strings additionally use offsets into a data buffer.

This is a memory model and interchange specification. It is distinct from Parquet's persistent storage format and from an execution engine that plans queries.

Prerequisites​

Complete Columnar Storage and Parquet. Basic familiarity with arrays, bytes, and null values is helpful.

Learning goals​

  • Identify the logical type, length, null count, and buffers of an Arrow array.
  • Explain how a validity bitmap separates nullness from stored value bytes.
  • Distinguish a zero-copy slice from a conversion that may allocate or copy.
  • Recognize tables, record batches, and chunked arrays as different structures.

Worked example​

The array below has four logical slots and one null. buffers() exposes the validity and values buffers for this fixed-width type.

import pyarrow as pa

amounts = pa.array([4000, 6000, None, 8000], type=pa.int64())
validity, values = amounts.buffers()
window = amounts.slice(1, 2)
_, window_values = window.buffers()

print(amounts.type, len(amounts), amounts.null_count)
print(validity.size, values.size)
print(values.address == window_values.address, window.offset)

The values buffer holds four 64-bit slots, so it is 32 bytes. The validity bitmap needs one byte for these four positions. The slice reports offset 1 and shares this implementation's values buffer. It creates a new array view and metadata without copying the underlying values.

That observation does not justify saying “Arrow is always zero-copy.” A cast to a different physical type, combining chunks, changing null representation, crossing an interface that cannot consume Arrow buffers, or converting to an incompatible pandas representation may allocate and copy. Even when buffers are shared, wrapper objects and metadata still exist, and their lifetime must keep the underlying memory alive.

Tables, batches, and chunks​

An Arrow RecordBatch is a fixed set of equal-length arrays under one schema. A Table can represent each field as a ChunkedArray, allowing several physical chunks to form one logical column. Chunking avoids mandatory concatenation, but algorithms and consumers differ in how well they handle many chunks. combine_chunks() can simplify the layout by allocating new buffers.

Arrow IPC serializes Arrow-formatted record batches for storage or transport. It preserves the Arrow representation more directly than converting through CSV, but IPC still involves I/O and may involve copying, memory mapping, decompression, or device transfer. The C Data Interface standardizes ownership-aware exchange of Arrow arrays between runtimes; it does not make arbitrary objects compatible without conversion.

Limitations and common mistakes​

  • Null and zero are different: the validity bitmap determines whether a slot is logically present.
  • Buffer sharing is an implementation property to verify at a specific boundary, not a blanket product guarantee.
  • Arrow format compatibility does not imply that every library supports every extension type or preserves all metadata.
  • Slicing retains the referenced buffer, so a small view can keep a much larger allocation alive.
  • Untrusted or invalid buffers are a security concern; use maintained readers and validate external inputs.

Practice​

Create a string array with one null and inspect its three buffers. Explain why the offsets buffer is needed and why its size differs from the UTF-8 data buffer.

Next steps​

Continue with Compression, Encoding, and Partitioning. The later pandas, Polars, Arrow, and DuckDB lesson revisits interchange boundaries.

Primary sources​

Checked 2026-10-15: Arrow columnar format specification, Arrow C Data Interface, PyArrow data structures, and Arrow security considerations.