Schemas, Types, and Evolution

A schema names fields and assigns types, nullability, and structure. A useful data contract adds meaning: grain, keys, units, allowed values, ownership, freshness, and compatibility policy. A file can satisfy its physical schema while violating the business contract—for example, by duplicating an order ID or storing a negative amount.

Schema evolution is a relationship between a producer change and specific consumers. Adding a nullable field may be backward-compatible for a reader that ignores unknown fields, but breaking for a strict reader. Widening an integer may preserve values in one engine and overflow another consumer's narrower type.

Prerequisites

Complete CSV, JSON, and Schema Inference and Compression, Encoding, and Partitioning.

Learning goals

• Separate physical schema validation from semantic contract tests.
• Classify additive, subtractive, renamed, narrowed, and meaning-changing updates.
• Read old and new records under one explicit compatible schema.
• Plan migrations instead of relying on inference to reconcile versions.

Worked example

Version 2 of the repository event fixture adds a nullable currency field. Supplying the target schema makes the expected evolution explicit: old records receive null for the absent field, while unexpected fields would be rejected.

from pathlib import Path
import pyarrow as pa
import pyarrow.json as pajson

root = Path("fixtures/data-engineering/files-tools")
event_schema = pa.schema([
    ("event_id", pa.int64()),
    ("occurred_at", pa.timestamp("s", tz="UTC")),
    ("amount", pa.float64()),
    ("currency", pa.string()),
])
options = pajson.ParseOptions(
    explicit_schema=event_schema,
    unexpected_field_behavior="error",
)
v1 = pajson.read_json(root / "events-v1.jsonl", parse_options=options)
v2 = pajson.read_json(root / "events-v2.jsonl", parse_options=options)
events = pa.concat_tables([v1, v2])

print(events.schema)
print(events.column("currency").to_pylist())

The combined column is [None, None, 'EUR', 'EUR']. This policy preserves all four records, but downstream consumers must decide what a missing currency means. If amount was previously documented as EUR, the migration might backfill EUR; if its meaning was unknown, such a backfill would invent data.

Compatibility is consumer-specific

Ask at least three questions for every change:

1. Can existing readers parse the new representation?
2. Can new readers interpret historical data?
3. Does the field still mean the same thing?

Adding an optional field often passes the first two questions. Renaming a field usually needs a period in which both names are supported or a versioned transformation. Narrowing a type, making a nullable field required, reusing an enum value, changing units, or changing timestamp timezone semantics can be breaking even when the bytes still parse.

Prefer explicit versions or compatibility rules at system boundaries. Validate producers before publication and run contract tests against representative consumers. Preserve raw immutable inputs where policy permits so a corrected parser can replay them.

Limitations and common mistakes

• “Nullable” does not explain why data is missing or whether a consumer may substitute a default.
• A type cast that succeeds can still truncate precision, change timezone meaning, or reinterpret identifiers.
• Union-by-name features help align files; they do not decide semantic compatibility.
• Default values can conceal whether the producer actually observed a value.
• Deleting or renaming a field requires consumer inventory and a deprecation window, not just a writer change.

Practice

Classify these changes for an existing consumer: customer_id from integer to string, a new nullable source field, amount from EUR cents to EUR, and removal of status. For each, specify the migration evidence you would require.

Next steps

Use these contracts with an embedded engine in DuckDB for Local Analytics, then compare DataFrame and query-engine boundaries in later lessons.

Primary sources

Checked 2026-10-15: Arrow schema metadata, Arrow type system, Parquet logical types, and DuckDB multi-file schema handling.