Choosing a Data Processing Tool

A defensible tool choice starts with a workload description, not a leaderboard. Specify data sources and formats, logical schema, transformations, result size, memory budget, latency needs, team interface, downstream ecosystem, deployment target, concurrency, and operational ownership. Then test the smallest representative slice that could disprove your choice.

Prerequisites

Complete the preceding files-and-tools lessons, especially pandas, Polars, Arrow, and DuckDB.

Learning goals

• Translate a data task into selection criteria and correctness invariants.
• Choose a starting tool without claiming one universal winner.
• Recognize when a format, interchange layer, library, or query engine solves a different part of the system.
• Record evidence and a reevaluation trigger.

Worked example

Suppose a data analyst must read monthly local Parquet extracts, join them to a small reference table, aggregate a stable report, and hand the result to a plotting library. The data currently fits in memory but is growing.

Constraint	Evidence to collect	Plausible starting point
Team maintains SQL transformations	Query readability, plan, Parquet pushdown, export boundary	DuckDB
Existing notebook and library integration dominate	pandas dtype fidelity, peak memory, downstream compatibility	pandas
Expression pipeline and deferred scans are useful	Polars plan, streaming support, deterministic result	Polars
Cross-language columnar exchange is the main boundary	Arrow type support, chunking, buffer ownership, IPC needs	Arrow interfaces plus an execution library

This table is not a ranking. The same project might query with DuckDB, exchange an Arrow table, and use pandas for a specialist library. That combination is justified only if schema and ownership checks make the boundaries reliable.

Define result invariants before performance tests: row grain, key uniqueness, null policy, decimal precision, timezone, ordering, and reconciled totals. Use production-like formats and distributions. Measure elapsed time, peak memory, bytes read or written, startup overhead, and operational complexity under controlled versions and hardware. A performance difference matters only after results agree.

A lightweight decision record

Record:

1. workload and representative dataset;
2. required interfaces and deployment constraints;
3. candidate versions and configurations;
4. correctness results and bounded measurements;
5. rejected alternatives and why;
6. decision owner and date;
7. trigger for reevaluation, such as a tenfold volume increase or a new multi-user requirement.

Prefer the simplest tool that satisfies the current constraints and leaves a clear boundary for change. Avoid adopting distributed infrastructure merely because the dataset might grow; also avoid ignoring measured memory or concurrency limits because the current sample is small.

Limitations and common mistakes

• Vendor or project benchmarks may use different hardware, queries, data distributions, tuning, and competitors' versions.
• Familiar syntax reduces training cost but does not remove correctness, memory, or operational limits.
• Combining tools adds conversions, dependencies, failure modes, and version compatibility work.
• Local embedded tools do not automatically provide remote multi-user access, governance, scheduling, or high availability.
• A decision record expires when its workload assumptions change.

Practice

Write a one-page decision record for a daily 20 GB CSV ingestion that produces a 500 MB Parquet report. Include malformed-row policy, schema contract, memory budget, benchmark design, chosen boundary, and reevaluation trigger.

Next steps

You can now distinguish files, memory models, DataFrames, and engines while making a bounded tool choice. Continue with ETL vs. ELT to begin the reliable-pipelines sequence, and revisit Schemas, Types, and Evolution when defining pipeline contracts.

Primary sources

Checked 2026-10-15: pandas scaling guidance, Polars lazy API, Arrow FAQ, DuckDB benchmark guide, and DuckDB environment guidance.