Pandas Arrow CSV And JSON IO Notes

pandas had a useful week for data engineering users, with 25 recent commits touching 67 files. The theme is not a single feature launch. It is a set of IO and dtype fixes that make CSV, JSON, datetime, Arrow, and HDF5 behavior less surprising in production pipelines.

The most visible performance work landed around the C CSV parser. The project added xsimd build support in the SIMD build change, then used that base in the C tokenizer SIMD change. That is the right layer to improve. CSV still sits in the hot path for lake ingest, partner feeds, compliance exports, and notebook handoffs.

The related Arrow string work is just as important. In the Arrow string parser change, read_csv can emit pyarrow backed string arrays directly from the C parser. That reduces the amount of Python side conversion work needed after parse time. It also puts more pressure on pipeline tests to cover the exact dtype backend used in production, not only default object strings.

The changed files point at the real surface area: pandas/_libs/parsers.pyx, pandas/_libs/meson.build, and pandas/tests/io/parser/test_c_parser_only.py. This is mostly internal engine work, but the output contract is user visible. If a workload depends on string dtype identity, null behavior, or Arrow conversion cost, this is worth testing before the next pandas upgrade.

Two bug fixes move pandas toward fail fast behavior in parsing paths. The JSON pyarrow options change makes read_json(engine="pyarrow") raise when callers pass options that the backend silently ignored. That can break code that passed broad option sets across engines, but the old behavior was worse for batch jobs. It let a job appear configured while one backend did something else.

The datetime fix is similar. The origin coercion fix addresses a path where to_datetime with an origin ignored errors='coerce'. For ETL, that matters because bad timestamps are often handled by explicit coercion and later quality checks. Silent divergence around that contract can move invalid records into a dataset as exceptions, wrong values, or missed nulls depending on the surrounding code.

There was also a JSON memory fix. The timezone timestamp serialization fix addressed a memory leak when serializing timestamps with timezones. It is easy to dismiss that as small unless pandas sits inside a long running worker, API process, or scheduler task that serializes frames repeatedly. The files to watch are pandas/io/json/_json.py, pandas/core/tools/datetimes.py, and pandas/tests/tools/test_to_datetime.py.

The most recent commit, the pyarrow nightly CI fix, is a small maintenance patch with a large signal. pyarrow 24 deprecates the feather read and write entry points pandas currently calls. pandas now suppresses those warnings in pandas/io/feather_format.py while the project tracks migration toward the pyarrow IPC path.

That does not change the public feather API yet. It does tell operators that Arrow dependency drift is active. Teams that pin pandas but allow pyarrow to float can still hit warning volume, test noise, or backend behavior changes first in CI. The pixi lock update and dependency bot commits are part of that same story: pandas is keeping its own test matrix current so downstream users see fewer surprises later.

The HDF5 note is another compatibility marker. The HDF5 documentation update records that removal of pandas_version metadata can break reads by older pandas versions. That is not just documentation for archive readers. It affects rolling upgrades where old and new jobs share HDF5 artifacts.

Several fixes are not IO features, but they still matter for pipeline correctness. The DataFrame.loc assignment fix addresses partial assignment upcasting dtype for untouched columns. In production data code, unwanted dtype drift is a common source of noisy diffs, failed schema checks, and slower writes.

Index and construction fixes point in the same direction. The pyarrow resample index name fix keeps a result index name from being dropped when resample(on=...) uses a pyarrow column. The dataclass constructor fix covers a list like dataclass input path. The infer dtype refactor removes recursion from object dtype inference.

None of these changes should be sold as a new workflow. They are maintenance on the sharp edges around pandas/core/frame.py and dtype inference. That is still valuable. Data platforms often fail in boring places: a dtype changed, an index name vanished, or an input wrapper behaved differently from a normal list.

  • read_csv speed work now depends on SIMD build detection and xsimd packaging. Test on the same wheel class used in production.
  • read_json(engine="pyarrow") may fail earlier when callers pass options that backend never honored. Treat that as a config cleanup task, not just a pandas issue.
  • Feather migration is not done yet. The current change suppresses warnings while pandas tracks the pyarrow IPC path.