Pandas Expands inplace Deprecation And CSV Parser Fixes

pandas is the Python dataframe library most ETL stacks still call for tabular transforms. This week on main landed 54 commits, 313 files, 10661 insertions, and 8404 deletions. The useful signal is a wider inplace deprecation plus C engine read_csv bugs that drop, pad, or silently truncate rows depending on chunk layout.

The inplace keyword on dropna and drop_duplicates now raises Pandas4Warning whenever you pass it. A later commit does the same for sort_values and sort_index. The warning lives in pandas/core/frame.py and the matching Series methods. It fires even for inplace=False. Omitting the keyword is the only quiet path.

This is PDEP-8 continuing after rename and drop. The keyword is marked for removal in pandas 4.0. Copy on write already made most in place calls copy anyway, so the speed argument for inplace=True has been weak for a while. The remaining cost is churn. Jobs that wrote df.dropna(inplace=True) will warn on 3.1 and fail later.

df = df.sort_values("event_ts")
# df.sort_values("event_ts", inplace=True)  # Pandas4Warning on 3.1
# df.sort_values("event_ts", inplace=False) # same warning; the flag is the trigger

Pipelines that already assign keep working without warnings. Notebooks and generated wrappers that stamp inplace=True on every call will fill logs on 3.1 and break on 4.0. Downstream libraries that wrap these methods need to stop forwarding the keyword.

The C engine in pandas/_libs/parsers.pyx still has chunk bookkeeping bugs that show up in production files, not toy CSVs.

One fix is field counts at chunk boundaries. With chunksize, iterator=True, or nrows, a line with too many fields at the start of a chunk skipped on_bad_lines. The line was truncated to the table width and kept. With default low_memory=True, a run of short lines that crossed an internal buffer boundary could raise ParserError even though the same file parsed with low_memory=False. The tokenizer used buffer slot numbers to decide whether a line was still header. After parser_consume_rows shifted data into slot 0, those exemptions applied to real rows. The fix tracks header_done and prev_line_fields across the shift.

Separator encoding was also wrong. The C engine tokenizes a utf-8 byte stream. The fallback check used sys.getfilesystemencoding(). On a latin-1 host, a single character separator that is two bytes in utf-8 could be handed to the C engine, split the file wrong, or raise UnicodeDecodeError. The check now encodes the separator as utf-8.

Related I/O fixes from the same window:

  • Integer sentinels. int64 min -9223372036854775808 and uint64 max 18446744073709551615 were read as missing with dtype_backend set to numpy_nullable or pyarrow.
  • Carriage return as sep. sep="\r" used to produce one column plus NaN padding. It now raises ValueError, matching "\n".
  • Buffers without mode. mmap.mmap and boto3 streams ignored encoding and decoded as utf-8.
  • Handle leak. A reader that raised while opening a ZIP or TAR with more than one file left the handle open. On Windows that blocks rename and delete.

If a nightly job sometimes fails read_csv and sometimes silently drops a bad line, look at chunk size and low_memory before blaming the file.

to_numeric and Python engine read_csv inference allocated a full extra set of typed buffers up front. maybe_convert_numeric reserved float64, complex128, int64, uint64, object, and bool arrays for every column. A column of floats paid about six times the memory of the array actually returned. The new path allocates the float buffer immediately and the rest only when a value of that kind shows up.

The same change closed a correctness hole. Values seen before the first complex were left uninitialized, so to_numeric(Series(["1.5", 2j])) could return 0j instead of 1.5+0j.

DataFrame.where, mask, and clip skipped a redundant fillna plus infer_objects pass when the condition is already numpy bool. The win is largest when cond does not share the frame index. Boolean __setitem__ sits on the same path. Neither change needs a config flag. Wide CSV ingest and where over misaligned masks should just get cheaper on 3.1.

hash_pandas_object treated equal floats as distinct when the bit pattern differed. 0.0 and -0.0 hashed apart. So did NaN values with different sign or payload bits. Dedup, factorize, and anything keyed on hash_pandas_object could split rows that compare equal. The fix canonicalizes those patterns before hashing, including the real and imag parts of complex64 and complex128.

This is the kind of bug that survives in a warehouse for years. Joins look a little fatter. Group counts do not match unique(). Nobody suspects the hash.

The 3.1 whatsnew file took 23 commits this window. That is where these notes are accumulating. Treat main as 3.1 development, not a patch on 3.0.

Scan jobs for inplace= on sort_values, sort_index, dropna, and drop_duplicates. Passing the keyword at all will warn. Assignment form is the replacement.

Run CSV fixtures that use chunksize or default low_memory against 3.1 nightlies. A file that parsed only because a bad line sat at a chunk start will now raise or skip as on_bad_lines says.

Extension authors should switch pandas.core.accessor.CachedAccessor to Accessor. The alias now warns.