petl v1.7.24 Fixes Avro Decimal Inference

The Python ETL library petl tagged v1.7.24 on 19 August 2026. The only code change is a fix to Avro schema inference for Python Decimal values when toavro() runs without an explicit schema. Zero amounts, scientific notation, and mixed fractional scales no longer crash the writer or emit a decimal logical type that Apache Avro rejects.

The full release notes and downloads are on the GitHub release page. The tag is a patch over v1.7.23, not a prerelease.

When toavro() has no schema= argument it inspects column values. A Decimal column becomes a bytes field with logicalType set to decimal. Precision and scale come from precision_and_scale() in petl/io/avro.py.

That helper was a copy of fastavro’s prepare_bytes_decimal() encoder. An encoder rescales a value to a scale the schema already fixed. Inference has no scale yet, so delta was hardcoded to 1 and the leftover comment # delta = exp + scale stayed in the file. Digit count then used math.ceil(math.log10(abs(inumber))).

Any zero trips that call. Decimal('0'), Decimal('0.00'), Decimal('-0'), and Decimal('0E+2') all have digits (0,), so inumber is 0 and log10(0) raises ValueError: math domain error. A zero balance is a normal row. Nested arrays and nested records recurse through _get_definition_from_type_of.

This write aborted:

from decimal import Decimal
import petl as etl

etl.toavro(
    [['id', 'amount'], [1, Decimal('0.00')], [2, Decimal('12.34')]],
    'out.avro',
)

The Avro spec says scale must be zero or a positive integer less than or equal to precision. The old floor max(prec, 8) was applied too early. Decimal('1E-9') therefore inferred precision 8 and scale 9. fastavro refused the schema with SchemaParseException: decimal scale must be less than or equal to the precision of 8.

A positive exponent was counted as fractional digits. abs(exp) turned Decimal('1E+2'), an integer, into scale 2. A decimal is unscaled * 10**-scale, so an integral value has scale 0.

Pull request 706 by gaoflow now counts digits from the coefficient and the exponent:

scale = -exp if exp < 0 else 0
int_digits = len(digits) + exp
prec = max(int_digits + scale, scale, 1)

The math import is gone from petl/io/avro.py. max(prec, 8) remains, but it now applies only to the emitted precision so the floor does not feed back into the running estimate.

Non finite Decimal values (NaN, Infinity) have no encoding under any Avro decimal schema. They used to fail inside the digit count with the same domain error. They now raise ValueError that names the value.

Inference still samples the column. The old combine took max(prec) and max(scale) independently. Writing then rescales each value to the schema scale, which widens the unscaled integer by the gap between the two scales.

Take a column [Decimal('123456789.1'), Decimal('0.12345678901')]. The inferred type was precision 12, scale 11. At that scale the first value needs 20 unscaled digits. fastavro only checks len(digits) against precision, so the write succeeded and the file declared a precision that did not cover its contents. A stricter reader can reject that file.

Avro readers take precision and scale from the schema, not from the payload. An inferred schema that understates precision is a silent contract break for any job that treats the file as typed decimal data.

The new combine tracks integer part digits and scale separately, then sets precision so every sampled value fits after rescale. Tests landed in petl/test/io/test_avro.py covering those cases plus nested structures and a table of (value, precision, scale) pairs. The pull request reports an audit of 43 Decimal column shapes, 21 of which failed on v1.7.23 (crash, rejected schema, or precision too small) and none after the change.

Callers that already pass schema= to toavro() do not use this path. This release does not change encoding under an explicit schema.

Callers that let petl infer the schema for Decimal columns will see different precision and scale than v1.7.23. That is the fix, not a side effect.

Two operator visible shifts:

  • Tables that raised ValueError: math domain error on a zero now write an Avro file.
  • Tables with NaN or Infinity now fail with an explicit ValueError instead of the same domain error.

If a job stored an inferred schema from an older write and compares it to a new file, infer again on v1.7.24 before treating a mismatch as a pipeline bug. The changelog from v1.7.23 to v1.7.24 is this single pull request.