petl Week: cutout, valuecounts and Pivot Correctness Fixes


petl is a Python library for extracting, transforming and loading tabular data. The last seven days brought 26 commits, 53 files changed and 988 insertions, most of it edge case fixes queued under version 1.7.33 in the changelog. Nothing here is a new feature. It is a cleanup of places where petl silently did the wrong thing.

The most recent commit, fix negative field indexes in cutout, changes itercutout in petl/transform/basics.py. Negative indexes now count from the end of the header and remove the matching column. Indexes outside the header raise FieldSelectionError. The fix also reaches unjoin() when called with an explicit key and a negative value index.

The core of the change is small:

for i in asindices(hdr, spec):
    if i < -len(hdr):
        raise FieldSelectionError(i)
    indicesout.add(i if i >= 0 else len(hdr) + i)

The changelog says negative indexes now remove the selected columns, which implies they did not before. If a pipeline passed -1 and still got the column back, that was a silent bug. After the upgrade the output has fewer columns. Check for that, and check any code that depended on the old shape.

The same theme shows up in petl/util/counting.py. One commit accepts a single tuple or list in valuecounter() and valuecounts(), matching what values() already did. A follow up raises FieldSelectionError for a nested field selection so malformed input fails loudly. Two more commits are Python 2.7 CI repairs: one keeps the file ASCII only, one makes a test deterministic. So the project still tests on Python 2.7.

Real data has mixed types and missing values. Most of the pivot work sits in petl/transform/reshape.py.

Separately, recast handles mixed type variable names, coalesce compares missing values by equality, and fill all columns by position when no fields are given. The tied category fix is the one that changes numbers. If a pivot output ever looked short, rerun it.

The IO fixes touch resource handling and output safety.

Preserve existing data when appending to MemorySource changes petl/io/sources.py. The title says it all: appending should keep what the buffer already held. Close remote compressed response streams makes sure the stream is closed after reading. For long running jobs that read many remote files, unclosed streams are the kind of resource problem that only shows up under load.

Preserve iterator headers when rereading text touches petl/io/text.py. Clear merged cells when replacing an XLSX sheet clears merged ranges when a sheet is replaced, so the old layout does not leak into the new data.

The security relevant one is HTML escape cell and header text in tohtml() and teehtml(). Cell and header text is now escaped. Markup in cell values would otherwise land in the page as is. If you render untrusted data into reports with tohtml(), that is an injection path, and the fix closes it. The tradeoff: anyone who relied on embedding HTML in cells to get links or styling will see literal text now.

  • The work is recorded in docs/changes.rst under 1.7.33, which is not released yet as far as the input shows. Read it before pinning.
  • Audit calls to cutout() with negative indexes and tohtml() with embedded markup. Both change output.
  • Python 2.7 CI jobs still shape code style, such as the ASCII only rule in counting.py.