petl is a Python library for extracting, transforming and loading tabular data. The last seven days brought 26 commits, 53 files changed and 988 insertions, most of it edge case fixes queued under version 1.7.33 in the changelog. Nothing here is a new feature. It is a cleanup of places where petl silently did the wrong thing.
Field selection: cutout and valuecounts ¶
The most recent commit, fix negative field indexes in cutout, changes itercutout in petl/transform/basics.py. Negative indexes now count from the end of the header and remove the matching column. Indexes outside the header raise FieldSelectionError. The fix also reaches unjoin() when called with an explicit key and a negative value index.
The core of the change is small:
for i in asindices(hdr, spec):
if i < -len(hdr):
raise FieldSelectionError(i)
indicesout.add(i if i >= 0 else len(hdr) + i)
The changelog says negative indexes now remove the selected columns, which implies they did not before. If a pipeline passed -1 and still got the column back, that was a silent bug. After the upgrade the output has fewer columns. Check for that, and check any code that depended on the old shape.
The same theme shows up in petl/util/counting.py. One commit accepts a single tuple or list in valuecounter() and valuecounts(), matching what values() already did. A follow up raises FieldSelectionError for a nested field selection so malformed input fails loudly. Two more commits are Python 2.7 CI repairs: one keeps the file ASCII only, one makes a test deterministic. So the project still tests on Python 2.7.
Pivot, recast and coalesce with messy data ¶
Real data has mixed types and missing values. Most of the pivot work sits in petl/transform/reshape.py.
- Use PETL mixed type ordering for pivot categories stops category sorting from depending on Python’s native comparison.
- Keep all pivot values when distinct categories sort as ties fixes a data loss case. Two categories that compare equal in ordering no longer collapse into one.
- Several test commits cover mixed, missing and tied categories across the different sort paths, and a docs commit records how aggregation behaves for ties.
Separately, recast handles mixed type variable names, coalesce compares missing values by equality, and fill all columns by position when no fields are given. The tied category fix is the one that changes numbers. If a pivot output ever looked short, rerun it.
IO: sources, text, XLSX and HTML ¶
The IO fixes touch resource handling and output safety.
Preserve existing data when appending to MemorySource changes petl/io/sources.py. The title says it all: appending should keep what the buffer already held. Close remote compressed response streams makes sure the stream is closed after reading. For long running jobs that read many remote files, unclosed streams are the kind of resource problem that only shows up under load.
Preserve iterator headers when rereading text touches petl/io/text.py. Clear merged cells when replacing an XLSX sheet clears merged ranges when a sheet is replaced, so the old layout does not leak into the new data.
The security relevant one is HTML escape cell and header text in tohtml() and teehtml(). Cell and header text is now escaped. Markup in cell values would otherwise land in the page as is. If you render untrusted data into reports with tohtml(), that is an injection path, and the fix closes it. The tradeoff: anyone who relied on embedding HTML in cells to get links or styling will see literal text now.
What to watch ¶
- The work is recorded in docs/changes.rst under 1.7.33, which is not released yet as far as the input shows. Read it before pinning.
- Audit calls to
cutout()with negative indexes andtohtml()with embedded markup. Both change output. - Python 2.7 CI jobs still shape code style, such as the ASCII only rule in
counting.py.