Skip to content

fix: resolve string aggfun to callable in table_pivot step#1788

Open
apoorvdarshan wants to merge 1 commit into
frictionlessdata:mainfrom
apoorvdarshan:fix/table-pivot-string-aggfun-1764
Open

fix: resolve string aggfun to callable in table_pivot step#1788
apoorvdarshan wants to merge 1 commit into
frictionlessdata:mainfrom
apoorvdarshan:fix/table-pivot-string-aggfun-1764

Conversation

@apoorvdarshan

Copy link
Copy Markdown

Summary

A table-pivot step defined via a JSON/YAML descriptor with "aggfun": "sum" fails, while the equivalent Python API steps.table_pivot(..., aggfun=sum) works, even though the docs (and the step's own docstring) say descriptors mirror the Python API and that aggfun "can be sum, max, min, len etc.".

Reproducer from the issue:

json_descriptor = {
    "steps": [
        {"type": "table-normalize"},
        {"type": "table-pivot", "f1": "region", "f2": "gender", "f3": "units", "aggfun": "sum"},
    ]
}
Resource(data=data).transform(Pipeline.from_descriptor(json_descriptor))

fails with:

FrictionlessException: [step-error] Step is not valid: "table_pivot" raises
"[source-error] ... 'str' object is not callable"

Root cause: aggfun is typed Any with an empty metadata profile, so a JSON string like "sum" passes validation and is handed straight to petl's table.pivot(f1, f2, f3, aggfun), which requires a callable, not a string.

Fix

In frictionless/steps/table/table_pivot.py, transform_resource now resolves a string aggfun to the corresponding callable before calling petl.pivot, via a small map of the standard petl aggregation names:

sum, max, min, len, count (alias for len), mean (statistics.mean), first, last.

An actual callable aggfun is passed through unchanged (the existing Python-API behavior). An unrecognized name raises a clear StepError (aggregation function "<name>" is not supported) instead of the opaque petl TypeError. The docstring was updated to list the supported names.

Tests

Added test_step_table_pivot_string_aggfun_issue_1764 in frictionless/steps/table/__spec__/test_table_pivot.py, mirroring the existing test_step_table_pivot_issue_1220 but building the pipeline from a descriptor with "aggfun": "sum". It fails on main ('str' object is not callable) and passes with this change. Full frictionless/steps, frictionless/pipeline, and frictionless/transformer suites pass; ruff check, ruff format --check, and pyright are clean on the changed files.

Note on scope

This fixes the descriptor -> callable (input) direction, which is the reported failure and matches the documented usage. The issue also mentions that pipeline.to_json() fails when a Python callable was passed directly (e.g. the builtin sum), because a function is not JSON serializable. Reliably serializing an arbitrary callable back to a descriptor name is a separate design decision (lambdas / user functions have no canonical name), so it is intentionally left out here; using a string aggfun in the descriptor now round-trips cleanly through to_json/from_descriptor.

Disclosure: prepared with AI assistance; reviewed and verified locally.

The table_pivot step passed `aggfun` straight to petl's `table.pivot`,
which requires a callable. When the step is defined via a descriptor
(JSON/YAML), `aggfun` arrives as a string such as "sum", causing
`'str' object is not callable`, even though the Python API with a
callable works and the docstring advertises string names.

Resolve a string `aggfun` to the corresponding callable via a small map
of the standard petl aggregation names (sum, max, min, len, count, mean,
first, last) before calling petl.pivot. Actual callables are passed
through unchanged. An unknown name raises a clear StepError.

Fixes frictionlessdata#1764.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Pipeline.from_descriptor does not work for all steps

1 participant