fix: resolve string aggfun to callable in table_pivot step#1788
Open
apoorvdarshan wants to merge 1 commit into
Open
fix: resolve string aggfun to callable in table_pivot step#1788apoorvdarshan wants to merge 1 commit into
apoorvdarshan wants to merge 1 commit into
Conversation
The table_pivot step passed `aggfun` straight to petl's `table.pivot`, which requires a callable. When the step is defined via a descriptor (JSON/YAML), `aggfun` arrives as a string such as "sum", causing `'str' object is not callable`, even though the Python API with a callable works and the docstring advertises string names. Resolve a string `aggfun` to the corresponding callable via a small map of the standard petl aggregation names (sum, max, min, len, count, mean, first, last) before calling petl.pivot. Actual callables are passed through unchanged. An unknown name raises a clear StepError. Fixes frictionlessdata#1764.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
table-pivotstep defined via a JSON/YAML descriptor with"aggfun": "sum"fails, while the equivalent Python APIsteps.table_pivot(..., aggfun=sum)works, even though the docs (and the step's own docstring) say descriptors mirror the Python API and thataggfun"can besum,max,min,lenetc.".Reproducer from the issue:
fails with:
Root cause:
aggfunis typedAnywith an empty metadata profile, so a JSON string like"sum"passes validation and is handed straight to petl'stable.pivot(f1, f2, f3, aggfun), which requires a callable, not a string.Fix
In
frictionless/steps/table/table_pivot.py,transform_resourcenow resolves a stringaggfunto the corresponding callable before callingpetl.pivot, via a small map of the standard petl aggregation names:sum,max,min,len,count(alias forlen),mean(statistics.mean),first,last.An actual callable
aggfunis passed through unchanged (the existing Python-API behavior). An unrecognized name raises a clearStepError(aggregation function "<name>" is not supported) instead of the opaque petlTypeError. The docstring was updated to list the supported names.Tests
Added
test_step_table_pivot_string_aggfun_issue_1764infrictionless/steps/table/__spec__/test_table_pivot.py, mirroring the existingtest_step_table_pivot_issue_1220but building the pipeline from a descriptor with"aggfun": "sum". It fails onmain('str' object is not callable) and passes with this change. Fullfrictionless/steps,frictionless/pipeline, andfrictionless/transformersuites pass;ruff check,ruff format --check, andpyrightare clean on the changed files.Note on scope
This fixes the descriptor -> callable (input) direction, which is the reported failure and matches the documented usage. The issue also mentions that
pipeline.to_json()fails when a Python callable was passed directly (e.g. the builtinsum), because a function is not JSON serializable. Reliably serializing an arbitrary callable back to a descriptor name is a separate design decision (lambdas / user functions have no canonical name), so it is intentionally left out here; using a stringaggfunin the descriptor now round-trips cleanly throughto_json/from_descriptor.Disclosure: prepared with AI assistance; reviewed and verified locally.