Preserve nullable integer columns in to_json/to_csv/to_sql#8366
Open
ebarkhordar wants to merge 1 commit into
Open
Preserve nullable integer columns in to_json/to_csv/to_sql#8366ebarkhordar wants to merge 1 commit into
ebarkhordar wants to merge 1 commit into
Conversation
The JSON, CSV and SQL writers call batch.to_pandas() with pyarrow's default integer_object_nulls=False, which casts any integer column containing a null to float64. This dropped the integer type on export and corrupted values past 2**53, which float64 cannot represent exactly. Pass integer_object_nulls=True so a null-containing integer column keeps Python integers (object dtype) and is written as integers. Columns without nulls are unaffected. Fixes huggingface#8365
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Dataset.to_json,to_csvandto_sqlwrite a nullable integer column as floating point rather than integers.Fixes #8365.
Root cause
The three writers build a pandas frame with a bare
batch.to_pandas():io/json.py_batch_jsonio/csv.py_batch_csvio/sql.py_batch_sqlpyarrow's
to_pandasdefaults tointeger_object_nulls=False, which casts any integer column that contains a null tofloat64. On export that drops the integer type, and for values beyond2**53it corrupts the value, since those are not representable in float64. Ato_jsonthenload_dataset("json")round-trip silently changesint64intofloat64.Fix
Pass
integer_object_nulls=Trueto the three writers'to_pandas()calls. A null-containing integer column then materializes as Python integers (object dtype) and is written as integers; the null stays null. Columns with no nulls are untouched by the flag, so their numpy dtype is preserved.Tradeoff:
integer_object_nulls=Trueuses object dtype for integer columns that contain a null, which costs memory and speed for those columns only. An arrow-backed nullable dtype (pandasInt64via atypes_mapper) would avoid object dtype but touches more of the export path. Glad to switch to that approach if you prefer it.Verification
Ran in a clean container against this branch (datasets 5.0.1.dev0, pyarrow 25.0.0, pandas 3.0.5):
int64/uint64/int32. They fail onmain(the column exports as float, e.g.9007199254740992.0, and SQLite storesreal) and pass here (exact integers, SQLiteinteger).tests/io/test_json.py tests/io/test_csv.py tests/io/test_sql.py: 96 passed, no regressions.ruff checkandruff format --checkclean on the changed files.I did not run the full non-io test suite or the Windows and minimum-deps CI matrix locally.