Skip to content

Preserve nullable integer columns in to_json/to_csv/to_sql#8366

Open
ebarkhordar wants to merge 1 commit into
huggingface:mainfrom
ebarkhordar:fix/8365-nullable-int-export-float
Open

Preserve nullable integer columns in to_json/to_csv/to_sql#8366
ebarkhordar wants to merge 1 commit into
huggingface:mainfrom
ebarkhordar:fix/8365-nullable-int-export-float

Conversation

@ebarkhordar

Copy link
Copy Markdown
Contributor

What this fixes

Dataset.to_json, to_csv and to_sql write a nullable integer column as floating point rather than integers.

Fixes #8365.

Root cause

The three writers build a pandas frame with a bare batch.to_pandas():

  • io/json.py _batch_json
  • io/csv.py _batch_csv
  • io/sql.py _batch_sql

pyarrow's to_pandas defaults to integer_object_nulls=False, which casts any integer column that contains a null to float64. On export that drops the integer type, and for values beyond 2**53 it corrupts the value, since those are not representable in float64. A to_json then load_dataset("json") round-trip silently changes int64 into float64.

Fix

Pass integer_object_nulls=True to the three writers' to_pandas() calls. A null-containing integer column then materializes as Python integers (object dtype) and is written as integers; the null stays null. Columns with no nulls are untouched by the flag, so their numpy dtype is preserved.

Tradeoff: integer_object_nulls=True uses object dtype for integer columns that contain a null, which costs memory and speed for those columns only. An arrow-backed nullable dtype (pandas Int64 via a types_mapper) would avoid object dtype but touches more of the export path. Glad to switch to that approach if you prefer it.

Verification

Ran in a clean container against this branch (datasets 5.0.1.dev0, pyarrow 25.0.0, pandas 3.0.5):

  • Added regression tests for all three writers, parametrized over int64/uint64/int32. They fail on main (the column exports as float, e.g. 9007199254740992.0, and SQLite stores real) and pass here (exact integers, SQLite integer).
  • tests/io/test_json.py tests/io/test_csv.py tests/io/test_sql.py: 96 passed, no regressions.
  • ruff check and ruff format --check clean on the changed files.

I did not run the full non-io test suite or the Windows and minimum-deps CI matrix locally.

The JSON, CSV and SQL writers call batch.to_pandas() with pyarrow's default
integer_object_nulls=False, which casts any integer column containing a null
to float64. This dropped the integer type on export and corrupted values past
2**53, which float64 cannot represent exactly. Pass integer_object_nulls=True
so a null-containing integer column keeps Python integers (object dtype) and is
written as integers. Columns without nulls are unaffected.

Fixes huggingface#8365
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

to_json/to_csv/to_sql cast nullable integer columns to float, corrupting values past 2**53

1 participant