Prevent XLS precision loss for large integers#637
Conversation
XLS stores numbers as 64-bit floats with 53-bit mantissa, so integers above 2^53-1 lose precision. Convert such integers to text strings to preserve exact values. Fixes jazzband#197
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #637 +/- ##
==========================================
- Coverage 93.14% 93.12% -0.03%
==========================================
Files 29 29
Lines 3226 3229 +3
==========================================
+ Hits 3005 3007 +2
- Misses 221 222 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for the PR, please add tests. I've just merged a fix for the lint teyit failure, please also rebase/merge |
|
Another approach would have been to document that the XLS format is not adequate to store large integers. Is that format not on a disappearing trend? |
|
I'm fine with either approach. This fix seems simple enough, but I expect XLS use is very low, and outputting XLS even lower. A docs change is safer, and we've not exactly had a huge number of requests for this. |
Summary
Problem
XLS (Excel 97-2003) stores numeric values as 64-bit IEEE 754 floats with a 53-bit mantissa. Integers larger than 9,007,199,254,740,991 (2^53 - 1) lose precision when stored as numbers. For example,
568883383628111872becomes568883383628112000.Fix
In
dset_sheet, detect integers that exceed the float precision limit and convert them to strings before writing. Small integers continue to be stored as numbers for normal behavior.Testing
Verified that:
568883383628111872) are stored as text (cell type 1) and read back with full precision42) continue to be stored as numbers (cell type 2)Fixes #197