[Go] RecordFromJSON does not handle integer values not representable by double

[lidavidm]

Describe the bug, including details regarding any error messages, version, and platform.

This limits the usefulness of RecordFromJSON for testing, and implies we may have been writing incorrect tests.

You silently get wrong data:

package main

import (
	"bytes"
	"fmt"
	"log"

	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/arrow-go/v18/arrow/memory"
)

func main() {
	mem := memory.DefaultAllocator
	dt := arrow.FixedWidthTypes.Duration_s
	record, _, err := array.FromJSON(mem, dt, bytes.NewReader([]byte("[9223372036854775807]")))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(record)
}

Result:

[-9223372036854775808]

Commonly, large integers are represented as strings in JSON to get around this. That doesn't work:

package main

import (
	"bytes"
	"fmt"
	"log"

	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/arrow-go/v18/arrow/memory"
)

func main() {
	mem := memory.DefaultAllocator
	dt := arrow.FixedWidthTypes.Duration_s
	record, _, err := array.FromJSON(mem, dt, bytes.NewReader([]byte(`["9223372036854775807"]`)))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(record)
}

Result:

2026/05/07 15:36:14 json: cannot unmarshal 9223372036854775807 into Go value of type arrow.Duration
exit status 1

Component(s)

Other

[zeroshade]

Have you tried using the https://pkg.go.dev/github.com/apache/arrow-go/v18/arrow/array#WithUseNumber option? That should solve the issue for most cases (unless we missed one)

[zeroshade]

Ah, it looks like the string handling expects it to be in the form "3h2m5s" etc. for go's ParseDuration rather than being a number, that's why it fails in the string case.

Using WithUseNumber does fix the first case though.

package main

import (
	"bytes"
	"fmt"
	"log"

	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/arrow-go/v18/arrow/memory"
)

func main() {
	mem := memory.DefaultAllocator
	dt := arrow.FixedWidthTypes.Duration_s
	record, _, err := array.FromJSON(mem, dt, bytes.NewReader([]byte("[9223372036854775807]")), array.WithUseNumber())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(record)
}

[9223372036854775807]

[lidavidm]

Is there any reason not to enable that by default? Silently corrupting data seems bad...

[zeroshade]

Mostly that it does have a slight effect on performance potentially, though unless you're converting very large json blobs you probably wouldn't notice. Essentially using json.Number will keep it as a string and parse on the fly rather than creating the float64 during the json.Unmarshal in the first place. So it would cause a bunch of extra allocations/string parsing if you don't need it.

I suppose we could deprecate the WithUseNumber option, make it a no-op, add a new WithNoUseNumber option and make using the json.Number the default.

[lidavidm]

I guess it's workable then. I'll just adjust the test utilities.