GH-50567: [C++] Introduce JsonWriter and migrate integration JSON writer by Reranko05 · Pull Request #50568 · apache/arrow

Reranko05 · 2026-07-21T04:46:16Z

What changes are included?

This PR introduces a reusable JsonWriter wrapper around simdjson's JSON builder API and migrates the integration JSON writer to use it instead of RapidJSON.

Specifically, this PR:

Adds a reusable JsonWriter abstraction in arrow/json.
Migrates the integration JSON writer implementation to JsonWriter.
Updates the integration tests to use JsonWriter.
Adds unit tests for JsonWriter.

This is part of the incremental migration from RapidJSON to simdjson.

Performance

I compared the serialization performance of IntegrationJsonWriter::WriteRecordBatch() + Finish() using a temporary benchmark with a 1M-row RecordBatch. The benchmark was run 10 times on both main and this branch.

Branch	Average Time
`main`	2157.4 ms
This PR	2153.6 ms

No measurable performance regression was observed.

Are these changes tested?

Yes.

I added unit tests for JsonWriter and verified that both JSON and integration tests pass locally:

arrow-json-test
arrow-json-integration-test

Are there any user-facing changes?

No.

GitHub Issue: [C++] Introduce JsonWriter and migrate integration JSON writer #50567

github-actions · 2026-07-21T04:46:40Z

⚠️ GitHub issue #50567 has been automatically assigned in GitHub to PR creator.

Reranko05 · 2026-07-21T06:29:27Z

@kou could you review this PR when you have a chance?

kou

Could you check performance difference?

Reranko05 · 2026-07-21T07:31:38Z

@kou, I compared the serialization performance of IntegrationJsonWriter::WriteRecordBatch() + Finish() using a temporary benchmark with a 1M-row RecordBatch. I ran the benchmark 10 times on both main and this branch. After excluding one obvious outlier from each run, the averages were:

main: 2157.4 ms
this branch: 2153.6 ms

I didn't observe any measurable performance regression.

Reranko05 · 2026-07-21T08:38:50Z

@kou I investigated the failing workflows and they appear to be unrelated to this PR. Could you please verify?

rok

Thanks for working on this @Reranko05! Exciting to see this work!

We should probably add meson support for json_writer, see (cpp/src/arrow/meson.build, cpp/src/arrow/json/meson.build, cpp/src/arrow/integration/meson.build).

rok · 2026-07-21T12:45:33Z

+ private:
+  void MaybeComma();
+
+  simdjson::builder::string_builder builder_;


We currently require simdjson 3.0.0, but builder API was introduced in simdjson 4.0.0. Are we sure this will build for 3.0.0 <= version < 4.0.0? Can you do a (local) test to show what's the minimum simdjson version we need for introduced features?

https://repology.org/project/simdjson/versions - ubuntu seems to lag a bit.

Thanks, that's a good point. Ubuntu is lagging so bumping up the minimum version will not work.

I will look into reworking the implementation to remain compatible with 3.x.

I am not sure writing is possible pre-4.x so we likely want to raise the version. If we do so, how do we support ubuntu<25?
cc @lemire

rok · 2026-07-21T12:50:27Z

Another thought - do we have conbench cpp micro benchmarks to monitor improvement of json parsing and writing?

Copilot

Pull request overview

This PR adds a reusable arrow::json::JsonWriter wrapper around simdjson’s builder API and migrates the C++ integration JSON writer (and its tests) away from RapidJSON’s writer, as part of the incremental RapidJSON → simdjson migration.

Changes:

Added arrow::json::JsonWriter (header/impl) plus unit tests.
Migrated integration JSON serialization code paths to emit JSON using JsonWriter.
Updated CMake/test linkage and CI environment to ensure simdjson is available/linked where needed.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cpp/src/arrow/json/json_writer.h	Introduces the `JsonWriter` public API.
cpp/src/arrow/json/json_writer.cc	Implements `JsonWriter` on top of simdjson builder primitives.
cpp/src/arrow/json/json_writer_test.cc	Adds unit coverage for basic writer operations.
cpp/src/arrow/json/CMakeLists.txt	Adds the new unit test and links simdjson for the test target.
cpp/src/arrow/integration/json_internal.h	Updates writer function signatures to accept `arrow::json::JsonWriter*`.
cpp/src/arrow/integration/json_internal.cc	Migrates integration JSON emission to `JsonWriter` (writer-side).
cpp/src/arrow/integration/json_integration.cc	Switches `IntegrationJsonWriter` implementation to `JsonWriter`.
cpp/src/arrow/integration/json_integration_test.cc	Updates integration tests to use `JsonWriter` for schema/array JSON generation.
cpp/src/arrow/integration/CMakeLists.txt	Links simdjson where integration tests/executables now depend on it.
cpp/src/arrow/CMakeLists.txt	Adds `json/json_writer.cc` to Arrow JSON sources and links simdjson for integration targets.
ci/docker/ubuntu-24.04-cpp.dockerfile	Sets `simdjson_SOURCE=BUNDLED` to ensure simdjson availability in CI images.


 namespace arrow::internal::integration::json {

 /// \brief Append integration test Schema format to rapidjson writer


+  ArrayWriter(const std::string& name, const Array& array, JsonWriter* writer)
      : name_(name), array_(array), writer_(writer) {}


Reranko05 · 2026-07-21T14:35:10Z

Another thought - do we have conbench cpp micro benchmarks to monitor improvement of json parsing and writing?

I wasn't able to find any existing Conbench C++ microbenchmarks covering JSON parsing or writing. I did run a local benchmark comparing IntegrationJsonWriter::WriteRecordBatch() + Finish() on this branch against main and didn't observe any measurable regression. If I missed an existing Conbench benchmark, please let me know and I will run this against it.

rok · 2026-07-21T15:09:34Z

If I missed an existing Conbench benchmark, please let me know and I will run this against it.

It then seems we currently don't benchmark serializing and deserializing json. It would be useful to introduce some, but perhaps that is out of scope for this ticket.

rok · 2026-07-21T15:15:35Z

See this PR for an example how to implement a benchmark.

pitrou · 2026-07-21T15:30:53Z

It then seems we currently don't benchmark serializing and deserializing json. It would be useful to introduce some, but perhaps that is out of scope for this ticket.

It doesn't sound useful to benchmark JSON parsing for integration testing.

rok · 2026-07-21T15:32:18Z

It seems we do have some - search for json on http://conbench.arrow-dev.org/c-benchmarks/.
Here are examples: ReadJSONBlockWithSchemaSingleThread, ReadJSONBlockWithSchemaMultiThread. Source code.

pitrou · 2026-07-21T15:35:05Z

They are testing the JSONL reader, not the JSON integration reader.

rok · 2026-07-21T15:55:23Z

It doesn't sound useful to benchmark JSON parsing for integration testing.

yeah, that is not very interesting indeed.

@Reranko05 sorry for bringing up benchmarks here, we likely don't need new ones here.

@pitrou did we ever discuss JSONL writer?

pitrou · 2026-07-21T16:07:34Z

@pitrou did we ever discuss JSONL writer?

There's probably an issue open for it :)

rok · 2026-07-21T16:30:32Z

@pitrou did we ever discuss JSONL writer?

There's probably an issue open for it :)

There is an issue and a PR.

apacheGH-35460: Add JsonWriter wrapper for simdjson builder API

2b7f64e

github-actions Bot added Component: C++ awaiting review Awaiting review labels Jul 21, 2026

Reranko05 force-pushed the gh-35460-json-writer-v2 branch 3 times, most recently from fd69d50 to bb7fd90 Compare July 21, 2026 06:02

kou reviewed Jul 21, 2026

View reviewed changes

Comment thread cpp/src/arrow/integration/json_internal.cc Outdated

kou added the CI: Extra Run extra CI label Jul 21, 2026

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Jul 21, 2026

apacheGH-35460: Migrate integration RapidJSON writer to JsonWriter

c983eb0

Reranko05 force-pushed the gh-35460-json-writer-v2 branch from bb7fd90 to c983eb0 Compare July 21, 2026 12:41

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 21, 2026

rok requested changes Jul 21, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jul 21, 2026

rok requested a review from Copilot July 21, 2026 13:06

Copilot started reviewing on behalf of rok July 21, 2026 13:07 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

pitrou reviewed Jul 21, 2026

View reviewed changes

Comment thread cpp/src/arrow/integration/json_internal.h Outdated

Comment thread cpp/src/arrow/json/json_writer_internal.h

Comment thread cpp/src/arrow/json/json_writer.h Outdated

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 21, 2026

Reranko05 force-pushed the gh-35460-json-writer-v2 branch from cfe491e to a25a245 Compare July 21, 2026 14:45

Make JsonWriter internal and update Meson

106ee16

Reranko05 force-pushed the gh-35460-json-writer-v2 branch from a25a245 to 106ee16 Compare July 21, 2026 14:50


		namespace arrow::internal::integration::json {

		/// \brief Append integration test Schema format to rapidjson writer

		ArrayWriter(const std::string& name, const Array& array, JsonWriter* writer)
		: name_(name), array_(array), writer_(writer) {}

Uh oh!

Conversation

Reranko05 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are included?

Performance

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

Reranko05 commented Jul 21, 2026

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reranko05 commented Jul 21, 2026

Uh oh!

Reranko05 commented Jul 21, 2026

Uh oh!

rok left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reranko05 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reranko05 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reranko05 commented Jul 21, 2026

Uh oh!

rok commented Jul 21, 2026

Uh oh!

rok commented Jul 21, 2026

Uh oh!

pitrou commented Jul 21, 2026

Uh oh!

rok commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Jul 21, 2026

Uh oh!

rok commented Jul 21, 2026

Uh oh!

pitrou commented Jul 21, 2026

Uh oh!

rok commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Reranko05 commented Jul 21, 2026 •

edited

Loading

rok left a comment •

edited

Loading

Reranko05 Jul 21, 2026 •

edited

Loading

rok Jul 21, 2026 •

edited

Loading

rok commented Jul 21, 2026 •

edited

Loading