AVRO-4258: [java] remove some allocations from decoder.getBytes() and Utf8 handling by mkeskells · Pull Request #3776 · apache/avro

mkeskells · 2026-05-12T23:06:51Z

Contribution Checklist

What is the purpose of the change

To reduce some allocations found during profiling for other changes (#3746)

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.
There is some trivial addition to the tests but only to ensure the methods are called

Note - I can't verify the change locally as Arvo doesn't build for me locally

Documentation

Does this pull request introduce a new feature? (no)

…ndling

mkeskells · 2026-06-01T08:36:47Z

Hi @opwvhk
I see spotless, and test errors on the run you started (thanks), but they seem to be on files I have not chnaged.
I am travelling today, so can only look on my phone so cant be sure

opwvhk · 2026-06-01T08:55:01Z

I've taken the liberty to run spotless.

mkeskells · 2026-06-01T10:47:18Z

Hi @opwvhk
feel free to make any chnages you feel appropriate, I am not precious on my PRs!
I cant run spotless before push, as it corrupts the code to a state where it doesnt compile
I am travelling and on hoiday, so I dont have a dev env with me, so happy to take any input on the 3 PRs I have open

mkeskells · 2026-06-01T10:59:53Z

I have merged main

Now colocated with `readBytes(ByteBuffer)` and `skipBytes()` implementations.

move the implementation to child classes reset utf8 string

…58-getBytes # Conflicts: # lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java # lang/java/avro/src/main/java/org/apache/avro/io/Decoder.java # lang/java/avro/src/main/java/org/apache/avro/io/ResolvingDecoder.java # lang/java/avro/src/main/java/org/apache/avro/util/Utf8.java

mkeskells · 2026-06-01T15:54:25Z

Hi @opwvhk
I did some chnages while offline (much the same, push down and utf fixes), and merged with yours

Fokko

Nice!

mkeskells · 2026-06-08T14:10:11Z

Hi
I don't know the process here, is this able to be merged?
Also if someone can advise me on the best way to progress these issues, I can learn and use that knowledge on the other PRs that I have :-)

Copilot

Pull request overview

This PR introduces a new Decoder.readBytes() API that returns a raw byte[] to reduce allocations (avoiding intermediate ByteBuffer creation) and updates several call sites (bytes→string promotion, file header metadata parsing, conversions) to use it.

Changes:

Add Decoder.readBytes() and implement/override it in core decoder implementations (BinaryDecoder, DirectBinaryDecoder, JsonDecoder, ResolvingDecoder, ValidatingDecoder).
Switch internal call sites that immediately materialize byte[] (or promote bytes→string/Utf8) to use the new readBytes() path.
Add small tests to ensure the new readBytes() method is exercised for EOF/negative length/VM limit cases.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
lang/java/avro/src/main/java/org/apache/avro/io/Decoder.java	Adds new `readBytes()` API intended to avoid `ByteBuffer` allocations.
lang/java/avro/src/main/java/org/apache/avro/io/BinaryDecoder.java	Implements `readBytes()` returning an exactly-sized byte array.
lang/java/avro/src/main/java/org/apache/avro/io/DirectBinaryDecoder.java	Implements `readBytes()` for direct (non-buffering) decoding.
lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java	Implements `readBytes()` for JSON decoding.
lang/java/avro/src/main/java/org/apache/avro/io/ResolvingDecoder.java	Uses `readBytes()` for bytes/string resolution and removes some intermediate allocations.
lang/java/avro/src/main/java/org/apache/avro/io/ValidatingDecoder.java	Adds validating wrapper implementation of `readBytes()`.
lang/java/avro/src/main/java/org/apache/avro/io/FastReaderBuilder.java	Uses new raw-bytes API for bytes→string promotion and Utf8 reuse.
lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectDatumReader.java	Uses raw-bytes API when reader expects `byte[]`.
lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java	Reads header metadata values via `readBytes()` directly into `byte[]`.
lang/java/avro/src/main/java/org/apache/avro/file/DataFileReader12.java	Reads metadata values via `readBytes()` directly into `byte[]`.
lang/java/avro/src/main/java/org/apache/avro/Conversions.java	Uses `readBytes()` directly instead of `ByteBuffer.array()`.
lang/java/avro/src/test/java/org/apache/avro/io/TestBinaryDecoder.java	Adds basic coverage for `readBytes()` EOF/negative length/VM limit.
lang/java/avro/src/test/java/org/apache/avro/util/TestUtf8.java	Adds light coverage to ensure the new `Utf8.set(byte[])` path is exercised.
lang/java/avro/src/main/java/org/apache/avro/util/Utf8.java	Adds `Utf8.set(byte[])` for reuse without extra wrapper allocations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mkeskells · 2026-06-21T11:03:07Z

+  /**
+   * Reads a byte-string written by {@link Encoder#writeBytes}.
+   * <p>
+   * This is useful when you want to avoid the creation of a ByteBuffer, and only
+   * want the byte[], e.g.:
+   *
+   * <pre>
+   * ByteBuffer buffer = decoder.readBytes(null);
+   * byte[] array = buffer.array();
+   * </pre>
+   *
+   * @throws AvroTypeException If this is a stateful reader and byte-string is not
+   *                           the type of the next value to be read
+   */
+  public abstract byte[] readBytes() throws IOException;


I am not aware of the expectations on up/downgrade, and if this is appropriate. Happy to make a change if i is the right thing to do, but it looks to me as if if that is the case the default implementation should be removed on the next opportunity for a braking change, be that major or minor version, based on the policy of the avro project

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

AVRO-4258 remove some allocations from decoder.getBytes() and Utf8 ha…

059a5b0

…ndling

github-actions Bot added the Java Pull Requests for Java binding label May 12, 2026

mkeskells mentioned this pull request May 12, 2026

AVRO-4249: [java] provide a cache of schema to avoid building #3746

Open

opwvhk approved these changes Jun 1, 2026

View reviewed changes

AVRO-4258: Run spotless:apply

ed5e115

mkeskells and others added 6 commits June 1, 2026 12:00

Merge branch 'main' into AVRO-4258-getBytes

9fa67ec

AVRO-4258: Cleanup code

2d638f7

AVRO-4258: Set length with byte array

e5ef7b2

AVRO-4258: Push readBytes() implementation down

bf69352

Now colocated with `readBytes(ByteBuffer)` and `skipBytes()` implementations.

AVRO-4258

2b76d23

move the implementation to child classes reset utf8 string

Fokko approved these changes Jun 2, 2026

View reviewed changes

iemejia requested a review from Copilot June 20, 2026 14:27

Copilot started reviewing on behalf of iemejia June 20, 2026 14:27 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

unused import

40c0a54

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AVRO-4258: [java] remove some allocations from decoder.getBytes() and Utf8 handling #3776

AVRO-4258: [java] remove some allocations from decoder.getBytes() and Utf8 handling #3776
mkeskells wants to merge 9 commits into
apache:mainfrom
mkeskells:AVRO-4258-getBytes

mkeskells commented May 12, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

opwvhk commented Jun 1, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

Fokko left a comment

Uh oh!

mkeskells commented Jun 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

mkeskells Jun 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mkeskells commented May 12, 2026

Contribution Checklist

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

opwvhk commented Jun 1, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

mkeskells commented Jun 1, 2026

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

mkeskells commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mkeskells Jun 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mkeskells commented Jun 8, 2026 •

edited

Loading