fix: use Go-compatible JSON encoding for OCI manifests by sajayantony · Pull Request #348 · oras-project/oras-dotnet

sajayantony · 2026-02-23T23:05:46Z

Fix: `+` escaped as `\u002B` in JSON serialization

Fixes #346

Problem

System.Text.Json's default JavaScriptEncoder escapes + as \u002B. OCI media types like application/vnd.oci.image.manifest.v1+json become ...v1\u002Bjson, producing different digests than oras-go and breaking CAS interoperability.

Root cause: JavaScriptEncoder.Default hardcodes + on its block list. AllowCharacter('+') doesn't work. UnsafeRelaxedJsonEscaping fixes + but stops escaping <>& — unsafe.

Solution

Custom JsonConverter classes with Utf8JsonWriter.WriteRawValue() and a Go-compatible escape function matching encoding/json.Marshal semantics.

Design

Call Sites
  Packer · ManifestStore · Repository · Registry
  FetchableExtensions · ResponseException · Error
        |
        v
  OciJsonSerializer (internal)
    SerializeToUtf8Bytes()  <- size guard (4 MiB)
    Deserialize(byte[])
    Deserialize(string)
    DeserializeAsync(Stream)
    FormatErrorDetail()
    EscapeJsonString()      <- Go compat escaping
        |
        | registers
    +---+---+
    v       v
  OciStringConverter    OciDictionaryConverter
  (string values)       (annotation dict keys+values)
                        (UTF-8 bytewise key sort)

Why two converters? JsonConverter<string> is NOT invoked for dictionary keys — only values. OciDictionaryConverter handles annotation dictionaries where both keys and values may contain +.

Escaping Rules (Go `encoding/json.Marshal` compatible)

Input	Output	Input	Output
`"`	`\"`	`\`	`\\`
`\b \f \n \r \t`	`\b \f \n \r \t`	`< > &`	`\u003c \u003e \u0026`
U+2028, U+2029	`\u2028 \u2029`	0x00–0x1F (other)	`\uXXXX`
`+`	`+` (literal — the fix)	all other	literal

Changes

New files (all internal):

Serialization/OciJsonSerializer.cs — single entry point for all JSON ops
Serialization/OciStringConverter.cs — string value converter
Serialization/OciDictionaryConverter.cs — annotation dictionary converter (UTF-8 bytewise key sort for Go compatibility)
Content/OciLimits.cs — shared MaxManifestBytes constant (4 MiB)

Modified call sites:

Packer.cs, ManifestStore.cs, Repository.cs, Registry.cs, FetchableExtensions.cs, ResponseException.cs, Error.cs — all JSON through OciJsonSerializer
Index.cs — GenerateIndex() now delegates to OciJsonSerializer.SerializeToUtf8Bytes
CopyGraphOptions.cs, RepositoryOptions.cs — use shared OciLimits.MaxManifestBytes

Tests (17 new test methods, 438 total passing):

OciStringConverterTest.cs — escape branch coverage + round-trip
OciDictionaryConverterTest.cs — keys, values, empty, null, invalid types, round-trip, UTF-8 key sort order
ManifestSerializationTest.cs — JSON fixture deserialization, annotation fidelity, index, round-trip, size limit
PackerTest.cs — 2 regression tests checking raw bytes for literal +

Key Design Decisions

Layered architecture: All JSON serialization AND deserialization goes through OciJsonSerializer
Domain types clean: Index.GenerateIndex() stays on Index but delegates to OciJsonSerializer for encoding
Internal only: Zero public API surface added
Size guard: 4 MiB limit on serialized output (matches oras-go)
Shared constant: OciLimits.MaxManifestBytes replaces duplicate private consts
UTF-8 key sort: OciDictionaryConverter sorts annotation keys by UTF-8 byte sequence (not UTF-16 ordinal) to match Go's encoding/json.Marshal behavior

Copilot

Pull request overview

This PR fixes a critical JSON serialization issue where System.Text.Json's default encoder escapes + as \u002B, breaking content-addressable storage (CAS) interoperability with oras-go and other OCI tools. The solution introduces a custom JSON serialization layer with Go-compatible escaping rules.

Changes:

Introduced internal OciJsonSerializer with custom converters (OciStringConverter, OciDictionaryConverter) that bypass .NET's default encoder to preserve literal + characters while maintaining proper escaping for quotes, backslashes, control characters, and HTML-sensitive characters (<, >, &)
Centralized all OCI content JSON operations through OciJsonSerializer across 7 source files
Moved GenerateIndex from Index class to OciJsonSerializer for architectural consistency
Consolidated duplicate 4 MiB size limit constants into shared OciLimits.MaxManifestBytes
Updated Copilot instructions with serialization conventions and testing best practices learned from this fix

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`OciJsonSerializer.cs`	Central serialization entry point with Go-compatible `EscapeJsonString` method and 4 MiB size guard
`OciStringConverter.cs`	Custom `JsonConverter<string>` using `WriteRawValue` to bypass default encoder
`OciDictionaryConverter.cs`	Custom dictionary converter handling both keys and values (framework doesn't invoke string converter for dict keys)
`OciLimits.cs`	Shared 4 MiB manifest size constant
`Packer.cs`	Updated to use `OciJsonSerializer.SerializeToUtf8Bytes`
`ManifestStore.cs`	Updated to use `OciJsonSerializer.DeserializeAsync` and `GenerateIndex`
`Repository.cs`	Updated to use `OciJsonSerializer.Deserialize` for OCI content
`Registry.cs`	Updated to use `OciJsonSerializer.Deserialize`
`FetchableExtensions.cs`	Updated to use `OciJsonSerializer.Deserialize`
`ResponseException.cs`	Updated to use `OciJsonSerializer.Deserialize` for error responses
`Error.cs`	Updated to use `OciJsonSerializer.FormatErrorDetail`
`Index.cs`	Removed `GenerateIndex` method (moved to serializer)
`CopyGraphOptions.cs`	Uses `OciLimits.MaxManifestBytes` instead of duplicate constant
`RepositoryOptions.cs`	Uses `OciLimits.MaxManifestBytes` instead of duplicate constant
`OciStringConverterTest.cs`	Tests escaping rules, round-trip, verifies literal `+` not `\u002B`
`OciDictionaryConverterTest.cs`	Tests key/value escaping, null handling, invalid types, round-trip
`ManifestSerializationTest.cs`	Wire-format JSON fixtures, annotation fidelity, size limit enforcement
`PackerTest.cs`	Two new regression tests inspecting raw bytes for literal `+`
`.github/copilot-instructions.md`	Trimmed 28%, added serialization/testing conventions

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs

src/OrasProject.Oras/Serialization/OciDictionaryConverter.cs

.github/copilot-instructions.md

tests/OrasProject.Oras.Tests/Serialization/OciDictionaryConverterTest.cs

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

codecov · 2026-02-24T00:23:38Z

Codecov Report

❌ Patch coverage is 94.87179% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.09%. Comparing base (386efee) to head (65f18ef).

Files with missing lines	Patch %	Lines
...rc/OrasProject.Oras/Content/FetchableExtensions.cs	0.00%	0 Missing and 2 partials ⚠️
.../OrasProject.Oras/Registry/Remote/ManifestStore.cs	85.71%	0 Missing and 2 partials ⚠️
src/OrasProject.Oras/Registry/Remote/Repository.cs	80.00%	0 Missing and 2 partials ⚠️
...oject.Oras/Serialization/OciDictionaryConverter.cs	95.65%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #348      +/-   ##
==========================================
+ Coverage   91.78%   92.09%   +0.31%     
==========================================
  Files          64       67       +3     
  Lines        2752     2886     +134     
  Branches      365      374       +9     
==========================================
+ Hits         2526     2658     +132     
- Misses        138      139       +1     
- Partials       88       89       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

src/OrasProject.Oras/Registry/Remote/Registry.cs

src/OrasProject.Oras/Serialization/OciDictionaryConverter.cs

tests/OrasProject.Oras.Tests/Serialization/OciStringConverterTest.cs

src/OrasProject.Oras/Registry/Remote/Exceptions/ResponseException.cs

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs

sajayantony · 2026-02-26T07:17:16Z

Resolved by Copilot: Added section comments throughout the EscapeJsonString switch: JSON structural characters, Named control character escapes, HTML-sensitive characters (Go escapes these), Unicode line/paragraph separators (Go escapes these), and a default-case comment explaining it escapes remaining control chars (U+0000-U+001F) as \uXXXX while passing all other characters through literally, including +.

sajayantony · 2026-02-26T07:17:27Z

Resolved by Copilot: Good idea. The serialization path is only used for manifests/indexes (bounded at 4 MiB by OciLimits.MaxManifestBytes), so the overhead relative to network I/O should be negligible. That said, adding a benchmark suite would be valuable for tracking regressions — filed as a follow-up concern.

sajayantony · 2026-02-26T07:17:40Z

Resolved by Copilot: Added a comment explaining the relationship — _maxMetadataBytes defaults to OciLimits.MaxManifestBytes because manifests are the primary metadata cached during copy operations. The constant was extracted to OciLimits so both the serializer size check and this default share the same source of truth.

sajayantony · 2026-02-26T07:17:49Z

Resolved by Copilot: Named the tuple elements — (Descriptor Descriptor, byte[] Content). Also added a tuple naming convention to copilot-instructions.md.

akashsinghal

Caution

This review was generated by Copilot using Claude Opus 4.6, mimicking the review style of shizhMSFT. This is not the real shizhMSFT.

Solid fix for a real CAS interoperability problem. The escaping logic matches Go's encoding/json.Marshal semantics correctly, and the converter architecture (separate string + dictionary converters) is the right approach given System.Text.Json's limitation that JsonConverter<string> doesn't intercept dictionary keys. A few observations below.

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs

tests/OrasProject.Oras.Tests/Serialization/ManifestSerializationTest.cs

.github/copilot-instructions.md

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs

- Revert copilot-instructions.md to upstream (split to issue oras-project#356) - Move GenerateIndex back to Index.cs, call OciJsonSerializer internally - Add byte-level idempotency assertion in round-trip tests - Document size-guard materialization behavior in OciJsonSerializer Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Sajay Antony <sajaya@microsoft.com>

sajayantony · 2026-03-05T07:21:56Z

Addressed all 4 unresolved review threads in commit 6eacd7d:

GenerateIndex in serializer — Moved back to Index.cs. Calls OciJsonSerializer.SerializeToUtf8Bytes internally so the serializer stays focused on encoding only.
Byte-level idempotency test — Round-trip test now asserts serialize-deserialize-serialize produces byte-identical output for CAS compatibility.
copilot-instructions.md rewrite — Reverted to upstream. Filed docs: trim and update .github/copilot-instructions.md #356 to track the rewrite as a separate PR.
Size-guard materialization — Added remarks documenting that SerializeToUtf8Bytes fully materializes before the size check.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

You can also share your feedback on Copilot code review. Take the survey.

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs

Copilot · 2026-03-05T07:23:40Z

src/OrasProject.Oras/Oci/Index.cs

+    internal static (Descriptor Descriptor, byte[] Content) GenerateIndex(
+        IList<Descriptor> manifests)
    {
        var index = new Index(manifests);
-        var indexContent = JsonSerializer.SerializeToUtf8Bytes(index);
-        return (Descriptor.Create(indexContent, Oci.MediaType.ImageIndex), indexContent);
+        var indexContent =
+            OciJsonSerializer.SerializeToUtf8Bytes(index);
+        return (
+            Descriptor.Create(indexContent, Oci.MediaType.ImageIndex),
+            indexContent);


PR description mentions Index.GenerateIndex() being removed/moved into the serializer, but this method still exists (now just using OciJsonSerializer). Either update the PR description to match the implementation, or remove/move the method if the intent is to centralize index generation in OciJsonSerializer.

@copilot apply changes based on this feedback

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-05T07:28:36Z

src/OrasProject.Oras/Serialization/OciDictionaryConverter.cs

+        var first = true;
+        foreach (var kvp in value.OrderBy(k => k.Key, StringComparer.Ordinal))
+        {


OciDictionaryConverter.Write sorts keys with StringComparer.Ordinal, which compares UTF-16 code units. Go’s encoding/json sorts map keys by Go string order (bytewise over UTF-8), which can differ from UTF-16 ordinal ordering for non-BMP characters (surrogate pairs) and some BMP ranges, potentially reintroducing cross-runtime digest mismatches for non-ASCII annotation keys. Consider sorting by UTF-8 byte sequence (e.g., Encoding.UTF8.GetBytes(key) with lexicographic compare) to match Go’s ordering semantics.

@copilot apply changes based on this feedback

ridhoq

LGTM

Custom JsonConverter classes with Utf8JsonWriter.WriteRawValue() and a Go-compatible escape function matching encoding/json.Marshal semantics. - OciJsonSerializer: single entry point for all JSON serialization - OciStringConverter: Go-compatible string escaping (literal '+') - OciDictionaryConverter: annotation dict with UTF-8 bytewise key sort - OciLimits: shared MaxManifestBytes constant (4 MiB) - All call sites routed through OciJsonSerializer Fixes oras-project#346 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Sajay Antony <sajaya@microsoft.com>

sajayantony requested a review from shizhMSFT as a code owner February 23, 2026 23:05

Copilot AI review requested due to automatic review settings February 23, 2026 23:05

sajayantony requested a review from Wwwsylvia as a code owner February 23, 2026 23:05

Copilot started reviewing on behalf of sajayantony February 23, 2026 23:06 View session

sajayantony marked this pull request as draft February 23, 2026 23:09

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from 52000ac to 017a00b Compare February 23, 2026 23:09

Copilot AI reviewed Feb 23, 2026

View reviewed changes

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from 017a00b to 3f67c33 Compare February 23, 2026 23:12

sajayantony marked this pull request as ready for review February 23, 2026 23:13

Copilot AI review requested due to automatic review settings February 23, 2026 23:13

Copilot started reviewing on behalf of sajayantony February 23, 2026 23:13 View session

Copilot AI reviewed Feb 23, 2026

View reviewed changes

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from 3f67c33 to 57092d1 Compare February 23, 2026 23:28

Copilot AI review requested due to automatic review settings February 23, 2026 23:29

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from 57092d1 to 0489b9a Compare February 23, 2026 23:29

Copilot started reviewing on behalf of sajayantony February 23, 2026 23:30 View session

Copilot AI reviewed Feb 23, 2026

View reviewed changes

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from 0489b9a to 14934ba Compare February 23, 2026 23:35

Copilot AI review requested due to automatic review settings February 24, 2026 01:21

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from 14934ba to af9d966 Compare February 24, 2026 01:21

Copilot started reviewing on behalf of sajayantony February 24, 2026 01:21 View session

Copilot AI reviewed Feb 24, 2026

View reviewed changes

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from af9d966 to cd1e45b Compare February 24, 2026 01:49

Copilot AI review requested due to automatic review settings February 24, 2026 02:00

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from cd1e45b to f16fe71 Compare February 24, 2026 02:00

Copilot started reviewing on behalf of sajayantony February 24, 2026 02:01 View session

Copilot AI reviewed Feb 24, 2026

View reviewed changes

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs Show resolved Hide resolved

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from f16fe71 to ba708a3 Compare February 24, 2026 02:39

Copilot AI review requested due to automatic review settings February 24, 2026 02:46

ridhoq reviewed Feb 24, 2026

View reviewed changes

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs Show resolved Hide resolved

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs Show resolved Hide resolved

sajayantony force-pushed the agent/fix-plus-escaping-346 branch 2 times, most recently from 1d6f5f4 to d7a9498 Compare February 26, 2026 07:25

akashsinghal reviewed Mar 3, 2026

View reviewed changes

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs Outdated Show resolved Hide resolved

akashsinghal reviewed Mar 3, 2026

View reviewed changes

tests/OrasProject.Oras.Tests/Serialization/ManifestSerializationTest.cs Show resolved Hide resolved

akashsinghal reviewed Mar 3, 2026

View reviewed changes

.github/copilot-instructions.md Show resolved Hide resolved

akashsinghal reviewed Mar 3, 2026

View reviewed changes

src/OrasProject.Oras/Serialization/OciJsonSerializer.cs Show resolved Hide resolved

sajayantony mentioned this pull request Mar 4, 2026

docs: trim and update .github/copilot-instructions.md #356

Open

Copilot AI review requested due to automatic review settings March 5, 2026 07:19

Copilot started reviewing on behalf of sajayantony March 5, 2026 07:19 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings March 5, 2026 07:24

Copilot started reviewing on behalf of sajayantony March 5, 2026 07:24 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

ridhoq approved these changes Mar 6, 2026

View reviewed changes

ridhoq mentioned this pull request Mar 6, 2026

[BUG] OciImageManifest serialization escapes + as \u002B in media types Azure/azure-sdk-for-net#56319

Open

sajayantony force-pushed the agent/fix-plus-escaping-346 branch 2 times, most recently from 7df89de to acfef7c Compare March 30, 2026 22:25

sajayantony force-pushed the agent/fix-plus-escaping-346 branch from acfef7c to 65f18ef Compare March 30, 2026 22:27

Conversation

sajayantony commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fix: + escaped as \u002B in JSON serialization

Problem

Solution

Design

Escaping Rules (Go encoding/json.Marshal compatible)

Changes

Key Design Decisions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

codecov bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sajayantony commented Feb 26, 2026

Uh oh!

sajayantony commented Feb 26, 2026

Uh oh!

sajayantony commented Feb 26, 2026

Uh oh!

sajayantony commented Feb 26, 2026

Uh oh!

akashsinghal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sajayantony commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

sajayantony Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

sajayantony commented Feb 23, 2026 •

edited

Loading

Fix: `+` escaped as `\u002B` in JSON serialization

Escaping Rules (Go `encoding/json.Marshal` compatible)

codecov bot commented Feb 24, 2026 •

edited

Loading