Skip to content

feat(sdk): TinyGo WASM canary builds and spike plan#3063

Draft
pflynn-virtru wants to merge 37 commits intomainfrom
feat/wasm-tinygo-canary
Draft

feat(sdk): TinyGo WASM canary builds and spike plan#3063
pflynn-virtru wants to merge 37 commits intomainfrom
feat/wasm-tinygo-canary

Conversation

@pflynn-virtru
Copy link
Member

@pflynn-virtru pflynn-virtru commented Feb 9, 2026

Summary

  • Add TinyGo WASM canary programs under sdk/experimental/tdf/wasm/ that exercise the stdlib and third-party packages needed for a hybrid WASM TDF core engine
  • Migrate sdk/experimental/tdf/ manifest, assertion, and key-access structs from encoding/json to tinyjson codegen for TinyGo WASM compatibility — includes generated marshal/unmarshal code and updated tests
  • Add tinyjson and zipstream canaries that validate these migrations compile and round-trip correctly under TinyGo
  • Add informational CI workflow (.github/workflows/tinygo-wasm-canary.yaml) that compiles each canary with TinyGo targeting wasip1 — expected to have failures until spike work is complete
  • Add spike plan ADR (docs/adr/spike-wasm-core-tinygo-hybrid.md) documenting the TinyGo hybrid architecture, host crypto ABI (8 functions), go/no-go criteria, and task breakdown
  • Add TDF decrypt benchmarks (sdk/benchmark_test.go) measuring pure decrypt throughput at 1MB–2GB using direct key injection (no KAS dependency)
  • Add experimental TDF Writer encrypt benchmarks (sdk/experimental/tdf/benchmark_test.go) measuring streaming encrypt throughput, single-segment performance, and full TDF assembly
  • Add hostcrypto package (sdk/experimental/tdf/wasm/hostcrypto/) with typed Go wrappers for all go:wasmimport host functions — the WASM-side ABI layer for crypto and I/O
  • Add Wazero host module (sdk/experimental/tdf/wasm/host/) implementing the host side of the crypto/IO ABI — registers "crypto" and "io" modules with Wazero, delegating all crypto to lib/ocrypto. Includes 20 tests covering ABI conformance, round-trip correctness, error paths, and OOB handling.
  • Implement single-segment TDF3 encrypt in WASM — the first real TDF logic running inside the WASM sandbox. All crypto delegated to host via hostcrypto; manifest construction, policy binding (HS256), integrity computation, and ZIP assembly run inside WASM using tinyjson types and zipstream.
  • Implement single- and multi-segment TDF3 decrypt in WASM — ZIP parsing, manifest validation, segment integrity verification, root signature check, and AES-GCM decryption all run inside the WASM sandbox with crypto delegated to host.
  • Fix WASM decrypt OOM at large payloads — eliminate ~3× payload memory overhead by replacing ptrToBytes copies with zero-copy ptrToSlice, writing decrypted segments directly into the host-provided output buffer via AesGcmDecryptInto, and removing intermediate accumulation. Peak WASM Go-heap usage during decrypt drops from ~300 MB to ~0 for a 100 MB TDF.
  • Add 11 end-to-end encrypt integration tests — compile WASM module, load in wazero, exercise tdf_encrypt: round-trip decrypt, manifest field validation, policy binding, segment/root integrity, UUID format, attributes, empty plaintext, deterministic sizes, and error paths. Includes wazero proc_exit fix for Go 1.25 wasip1 and GC-safety fixes for WASM malloc.
  • Add cross-SDK benchmark (examples/cmd/benchmark_cross_sdk.go) comparing Production SDK, Experimental Writer, and WASM across payload sizes from 256 B to 100 MB.

Canary programs

Canary Packages tested Expected
base64hex encoding/base64, encoding/hex pass
zipwrite encoding/binary, hash/crc32, bytes, sort, sync pass
tinyjson tinyjson codegen manifest + assertion round-trip pass
zipstream production zipstream writer: TDF ZIP creation + CRC32 combine + ZIP64 pass
iocontext io, context, strings, strconv, fmt, errors fail
stdjson encoding/json with TDF manifest structs (superseded by tinyjson) fail
wasm full WASM module — go:wasmimport host ABI + tdf package fail

Host crypto ABI

Host function Wazero implementation Tests
random_bytes ocrypto.RandomBytes(n) happy path, OOB write
aes_gcm_encrypt ocrypto.NewAESGcm(key).Encrypt(pt) round-trip, empty key
aes_gcm_decrypt ocrypto.NewAESGcm(key).Decrypt(ct) round-trip, empty key, empty ct, corrupted ct
hmac_sha256 ocrypto.CalculateSHA256Hmac(key, data) vs direct ocrypto, OOB output
rsa_oaep_sha1_encrypt ocrypto.NewAsymEncryption(pem).Encrypt(pt) round-trip, wrong key
rsa_oaep_sha1_decrypt ocrypto.NewAsymDecryption(pem).Decrypt(ct) round-trip, wrong key
rsa_generate_keypair ocrypto.NewRSAKeyPair(bits) PEM validation + round-trip
get_last_error thread-safe string (set/get/clear) retrieval, clear-after-read, truncation, no-error
read_input cfg.Input.Read(buf) data flow, EOF, nil reader
write_output cfg.Output.Write(buf) data flow, nil writer

WASM TDF encrypt (Task 3.1)

Single-segment TDF3 encrypt running entirely inside the WASM sandbox:

Step Operation Implementation
1 DEK generation hostcrypto.RandomBytes(32)
2 Key wrapping hostcrypto.RsaOaepSha1Encrypt(kasPub, dek)
3 Policy construction tinyjson types + MarshalJSON()
4 Policy binding HMAC-SHA256(dek, base64Policy) → hex → base64
5 AES-GCM encrypt hostcrypto.AesGcmEncrypt(dek, plaintext)
6 Segment integrity HMAC-SHA256(dek, cipher) → base64
7 Root signature HMAC-SHA256(dek, segmentSig) → base64
8 ZIP assembly zipstream WriteSegment + Finalize

Encrypt integration tests (11 tests)

Test Validates
TestTDFEncryptRoundTrip Full encrypt → unwrap DEK → AES-GCM decrypt → plaintext matches
TestTDFEncryptManifestFields All manifest fields (version, method, key access, payload, segments)
TestTDFEncryptPolicyBinding HMAC-SHA256 double-encoding matches recomputed binding
TestTDFEncryptSegmentIntegrity Segment hash + root signature match recomputed HMAC values
TestTDFEncryptPolicyUUID UUID v4 format validation
TestTDFEncryptWithAttributes Attributes round-trip through policy JSON
TestTDFEncryptEmptyPlaintext Empty input → 28-byte payload (nonce + tag only)
TestTDFEncryptDeterministicSizes Consistent sizes across encryptions, different ciphertext
TestTDFEncryptErrorInvalidKey Invalid PEM → error message
TestTDFEncryptErrorBufferTooSmall Undersized output buffer → "buffer too small" error
TestTDFEncryptGetErrorClearsAfterRead Error cleared after get_error retrieval

Cross-SDK Benchmarks

Measured on a single dev machine (Apple Silicon M3 Max); numbers are indicative, not normative. 5 iterations per size.

Encrypt

Payload Production SDK Exp. Writer WASM
256 B 2.8 ms 0.1 ms 0.3 ms
1 KB 0.3 ms 0.1 ms 0.2 ms
16 KB 0.1 ms 0.1 ms 0.2 ms
64 KB 0.3 ms 0.1 ms 0.9 ms
256 KB 0.2 ms 0.4 ms 1.6 ms
1 MB 0.6 ms 1.0 ms 5.1 ms
10 MB 5.4 ms 2.3 ms 42.9 ms
100 MB 52.5 ms 17.2 ms 424.7 ms

Decrypt

Payload Production SDK* WASM**
256 B 27.9 ms 1.2 ms
1 KB 20.2 ms 1.2 ms
16 KB 19.3 ms 1.3 ms
64 KB 19.8 ms 1.4 ms
256 KB 20.7 ms 1.4 ms
1 MB 18.3 ms 3.9 ms
10 MB 21.9 ms 17.4 ms
100 MB 44.7 ms 328.5 ms

*Production SDK: includes KAS rewrap network latency (~20 ms round-trip to localhost)
**WASM: includes local RSA-OAEP DEK unwrap (no network); in production the host would call KAS for rewrap

WASM decrypt memory optimization

Allocation Before After
TDF input copy (ptrToBytes) payload size 0 (zero-copy ptrToSlice)
AES-GCM decrypt intermediate segment size 0 (AesGcmDecryptInto writes to output)
Accumulated plaintext slice payload size 0 (direct write to output buffer)
Output copy in tdfDecrypt payload size 0 (already in output buffer)
Peak Go heap ~3× payload ~0

Microbenchmarks

Benchmark Path Throughput
BenchmarkDecrypt/1MB–2GB sdk/benchmark_test.go ~1.4–1.6 GB/s (WriteTo)
BenchmarkStreamDecrypt (100MB, 32KB reads) sdk/benchmark_test.go ~23 MB/s (ReadAt overhead)
BenchmarkWriterEncrypt/1MB–1GB sdk/experimental/tdf/benchmark_test.go ~1.1–1.2 GB/s
BenchmarkWriterWriteSegment (2MB) sdk/experimental/tdf/benchmark_test.go ~1.25 GB/s
BenchmarkWriterAssemble/1MB–100MB sdk/experimental/tdf/benchmark_test.go ~1.0–1.1 GB/s

CI workflow

  • Informational only — not in the ci aggregation job, does not block PRs
  • Matrix strategy with fail-fast: false — all canaries run independently
  • Reports binary size and build/run status in job summary
  • Triggers on changes to sdk/experimental/tdf/**, sdk/internal/zipstream/**, sdk/manifest.go, lib/ocrypto/**

Test plan

  • CI workflow runs successfully (individual canaries may fail as expected)
  • base64hex, zipwrite, tinyjson, and zipstream canaries pass TinyGo compilation and execution
  • All canaries compile with standard Go (go build ./sdk/experimental/tdf/wasm/...)
  • Workflow does not appear in ci aggregation job needs list
  • Decrypt benchmarks pass: cd sdk && go test -bench=BenchmarkDecrypt -short -run=^$ .
  • Experimental writer benchmarks pass: cd sdk/experimental/tdf && go test -bench=Benchmark -short -run=^$ .
  • Wazero host module tests pass: go test -v ./sdk/experimental/tdf/wasm/host/ (44 host ABI + integration tests)
  • WASM encrypt integration tests pass: go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFEncrypt (11 end-to-end tests)
  • WASM decrypt integration tests pass: go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFDecrypt (13 end-to-end tests)
  • Cross-SDK benchmark runs without OOM at 100 MB: cd examples && go run . benchmark-cross-sdk -e ... --sizes "256,1024,16384,65536,262144,1048576,10485760,104857600"
  • Full SDK builds: go build ./sdk/...
  • WASM module compiles: GOOS=wasip1 GOARCH=wasm go build ./sdk/experimental/tdf/wasm/

Related: SDK-WASM-1 (Jira)

🤖 Generated with Claude Code

pflynn-virtru and others added 2 commits February 9, 2026 11:35
ADR documenting the TinyGo hybrid WASM architecture spike (SDK-WASM-1),
including host crypto ABI, go/no-go criteria, and task breakdown.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paul Flynn <pflynn@virtru.com>
@github-actions github-actions bot added comp:ci Github Actions Work comp:sdk A software development kit, including library, for client applications and inter-service communicati docs Documentation size/l labels Feb 9, 2026
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @pflynn-virtru, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request lays the groundwork for a significant architectural shift by exploring the feasibility of a TinyGo-compiled WebAssembly (WASM) core engine for TDF operations. It introduces a set of targeted canary programs to assess TinyGo's compatibility with crucial Go standard library components and establishes a detailed spike plan. The primary aim is to validate a hybrid approach where cryptographic primitives are delegated to the host environment, while core TDF logic resides within a compact and portable WASM module, paving the way for enhanced flexibility and deployment options.

Highlights

  • TinyGo WASM Canary Programs: Introduced several TinyGo WASM canary programs under 'sdk/experimental/tdf/wasm/' to test the compatibility and functionality of various standard library packages (e.g., encoding/base64, encoding/hex, io, context, encoding/json, hash/crc32) essential for building a hybrid WASM TDF core engine.
  • Informational CI Workflow: Implemented a new informational CI workflow ('.github/workflows/tinygo-wasm-canary.yaml') that compiles and runs these TinyGo canaries targeting 'wasip1'. This workflow is non-blocking and provides early feedback on TinyGo's behavior with specific TDF-related code patterns, with some expected failures for certain canaries.
  • Spike Plan Architectural Decision Record (ADR): Added a comprehensive Architectural Decision Record ('docs/adr/spike-wasm-core-tinygo-hybrid.md') detailing a 2-week spike plan. This ADR outlines the objective, go/no-go criteria, proposed hybrid architecture, host crypto ABI (8 functions), task breakdown, measurement strategies, and identified technical risks for validating a TinyGo-compiled WASM module for TDF3 single-segment encrypt/decrypt.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • docs/adr/spike-wasm-core-tinygo-hybrid.md
    • Added a new Architectural Decision Record (ADR) titled 'SDK-WASM-1: Spike — TinyGo Hybrid WASM Core Engine'.
    • Defined the objective to validate TinyGo-compiled WASM for TDF3 single-segment encrypt/decrypt with host-delegated crypto.
    • Established five clear Go/No-Go criteria for the spike, covering compilation, host function interoperability, binary size, TDF validity, and performance.
    • Detailed the proposed hybrid architecture, illustrating the separation of WASM-internal logic (e.g., manifest handling, ZIP writing) from 8 host-imported crypto functions (e.g., AES-GCM, RSA-OAEP, HMAC-SHA256).
    • Specified the Host Function ABI conventions for data exchange (shared linear memory), parameter types (ptr, len pairs), error reporting, and output sizing.
    • Outlined a 4-phase task breakdown for the 10-day spike, including scaffolding, adapting manifest structs and zipstream, implementing the Wazero host crypto interface, and developing TDF encrypt with round-trip validation.
    • Identified explicitly out-of-scope items for the spike, such as NanoTDF, EC key wrapping, and in-WASM decrypt.
    • Listed dependencies and prerequisites, including TinyGo, Wazero, tinyjson codegen, and wasm-opt.
    • Analyzed key technical risks (e.g., tinyjson compatibility, encoding/binary issues, shared memory pointer fragility) and proposed mitigation strategies.
    • Provided an estimated effort breakdown for each phase and a vision for future ABI evolution to support EC-wrapped TDFs.
  • sdk/experimental/tdf/wasm/base64hex/main.go
    • Added a new TinyGo canary program to verify the correct round-trip encoding and decoding functionality of 'encoding/base64' and 'encoding/hex', which are crucial for TDF policy and assertion hash encoding.
  • sdk/experimental/tdf/wasm/iocontext/main.go
    • Added a new TinyGo canary program to test the compatibility of various standard library packages under TinyGo, including 'io' (Reader, Writer, ReadFull, MultiReader), 'context' (cancellation), 'strings' (Contains, ToLower, Split), 'strconv' (Itoa, Atoi), 'fmt' (Sprintf), and 'errors' (Errorf, Is), all of which are extensively used in TDF logic.
  • sdk/experimental/tdf/wasm/stdjson/main.go
    • Added a new TinyGo canary program that attempts to marshal and unmarshal TDF manifest structs using the standard 'encoding/json' package. This canary is explicitly expected to fail under TinyGo due to its known limitations with 'reflect' features, serving as a baseline for the planned 'tinyjson' integration.
  • sdk/experimental/tdf/wasm/wasmimport/main.go
    • Added a new TinyGo canary program to validate the compilation of 'go:wasmimport' directives for 8 crypto host functions and 'go:wasmexport' directives for 'malloc', 'free', and 'tdf_encrypt'. This program demonstrates the core pattern for the hybrid WASM architecture, ensuring TinyGo can correctly handle host-guest function interfaces.
  • sdk/experimental/tdf/wasm/zipwrite/main.go
    • Added a new TinyGo canary program to test the functionality of 'encoding/binary' (Write, Read with LittleEndian), 'hash/crc32' (ChecksumIEEE, incremental CRC), 'sort.Ints', 'sync.Mutex', and 'bytes.Buffer' composition. These operations are fundamental to the 'zipstream' package, which is responsible for writing TDF ZIP archives.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/tinygo-wasm-canary.yaml
Activity
  • The pull request was opened to introduce the initial set of TinyGo WASM canary builds and a detailed spike plan.
  • An informational CI workflow ('tinygo-wasm-canary.yaml') has been configured to automatically compile and execute these canary programs, providing early feedback on TinyGo compatibility without blocking the PR.
  • Canary programs for 'base64hex' and 'zipwrite' are expected to compile and pass, while 'iocontext' and 'stdjson' are anticipated to fail, as outlined in the PR description, to gather critical data for the spike.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.


Go code, now WASM bound, TinyGo makes it small and fast, New frontiers are found.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive spike plan (ADR) for a TinyGo-based WASM core engine and adds several canary programs to test the compatibility of necessary Go standard library packages with TinyGo. The ADR is well-structured and detailed, covering architecture, ABI, risks, and a task breakdown. The canary programs are good initial steps to de-risk the effort.

My review focuses on improving the clarity of the ADR and addressing a potential memory safety issue in the wasmimport canary. I've suggested a clarification in the ADR's objective to align it better with the documented scope. I've also recommended adding a crucial comment to the wasmMalloc implementation to highlight its dependency on the -gc=leaking build flag for memory safety, which is critical for future maintenance.

Comment on lines +44 to +45
buf := make([]byte, size)
return uint32(uintptr(unsafe.Pointer(&buf[0])))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current implementation of wasmMalloc has a potential memory safety issue. The buf slice is allocated within the function, and returning a pointer to its underlying data can lead to a dangling pointer if the garbage collector reclaims buf after the function returns. While this is mitigated by compiling with the -gc=leaking flag as mentioned in the ADR, this critical dependency is not apparent from the code itself. For better maintainability and to prevent accidental misuse, I recommend adding a comment explaining why this is safe under the specific build conditions and highlighting the intentional memory leak.

	buf := make([]byte, size)
	// NOTE: This leaks memory. This is only safe because the module is compiled
	// with the `-gc=leaking` flag, which prevents the garbage collector from
	// reclaiming the memory.
	return uint32(uintptr(unsafe.Pointer(&buf[0])))

## Objective

Validate that a TinyGo-compiled WASM module can perform TDF3 single-segment
encrypt/decrypt with all crypto delegated to host functions, producing output
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The objective states encrypt/decrypt, but the scope of the spike seems to be focused on implementing encryption within the WASM module and then validating the output using the existing Go SDK for decryption. The 'Explicitly Out of Scope' section also mentions 'Decrypt inside WASM' is for a future milestone (M2). To better reflect the spike's goal, consider clarifying that only encryption will be performed by the WASM module by removing /decrypt.

Suggested change
encrypt/decrypt with all crypto delegated to host functions, producing output
encrypt with all crypto delegated to host functions, producing output

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

X-Test Failure Report

…e logic

- Removed `wasmimport` module and redundant crypto directives.
- Consolidated `calculateSignature` logic into `writer.go` and removed it from `manifest.go`.
- Updated TinyGo canary workflow to reflect WASM module restructuring.
- name: "Report result"
if: always()
run: |
BUILD="${{ steps.build.outcome }}"

Check notice

Code scanning / zizmor

code injection via template expansion Note

code injection via template expansion
@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 158.295972ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 84.349628ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 370.740028ms
Throughput 269.73 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 38.084350774s
Average Latency 378.226999ms
Throughput 131.29 requests/second

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2026

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 184.733222ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 117.453566ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 369.118001ms
Throughput 270.92 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 40.022505123s
Average Latency 398.910648ms
Throughput 124.93 requests/second

pflynn-virtru and others added 2 commits February 12, 2026 14:43
Adds self-contained decrypt benchmarks that construct valid TDFs
programmatically and inject payload keys directly, bypassing KAS.
Covers 1MB–2GB WriteTo path and 100MB streaming Read() path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paul Flynn <pflynn@virtru.com>
Benchmarks for the experimental streaming TDF Writer covering:
- End-to-end encrypt (NewWriter + WriteSegment + Finalize)
- Single segment encrypt throughput (WriteSegment only)
- Full TDF assembly (segments + finalize bytes)
Sizes: 1MB, 100MB, 1GB (short-mode skippable).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Paul Flynn <pflynn@virtru.com>
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 208.100912ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 103.652178ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 377.780866ms
Throughput 264.70 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.015039822s
Average Latency 388.838384ms
Throughput 128.16 requests/second

…ibility

Replace reflection-based encoding/json with tinyjson (CosmWasm fork) codegen
for manifest, assertion, and policy structs in the TDF write path. This is a
prerequisite for the WASM core engine spike (SDK-WASM-1) since encoding/json
panics at runtime under TinyGo.

Changes:
- Add tinyjson codegen for manifest.go (13 types) and assertion_types.go (3 types)
- Change KeyAccess.PolicyBinding from interface{} to concrete PolicyBinding type
- Replace json.Marshal with .MarshalJSON() in writer.go, key_access.go, assertion.go
- Move Assertion/Statement/Binding structs to assertion_types.go for codegen
- Drop polymorphic Statement.Value UnmarshalJSON (reader concern, out of WASM scope)
- Add tinyjson TinyGo canary with manifest/policy/assertion round-trip validation
- Add wasm/Makefile with toolcheck, build, run, generate targets
- Add wasm/README.md with TinyGo/tinyjson/wasmtime install instructions

Canary results: tinyjson module compiles to 62KB raw / 29KB gzipped WASM,
all round-trip tests pass under wasmtime.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 147.955649ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 71.833559ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 364.34748ms
Throughput 274.46 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 38.763645413s
Average Latency 385.728778ms
Throughput 128.99 requests/second

…r WASM

Copy production zipstream writer code (5 files from sdk/internal/zipstream/)
into a standalone canary module and verify it compiles and runs correctly
under TinyGo WASM. This completes Phase 1 (Foundation) of the WASM core
engine spike (SDK-WASM-1).

The canary exercises:
- Single-segment TDF ZIP creation (header + manifest + central directory)
- Multi-segment out-of-order writing (3 segments in order 2, 0, 1)
- ZIP64 mode (ZIP64 EOCD + locator signatures)
- CRC32 combine (multi-part checksum matches direct computation)

Key finding: time.Time and time.Now() work correctly under TinyGo — the only
identified risk for zipstream compatibility.

Binary size: 113KB raw / 59KB gzipped (well under 300KB budget).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

X-Test Failure Report

opentdfplatform5SPVSE.dockerbuild
cukes-report
✅ js-v0.9.0

Use grouped redirects ({ cmd1; cmd2; } >> file) instead of individual
redirects to satisfy shellcheck/actionlint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
if: always()
run: |
BUILD="${{ steps.build.outcome }}"
RUN="${{ steps.run.outcome }}"

Check notice

Code scanning / zizmor

code injection via template expansion Note

code injection via template expansion
@github-actions
Copy link
Contributor

X-Test Failure Report

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 198.401685ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 100.205609ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 369.525036ms
Throughput 270.62 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 40.838386268s
Average Latency 407.154076ms
Throughput 122.43 requests/second

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 198.31669ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 98.752819ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 374.098817ms
Throughput 267.31 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 41.356875227s
Average Latency 412.300025ms
Throughput 120.90 requests/second

Document the three I/O models evaluated (WASM-drives, host-drives,
hybrid) and recommend hybrid streaming I/O for M2 with read_input
and write_output host imports. Update ABI evolution to 13 functions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…opies

Replace ptrToBytes with zero-copy ptrToSlice for TDF input and DEK,
write decrypted segments directly into the host-provided output buffer
via new AesGcmDecryptInto, removing ~3x payload memory overhead that
caused OOM at 100MB.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 205.737959ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 100.050583ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 376.974702ms
Throughput 265.27 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.392837671s
Average Latency 392.559685ms
Throughput 126.93 requests/second

pflynn-virtru and others added 6 commits February 19, 2026 15:43
Update benchmark results doc to include Java WASM (Chicory) and
TypeScript WASM (Node.js WebAssembly) columns, with instructions for
running WASM benchmarks across all three SDKs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Populate Java WASM (Chicory) and TypeScript WASM (Node.js) encrypt
columns with actual benchmark data. WASM decrypt columns remain blocked
on rebuilding tdfcore.wasm with tdf_decrypt export. Add analysis of
Chicory interpreter overhead and TS WASM near-native performance.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All three WASM hosts now report encrypt and decrypt benchmarks after
rebuilding tdfcore.wasm with tdf_decrypt export (TinyGo reactor mode).

Key results:
- TS WASM decrypt: 1.0-2.7 ms (matches Go WASM, 15-60x faster than SDK+KAS)
- Java WASM decrypt: 2.8-27.4 ms (Chicory interpreter overhead)
- Go WASM decrypt: 1.2-2.5 ms (baseline)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move DEK unwrap inside the benchmark loop so WASM decrypt timing
includes the full host-side flow (RSA-OAEP unwrap + AES-GCM decrypt).
Also align decrypt table footnotes with Java/TS benchmark format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete benchmark matrix now covers 256B through 100MB across all
three WASM hosts. Key findings at large sizes:
- TS WASM decrypt: 100MB in 115ms (~870 MB/s), fastest WASM host
- Go WASM decrypt: 100MB in 266ms (~376 MB/s)
- Java/Chicory WASM decrypt: 100MB in 2,254ms (interpreter overhead)
- WASM encrypt OOMs at 100MB (TinyGo gc=leaking memory limit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update SDK-WASM-1 spike document from Draft to Complete:
- Fill in Go/No-Go criteria results (all 5 PASS)
- Add full Results section with binary size, correctness, and
  performance data from cross-SDK benchmarks (256B-100MB)
- Update scope table: decrypt and multi-segment now implemented
- Add risks (100MB OOM, Chicory perf) and M2 recommendations
- Link to detailed benchmark results doc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

X-Test Failure Report

opentdfplatformEUQXSB.dockerbuild

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 199.260444ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 102.210866ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 367.188321ms
Throughput 272.34 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 40.217059655s
Average Latency 400.600787ms
Throughput 124.33 requests/second

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 202.086698ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 102.911032ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 371.457456ms
Throughput 269.21 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 40.114235099s
Average Latency 399.719044ms
Throughput 124.64 requests/second

Revised web-sdk benchmark numbers: TS WASM encrypt unchanged (24-130x
faster than TS SDK due to cached key + no framework overhead), but TS
WASM decrypt now includes KAS rewrap over HTTP, matching native TS SDK
performance (~46-84ms at small sizes). Updated both docs to distinguish
Go/Java WASM (local RSA unwrap) from TS WASM (KAS-inclusive) decrypt.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 184.907301ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 95.423125ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 381.250373ms
Throughput 262.29 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 40.391473458s
Average Latency 401.351779ms
Throughput 123.79 requests/second

pflynn-virtru and others added 2 commits February 23, 2026 12:28
Switch tdf_encrypt from flat-buffer API (plaintext + output in WASM
linear memory) to streaming I/O via read_input/write_output host
callbacks. This enables 100MB+ payloads without WASM OOM by streaming
data through host callbacks instead of copying everything into linear
memory.

- Replace 13-param flat-buffer signature with 10-param streaming
  signature (plaintextSize:i64 replaces ptPtr/ptLen/outPtr/outCapacity)
- Add encryptStream() with reusable ptBuf/ctBuf per segment (~2x
  segmentSize memory regardless of total file size)
- Add AesGcmEncryptInto() for zero-alloc encryption into caller buffer
- Change IOConfig to IOState with mutex for swappable Reader/Writer
- Switch all //go:wasmexport to //export for TinyGo compatibility
  (//go:wasmexport traps after proc_exit in TinyGo's wrapper)
- Add tdfcore TinyGo build target to Makefile
- Add TestTDFEncryptStreamLargePayload (1MB, 16 segments, round-trip)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… numbers

- Add Java SDK column to encrypt/decrypt tables (end-to-end with KAS)
- Update Java WASM 100MB: OOM → 14,228ms (streaming I/O)
- Add Java WASM decrypt numbers with +25ms KAS estimate
- Mark OOM risk as resolved (streaming read_input/write_output)
- Mark streaming I/O as implemented in scope and M2 status
- Note TinyGo //export vs //go:wasmexport finding (risk #4)
- Update architecture diagram with //export and I/O imports
- Update tdf_encrypt signature to streaming 10-param version

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

X-Test Failure Report

opentdfplatformLBSIVE.dockerbuild

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 163.419546ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 81.220416ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 373.610692ms
Throughput 267.66 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 38.868603826s
Average Latency 386.965028ms
Throughput 128.64 requests/second

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 194.395147ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 96.943332ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 362.470765ms
Throughput 275.88 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 40.421072677s
Average Latency 402.390573ms
Throughput 123.70 requests/second

pflynn-virtru and others added 2 commits February 23, 2026 12:51
- examples/cmd/wasm_verify.go: TinyGo compilation, real read_input/
  write_output I/O callbacks, 10-param streaming tdf_encrypt
- examples/cmd/benchmark_cross_sdk.go: add 10MB/100MB default sizes,
  auto segment size selection for large payloads
- host/encrypt_test.go: TinyGo build, testing.TB for benchmark compat
- host/decrypt_test.go: dynamic output buffer sizing, testing.TB

Go/wazero TinyGo benchmark (3 iterations):
  Encrypt: 0.1ms (16KB) → 168ms (100MB)
  Decrypt: 1.2ms (16KB) → 224ms (100MB)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TinyGo build (v0.40.1) dramatically improves WASM performance:
- Encrypt: ~2-3x of native Go SDK (down from ~8-10x with std Go)
- 100MB encrypt: 168ms (was 544ms), decrypt: 224ms (was 266ms)
- All benchmarks and hosts now use TinyGo binary (150KB)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 176.663899ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 97.313944ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 390.922139ms
Throughput 255.81 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.713115532s
Average Latency 395.577125ms
Throughput 125.90 requests/second

pflynn-virtru and others added 2 commits February 23, 2026 13:06
Browser WASM (Chromium/V8 via Playwright, TinyGo streaming):
  16KB: 0.6ms, 64KB: 3.8ms, 256KB: 3.2ms, 1MB: 11.9ms, 10MB: 117.4ms
Browser is ~5-15x slower than Go/wazero due to async crypto bridge
overhead (Worker+SharedArrayBuffer+Atomics for SubtleCrypto calls).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TS SDK encrypt/decrypt columns to benchmark tables. Update browser
WASM 100MB from untested to 1,144.8ms (4.6x faster than TS SDK).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 191.599931ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 93.452544ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 361.502567ms
Throughput 276.62 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 38.920502481s
Average Latency 387.132951ms
Throughput 128.47 requests/second

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:ci Github Actions Work comp:sdk A software development kit, including library, for client applications and inter-service communicati docs Documentation size/l

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant