feat(sdk): TinyGo WASM canary builds and spike plan#3063
feat(sdk): TinyGo WASM canary builds and spike plan#3063pflynn-virtru wants to merge 37 commits intomainfrom
Conversation
… and zipwrite modules
ADR documenting the TinyGo hybrid WASM architecture spike (SDK-WASM-1), including host crypto ABI, go/no-go criteria, and task breakdown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Summary of ChangesHello @pflynn-virtru, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request lays the groundwork for a significant architectural shift by exploring the feasibility of a TinyGo-compiled WebAssembly (WASM) core engine for TDF operations. It introduces a set of targeted canary programs to assess TinyGo's compatibility with crucial Go standard library components and establishes a detailed spike plan. The primary aim is to validate a hybrid approach where cryptographic primitives are delegated to the host environment, while core TDF logic resides within a compact and portable WASM module, paving the way for enhanced flexibility and deployment options. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Go code, now WASM bound, TinyGo makes it small and fast, New frontiers are found. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive spike plan (ADR) for a TinyGo-based WASM core engine and adds several canary programs to test the compatibility of necessary Go standard library packages with TinyGo. The ADR is well-structured and detailed, covering architecture, ABI, risks, and a task breakdown. The canary programs are good initial steps to de-risk the effort.
My review focuses on improving the clarity of the ADR and addressing a potential memory safety issue in the wasmimport canary. I've suggested a clarification in the ADR's objective to align it better with the documented scope. I've also recommended adding a crucial comment to the wasmMalloc implementation to highlight its dependency on the -gc=leaking build flag for memory safety, which is critical for future maintenance.
| buf := make([]byte, size) | ||
| return uint32(uintptr(unsafe.Pointer(&buf[0]))) |
There was a problem hiding this comment.
The current implementation of wasmMalloc has a potential memory safety issue. The buf slice is allocated within the function, and returning a pointer to its underlying data can lead to a dangling pointer if the garbage collector reclaims buf after the function returns. While this is mitigated by compiling with the -gc=leaking flag as mentioned in the ADR, this critical dependency is not apparent from the code itself. For better maintainability and to prevent accidental misuse, I recommend adding a comment explaining why this is safe under the specific build conditions and highlighting the intentional memory leak.
buf := make([]byte, size)
// NOTE: This leaks memory. This is only safe because the module is compiled
// with the `-gc=leaking` flag, which prevents the garbage collector from
// reclaiming the memory.
return uint32(uintptr(unsafe.Pointer(&buf[0])))| ## Objective | ||
|
|
||
| Validate that a TinyGo-compiled WASM module can perform TDF3 single-segment | ||
| encrypt/decrypt with all crypto delegated to host functions, producing output |
There was a problem hiding this comment.
The objective states encrypt/decrypt, but the scope of the spike seems to be focused on implementing encryption within the WASM module and then validating the output using the existing Go SDK for decryption. The 'Explicitly Out of Scope' section also mentions 'Decrypt inside WASM' is for a future milestone (M2). To better reflect the spike's goal, consider clarifying that only encryption will be performed by the WASM module by removing /decrypt.
| encrypt/decrypt with all crypto delegated to host functions, producing output | |
| encrypt with all crypto delegated to host functions, producing output |
X-Test Failure Report |
…e logic - Removed `wasmimport` module and redundant crypto directives. - Consolidated `calculateSignature` logic into `writer.go` and removed it from `manifest.go`. - Updated TinyGo canary workflow to reflect WASM module restructuring.
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Adds self-contained decrypt benchmarks that construct valid TDFs programmatically and inject payload keys directly, bypassing KAS. Covers 1MB–2GB WriteTo path and 100MB streaming Read() path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Benchmarks for the experimental streaming TDF Writer covering: - End-to-end encrypt (NewWriter + WriteSegment + Finalize) - Single segment encrypt throughput (WriteSegment only) - Full TDF assembly (segments + finalize bytes) Sizes: 1MB, 100MB, 1GB (short-mode skippable). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…ibility
Replace reflection-based encoding/json with tinyjson (CosmWasm fork) codegen
for manifest, assertion, and policy structs in the TDF write path. This is a
prerequisite for the WASM core engine spike (SDK-WASM-1) since encoding/json
panics at runtime under TinyGo.
Changes:
- Add tinyjson codegen for manifest.go (13 types) and assertion_types.go (3 types)
- Change KeyAccess.PolicyBinding from interface{} to concrete PolicyBinding type
- Replace json.Marshal with .MarshalJSON() in writer.go, key_access.go, assertion.go
- Move Assertion/Statement/Binding structs to assertion_types.go for codegen
- Drop polymorphic Statement.Value UnmarshalJSON (reader concern, out of WASM scope)
- Add tinyjson TinyGo canary with manifest/policy/assertion round-trip validation
- Add wasm/Makefile with toolcheck, build, run, generate targets
- Add wasm/README.md with TinyGo/tinyjson/wasmtime install instructions
Canary results: tinyjson module compiles to 62KB raw / 29KB gzipped WASM,
all round-trip tests pass under wasmtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…r WASM Copy production zipstream writer code (5 files from sdk/internal/zipstream/) into a standalone canary module and verify it compiles and runs correctly under TinyGo WASM. This completes Phase 1 (Foundation) of the WASM core engine spike (SDK-WASM-1). The canary exercises: - Single-segment TDF ZIP creation (header + manifest + central directory) - Multi-segment out-of-order writing (3 segments in order 2, 0, 1) - ZIP64 mode (ZIP64 EOCD + locator signatures) - CRC32 combine (multi-part checksum matches direct computation) Key finding: time.Time and time.Now() work correctly under TinyGo — the only identified risk for zipstream compatibility. Binary size: 113KB raw / 59KB gzipped (well under 300KB budget). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Use grouped redirects ({ cmd1; cmd2; } >> file) instead of individual
redirects to satisfy shellcheck/actionlint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Document the three I/O models evaluated (WASM-drives, host-drives, hybrid) and recommend hybrid streaming I/O for M2 with read_input and write_output host imports. Update ABI evolution to 13 functions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…opies Replace ptrToBytes with zero-copy ptrToSlice for TDF input and DEK, write decrypted segments directly into the host-provided output buffer via new AesGcmDecryptInto, removing ~3x payload memory overhead that caused OOM at 100MB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Update benchmark results doc to include Java WASM (Chicory) and TypeScript WASM (Node.js WebAssembly) columns, with instructions for running WASM benchmarks across all three SDKs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Populate Java WASM (Chicory) and TypeScript WASM (Node.js) encrypt columns with actual benchmark data. WASM decrypt columns remain blocked on rebuilding tdfcore.wasm with tdf_decrypt export. Add analysis of Chicory interpreter overhead and TS WASM near-native performance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All three WASM hosts now report encrypt and decrypt benchmarks after rebuilding tdfcore.wasm with tdf_decrypt export (TinyGo reactor mode). Key results: - TS WASM decrypt: 1.0-2.7 ms (matches Go WASM, 15-60x faster than SDK+KAS) - Java WASM decrypt: 2.8-27.4 ms (Chicory interpreter overhead) - Go WASM decrypt: 1.2-2.5 ms (baseline) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move DEK unwrap inside the benchmark loop so WASM decrypt timing includes the full host-side flow (RSA-OAEP unwrap + AES-GCM decrypt). Also align decrypt table footnotes with Java/TS benchmark format. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete benchmark matrix now covers 256B through 100MB across all three WASM hosts. Key findings at large sizes: - TS WASM decrypt: 100MB in 115ms (~870 MB/s), fastest WASM host - Go WASM decrypt: 100MB in 266ms (~376 MB/s) - Java/Chicory WASM decrypt: 100MB in 2,254ms (interpreter overhead) - WASM encrypt OOMs at 100MB (TinyGo gc=leaking memory limit) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update SDK-WASM-1 spike document from Draft to Complete: - Fill in Go/No-Go criteria results (all 5 PASS) - Add full Results section with binary size, correctness, and performance data from cross-SDK benchmarks (256B-100MB) - Update scope table: decrypt and multi-segment now implemented - Add risks (100MB OOM, Chicory perf) and M2 recommendations - Link to detailed benchmark results doc Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Revised web-sdk benchmark numbers: TS WASM encrypt unchanged (24-130x faster than TS SDK due to cached key + no framework overhead), but TS WASM decrypt now includes KAS rewrap over HTTP, matching native TS SDK performance (~46-84ms at small sizes). Updated both docs to distinguish Go/Java WASM (local RSA unwrap) from TS WASM (KAS-inclusive) decrypt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Switch tdf_encrypt from flat-buffer API (plaintext + output in WASM linear memory) to streaming I/O via read_input/write_output host callbacks. This enables 100MB+ payloads without WASM OOM by streaming data through host callbacks instead of copying everything into linear memory. - Replace 13-param flat-buffer signature with 10-param streaming signature (plaintextSize:i64 replaces ptPtr/ptLen/outPtr/outCapacity) - Add encryptStream() with reusable ptBuf/ctBuf per segment (~2x segmentSize memory regardless of total file size) - Add AesGcmEncryptInto() for zero-alloc encryption into caller buffer - Change IOConfig to IOState with mutex for swappable Reader/Writer - Switch all //go:wasmexport to //export for TinyGo compatibility (//go:wasmexport traps after proc_exit in TinyGo's wrapper) - Add tdfcore TinyGo build target to Makefile - Add TestTDFEncryptStreamLargePayload (1MB, 16 segments, round-trip) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… numbers - Add Java SDK column to encrypt/decrypt tables (end-to-end with KAS) - Update Java WASM 100MB: OOM → 14,228ms (streaming I/O) - Add Java WASM decrypt numbers with +25ms KAS estimate - Mark OOM risk as resolved (streaming read_input/write_output) - Mark streaming I/O as implemented in scope and M2 status - Note TinyGo //export vs //go:wasmexport finding (risk #4) - Update architecture diagram with //export and I/O imports - Update tdf_encrypt signature to streaming 10-param version Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
- examples/cmd/wasm_verify.go: TinyGo compilation, real read_input/ write_output I/O callbacks, 10-param streaming tdf_encrypt - examples/cmd/benchmark_cross_sdk.go: add 10MB/100MB default sizes, auto segment size selection for large payloads - host/encrypt_test.go: TinyGo build, testing.TB for benchmark compat - host/decrypt_test.go: dynamic output buffer sizing, testing.TB Go/wazero TinyGo benchmark (3 iterations): Encrypt: 0.1ms (16KB) → 168ms (100MB) Decrypt: 1.2ms (16KB) → 224ms (100MB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TinyGo build (v0.40.1) dramatically improves WASM performance: - Encrypt: ~2-3x of native Go SDK (down from ~8-10x with std Go) - 100MB encrypt: 168ms (was 544ms), decrypt: 224ms (was 266ms) - All benchmarks and hosts now use TinyGo binary (150KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Browser WASM (Chromium/V8 via Playwright, TinyGo streaming): 16KB: 0.6ms, 64KB: 3.8ms, 256KB: 3.2ms, 1MB: 11.9ms, 10MB: 117.4ms Browser is ~5-15x slower than Go/wazero due to async crypto bridge overhead (Worker+SharedArrayBuffer+Atomics for SubtleCrypto calls). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TS SDK encrypt/decrypt columns to benchmark tables. Update browser WASM 100MB from untested to 1,144.8ms (4.6x faster than TS SDK). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Summary
sdk/experimental/tdf/wasm/that exercise the stdlib and third-party packages needed for a hybrid WASM TDF core enginesdk/experimental/tdf/manifest, assertion, and key-access structs fromencoding/jsonto tinyjson codegen for TinyGo WASM compatibility — includes generated marshal/unmarshal code and updated teststinyjsonandzipstreamcanaries that validate these migrations compile and round-trip correctly under TinyGo.github/workflows/tinygo-wasm-canary.yaml) that compiles each canary with TinyGo targetingwasip1— expected to have failures until spike work is completedocs/adr/spike-wasm-core-tinygo-hybrid.md) documenting the TinyGo hybrid architecture, host crypto ABI (8 functions), go/no-go criteria, and task breakdownsdk/benchmark_test.go) measuring pure decrypt throughput at 1MB–2GB using direct key injection (no KAS dependency)sdk/experimental/tdf/benchmark_test.go) measuring streaming encrypt throughput, single-segment performance, and full TDF assemblyhostcryptopackage (sdk/experimental/tdf/wasm/hostcrypto/) with typed Go wrappers for allgo:wasmimporthost functions — the WASM-side ABI layer for crypto and I/Osdk/experimental/tdf/wasm/host/) implementing the host side of the crypto/IO ABI — registers"crypto"and"io"modules with Wazero, delegating all crypto tolib/ocrypto. Includes 20 tests covering ABI conformance, round-trip correctness, error paths, and OOB handling.hostcrypto; manifest construction, policy binding (HS256), integrity computation, and ZIP assembly run inside WASM using tinyjson types and zipstream.ptrToBytescopies with zero-copyptrToSlice, writing decrypted segments directly into the host-provided output buffer viaAesGcmDecryptInto, and removing intermediate accumulation. Peak WASM Go-heap usage during decrypt drops from ~300 MB to ~0 for a 100 MB TDF.tdf_encrypt: round-trip decrypt, manifest field validation, policy binding, segment/root integrity, UUID format, attributes, empty plaintext, deterministic sizes, and error paths. Includes wazero proc_exit fix for Go 1.25 wasip1 and GC-safety fixes for WASM malloc.examples/cmd/benchmark_cross_sdk.go) comparing Production SDK, Experimental Writer, and WASM across payload sizes from 256 B to 100 MB.Canary programs
base64hexencoding/base64,encoding/hexzipwriteencoding/binary,hash/crc32,bytes,sort,synctinyjsonzipstreamiocontextio,context,strings,strconv,fmt,errorsstdjsonencoding/jsonwith TDF manifest structs (superseded bytinyjson)wasmgo:wasmimporthost ABI +tdfpackageHost crypto ABI
random_bytesocrypto.RandomBytes(n)aes_gcm_encryptocrypto.NewAESGcm(key).Encrypt(pt)aes_gcm_decryptocrypto.NewAESGcm(key).Decrypt(ct)hmac_sha256ocrypto.CalculateSHA256Hmac(key, data)rsa_oaep_sha1_encryptocrypto.NewAsymEncryption(pem).Encrypt(pt)rsa_oaep_sha1_decryptocrypto.NewAsymDecryption(pem).Decrypt(ct)rsa_generate_keypairocrypto.NewRSAKeyPair(bits)get_last_errorread_inputcfg.Input.Read(buf)write_outputcfg.Output.Write(buf)WASM TDF encrypt (Task 3.1)
Single-segment TDF3 encrypt running entirely inside the WASM sandbox:
hostcrypto.RandomBytes(32)hostcrypto.RsaOaepSha1Encrypt(kasPub, dek)MarshalJSON()HMAC-SHA256(dek, base64Policy)→ hex → base64hostcrypto.AesGcmEncrypt(dek, plaintext)HMAC-SHA256(dek, cipher)→ base64HMAC-SHA256(dek, segmentSig)→ base64WriteSegment+FinalizeEncrypt integration tests (11 tests)
TestTDFEncryptRoundTripTestTDFEncryptManifestFieldsTestTDFEncryptPolicyBindingTestTDFEncryptSegmentIntegrityTestTDFEncryptPolicyUUIDTestTDFEncryptWithAttributesTestTDFEncryptEmptyPlaintextTestTDFEncryptDeterministicSizesTestTDFEncryptErrorInvalidKeyTestTDFEncryptErrorBufferTooSmallTestTDFEncryptGetErrorClearsAfterReadget_errorretrievalCross-SDK Benchmarks
Measured on a single dev machine (Apple Silicon M3 Max); numbers are indicative, not normative. 5 iterations per size.
Encrypt
Decrypt
*Production SDK: includes KAS rewrap network latency (~20 ms round-trip to localhost)
**WASM: includes local RSA-OAEP DEK unwrap (no network); in production the host would call KAS for rewrap
WASM decrypt memory optimization
ptrToBytes)ptrToSlice)AesGcmDecryptIntowrites to output)plaintextslicetdfDecryptMicrobenchmarks
BenchmarkDecrypt/1MB–2GBsdk/benchmark_test.goBenchmarkStreamDecrypt(100MB, 32KB reads)sdk/benchmark_test.goBenchmarkWriterEncrypt/1MB–1GBsdk/experimental/tdf/benchmark_test.goBenchmarkWriterWriteSegment(2MB)sdk/experimental/tdf/benchmark_test.goBenchmarkWriterAssemble/1MB–100MBsdk/experimental/tdf/benchmark_test.goCI workflow
ciaggregation job, does not block PRsfail-fast: false— all canaries run independentlysdk/experimental/tdf/**,sdk/internal/zipstream/**,sdk/manifest.go,lib/ocrypto/**Test plan
base64hex,zipwrite,tinyjson, andzipstreamcanaries pass TinyGo compilation and executiongo build ./sdk/experimental/tdf/wasm/...)ciaggregation jobneedslistcd sdk && go test -bench=BenchmarkDecrypt -short -run=^$ .cd sdk/experimental/tdf && go test -bench=Benchmark -short -run=^$ .go test -v ./sdk/experimental/tdf/wasm/host/(44 host ABI + integration tests)go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFEncrypt(11 end-to-end tests)go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFDecrypt(13 end-to-end tests)cd examples && go run . benchmark-cross-sdk -e ... --sizes "256,1024,16384,65536,262144,1048576,10485760,104857600"go build ./sdk/...GOOS=wasip1 GOARCH=wasm go build ./sdk/experimental/tdf/wasm/Related: SDK-WASM-1 (Jira)
🤖 Generated with Claude Code