Improve Canton openApi Docs generating directly from annotated protos#2
Open
monsieurleberre wants to merge 4 commits into
Open
Improve Canton openApi Docs generating directly from annotated protos#2monsieurleberre wants to merge 4 commits into
monsieurleberre wants to merge 4 commits into
Conversation
…e options Demonstrates Option 4 from the C# SDK proposal appendix: annotate Canton's .proto files with google.api.http so the JSON HTTP surface can be derived from the protos directly, bypassing tapir's translation layer. Scope (one service end-to-end): - community/ledger-api-proto/.../v2/command_service.proto: imports google/api/annotations.proto and adds an option (google.api.http) block to each of the three RPCs (SubmitAndWait, SubmitAndWaitForTransaction, SubmitAndWaitForReassignment), matching the existing tapir routes. - All v2 .proto files (22) plus value.proto: add option go_package next to the existing csharp_namespace/java_package options. Required by every off-the-shelf OpenAPI generator in the ecosystem. Harmless to existing C#/Java/Scala codegen, which does not read this option. Verified with `buf build`: clean compile. Verified end-to-end OpenAPI generation with buf.build/grpc-ecosystem/openapiv2:v2.28.0: the output fixes all four type-system defects catalogued in the C# SDK proposal (numbered duplicate schemas, single-key oneOf wrappers, inline duplicated enums, untyped DAML payload fields). See peacefulstudio/documentation/dev-funding/openApi/upstream-fork-experiment.md for the full write-up.
Builds on the CommandService experiment by annotating every other v2 RPC
the JSON API exposes over HTTP, plus adding go_package to the admin/,
testing/, and interactive/ sub-packages so off-the-shelf OpenAPI
generators can resolve the full proto closure.
Annotated services (route source: community/ledger/ledger-json-api Js*Service.scala):
v2/
- CommandSubmissionService: Submit, SubmitReassignment
- CommandCompletionService: CompletionStream
- ContractService: GetContract
- EventQueryService: GetEventsByContractId
- PackageService: ListPackages, GetPackage,
GetPackageStatus, ListVettedPackages
- StateService: GetActiveContracts, GetActiveContractsPage,
GetConnectedSynchronizers, GetLedgerEnd,
GetLatestPrunedOffsets
- UpdateService: GetUpdates, GetUpdateByOffset,
GetUpdateById, GetUpdatesPage
- VersionService: GetLedgerApiVersion
v2/admin/
- IdentityProviderConfigService: 5 RPCs
- PackageManagementService: UploadDarFile, ValidateDarFile,
UpdateVettedPackages
- PartyManagementService: 7 HTTP-exposed RPCs
- UserManagementService: 9 RPCs
v2/interactive/
- InteractiveSubmissionService: 6 RPCs
Each annotation matches the existing tapir route (method + path + body
mapping) so the proto-derived OpenAPI is comparable to the current
tapir-emitted document.
Scope notes:
- Server-streaming RPCs (CompletionStream, GetActiveContracts,
GetUpdates) get only the canonical POST/list mapping; the WebSocket
GET form is a JSON-API-only convenience that has no representation in
google.api.http and remains in the AsyncAPI doc.
- Legacy JSON-only paths (/v2/updates/flats|trees|transaction-*,
/v2/authenticated-user, /v2/definitions/*) are not annotated — they
either lack a 1:1 proto RPC or are non-gRPC custom endpoints.
- Non-HTTP-exposed RPCs (TimeService, CommandInspectionService,
ParticipantPruningService, PackageManagementService.ListKnownPackages,
PartyManagementService.UpdatePartyIdentityProviderId) are deliberately
left unannotated.
Verified locally with `buf build community/ledger-api-proto/src/main/protobuf`:
clean compile.
The JSON API exposes this as GET with a JSON body containing a required
EventFormat message — non-standard REST, and rejected by
grpc-ecosystem/openapiv2 ("must not set request body when http method
is GET"). Switching to POST is the canonical mapping for a request
with a non-trivial body, and is the single intentional deviation from
the existing tapir route for the proto-derived OpenAPI.
Mirrors the experiment-evidence pattern from the single-service experiment, scaled to the full HTTP API: - openapi-3.0/openapi.yaml — OpenAPI 3.0.3 YAML generated via buf.build/community/google-gnostic-openapi:v0.7.0; same major version as the existing community/ledger/ledger-json-api/.../openapi.yaml, so the two can be diffed directly. - swagger-2.0/canton-ledger-api.swagger.yaml — Swagger 2.0 YAML via buf.build/grpc-ecosystem/openapiv2:v2.28.0; same plugin as the upstream CommandService experiment, merged across every annotated service. - coverage-comparison/ — sorted path inventories and the diff between the JSON API spec and the proto-derived spec, after normalizing path-parameter names (proto field names ↔ tapir placeholder names). 9 paths-only-in-JSON-API (all deliberate skips: 1 custom JSON-only endpoint + 8 deprecated legacy aliases), 0 paths-only-in-proto. - reproduce.sh — one-shot regen of every artefact in this directory. Internal to the Peaceful Studio fork. To be deleted before any upstream PR to digital-asset/canton.
monsieurleberre
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before : SwaggerHub with current openApi Docs
After : SwaggerHub with proto generated openApi Docs
Extends #1 from a single service to every HTTP-exposed RPC of the JSON Ledger API.
Builds on the same design and rationale — see #1 for the why.
What's done
community/ledger-api-proto:go_packageoption added to admin/, testing/, interactive/ protos that experiment: google.api.http annotations on CommandService (Peaceful Studio internal) #1 didn't touch.buf buildpasses oncommunity/ledger-api-proto/src/main/protobuf.http-annotations-evidence/:openapi-3.0/openapi.yaml(3.0.3, comparable to the existing JSON-API yaml) — SwaggerHub: https://app.swaggerhub.com/apis/peacefulstudio/canton-ledger-api-proto-derived/v2#/swagger-2.0/canton-ledger-api.swagger.yaml(same plugin as experiment: google.api.http annotations on CommandService (Peaceful Studio internal) #1).coverage-comparison/— diff vs the existing yaml shows 9 paths only in JSON-API (all deliberate skips) and 0 paths only in proto.reproduce.shregenerates every artefact from the protos.Intentional deviations from tapir
StateService.GetActiveContractsPageis POST, not GET. The tapir route uses GET-with-body, whichgrpc-ecosystem/openapiv2rejects and which OpenAPI generally disallows. The request carries a requiredEventFormatmessage that doesn't flatten to query params.CompletionStream,GetActiveContracts,GetUpdates) get the POST list mapping only. The WebSocket GET form is JSON-API-only and stays in the AsyncAPI doc.Deliberately not annotated
/v2/commands/submit-and-wait-for-transaction-tree,/v2/package-vetting,/v2/updates/{flats,trees,transaction-*})./v2/authenticated-user— custom JSON-only, no gRPC backing.TimeService,CommandInspectionService,ParticipantPruningService,PackageManagementService.ListKnownPackages,PartyManagementService.UpdatePartyIdentityProviderId).Left to do (not in this PR)
GetActiveContractsPageto POST so the JSON-API and the proto-derived spec agree on that one route.additional_bindingsfor the 3.4 → 3.5 transition window, or treat their removal from the spec as a feature of this change.http-annotations-evidence/and split into the minimal commits before any PR to digital-asset/canton.Test plan
buf build community/ledger-api-proto/src/main/protobufis clean.bash http-annotations-evidence/reproduce.shreproduces the committed yamls byte-for-byte.go_packageis inert for them — proven in experiment: google.api.http annotations on CommandService (Peaceful Studio internal) #1).