diff --git a/cli/azd/cmd/container.go b/cli/azd/cmd/container.go index 4aa90b41155..61a0c5194b2 100644 --- a/cli/azd/cmd/container.go +++ b/cli/azd/cmd/container.go @@ -1035,6 +1035,7 @@ func registerCommonDependencies(container *ioc.NestedContainer) { ) container.MustRegisterSingleton(grpcserver.NewAiModelService) container.MustRegisterScoped(grpcserver.NewCopilotService) + container.MustRegisterSingleton(grpcserver.NewTelemetryService) // Required for nested actions called from composite actions like 'up' registerAction[*cmd.ProvisionAction](container, "azd-provision-action") diff --git a/cli/azd/docs/extensions/extension-framework.md b/cli/azd/docs/extensions/extension-framework.md index 60f550d995f..7bbac014c92 100644 --- a/cli/azd/docs/extensions/extension-framework.md +++ b/cli/azd/docs/extensions/extension-framework.md @@ -1104,6 +1104,7 @@ Extensions can declare the following capabilities in their manifest: - **`provisioning-provider`**: Provide a custom infrastructure provisioning experience (alternative to Bicep / Terraform) - **`validation-provider`**: Contribute validation checks to azd's provision validation and future validation pipelines - **`metadata`**: Provide comprehensive metadata about commands and configuration schemas +- **`telemetry`**: Report usage attributes declared in the extension's official registry entry (see [Extension Telemetry Fields](./extension-telemetry-fields.md)) #### Complete Extension Manifest Example diff --git a/cli/azd/docs/extensions/extension-sdk-reference.md b/cli/azd/docs/extensions/extension-sdk-reference.md index ba91ef45e3a..64e805f7396 100644 --- a/cli/azd/docs/extensions/extension-sdk-reference.md +++ b/cli/azd/docs/extensions/extension-sdk-reference.md @@ -519,9 +519,51 @@ gRPC client connecting to the azd framework. Auto-discovers the socket via | `Extension()` | `ExtensionServiceClient` | | `Account()` | `AccountServiceClient` | | `Ai()` | `AiModelServiceClient` | +| `Telemetry()` | `TelemetryServiceClient` | Always call `defer client.Close()` after creation. +#### TelemetryService + +`Telemetry().ReportUsageAttribute(ctx, &azdext.ReportUsageAttributeRequest{Key, Value})` +lets an authenticated extension report one usage attribute value. The +extension declares the key and its closed value set in its entry in the +official azd registry; `azd` core owns no product-specific fields, so adding a +new field is a registry change rather than a core release. + +The host validates every call and fails closed. It requires the extension to +be installed from the official `azd` registry, to carry the `telemetry` +capability, and to have declared both the key and the exact value. Accepted +values are recorded on a dedicated `ext.usage` span that shares the command's +trace, so downstream queries join it to the originating command on +`operation_Id`. Extensions cannot choose the span, classification, purpose, +hashing, or aggregation. + +Declare fields in the registry entry alongside the capability: + +```json +{ + "version": "1.0.0", + "capabilities": ["telemetry"], + "telemetry": [ + { + "key": "ext.contoso.tools.deploy.mode", + "allowedValues": ["code", "container"] + } + ] +} +``` + +Keys must be namespaced as `ext..`, and values are +limited to lowercase alphanumerics with `_`, `-`, and `.` so registry names, +URLs, and paths cannot be smuggled through a value. See +[Extension Telemetry Fields](./extension-telemetry-fields.md) for the full +declaration rules and review process. + +The call is best-effort. An older azd host returns `Unimplemented`. Treat any +failure as a no-op and never let it change command behavior. Report a value as +soon as it is known so a later failure still retains it. + ### ConfigHelper ```go diff --git a/cli/azd/docs/extensions/extension-telemetry-fields.md b/cli/azd/docs/extensions/extension-telemetry-fields.md new file mode 100644 index 00000000000..9206b61cc83 --- /dev/null +++ b/cli/azd/docs/extensions/extension-telemetry-fields.md @@ -0,0 +1,106 @@ +# Extension Telemetry Fields + +This guide is for extension authors who need `azd` to record a bounded usage +signal on their behalf — for example, which deployment mode a user picked. + +`azd` core owns no product-specific telemetry fields. Every field an extension +can report is declared in that extension's entry in the official `azd` +registry, and `azd` only enforces the declaration. Adding a field is therefore +a registry pull request in this repository, not a core release. + +See [ADR-001](../../../../docs/architecture/adr-001-extension-telemetry-declarations.md) +for the reasoning behind this design. + +## What you can and cannot report + +| | | +|---|---| +| You choose | The key name and the closed set of values it may take | +| `azd` chooses | Which span the value lands on, its classification, and how it is exported | +| Always rejected | Free-form values, values outside your declared set, keys you did not declare | + +Values must be a closed enum because that is what makes a privacy review +possible. A shape-only rule (length and charset) would let someone pack a +container image reference into a value and leak a private registry name, so +the charset excludes `:` and `/` and the host compares each value against +your declaration. + +## Step 1: Declare the capability + +Add `telemetry` to `capabilities` in your `extension.yaml`: + +```yaml +capabilities: + - custom-commands + - telemetry +``` + +## Step 2: Declare the fields in the registry entry + +Add a `telemetry` array to your version entry in the official registry +(`cli/azd/extensions/registry.json`). Unlike `capabilities`, this is not +copied from `extension.yaml` — it only exists in the reviewed registry entry: + +```json +{ + "version": "1.0.0", + "capabilities": ["custom-commands", "telemetry"], + "telemetry": [ + { + "key": "ext.contoso.tools.deploy.mode", + "allowedValues": ["code", "container", "unknown"] + } + ] +} +``` + +Rules enforced when the registry is validated: + +| Rule | Limit | +|---|---| +| Key namespace | Must start with `ext..` | +| Key length | 128 characters | +| Fields per version | 16 | +| Values per field | 1–32, unique | +| Value length | 64 characters | +| Value charset | Lowercase alphanumerics plus `_`, `-`, `.` | +| Capability | `telemetry` must be present when fields are declared | + +The pull request that adds these values is the privacy review. Expect +reviewers to ask what each value means and why it is needed. + +## Step 3: Report from your extension + +```go +_, err := client.Telemetry().ReportUsageAttribute( + ctx, + &azdext.ReportUsageAttributeRequest{ + Key: "ext.contoso.tools.deploy.mode", + Value: "container", + }, +) +``` + +Treat the call as best-effort. Older `azd` hosts return `Unimplemented`, and +every rejection is a plain error. Never let the result change command +behavior, and never retry. Report the value as soon as it is known so a later +failure in your command still keeps the signal. + +## Why a call might be rejected + +| Status | Cause | +|---|---| +| `Unauthenticated` | The request did not carry the host-issued extension token | +| `PermissionDenied` | The extension was not installed from the official `azd` registry, is missing the `telemetry` capability, or is not installed | +| `FailedPrecondition` | The stored declaration no longer passes validation | +| `InvalidArgument` | The key was not declared, or the value is not in the declared set | + +Error messages never echo the key or value you sent, so use the status code +plus your own declaration to diagnose. + +## Where the data lands + +Each accepted value becomes an `ext.usage` span carrying `extension.id`, +`extension.version`, and your attribute. The span shares the command's trace, +so it joins to the originating command on `operation_Id` in Application +Insights. diff --git a/cli/azd/extensions/extension.schema.json b/cli/azd/extensions/extension.schema.json index 2a8340e59ff..27e5f048585 100644 --- a/cli/azd/extensions/extension.schema.json +++ b/cli/azd/extensions/extension.schema.json @@ -112,7 +112,7 @@ "capabilities": { "type": "array", "title": "Capabilities", - "description": "List of capabilities provided by the extension. Supported values: custom-commands, lifecycle-events, mcp-server, service-target-provider, framework-service-provider, provisioning-provider, validation-provider, metadata. Select one or more from the allowed list. Each value must be unique. Not required for extension packs, which declare dependencies instead and have no executable.", + "description": "List of capabilities provided by the extension. Supported values: custom-commands, lifecycle-events, mcp-server, service-target-provider, framework-service-provider, provisioning-provider, validation-provider, metadata, telemetry. Select one or more from the allowed list. Each value must be unique. Not required for extension packs, which declare dependencies instead and have no executable.", "minItems": 1, "uniqueItems": true, "items": { @@ -159,11 +159,17 @@ "title": "Validation Provider", "description": "Validation provider enables extensions to contribute checks to azd validation pipelines." }, - { - "type": "string", - "const": "metadata", + { + "type": "string", + "const": "metadata", "title": "Metadata", "description": "Metadata capability enables extensions to provide comprehensive metadata about their commands and capabilities via a metadata command." + }, + { + "type": "string", + "const": "telemetry", + "title": "Telemetry", + "description": "Telemetry capability enables extensions to report usage attributes declared in their official registry entry. Only extensions installed from the official azd registry may report." } ] } diff --git a/cli/azd/extensions/microsoft.azd.extensions/internal/cmd/publish.go b/cli/azd/extensions/microsoft.azd.extensions/internal/cmd/publish.go index 7fec51145e2..5ef71c59cee 100644 --- a/cli/azd/extensions/microsoft.azd.extensions/internal/cmd/publish.go +++ b/cli/azd/extensions/microsoft.azd.extensions/internal/cmd/publish.go @@ -474,6 +474,11 @@ func addOrUpdateExtension( Dependencies: extensionMetadata.Dependencies, Providers: extensionMetadata.Providers, Artifacts: artifacts, + // Telemetry declarations and MCP config are authored + // directly in the registry and reviewed there, so carry + // them forward instead of dropping them on republish. + Telemetry: v.Telemetry, + McpConfig: v.McpConfig, } return diff --git a/cli/azd/extensions/registry.schema.json b/cli/azd/extensions/registry.schema.json index 5ab7d22be42..2ddbd6014c1 100644 --- a/cli/azd/extensions/registry.schema.json +++ b/cli/azd/extensions/registry.schema.json @@ -81,9 +81,42 @@ "service-target-provider", "framework-service-provider", "provisioning-provider", - "validation-provider", - "metadata" - ] + "validation-provider", + "metadata", + "telemetry" + ] + } + }, + "telemetry": { + "type": "array", + "description": "Usage attributes this version may report through the telemetry service. Requires the 'telemetry' capability.", + "maxItems": 16, + "items": { + "type": "object", + "properties": { + "key": { + "type": "string", + "description": "Telemetry attribute name, namespaced as 'ext..'.", + "maxLength": 128, + "pattern": "^ext\\.[a-z0-9]([a-z0-9-]*[a-z0-9])?(\\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$" + }, + "allowedValues": { + "type": "array", + "description": "Closed set of values that may be reported for this key. Free-form values are always rejected.", + "minItems": 1, + "maxItems": 32, + "uniqueItems": true, + "items": { + "type": "string", + "maxLength": 64, + "pattern": "^[a-z0-9]([a-z0-9_.-]*[a-z0-9])?$" + } + } + }, + "required": [ + "key", + "allowedValues" + ] } }, "usage": { diff --git a/cli/azd/grpc/proto/telemetry.proto b/cli/azd/grpc/proto/telemetry.proto new file mode 100644 index 00000000000..3b5256eaf95 --- /dev/null +++ b/cli/azd/grpc/proto/telemetry.proto @@ -0,0 +1,31 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. +syntax = "proto3"; + +package azdext; + +option go_package = "github.com/azure/azure-dev/cli/azd/pkg/azdext"; + +// TelemetryService accepts usage attribute values from authenticated +// extensions. Every key and value must appear in the telemetry declaration +// the extension published in the official azd registry, which is where those +// fields are reviewed. The host decides which telemetry span a value lands on +// and owns classification, purpose, hashing, and aggregation. +service TelemetryService { + // ReportUsageAttribute reports one declared usage attribute value. + rpc ReportUsageAttribute(ReportUsageAttributeRequest) + returns (ReportUsageAttributeResponse); +} + +message ReportUsageAttributeRequest { + // Key must match a telemetry field the extension declared in the registry. + string key = 1; + + // Value must be in the allowed set the extension declared for key. + string value = 2; +} + +message ReportUsageAttributeResponse { + // Accepted reports whether the host recorded the value. + bool accepted = 1; +} diff --git a/cli/azd/internal/grpcserver/extension_claims.go b/cli/azd/internal/grpcserver/extension_claims.go index a4273c57a36..6a8e166c04d 100644 --- a/cli/azd/internal/grpcserver/extension_claims.go +++ b/cli/azd/internal/grpcserver/extension_claims.go @@ -23,6 +23,7 @@ func GenerateExtensionToken(extension *extensions.Extension, serverInfo *ServerI ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour * 1)), }, Capabilities: extension.Capabilities, + Source: extension.Source, } jwtToken, err := jwt.NewWithClaims(jwt.SigningMethodHS256, claims).SignedString(serverInfo.SigningKey) diff --git a/cli/azd/internal/grpcserver/main_test.go b/cli/azd/internal/grpcserver/main_test.go new file mode 100644 index 00000000000..8786c36d7a0 --- /dev/null +++ b/cli/azd/internal/grpcserver/main_test.go @@ -0,0 +1,29 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package grpcserver + +import ( + "os" + "testing" + + "go.opentelemetry.io/otel" + tracesdk "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/sdk/trace/tracetest" +) + +// The OTel global provider only ever delegates to the first provider +// installed in a process, and internal/tracing resolves its tracer at +// package init. A provider installed inside a single test would therefore +// be silently ignored, so the package installs exactly one here and tests +// share the recorder. +var ( + testSpanRecorder = tracetest.NewSpanRecorder() + testTracerProvider = tracesdk.NewTracerProvider( + tracesdk.WithSpanProcessor(testSpanRecorder)) +) + +func TestMain(m *testing.M) { + otel.SetTracerProvider(testTracerProvider) + os.Exit(m.Run()) +} diff --git a/cli/azd/internal/grpcserver/prompt_service_test.go b/cli/azd/internal/grpcserver/prompt_service_test.go index 99868e564c9..5fac998dc8b 100644 --- a/cli/azd/internal/grpcserver/prompt_service_test.go +++ b/cli/azd/internal/grpcserver/prompt_service_test.go @@ -622,6 +622,7 @@ func setupTestServer(t *testing.T, promptSvc azdext.PromptServiceServer) ( azdext.UnimplementedCopilotServiceServer{}, azdext.UnimplementedProvisioningServiceServer{}, azdext.UnimplementedValidationServiceServer{}, + azdext.UnimplementedTelemetryServiceServer{}, ) serverInfo, err := server.Start() diff --git a/cli/azd/internal/grpcserver/server.go b/cli/azd/internal/grpcserver/server.go index 48779be0d8d..03abc800c5c 100644 --- a/cli/azd/internal/grpcserver/server.go +++ b/cli/azd/internal/grpcserver/server.go @@ -44,6 +44,7 @@ type Server struct { copilotService azdext.CopilotServiceServer provisioningService azdext.ProvisioningServiceServer validationService azdext.ValidationServiceServer + telemetryService azdext.TelemetryServiceServer } func NewServer( @@ -64,6 +65,7 @@ func NewServer( copilotService azdext.CopilotServiceServer, provisioningService azdext.ProvisioningServiceServer, validationService azdext.ValidationServiceServer, + telemetryService azdext.TelemetryServiceServer, ) *Server { return &Server{ projectService: projectService, @@ -83,6 +85,7 @@ func NewServer( copilotService: copilotService, provisioningService: provisioningService, validationService: validationService, + telemetryService: telemetryService, } } @@ -97,10 +100,12 @@ func (s *Server) Start() (*ServerInfo, error) { s.grpcServer = grpc.NewServer( grpc.ChainUnaryInterceptor( s.errorWrappingInterceptor(), + s.traceContextInterceptor(), s.tokenAuthInterceptor(&serverInfo), ), grpc.ChainStreamInterceptor( s.errorWrappingStreamInterceptor(), + s.traceContextStreamInterceptor(), s.tokenAuthStreamInterceptor(&serverInfo), ), ) @@ -132,6 +137,7 @@ func (s *Server) Start() (*ServerInfo, error) { azdext.RegisterCopilotServiceServer(s.grpcServer, s.copilotService) azdext.RegisterProvisioningServiceServer(s.grpcServer, s.provisioningService) azdext.RegisterValidationServiceServer(s.grpcServer, s.validationService) + azdext.RegisterTelemetryServiceServer(s.grpcServer, s.telemetryService) serverInfo.Address = fmt.Sprintf("127.0.0.1:%d", randomPort) serverInfo.Port = randomPort @@ -248,7 +254,7 @@ func (s *Server) tokenAuthStreamInterceptor(serverInfo *ServerInfo) grpc.StreamS } // Wrap the stream to inject validated claims into its context - wrappedStream := &authenticatedStream{ + wrappedStream := &contextStream{ ServerStream: ss, ctx: ctx, } @@ -257,13 +263,14 @@ func (s *Server) tokenAuthStreamInterceptor(serverInfo *ServerInfo) grpc.StreamS } } -// authenticatedStream wraps a grpc.ServerStream to provide a context with validated claims. -type authenticatedStream struct { +// contextStream wraps a grpc.ServerStream to override the context seen by the +// handler, for example to carry validated claims or trace context. +type contextStream struct { grpc.ServerStream ctx context.Context } -func (s *authenticatedStream) Context() context.Context { +func (s *contextStream) Context() context.Context { return s.ctx } diff --git a/cli/azd/internal/grpcserver/server_test.go b/cli/azd/internal/grpcserver/server_test.go index 5289e31344e..439c7481ca8 100644 --- a/cli/azd/internal/grpcserver/server_test.go +++ b/cli/azd/internal/grpcserver/server_test.go @@ -35,6 +35,24 @@ import ( // Test_Server_Start validates the start and stop flows of the gRPC server, // and confirms the expected behavior for authenticated and unauthenticated requests. func Test_Server_Start(t *testing.T) { + // The reporting extension is installed from the official registry and + // declares one bounded field, which is what the host validates against. + reportingExtension := &extensions.Extension{ + Id: "azd.internal.telemetry", + Version: "1.0.0", + Source: extensions.MainRegistryName, + Capabilities: []extensions.CapabilityType{ + extensions.TelemetryCapability, + }, + Telemetry: []extensions.TelemetryFieldDeclaration{ + { + Key: "ext.azd.internal.telemetry.deploy.mode", + AllowedValues: []string{"code", "container"}, + }, + }, + Namespace: "test", + } + server := NewServer( azdext.UnimplementedProjectServiceServer{}, azdext.UnimplementedEnvironmentServiceServer{}, @@ -53,6 +71,7 @@ func Test_Server_Start(t *testing.T) { azdext.UnimplementedCopilotServiceServer{}, azdext.UnimplementedProvisioningServiceServer{}, azdext.UnimplementedValidationServiceServer{}, + newTelemetryService(stubExtensionLookup{reportingExtension}), ) serverInfo, err := server.Start() @@ -119,6 +138,73 @@ func Test_Server_Start(t *testing.T) { require.True(t, ok) require.Equal(t, codes.Unauthenticated, st.Code()) }) + + t.Run("TelemetryAccepted", func(t *testing.T) { + accessToken, err := GenerateExtensionToken(reportingExtension, serverInfo) + require.NoError(t, err) + + ctx := azdext.WithAccessToken(t.Context(), accessToken) + client, err := azdext.NewAzdClient(azdext.WithAddress(serverInfo.Address)) + require.NoError(t, err) + + resp, err := client.Telemetry().ReportUsageAttribute(ctx, &azdext.ReportUsageAttributeRequest{ + Key: "ext.azd.internal.telemetry.deploy.mode", + Value: "code", + }) + require.NoError(t, err) + require.True(t, resp.Accepted) + }) + + t.Run("TelemetryUndeclaredValue", func(t *testing.T) { + accessToken, err := GenerateExtensionToken(reportingExtension, serverInfo) + require.NoError(t, err) + + ctx := azdext.WithAccessToken(t.Context(), accessToken) + client, err := azdext.NewAzdClient(azdext.WithAddress(serverInfo.Address)) + require.NoError(t, err) + + _, err = client.Telemetry().ReportUsageAttribute(ctx, &azdext.ReportUsageAttributeRequest{ + Key: "ext.azd.internal.telemetry.deploy.mode", + Value: "byo_image", + }) + st, ok := status.FromError(err) + require.True(t, ok) + require.Equal(t, codes.InvalidArgument, st.Code()) + }) + + t.Run("TelemetryMissingCapability", func(t *testing.T) { + // The base extension only declares CustomCommandCapability. + accessToken, err := GenerateExtensionToken(extension, serverInfo) + require.NoError(t, err) + + ctx := azdext.WithAccessToken(t.Context(), accessToken) + client, err := azdext.NewAzdClient(azdext.WithAddress(serverInfo.Address)) + require.NoError(t, err) + + _, err = client.Telemetry().ReportUsageAttribute(ctx, &azdext.ReportUsageAttributeRequest{ + Key: "ext.azd.internal.telemetry.deploy.mode", + Value: "code", + }) + st, ok := status.FromError(err) + require.True(t, ok) + require.Equal(t, codes.PermissionDenied, st.Code()) + }) + + t.Run("TelemetryMissingToken", func(t *testing.T) { + client, err := azdext.NewAzdClient(azdext.WithAddress(serverInfo.Address)) + require.NoError(t, err) + + _, err = client.Telemetry().ReportUsageAttribute( + t.Context(), + &azdext.ReportUsageAttributeRequest{ + Key: "ext.azd.internal.telemetry.deploy.mode", + Value: "code", + }, + ) + st, ok := status.FromError(err) + require.True(t, ok) + require.Equal(t, codes.Unauthenticated, st.Code()) + }) } // Test_Server_StreamInterceptor validates that the streaming RPC interceptor @@ -142,6 +228,7 @@ func Test_Server_StreamInterceptor(t *testing.T) { azdext.UnimplementedCopilotServiceServer{}, azdext.UnimplementedProvisioningServiceServer{}, azdext.UnimplementedValidationServiceServer{}, + azdext.UnimplementedTelemetryServiceServer{}, ) serverInfo, err := server.Start() @@ -398,11 +485,11 @@ func requireAuthErrorInfo(t *testing.T, st *status.Status) *errdetails.ErrorInfo return nil } -func TestAuthenticatedStream_Context(t *testing.T) { +func TestContextStream_Context(t *testing.T) { t.Parallel() ctx := context.WithValue(t.Context(), struct{ key string }{key: "test"}, "value") - stream := &authenticatedStream{ + stream := &contextStream{ ctx: ctx, } @@ -618,7 +705,7 @@ func TestValidateAuthToken_InvalidToken(t *testing.T) { func TestNewServer(t *testing.T) { t.Parallel() - s := NewServer(nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + s := NewServer(nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) require.NotNil(t, s) assert.Nil(t, s.grpcServer, "grpcServer should be nil before Start") } diff --git a/cli/azd/internal/grpcserver/telemetry_service.go b/cli/azd/internal/grpcserver/telemetry_service.go new file mode 100644 index 00000000000..7fda0442de1 --- /dev/null +++ b/cli/azd/internal/grpcserver/telemetry_service.go @@ -0,0 +1,124 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package grpcserver + +import ( + "context" + "slices" + "strings" + + "github.com/azure/azure-dev/cli/azd/internal/tracing" + "github.com/azure/azure-dev/cli/azd/internal/tracing/events" + "github.com/azure/azure-dev/cli/azd/internal/tracing/fields" + "github.com/azure/azure-dev/cli/azd/pkg/azdext" + "github.com/azure/azure-dev/cli/azd/pkg/extensions" + "go.opentelemetry.io/otel/attribute" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// installedExtensionLookup resolves the installed extension record for a +// signed extension id. *extensions.Manager satisfies it. +type installedExtensionLookup interface { + GetInstalled(options extensions.FilterOptions) (*extensions.Extension, error) +} + +// telemetryService implements azdext.TelemetryServiceServer. +type telemetryService struct { + azdext.UnimplementedTelemetryServiceServer + extensions installedExtensionLookup +} + +// NewTelemetryService creates the telemetry gRPC service. The extension +// manager supplies the telemetry declarations an extension published in the +// registry it was installed from; the host owns no product-specific fields. +// Returning the interface type lets the IoC container satisfy the +// azdext.TelemetryServiceServer parameter on NewServer without an adapter. +func NewTelemetryService(manager *extensions.Manager) azdext.TelemetryServiceServer { + return newTelemetryService(manager) +} + +func newTelemetryService(lookup installedExtensionLookup) *telemetryService { + return &telemetryService{extensions: lookup} +} + +// ReportUsageAttribute records one usage attribute value that the calling +// extension declared in the official azd registry. It fails closed: callers +// without validated claims, extensions installed from any other source, +// extensions without the telemetry capability, undeclared keys, and values +// outside the declared set are all rejected before anything is recorded. +// Rejected caller text is never echoed into the returned error. +func (s *telemetryService) ReportUsageAttribute( + ctx context.Context, + req *azdext.ReportUsageAttributeRequest, +) (*azdext.ReportUsageAttributeResponse, error) { + claims, err := extensions.GetClaimsFromContext(ctx) + if err != nil { + return nil, status.Error(codes.Unauthenticated, "validated extension claims are required") + } + + if req == nil || + req.Key == "" || req.Value == "" || + len(req.Key) > extensions.MaxTelemetryKeyLength || + len(req.Value) > extensions.MaxTelemetryValueLength { + return nil, status.Error(codes.InvalidArgument, "telemetry key and value are required") + } + + // Only extensions published through the official registry may report + // telemetry, because that registry is where declared fields are reviewed. + // Source is host-signed, so an extension cannot claim a better origin. + if !strings.EqualFold(claims.Source, extensions.MainRegistryName) { + return nil, status.Error(codes.PermissionDenied, + "telemetry requires an extension installed from the official registry") + } + + if !slices.Contains(claims.Capabilities, extensions.TelemetryCapability) { + return nil, status.Error(codes.PermissionDenied, "extension lacks the telemetry capability") + } + + extension, err := s.extensions.GetInstalled(extensions.FilterOptions{Id: claims.Subject}) + if err != nil { + return nil, status.Error(codes.PermissionDenied, "extension is not installed") + } + + // Re-validate the stored declaration. The installed record lives in user + // config, so shape is enforced again here rather than trusted from disk. + if issues := extensions.ValidateTelemetryDeclarations( + extension.Id, extension.Telemetry); len(issues) > 0 { + return nil, status.Error(codes.FailedPrecondition, "telemetry declarations are invalid") + } + + if !isDeclaredValue(extension.Telemetry, req.Key, req.Value) { + return nil, status.Error(codes.InvalidArgument, "telemetry value is not declared") + } + + // Record a dedicated span rather than augmenting the command span. The + // extension's trace context arrives over gRPC metadata, so this span + // shares the command's trace and joins on operation_Id downstream. + _, span := tracing.Start(ctx, events.ExtensionUsageEvent) + span.SetAttributes( + fields.ExtensionId.String(extension.Id), + fields.ExtensionVersion.String(extension.Version), + attribute.String(req.Key, req.Value), + ) + span.End() + + return &azdext.ReportUsageAttributeResponse{Accepted: true}, nil +} + +// isDeclaredValue reports whether key is declared and value is in that key's +// declared closed set. +func isDeclaredValue( + declarations []extensions.TelemetryFieldDeclaration, + key string, + value string, +) bool { + for _, declaration := range declarations { + if declaration.Key == key { + return slices.Contains(declaration.AllowedValues, value) + } + } + + return false +} diff --git a/cli/azd/internal/grpcserver/telemetry_service_test.go b/cli/azd/internal/grpcserver/telemetry_service_test.go new file mode 100644 index 00000000000..5abc7b91fa0 --- /dev/null +++ b/cli/azd/internal/grpcserver/telemetry_service_test.go @@ -0,0 +1,311 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package grpcserver + +import ( + "context" + "strings" + "testing" + + "github.com/azure/azure-dev/cli/azd/internal/tracing/events" + "github.com/azure/azure-dev/cli/azd/internal/tracing/fields" + "github.com/azure/azure-dev/cli/azd/pkg/azdext" + "github.com/azure/azure-dev/cli/azd/pkg/extensions" + "github.com/golang-jwt/jwt/v5" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/attribute" + tracesdk "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/trace" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +const testTelemetryKey = "ext.azd.internal.telemetry.deploy.mode" + +// stubExtensionLookup returns a fixed installed extension record, or a +// not-found error when the requested id does not match. +type stubExtensionLookup struct { + extension *extensions.Extension +} + +func (s stubExtensionLookup) GetInstalled( + options extensions.FilterOptions, +) (*extensions.Extension, error) { + if s.extension == nil || s.extension.Id != options.Id { + return nil, extensions.ErrInstalledExtensionNotFound + } + + return s.extension, nil +} + +func testExtension() *extensions.Extension { + return &extensions.Extension{ + Id: "azd.internal.telemetry", + Version: "1.0.0", + Source: extensions.MainRegistryName, + Capabilities: []extensions.CapabilityType{ + extensions.TelemetryCapability, + }, + Telemetry: []extensions.TelemetryFieldDeclaration{ + { + Key: testTelemetryKey, + AllowedValues: []string{"code", "container"}, + }, + }, + } +} + +// callWith runs the handler as the given extension, mirroring how the auth +// interceptor injects host-signed claims. +func callWith( + t *testing.T, + extension *extensions.Extension, + req *azdext.ReportUsageAttributeRequest, +) (*azdext.ReportUsageAttributeResponse, error) { + t.Helper() + + return callWithContext(t, t.Context(), extension, req) +} + +// callWithContext is callWith with a caller-supplied context, so a test can +// place the call inside an existing trace. +func callWithContext( + t *testing.T, + ctx context.Context, + extension *extensions.Extension, + req *azdext.ReportUsageAttributeRequest, +) (*azdext.ReportUsageAttributeResponse, error) { + t.Helper() + + service := newTelemetryService(stubExtensionLookup{extension}) + ctx = extensions.WithClaimsContext(ctx, &extensions.ExtensionClaims{ + RegisteredClaims: jwt.RegisteredClaims{Subject: extension.Id}, + Capabilities: extension.Capabilities, + Source: extension.Source, + }) + + return service.ReportUsageAttribute(ctx, req) +} + +// usageSpansIn returns the recorded ext.usage spans belonging to traceId. +// The recorder is shared by the whole package, so filtering on the trace +// started by a single test keeps it independent of every other test. +func usageSpansIn(traceId trace.TraceID) []tracesdk.ReadOnlySpan { + matched := []tracesdk.ReadOnlySpan{} + + for _, span := range testSpanRecorder.Ended() { + if span.Name() != events.ExtensionUsageEvent { + continue + } + + if span.SpanContext().TraceID() == traceId { + matched = append(matched, span) + } + } + + return matched +} + +func requireCode(t *testing.T, err error, expected codes.Code) { + t.Helper() + + st, ok := status.FromError(err) + require.True(t, ok) + require.Equal(t, expected, st.Code()) +} + +func Test_TelemetryService_AcceptsDeclaredValue(t *testing.T) { + t.Parallel() + + // Start a command span first so the usage span can be checked to share + // its trace, which is what makes the operation_Id join work downstream. + ctx, command := testTracerProvider.Tracer("test").Start(t.Context(), "cmd.deploy") + defer command.End() + + extension := testExtension() + resp, err := callWithContext(t, ctx, extension, &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "container", + }) + + require.NoError(t, err) + require.True(t, resp.Accepted) + + recorded := usageSpansIn(command.SpanContext().TraceID()) + require.Len(t, recorded, 1) + + // Spans also carry process-global attributes, so assert only the ones + // this feature owns rather than matching the whole set. + attributes := map[attribute.Key]attribute.Value{} + for _, attr := range recorded[0].Attributes() { + attributes[attr.Key] = attr.Value + } + + for _, expected := range []attribute.KeyValue{ + fields.ExtensionId.String(extension.Id), + fields.ExtensionVersion.String(extension.Version), + attribute.String(testTelemetryKey, "container"), + } { + require.Contains(t, attributes, expected.Key) + require.Equal(t, expected.Value, attributes[expected.Key]) + } +} + +func Test_TelemetryService_RecordsNoSpanWhenRejected(t *testing.T) { + t.Parallel() + + ctx, command := testTracerProvider.Tracer("test").Start(t.Context(), "cmd.deploy") + defer command.End() + + _, err := callWithContext(t, ctx, testExtension(), &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "byo_image", + }) + + requireCode(t, err, codes.InvalidArgument) + require.Empty(t, usageSpansIn(command.SpanContext().TraceID())) +} + +func Test_TelemetryService_RequiresClaims(t *testing.T) { + t.Parallel() + + service := newTelemetryService(stubExtensionLookup{testExtension()}) + _, err := service.ReportUsageAttribute(t.Context(), &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "code", + }) + + requireCode(t, err, codes.Unauthenticated) +} + +func Test_TelemetryService_RejectsInvalidRequests(t *testing.T) { + t.Parallel() + + tests := map[string]*azdext.ReportUsageAttributeRequest{ + "nil": nil, + "empty key": {Key: "", Value: "code"}, + "empty value": {Key: testTelemetryKey, Value: ""}, + "long key": { + Key: strings.Repeat("k", extensions.MaxTelemetryKeyLength+1), + Value: "code", + }, + "long value": { + Key: testTelemetryKey, + Value: strings.Repeat("v", extensions.MaxTelemetryValueLength+1), + }, + } + + for name, req := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, err := callWith(t, testExtension(), req) + requireCode(t, err, codes.InvalidArgument) + }) + } +} + +func Test_TelemetryService_RejectsUnofficialSource(t *testing.T) { + t.Parallel() + + extension := testExtension() + extension.Source = "dev" + + _, err := callWith(t, extension, &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "code", + }) + + requireCode(t, err, codes.PermissionDenied) +} + +func Test_TelemetryService_RejectsMissingSource(t *testing.T) { + t.Parallel() + + // An install predating source tracking must fail closed rather than be + // treated as coming from the official registry. + extension := testExtension() + extension.Source = "" + + _, err := callWith(t, extension, &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "code", + }) + + requireCode(t, err, codes.PermissionDenied) +} + +func Test_TelemetryService_RejectsMissingCapability(t *testing.T) { + t.Parallel() + + extension := testExtension() + extension.Capabilities = []extensions.CapabilityType{ + extensions.CustomCommandCapability, + } + + _, err := callWith(t, extension, &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "code", + }) + + requireCode(t, err, codes.PermissionDenied) +} + +func Test_TelemetryService_RejectsUninstalledExtension(t *testing.T) { + t.Parallel() + + service := newTelemetryService(stubExtensionLookup{}) + ctx := extensions.WithClaimsContext(t.Context(), &extensions.ExtensionClaims{ + RegisteredClaims: jwt.RegisteredClaims{Subject: "azd.internal.telemetry"}, + Capabilities: []extensions.CapabilityType{extensions.TelemetryCapability}, + Source: extensions.MainRegistryName, + }) + + _, err := service.ReportUsageAttribute(ctx, &azdext.ReportUsageAttributeRequest{ + Key: testTelemetryKey, + Value: "code", + }) + + requireCode(t, err, codes.PermissionDenied) +} + +func Test_TelemetryService_RejectsUndeclaredKeyOrValue(t *testing.T) { + t.Parallel() + + tests := map[string]*azdext.ReportUsageAttributeRequest{ + "undeclared key": {Key: "ext.azd.internal.telemetry.other", Value: "code"}, + "undeclared value": {Key: testTelemetryKey, Value: "byo_image"}, + "core key": {Key: "agent.deploy.mode", Value: "code"}, + } + + for name, req := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, err := callWith(t, testExtension(), req) + requireCode(t, err, codes.InvalidArgument) + }) + } +} + +func Test_TelemetryService_RejectsTamperedDeclaration(t *testing.T) { + t.Parallel() + + // The installed record lives in user config, so a locally widened + // declaration must be rejected instead of trusted. + extension := testExtension() + extension.Telemetry = []extensions.TelemetryFieldDeclaration{ + { + Key: "agent.deploy.mode", + AllowedValues: []string{"code"}, + }, + } + + _, err := callWith(t, extension, &azdext.ReportUsageAttributeRequest{ + Key: "agent.deploy.mode", + Value: "code", + }) + + requireCode(t, err, codes.FailedPrecondition) +} diff --git a/cli/azd/internal/grpcserver/trace_context.go b/cli/azd/internal/grpcserver/trace_context.go new file mode 100644 index 00000000000..b130b48fc15 --- /dev/null +++ b/cli/azd/internal/grpcserver/trace_context.go @@ -0,0 +1,70 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package grpcserver + +import ( + "context" + + "go.opentelemetry.io/otel/propagation" + "google.golang.org/grpc" + "google.golang.org/grpc/metadata" +) + +// W3C trace context metadata keys sent by the extension SDK. +const ( + traceparentHeader = "traceparent" + tracestateHeader = "tracestate" +) + +// traceContextInterceptor restores the caller's W3C trace context so spans the +// host records while serving an extension call join the command's trace +// instead of starting an unrelated root span. +func (s *Server) traceContextInterceptor() grpc.UnaryServerInterceptor { + return func( + ctx context.Context, + req any, + info *grpc.UnaryServerInfo, + handler grpc.UnaryHandler, + ) (any, error) { + return handler(withIncomingTraceContext(ctx), req) + } +} + +// traceContextStreamInterceptor is the streaming counterpart of +// traceContextInterceptor. +func (s *Server) traceContextStreamInterceptor() grpc.StreamServerInterceptor { + return func( + srv any, + ss grpc.ServerStream, + info *grpc.StreamServerInfo, + handler grpc.StreamHandler, + ) error { + return handler(srv, &contextStream{ + ServerStream: ss, + ctx: withIncomingTraceContext(ss.Context()), + }) + } +} + +// withIncomingTraceContext extracts W3C trace context from gRPC metadata. +// A missing or malformed traceparent leaves the context untouched, so calls +// from older extensions keep working. +func withIncomingTraceContext(ctx context.Context) context.Context { + md, ok := metadata.FromIncomingContext(ctx) + if !ok { + return ctx + } + + parent := md.Get(traceparentHeader) + if len(parent) == 0 || parent[0] == "" { + return ctx + } + + carrier := propagation.MapCarrier{traceparentHeader: parent[0]} + if state := md.Get(tracestateHeader); len(state) > 0 { + carrier[tracestateHeader] = state[0] + } + + return propagation.TraceContext{}.Extract(ctx, carrier) +} diff --git a/cli/azd/internal/grpcserver/trace_context_test.go b/cli/azd/internal/grpcserver/trace_context_test.go new file mode 100644 index 00000000000..7ed8644a90e --- /dev/null +++ b/cli/azd/internal/grpcserver/trace_context_test.go @@ -0,0 +1,64 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package grpcserver + +import ( + "testing" + + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/trace" + "google.golang.org/grpc/metadata" +) + +const ( + testTraceId = "68f1c4f4ef5346e69d7f196761d10c68" + testSpanId = "7fbdc197a52f4825" +) + +func testTraceparent() string { + return "00-" + testTraceId + "-" + testSpanId + "-01" +} + +func Test_WithIncomingTraceContext_RestoresCallerTrace(t *testing.T) { + t.Parallel() + + md := metadata.Pairs( + traceparentHeader, testTraceparent(), + tracestateHeader, "vendor=value", + ) + ctx := metadata.NewIncomingContext(t.Context(), md) + + spanContext := trace.SpanContextFromContext(withIncomingTraceContext(ctx)) + + require.True(t, spanContext.IsValid()) + require.Equal(t, testTraceId, spanContext.TraceID().String()) + require.Equal(t, testSpanId, spanContext.SpanID().String()) + require.Equal(t, "vendor=value", spanContext.TraceState().String()) +} + +func Test_WithIncomingTraceContext_LeavesContextAlone(t *testing.T) { + t.Parallel() + + tests := map[string]metadata.MD{ + "no metadata": nil, + "no traceparent": metadata.Pairs("authorization", "token"), + "empty traceparent": metadata.Pairs(traceparentHeader, ""), + "malformed": metadata.Pairs(traceparentHeader, "not-a-traceparent"), + "unsupported format": metadata.Pairs(traceparentHeader, "99-"+testTraceId), + } + + for name, md := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + ctx := t.Context() + if md != nil { + ctx = metadata.NewIncomingContext(ctx, md) + } + + require.False(t, trace.SpanContextFromContext( + withIncomingTraceContext(ctx)).IsValid()) + }) + } +} diff --git a/cli/azd/internal/tracing/events/events.go b/cli/azd/internal/tracing/events/events.go index 5e89fb7cbf7..8c36bb2059b 100644 --- a/cli/azd/internal/tracing/events/events.go +++ b/cli/azd/internal/tracing/events/events.go @@ -32,6 +32,10 @@ const ( ExtensionUpgradeEvent = "ext.upgrade" // ExtensionPromoteEvent tracks a registry promotion (e.g., dev → main). ExtensionPromoteEvent = "ext.promote" + // ExtensionUsageEvent carries one usage attribute an extension reported + // through the telemetry service. The attribute key and value come from + // the extension's registry declaration, validated by the host. + ExtensionUsageEvent = "ext.usage" ) // Copilot agent related events. diff --git a/cli/azd/pkg/azdext/azd_client.go b/cli/azd/pkg/azdext/azd_client.go index a2f6fb9031b..9d02d80141a 100644 --- a/cli/azd/pkg/azdext/azd_client.go +++ b/cli/azd/pkg/azdext/azd_client.go @@ -9,6 +9,7 @@ import ( "os" "strings" + "go.opentelemetry.io/otel/propagation" "google.golang.org/grpc" "google.golang.org/grpc/credentials" "google.golang.org/grpc/credentials/insecure" @@ -82,15 +83,47 @@ func isLocalhostAddress(address string) bool { return ip != nil && ip.IsLoopback() } -// WithAccessToken sets the access token for the `azd` client into a new Go context. +// WithAccessToken sets the access token for the `azd` client into a new Go +// context. It also forwards the W3C trace context so telemetry the host +// records while serving the call joins the azd command's trace instead of +// starting an unrelated one. func WithAccessToken(ctx context.Context, params ...string) context.Context { tokenValue := strings.Join(params, "") if tokenValue == "" { tokenValue = os.Getenv("AZD_ACCESS_TOKEN") } - md := metadata.Pairs("authorization", tokenValue) - return metadata.NewOutgoingContext(ctx, md) + pairs := append([]string{"authorization", tokenValue}, traceContextPairs(ctx)...) + + return metadata.NewOutgoingContext(ctx, metadata.Pairs(pairs...)) +} + +// traceContextPairs returns W3C trace context metadata pairs. A span already +// on ctx wins; otherwise the TRACEPARENT/TRACESTATE variables azd sets on the +// extension process are used, which is the common case because extensions +// build their context from scratch. +func traceContextPairs(ctx context.Context) []string { + carrier := propagation.MapCarrier{} + propagation.TraceContext{}.Inject(ctx, carrier) + + parent := carrier.Get(TraceparentKey) + state := carrier.Get(TracestateKey) + + if parent == "" { + parent = os.Getenv(TraceparentEnv) + state = os.Getenv(TracestateEnv) + } + + if parent == "" { + return nil + } + + pairs := []string{TraceparentKey, parent} + if state != "" { + pairs = append(pairs, TracestateKey, state) + } + + return pairs } // NewAzdClient creates a new `azd` client. @@ -262,3 +295,14 @@ func (c *AzdClient) Validation() ValidationServiceClient { return c.validationClient } + +// Telemetry returns the telemetry service client used to contribute +// host-validated command usage attributes. +// +// A fresh client is returned on each call rather than caching it on the +// AzdClient struct. Service target providers can deploy services concurrently, +// so an unsynchronized lazily-written cache field could race on first use. The +// generated client wrapper is cheap and shares the existing connection. +func (c *AzdClient) Telemetry() TelemetryServiceClient { + return NewTelemetryServiceClient(c.connection) +} diff --git a/cli/azd/pkg/azdext/azd_client_test.go b/cli/azd/pkg/azdext/azd_client_test.go index 06bd5c46981..21c3d9fa551 100644 --- a/cli/azd/pkg/azdext/azd_client_test.go +++ b/cli/azd/pkg/azdext/azd_client_test.go @@ -7,8 +7,65 @@ import ( "testing" "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel/trace" + "google.golang.org/grpc/metadata" ) +const ( + testTraceId = "68f1c4f4ef5346e69d7f196761d10c68" + testSpanId = "7fbdc197a52f4825" +) + +func testTraceparent() string { + return "00-" + testTraceId + "-" + testSpanId + "-01" +} + +func Test_WithAccessToken_PropagatesEnvTraceContext(t *testing.T) { + t.Setenv(TraceparentEnv, testTraceparent()) + t.Setenv(TracestateEnv, "vendor=value") + + md, ok := metadata.FromOutgoingContext(WithAccessToken(t.Context(), "token")) + + require.True(t, ok) + require.Equal(t, []string{"token"}, md.Get("authorization")) + require.Equal(t, []string{testTraceparent()}, md.Get(TraceparentKey)) + require.Equal(t, []string{"vendor=value"}, md.Get(TracestateKey)) +} + +func Test_WithAccessToken_PrefersContextSpan(t *testing.T) { + t.Setenv(TraceparentEnv, testTraceparent()) + + traceId, err := trace.TraceIDFromHex("11111111111111111111111111111111") + require.NoError(t, err) + spanId, err := trace.SpanIDFromHex("2222222222222222") + require.NoError(t, err) + + ctx := trace.ContextWithSpanContext(t.Context(), trace.NewSpanContext(trace.SpanContextConfig{ + TraceID: traceId, + SpanID: spanId, + TraceFlags: trace.FlagsSampled, + })) + + md, ok := metadata.FromOutgoingContext(WithAccessToken(ctx, "token")) + + require.True(t, ok) + require.Equal(t, []string{ + "00-11111111111111111111111111111111-2222222222222222-01", + }, md.Get(TraceparentKey)) +} + +func Test_WithAccessToken_OmitsMissingTraceContext(t *testing.T) { + t.Setenv(TraceparentEnv, "") + t.Setenv(TracestateEnv, "") + + md, ok := metadata.FromOutgoingContext(WithAccessToken(t.Context(), "token")) + + require.True(t, ok) + require.Equal(t, []string{"token"}, md.Get("authorization")) + require.Empty(t, md.Get(TraceparentKey)) + require.Empty(t, md.Get(TracestateKey)) +} + func Test_IsLocalhostAddress(t *testing.T) { tests := []struct { name string diff --git a/cli/azd/pkg/azdext/telemetry.pb.go b/cli/azd/pkg/azdext/telemetry.pb.go new file mode 100644 index 00000000000..b6f655f1c60 --- /dev/null +++ b/cli/azd/pkg/azdext/telemetry.pb.go @@ -0,0 +1,188 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.36.10 +// protoc v7.35.0 +// source: telemetry.proto + +package azdext + +import ( + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" + unsafe "unsafe" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +type ReportUsageAttributeRequest struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Key must match a telemetry field the extension declared in the registry. + Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` + // Value must be in the allowed set the extension declared for key. + Value string `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ReportUsageAttributeRequest) Reset() { + *x = ReportUsageAttributeRequest{} + mi := &file_telemetry_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ReportUsageAttributeRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ReportUsageAttributeRequest) ProtoMessage() {} + +func (x *ReportUsageAttributeRequest) ProtoReflect() protoreflect.Message { + mi := &file_telemetry_proto_msgTypes[0] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ReportUsageAttributeRequest.ProtoReflect.Descriptor instead. +func (*ReportUsageAttributeRequest) Descriptor() ([]byte, []int) { + return file_telemetry_proto_rawDescGZIP(), []int{0} +} + +func (x *ReportUsageAttributeRequest) GetKey() string { + if x != nil { + return x.Key + } + return "" +} + +func (x *ReportUsageAttributeRequest) GetValue() string { + if x != nil { + return x.Value + } + return "" +} + +type ReportUsageAttributeResponse struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Accepted reports whether the host recorded the value. + Accepted bool `protobuf:"varint,1,opt,name=accepted,proto3" json:"accepted,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *ReportUsageAttributeResponse) Reset() { + *x = ReportUsageAttributeResponse{} + mi := &file_telemetry_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *ReportUsageAttributeResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ReportUsageAttributeResponse) ProtoMessage() {} + +func (x *ReportUsageAttributeResponse) ProtoReflect() protoreflect.Message { + mi := &file_telemetry_proto_msgTypes[1] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ReportUsageAttributeResponse.ProtoReflect.Descriptor instead. +func (*ReportUsageAttributeResponse) Descriptor() ([]byte, []int) { + return file_telemetry_proto_rawDescGZIP(), []int{1} +} + +func (x *ReportUsageAttributeResponse) GetAccepted() bool { + if x != nil { + return x.Accepted + } + return false +} + +var File_telemetry_proto protoreflect.FileDescriptor + +const file_telemetry_proto_rawDesc = "" + + "\n" + + "\x0ftelemetry.proto\x12\x06azdext\"E\n" + + "\x1bReportUsageAttributeRequest\x12\x10\n" + + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + + "\x05value\x18\x02 \x01(\tR\x05value\":\n" + + "\x1cReportUsageAttributeResponse\x12\x1a\n" + + "\baccepted\x18\x01 \x01(\bR\baccepted2u\n" + + "\x10TelemetryService\x12a\n" + + "\x14ReportUsageAttribute\x12#.azdext.ReportUsageAttributeRequest\x1a$.azdext.ReportUsageAttributeResponseB/Z-github.com/azure/azure-dev/cli/azd/pkg/azdextb\x06proto3" + +var ( + file_telemetry_proto_rawDescOnce sync.Once + file_telemetry_proto_rawDescData []byte +) + +func file_telemetry_proto_rawDescGZIP() []byte { + file_telemetry_proto_rawDescOnce.Do(func() { + file_telemetry_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_telemetry_proto_rawDesc), len(file_telemetry_proto_rawDesc))) + }) + return file_telemetry_proto_rawDescData +} + +var file_telemetry_proto_msgTypes = make([]protoimpl.MessageInfo, 2) +var file_telemetry_proto_goTypes = []any{ + (*ReportUsageAttributeRequest)(nil), // 0: azdext.ReportUsageAttributeRequest + (*ReportUsageAttributeResponse)(nil), // 1: azdext.ReportUsageAttributeResponse +} +var file_telemetry_proto_depIdxs = []int32{ + 0, // 0: azdext.TelemetryService.ReportUsageAttribute:input_type -> azdext.ReportUsageAttributeRequest + 1, // 1: azdext.TelemetryService.ReportUsageAttribute:output_type -> azdext.ReportUsageAttributeResponse + 1, // [1:2] is the sub-list for method output_type + 0, // [0:1] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_telemetry_proto_init() } +func file_telemetry_proto_init() { + if File_telemetry_proto != nil { + return + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: unsafe.Slice(unsafe.StringData(file_telemetry_proto_rawDesc), len(file_telemetry_proto_rawDesc)), + NumEnums: 0, + NumMessages: 2, + NumExtensions: 0, + NumServices: 1, + }, + GoTypes: file_telemetry_proto_goTypes, + DependencyIndexes: file_telemetry_proto_depIdxs, + MessageInfos: file_telemetry_proto_msgTypes, + }.Build() + File_telemetry_proto = out.File + file_telemetry_proto_goTypes = nil + file_telemetry_proto_depIdxs = nil +} diff --git a/cli/azd/pkg/azdext/telemetry_grpc.pb.go b/cli/azd/pkg/azdext/telemetry_grpc.pb.go new file mode 100644 index 00000000000..75851425422 --- /dev/null +++ b/cli/azd/pkg/azdext/telemetry_grpc.pb.go @@ -0,0 +1,138 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +// Code generated by protoc-gen-go-grpc. DO NOT EDIT. +// versions: +// - protoc-gen-go-grpc v1.5.1 +// - protoc v7.35.0 +// source: telemetry.proto + +package azdext + +import ( + context "context" + grpc "google.golang.org/grpc" + codes "google.golang.org/grpc/codes" + status "google.golang.org/grpc/status" +) + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the grpc package it is being compiled against. +// Requires gRPC-Go v1.64.0 or later. +const _ = grpc.SupportPackageIsVersion9 + +const ( + TelemetryService_ReportUsageAttribute_FullMethodName = "/azdext.TelemetryService/ReportUsageAttribute" +) + +// TelemetryServiceClient is the client API for TelemetryService service. +// +// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. +// +// TelemetryService accepts usage attribute values from authenticated +// extensions. Every key and value must appear in the telemetry declaration +// the extension published in the official azd registry, which is where those +// fields are reviewed. The host decides which telemetry span a value lands on +// and owns classification, purpose, hashing, and aggregation. +type TelemetryServiceClient interface { + // ReportUsageAttribute reports one declared usage attribute value. + ReportUsageAttribute(ctx context.Context, in *ReportUsageAttributeRequest, opts ...grpc.CallOption) (*ReportUsageAttributeResponse, error) +} + +type telemetryServiceClient struct { + cc grpc.ClientConnInterface +} + +func NewTelemetryServiceClient(cc grpc.ClientConnInterface) TelemetryServiceClient { + return &telemetryServiceClient{cc} +} + +func (c *telemetryServiceClient) ReportUsageAttribute(ctx context.Context, in *ReportUsageAttributeRequest, opts ...grpc.CallOption) (*ReportUsageAttributeResponse, error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + out := new(ReportUsageAttributeResponse) + err := c.cc.Invoke(ctx, TelemetryService_ReportUsageAttribute_FullMethodName, in, out, cOpts...) + if err != nil { + return nil, err + } + return out, nil +} + +// TelemetryServiceServer is the server API for TelemetryService service. +// All implementations must embed UnimplementedTelemetryServiceServer +// for forward compatibility. +// +// TelemetryService accepts usage attribute values from authenticated +// extensions. Every key and value must appear in the telemetry declaration +// the extension published in the official azd registry, which is where those +// fields are reviewed. The host decides which telemetry span a value lands on +// and owns classification, purpose, hashing, and aggregation. +type TelemetryServiceServer interface { + // ReportUsageAttribute reports one declared usage attribute value. + ReportUsageAttribute(context.Context, *ReportUsageAttributeRequest) (*ReportUsageAttributeResponse, error) + mustEmbedUnimplementedTelemetryServiceServer() +} + +// UnimplementedTelemetryServiceServer must be embedded to have +// forward compatible implementations. +// +// NOTE: this should be embedded by value instead of pointer to avoid a nil +// pointer dereference when methods are called. +type UnimplementedTelemetryServiceServer struct{} + +func (UnimplementedTelemetryServiceServer) ReportUsageAttribute(context.Context, *ReportUsageAttributeRequest) (*ReportUsageAttributeResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ReportUsageAttribute not implemented") +} +func (UnimplementedTelemetryServiceServer) mustEmbedUnimplementedTelemetryServiceServer() {} +func (UnimplementedTelemetryServiceServer) testEmbeddedByValue() {} + +// UnsafeTelemetryServiceServer may be embedded to opt out of forward compatibility for this service. +// Use of this interface is not recommended, as added methods to TelemetryServiceServer will +// result in compilation errors. +type UnsafeTelemetryServiceServer interface { + mustEmbedUnimplementedTelemetryServiceServer() +} + +func RegisterTelemetryServiceServer(s grpc.ServiceRegistrar, srv TelemetryServiceServer) { + // If the following call pancis, it indicates UnimplementedTelemetryServiceServer was + // embedded by pointer and is nil. This will cause panics if an + // unimplemented method is ever invoked, so we test this at initialization + // time to prevent it from happening at runtime later due to I/O. + if t, ok := srv.(interface{ testEmbeddedByValue() }); ok { + t.testEmbeddedByValue() + } + s.RegisterService(&TelemetryService_ServiceDesc, srv) +} + +func _TelemetryService_ReportUsageAttribute_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ReportUsageAttributeRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(TelemetryServiceServer).ReportUsageAttribute(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: TelemetryService_ReportUsageAttribute_FullMethodName, + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(TelemetryServiceServer).ReportUsageAttribute(ctx, req.(*ReportUsageAttributeRequest)) + } + return interceptor(ctx, in, info, handler) +} + +// TelemetryService_ServiceDesc is the grpc.ServiceDesc for TelemetryService service. +// It's only intended for direct use with grpc.RegisterService, +// and not to be introspected or modified (even as a copy) +var TelemetryService_ServiceDesc = grpc.ServiceDesc{ + ServiceName: "azdext.TelemetryService", + HandlerType: (*TelemetryServiceServer)(nil), + Methods: []grpc.MethodDesc{ + { + MethodName: "ReportUsageAttribute", + Handler: _TelemetryService_ReportUsageAttribute_Handler, + }, + }, + Streams: []grpc.StreamDesc{}, + Metadata: "telemetry.proto", +} diff --git a/cli/azd/pkg/extensions/claims.go b/cli/azd/pkg/extensions/claims.go index 1d302388d8a..2e0b69d5106 100644 --- a/cli/azd/pkg/extensions/claims.go +++ b/cli/azd/pkg/extensions/claims.go @@ -14,6 +14,9 @@ import ( type ExtensionClaims struct { jwt.RegisteredClaims Capabilities []CapabilityType `json:"cap,omitempty"` + // Source is the registry the extension was installed from. It is signed by + // the host so an extension cannot claim a more trusted origin than it has. + Source string `json:"src,omitempty"` } // extensionClaimsKeyType is the context key for storing validated extension claims. diff --git a/cli/azd/pkg/extensions/extension.go b/cli/azd/pkg/extensions/extension.go index 4fa288bf957..19113ff24af 100644 --- a/cli/azd/pkg/extensions/extension.go +++ b/cli/azd/pkg/extensions/extension.go @@ -27,6 +27,9 @@ type Extension struct { Providers []Provider `json:"providers,omitempty"` McpConfig *McpConfig `json:"mcp,omitempty"` LastUpdateWarning string `json:"lastUpdateWarning,omitempty"` + // Telemetry are the usage attributes this version declared in the registry + // it was installed from. azd validates every reported value against it. + Telemetry []TelemetryFieldDeclaration `json:"telemetry,omitempty"` stdin *bytes.Buffer stdout *output.DynamicMultiWriter diff --git a/cli/azd/pkg/extensions/manager.go b/cli/azd/pkg/extensions/manager.go index 65fae091afc..0c799e32358 100644 --- a/cli/azd/pkg/extensions/manager.go +++ b/cli/azd/pkg/extensions/manager.go @@ -752,6 +752,7 @@ func (m *Manager) installInternal( Source: extension.Source, Providers: selectedVersion.Providers, McpConfig: selectedVersion.McpConfig, + Telemetry: selectedVersion.Telemetry, } if err := m.userConfig.Set(installedConfigKey, extensions); err != nil { diff --git a/cli/azd/pkg/extensions/registry.go b/cli/azd/pkg/extensions/registry.go index 18427236f68..bc9b9d96042 100644 --- a/cli/azd/pkg/extensions/registry.go +++ b/cli/azd/pkg/extensions/registry.go @@ -57,6 +57,10 @@ const ( // Validation provider enables extensions to contribute validation checks // to azd's validation pipeline (e.g. provision checks during provisioning) ValidationProviderCapability CapabilityType = "validation-provider" + // Telemetry enables extensions to report the usage attributes they declare + // in the official registry. azd validates every reported key and value + // against that declaration. + TelemetryCapability CapabilityType = "telemetry" ) type ProviderType string @@ -139,6 +143,9 @@ type ExtensionVersion struct { EntryPoint string `json:"entryPoint,omitempty"` // McpConfig is the MCP server configuration for this extension version McpConfig *McpConfig `json:"mcp,omitempty"` + // Telemetry declares the usage attributes this version may report through + // the telemetry service. Requires the telemetry capability. + Telemetry []TelemetryFieldDeclaration `json:"telemetry,omitempty"` } // ExtensionArtifact represents the artifact information of an extension diff --git a/cli/azd/pkg/extensions/telemetry_declaration.go b/cli/azd/pkg/extensions/telemetry_declaration.go new file mode 100644 index 00000000000..411af698522 --- /dev/null +++ b/cli/azd/pkg/extensions/telemetry_declaration.go @@ -0,0 +1,163 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package extensions + +import ( + "fmt" + "regexp" + "strings" +) + +// Bounds for extension-declared telemetry fields. They keep registry entries +// small and are re-checked at runtime so a tampered local install cannot +// widen them. +const ( + // MaxTelemetryFields is the number of fields one extension version may + // declare. + MaxTelemetryFields = 16 + // MaxTelemetryKeyLength bounds a declared attribute key. + MaxTelemetryKeyLength = 128 + // MaxTelemetryAllowedValues bounds the closed value set of one field. + MaxTelemetryAllowedValues = 32 + // MaxTelemetryValueLength bounds a single declared value. + MaxTelemetryValueLength = 64 +) + +// TelemetryKeyPrefix namespaces every extension-declared telemetry key so +// extension values can never collide with, or masquerade as, a core field. +const TelemetryKeyPrefix = "ext." + +// telemetryKeySegmentRegex matches one dot-separated segment of a key. +var telemetryKeySegmentRegex = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?$`) + +// telemetryValueRegex bounds declared values to a conservative charset. It +// deliberately excludes ':' and '/' so a declaration cannot smuggle registry +// names, URLs, or paths into a value. +var telemetryValueRegex = regexp.MustCompile(`^[a-z0-9]([a-z0-9_.-]*[a-z0-9])?$`) + +// TelemetryFieldDeclaration declares one usage attribute that an extension +// version may report at runtime. The registry entry is the source of truth, +// so azd core carries no product-specific telemetry semantics: adding a field +// is a registry change, not a core release. +type TelemetryFieldDeclaration struct { + // Key is the telemetry attribute name. It must be "ext." followed by the + // declaring extension id and at least one more dot-separated segment. + Key string `json:"key"` + // AllowedValues is the closed set of values the extension may report for + // Key. Free-form values are never accepted. + AllowedValues []string `json:"allowedValues"` +} + +// TelemetryKeyPrefixFor returns the key prefix an extension must use for +// every telemetry field it declares. +func TelemetryKeyPrefixFor(extensionId string) string { + return TelemetryKeyPrefix + extensionId + "." +} + +// ValidateTelemetryDeclarations reports every way declarations violate the +// host telemetry contract: namespaced keys, bounded closed value sets, and a +// conservative charset. It runs at publish time against the registry and +// again at runtime against the installed record, so it must stay free of any +// product-specific knowledge. +func ValidateTelemetryDeclarations( + extensionId string, + declarations []TelemetryFieldDeclaration, +) []string { + if len(declarations) == 0 { + return nil + } + + var issues []string + + if len(declarations) > MaxTelemetryFields { + issues = append(issues, fmt.Sprintf( + "telemetry declares %d fields (max %d)", len(declarations), MaxTelemetryFields)) + } + + prefix := TelemetryKeyPrefixFor(extensionId) + seenKeys := map[string]struct{}{} + + for i, declaration := range declarations { + at := fmt.Sprintf("telemetry[%d]", i) + + if _, duplicate := seenKeys[declaration.Key]; duplicate { + issues = append(issues, fmt.Sprintf("%s: duplicate key '%s'", at, declaration.Key)) + } + seenKeys[declaration.Key] = struct{}{} + + issues = append(issues, validateTelemetryKey(at, prefix, declaration.Key)...) + issues = append(issues, validateTelemetryValues(at, declaration.AllowedValues)...) + } + + return issues +} + +func validateTelemetryKey(at string, prefix string, key string) []string { + if key == "" { + return []string{fmt.Sprintf("%s: missing required field 'key'", at)} + } + + if len(key) > MaxTelemetryKeyLength { + return []string{fmt.Sprintf( + "%s: key exceeds %d characters", at, MaxTelemetryKeyLength)} + } + + if !strings.HasPrefix(key, prefix) { + return []string{fmt.Sprintf( + "%s: key '%s' must start with '%s'", at, key, prefix)} + } + + suffix := strings.TrimPrefix(key, prefix) + if suffix == "" { + return []string{fmt.Sprintf( + "%s: key '%s' needs at least one segment after '%s'", at, key, prefix)} + } + + for segment := range strings.SplitSeq(suffix, ".") { + if !telemetryKeySegmentRegex.MatchString(segment) { + return []string{fmt.Sprintf( + "%s: key '%s' has invalid segment '%s' "+ + "(lowercase alphanumeric and hyphens)", at, key, segment)} + } + } + + return nil +} + +func validateTelemetryValues(at string, values []string) []string { + if len(values) == 0 { + return []string{fmt.Sprintf("%s: allowedValues must declare at least one value", at)} + } + + var issues []string + + if len(values) > MaxTelemetryAllowedValues { + issues = append(issues, fmt.Sprintf( + "%s: declares %d allowed values (max %d)", + at, len(values), MaxTelemetryAllowedValues)) + } + + seen := map[string]struct{}{} + + for _, value := range values { + if _, duplicate := seen[value]; duplicate { + issues = append(issues, fmt.Sprintf("%s: duplicate allowed value '%s'", at, value)) + } + seen[value] = struct{}{} + + if len(value) > MaxTelemetryValueLength { + issues = append(issues, fmt.Sprintf( + "%s: allowed value exceeds %d characters", at, MaxTelemetryValueLength)) + continue + } + + if !telemetryValueRegex.MatchString(value) { + issues = append(issues, fmt.Sprintf( + "%s: invalid allowed value '%s' "+ + "(lowercase alphanumeric, '_', '-', and '.')", at, value)) + } + } + + return issues +} diff --git a/cli/azd/pkg/extensions/telemetry_declaration_test.go b/cli/azd/pkg/extensions/telemetry_declaration_test.go new file mode 100644 index 00000000000..0b25cbc72f8 --- /dev/null +++ b/cli/azd/pkg/extensions/telemetry_declaration_test.go @@ -0,0 +1,198 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +package extensions + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +const testExtensionId = "azure.ai.agents" + +func Test_ValidateTelemetryDeclarations_Valid(t *testing.T) { + t.Parallel() + + issues := ValidateTelemetryDeclarations(testExtensionId, []TelemetryFieldDeclaration{ + { + Key: "ext.azure.ai.agents.deploy.mode", + AllowedValues: []string{"code", "container", "byo_image"}, + }, + { + Key: "ext.azure.ai.agents.deploy-target", + AllowedValues: []string{"aca"}, + }, + }) + + require.Empty(t, issues) +} + +func Test_ValidateTelemetryDeclarations_Empty(t *testing.T) { + t.Parallel() + + require.Empty(t, ValidateTelemetryDeclarations(testExtensionId, nil)) +} + +func Test_ValidateTelemetryDeclarations_RejectsBadKeys(t *testing.T) { + t.Parallel() + + tests := map[string]string{ + "missing key": "", + "core namespace": "agent.deploy.mode", + "other extension": "ext.contoso.tools.deploy.mode", + "prefix only": "ext.azure.ai.agents.", + "uppercase segment": "ext.azure.ai.agents.Mode", + "empty segment": "ext.azure.ai.agents..mode", + "too long": "ext.azure.ai.agents." + strings.Repeat("m", MaxTelemetryKeyLength), + } + + for name, key := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + issues := ValidateTelemetryDeclarations(testExtensionId, []TelemetryFieldDeclaration{ + {Key: key, AllowedValues: []string{"code"}}, + }) + + require.NotEmpty(t, issues) + }) + } +} + +func Test_ValidateTelemetryDeclarations_RejectsBadValues(t *testing.T) { + t.Parallel() + + tests := map[string][]string{ + "no values": {}, + "empty value": {""}, + "registry name": {"byo_image:myregistry.azurecr.io/foo"}, + "path": {"images/foo"}, + "uppercase": {"Code"}, + "duplicate": {"code", "code"}, + "too long": {strings.Repeat("v", MaxTelemetryValueLength+1)}, + "too many entries": make([]string, MaxTelemetryAllowedValues+1), + } + + for name, values := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + issues := ValidateTelemetryDeclarations(testExtensionId, []TelemetryFieldDeclaration{ + {Key: "ext.azure.ai.agents.deploy.mode", AllowedValues: values}, + }) + + require.NotEmpty(t, issues) + }) + } +} + +func Test_ValidateTelemetryDeclarations_RejectsOversizedSets(t *testing.T) { + t.Parallel() + + declarations := make([]TelemetryFieldDeclaration, 0, MaxTelemetryFields+1) + for i := range MaxTelemetryFields + 1 { + declarations = append(declarations, TelemetryFieldDeclaration{ + Key: "ext.azure.ai.agents.field" + string(rune('a'+i)), + AllowedValues: []string{"code"}, + }) + } + + require.NotEmpty(t, ValidateTelemetryDeclarations(testExtensionId, declarations)) +} + +func Test_ValidateTelemetryDeclarations_RejectsDuplicateKeys(t *testing.T) { + t.Parallel() + + issues := ValidateTelemetryDeclarations(testExtensionId, []TelemetryFieldDeclaration{ + {Key: "ext.azure.ai.agents.deploy.mode", AllowedValues: []string{"code"}}, + {Key: "ext.azure.ai.agents.deploy.mode", AllowedValues: []string{"container"}}, + }) + + require.NotEmpty(t, issues) +} + +func Test_ValidateRegistry_TelemetryRequiresCapability(t *testing.T) { + t.Parallel() + + metadata := []*ExtensionMetadata{ + { + Id: testExtensionId, + DisplayName: "Agents", + Description: "Test extension", + Versions: []ExtensionVersion{ + { + Version: "1.0.0", + Telemetry: []TelemetryFieldDeclaration{ + { + Key: "ext.azure.ai.agents.deploy.mode", + AllowedValues: []string{"code"}, + }, + }, + Artifacts: map[string]ExtensionArtifact{ + "linux/amd64": { + URL: "https://example.com/agents.zip", + Checksum: ExtensionChecksum{Algorithm: "sha256", Value: "abc"}, + }, + }, + }, + }, + }, + } + + result := ValidateExtensions(metadata, false) + + require.False(t, result.Valid) + require.Contains(t, strings.Join(collectMessages(result), "\n"), "telemetry") +} + +func Test_ValidateRegistry_TelemetryValidWithCapability(t *testing.T) { + t.Parallel() + + metadata := []*ExtensionMetadata{ + { + Id: testExtensionId, + DisplayName: "Agents", + Description: "Test extension", + Versions: []ExtensionVersion{ + { + Version: "1.0.0", + Capabilities: []CapabilityType{TelemetryCapability}, + Telemetry: []TelemetryFieldDeclaration{ + { + Key: "ext.azure.ai.agents.deploy.mode", + AllowedValues: []string{"code"}, + }, + }, + Artifacts: map[string]ExtensionArtifact{ + "linux/amd64": { + URL: "https://example.com/agents.zip", + Checksum: ExtensionChecksum{Algorithm: "sha256", Value: "abc"}, + }, + }, + }, + }, + }, + } + + result := ValidateExtensions(metadata, false) + + require.True(t, result.Valid, strings.Join(collectMessages(result), "\n")) +} + +func collectMessages(result *RegistryValidationResult) []string { + messages := []string{} + + for _, issue := range result.Issues { + messages = append(messages, issue.Message) + } + + for _, extension := range result.Extensions { + for _, issue := range extension.Issues { + messages = append(messages, issue.Message) + } + } + + return messages +} diff --git a/cli/azd/pkg/extensions/validate_registry.go b/cli/azd/pkg/extensions/validate_registry.go index ed00edda141..985f129f7d8 100644 --- a/cli/azd/pkg/extensions/validate_registry.go +++ b/cli/azd/pkg/extensions/validate_registry.go @@ -32,6 +32,7 @@ var ValidCapabilities = []CapabilityType{ MetadataCapability, ProvisioningProviderCapability, ValidationProviderCapability, + TelemetryCapability, } // validChecksumAlgorithms defines the supported checksum algorithms. @@ -218,7 +219,7 @@ func validateExtension(ext *ExtensionMetadata, strict bool) ExtensionValidationR // Validate each version for i, ver := range ext.Versions { - validateVersion(&result, i, &ver, strict) + validateVersion(&result, ext.Id, i, &ver, strict) } // Find latest version using semver ordering @@ -262,7 +263,13 @@ func findLatestVersion(versions []ExtensionVersion) *ExtensionVersion { } // validateVersion validates a single version entry within an extension. -func validateVersion(result *ExtensionValidationResult, index int, ver *ExtensionVersion, strict bool) { +func validateVersion( + result *ExtensionValidationResult, + extensionId string, + index int, + ver *ExtensionVersion, + strict bool, +) { prefix := fmt.Sprintf("versions[%d]", index) // Validate semver format using the semver package @@ -281,6 +288,17 @@ func validateVersion(result *ExtensionValidationResult, index int, ver *Extensio } } + // Validate telemetry declarations. Every field an extension may report at + // runtime is declared here and reviewed as part of the registry change. + for _, issue := range ValidateTelemetryDeclarations(extensionId, ver.Telemetry) { + result.addError(fmt.Sprintf("%s: %s", prefix, issue)) + } + + if len(ver.Telemetry) > 0 && !slices.Contains(ver.Capabilities, TelemetryCapability) { + result.addError(fmt.Sprintf("%s: telemetry declarations require the '%s' capability", + prefix, TelemetryCapability)) + } + // Enforce that each version has at least one artifact or dependency hasArtifacts := len(ver.Artifacts) > 0 hasDependencies := len(ver.Dependencies) > 0 diff --git a/docs/README.md b/docs/README.md index a36a0eff4fe..29a470ac2be 100644 --- a/docs/README.md +++ b/docs/README.md @@ -43,6 +43,7 @@ System overviews, design context, and decision records. - [Extension Framework](architecture/extension-framework.md) — gRPC-based extension system architecture - [Provisioning Pipeline](architecture/provisioning-pipeline.md) — How infrastructure provisioning works - [Telemetry Architecture](architecture/telemetry.md) — How azd collects and exports telemetry +- [ADR-001: Extension Telemetry Declarations](architecture/adr-001-extension-telemetry-declarations.md) — Why extension telemetry fields live in the registry - [ADR Template](architecture/adr-template.md) — Template for lightweight architecture decision records --- diff --git a/docs/architecture/adr-001-extension-telemetry-declarations.md b/docs/architecture/adr-001-extension-telemetry-declarations.md new file mode 100644 index 00000000000..381c4a9b3d9 --- /dev/null +++ b/docs/architecture/adr-001-extension-telemetry-declarations.md @@ -0,0 +1,99 @@ +# ADR-001: Extension telemetry is declared in the registry, not in azd core + +**Status:** Proposed + +**Date:** 2026-07-27 + +## Context + +Extensions need to report a small number of bounded usage signals so the team +can answer product questions like which deployment mode people actually pick. +Two things made this awkward: + +1. The first design put an allowlist of product fields (key, allowed values, + eligible commands, required capability) inside `azd` core. That means every + new field an extension wants is a core PR, a core release, and a wait for + users to upgrade `azd` before any data arrives. It also puts product + semantics from one Foundry product into the CLI that hosts all of them. +2. Extensions cannot simply be trusted to send arbitrary telemetry. The values + need a privacy review, and a bounded set is what makes that review possible. + +The constraint is therefore: reviewable and bounded, but not on the core +release path, and with no product-specific knowledge in core. + +## Decision + +**Field declarations move to the extension registry entry; `azd` core only +enforces them.** + +- `ExtensionVersion` gains `telemetry: [{ key, allowedValues }]`, following the + same "registry declares, core enforces" pattern already used for + `capabilities`, `providers`, and `mcp`. +- Publish-time validation (`ValidateTelemetryDeclarations`) enforces the shape: + keys namespaced as `ext..`, a non-empty closed value + set, and a charset that excludes `:` and `/` so registry names, URLs, and + paths cannot be smuggled through a value. Bounds cap the number of fields, + values, and lengths. +- A dedicated `telemetry` capability gates the feature, and the extension's + install `Source` is carried in the host-signed JWT claims. Only extensions + installed from the official `azd` registry are allowed to report. +- At runtime the host re-validates the stored declaration before use, because + the installed record lives in user-writable config. +- Accepted values are recorded on a new `ext.usage` span rather than being + appended to the command span. +- The RPC is named `ReportUsageAttribute`, deliberately free of any span + wording, so the host can change where values land without a breaking SDK + change. + +The privacy gate does not disappear: the official registry lives in this +repository, so adding a field is still a reviewed PR by the same people. It +just no longer occupies a core release slot. + +## Consequences + +**Easier** + +- Adding a telemetry field is a registry PR. Users get it by upgrading the + extension; `azd` core does not ship. +- `azd` core contains zero product semantics for extension telemetry. The + schema documents one class of field rather than one row per product concept. +- The trust boundary is explicit: the capability and the install `Source` are + carried in host-signed claims, so an extension cannot assert either of them + on the request itself. + +**More difficult** + +- Reviewers must read registry changes as telemetry changes. This needs to be + part of the registry review checklist, not just tribal knowledge. +- The new span does not sit on the same row as the command in App Insights. + Queries must join on `operation_Id`. This is reliable because `azd` does not + sample and the exporter writes the trace ID to `operation_Id`. +- Trace context has to cross the gRPC boundary for that join to work, so the + extension SDK now forwards the W3C trace context headers and the server + extracts them. +- Those claims are signed from the installed record, which lives in + user-writable config. This is the same trust level every existing capability + gate already depends on, so the feature does not weaken it, but it is not + proof of provenance. Hardening the installed record is tracked separately. + +## Alternatives Considered + +**Keep the allowlist in core.** Simplest to review, and it keeps every value in +one Go file. Rejected because it puts one product's vocabulary in the CLI that +hosts all products, and because it forces a core release for each new field — +the maintenance overhead this design was asked to remove. + +**Let extensions send free-form key/value pairs, validated only by shape** +(length, charset, cardinality). Rejected: an extension could pack a container +image reference into a value and leak a private registry name. A closed, +declared value set is what makes the privacy review meaningful. + +**Put declarations in the signed JWT claims instead of looking them up.** +Rejected: the token travels on every RPC, so declarations would add constant +per-call overhead for a feature used a handful of times per command. + +**Augment the existing command span instead of creating `ext.usage`.** +Rejected: it requires a process-global command-usage scope stack that has to be +opened and closed by middleware and kept correct across nested and concurrent +commands. Supporting both models would mean maintaining that machinery *and* +the new span path. diff --git a/docs/architecture/extension-framework.md b/docs/architecture/extension-framework.md index 6f961f5430a..0e5ce7ad5de 100644 --- a/docs/architecture/extension-framework.md +++ b/docs/architecture/extension-framework.md @@ -58,6 +58,7 @@ Extensions declare their capabilities in `extension.yaml`: | `service-target-provider` | Add deployment support for new hosting targets | | `provisioning-provider` | Add custom infrastructure provisioning support | | `metadata` | Provide metadata about commands and capabilities | +| `telemetry` | Report usage attributes declared in the extension's registry entry | ## Available gRPC Services diff --git a/docs/architecture/telemetry.md b/docs/architecture/telemetry.md index bee6971d45f..a5482f4ea9d 100644 --- a/docs/architecture/telemetry.md +++ b/docs/architecture/telemetry.md @@ -183,6 +183,12 @@ flowchart LR - Auth: `ext.auth.*` - Dependency: `ext.dependency.*` - Extension lifecycle events: `ext.install`, `ext.upgrade`, `ext.promote` +- Extensions installed from the official registry and carrying the `telemetry` + capability can report **declared usage attributes** via + `TelemetryService.ReportUsageAttribute`. Each accepted value becomes an + `ext.usage` span sharing the command's trace. The key and its allowed values + come from the extension's registry entry, not from azd core — see + [ADR-001](./adr-001-extension-telemetry-declarations.md). ## Consent & Privacy diff --git a/docs/concepts/glossary.md b/docs/concepts/glossary.md index 86836ad5275..8dec4393c55 100644 --- a/docs/concepts/glossary.md +++ b/docs/concepts/glossary.md @@ -62,7 +62,11 @@ A JSON manifest that lists available extensions, their versions, capabilities, a ### Extension Capabilities -The set of features an extension provides. Valid capabilities are: `custom-commands`, `lifecycle-events`, `mcp-server`, `service-target-provider`, `framework-service-provider`, `provisioning-provider`, and `metadata`. See [Extension Framework](../architecture/extension-framework.md) for details. +The set of features an extension provides. Valid capabilities are: `custom-commands`, `lifecycle-events`, `mcp-server`, `service-target-provider`, `framework-service-provider`, `provisioning-provider`, `validation-provider`, `metadata`, and `telemetry`. See [Extension Framework](../architecture/extension-framework.md) for details. + +### Telemetry Declaration + +The `telemetry` array in an extension's registry entry, listing each usage attribute key the extension may report and the closed set of values allowed for it. `azd` core owns no product-specific telemetry fields; it only enforces what the registry declares. See [ADR-001](../architecture/adr-001-extension-telemetry-declarations.md). ### Provisioning Provider diff --git a/docs/reference/telemetry-data.md b/docs/reference/telemetry-data.md index 47f001122e0..698682f8ea4 100644 --- a/docs/reference/telemetry-data.md +++ b/docs/reference/telemetry-data.md @@ -74,6 +74,7 @@ Commands follow the pattern `cmd.` where spaces become dots. | `ext.install` | Extension installation | | `ext.upgrade` | Extension upgrade attempt | | `ext.promote` | Registry promotion (e.g., dev → main) | +| `ext.usage` | Usage attribute reported by an extension, limited to the values declared in its official registry entry | ### Agent & Copilot Events diff --git a/docs/specs/metrics-audit/feature-telemetry-matrix.md b/docs/specs/metrics-audit/feature-telemetry-matrix.md index 5e9fea1358a..548f4f96fb0 100644 --- a/docs/specs/metrics-audit/feature-telemetry-matrix.md +++ b/docs/specs/metrics-audit/feature-telemetry-matrix.md @@ -166,3 +166,4 @@ privacy review covers every emission point. | **Agent troubleshoot middleware** | Triggered on command failure when troubleshooting is engaged | `agent.troubleshoot` | Error chain attributes, hashed error fields | Emitted from `cmd/middleware/error.go` | | **Up-graph performance** | `up` (graph execution) | (none — enriches the `up` command span) | `perf.provision_duration_ms`, `perf.deploy_duration_ms`, `perf.total_duration_ms` | Emitted from `internal/cmd/up_graph.go` after the graph completes; provision/deploy durations set only when those phases run | | **VS RPC** | `vs-server` long-running session | `vsrpc.*` (event prefix) | Per-RPC attributes documented in `telemetry-schema.md` | Long-running RPC server for VS integration | +| **Extension telemetry service** | Extension calls `ReportUsageAttribute` over the extension gRPC API | `ext.usage` | `extension.id`, `extension.version`, plus the single registry-declared attribute key/value carried by the request | Requires the `telemetry` capability and an install from the official `azd` registry. Keys and their closed value sets are declared per version in the registry, not in core; the host rejects any key outside the `ext..` namespace and any value outside the declared set | diff --git a/docs/specs/metrics-audit/telemetry-schema.md b/docs/specs/metrics-audit/telemetry-schema.md index d4bd317c0a2..85883bcc404 100644 --- a/docs/specs/metrics-audit/telemetry-schema.md +++ b/docs/specs/metrics-audit/telemetry-schema.md @@ -19,6 +19,7 @@ OpenTelemetry span name or event name. | `ExtensionInstallEvent` | `ext.install` | Extension install/upgrade event | | `ExtensionUpgradeEvent` | `ext.upgrade` | Single extension upgrade attempt | | `ExtensionPromoteEvent` | `ext.promote` | Extension registry promotion (e.g., dev → main) | +| `ExtensionUsageEvent` | `ext.usage` | One usage attribute reported by an extension through the telemetry service | | `CopilotInitializeEvent` | `copilot.initialize` | Copilot initialization event | | `CopilotSessionEvent` | `copilot.session` | Copilot session lifecycle event | | `ProvisionValidationEvent` | `validation.provision` | Local provision validation outcome | @@ -215,6 +216,34 @@ not emitted by azd spans. | Dependency of | `extension.dependency_of` | SystemMetadata | FeatureInsight | Parent extension for a dependency upgrade | | Dependency upgrade count | `extension.dependency_upgrade_count` | SystemMetadata | FeatureInsight | Recursive dependency upgrade count | +#### Extension-contributed usage attributes + +Extensions do not have individual fields listed in this document. Instead they +declare the attributes they may report in their entry in the official azd +extension registry, and `azd` records each accepted value on an `ext.usage` +span alongside `extension.id` and `extension.version`. + +`azd` core carries no product-specific telemetry semantics for these fields. +The following rules are enforced by the host and are what this schema +guarantees about the whole class: + +| Rule | Enforcement | +|------|-------------| +| Key namespace | Must be `ext..[.…]`, lowercase alphanumeric and hyphens | +| Values | Must be one of the closed set declared for that key in the registry; free-form values are always rejected | +| Value charset | Lowercase alphanumeric with `_`, `-`, and `.`; `:` and `/` are excluded so registry names, URLs, and paths cannot be smuggled through | +| Classification | Always `SystemMetadata` | +| Purpose | Always `FeatureInsight` | +| Trust | Only extensions installed from the official `azd` registry, carrying the `telemetry` capability, may report values | +| Review | Adding or changing a declared field is a change to the official registry in this repository and is reviewed like any other telemetry change | + +Because `ext.usage` spans share the command's trace, they join the originating +command in Kusto on `operation_Id`. See +[ADR-001](../../architecture/adr-001-extension-telemetry-declarations.md) for +the design rationale and +[Extension Telemetry Fields](../../../cli/azd/docs/extensions/extension-telemetry-fields.md) +for the author-facing declaration rules. + ### Update | Field | OTel Key | Classification | Purpose |