feat: Implement OpenTelemetry telemetry for backend with Application Insights and Azure Monitor Private Link

[Harmanpreet-Microsoft]

Purpose

This pull request introduces comprehensive Application Insights (Azure Monitor) integration for backend telemetry and observability, with a focus on using OpenTelemetry for selective and non-intrusive instrumentation. The changes ensure that FastAPI HTTP requests, Python logs, and custom/manual spans are sent to Application Insights, while avoiding interference with async operations (such as those in Semantic Kernel). Additionally, helper utilities are added for easy manual tracing of Azure SDK operations, and infrastructure changes are made to support Azure Monitor Private Link for secure telemetry ingestion.

The most important changes are:

Backend Telemetry Integration and Instrumentation:

• Introduced OpenTelemetry-based Application Insights integration, exporting FastAPI traces, Python logs, and custom spans to Application Insights. This is done with fine-grained control to avoid interfering with async generators and sensitive code paths. (src/backend/app.py, src/backend/requirements.txt) [1] [2] [3] [4]
• Added OpenTelemetry FastAPI instrumentation and Azure Monitor exporters, with configuration based on the presence of the Application Insights connection string. Logging and tracing are now exported only if configured. (src/backend/app.py)

Telemetry Utilities and Custom Event Tracking:

• Introduced a common.telemetry package with decorators and context managers for easy manual tracing of operations (async and sync) and adding attributes to spans, enabling detailed telemetry for Azure SDK calls and business logic. (src/backend/common/telemetry/__init__.py, src/backend/common/telemetry/telemetry_helper.py) [1] [2]
• Updated custom event tracking to use OpenTelemetry spans and events instead of the deprecated Azure Monitor events extension, ensuring events appear in Application Insights. (src/backend/api/event_utils.py)

Infrastructure: Secure Azure Monitor Private Link Support:

• Added modules and configuration for Azure Monitor Private Link Scope (AMPLS), Data Collection Endpoint (DCE), and required DNS zones, ensuring all telemetry ingestion and queries are routed through private endpoints for enhanced security. (infra/main.bicep) [1] [2] [3] [4] [5]

Cleanup and Refactoring:

• Removed legacy Application Insights configuration and commented-out code from the API routes, as telemetry is now handled via OpenTelemetry and centralized configuration. (src/backend/api/api_routes.py) [1] [2]

These changes significantly improve observability, security, and maintainability of telemetry in the backend, while providing flexible tools for developers to add custom tracing where needed.

• ...

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information