Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 65 additions & 1 deletion pipeline/outputs/azure_kusto.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,41 @@

The _Kusto_ output plugin lets you ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/products/data-explorer/) cluster, using the [Queued Ingestion](https://learn.microsoft.com/en-us/kusto/api/netfx/about-kusto-ingest?view=azure-data-explorer&preserve-view=true&tabs=csharp#queued-ingestion) mechanism. This output plugin can also be used to ingest logs into an [Eventhouse](https://blog.fabric.microsoft.com/en-us/blog/eventhouse-overview-handling-real-time-data-with-microsoft-fabric/) cluster in Microsoft Fabric Real Time Analytics.

## Ingest into Azure Data Explorer: create a Kusto cluster and database
## Authentication Methods

Check warning on line 9 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'Authentication Methods' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'Authentication Methods' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 9, "column": 4}}}, "severity": "INFO"}

Fluent-Bit can use various authentication methods to connect to your Azure Data Explorer cluster:

Check warning on line 11 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.FluentBitCasing] Use the proper noun 'Fluent Bit' with correct casing and spacing. Raw Output: {"message": "[FluentBit.FluentBitCasing] Use the proper noun 'Fluent Bit' with correct casing and spacing.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 11, "column": 1}}}, "severity": "WARNING"}

### Service Principal Authentication (Default)

Check warning on line 13 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'Service Principal Authentication (Default)' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'Service Principal Authentication (Default)' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 13, "column": 5}}}, "severity": "INFO"}

For service principal authentication, you'll need to create an Azure AD application:

- [Register an Application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application)
- [Add a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret)
- [Authorize the app in your database](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers#azure-ad-tenants)

Configure Fluent Bit with your application's `tenant_id`, `client_id`, and `client_secret`.

### Managed Identity Authentication

Check warning on line 23 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'Managed Identity Authentication' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'Managed Identity Authentication' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 23, "column": 5}}}, "severity": "INFO"}

When running on Azure services that support Managed Identities (such as Azure VMs, AKS, or App Service):

Check warning on line 25 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'AKS', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'AKS', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 25, "column": 84}}}, "severity": "INFO"}

1. [Assign the managed identity appropriate permissions to your Kusto database](https://learn.microsoft.com/en-us/azure/data-explorer/configure-managed-identities-cluster)
2. Configure Fluent Bit with `auth_type` set to `managed_identity`
3. For system-assigned identity, set `client_id` to `system`
4. For user-assigned identity, set `client_id` to the managed identity's client ID (GUID)

Check warning on line 30 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Acronyms] Spell out 'GUID', if it's unfamiliar to the audience. Raw Output: {"message": "[FluentBit.Acronyms] Spell out 'GUID', if it's unfamiliar to the audience.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 30, "column": 85}}}, "severity": "INFO"}

### Workload Identity Authentication

Check warning on line 32 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'Workload Identity Authentication' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'Workload Identity Authentication' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 32, "column": 5}}}, "severity": "INFO"}

For Kubernetes environments using Azure Workload Identity:

1. [Set up Azure Workload Identity in your Kubernetes cluster](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster)
2. Configure your pod to use a service account with Workload Identity Federation
3. Configure Fluent Bit with:
- `auth_type` set to `workload_identity`
- `tenant_id` and `client_id` of your Azure AD application
- `workload_identity_token_file` pointing to your token file path (typically `/var/run/secrets/azure/tokens/azure-identity-token`)

Check warning on line 41 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.WordList] Use 'path' instead of 'file path'. Raw Output: {"message": "[FluentBit.WordList] Use 'path' instead of 'file path'.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 41, "column": 60}}}, "severity": "INFO"}

## For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database

Check warning on line 43 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 43, "column": 4}}}, "severity": "INFO"}

Create an Azure Data Explorer cluster in one of the following ways:

Expand Down Expand Up @@ -197,6 +231,36 @@
Io_Timeout 60s
```

#### Managed Identity Authentication

Check warning on line 234 in pipeline/outputs/azure_kusto.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.Headings] 'Managed Identity Authentication' should use sentence-style capitalization. Raw Output: {"message": "[FluentBit.Headings] 'Managed Identity Authentication' should use sentence-style capitalization.", "location": {"path": "pipeline/outputs/azure_kusto.md", "range": {"start": {"line": 234, "column": 6}}}, "severity": "INFO"}

```
[OUTPUT]
Match *
Name azure_kusto
Auth_Type managed_identity
Client_Id <managed_identity_client_id> # Use 'system' for system-assigned managed identity
Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
Database_Name <database_name>
Table_Name <table_name>
# Additional parameters as needed
```

#### Workload Identity Authentication

```
[OUTPUT]
Match *
Name azure_kust
Auth_Type workload_identity
Tenant_Id <tenant_id>
Client_Id <client_id>
Workload_Identity_Token_File /var/run/secrets/azure/tokens/azure-identity-token
Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
Database_Name <database_name>
Table_Name <table_name>
# Additional parameters as needed
```

{% endtab %}
{% endtabs %}

Expand Down
Loading