Skip to content

Update BYOC Logs sizing tier resources #38313

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shuheiktgw wants to merge 1 commit into
base: master
Choose a base branch
from
+22 −13
Open

Update BYOC Logs sizing tier resources #38313

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 22 additions & 13 deletions content/en/byoc-logs/operate/sizing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,19 +134,28 @@ Use standard-tier object storage (for example, S3 Standard, GCS Standard) for ac

## Helm chart sizing tiers

The BYOC Logs Helm chart provides predefined sizing tiers through the `indexer.podSize` and `searcher.podSize` parameters. Each tier sets the vCPU and memory resource limits for a pod, and automatically configures component-specific settings.

| Size | vCPUs | Memory |
|------|-------|--------|
| medium | 1 | 4 GB |
| large | 2 | 8 GB |
| xlarge | 4 | 16 GB |
| 2xlarge | 8 | 32 GB |
| 4xlarge | 16 | 64 GB |
| 6xlarge | 24 | 96 GB |
| 8xlarge | 32 | 128 GB |

Values defining the ingest queue sizes and search cache sizes are automatically applied when you set `indexer.podSize` in the [Helm chart](https://github.com/DataDog/helm-charts/blob/main/charts/cloudprem/sizing-map.yaml). For more details on each parameter, you can check the Quickwit documentation for [indexer parameters][2], [ingest api parameters][3] and [searcher parameters][3].
The BYOC Logs Helm chart provides predefined resource tiers through the `indexer.podSize` and `searcher.podSize` parameters. `podSize` selects the pod's resource requirements and related Quickwit tuning parameters. The default `podSize` is `xlarge` for both components. Each preset is designed to leave room on a matching node for Kubernetes system components, DaemonSets, and add-ons.

The presets account for resources reserved for Kubernetes system components. The reservation amounts are based on the [GKE node reservation calculation](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/plan-node-sizes#resource_reservations). An additional 250m CPU and 512Mi memory per node is reserved for DaemonSets and add-ons:

```text
Actual CPU request = nominal pod CPU - Kubernetes system CPU reservation - 250m
Actual memory request/limit = nominal pod memory - Kubernetes system memory reservation - 512Mi
```

| `podSize` | Nominal CPU request | Actual CPU request | Nominal memory request/limit | Actual memory request/limit |
|---|---:|---:|---:|---:|
| `medium` | 1 | 690m | 4Gi | 2460Mi |
| `large` | 2 | 1680m | 8Gi | 5736Mi |
| `xlarge` | 4 | 3670m | 16Gi | 13109Mi |
| `2xlarge` | 8 | 7660m | 32Gi | 28510Mi |
| `4xlarge` | 16 | 15640m | 64Gi | 59312Mi |
| `6xlarge` | 24 | 23620m | 96Gi | 90114Mi |
| `8xlarge` | 32 | 31600m | 128Gi | 120916Mi |

The presets do not set a CPU limit, allowing a pod to use idle CPU on its node without being throttled. Memory requests and limits are equal to keep memory usage within the allocatable node capacity.

Values defining the ingest queue sizes and search cache sizes are automatically applied for the selected tier. See the [Helm chart sizing map][1] for the complete configuration. For more details on each parameter, see the Quickwit documentation for [indexer parameters][2], [ingest API parameters][3], and [searcher parameters][4].

## Further reading

Expand Down
Loading