Skip to content

Introduce standard storage alternative #895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matheuscscp merged 1 commit into from
Jun 15, 2026
Merged

Introduce standard storage alternative #895

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/spec/v1/imagerepositories.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,27 @@ Events:
Normal Succeeded 17s image-reflector-controller successful scan, found 211 tags
```

## Controller storage

The controller stores scanned tags in a rebuildable on-disk cache at
`--storage-path` (`/data` by default). BadgerDB is the default backend, and
`--storage-value-log-file-size` applies only to that backend.

When the `FluxStorage` feature gate is enabled, the controller uses a filesystem

@stefanprodan stefanprodan Jun 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks so nice! We need a benchmark to see how it does with 1K images and 10K tags vs BadgerDB in terms of CPU/IO/MEM (with a default of 64 KiB instead of 1 KiB to disable gzip). I would really like to turn this feature gate on by default in Flux 2.10 if the benchmark is favorable.

@stefanprodan stefanprodan Jun 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A more relevant benchmark for large setups would be 5K images with 2M tags in total, 60-80MiB (uncompressed on disk).

@matheuscscp matheuscscp Jun 14, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BadgerDB vs Flux storage — benchmark

Single-node kind, one controller (--concurrent=10, threshold 64 KiB). Unique tags per repo.
Each ImageRepository has 1 ImagePolicy, so write path (scan→SetTags) and read path (Tags) both run.
Metrics from the controller process: CPU process_cpu_seconds_total, RSS process_resident_memory_bytes,
disk du /data, IO /proc/1/io.

Throughput (one full scan pass)

tags (repos×tags) backend CPU s peak RSS disk disk write disk read syscall write syscall read wall
10K (1K×10) badger 23 199 MiB <1 MiB 0 MiB 0 17 MiB 48 MiB 9s
10K (1K×10) flux 29 111 MiB 7 MiB 3 MiB 0 18 MiB 50 MiB 8s
2M (5K×400) badger 155 366 MiB 37 MiB 101 MiB 0 89 MiB 316 MiB 19s
2M (5K×400) flux 172 210 MiB 97 MiB 78 MiB 0 159 MiB 412 MiB 18s
25M (5K×5K) badger 241 1441 MiB 418 MiB 1705 MiB 0 103 MiB 1134 MiB 32s
25M (5K×5K) flux 477 228 MiB 234 MiB 214 MiB 1 319 MiB 1425 MiB 43s

2M: 14 KiB files, no gzip. 25M: 170 KiB files, gzip kicks in (>64 KiB).

Idle (zero objects, 60s window)

state backend idle CPU (cores) GC/60s RSS heap in-use next-GC
empty DB badger 0.026 0 52 MiB 97 MiB 180 MiB
empty DB flux 0.002 0 49 MiB 9 MiB 11 MiB

Memory sweep — loaded+orphaned badger (2M tags, objects deleted, default GOGC, no GOMEMLIMIT)

mem limit (=request) idle CPU (cores) GC/60s RSS heap in-use restarts
8Gi 0.019 0 52 MiB 97 MiB 0
2Gi 0.017 0 53 MiB 97 MiB 0
1Gi 0.018 0 53 MiB 97 MiB 0
512Mi 0.018 0 53 MiB 97 MiB 0

Conclusions

  • Memory: flux wins big. 40% less at 2M, 6× less at 25M. Even empty, badger holds 10× the heap (arena + cache at open) — that's the Higher CPU usage without load #333 overhead.
  • Disk: badger smaller under 64 KiB, flux smaller over it (gzip). Small either way (<250 MiB at 25M).
  • CPU: about equal, except flux doubles when gzip turns on (huge tag sets only). 64 KiB default keeps gzip off for normal repos — right call.
  • Badger writes way more to disk (LSM compaction): 8× at 25M.
  • "Bump memory fixes CPU" (Higher CPU usage without load #333): not at idle. Idle badger does 0 GC, so memory changes nothing (IRC sets no GOMEMLIMIT). Core-pinning needs active churn, not idle.
  • Ship FluxStorage on by default (2.10). Less memory, less disk-write, equal CPU normally, and it frees disk on delete (badger never does — Delete is a no-op).

backend based on `github.com/fluxcd/pkg/artifact/storage`. Tags are stored per
`ImageRepository` object, keyed by `<namespace>/<name>` instead of canonical
image name, so repositories with different credentials do not share cached tag
sets. The file layout is:

```text
imagerepository/<namespace>/<name>/tags.txt
imagerepository/<namespace>/<name>/tags.txt.gz
```

Each file contains one OCI tag per line. Files are compressed when the
uncompressed content is at least `--storage-compression-threshold` KiB
(default `64`). Switching the `FluxStorage` gate on or off wipes the tag cache.

## Writing an ImageRepository spec

As with all other Kubernetes config, an ImageRepository needs `apiVersion`,
Expand Down
17 changes: 13 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ require (
github.com/fluxcd/pkg/apis/acl v0.10.0
github.com/fluxcd/pkg/apis/event v0.27.0
github.com/fluxcd/pkg/apis/meta v1.30.0
github.com/fluxcd/pkg/artifact v0.18.0
github.com/fluxcd/pkg/auth v0.54.0
github.com/fluxcd/pkg/cache v0.14.0
github.com/fluxcd/pkg/runtime v0.110.0
Expand All @@ -27,7 +28,6 @@ require (
k8s.io/api v0.36.1
k8s.io/apimachinery v0.36.1
k8s.io/client-go v0.36.1
k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2
sigs.k8s.io/controller-runtime v0.24.1
sigs.k8s.io/yaml v1.6.0
)
Expand Down Expand Up @@ -66,17 +66,22 @@ require (
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chai2010/gettext-go v1.0.3 // indirect
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgraph-io/ristretto/v2 v2.2.0 // indirect
github.com/docker/cli v29.4.3+incompatible // indirect
github.com/docker/docker-credential-helpers v0.9.3 // indirect
github.com/docker/docker-credential-helpers v0.9.5 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emicklei/go-restful/v3 v3.13.0 // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
github.com/fatih/color v1.18.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fluxcd/cli-utils v1.2.1 // indirect
github.com/fluxcd/pkg/lockedfile v0.8.0 // indirect
github.com/fluxcd/pkg/oci v0.68.0 // indirect
github.com/fluxcd/pkg/sourceignore v0.18.0 // indirect
github.com/fluxcd/pkg/tar v1.2.0 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
github.com/go-errors/errors v1.5.1 // indirect
Expand All @@ -100,6 +105,7 @@ require (
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.18.6 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
github.com/mailru/easyjson v0.9.0 // indirect
Expand All @@ -110,7 +116,8 @@ require (
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nxadm/tail v1.4.11 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.1-0.20220411205349-bde1400a84be // indirect
github.com/opencontainers/go-digest/blake3 v0.0.0-20250813155314-89707e38ad1a // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
Expand All @@ -119,12 +126,13 @@ require (
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/cobra v1.10.2 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/xlab/treeprint v1.2.0 // indirect
github.com/zeebo/blake3 v0.2.3 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
go.opentelemetry.io/otel v1.43.0 // indirect
Expand Down Expand Up @@ -156,6 +164,7 @@ require (
k8s.io/klog/v2 v2.140.0 // indirect
k8s.io/kube-openapi v0.0.0-20260317180543-43fb72c5454a // indirect
k8s.io/kubectl v0.36.1 // indirect
k8s.io/utils v0.0.0-20260210185600-b8788abfbbc2 // indirect
sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
sigs.k8s.io/kustomize/api v0.21.1 // indirect
sigs.k8s.io/kustomize/kyaml v0.21.1 // indirect
Expand Down
Loading
Loading