diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d1f6df..8f2df8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.7.6] - 2026-03-28 + +### Changed + +- Rewrite README to clarify project motivation: overcome the 5-minute Hubble ring buffer limit using Loki for long-term flow history. +- Document zero-trust workflow: start from default-deny, filter on audit verdicts, and generate policies for the gaps. +- Promote Loki backend as the recommended flow source in Quick Start. ## [0.7.5] - 2026-03-28 ### Fixed diff --git a/README.md b/README.md index c73b6e5..2c5c5a9 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,26 @@ # hubble-audit2policy -Generate least-privilege [CiliumNetworkPolicy](https://docs.cilium.io/en/stable/security/policy/) YAML from observed Hubble traffic -- no manual flow capture required. +Generate least-privilege [CiliumNetworkPolicy](https://docs.cilium.io/en/stable/security/policy/) YAML from real observed traffic -- no enterprise license required. -Point it at your cluster and get policies. The tool connects to [Hubble](https://github.com/cilium/hubble) directly, queries [Grafana Loki](https://grafana.com/oss/loki/), or reads a file you already have. It produces per-workload CiliumNetworkPolicy files with ingress and egress rules, enriched with real Cilium endpoint labels. +## Why this exists -Built for platform engineers, SREs, and security engineers who need to bootstrap or audit Kubernetes network policies from real observed traffic rather than guessing. +Cilium OSS gives you Hubble for flow observability, but it has a hard limit: Hubble keeps only 5 minutes of flows in its ring buffer before they are gone. There is no built-in way to look back over hours or days of traffic and generate network policies from what actually happened. Cisco's/Isovalent's enterprise offering, Timescape, fills that gap, but it requires an enterprise license. + +This tool takes a different approach. If you are already shipping Hubble flows to Grafana Loki (or any log pipeline), you have a long-term record of every connection your workloads make. hubble-audit2policy queries that record and turns it into per-workload CiliumNetworkPolicy files with proper ingress and egress rules, enriched with real Cilium endpoint labels. + +The key benefit is **capturing workload ground noise** -- the full picture of what your services actually talk to, including the traffic that only happens during UAT, development, cron jobs, nightly batches, cache warming, leader elections, or once-a-week reconciliation loops. With only a 5-minute ring buffer, you will never catch all of that by watching live. Loki keeps it all, and this tool reads it all back. + +The typical workflow is to start from a zero-trust baseline (default-deny policies) and then use this tool filtered to audit verdicts only (`--verdict audit`) to see what traffic would be blocked. Those audit flows represent the gaps in your policy -- the legitimate connections that need to be allowed. hubble-audit2policy turns them directly into the CiliumNetworkPolicy rules that fill those gaps. + +You get the same outcome as an enterprise policy-generation feature, using infrastructure you probably already run. + +## How it works + +1. Hubble flows get shipped to Loki (via fluentd, promtail, or whatever you use) +2. hubble-audit2policy queries Loki for flows over any time range you choose +3. It groups flows by workload, resolves real Cilium endpoint labels from the cluster, and writes one CiliumNetworkPolicy YAML per workload + +It also works with live Hubble streams and plain JSON files if that is what you have, but the Loki backend is where the real value lives. ## Installation @@ -20,23 +36,25 @@ pip install -e ".[dev]" ## Quick Start -### One command, three ways to get flows +### From Loki (recommended) -Pick whichever fits your setup -- the tool handles the rest: - -**Live from the cluster** -- auto-detects Hubble, streams flows, shows an interactive TUI: +Query your existing log pipeline -- no port-forwarding, no time pressure, full history: ```bash -hubble-audit2policy --watch -o policies/ +hubble-audit2policy --from loki --loki-url http://loki:3100 -o policies/ ``` -**From Grafana Loki** -- queries your existing log pipeline, no port-forwarding needed: +### Live from the cluster + +Auto-detects Hubble, streams flows, shows an interactive TUI: ```bash -hubble-audit2policy --from loki --loki-url http://loki:3100 -o policies/ +hubble-audit2policy --watch -o policies/ ``` -**From a file** -- if you already have exported flows: +### From a file + +If you already have exported flows: ```bash hubble-audit2policy flows.json -o policies/ @@ -48,19 +66,39 @@ All three produce the same output: one CiliumNetworkPolicy YAML per workload. ```bash # Preview policies on stdout without writing files: -hubble-audit2policy --watch --dry-run +hubble-audit2policy --from loki --loki-url http://loki:3100 --dry-run # Write all policies into a single multi-document YAML: hubble-audit2policy --from loki --loki-url http://loki:3100 --single-file policies/all.yaml # Scope to specific namespaces: -hubble-audit2policy --watch -n monitoring -n default -o policies/ +hubble-audit2policy --from loki --loki-url http://loki:3100 -n monitoring -n default -o policies/ + +# Custom time window -- last 24 hours: +hubble-audit2policy --from loki --loki-url http://loki:3100 --since 24h -o policies/ # Print a flow frequency report (works with any source): -hubble-audit2policy --watch --report +hubble-audit2policy --from loki --loki-url http://loki:3100 --report hubble-audit2policy flows.json --report-only ``` +## Loki Backend + +Query a Grafana Loki instance directly -- ideal when Hubble flows are already being shipped to Loki via fluentd or another collector: + +```bash +# All flows from the last hour: +hubble-audit2policy --from loki --loki-url http://loki:3100 --dry-run + +# Scoped to a namespace with a custom time window: +hubble-audit2policy --from loki --loki-url http://loki:3100 --since 2h --until 30m -n kube-system -o policies/ + +# Custom LogQL selector (adjust to match your labels): +hubble-audit2policy --from loki --loki-url http://loki:3100 --loki-query '{namespace="hubble"}' +``` + +All existing filters (`-n`, `--verdict`, `--label-key`, `--report`, etc.) work identically with the Loki backend. + ## Live Watch Mode Watch mode spawns `hubble observe` internally and continuously refreshes a flow-frequency report in a curses-based TUI. No separate terminal or manual capture needed -- just run: @@ -106,7 +144,7 @@ hubble-audit2policy --watch --dry-run # preview selected policies on stdout | Key | Action | |-----|--------| -| `j/↓`, `k/↑` | Scroll down / up one line | +| `j/down`, `k/up` | Scroll down / up one line | | `d/PgDn`, `u/PgUp` | Scroll half a page | | `g/Home`, `G/End` | Jump to top / bottom | | `Space` | Pause/resume (normal); toggle selection (select mode) | @@ -115,23 +153,6 @@ hubble-audit2policy --watch --dry-run # preview selected policies on stdout | `Esc` | Exit select mode and clear selections | | `q / Ctrl-C` | Quit (last report is printed to the terminal) | -## Loki Backend - -Query a Grafana Loki instance directly -- ideal when Hubble flows are already being shipped to Loki via fluentd or another collector: - -```bash -# All flows from the last hour: -hubble-audit2policy --from loki --loki-url http://loki:3100 --dry-run - -# Scoped to a namespace with a custom time window: -hubble-audit2policy --from loki --loki-url http://loki:3100 --since 2h --until 30m -n kube-system -o policies/ - -# Custom LogQL selector (adjust to match your labels): -hubble-audit2policy --from loki --loki-url http://loki:3100 --loki-query '{namespace="hubble"}' -``` - -All existing filters (`-n`, `--verdict`, `--label-key`, `--report`, etc.) work identically with the Loki backend. - ## Cluster Enrichment By default the tool queries `cilium endpoint list` and `cilium endpoint get` on each Cilium DaemonSet pod to resolve the authoritative security-relevant labels for every workload seen in the flows. This produces accurate `endpointSelector` and `matchLabels` in the generated policies instead of a simple `app` label fallback. diff --git a/pyproject.toml b/pyproject.toml index 677684e..1e01e51 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "hubble-audit2policy" -version = "0.7.5" +version = "0.7.6" description = "Generate least-privilege CiliumNetworkPolicy YAML from Hubble flow logs." readme = "README.md" license = "Apache-2.0"