Skip to content

feat: Add initial funnel k8s cleanup command#1439

Open
lbeckman314 wants to merge 2 commits into
developfrom
feat/k8s-cleanup-cmd
Open

feat: Add initial funnel k8s cleanup command#1439
lbeckman314 wants to merge 2 commits into
developfrom
feat/k8s-cleanup-cmd

Conversation

@lbeckman314

@lbeckman314 lbeckman314 commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Description

Adds an initial funnel kubernetes cleanup command that deletes orphaned Funnel-managed Kubernetes resources (PVs, ServiceAccounts, executor jobs) whose task is no longer present or is in a terminal state. It is designed to run inside the cluster as a CronJob — the "external reconciler" — so that resource cleanup is decoupled from the Funnel Server lifecycle and multiple server replicas do not race to clean the same resources.

The command reuses the running server's configuration and connects to the same cluster; it does not stand up a second compute backend or require a worker job template.

Changes

  • cmd/kubernetes/kubernetes.go
    • New funnel kubernetes cleanup subcommand. Opens only the configured Funnel database (no HTTP/gRPC server) and runs CleanOrphanedResources.
    • Add a k8s alias for the kubernetes command group, matching the backend name which already accepts both kubernetes and k8s.
  • compute/kubernetes/backend.go
    • New NewCleanupBackend constructor: a cleanup-only backend that, unlike NewBackend, does not require a worker job template and does not start the reconcile goroutine. It only needs a k8s client, a DB reader, and config.
    • Resolves the target namespace from config, falling back to JobsNamespace and then the pod's ServiceAccount namespace.
  • util/k8sutil/k8s.go
    • Add InClusterNamespace, which reads the namespace from the pod's mounted ServiceAccount, so the CronJob works without an explicitly configured namespace.

Motivation and Context

K8s resource cleanup currently runs inside the Funnel Server via the reconcile goroutine, coupling it to the server lifecycle and causing replicas to race. This command is the CLI entry point for running that cleanup as an independent, schedulable CronJob.

Related PRs:

How Has This Been Tested?

  • Builds and go vet pass locally.
  • funnel k8s --help / funnel kubernetes --help resolve to the same command group (alias verified).
  • The command is intended to run in-cluster (it uses in-cluster config). It can be exercised by triggering the CronJob from feat: Add initial CronJob as external reconciler for resource cleanup helm-charts#22, e.g. kubectl create job --from=cronjob/funnel-cleanup-<namespace> funnel-cleanup-manual.

Types of Changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • I have updated the documentation accordingly (link here).
  • I have tested that this feature locally.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • Reviewer has tested this feature locally

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>
@github-actions

Copy link
Copy Markdown

Gen3 Integration Tests

filepath passed skipped SUBTOTAL
tests/test_gen3_workflow.py 25 4 29
TOTAL 25 4 29

@lbeckman314 lbeckman314 marked this pull request as ready for review June 26, 2026 04:05
@github-actions

Copy link
Copy Markdown

Gen3 Integration Tests

filepath passed skipped SUBTOTAL
tests/test_gen3_workflow.py 25 4 29
TOTAL 25 4 29

@lbeckman314 lbeckman314 added this to the Helm Chart v0.1.99-rc.39 milestone Jul 3, 2026
@lbeckman314 lbeckman314 self-assigned this Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant