feat: Add initial funnel k8s cleanup command#1439
Open
lbeckman314 wants to merge 2 commits into
Open
Conversation
Signed-off-by: Liam Beckman <lbeckman314@gmail.com>
Gen3 Integration Tests
|
Gen3 Integration Tests
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an initial
funnel kubernetes cleanupcommand that deletes orphaned Funnel-managed Kubernetes resources (PVs, ServiceAccounts, executor jobs) whose task is no longer present or is in a terminal state. It is designed to run inside the cluster as a CronJob — the "external reconciler" — so that resource cleanup is decoupled from the Funnel Server lifecycle and multiple server replicas do not race to clean the same resources.The command reuses the running server's configuration and connects to the same cluster; it does not stand up a second compute backend or require a worker job template.
Changes
cmd/kubernetes/kubernetes.gofunnel kubernetes cleanupsubcommand. Opens only the configured Funnel database (no HTTP/gRPC server) and runsCleanOrphanedResources.k8salias for thekubernetescommand group, matching the backend name which already accepts bothkubernetesandk8s.compute/kubernetes/backend.goNewCleanupBackendconstructor: a cleanup-only backend that, unlikeNewBackend, does not require a worker job template and does not start the reconcile goroutine. It only needs a k8s client, a DB reader, and config.JobsNamespaceand then the pod's ServiceAccount namespace.util/k8sutil/k8s.goInClusterNamespace, which reads the namespace from the pod's mounted ServiceAccount, so the CronJob works without an explicitly configured namespace.Motivation and Context
K8s resource cleanup currently runs inside the Funnel Server via the reconcile goroutine, coupling it to the server lifecycle and causing replicas to race. This command is the CLI entry point for running that cleanup as an independent, schedulable CronJob.
Related PRs:
reconcileOnce) can be invoked externally. Once those land, this command should call the full single-pass reconcile rather than onlyCleanOrphanedResources(tracked by aTODOincmd/kubernetes/kubernetes.go).funnel kubernetes cleanup --config /etc/config/funnel-server.yamland derives its schedule fromKubernetes.ReconcileRate.How Has This Been Tested?
go vetpass locally.funnel k8s --help/funnel kubernetes --helpresolve to the same command group (alias verified).kubectl create job --from=cronjob/funnel-cleanup-<namespace> funnel-cleanup-manual.Types of Changes
Checklist