Add HELM_FORCE_UPDATE and --helm-force flag

[krzysztofczyz-da]

Fixes #4940

I think it makes sense to have a flag to switch this on and off as Helm's --force should not be used normally (as replace can nuke your apps' availablity, particulary damaging if done for a slow starting app). Because of this current implementation adds HELM_FORCE_UPDATE env var which is false by default and --helm-force flag to cncluster which conditionally turns it to true - I assume most use cases will be applied from localhost.

[martinflorian-da]

Thanks!

--force should not be used normally (as replace can nuke your apps' availablity, particulary damaging if done for a slow starting app)

Hmm so my understanding is that since we chain this after Pulumi, a Helm upgrade is (still) only triggered is something actually changed in the Pulumi state. And then you'd get a restart, true, but AFAICT we almost always get a restart when we change things even without force... I'd say let's just test it by merging and if it causes problems we can revert before it reaches MainNet. I.e., default the config thing to true.

I assume most use cases will be applied from localhost.

So the main need is actually the long-running clusters (CILR, DevNet, TestNet, MainNet). These are managed by the operator, so AFAIU we'd need to deploy the operator once to enable this "feature"... and then it's enabled for all help operations.

env var

We're trying to reduce the number of weird env vars; can we add this to the config.yaml schema instead?

And let's do a CI scratchnet test before merging, maybe an upgrade test as this actually changes many things that are already deployed.

[martinflorian-da]

I actually don't think we need that extra cncluster feature tbh; assuming you add this to config.yaml, the "edit config.yaml and rerun" is a quite established flow by now.

[martinflorian-da]

I also suggest doing a basic "does it do what it should" test if you haven't already. If you're quick, you could just kubectl edit some deployment while the CI upgrade test is running (but before the upgrade step) to add some extra env var, and then confirm that the operator removed your manual changes again on the upgrade step.