Add abitility to set QPS and Burst limits for api client

[Demch1k]

We have a problem in our big kubernetes clusters with 300+ postgres clusters.
When operator starts reconcile it can stack creating new clusters for 10-15 minutes and We can see in the logs messages like that:

I0618 03:13:56.195420 1 request.go:697] Waited for 17.19394453s due to client-side throttling, not priority and fairness, request: POST:https://10.141.64.1:443/apis/apps/v1/namespaces/postgres-ns/statefulsets

Especially in working hours when our developers create new environments.
This PR adds ability to set kubernetes qps and burst limits for client.

[Demch1k]

Any updates ?

[FxKu]

Can you not make these variables also configurable similar to NoDatabaseAccess and NoTeamsAPI. We can use 5 and 10 as default values.

[Demch1k]

@FxKu I have reverted my previous changes because we can't change qps and burst limits for kube client after initialization.
Without kube client we can't obtain crd with limits. Only one way to pass it is used cli args.

[FxKu]

Config changes always require a replacement of the operator pod. Doesn't this initialize the client as well each time?

[Demch1k]

@FxKu I mean that you can't change kubernetes client parameters after client has been initilized.
To be able to read it from operator configs you need initilize kubernetes client first and read custom resource. After that you must reinitilize kubernetes client to apply new values like QPS or Burst but I don't think that is possible. If you know how to do it, help me please with it.

[Demch1k]

@FxKu We have been testing this fix in our k8s cluster with 200+ postgres clusters for 2 weeks now and it really helps.

[Demch1k]

@FxKu any updates?

[FxKu]

@FxKu We have been testing this fix in our k8s cluster with 200+ postgres clusters for 2 weeks now and it really helps.

Can you add a chapter in the docs on how to do this (administrator.md is a good place, I think)? I guess many users would be interested.

[FxKu]

@Demch1k PR is now green after I have increased the qps and burst value slightly. Could you provide a small paragraph to our docs to show how people can scale the operator so it can handle 200+ clusters. This would be much appreciated. I can then merge it tomorrow so it will be included in the release.

[hughcapet]

👍

[Demch1k]

@FxKu Sorry, I was totally running of time. Hope will have time soon

[FxKu]

👍

[FxKu]

Thanks @Demch1k. We merge it now, but a follow PR would be welcome.

[chobostar]

Hi @Demch1k, thanks for the improvement — this seems very useful.

A couple of questions:

1. Given your experience supporting 300+ Postgres clusters, what QPS and burst values have you found work best in practice?
2. I noticed that the Helm chart (https://github.com/zalando/postgres-operator/blob/master/charts/postgres-operator/templates/deployment.yaml#L36-L71) doesn’t currently support passing CLI args. I assume you're deploying the operator with a custom patch or without using the Helm chart?

[chobostar]

@FxKu I brought the docs #2892, RFR