From a25a9f0fae7b9e9f71501a5d903dd7a1c36f2c1d Mon Sep 17 00:00:00 2001 From: Cailey Jones Date: Tue, 21 Jul 2020 12:30:55 -0700 Subject: [PATCH 1/5] cronjob doc init --- docs/CronjobManagement.md | 58 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 docs/CronjobManagement.md diff --git a/docs/CronjobManagement.md b/docs/CronjobManagement.md new file mode 100644 index 000000000..20e2856ec --- /dev/null +++ b/docs/CronjobManagement.md @@ -0,0 +1,58 @@ +--- +title: Cronjob Management +resourceType: Documentation +personas: + - Developer +tags: + - openshift + - ha + - maintenance + - cronjobs + - devops +description: Running cronjobs in a community-minded way. +--- + +# Cronjob Management + +Cronjobs are a common and useful tool on Openshift, just as they are on any legacy platform. +There are, however, a few key things to note about how cronjobs work in a containerized platform that development teams should consider when running cronjobs. + +## How Openshift Cronjobs Work + +A cronjob is scheduled on Openshift in almost exactly the same way as it would be on a normal server, using the same `* * * * *` format. +However, instead of spinning up a file to be run on the given schedule, a cronjob on Openshift spins up a new pod to perform the task in question. +This has its pros and cons. + +The benefit is likely rather obvious: you don't have a pod running, taking up resources, unless the cronjob is actually running. +Once the scheduled moment hits, the system spins up a pod, performs the task, and then shuts right back down again, leaving your quota free for you to use in other ways. + +The drawback, however, is that you're required to spin up a pod every time you want to run the job, which creates a little overhead on each job. +Now, this isn't a concern if you're only running the job twice a day. +However, if you have more stars in your schedule than anything else, you may run into an issue where you have a new pod spinning up every single minute. +This is no longer an acceptable amount of overhead, and may even interfere with the correct running of the job itself as well as placing unnecessary strain on the platform as a whole. + +## How Can I Find Out If My CronJobs Run Too Often? + +We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. +If you have a job that runs every 15 minutes but takes 10 seconds to run, spinning up a new pod every time might be worthwhile, since that's one less pod on the platform for 14 and a half minutes. +On the other hand, if your job takes 12 minutes to run, it might no longer be worthwhile to spend the overhead shutting the pod down for only 3 minutes in each window. + +There is a reason why teams on this platform are given so much freedom to make their own decisions - we trust you to be educated on best practices, to be community-minded, and to understand your application better than we ever could. +**Use your own best judgement when deciding if your cronjob runs too often.** + +To check and see what cronjobs you have on your project that run very often, try this command: + +`oc get cronjob | grep -E '\* \* \* \* \*|\*/1' | grep -v True` + +This version of the command will only pull up those cronjobs running once per minute - if you want to find cronjobs with other schedules, just adjust the grep filter accordingly. + +## What Do I Do If I Have A Cronjob That Runs Often? + +The first and most obvious question is whether it *needs* to run that often. If not, that's a very easy fix! + +But often, there is a very good reason why a cronjob would need to run so regularly. We are absolutely not asking that you reduce the regularity of such jobs, especially not if it would negatively impact your application in some way. +In this situation, the correct solution is to put your cronjon into a **persistent pod**. +This just means that, instead of spinning up a new pod every time the platform tries to run your job, it spins up one pod, one time, and that pod stays up, running the job repeatedly. +This eliminates the need for a pod to bounce up and down very often. + + From 9c9c1d532a0cb2244b8ec23ed7c937feeca03b9e Mon Sep 17 00:00:00 2001 From: Cailey Jones Date: Tue, 21 Jul 2020 12:38:01 -0700 Subject: [PATCH 2/5] cronjob doc init --- docs/CronjobManagement.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/CronjobManagement.md b/docs/CronjobManagement.md index 20e2856ec..5f49b10c3 100644 --- a/docs/CronjobManagement.md +++ b/docs/CronjobManagement.md @@ -37,8 +37,15 @@ We don't have a handy rule-of-thumb to decide how often is too often - while it If you have a job that runs every 15 minutes but takes 10 seconds to run, spinning up a new pod every time might be worthwhile, since that's one less pod on the platform for 14 and a half minutes. On the other hand, if your job takes 12 minutes to run, it might no longer be worthwhile to spend the overhead shutting the pod down for only 3 minutes in each window. +As a general rule you should take a closer look at any cronjob that runs more often than every 10 minutes. +This is not to say that every job that runs that often is a problem and *requires* tweaking, but rather that these jobs require a little extra checking to determine *if* they require adjustment. + +Similarly, you should take a second look at any job that requires more than a minute or two to complete. + +However, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part. There is a reason why teams on this platform are given so much freedom to make their own decisions - we trust you to be educated on best practices, to be community-minded, and to understand your application better than we ever could. **Use your own best judgement when deciding if your cronjob runs too often.** +And remember that this is a shared platform - don't use more resources than you actually require. To check and see what cronjobs you have on your project that run very often, try this command: From 8d957548000594942d55d55e68402d7e26b6f30c Mon Sep 17 00:00:00 2001 From: Cailey Jones Date: Tue, 21 Jul 2020 15:01:01 -0700 Subject: [PATCH 3/5] add jobber reference --- docs/CronjobManagement.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/CronjobManagement.md b/docs/CronjobManagement.md index 5f49b10c3..2f423438a 100644 --- a/docs/CronjobManagement.md +++ b/docs/CronjobManagement.md @@ -62,4 +62,6 @@ In this situation, the correct solution is to put your cronjon into a **persiste This just means that, instead of spinning up a new pod every time the platform tries to run your job, it spins up one pod, one time, and that pod stays up, running the job repeatedly. This eliminates the need for a pod to bounce up and down very often. - +There are a number of ways to approach creating this kind of persistent pod. +A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. +If your team finds a good solution that works well, we invite you to include a link to your deployment config on this document! From f59843b4ecc9d1214bc0f62a6f9e24a9d1470300 Mon Sep 17 00:00:00 2001 From: Cailey Jones Date: Mon, 27 Jul 2020 12:20:47 -0700 Subject: [PATCH 4/5] requested updates --- docs/CronjobManagement.md | 49 ++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 13 deletions(-) diff --git a/docs/CronjobManagement.md b/docs/CronjobManagement.md index 2f423438a..331537f07 100644 --- a/docs/CronjobManagement.md +++ b/docs/CronjobManagement.md @@ -1,7 +1,7 @@ --- title: Cronjob Management resourceType: Documentation -personas: +personas: - Developer tags: - openshift @@ -14,35 +14,33 @@ description: Running cronjobs in a community-minded way. # Cronjob Management -Cronjobs are a common and useful tool on Openshift, just as they are on any legacy platform. +Cronjobs are a common and useful tool on Openshift, just as they are on any legacy platform. There are, however, a few key things to note about how cronjobs work in a containerized platform that development teams should consider when running cronjobs. ## How Openshift Cronjobs Work A cronjob is scheduled on Openshift in almost exactly the same way as it would be on a normal server, using the same `* * * * *` format. However, instead of spinning up a file to be run on the given schedule, a cronjob on Openshift spins up a new pod to perform the task in question. -This has its pros and cons. +This has its pros and cons. The benefit is likely rather obvious: you don't have a pod running, taking up resources, unless the cronjob is actually running. Once the scheduled moment hits, the system spins up a pod, performs the task, and then shuts right back down again, leaving your quota free for you to use in other ways. The drawback, however, is that you're required to spin up a pod every time you want to run the job, which creates a little overhead on each job. -Now, this isn't a concern if you're only running the job twice a day. +Now, this isn't a concern if you're only running the job twice a day. However, if you have more stars in your schedule than anything else, you may run into an issue where you have a new pod spinning up every single minute. This is no longer an acceptable amount of overhead, and may even interfere with the correct running of the job itself as well as placing unnecessary strain on the platform as a whole. ## How Can I Find Out If My CronJobs Run Too Often? -We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. -If you have a job that runs every 15 minutes but takes 10 seconds to run, spinning up a new pod every time might be worthwhile, since that's one less pod on the platform for 14 and a half minutes. -On the other hand, if your job takes 12 minutes to run, it might no longer be worthwhile to spend the overhead shutting the pod down for only 3 minutes in each window. +### Cronjobs Scheduled Too Often -As a general rule you should take a closer look at any cronjob that runs more often than every 10 minutes. -This is not to say that every job that runs that often is a problem and *requires* tweaking, but rather that these jobs require a little extra checking to determine *if* they require adjustment. +We don't have a handy rule-of-thumb to decide how often is too often - while it would be nice to say "any job that runs more often than once every 15 minutes," the reality is that it depends on the nature of the job. -Similarly, you should take a second look at any job that requires more than a minute or two to complete. +If you find that you need to run a cronjob every minute, the Platform Team has decided that you *cannot* do this using the Openshift cronjob object. +If every minute is necessary for your app, you *must* use a persistent pod. -However, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part. +Outside of this limitation, however, we are not going to provide specific limits on what counts as "too often" or "too long" and which *require* action on your part. There is a reason why teams on this platform are given so much freedom to make their own decisions - we trust you to be educated on best practices, to be community-minded, and to understand your application better than we ever could. **Use your own best judgement when deciding if your cronjob runs too often.** And remember that this is a shared platform - don't use more resources than you actually require. @@ -53,15 +51,40 @@ To check and see what cronjobs you have on your project that run very often, try This version of the command will only pull up those cronjobs running once per minute - if you want to find cronjobs with other schedules, just adjust the grep filter accordingly. +### Cronjobs with a Bad Ratio of Work to Overhead + +Another way to determine whether your cronjobs run too often is to consider how much platform overhead they require. +This can be calculated in terms of work per hour, like so: + +First, you must determine how long your job takes to run, and determine how much time per hour it spends running, like so: +`Job work = 5 seconds x 4 times per hour = 20 seconds per hour` + +Then, determine how long it takes the pod to spin up every time it runs: +`Overhead work = 1 minute x 4 times per hour = 4 minutes per hour` + +Because the overhead of spinning up the pod requires more work than actually running the job does, it may be better for your job (and for the platform) if you simply put this job onto a persistent pod and ensure that the pod is running with extremely low requests. + +On the other hand, if you end up with a job that runs only once per day and takes a significant amount of time to run, you might end up with a calculation that looks more like this: + +```bash +Job work = 20 minutes x 1 time per day = 20 minutes per day +Overhead work = 1 minute x 1 time per day = 1 minute per day +``` + +In this case, the job work is significantly *higher* than the overhead work - this is the kind of cronjob where spinning up a pod is clearly worthwhile. + +Again, we can't really give you specific numbers regarding what ratio of work to overhead is the correct "cut-off" point - it still depends on the needs of the app. +However, if your overhead is higher than your actual cronjob work, it's definitely time to consider whether a persistent pod would be a better choice. + ## What Do I Do If I Have A Cronjob That Runs Often? The first and most obvious question is whether it *needs* to run that often. If not, that's a very easy fix! But often, there is a very good reason why a cronjob would need to run so regularly. We are absolutely not asking that you reduce the regularity of such jobs, especially not if it would negatively impact your application in some way. -In this situation, the correct solution is to put your cronjon into a **persistent pod**. +In this situation, the correct solution is to put your cronjon into a **persistent pod**. This just means that, instead of spinning up a new pod every time the platform tries to run your job, it spins up one pod, one time, and that pod stays up, running the job repeatedly. This eliminates the need for a pod to bounce up and down very often. -There are a number of ways to approach creating this kind of persistent pod. +There are a number of ways to approach creating this kind of persistent pod. A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. If your team finds a good solution that works well, we invite you to include a link to your deployment config on this document! From 197187645b0d1e9c0ca072f35559a392e7ddad02 Mon Sep 17 00:00:00 2001 From: Cailey Jones Date: Mon, 27 Jul 2020 13:13:00 -0700 Subject: [PATCH 5/5] add backup container mention --- docs/CronjobManagement.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/CronjobManagement.md b/docs/CronjobManagement.md index 331537f07..a1a67dc69 100644 --- a/docs/CronjobManagement.md +++ b/docs/CronjobManagement.md @@ -86,5 +86,8 @@ This just means that, instead of spinning up a new pod every time the platform t This eliminates the need for a pod to bounce up and down very often. There are a number of ways to approach creating this kind of persistent pod. + A good option might be [Jobber](https://hub.docker.com/_/jobber) - the platform team has not made extensive use of this, but development teams are welcome to test it (or any other solution) out as a problem. If your team finds a good solution that works well, we invite you to include a link to your deployment config on this document! + +There is also an example of a long-running pod that implements a cronjob in the [backup container](https://github.com/BCDevOps/backup-container).