Run Cron Jobs with Jenkins Efficiently

Jenkins is one of the most popular CI / CD platforms available today. Its strength lies in the vast plugin ecosystem which lets you customize almost everything about it. A large and active community has contributed hundreds of plugins to date.

One of the lesser-known facts about Jenkins is it could be used as a very good job running tool that can replace traditional crontab on servers. Jenkins pipeline gives you fine-grained control over simple to complex jobs orchestration needs.

Background

We were using Linux crontab to execute a number of jobs every midnight for a high traffic e-commerce application. Many of these jobs are inter-dependent.

For example,

1. We receive product price feed from ERP system at a specific time at night. So, we process the feed at that time and then run the next jobs such as:
2. Update prices in cache storage so that new prices would reflect on the frontend.
3. Calculate discounts based on updated price for each product
4. Maps discounted products to “Sale” product category which appears as a separate page on frontend to attract customers.

There are multiple challenges with this

1. If price feed is not received someday, subsequent jobs should not be executed. In other words, a job execution depends on the successful execution of the preceding job.
2. How long the execution will take for each job could vary and depends on the number of product prices received in the feed. That means you can not schedule the next jobs at a pre-defined time. You might end up starting it before the completion of the previous one.
3. Sometimes a job may stall waiting for system resources or it can result in failure. However, this lack of resources could be temporary, for a small duration and job should complete successfully if re-attempted.

Managing all these challenges with traditional crontab was a nightmare. We often end up rescheduling manually all subsequent jobs if current job delays or fails.

That's when we realized, the platform we are using for deployments(Jenkins) could also be used to manage the jobs effectively and overcome all these challenges.

Solution — Jenkins Pipeline

Declarative pipeline comes as a recommended plugin with Jenkins and offers handy features such as

1. Retry a job on failure
2. Specify a timeout period for a job
3. Run jobs sequentially or in parallel.
4. Ensure only one (or specified number) instance of the job runs at a time

As a default, a pipeline step depends on the completion of the previous step. So, you don’t have to schedule subsequent jobs at a fixed time like you do in crontab. They are executed only after the previous job completes. Additionally, if you want, you can handle the failure of a job by either retrying the job or move forward in the pipeline to execute the next job.

Now, a pipeline step is meant for deployment steps such as running tests, build code etc. but its a generalized block that could be used to execute any type of job.

Jenkins maintains the execution log of each step and the history of pipeline runs. Which makes its easy to refer and debug whenever needed.

In Jenkins, I can run commands remotely using SSH plugin which helps to manage all jobs from a separate server. This means I can have a separate job managing server to remotely run jobs on all target servers instead of managing crontab on each of them.

Pipeline also shows the visual representation of steps being executed and time taken for each. It helps to track jobs progress on the dashboard as shown below.

Add-ons

You can pause the pipeline if needed in specific situations e.g. production releases at the same time. You can resume the pipeline from the same point once you are done with the release process. You can send customized email notifications based on job execution results. e.g. send notification only if the job fails and include execution logs as an attachment.

Apart from these, Jenkins brings a lot of options, features, utilities to satisfy your business needs.

Syntax

Jenkins uses Groovy syntax to define pipelines and its very easy to learn. Here is the sample jobs pipeline:

There are other tools and solutions available in the market but Jenkins is open source, easy to set up, and backed by the powerful community. It's worth evaluating before building a custom solution or purchasing a complex tool with a long learning curve.

References

• Jenkins: https://jenkins.io/
• Jenkins pipeline: https://jenkins.io/doc/book/pipeline/syntax