So you want to run a cron job... with devops
The other day, I found myself wanting to run a task as a cron. I’ve set up plenty of crontabs in my day, but my current work setup doesn’t have anything like the “one specified server where all the crons run” like most previous jobs have had. We do, however, have a Nomad cluster where all of our important applications and workloads run, so that seemed like a sensible place to put this as well. Naturally, getting this all figured up took some yakshaving and tears, so for posterity I’m documenting what I got set up.
This tutorial will cover how to use Circle, Docker, and Nomad to run what is basically a cron job. It assumes that you have a CircleCI account, a quay.io account, and a working Nomad cluster all set up already, as well as Docker installed and running on your local dev machine. (For bonus points, it also assumes you have Vault and Terraform running, though those are optional if your cron doesn’t need any secrets.) This post will in no way cover how to set up a Nomad cluster or anything like that; it covers only all the moving pieces I found necessary to glue this cron job together. Ready? Me neither, but let’s get going!
The first thing you’ll want to have is the script you want to deploy as your cron job. To keep things simple and/or ridiculous, let’s say that you have a bash script called cow_cron.sh that expects to get some information from environment variables.
#!/bin/bash
cowsay $SECRET_COW_SPEECHWhy would you need a cron job to do this? I dunno, look at how cute it is though:
___________
< honk >
-----------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||We’re going to pretend that you’re going to set this all up in a github repo, because that’s what you would do if you had a real script that you needed to run regularly and not just a noisy cow.
The next thing we want to do is to create a Docker image that has our bash script all packaged up in it. Our minimal Dockerfile, which will install the cowsay package (and set up the path to use it) and our script, looks like this:
FROM ubuntu:latest
RUN apt-get update && apt-get install -y cowsay
ENV PATH /usr/games:$PATH
COPY ./cron.sh /usr/bin/cow_cron.sh
RUN chmod +x /usr/bin/cow_cron.sh
ENTRYPOINT ["/usr/bin/cow_cron.sh"]From within the directory where you have your bash script and your Dockerfile, you should now be able to run docker build -t cowcron . and have that complete successfully. Now, you can run docker run -it cowcron and…
…it seems to do nothing but hang?
Aha, if you don’t pass a string into cowsay, it will wait for you to type something on STDIN. And while we intended to pass in the value of the SECRET_COW_SPEECH env var, we haven’t actually set that env var anywhere yet. We’ll do that with Nomad, but before we can do that, we have to put our Docker image somewhere were Nomad can find it.
By configuring a CircleCI job in our github repo, we can make sure that a fresh version of our Docker image gets built every time we make changes to our very important and complicated bash script. To do this, you’ll want to create a .circleci/config.yml file that looks something like this:
---
version: 2
workflows:
version: 2
build:
jobs:
- docker-upload:
filters:
branches:
only: main
jobs:
docker-upload:
docker:
- image: cimg/base:2020.01
steps:
- checkout
- attach_workspace:
at: "."
- setup_remote_docker:
docker_layer_caching: true
- run:
name: Build cowcron docker container
command: |
echo "$QUAY_PASSWORD" | docker login --username "$QUAY_USERNAME" --password-stdin quay.io
docker build -t quay.io/yourname/cowcron:$CIRCLE_SHA1 .
- deploy:
name: Push image to quay
command: |
docker push quay.io/yourname/cowcron:$CIRCLE_SHA1Before this will work (the yaks, they’re coming), you will have to set up this repository to work in Circle, and set up the QUAY_USERNAME and QUAY_PASSWORD as environment variables in the Project Settings for this Circle project. You’ll also need to set up this repository in your Quay account, including giving whichever account the Quay username and password are associated with write access to the Quay repo - if you don’t do this, the Circle job won’t have permissions to upload the new image once it’s created. Assuming you get all that configured, this Circle job should build and upload a new version of the Docker container on every commit to your main branch, tagging it with the hash of the last commit in the build.
Sound complicated? Yeah, it’s only gonna get worse.
The next step is to set up the Nomad job. Wherever you configure your Nomad job specs, you’ll want to create a new one for your delightful and very useful cow cron. It will look something like this:
job "cowcron" {
type = "batch"
periodic {
cron = "@daily"
}
group "cowgroup" {
task "cowtask" {
driver = "docker"
leader = true
config {
image = "quay.io/yourname/cowcron:HASH_FROM_CIRCLE”
}
vault {
policies = ["cowcron-vault-policy"]
}
template {
destination = "secrets/file.env"
env = true
splay = "5m"
data = <<-__EOF
SECRET_COW_SPEECH="{{ with secret `secret/cowcron` }}{{ .Data.data.secret_cow_speech }}{{ end }}"
__EOF
}
resources {
cpu = 256
memory = 256
}
}
}
}This will define a Nomad job that, using the Docker driver, will pull down your image from Quay and run it. (If your Quay repo is private, you’ll need to configure Nomad authentication to be able to pull the image.) The HASH_FROM_CIRCLE needs to match the tag of a valid Docker image in your quay repo. Because of the ENTRYPOINT in the Dockerfile that we defined, our cow script will run whenever this job is invoked, which uses the periodic stanza to run it daily. The template block in the Nomad jobspec is what allows you to set environment variables (in this case, the SECRET_COW_SPEECH that we were trying to set earlier) that will get magically put into the environment of your running Docker container.
Wait, hang on, what the heck is that Vault stuff in there?
Well, for some reason, this cron is one where a cow just says some Very Secret things out loud. Your security team would probably prefer that you VERY MUCH NOT DO THIS, WHY WOULD YOU DO THIS, but there are certainly real-world situations where a script might expect something like an API key to exist as an env var, and in those sorts of situations, your security friends would probably prefer you not pass those around in plaintext, so we’ll show you how to set up secret env vars with secrets from Vault.
As with the rest of this tutorial, there’s a lot of assumptions here, like you having a working Vault cluster set up already. The example above assumes that using the generic secret backend, you define a path called cowcron that has a key called secret_cow_speech, the value of which is the thing that your helpful cow will say.
Fun fact, if you typo the path or the key of the secret, Nomad will helpfully NOT throw an error and will happily continue on its way, filling that part of your template with absolutely nothing at all, so watch out for that.
The last piece of the puzzle here is a Vault policy that will allow this particular Nomad job to access only this particular secret. How do we do that? I’m glad you asked. For extra Devops Points, let’s set one up with Terraform:
data "vault_policy_document" "cowcron-vault-policy" {
rule {
path = "secret/cowcron”
capabilities = ["read"]
description = "keep it secret, keep it safe"
}
}
resource "vault_policy" "cowcron-vault-policy" {
name = "cowcron-vault-policy"
policy = data.vault_policy_document.cowcron-vault-policy.hcl
}Apply that in your Terraform setup of choice, and now, you should be able to run your Nomad job that talks to your Vault cluster so that a noisy little cow can yell your secrets into the void of a Docker container. You can either wait for whenever you scheduled your cron to run, or you can force it to run immediately by running nomad job periodic force cowcron. Congratulations, you have now deployed a cron job With Devops TM!
And with that, I’ll leave you with this meme, which is the only thing my brain finds funny these days:
