bashdevops

Simple templating engine in Bash

TL;DR: Checkout the template engine at github.

When working with pods in Kubernetes I had the need to define environment variables for the pod from a file. This is currently not supported in the currently released version of Kubernetes (1.1). What I wanted to do was to have a template file, for example `pod-template.yaml`, that looked like this:

kind: "ReplicationController"
apiVersion: "v1"
metadata:
  name: "my-pod-{{VERSION}}"
  labels:
    name: "my-pod"
    version: "{{VERSION}}"
spec:
  replicas: 2
  selector:
    name: "my-pod"
    version: "{{VERSION}}"
  template:
    metadata:
      labels:
        name: "my-pod"
        version: "{{VERSION}}"
    spec:
      containers:
        - name: "my-pod"
          image: "gcr.io/my-project/my-pod:{{VERSION}}"
          env:
            - name: "TIMEOUT"
              value: {{TIMEOUT}}
            - name: "URL"
              value: "{{URL}}"
          ports:
            - containerPort: {{API_PORT}}
              name: "api"
              protocol: "TCP"
            - containerPort: {{ADMIN_PORT}}
              name: "admin"
              protocol: "TCP"

and be able to replace things inside the double brackets (placeholders) with values defined externally to the `pod-template.yaml`. This is of course nothing new and there’s a gazillion of various templating engines out there for various programming languages. But after quite a lot of googling and looking through stackoverflow I couldn’t find anything that worked the way I wanted to in bash.

What made this particular use case a bit tricky was that some of the placeholder values were defined in a file while others had to be passed as parameters to the template engine on usage. For example the value of the `{{VERSION}}` placeholder is generated by our continuous integration server while all others (`{{TIMEOUT}}`, `{{URL}}` etc) are defined in an (encrypted) configuration file checked into our code repository. So I needed a template engine that could handle both of these cases at the same time. The closest thing I found was the bash-templater projected created by Sébastien Lavoie which allowed me to do like this:

$ VERSION=1.2.3 TIMEOUT=1000 URL="http://www.google.com" API_PORT=8080 ADMIN_PORT=8081 \ 
  ./templater.sh pod-template.yaml

I.e. it takes environment variables and replaces the placeholders defined in the template file with these values. The outcome looks like this:

kind: "ReplicationController"
apiVersion: "v1"
metadata:
  name: "my-pod-1.2.3"
  labels:
    name: "my-pod"
    version: "1.2.3"
spec:
  replicas: 2
  selector:
    name: "my-pod"
    version: "1.2.3"
  template:
    metadata:
      labels:
        name: "my-pod"
        version: "1.2.3"
    spec:
      containers:
        - name: "my-pod"
          image: "gcr.io/my-project/my-pod:1.2.3"
          env:
            - name: "TIMEOUT"
              value: 1000
            - name: "URL"
              value: "http://www.google.se"
          ports:
            - containerPort: 8080
              name: "api"
              protocol: "TCP"
            - containerPort: 8081
              name: "admin"
              protocol: "TCP"

The problem (again) was that our configuration was defined in a file (`config.properties`) that looked something like this:

# The timeout in milliseconds
TIMEOUT=1000 
# The URL
URL="http://www.google.com" 
# External API port
API_PORT=8080 
# Internal admin port
ADMIN_PORT=8081

So I ended up forking `bash-templater` to my github repository and added support for passing in values as parameters as well as reading values from a file. Usage:

$ VERSION=1.2.3 ./templater.sh pod-template.yaml -f config.properties > pod.yaml

Volia, all requirements fulfilled! The result of the transformation is stored in the `pod.yaml` file.

I also added some other tweaks, for example suppression of warning messages (that will otherwise be included in the resulting file and mess up the outcome) if you define entries in `config.properties` that are not found as placeholders in template. Just add the `-s` (silent) argument to ignore these warnings:

$ VERSION=1.2.3 ./templater.sh pod-template.yaml -f config.properties -s > pod.yaml

I hope that this will come to use for someone else as well.

13 thoughts on “Simple templating engine in Bash

  1. export var1=”test”
    envsubst generated.yaml

    $var1 in template.yaml, and will substituted as “test” in generated.yaml

    But, i cannot find a way to escape if using envsubst.
    e.g. i want to keep ${var_to_keep} not substituted by envsubst

  2. Johan, hello. Once again, thanks.

    I’ve run into an issue where supplying a template with no variables (ie: {{}}) throws a warning and an error:

    Warning: No variable was found in services/portal.yml, syntax is {{VAR}}
    sed: 1: “services/portal.yml”: unterminated substitute in regular expression

    I meant to open an issue in the repo but they’re disabled. This is just a heads up. I’ll try to submit a PR asap.

    1. Hi Luis,

      Thanks for your comment. I actually didn’t know that issues were disabled, I’ll fix this. Also a PR would be great 🙂

      Regards,
      /Johan

Leave a Reply

Your email address will not be published. Required fields are marked *