Simple templating engine in Bash

TL;DR: Checkout the template engine at github.

When working with pods in Kubernetes I had the need to define environment variables for the pod from a file. This is currently not supported in the currently released version of Kubernetes (1.1). What I wanted to do was to have a template file, for example pod-template.yaml, that looked like this:

kind: "ReplicationController"
apiVersion: "v1"
metadata:
  name: "my-pod-{{VERSION}}"
  labels:
    name: "my-pod"
    version: "{{VERSION}}"
spec:
  replicas: 2
  selector:
    name: "my-pod"
    version: "{{VERSION}}"
  template:
    metadata:
      labels:
        name: "my-pod"
        version: "{{VERSION}}"
    spec:
      containers:
        - name: "my-pod"
          image: "gcr.io/my-project/my-pod:{{VERSION}}"
          env:
            - name: "TIMEOUT"
              value: {{TIMEOUT}}
            - name: "URL"
              value: "{{URL}}"
          ports:
            - containerPort: {{API_PORT}}
              name: "api"
              protocol: "TCP"
            - containerPort: {{ADMIN_PORT}}
              name: "admin"
              protocol: "TCP"

kind: "ReplicationController"

apiVersion: "v1"

metadata:

labels:

version: "{{VERSION}}"

spec:

replicas: 2

selector:

version: "{{VERSION}}"

template:

metadata:

labels:

version: "{{VERSION}}"

spec:

containers:

- name: "my-pod"

image: "gcr.io/my-project/my-pod:{{VERSION}}"

env:

- name: "TIMEOUT"

value: {{TIMEOUT}}

- name: "URL"

value: "{{URL}}"

ports:

- containerPort: {{API_PORT}}

protocol: "TCP"

- containerPort: {{ADMIN_PORT}}

protocol: "TCP"

and be able to replace things inside the double brackets (placeholders) with values defined externally to the pod-template.yaml. This is of course nothing new and there’s a gazillion of various templating engines out there for various programming languages. But after quite a lot of googling and looking through stackoverflow I couldn’t find anything that worked the way I wanted to in bash.

What made this particular use case a bit tricky was that some of the placeholder values were defined in a file while others had to be passed as parameters to the template engine on usage. For example the value of the {{VERSION}} placeholder is generated by our continuous integration server while all others ({{TIMEOUT}}, {{URL}} etc) are defined in an (encrypted) configuration file checked into our code repository. So I needed a template engine that could handle both of these cases at the same time. The closest thing I found was the bash-templater projected created by Sébastien Lavoie which allowed me to do like this:

$ VERSION=1.2.3 TIMEOUT=1000 URL="http://www.google.com" API_PORT=8080 ADMIN_PORT=8081 \ 
  ./templater.sh pod-template.yaml

1 2	$ VERSION=1.2.3 TIMEOUT=1000 URL="http://www.google.com" API_PORT=8080 ADMIN_PORT=8081 \ ./templater.sh pod-template.yaml

I.e. it takes environment variables and replaces the placeholders defined in the template file with these values. The outcome looks like this:

kind: "ReplicationController"
apiVersion: "v1"
metadata:
  name: "my-pod-1.2.3"
  labels:
    name: "my-pod"
    version: "1.2.3"
spec:
  replicas: 2
  selector:
    name: "my-pod"
    version: "1.2.3"
  template:
    metadata:
      labels:
        name: "my-pod"
        version: "1.2.3"
    spec:
      containers:
        - name: "my-pod"
          image: "gcr.io/my-project/my-pod:1.2.3"
          env:
            - name: "TIMEOUT"
              value: 1000
            - name: "URL"
              value: "http://www.google.se"
          ports:
            - containerPort: 8080
              name: "api"
              protocol: "TCP"
            - containerPort: 8081
              name: "admin"
              protocol: "TCP"

kind: "ReplicationController"

apiVersion: "v1"

metadata:

labels:

version: "1.2.3"

spec:

replicas: 2

selector:

version: "1.2.3"

template:

metadata:

labels:

version: "1.2.3"

spec:

containers:

- name: "my-pod"

image: "gcr.io/my-project/my-pod:1.2.3"

env:

- name: "TIMEOUT"

value: 1000

- name: "URL"

value: "http://www.google.se"

ports:

- containerPort: 8080

protocol: "TCP"

- containerPort: 8081

protocol: "TCP"

The problem (again) was that our configuration was defined in a file (config.properties) that looked something like this:

# The timeout in milliseconds
TIMEOUT=1000 
# The URL
URL="http://www.google.com" 
# External API port
API_PORT=8080 
# Internal admin port
ADMIN_PORT=8081

# The timeout in milliseconds

TIMEOUT=1000

# The URL

URL="http://www.google.com"

# External API port

API_PORT=8080

# Internal admin port

ADMIN_PORT=8081

So I ended up forking bash-templater to my github repository and added support for passing in values as parameters as well as reading values from a file. Usage:

$ VERSION=1.2.3 ./templater.sh pod-template.yaml -f config.properties > pod.yaml

1	$ VERSION=1.2.3 ./templater.sh pod-template.yaml -f config.properties > pod.yaml

Volia, all requirements fulfilled! The result of the transformation is stored in the pod.yaml file.

I also added some other tweaks, for example suppression of warning messages (that will otherwise be included in the resulting file and mess up the outcome) if you define entries in config.properties that are not found as placeholders in template. Just add the -s (silent) argument to ignore these warnings:

$ VERSION=1.2.3 ./templater.sh pod-template.yaml -f config.properties -s > pod.yaml

1	$ VERSION=1.2.3 ./templater.sh pod-template.yaml -f config.properties -s > pod.yaml

I hope that this will come to use for someone else as well.

12 Comments

Sami Niiranen February 26, 2016

Why not just use envsubst ?

- johanhaleby March 4, 2016
  
  I’m not familiar with envsubst, could you please elaborate?
  
Kubernetes Caveats and Workarounds March 4, 2016

[…] resides in the app-config.properties file. I’ve described this workaround in more detail here. Kubernetes 1.2 will give us ConfigMaps that’ll probably address the need for this […]

James March 10, 2016

A simple solution: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine

James March 10, 2016

export var1=”test”
envsubst generated.yaml

$var1 in template.yaml, and will substituted as “test” in generated.yaml

But, i cannot find a way to escape if using envsubst.
e.g. i want to keep ${var_to_keep} not substituted by envsubst

Luis April 15, 2016

Johan, hello. Once again, thanks.

I’ve run into an issue where supplying a template with no variables (ie: {{}}) throws a warning and an error:

Warning: No variable was found in services/portal.yml, syntax is {{VAR}}
sed: 1: “services/portal.yml”: unterminated substitute in regular expression

I meant to open an issue in the repo but they’re disabled. This is just a heads up. I’ll try to submit a PR asap.

- johanhaleby April 17, 2016
  
  Hi Luis,
  
  Thanks for your comment. I actually didn’t know that issues were disabled, I’ll fix this. Also a PR would be great 🙂
  
  Regards,
  /Johan
  
Mark May 3, 2017

Thanks for your article! Are you going to submit your work upstream to Sébastien’s repo?

- Johan Haleby May 4, 2017
  
  I already have provided a PR, see https://github.com/lavoiesl/bash-templater/pull/1, but it has not been accepted yet.
  
  - Mark May 13, 2017
    
    Ah, I missed that. Cool!
    
    I drew some inspiration from your and lavoiesl’s work: https://github.com/relaxdiego/renderest
    
zoobab August 27, 2018

Just wanted to say that j2cli does the same, as uses jinja syntax for templating.

You can also source properties files as well:

https://github.com/kolypto/j2cli

- Johan Haleby August 28, 2018
  
  Thanks for the tip

12 Comments

Leave a Reply Cancel reply