Streamlining Kubernetes deployment with Jsonnet: Part 1
Basics
In these two articles I’m going to introduce jsonnet as a clean and powerful alternative to helm charts and kustomize for configuring kubernetes resources.
Jsonnet is a general-purpose language for generating JSON, and I needed to play with it for a while to find the best way to apply it to kubernetes. I also found the official drinks-based tutorial a bit overwhelming at first.
I hope this more kubernetes-focussed introduction is helpful. I am including some approaches which I’ve found to work well in practice. Part two will provide a complete worked example for a non-trivial application deployment.
Installation
If you want to try out the examples in this article, you’ll need to install the jsonnet tool.
There are two versions: the C++ version or the go version. The two are kept carefully in line feature-wise, but I suggest you use the go version, because it includes deb packages for Debian/Ubuntu.
$ wget https://github.com/google/go-jsonnet/releases/download/v0.17.0/jsonnet-go_0.17.0_linux_amd64.deb
$ sudo dpkg -i jsonnet-go_0.17.0_linux_amd64.debThere is also a formatting tool jsonnetfmt, available as a separate package from the go repository, or included with the C++ version.
Basics
Jsonnet has much in common with Javascript. You have the same data types available: number, string, object, array, boolean, null, function.
A jsonnet program is just an expression. You can provide this on the command line⁰:
$ jsonnet -e "1+2"
3Or you can put it in a file:
$ echo "1+2" >test.jsonnet
$ jsonnet test.jsonnet
3Like Javascript, you don’t need to quote object keys. You can include comments (using // or /* ... */ forms); and you are allowed trailing commas¹ on the last element of an object or array.
Put this source in a file and run it through jsonnet:
{
// Use the database id
id: 123,
greeting: 'hello world',
}The rendered output is valid JSON:
{
"greeting": "hello world",
"id": 123
}Note that in the output, the fields of the object are sorted, so greeting comes before id. This helps keep the output stable between runs, which is useful for version control and for checking when something has changed.
Functional operators
Unlike Javascript though, Jsonnet is a pure functional language. This means there are no mutating operations. To modify a value, you create a new value derived from the previous one. For example, to replace the last value of an array, you take the previous values and concatenate them with a new value.
local arr1 = [1,2,5];
local arr2 = arr1[0:2] + [3];
arr2To update a field of an object, you “add” another object: the result is a new merged object with fields from the first overridden by fields from the second.
local obj1 = {
foo: 123,
bar: 321,
};
local obj2 = {
bar: 456,
};
obj1 + obj2This is such a pervasive operation in jsonnet, that when the right-hand side is an object literal you can omit the + operator entirely. This program outputs exactly the same:
local obj1 = {
foo: 123,
bar: 321,
};
obj1 {
bar: 456,
}To help with deeply nested objects, a field marked with +: is merged into the existing value instead of replacing it. For example:
local Womble = {
metadata: {
name: 'Bulgaria',
labels: {
relation: 'uncle',
},
},
};Womble {
metadata+: {
namespace: 'wimbledon',
},
}
Here, instead of replacing the entire metadata value, we just add (or replace) the namespace field within it. This works down multiple levels too:
...
Womble {
metadata+: {
namespace: 'wimbledon',
labels+: {
env: 'production',
},
},
}Hopefully the application to kubernetes is already becoming apparent (and will be familiar to users of kustomize).
The +: syntax also works if the field contains an array value; the result is the concatenation of the arrays.
local obj = {
items: [123, 456],
};
obj {
items+: [789],
}Evaluation
Jsonnet is a lazily-evaluated language, which means that if a value is not used, it’s never evaluated — and therefore it doesn’t matter if it’s invalid in that context. Here’s an example:
local person = null;
local greeting = 'hello ' + person.name;
if person != null
then greeting
else 'Nobody is here'person.name would give an error if evaluated, but since greeting is not used in the output when person is null, everything is fine.
That example also introduced the conditional expression:
if bool then expr1 [else expr2]This is an expression which evaluates to a value, not a statement. It’s like bool ? expr1 : expr2 in Javascript, or expr1 if bool else expr2 in Python. If else expr2 is missing, then the value is null when the condition is false.
Iteration
There are no loops! If you want to create an array of similar objects dynamically, or an object with a variable set of fields, then you can use Python-style array comprehensions (or object comprehensions).
local domain = 'example.com';{
urls: [
'www-%d.%s' % [i, domain]
for i in std.range(1, 10)
],
}
My first kubernetes example
Let’s take a simple PersistentVolumeClaim resource. In YAML it looks like this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-pvc
labels:
app: docker-registry
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
# This is how much storage to request
storage: 20GiThere is an online conversion tool which can convert this automatically to jsonnet, although it loses any comments.
Here’s the result (after re-inserting the comment):
{
apiVersion: 'v1',
kind: 'PersistentVolumeClaim',
metadata: {
name: 'registry-pvc',
labels: {
app: 'docker-registry',
},
},
spec: {
accessModes: [
'ReadWriteOnce',
],
resources: {
requests: {
// This is how much storage to request
storage: '20Gi',
},
},
},
}Write this into a file pvc.jsonnet, process it using jsonnet pvc.jsonnet and it gets converted into true JSON.
$ jsonnet pvc.jsonnet
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"labels": {
"app": "docker-registry"
},
"name": "registry-pvc"
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "20Gi"
}
}
}
}That’s suitable for direct application to Kubernetes:
$ jsonnet pvc.jsonnet | kubectl apply -f -So far, so meh: it’s still just a static resource definition.
Parameterising via functions
It gets more interesting when we allow the user to parameterise the resource. You’ve seen one way already:
// This is how much storage to request
local volsize = '20Gi';{
apiVersion: 'v1',
kind: 'PersistentVolumeClaim',
metadata: {
name: 'registry-pvc',
labels: {
app: 'docker-registry',
},
},
spec: {
accessModes: [
'ReadWriteOnce',
],
resources: {
requests: {
storage: volsize, // << note here
},
},
},
}
Now the parameter is in an obvious place at the top of the file. However, the user still has to edit this file to adjust it.
What we want is to supply this as a separate parameter somehow. There are several ways to do this; the one I’m going to show here is to create a function value. Change pvc.jsonnet to:
function(conf) { // << note
apiVersion: 'v1',
kind: 'PersistentVolumeClaim',
metadata: {
name: 'registry-pvc',
labels: {
app: 'docker-registry',
},
},
spec: {
accessModes: [
'ReadWriteOnce',
],
resources: {
requests: {
storage: conf.volsize, // << note
},
},
},
}Note that unlike Javascript, the opening brace after function(conf) does not mark the function body: it’s simply the start of the object literal which the function returns². If you were returning some other value, such as a string, it wouldn’t be there.
If you run this file using just jsonnet pvc.jsonnet you’ll get an error, because a function can’t be rendered without giving its argument(s). This can be done inline at the command line:
$ jsonnet --tla-code "conf={volsize:'10Gi'}" pvc.jsonnet(“tla” stands for “top level argument”, and “code” because it’s a jsonnet value). Or you can put the arguments in a separate file, say conf.jsonnet:
{
volsize: '10Gi',
}and apply it:
$ jsonnet --tla-code-file conf=conf.jsonnet pvc.jsonnetThese work fine, and are certainly useful for testing, but that’s an awkward command to remember and type. So the way I prefer is to write a new jsonnet file which imports the other resource function and calls it. Let’s call it prod.jsonnet (for our production config):
local conf = {
volsize: '150Gi',
};
local pvc = import 'pvc.jsonnet';pvc(conf)
Now simply typingjsonnet prod.jsonnet generates the resource we want, with the parameters we want.
The function value from pvc.jsonnet is anonymous, but we bound it to local name pvc when importing. If you want to be really terse, you can eliminate the local names (which is how jsonnet evaluates the expression anyway):
(import 'pvc.jsonnet')({
volsize: '150Gi',
})Aside: you may have noticed that I passed an object (“conf”) for the configuration. At the moment there’s only one parameter, so I could have passed in volsize directly as a string. When I need more arguments, I could add more parameters to the function. But in practice, I find that a single configuration object works well and is flexible: if I add or remove parameters, less code has to change.
More function examples
There are several different ways to define functions, and it helps to be familiar with them all:
local add1 = function(x,y) x+y;local add2(x,y) = x+y;local lib = {
add3(x,y):: x+y,
};// Generate a result
[add1(1,2), add2(3,4), lib.add3(5,6)]
The eagle-eyed may ask, “why the double-colon in add3?” It’s not a typo! The double-colon defines a hidden object field, one which is not output when the object is rendered. Function values can’t be rendered, so if add3 were visible and you tried to print the value of lib, it would fail.
A normal field with : has “default” visibility. It is visible unless it’s overriding a hidden field, in which case the new field remains hidden. :: forces hidden, and ::: forces visible. This feature is very useful to hide partial results in objects which you don’t want to appear in the final output, and we’ll come back to it shortly.
The lib object demonstrates how to make a “library” of functions in jsonnet: just create an object containing one or more functions. You can put this object into a separate file — conventionally with extension .libsonnet — and then import it when required.
--- math.libsonnet ---
{
add(x,y):: x+y,
sub(x,y):: x-y,
}--- prog.jsonnet ---
local math = import 'math.libsonnet';
math.add(1,2)
Post-processing
So now we have a PVC resource. But what if the user wants to set a namespace? What if they want to set the storage class? What if they want to add annotations?
Well, we could add parameters for all of these things, in the style of helm, but then it becomes very messy — and we still won’t have anticipated all possible use cases.
// DON'T DO THIS!
function(conf)
local defaults = {
annotations: {},
namespace: 'default',
storageClass: null,
volumeName: null,
}; local c = defaults + conf; {
apiVersion: 'v1',
kind: 'PersistentVolumeClaim',
metadata: {
name: 'registry-pvc',
namespace: c.namespace,
labels: {
app: 'docker-registry',
},
annotations: c.annotations
},
spec: {
[if c.storageClass != null then "storageClassName"]: c.storageClass,
[if c.volumeName != null then "volumeName"]: c.volumeName,
accessModes: [
'ReadWriteOnce',
],
resources: {
requests: {
storage: c.volsize,
},
},
},
}
There’s nothing actually wrong with this: it’s just wasted effort and unneeded complexity, because the user already has the tools to patch the resource once it has been returned.
Let’s stick with the original, simple function version of pvc.jsonnet. Edit prod.jsonnet so it contains:
local conf = {
volsize: '150Gi',
};
local pvc = import 'pvc.jsonnet';// Apply whatever adjustments you need here
pvc(conf) {
metadata+: {
namespace: 'prod',
},
spec+: {
storageClass: 'local-path',
},
}
Run it, and here’s the result:
$ jsonnet prod.jsonnet
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"labels": {
"app": "docker-registry"
},
"name": "registry-pvc",
"namespace": "prod"
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "150Gi"
}
},
"storageClass": "local-path"
}
}We now have the best of both worlds. There’s helm-style configuration for important parameters (such as those which have no sensible default, or need to be consistent between multiple resources); and flexible kustomize-style local overrides to suit the local environment.
self and super
For the last part of this article, I want to introduce self and super.
self refers to the current object, at the current level in the object hierarchy.
{
foo:: 123,
bar:: 456,
baz:: 789,
items: [self.foo, self.bar, self.baz],
}This resolves to:
{
"items": [
123,
456,
789
]
}(foo, bar and baz are missing in the output because I chose to make them hidden — although I didn’t have to). Notice that if we override one of the fields in a derived object, the dependent fields pick up the new value too.
local obj = {
foo:: 123,
bar:: 456,
baz:: 789,
items: [self.foo, self.bar, self.baz],
};obj {
foo: 321,
}
// renders to: {"items": [321, 456, 789]}
Combined with lazy evaluation, this gives a different way to build a dynamic kubernetes resource, without using a function at all. Below, I have made a hidden conf value at the top level:
// --- pvc2.jsonnet ---
{
local outer = self, conf:: {
volsize: '20Gi',
}, apiVersion: 'v1',
kind: 'PersistentVolumeClaim',
metadata: {
name: 'registry-pvc',
labels: {
app: 'docker-registry',
},
},
spec: {
accessModes: [
'ReadWriteOnce',
],
resources: {
requests: {
storage: outer.conf.volsize, // << note
},
},
},
}
This jsonnet file can be run as-is. But a consumer can also override conf parameters, and it will render with the new configuration:
local pvc = import 'pvc2.jsonnet';
pvc {
conf+: {
volsize: '150Gi',
}
}You can see that self has different values at different positions in the tree of objects, so I needed to bind it to another name to access conf “at a distance”³. This is all very cool, but I suggest you don’t over-use it: functions are a clearer way of passing parameters.
super is used when combining two objects (obj1 + obj2), and in obj2 you want to access a value from obj1 which would otherwise be hidden by the replacement field from obj2. This is especially useful for arrays.
For array transformations other than a simple append, you’ll need to replace the field with an entirely new array value derived from the previous one. For example, this is how you can remove the front value from an array:
local obj = {
foo:: 123,
bar:: 456,
baz:: 789,
items: [self.foo, self.bar, self.baz],
};obj {
items: super.items[1:],
}
This is replacing the items field with an expression which uses the items value from the first object. Think of the first object as sitting in a layer beneath the second: it’s hidden by the second, but still accessible via super.
Documentation
I hope that has given you a basic flavour of jsonnet, and now you may wish to review the documentation. The information is spread around a bit⁴: I found the language reference to be the most approachable, but it does not mention some features which are shown in the tutorial, and vice versa. You should also skim through the standard library.
In part two, I’m going to show some specific patterns that I’ve found helpful when composing kubernetes resources, with a full worked example.
⁰ Beware, the expression does not necessarily follow -e immediately; there could be other flags. So jsonnet -e '-42' fails, because it treats -4 as if it were a flag. To fix, use jsonnet -e -- '-42'
¹ It’s good style to include the trailing commas, and jsonnetfmt will add them for you.
² When returning an object, you could write either:
function(args) {
field: value,
}or:
function(args)
{
field: value,
}jsonnetfmt permits (and preserves) both styles. The former uses less indentation, and is consistent with how you create nested objects. The latter lets you insert ‘local’ definitions prior to the returned object without re-indenting (although you can also put ‘local’ definitions inside the object). Use whichever you prefer.
³ There’s also a special variable $ which means “the top-level object” in the current expression.
⁴ In addition: some language features are mentioned but not defined, such as arr[10:20:2] — you’re expected to go to Python documentation to understand that. And some keywords are not documented at all. But these documentation weaknesses are really the only problems I’ve found with jsonnet.
