Kubernetes Operator Design Patterns: how to make complex custom resources user-friendly

The Kubernetes Operator offers custom resources for users to configure their applications or services. The custom resource is the only source of truth to configure the operand. However, the operand the operator manages, can be complex enough, so that the custom resource itself can become relatively complex as well, in order to cover all the configuration options. Human operators can certainly work on and edit the custom resources directly, but it requires the them to understand the whole structure of the custom resource, knowing where exactly to put the parameters into the correct fields. They also need to remember all the existing configurations, when they try to add new configurations. As the custom resource grows complex, the configuration human operators manage also becomes complex.

An example of Kubernetes Operator with complex CRDs:

• Knative Operator: link to the CRDs

How to make it easier for the user to modify complex custom resources?

Here is the diagram on how the end-user manages Kubernetes cluster the via the Kubernetes operator:

There is nothing we can do to the interaction from the custom resources to the Kubernetes cluster, since it is the nature of the Kubernetes operators to leverage the Kubernetes capabilities to manage the software. What we can design for the users is to change the way they interact to the custom resources as the part indicated below.

Instead of creating and editing the custom resources directly, we can create a layer that is able to receive the parameters, and convert them into the custom resources with the correct configuration. This layer shields the users from the complexities of orchestrating the custom resources with the new and the existing configurations.

A good candidate of this layer is a CLI tool. This CLI should be capable of doing the following two tasks:

• Able to accept the parameters from the user in a systematic way, so that the user only cares the new configuration.
• Able to retrieve the existing custom resource, so that it can merge the new configuration with the existing ones, and apply the new changes with the existing configurations.

We can describe the interaction with custom resources with the following diagram:

It is straightforward for all CLI implementations to support and accept the parameters. If you implement the CLI tool in Golang, you can use the library, like Cobra.

As for Knative Operator, we implemented a CLI plugin, which is able to receive user-input parameters, retrieve the existing custom resources, and apply the changes to the custom resources. You can access the CLI plugin for Knative Operator here.

How about integrating the new configuration with the existing configurations for the custom resource?

Different programming languages have different libraries to manipulate YAML format content. We focus on Golang in this article. After we define the CRDs in the Golang Kubernetes operator, we can generate client code the operator to access the custom resource. In the CLI tool, we can use the generated client to get the existing CR, containing the existing configurations.

To modify the existing CR with the new configuration, one option is to change the CR object directly, and update the CR against the cluster. It sounds easy, but it is not as easy as we think, because the structure of the CR is complicated. The user can configure any field, so in the business logic, we probably need to iterate all the fields to determine where to reflect the changes.

Another option is to convert the Golang object into YAML format content, and change the yaml accordingly to apply against the cluster. There are several libraries available to parse and edit the yaml content: helm, kustomize, etc. They used to be my candidates to be honest, but all of them are lack of the capability to use the existing CR as the template. Helm and Kustomize are both good at merging the existing template with the parameters and the values. It is a two-way merging. What we need here is like a three-way merging: existing CR, template with parameters and the values.

Is there a Golang library able to parse the yaml in this three-way merging mechanism? Yes. The answer is Carvel-ytt, a templating tool that understands YAML structure. It helps you easily configure complex software via reusable templates and user provided values. With this library, we can use the yaml of the existing CR as the base template, the template yaml with parameters as the overlay, the value file as the input of the values. Carvel-ytt is able to use three of these yaml content as the input to generate the output, which is ultimately the final CR we will apply against the cluster. In the business logic, no matter what fields we would like to change in the CR, we can always convert it into using the existing CR as the base, parameterizing what need to be changed as overlay, and using the values of the parameters as the values. Carvel-ytt can generate the output based on the three yaml files. Finally, we apply the content of the output to reflect the changes of the custom resource.

All changes on the custom resource can be converted into the following steps:

1. Retrieve the existing custom resource in the Kubernetes cluster and transform the Golang object into the content in YAML format.
2. Create the overlay template to define what fields need to be changed.
3. Create the file of values based on the overlay template.
4. Three of the above files can be taken as the input to the carvel-ytt interface to generate the ultimate content in YAML.

You can find the CLI plugin of the Knative Operator as one reference implementation of the design pattern described as above. There is a struct called YttProcessor to generate the output based on the input of the base, overlay and value:

This CLI tool wraps up the complexity of merging the new configuration with the existing configurations for the custom resources. The users only focus on the parameters they want to configure without knowing the structure of the custom resources and existing configurations. Kubernetes operators ease down the burden to manage Kubernetes applications. The CLI tool can ease down the effort to manage the complex custom resources. I hope this article helps with your design of Kubernetes Operators.

Don’t want to derail? Follow Vincent!