How to create conversion webhook for my operator with operator-sdk

You must have searched online a lot of other materials to learn how to create the webhooks with operator-sdk before reading this article. It is lucky for you to find this one, because this article will guarantee you an error-free process, and you don’t even need to look for more.

Prerequisites:

  • Git
  • Ko
  • Kubernetes cluster. You can install Minikube or Docker Desktop for you environment. Any Kubernetes service will work fine. I was using Kubernetes v1.24.1.

Prepare your workstation:

  1. Build and install operator-sdk with the latest commit

You can download the the latest v1.22.1 version here or build the operator-sdk binary based on the commit `87cdc50`. Make sure you build one based on a commit, no earlier than that.

To build the binary from the source code, go to the path $GOPATH/src/github.com/operator-framework in your terminal:

Download the source code of operator-sdk:

Go to the home directory of the operator-sdk:

Build the binaries:

The binary operator-sdk is automatically installed under $GOPATH/bin. Make sure your environment variable $PATH includes $GOPATH/bin. You can verify the installation with the command:

It will show you the path of the operator-sdk.

2. Install cert-manager

Since the webhook requires a TLS certificate that the apiserver is configured to trust, install the cert-manager with the following command:

You can visit the release page to choose the specific version you need.

Create the classic operator example:

Let’s leverage the well-known memcached-operator presented by all operator-sdk tutorials. However, we will create v1alpha1 and v1beta1 as two versions of the CRD. Use Git to manage your source code, because we plan to create two branches for the source code, using one for the v1alpha1 CRD and the other for the v1beta1 CRD.

Create the project:

Create a work directory under $GOPATH/src for the project. This article picks up the path github.com/example.

Initialize the project with Git:

Initialize the project with operator-sdk:

The domain and repo names can be changed based on your needs.

Create a new API and controller for the v1alpha1 version:

This command scaffolds the Memcached resource API and the controller. Next, let’s change the file api/v1alpha1/memcached_types.go .

Define the API for the Memcached Custom Resource(CR) as bellow:

Update the generated code by invoking the controller-gen utility to update the api/v1alpha1/zz_generated.deepcopy.go:

Generate the CRD manifests:

Let’s implement the controller in the easiest way. The reconcile loop does nothing except printing a message, and the controller only watches the changes on the newly created CR and the deployments it owns.

Go to the file controllers/memcached_controller.go . Change the function Reconcile into:

We hardcoded the pod names just for testing purpose.

Change the function SetupWithManager into

Specify permissions and generate RBAC manifests by adding the following contents:

Generate the ClusterRole manifest at config/rbac/role.yaml :

Change the CR sample at config/samples/cache_v1alpha1_memcached.yaml into the following contents:

By far, we have created the basic structure of the memcached operator for the v1alpha1 version CRD.

We would like to add the binary under bin, so we can remove the line bin in the file .gitignore.

Let’s save the work with the lovely Git. Add the following folders and files:

Leverage the git command to save the commit:

Create a branch to save the work of v1alpha1 CRD:

Build and test the operator:

You can choose any image repository to save the images. In the following steps, we take docker.io as the image repository. Specify the variable $USER and build the image:

We use the tag v0.0.1 for v1alpha1 CRD. Replace <name> with your name registered with docker.io.

After the image is successfully published, run the following command to deploy the operator:

Check the deployment of the operator with:

We can see the output as below:

Check the log with the command:

We need to specify the container’s name manager here, as the deployment launched two containers.

Create the v1alpha1 CR with the command:

Once this command runs, we should see a new log message:

This means the reconcile loop is called, as the CR is created.

Check the CR:

This command will yield the contents of the CR in the yaml format as below.

Now we have everything for v1alpha1 CRD in place.

To remove the CR:

To remove the operator:

Create a new CRD version v1beta1:

Switch back to the master branch, and continue the development.

Create the new v1beta1 API:

We do not need to create the controller, because we have already had one. We only need the resource. Make the following options after running the above command:

The only thing we need to change is to allow the controller to reconcile based on the v1beta1 CR, not the v1alpha1 CR any more. To develop a Kubernetes operator, avoid reconciling on multiple versions of the same CRs for the controllers.

Create a field called replicaSize for the v1beta1 CRD. This is the crucial and only change comparing to v1alpha1 CRD. Let’s change the file api/v1beta1/memcached_types.go .

Define the API for the Memcached Custom Resource(CR) as bellow:

Add the marker +kubebuilder:storageversion to indicate v1beta1 will be the storage version:

Go to the file controllers/memcached_controller.go, and replace all existences of cachev1alpha1 with cachev1beta1, and all v1alpha1 with v1beta1.

Change the import from

to

Change the function Reconcile into

Change the function SetupWithManager into

Make sure the controller switch to watch v1beta1 CR.

Then, update the generated code and regenerate the CRD manifests:

Both v1alpha1 and v1beta1 CRDs are serving, but only v1beta1 CRD will be stored.

Change the CR sample at config/samples/cache_v1beta1_memcached.yaml into the following contents:

Create the conversion webhook for v1beta1 resource:

Next, we need to implement the conversion.Hub and conversion.Convertible interfaces for your CRD types. We will leverage the v1beta1 as the storage version and the Hub, which means any resource version can convert into v1beta1.

Create a file named memcached_conversion.go under api/v1beta1 with the contents:

The v1alpha1 resource needs to implement the conversion.Convertible interface, so that it is able to convert into and from v1beta1 resource. In this example, the attribute size in v1alpha1 matches replicaSize in v1beta1.

Create a file named memcached_conversion.go under api/v1alpha1 with the contents:

Do not forget to set the ObjectMeta.

Update the generated code and regenerate the CRD manifests once more:

Enable the webhook and certificate manager in manifests:

Go through a few kustomization.yaml files under config/crd, config/default, and config/webhook, and do a few uncomment and comment actions.

For config/crd/kustomization.yaml, uncomment the following lines

into

For config/default/kustomization.yaml, uncomment the following lines

into

and all the line below vars:

For config/webhook/kustomization.yaml, comment the following line:

as

Change the file config/crd/patches/webhook_in_memcacheds.yaml, by adding conversionReviewVersions: [“v1alpha1”, “v1beta1”] under spec.conversion.webhook like:

Save the v1beta1 work with Git:

Commit the change:

Build the image for v1beta1:

We use the tag v0.0.2 for v1beta1 CRD. Replace <name> with your name registered with docker.io.

Deploy the operator with v1beta1 resource:

Check the deployment of the operator with:

Check the log with the command:

Still create the v1alpha1 CR with the command:

This time, we will get the v1beta1 resource save in the cluster. The v1alpha1 resource is automatically converted into v1beta1.

Check the CR:

We will get something like:

The resource is saved in the cluster in terms of v1beta1, though created by v1alpha1. The field size: 3 of v1alpha1 was converted into replicaSize: 3 of v1beta1 in the storage with the webhook.

If you install the deployment of v1alpha1 first, and then install the deployment of v1beta1, you probably still see the `apiVersion: cache.example.com/v1alpha1`. The kubectl command leverages cache mechanism. That is why you still see the old APIVersion, even if you update the CRD to v1beta1. To remove the cache of kubectl, run the command:

Or you can run the following command to refresh the resources:

After running one of the above commands, you can try to check the CR again.

Migrate the existing v1alpha1 resource into v1beta1 resource:

The CRD define the storage version, but it only applies to the new resource creation. How shall we deal with the resources at the older versions, which have relady existed in the cluster?

Based on the offical Kubernetes documents, we can follow the instructions here. However, is there a better way, easier and automated?

YES!

We can leverage the tool, knative.dev/pkg/apiextensions/storageversion/cmd/migrate, available in Knative common package, to migrate the existing resources.

Switch back to master branch:

Add knative.dev/pkg v0.0.0–20210309024624–0f8d8de5949d into the file go.mod.

Create a directory called post-install under config/, to host all the yamls regarding the migration.

Create config/post-install/tools.go with the contents:

The purpose of this file is used to import the migration library.

Create config/post-install/clusterrole.yaml with the contents:

Create config/post-install/serviceaccount.yaml with the contents:

Create config/post-install/storage-version-migrator.yaml with the contents:

Specify the arg “memcacheds.cache.example.com” directly for this tool, so it will do the conversion.

Sync-up the go.sumby running

Generate the dependencies for the project:

Build the image for the migration tool:

We also tag it with 0.0.2, since it is the conversion to v1beta1. The image will be published at docker.io/$USER/migrate:0.0.2.

Replace image: ko://github.com/example/memcached-operator/vendor/knative.dev/pkg/apiextensions/storageversion/cmd/migrate with image:docker.io/$USER/migrate:0.0.2, in the file config/post-install/storage-version-migrator.yaml.

By far, the job is ready for the resource migration from v1alpha1 to v1beta1.

Save the work with Git:

Save it into another branch called v1beta1:

Demostrate how the resource migration works:

There are two git branches available: v1alpha1 and v1beta1. Make sure the Kubernetes cluster has a clean environment with no memcached operator installed or a fresh new cluster to run the following steps, but do not forget to install the cert-manager.

Go to the v1alpha1 branch:

Install the operator with the v1alpha1 resource:

Create the v1alpha1 resource:

Now, we have got the v1alpha1 resource saved in the cluster with the v1alpha1 memcached operator.

Verify the CR with the command:

It should be saved as v1alpha1 as below:

Go to the v1beta1 branch:

Install the operator with the v1beta1 resource:

The older version of the memcached operator is replaced with the newer version, but the it is still the v1alpha1 resource saved in the cluster.

Check the status of the CRD:

We can see the storage version:

Run the following command to migrate the resource:

Check the status of the CRD:

We can see the storage version has changed into:

Let’s check the CR:

Please be aware that the CR does not change immeieately after the migration job is complete. It may take a few minutes to accoumplish the transition. Once the migration is done for the CR, we can get the CR as below:

The size: 3 in v1alpha is converted into replicaSize: 3 in v1beta1.

Again, do not forget to clean up the cache of kubectl, if you still see `cache.example.com/v1alpha1` in the APIVersion. The command is

Webhook wants to mess up with me? Too young! Too simple! Sometimes naive!

This is how you create conversion webhook with operator-sdk to convert resources among different versions, and how you can migrate existing resources from the old version to the new version.

Follow Vincent, (and) you won’t derail!

--

--

A Chinese software engineer, used to study in Belgium and currently working in US, as Knative & Tekton Operator Lead and Istio Operator Contributor.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vincent Hou

A Chinese software engineer, used to study in Belgium and currently working in US, as Knative & Tekton Operator Lead and Istio Operator Contributor.