Kubernetes Operator journey EP4: controller-runtime VS knative/pkg for operator
In this episode, I will run a throughout comparison between controller-runtime and knative/pkg to see how it varies in terms of building Kubernetes operators.
1.Tools to automate the operator creation:
Controller-runtime based operator: there are several frameworks that can be used to facilitate the operator creation, including CR/CRD generation, RBAC generation, controller generation, etc.
You probably heard of multiple frameworks, but we suggest operator-sdk or kubebuilder. For tutorials on operator-sdk, you can visit: https://docs.okd.io/latest/operators/osdk-getting-started.html. For tutorials on kubebuilder, you can visit: https://book.kubebuilder.io/
Knative/pkg based operator: there is no framework available to generate everything, but as we introduced in previous episodes, knative/pkg based operator can use k8s.io/code-generator to generate source code of deepcopy,client,informer,lister, and use knative/pkg to generate source code of injection.
We have to say controller-runtime based operator is currently empowered by more automated tools than knative/pkg based operator. However, CR/CRD manifest and RBAC manifest remain the same, regardless of which package you decide to use for your operator.
2. Client used to access the resources:
Controller-runtime based operator: there is a universal client within controller-runtime to access all types of resources in Kubernetes, including the newly created custom resource.
Knative/pkg based operator: there will be three types of clients, if you use k8s.io/code-generator to generate the client code.
- A special client to access the custom resource
- A kubeclient to access the core resources of Kubernetes
- A dynamic client to access any resource in terms of GroupVersionResource or GroupVersionKind
3. How to register resources to monitor
Controller-runtime based operator: the controller has a Watch function to register the resources able to kick off the reconcile function. For example, if there is CR called CdnCluster created, to register this CR and all the deployments owned by this CR, we can use the following code:
Knative/pkg based operator: the generated informer will be used to register the resources to monitor. For example, if there is CR called KnativeServing created, with all the generated client and informer, to register this CR and all the deployments owned by this CR, we can use the following code:
4. How to register the new CRs in Kubernetes
Structurally, there may be some differences, but they both add the custom resources, by using runtime.Scheme within k8s.io/apimachinery.
Controller-runtime based operator: it is possible to involve multiple files to register the resources in Kubernetes. For example, the following snapshots demonstrate how the CR is registered:
Knative/pkg based operator: in Knative serving-operator, we use the following section to register the resource:
5. How to send events
They both use record.EventRecorder within k8s.io/client-go/tools, as the recorder for recording events with Kubernetes API.
Controller-runtime based operator: since manager within controller-runtime has encapsulated the EventRecorder, there is no need to implement anything.
Knative/pkg based operator: record.EventRecorder needs to be created in the reconciler, either get it from the controller or create it from scratch.
6. How to launch the controller
Controller-runtime based operator: controller-runtime has a type named manager, which can contain multiple controllers and launch them one by one. Here is an example of the source code in main.go. By the way, this manager provide Get function to retrieve the client and scheme.
Knative/pkg based operator: it has dramatically simplified the complexity of main.go, since Knative/pkg abstracts functions like, Main, MainWithContext
and MainWithConfig, which can be called directly to launch the controller. For example, in serving-operator, we created the main.go as below:
7.Publication in the operatorhub.io
Controller-runtime based operator: you can use operator-sdk as the framework, to generate the metadata necessary for you to submit the operator to operatorhub.io.
Knative/pkg based operator: operatorhub does not necessarily require that the operator has to be based on certain package, but this knative common package needs some default ConfigMaps. However, the metadata targetting operatorhub does not so far support ConfigMaps. There may some hacking work to do, if your operator need to create some ConfigMaps in manifest.
If there are any other differences you may notice between controller-runtime and knative/pkg based operators, welcome to comment below.
Follow Vincent, (and) you won’t derail!