Kubernetes Operator journey EP3: how to create a Kubernetes Operator based on Knative common packages

As we summarized in EP2, there are 8 steps to walk through, in order to create an operator. In this article, I will explain each step in details, by taking Knative eventing-operator as an example.

Knative Eventing is a system that is designed to address a common need for cloud native development and provides composable primitives to enable late-binding event sources and event consumers. Knative Eventing Operator aims to deploy and manage Knative Eventing in an automated way. It is an honor for me to take the initial development work of this operator. Since this is an official project under the Knative community, we choose Knative common packages as the foundation to build the controller, so that the consistent project structure can make sure contributors in the community have no hassle to shift their focuses from one project to another. Although, the common packages originate from and reside in Knative, any project can leverage it to build the operator, or controller of any other category. Now let’s create this eventing-operator.

1.Define the CustomResourceDefinition(CRD). It is straightforward that we need to define the name, group, version and kind as defined for any CRD. One special property we need to think of is the scope of this CRD. Should it be namespace-scoped or cluster-scoped? From my perspective, Knative Eventing will create cluster-scoped resources, but the most important deployments are under a certain namespace. I prefer to put the CR of the operator under the same namespace as the deployment, so I define the scope as namespaced. Besides that, I define some basic rules to validate the CRD properties. It is your option to find a location for the CRD files. For eventing-operator, the CRD is put under the directory config/crds, you can locate it via this link.

2.Define the Role-based access control (RBAC) of the operator. The CR to be created in the operator will be the owner of all the resources of Knative eventing. The CR is namespace-scoped, but the resources of Knative eventing can be namespace-scoped or cluster-scoped, so we need to define both the Role for namespace-scoped resources, and the ClusterRole for the cluster-scoped resources. We need to associate verbs with the resources under the apiGroups.

• To see how we define the Role and ClusterRole, please visit role.yml under the directory config.
• We create one service account called knative-eventing-operator for the operator in service_account.yaml. There is only one CR in this operator. It is sufficient to have only one service account.
• The RoleBinding and ClusterRoleBinding are also created to grant the service account the access in role_binding.yaml.

3.Create the deployment for the operator. We determine to launch a deployment, which creates a pod running constantly as a service for the eventing operator. Some crucial properties we need to think of include the service account name, image path for the container, and some environment variables. They are all easy to understand for this operator. You can find the manifest file here.

4.Create the CR object for extraction. We put all the source code files under the directory pkg/apis/eventing/v1alpha1. We create a file named knativeeventing_types.go, to include the type of the CR, Eventing. We also need to have the type EventingSpec, defining the desired state, and the type EventingStatus, indicating the actual state. Besides them, we probably need the type EventingList to map the list of the CRs.

If we would like to generate the openAPI definition file to be used in open API spec, we should add the tag “+k8s:openapi-gen=true” as comment for the types.

Since the CR is newly introduced in the kubernetes, we also need client code to access it via CRUP operations. We need to add the tag “+genclient” above the type of the CR, to generate default client verb functions (create, update, delete, get, list, update, patch, watch and depending on the existence of .Status field in the type the client is generated for also updateStatus).

We specifies runtime.Object interface as the interface to generate deepcopy for the type, by adding the tag “+k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object”.

To change and retrieve the status of the eventing CR, we create a file called knativeeventing_lifecycle.go, using apis.NewLivingConditionSet in knative/pkg, to access/modify the conditions of the CR. These methods will be called by the reconcile function.

To register the CR in Kubernetes, we need to add the newly defined types into supplied GroupVersionKind scheme, so we create a file called register.go.

5.We use two powerful tools k8s.io/code-generator and knative.dev/pkg(knative common packages) to generate the code of deepcopy,client,informer,lister and injection. All the generations have been consolidated in a single script called update-codegen.sh, under the directory hack. This script can be called anytime to generate or update the code under the directory client, where all of generated code is located.

• Deepcopy mean to have a method func (t* T) DeepCopy() *T for each type T.
• Client means the typed clientsets for CustomResource APIGroups.
• Informer offers an event based interface to react on changes of CustomResources on the server.
• Lister offers a read-only caching layer for GET and LIST requests.
• Injection offers a convenient way to access the informer.

6.We need to specify which resources to be monitored. Based on our need, we will watch the changes to the eventing CR and the changes to the deployments of Knative eventing. With the injection code, we can get the knativeEventingInformer and deploymentInformer, and add them into the event handlers. All the implementation can be found in the file controller.go under pkg/reconciler/knativeeventing/.

We also create a file named reconciler.go under the directory pkg/reconciler, instantiating an instance with NewBase function to implement of the common code for the reconciler, and adding the CR types into the scheme with init function.

7.The reconcile function is kicked-off each time there is a change happening to the resources we monitor in the event handlers. Currently, we have implemented knative installation, knative uninstallation and deployment restoration. Each time when the resource is changed, we will go through three steps, initialize the status, install the knative eventing by applying the manifest, and verify the deployment. Please visit the file knativeeventing.go for detailed information.

8.To launch the custom controller, it will be very convenient by calling sharedmain.MainWithConfig under the knative common packages. The file main.go only contains a few lines.

After going through the major steps, we are about to launch this new operator with the ‘ko command’. Welcome to skim the README of the eventing-operator for easy installation of Knative eventing!

Follow Vincent, (and) you won’t derail!