I’ve been writing Operators & Controllers for Kubernetes for a while now and there are many approaches that exist today that can help build/test/deploy a Controller/Operator. Typically these solutions watch the entire cluster requiring a large permission set across the entire cluster.

I’m going to discuss a bit about how I’ve accomplished this in the past and then dig into how to implement a ‘watch’ on the Kubernetes API server to get notified of events that happen for only a single namespace.

Watch via HTTP

One of my first controllers that I wrote was a tool to fetch secrets from Hashicorp Vault and expose them as Kubernetes secret. A great use-case for this is to allow for easy password rotation for accounts wanting to authenticate to a MySQL database. Application’s only need to know how to consume the credentials from a Kubernetes secret, when that secret needs rotated, it will get updated and the application can refresh its internal connections/state.

The interesting bit with how the secret manager got implemented initially is that it just used the REST API endpoints of Kubernetes to interact. We didn’t use (or even have) client-go at this point, but the simplicity of the integration was very interesting since it only used HTTP calls to list secrets, or watch for changes in the cluster.

In the following example you can see we are making a GET request to http://127.0.0.1:8001/api/v1/namespaces/default/secrets/secretname and looking at the response code the API server returns with so that we can validate if the Secret was found.

apiHost = "http://127.0.0.1:8001"
secretsEndpoint = fmt.Sprintf("/api/v1/namespaces/%s/secrets", namespace)

resp, err := http.Get(apiHost + secretsEndpoint + "/" + secretName)
    if err != nil {
    return err
}

if resp.StatusCode == 200 {
    // Found the secret!
} else if resp.StatusCode == 404 {
    // Secret is missing...
}

Requesting a specific resource isn’t all that interesting for a controller however, it’s important to allow it to be notified when something changes in the cluster. In our example, it’s for the Secrets resource.

Here’s how you could implement a watch using HTTP:

events := make(chan CustomSecretEvent)
errc := make(chan error, 1)
go func() {
for {
    resp, err := http.Get(apiHost + customSecretsWatchEndpoint)
    if err != nil {
        errc <- err
        time.Sleep(5 * time.Second)
        continue
    }
    if resp.StatusCode != 200 {
        errc <- errors.New("Invalid status code: " + resp.Status)
        time.Sleep(5 * time.Second)
        continue
    }
    decoder := json.NewDecoder(resp.Body)
    for {
        var event CustomSecretEvent
        err = decoder.Decode(&event)
        if err != nil {
            errc <- err
            break
        }
        events <- event
    }
}
}()

Watch via client-go

Client-go is a client used to talk to Kubernetes via golang. There are two “modes” that can be used to query for resources, a typed and a dynamic mode. For this post, I’ll focus on the typed mode. Setting up a watch with a client-go typed client is pretty straightforward.

First, let’s create a Kubernetes client, which sets up a connection to the API server:

// creates the in-cluster config
config, err := rest.InClusterConfig()
if err != nil {
    panic(err.Error())
}
// creates the clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
    panic(err.Error())
}

Next we’ll create a SharedInformerFactory which will allow us to setup Informers on resources to register for events (e.g. Add, Update, Delete) when they occur in the server. The second line in the following example registers for the events which are passed to a resource event handler (the specifics are out of scope for this example).

coreInformers := coreinformers.NewSharedInformerFactory(client, 0)
coreInformers.Core().V1().Secrets().Informer().AddEventHandler(&reh)

Finally we’ll start the informers which will give us first a list of all resources we’re watching for (as well as fill up a local cache), then update us when changes happen back to that resource event handler we passed to the informer AddEventHandler method.

// stop is a channel passed in to handle termination
coreInformers.WaitForCacheSync(stop)
coreInformers.Start(stop)

This is pretty simple to get up and running, but again back to our original problem statement, we’re watching for ALL secrets in the cluster when I only want to watch for a single namespace.

Watch a single namespace with client-go

The only change we’ll need to make to the previous examples is how we built up the SharedInformerFactory. To watch only a single namespace, we’ll need to pass the namespace we want to watch and everything else will be the same:

namespace := "root-ingressroutes"
coreInformer := coreinformers.NewSharedInformerFactoryWithOptions(client, 0, coreinformers.WithNamespace(namespace))

This version of coreInformer now only watches the namespace root-ingressroute for changes. Ideally as well, the service account passed to this controller would only have RBAC permissions to the namespace it needs to watch (e.g. root-ingressroute).

Conclusion

There’s much more to building out controllers with client-go! I recommend you also checking out kubebuilder which is a way to implement controllers without having to deal with much of the boilerplate overhead. Also, Joe Beda did a TGIK recently on kubebuilder which gives you a nice walk through of kubebuilder in action!