Deis Router on Arm

Overview

If you've followed me on Twitter or come to any of my talks at CodeMash, Pittsburgh Tech Fest, or coming up at Abstractions (Here and Here), you know that I'm into Kubernetes and deploying it out to a Raspberry Pi Cluster.

Why DNS?

In prep for my talks coming up at Abstractions, I wanted to make the demo's a little bit more realistic by using real dns names for the demos instead of using NodePorts (because who is going to access a site on the public internet like this: https://foo.com:30243)?

If you're not familiar with Kubernetes, NodePorts are a simple way to expose containers running in a cluster to the outside world. You access one the nodes in the cluster via this port and magically you are routed to the container. You can read more about NodePorts here.

Solutions

I tried a few solutions to get this dns routing to work.

First I tried to implement Ingress routing. This is a k8s native solution which lets you route traffic to the cluster, and a router inside the cluster then looks at the request and sends you to the appropriate container inside the cluster.

I struggled to get ingress working in Minikube on my Mac laptop. There was some errors with my certs and the service account. I then tried to deploy to my AWS cluster running on CoreOS. This time I got past the cert errors, but the Nginx controller wouldn't pick up the services.

Being a big fan of the work the Deis team has been doing. They have built / are building some really cool things and I encourage everyone to check it out.

They make a router which works similarly to Ingress but instead of deploying Ingress resources to your cluster, you just attach labels to the services.

You just need to deploy the router, add a label (router.deis.io/routable: "true") and an annotation (router.deis.io/domains: foo,bar,www.foobar.com) to a service and that's it!

Example:

Compiling for ARM

I love me some raspberry pi's and think it's totally cool that we can run kubernetes (and lots of other cool stuff) on small, cheap hardware. I think building out a k8s cluster by hand is a great learning path to understand how kubernetes works and how to troubleshoot (because stuff will ultimately go wrong).

Since the Deis router isn't compiled for ARM, that's up to us to do.

Build the Router Binary

1. On my mac I cloned the Deis Router repo down: git clone https://github.com/deis/router.git
2. Run make boostrap to get the environment running
3. Edit the make file and change GOARCH to be arm instead of amd64
4. Build the router binary for arm: make binary-build
5. Copy that file over to your raspberry pi
6. Clone the router repo onto your pi and move the router binary into ${GITREPO}/rootfs/opt/router/sbin/router

Build the helper containers:

The Deis team uses some base images they built as a common starting point. First, I had to update the base image (quay.io/deis/base:0.2.0). I did this by changing the root image to be hypriot/rpi-golang which turned into: stevesloka/deis-base-arm.

Then just update the base image in rootfs/Dockerfile to be the base image we created in the previous step.

My final image is here: stevesloka/deis-router-arm

Replication Controller

Here's my replication controller which I used to deploy the router to my pi:

apiVersion: v1  
kind: ReplicationController  
metadata:  
  name: deis-router
  namespace: deis
  labels:
    heritage: deis
spec:  
  replicas: 1
  selector:
    app: deis-router
  template:
    metadata:
      labels:
        app: deis-router
    spec:
      containers:
      - name: deis-router
        image: stevesloka/deis-router-arm:0.1.0
        imagePullPolicy: Always
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        ports:
        - containerPort: 8080
          hostPort: 80
        - containerPort: 6443
          hostPort: 443
        - containerPort: 2222
          hostPort: 2222
        - containerPort: 9090
          hostPort: 9090
        livenessProbe:
          httpGet:
            path: /healthz
            port: 9090
          initialDelaySeconds: 1
          timeoutSeconds: 1
        readinessProbe:
          httpGet:
            path: /healthz
            port: 9090
          initialDelaySeconds: 1
          timeoutSeconds: 1

Summary

Hope this helps anyone run this on their pi. I probably need to come up with a more automated way to build these images, but for now it should work out fine.

I was pinged on Twitter someone does have Ingress working on ARM here, but haven't yet taken a look yet, but will soon.

~ Steve