KinD, Kubernetes on your local machine and your CI

Creating a cluster on your local machine can be a frustrating task. Docker and k8s help but still, k8s installation can be intrusive and slow on the local machine. Remote clusters also have their downside, especially for debugging. E2E testing can be challenging as well especially when you have multiple tests that create side effects for each other, and it can be very expensive, having k8s cluster deployed for each PR.

Imagine you could run every time on a fresh cluster, just for a single interaction, without any side effects.

Meet KinD (Kubernetes in Docker) is a lightweight implementation of K8S primarily designed for testing Kubernetes itself. But it’s a great tool for any developer!

If you already using k8s and have a working CD you already did the hard part. The idea is simple. Create a fresh k8s cluster, deploy your application, and finally debug or test your application.

Real-world example

I have created a simple straight forward example to show how to use Scalecube-js on a Kubernetes cluster. Scalecube-js is a micro-services infrastructure opensource project.

In this example I used KinD for:

• Develop it locally (the application and Helm charts). I have a simple `start` script that creates a cluster and deploy my application with Helm
• Ensure it’s working without regression. I use Github Actions to run tests written with Jest, Jest is calling `start` script, and after the cluster is ready, runs the E2E tests
• Allow any contributor to run it without k8s cluster of his own. Anyone can try the application by using the `start` script. The only prerequisites are Docker, KinD, Helm, and Kubectl. If you want you can put everything in a Docker image and have only Docker as a dependency.

You can see the full example here: https://github.com/scalecube/scalecube-js/tree/develop/packages/examples/k8s.

How it’s works

Let’s take a look at my start script:

Step by step explanation:

Here I am simply adding GOPATH to PATH. I installed KinD on my local machine with Go. Don’t worry you don’t need to do it. I am using Go for other projects so it was my preferred way. You can find installation instruction here: https://kind.sigs.k8s.io/docs/user/quick-start, the simplest way is to download a single binary file.

Here I am checking I have all the tools I need. Docker, kubectl & KinD are must for any project In my case I use Helm to manage my cluster. I use this chart.

Here I am building a Docker image, you can also pull and Image from a remote registry.

Here we are creating our cluster and loading our image. We are reusing the cluster if it already exists to save time. The kinD load will load the image only if it does not exist on the cluster. If the user wants to start with a clean state he can simply delete the cluster.

Tips

• You can create multiple clusters just pass different names in ` — name` argument
• Creating a cluster can take a few seconds even up to a minute depending on the machine. Delete cluster happens almost instantly. K8S namespaces usually are the opposite creating is very fast and deleting namespace can take some time
• Kubernetes can pull images from remote registries (public or private) instead of using `kind load`. But I found that `kind load` is the most useful way. If you still want to pull images with private registries there some gotcha… see instructions.
• Don’t use “latest” tag, when you are using “latest” tag k8s change the pulling policy to always, then k8s will try pulling the image even if you loaded it. You can pull “latest” to the host machine and add tag (via docker tag command) for `kind` load command

` — config cluster` is telling KinD to use the following config file

In this configuration file, I am forwarding the container port 30080 to 8080 on my local machine. In this way, we can interact with the cluster. You can forward anything using this method, a service, container port even Ingress. If you want to temporarily forward some port you can use `kubectl port-forward`.

Using the config file you can also mount a directory or socket, see example below:

Let’s continue with our script

Here we are making sure `kubectl` is with the right context, the context will be kind-{cluster-name}. Then we are creating a namespace, in case it already exists we deleting it and creating a new namespace.

In the last step, we are using helm to install the namespace, — wait will cause the command to wait until the installation is done.

Now we are ready to start using our cluster!

How the CI work

The tests are regular tests written Jest, I simply create a cluster using the “start” script. Because I am using GitHub Actions for the CI and it runs the tests, everything I need is already installed (KinD, Docker, Helm, and Kubectl).

You can see it here: https://github.com/scalecube/scalecube-js/blob/develop/packages/examples/tests/greeting.spec.ts

Here we simply calling our script, when it’s finished we have cluster ready for tests. Our cluster is ready but the services still need a warm-up. There are few ways to handle it, I chose retry in this example. Another strategy, probably a better strategy for most cases, is to create the cluster and wait for the services to be ready. With Jest, you can do it with `globalSetup` file, run you “start cluster” script, sample your services if it’s ready (everyone has a different method for it), and then continue to the tests.

Something like:

Thanks for reading ;-)

If you enjoy it or want more on this subject please let me know. I want also to invite you to try out scalecube-js, Scalecube is an opensource toolkit for creating microservices/micro-frontends based systems. We just released version 0.2.0, it’s a major milestone, after more than a year rewriting scalecube-js and creating the second generation of scalecube-js, we are finally releasing LTS version.