I've been working full-time with Kubernetes for several years, writing manifests, running a bunch of kubectl apply, oc apply, helm upgrade --install, kustomize build | kubectl apply, et cetera. Generally, I'm happy with these interfaces, but I have not had a great time managing the configuration in source control, for a variety of reasons I'd like to expand upon separately. In any case, I've decided it might be a cool home project to try and build some tooling that solves exactly the problems I've had going from Git repository to Kubernetes cluster. This post outlines my plan.
tl;dr the repo is here
things I would like to be able to do
immediate goals:
- have manifests in a git repository be deployed to one or more clusters when I push to master
- deploy plain manifests, kustomize directories, and helm charts from the same repo
- configure the order items are deployed (for example, CRDs before other stuff)
- ensure that what's in the repository is always what's on the cluster
- manage the deployment system through CRDs and
kubectl
stretch goals:
- build an image from a Dockerfile in the repo, then consume it in a manifest
- "canary" deploy manifests on pull requests
disclaimer: I am sure there is a tool out there that can fit these needs, but I haven't found it yet, and making my own sounds more fun!
how I have decided to do it (for now, at least c:)
I'd like to use this project as a vehicle for testing a few tools I've recently been fascinated with: Kubebuilder and Tekton. So, I'm going to create a Kubernetes controller that watches GitHub repositories specified through a new Kubernetes Custom Resource, the Repo.
When changes are detected, Kubernetes manifests contained within the repository are deployed to one or more Kubernetes clusters via a Tekton pipeline. Tekton is certainly overkill for creating Kubernetes objects - I could easily have the Kubebuilder controller process manifests for me - but I believe the Tekton primitives will make it far easier to build some of the more advanced future-state items, so I'm running with it!
Some pieces of configuration, such as the repo's URL and the list of target Kubernetes clusters, will be managed through a Kubernetes Custom Resource. Other pieces, such as which files to deploy and which kind of files they are (helm, kustomize, etc) will be configured from a file contained within the repo.
assorted nonsense
I've dubbed the project alaska, and I honestly don't really know why. The repo is here. I'll write up a deep-dive once I've made some real progress. :D
o7,
rudeboy