Building ClusterDrill Lab: A Disposable, Exam-Realistic Kubernetes Lab on AWS

Francis Onah

8 min read

·

Oct 15, 2026

--

Listen

Share

Terraform, kubeadm and Cilium, a remote desktop in the browser, and the bugs that only showed up on a real cluster.

Press enter or click to view image in full size

In my CKAD preparation guide I explained why I stopped practising only on my laptop: the exam runs as a remote desktop in your browser, and that latency changes how fast you work. A single-node Minikube profile on a laptop also can’t give you a real cluster to break: no separate worker to drain, no node to SSH into, no network plugin you installed yourself.

I wanted a real multi-node cluster I could create when I needed it and throw away when I didn’t. That became ClusterDrill Lab. It’s also where ClusterDrill, my practice platform, runs when I’m not on my laptop.

This article covers how the lab is built, why it’s built that way, and the problems that only appeared when I ran it on a real AWS account.

What you get

One terraform apply and one bootstrap script give you:

• a control-plane node and one or more workers on EC2 (t3.medium by default, with arm64/Graviton supported)
• Kubernetes 1.33 built with kubeadm, containerd and the Cilium network plugin
• Helm, kustomize, Podman and the k alias for kubectl on every node
• a Headlamp dashboard
• optionally, ClusterDrill and a browser-based practice desktop

cd providers/aws
terraform init && terraform apply
terraform output -json > ../../bootstrap/outputs.json
cd ../../bootstrap
CLUSTERDRILL_DEPLOY=1 PRACTICE_GUI_DEPLOY=1 ./run.sh outputs.json ~/.ssh/<your-key>

Two layers and a four-value contract

The lab is split into two layers that know almost nothing about each other.

Terraform creates the machines. The AWS module creates a VPC, a public subnet, a security group, a key pair, one control-plane instance and the workers. That’s all it does.

A bootstrap script turns machines into a cluster. run.sh connects over SSH and installs everything else. It's plain Bash, and it has no AWS-specific code at all.

The two layers meet at exactly four values: the control-plane IP, the worker IPs, the SSH user and the SSH key name. Any provider that produces those four outputs can use the same bootstrap unchanged. GCP support, when I build it, is a new Terraform module, not a rewrite.

Keeping the layers apart also made the bootstrap testable. My CI runs the full bootstrap on an ordinary GitHub runner, with no cloud account involved.

Press enter or click to view image in full size

A managed service like EKS would have been quicker to set up. I chose kubeadm on plain EC2 instances for four reasons:

• It’s closer to the exam. The clusters you work on in the exam are self-managed, with a control plane you can inspect and nodes you can reach. kubeadm builds exactly that kind of cluster.
• Full control of the nodes. I can SSH into any node, open a node shell with kubectl debug node, and practise node-level scenarios a managed control plane would hide.
• No managed control-plane fee. EKS charges for every cluster’s control plane on top of the nodes. For a lab I create and destroy regularly, that fee adds up with nothing to show for it.
• Learning the bootstrap end to end. Writing kubeadm init, the network plugin install and the worker join myself taught me more than any course module on the topic.

The trade-off is that I own every failure mode. For a disposable lab, that’s a price worth paying.

Building the cluster in six stages

run.sh works through six numbered stages and prints a banner with the elapsed time for each:

1. Prepare every node: containerd, kubeadm, kubelet and kubectl (pinned to Kubernetes 1.33), plus the practice tools.
2. Initialise the control plane with kubeadm init, then install Cilium 1.16.5.
3. Join the workers.
4. Set up the practice desktop (opt-in).
5. Deploy ClusterDrill (opt-in).
6. Deploy Headlamp.

A few decisions made the bootstrap reliable rather than merely working:

• Every stage is idempotent. Each script checks its own “already done” marker before acting: admin.conf for the control plane, cilium status for the network plugin, kubelet.conf for a worker. If a run fails halfway, I just run it again and it picks up where it stopped.
• Transient failures are retried. SSH reachability is checked before every remote step, and downloads retry up to three times.
• Mistakes fail early. Terraform checks each instance type against the chosen CPU architecture, so an arm64-only instance paired with an amd64 image fails at terraform plan, not halfway through apply with one node created and the other missing.
• The API server certificate includes the public IP, so you can copy the kubeconfig to your own machine and kubectl still validates TLS.

A practice desktop in the browser

With PRACTICE_GUI_DEPLOY=1, each worker gets a lightweight XFCE desktop with a terminal and Firefox, reached through noVNC in an ordinary browser tab. There's no VNC client to install. The bootstrap copies the cluster's kubeconfig to the worker, so kubectl works straight away in the desktop's terminal. It even tunes Firefox's scroll speed, because VNC sends one click per mouse-wheel tick and the default felt sluggish.

The security boundary is deliberately narrow:

• The VNC server listens only on 127.0.0.1. Nothing outside the node can reach it directly.
• websockify on port 6080 is the only exposed piece, and the security group opens it only to your own IP range.
• A VNC password is generated on the first run and shown when the setup finishes.

If you’d rather skip the password, [NO-PASSWORD FLAG] turns it off. Use it with care: the desktop's terminal holds an admin kubeconfig, so without a password anyone who can reach port 6080 has admin access to the cluster. Only use it when the allowed IP range is yours alone.

Opt-in layers

By default you get a bare cluster plus Headlamp. Everything else is a flag:

• CLUSTERDRILL_DEPLOY=1 installs ClusterDrill from its published Helm chart, after every worker has joined. The version is pinned in compatibility.json, and a CI check makes sure the pins in the scripts never drift from it. The [ClusterDrill article]([link to the ClusterDrill article]) covers the chart itself.
• PRACTICE_GUI_DEPLOY=1 adds the desktop above.

Two smaller choices are worth explaining:

• Headlamp is installed from a static manifest, not its Helm chart. The chart’s defaults bind the dashboard to cluster-admin. My manifest gives it a read-only role, and you log in with a short-lived token you generate on demand. The scripts never print one.
• Podman instead of Docker Engine for the exam’s “build a container image” objective. Installing Docker Engine on a node would bring a second containerd and quietly break the kubelet.

Press enter or click to view image in full size

The problems that only showed up on a real cluster

The ClusterDrill install waited for a worker that hadn’t joined

The first time I tested the full flow on a real AWS lab, the ClusterDrill install sat at kubectl rollout status until its five-minute timeout, then failed the whole run.

The cause was ordering. The bootstrap installed ClusterDrill as part of the control-plane stage, before any worker had joined. A kubeadm control plane carries a NoSchedule taint, and ClusterDrill's Deployment doesn't tolerate it, so the pod had nowhere to run.

CI hadn’t caught it because the CI job builds a single-node cluster on one runner, and removes that taint so everything fits. Every real lab keeps it.

I split the install into its own script and reordered the run: control plane, then every worker joins, then ClusterDrill. I verified the fix on a mixed cluster, with an amd64 control plane and an arm64 worker. ClusterDrill landed on the arm64 worker and passed its health check from outside AWS.

The lesson I took from it: a test environment that differs from production in one small way can hide exactly the bug production will hit.

Firefox wouldn’t open

On the first desktop build, the Firefox icon was there, but double-clicking it did nothing. There was no error dialog either.

On Ubuntu, apt install firefox installs a small transitional package that pulls in the snap version. Snap's sandbox expects a proper login session from systemd-logind, and a VNC session never creates one. So snap refused to start Firefox before Firefox's own code ever ran. The only trace was a line in the logs: is not a snap cgroup for tag snap.firefox.firefox.

The fix was to install Firefox from Mozilla’s own apt repository, pinned above Ubuntu’s package. That needed --allow-downgrades, because Ubuntu's transitional package carries an inflated version number specifically so it wins over a real package of the same name.

Smaller lessons

• The screensaver locked me out. XFCE’s screensaver locks the session with the Linux user’s password, and the ubuntu user only has an SSH key. A locked desktop could never be unlocked, so the bootstrap now removes the screensaver.
• Interrupted runs couldn’t resume. kubeadm init and kubeadm join refuse to run twice, which is why every stage became idempotent.
• AWS rejected an apostrophe. Security-group descriptions only allow a limited character set, and “operator’s” wasn’t in it.

Lifecycle and cost

The lab is disposable by design. I destroy it when I’m done and recreate it the next time I need it. terraform destroy removes everything the module created and nothing else in the account. [Placeholder: your cost figure, e.g. what a typical practice session or month cost you.]

Why not just stop the instances overnight? The nodes have no Elastic IP, so a stopped instance comes back with a new public IP. The API server certificate was issued for the old one, so kubectl from your own machine fails TLS validation, and the saved Terraform outputs point at addresses that no longer exist. Fixing that by hand costs more effort than a fresh terraform apply.

The same thinking explains the non-goals:

• No high availability. One control plane is the intended scope.
• No automatic cost controls. Budget alerts depend on your own account, so set one up yourself.
• No backups. Re-provisioning is the recovery path; nothing in a practice lab is worth restoring.

Engineering practices

• A CI quality gate on every pull request: terraform fmt and validate, a Terraform security scan, ShellCheck, OS-detection tests, and an end-to-end bootstrap run with and without ClusterDrill.
• Least-exposure networking. SSH, the Kubernetes API, NodePorts and noVNC are open only to allowed_ssh_cidr. That variable has no default and rejects 0.0.0.0/0.
• No credentials held by the project. You own the AWS account, the SSH key and the Terraform state.
• Versioned releases with release-please, and Dependabot for Terraform provider and GitHub Actions updates.

What’s next, and how to try it

Next on the list is a GCP provider. The contract is ready; it needs a Terraform module that produces the same four outputs. Contributions are welcome.

To try it, you need AWS credentials, an SSH key pair and Terraform 1.5 or later. Clone a tagged release rather than main:

git clone --branch <latest-release-tag> --depth 1 \
  https://github.com/onahFran6/clusterdrill-lab.git

Then follow the quick start in the README: https://github.com/onahFran6/clusterdrill-lab

If you’re preparing for the CKAD, my preparation guide covers how I used this lab, and my ClusterDrill article covers the practice platform that runs on it.

ClusterDrill Lab is not affiliated with, endorsed by, or sponsored by the Linux Foundation or the Cloud Native Computing Foundation. “CKAD” is a trademark of the Linux Foundation.