k3d

https://k3d.io/v5.7.2/

k3d is a lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker. k3d makes it very easy to create single- and multi-node k3s clusters in docker, e.g. for local development on Kubernetes.

---

Installation

With nix

It’s as simple as (also install kubectl so you can interact with your kubernetes cluster):

{ pkgs, ... }: {
  home.packages = with pkgs; [ k3d kubectl ];
}

Launch

Create a default.yml with the following content:

---
# see https://k3d.io/v5.6.3/usage/configfile/ for complete config
apiVersion: k3d.io/v1alpha5
kind: Simple
servers: 1
agents: 0
image: docker.io/rancher/k3s:v1.30.1-k3s1
# ingress
ports:
  - port: 80:80
    nodeFilters:
      - server:0
# will use host docker registry
registries:
  create:
    name: registry.localhost
    host: "0.0.0.0"
    hostPort: "5000"

This will specify the version of k3s and create a local docker registry (useful for local tests). Then execute the following command:

$ # create the cluster
$ k3d cluster create --config default.yml
 
$ # wait a bit and you can see the cluster is created
$ k3d cluster list
NAME          SERVERS   AGENTS   LOADBALANCER
k3s-default   1/1       0/0      true
 
$ # or using kubectl
$ kubectl get nodes
NAME                       STATUS   ROLES                  AGE     VERSION
k3d-k3s-default-server-0   Ready    control-plane,master   3d21h   v1.30.1+k3s1

Issues

✅ Everything fails after some time

Success

The issue comes from the fact k3d has some limit on disk storage (5% if my memory serves me right). And I did not have much disk space, so the Pod were failing one after the other once I did not have much space disk left…

After some time (a few hours), every pods are in failure state and cannot be restored.

Not sure if it was because I was updating my home-manager at the same time…

It was not because of home-manager.

When looking at the pod status message, I got:

status:
  message: 'Pod was rejected: The node had condition: [DiskPressure]. '
  phase: Failed
  reason: Evicted
  startTime: "2024-07-05T09:35:02Z"

There’s an issue about this. They suggested adding some flags when creating the k3d cluster:

k3d cluster create \
  --k3s-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%@agent:*' \
  --k3s-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%@agent:*'