Having multiple homegrown k8s clusters at former work… I missed them (the sad individual I am) so I got mine

Steps to follow to get microk8s running on Ubuntu 24 and arm with great help of ChatGPT and ClaudeAI

If you follow instructions on how to install microk8s on Ampere VM running Ubuntu 24 in OCI, you’ll discover that while the install succeeds, pods can’t communicate to outside world. You may notice some of them restarting.

It would appear there are 2 separate issues that prevent it from working:

1. microk8s requires legacy iptables (seems to be due to some changes in Ubuntu 24)
2. some default routes issue (appears to be caused by default firewall rules in Ubuntu 24 image in OCI)

Note: This is a result of debugging with help of ChatGPT and Claude both of which have sent me to various other paths until I spoon fed them iptables-save output. Perhaps easier ways exist but this works and is reproducible.

Workaround legacy IP tables issue

Before installing microk8s:

sudo apt-get update
sudo apt-get install -y iptables arptables ebtables

And then, alternatives switch:

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

To verify:

sudo update-alternatives --display iptables
sudo update-alternatives --display ip6tables
sudo update-alternatives --display arptables
sudo update-alternatives --display ebtables

all should be displaying that legacy binaries are in use.

Reboot.

Install microk8s

Follow steps 1&2 from instructions.

Update iptables

It would appear that the following iptables rule is on the wrong place… moving it fixes connectivity issues.

sudo iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
sudo iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

This needs be done each time after restart. These notes will be updated to reflect more permanent solution.

At this stage, you should see 3 pods up and running:

microk8s kubectl -n kube-system get pods -o wide

NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
calico-kube-controllers-79949b87d-rznxr   1/1     Running   2          4d15h   10.1.205.202   master-2   <none>           <none>
calico-node-4lk9b                         1/1     Running   2          4d15h   10.0.0.17      master-2   <none>           <none>
coredns-ccd8f67bc-lrkk9                   1/1     Running   2          4d15h   10.1.205.203   master-2   <none>           <none>

You could run a shell (as in the end of this page) to verify connectivity. Your IPs maybe different.

External Access

Very little point in having K8s cluster not accessible from remote but also, we don’t want to simply expose it with no security. So let’s do it:

Enable RBAC

microk8s enable rbac

Have API Server advertise IP and listen on all IPs

Edit kube-apiserver e.g.

sudo vi /var/snap/microk8s/current/args/kube-apiserver

Add these two lines in replacing w.x.y.z with your own external IP

--bind-address=0.0.0.0
--advertise-address=w.x.y.z

Modify CSR to have external IP

sudo vi /var/snap/microk8s/current/certs/csr.conf.template

Add your IP by copying IP.2 as

IP.3 = w.x.y.z

Again, replace w.x.y.z with your own external IP.

Refresh Certs:

sudo microk8s refresh-certs --cert ca.crt 

Prepare client cert for kubectl config

Execute the following

mkdir ~/k8s-certs
cd ~/k8s-certs

# Generate your private key
openssl genrsa -out admin.key 2048

# Create certificate signing request
openssl req -new -key admin.key -out admin.csr -subj "/CN=admin/O=system:masters"

# Sign it with the cluster CA
sudo openssl x509 -req -in admin.csr \
-CA /var/snap/microk8s/current/certs/ca.crt \
-CAkey /var/snap/microk8s/current/certs/ca.key \
-CAcreateserial -out admin.crt -days 365

# Copy the CA cert for your kubeconfig
sudo cp /var/snap/microk8s/current/certs/ca.crt ./ca.crt

# Create the kubeconfig file
cat > ~/.kube/microk8s-external << EOF
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/ubuntu/k8s-certs/ca.crt
    server: https://w.x.y.z:16443
  name: microk8s-external
contexts:
- context:
    cluster: microk8s-external
    user: admin
  name: microk8s-external
current-context: microk8s-external
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate: /home/ubuntu/k8s-certs/admin.crt
    client-key: /home/ubuntu/k8s-certs/admin.key
EOF

Please modify, in the script above, w.x.y.z with your actual IP.

Open firewall and restart MicroK8s

sudo iptables -I INPUT 4 -p tcp -m tcp --dport 16443 -j ACCEPT

# Restart MicroK8s

microk8s stop
microk8s start

Firewalls

You will need to allow access to 16443 externally from any IP.

Helpful, random things

Get a basic ubuntu shell

kubectl run my-shell --rm -i --tty --image ubuntu -- bash

Get a very nice network debug shell

kubectl run my-shell --rm -i --tty --image nicolaka/netshoot -- bash