blog.dasrecht.net

With a min of max the opt!

ARM64 & k3s – The pitfalls

In the last blog post, I briefly went over my new setup, which is fully ARM-based. Now I want to dive a bit deeper into the issues I saw setting up my system on a fully arm based setup

VPN Setup

I run a setup with OpenVPN, Wireguard and Shadowsocks. The only thing I couldn’t get to run and build was Shadowsocks. Building the container failed, and since I have Wireguard in my setup, I am less inclined to still support Shadowsocks.

Most of my setup runs in containers, and the only thing I needed to do manually was building the OpenVPN containers as my upstream image isn’t cross-building ARM images, but that was mostly it. As I’m very pleased with the network speed of Wireguard across all my devices, I will most likely also drop OpenVPN in favour of running only Wireguard.

K3s Setup

As I’m also moving services away from traditional infrastructure towards Kubernetes, I also run K3s. The normal K3s installation works without issues, but as K3s is built for lightweight devices, this was to be expected.

Installing ArgoCD

ArgoCD doesn’t build ARM64 container images currently (Github Issue) But someone from the community started building those under alinbalutoiu/argocd. Those work great and without issues so far.

You can find the install.yml, which has been rewritten to the cross built images in the following Github Gist here (you might want to bump it to the most up-to-date image before applying)

Installing Lens Metrics Stack

Most of the engineers in our team started to use Lens for Kubernetes a while ago – The nice thing about it also comes with a metrics stack: one thing less to worry/think about. The only 2 changes I needed to do was removing the selector and upgrade the image to the 2.x stream of the images as of 2.x everything is built also for ARM64

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                  - linux
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
          - matchExpressions:
              - key: beta.kubernetes.io/os
                operator: In
                values:
                  - linux
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                  - amd64

The pitfall here is that the image moved from quay to the GCR registry. But this is written down in the Readme of kube-state-metrics. So if you are wondering why the v2.0.0 image doesn’t work when pulling from quay here you go:

k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-beta

RenovateBot

I run RenovateBot against my repositories for updates which will then be applied by ArgoCD but RenovateBot does not build under ARM yet due to ls-lint not being compatible yet. But there’s now also an Issue on renovatebot/renovate

I’ll get around the limitation of Renovate currently by running it as a docker container on my local machine in a cronjob. This does the job for now 🙂

Bastian