Angelesen #77

Here we go again, didn’t get to sort a few links in the past few weeks so they come out of the link-wrangler a little bit delayed. Enjoy the weekend 🎉

A Ship Got Stuck. So He Built A Website. (

It seems like, at one point, some journalists were using you as a resource, no?

Yes! It was hilarious but also a bit troubling — like, this is not how it’s supposed to work! I’m supposed to be looking at you! My favorite was when a journalist would tweet something and other people would like to the site to refute their reporting. I’m sitting here thinking ‘Wow you all have way too much faith in a random domain on the internet.’

Love the story – person registers domain world visits website and takes everything for granted.

🔒Leere Züge, leere Kassen: SBB unter Druck – wie weiter? (

Die GAs gingen 2020 um 60’000 zurück, ein Minus von 12 Prozent. Im Tarifverbund Nordwestschweiz, im Zürcher Verkehrsverbund und im Regionalverkehr Bern-Solothurn betrug das Minus bei den Abos sogar einen Viertel und mehr. Der Abwärtstrend hält ungebremst an. Der Branchenverband Alliance Swisspass erwartet für das laufende Jahr noch einmal 10 Prozent weniger GAs.

Tja, ich hab ebenfalls nach mehr als einem Jahrzehnt GA besitz dem ganzen den Rücken gekehrt, die letzten Jahre hatte ich es noch, weil es einfacher war – Fairtiq ist jetzt das neue und einfache und angenehme. Wenn die SBB und Konsorten in der Krise mit ihren Stammkunden ein bisschen besser umgegangen wären, ja dann wären es wohl nicht 10 % weniger Langzeitkunden.

Faktencheck zu Vogts Vorschlag – Kann das Gesundheitssystem 30’000 Neuinfektionen pro Tag verkraften? (

Sobald alle Risikopatienten geimpft sind, hält Arbeitgeber-Präsident Valentin Vogt 30’000 Neuinfektionen pro Tag für verkraftbar.


Here’s to the crazy ones: lessons learned at Apple (

Code lives a lot longer than you’d expect. Something I started off as a little side project a year into my time in Maps became a fundamental dependency in other teams’ flows. Receiving questions and PRs from people across the company about it long after I was out of that group became a source of pride as time went on.

Got burned by this a few times – Expect code to live for a very long time even if the Upstream is long EOL – Looking at sites still running on PHP 5.6!

Silos can be good because they allow people to focus, but they’re only good as long as there are well-defined interfaces at the boundaries between teams.

Super important – This is the biggest danger if you switch to well defined teams – that in the beginning there might not be enough people till you get the team structure fully setup.


Some of these skeletons are little-known but genuinely useful features, some of them are legacy oddities relied on by billions of connections daily, and some of them really shouldn’t exist at all.


Cache-Control: private, no-cache It looks like this means "don’t store this response anywhere", right?

Hahaha no.

HTTP Goodread 🙂

Police warn students to avoid science website (

But Max Bruce, the City of London police’s cyber protection officer, has urged universities to block the website on their networks because of the "threat posed by Sci-Hub to both the university and its students".

University meet Internet, Internet meet University

Beeper – All Your Chats In One App (

Might give that one a try to see how it holds up against Rambox

Experience: I tracked down my impostor (

I don’t understand why he chose me. I have an interesting niche subject, and love what I do, but I’m not the most prolific of academics. We never contacted the police, because we didn’t think he’d done anything illegal. It was really an issue of academic misconduct. The most upsetting thing – copying my hand tattoos – wasn’t something I could do anything about. He’s probably still walking around with those tattoos on his hands today.

Wow this is quite something

The French army is testing Boston Dynamics’ robot dog Spot in combat scenarios (

When they come for you – aim for the sensors – good luck

GitHub investigating crypto-mining campaign abusing its server infrastructure (

The attack involves forking a legitimate repository, adding malicious GitHub Actions to the original code, and then filing a Pull Request with the original repository in order to merge the code back into the original.

But the attack doesn’t rely on the original project owner approving the malicious Pull Request. Just filing the Pull Request is enough for the attack, Perdok said.

Neat, this attack vector is great!

Unter dem Rhein ist die größte Lithium-Quelle Europas (

Der Oberrheingraben ist Europas größte Lithium-Quelle. Es handelt sich hierbei um eine 300 Kilometer lange und bis zu 40 Kilometer breite Tiefebene zwischen Frankfurt und Basel. Unter der Wasseroberfläche des Grabens sind zehntausende Tonnen an Lithium verborgen, die Deutschland zu einem der größten Lithium-Produzenten der Welt machen könnten, wie das „Handelsblatt“ berichtet.

Wenigstens plündern wir in Zukunft unseren Vorgarten, nicht!

This blog is now hosted on a GPS/LTE modem (

For whatever reason, I thought it’d be fun to run my blog on this thing. Since we were working with limited resources (around 48M of space and the same amount of memory), and the fact that my blog is just a bunch of static files, I decided that something like nginx (as lightweight as it is) would be a bit overkill for my purposes.

darkhttpd seemed to fit the bill well. Single binary, no external dependencies, does GET and HEAD requests only. Perfect.


5 Jahre später – Nächste Wohnung?

Also eigentlich 4 Jahre und 11 Monate aber wollen wir mal nicht so kleinlich sein – 5 Jahre sind eine gute Zeitspanne quasi der Fünfjahresplan. Ich ziehe weiter, und für die Leute, die achsogerne fragen, ob ich wieder nach Bern komme: Ein definitives Jein!

Für die nächsten Monate werde ich grösstenteils losgelöst von einer festen Wohnung leben (auch mal in Bern, rumour has it, dass ich da Wurzeln habe). Mal schauen, wie das so funktionieren wird, mit dem Leben im Camper *hust* Vanlife *hust* habe ich ja schon Erfahrung gesammelt, die letzten 2 Jahre. Es “Vollzeit” zu leben, ist jedoch eine andere Dimension. Ich bin gespannt.

Mit einem weinenden und einem lachenden Auge gebe ich die Schlüssel der wirklich schönen Wohnung ab. Danke: Emma, Lewis, Sändu, Roger für die vergangenen Jahre in Altstetten!

ARM64 & k3s – The pitfalls

In the last blog post, I briefly went over my new setup, which is fully ARM-based. Now I want to dive a bit deeper into the issues I saw setting up my system on a fully arm based setup

VPN Setup

I run a setup with OpenVPN, Wireguard and Shadowsocks. The only thing I couldn’t get to run and build was Shadowsocks. Building the container failed, and since I have Wireguard in my setup, I am less inclined to still support Shadowsocks.

Most of my setup runs in containers, and the only thing I needed to do manually was building the OpenVPN containers as my upstream image isn’t cross-building ARM images, but that was mostly it. As I’m very pleased with the network speed of Wireguard across all my devices, I will most likely also drop OpenVPN in favour of running only Wireguard.

K3s Setup

As I’m also moving services away from traditional infrastructure towards Kubernetes, I also run K3s. The normal K3s installation works without issues, but as K3s is built for lightweight devices, this was to be expected.

Installing ArgoCD

ArgoCD doesn’t build ARM64 container images currently (Github Issue) But someone from the community started building those under alinbalutoiu/argocd. Those work great and without issues so far.

You can find the install.yml, which has been rewritten to the cross built images in the following Github Gist here (you might want to bump it to the most up-to-date image before applying)

Installing Lens Metrics Stack

Most of the engineers in our team started to use Lens for Kubernetes a while ago – The nice thing about it also comes with a metrics stack: one thing less to worry/think about. The only 2 changes I needed to do was removing the selector and upgrade the image to the 2.x stream of the images as of 2.x everything is built also for ARM64

          - matchExpressions:
              - key:
                operator: In
                  - linux
              - key:
                operator: In
                  - amd64
          - matchExpressions:
              - key:
                operator: In
                  - linux
              - key:
                operator: In
                  - amd64

The pitfall here is that the image moved from quay to the GCR registry. But this is written down in the Readme of kube-state-metrics. So if you are wondering why the v2.0.0 image doesn’t work when pulling from quay here you go:


I run RenovateBot against my repositories for updates which will then be applied by ArgoCD but RenovateBot does not build under ARM yet due to ls-lint not being compatible yet. But there’s now also an Issue on renovatebot/renovate

I’ll get around the limitation of Renovate currently by running it as a docker container on my local machine in a cronjob. This does the job for now 🙂

Angelesen #76

Here we go again, a week with a lot of Cookie-related fun (and games). It’s also good to check recommended Readings of Habi every once in a while (with many WhatsApp related articles) and obviously also Tobru, which releases his newsletter like clockwork! Anyways, enjoy the weekend!

Cookie Consent Speed.Run (

Since GDPR came into our lives, we’ve all had to struggle with obtaining our basic privacy rights. With each cookie banner we have all been honing our skills, learning to navigate ambiguous options and distrust obvious buttons.

This is great! I failed miserably!

H/T Dan

I don’t care about cookies 3.2.9 (

This browser extension removes cookie warnings from almost all websites and saves you thousands of unnecessary clicks!

Will give that a try for a few weeks and see how it works.

Signal on Twitter (

Signal registrations are through the roof; welcome everyone! Solidarity to the folks working on the WhatsApp outage. People outside of the tech industry will never understand how weird it sounds when someone says that they are "looking forward to some weekend downtime.

Use Signal, Use Tor.

Radikal für Delfine: Wie weit geht Sea Shepherd? (

30 Minuten die Mensch sich gönnen darf. Und gegebenenfalls auch kurz eine Spende bei Sea Shepperd lassen 🙂

Tinder will soon let you run a background check on a potential date through Garbo (

Somehow yes, but also a hard no on this.

Toronto swaps Google-backed, not-so-smart city plans for people-centred vision (

Sidewalk Labs pledged to make Toronto one of the world’s first “smart” cities. But skeptics saw a darker side, which included thousands of cameras monitoring streets, storefronts and parks, and harvesting data on the smallest movements.

Now, Canada’s largest city is moving towards a new vision of the future, in which affordability, sustainability and environmentally friendly design are prioritized over the trappings of new and often untested technologies.

Glad to see that Toronto goes not all in "smart-city" as France did with their Safe City approach – see Technopolice: calling out so-called "Safe Cities"

What Remote Work Really Does To Your Engineering Productivity (

Flexible hours can easily become an imposition. If you assume your engineers will set their own hours, employees with more seniority or status will implicitly dictate everyone else’s activities.

M.2 on a Raspberry Pi – the TOFU Compute Module 4 Carrier Board (

Now, a single x1 lane at gen 2 speeds tops out around 400 MiB/sec in real-world usage, so many NVMe drives are still underpowered connected to the Pi, but as you’ll see in a bit, a cheap KingSpec SSD was 3x faster for random IO than a similar SSD plugged in via USB 3.0.

Seeing the Compute Module 4 and all the work Jeff puts in to play around with it and find creative ways makes me wonder if the next iteration will be a compute module for my Infrastructure.

self-hosting git; or, how git servers actually work, and how to keep yours secure (

Guiding principles for the forks of Elasticsearch and Kibana – Stepping up: Elasticsearch & Kibana Fork (

Open source like we mean it. We are invested in this being a successful open source project for the long term. It’s all Apache 2.0. There’s no Contributor License Agreement. Easy.

🎉 That’s what we’re talking about!

H/T Scott

System separation in the Continental Europe Synchronous Area on 8 January 2021 – 2nd update (

The system separation resulted in a deficit of power (approx. -6.3 GW) in the North-West Area and a surplus of power (approx. +6.3 GW) in the South-East Area, resulting in turn in a frequency decrease in the North-West Area and a frequency increase in the South-East Area.

A few gotchas from the article.

  • It’s amazing to see the level of fault tolerance the entire system has when it gets out of balance (and 6.3GW is a lot out of balance)
  • A very distributed system with so many companies leads to weeks and weeks of work to find out what exactly went wrong
  • This makes our post-mortem reports look tiny in comparison

The real reason Okta spent $6.5B on Auth0 (

TL;DR: Increasing addressable market

Auth0 is developer-driven Okta is sales-driven

Getting the market from both sides makes sense for them.

Angelesen #75

Texas electricity firm files for bankruptcy citing $1.8 billion in claims from grid operator (

Brazos and others that committed to provide power to the grid – and could not – were required to buy replacement power at high rates and cover other firms’ unpaid fees.

The grid operator, the Electric Reliability Council of Texas (ERCOT), on Monday said that $2.46 billion in bills went unpaid, underscoring the financial stress on utilities and power marketers. ERCOT acts in part as a clearinghouse, collecting from power buyers and paying those who provide the electrons.

The market will regulate itself…

The Art of Reading More Effectively and Efficiently (

Good approach to reading in general and the different levels of reading books can have.

5G: The outsourced elephant in the room (

European service providers have however had reasons beyond balance-sheet gymnastics to outsource: we have traditionally not valued (telecommunication) engineering expertise. Instead for years service providers have glorified their marketing and finance departments.

Longread of the week on why it’s hard to achieve sovereignty over 5G infrastructure.

On-Boarding New Employees Remotely (

Great summary on how to successfully get new employees on-board in an all-remote company.

How often should I rotate my ssh keys? (

Anyway, back to our original question: how often should I rotate my ssh keys? More often than never! As often as you can. And make sure you retire old keys when you’re done with them.

Good reminder on rotating SSH Keys 🙂

Video shows NYPD’s new robotic dog in action in the Bronx (

They looked much cuter when they were dancing. But now they are cute and policing through the streets. And I can’t get around flashbacks of Black Mirror – Metalhead S4E5

IoTaWatt™ Open WiFi Electric Power Monitor (

As a residential whole-house monitor IoTaWatt measures mains, individual circuits, and solar. Any voltage or frequency, single phase or three-phase, with easy installation and configuration. As a standalone unit it saves and displays years of detailed data, while also capable of uploading real-time usage to any of several popular cloud services.

This looks super interesting and not too expensive to get it 🙂

IPO filing: Coinbase Global, Inc. (

IPO Filing: DigitalOcean Holdings, Inc. (

I paid for Spotify playlist placements so you don’t have to (

Paying for Spotify playlist promotion does not relate to an increase in followers. Do not use it if this is your goal.

A few interesting metrics for people in the spotify game.