This post should details the crazy journey on how i managed(or failed) to waste lots of my time and in the end got a rootless k3s cluster with bgp for loadbalancer IPs via metallb.

Your sanity might not survive this read. Mine did not survive the project :P

... insert (k3s) rootless working principle here ...

How does k3s setup the network:

  • k3s server is started - calls into rootlesskit https://github.com/k3s-io/k3s/blob/38a0b91c1a917d2866aee265bc7815424af3e701/pkg/rootless/rootless.go#L37
  • k3s server then forks itself(where?) to handle parent(outside netns) and child(inside netns) operations.
  • does not allow us to change from slirp4netns to something else like lxc-nic(easier to patch).
  • Rootlesskit creates external(slirp4netns) process to attach tap interface - https://github.com/rootless-containers/rootlesskit/blob/master/pkg/network/slirp4netns/slirp4netns.go#L176
    • slirp4netns is called with specific options, theres no intelligent return mechanism for interface config, hence we have to reuse whats there
  • Rootlesskit does child network configuration - https://github.com/rootless-containers/rootlesskit/blob/master/pkg/child/child.go#L156
    • where does the tap0 device comes from? https://github.com/rootless-containers/rootlesskit/blob/master/pkg/network/slirp4netns/slirp4netns.go#L182
    • we will need to remove this and redo some of the network configuration
  • k3s packages its own slirp4netns, need to override that to do our magic
    • ln -fs bin/slirp4netns .rancher/k3s/data/current/bin/slirp4netns
    • because k3s sets up the bin to be used here :( https://github.com/k3s-io/k3s/blob/0d23cfe038ef22d7ca899764e9aaeea8a39d4874/cmd/k3s/main.go#L189

Wrote slirp4netns wrapper(crude & insecure, will need to harden):

  • rootless wrapper:
    • will write environment infos as json for rootfull process to intercept and handle
    • will wait until marker file is there to wait for further startup
  • rootfull wrapper:
    • takes network information from json file
    • creates additional netns for support processes(i.e. attaching to lan network via dhcp)
    • dns, bgp(to lan, to cluster)
    • setups correct IP addresses on both sides
    • network links:
      • helper ns <-> host - attach do well defined bridge on host for dhcp
      • helper ns <-> cluster - uplink handling

Learnings:

  • lsns -t net is freaking awesome
  • you can address network namespaces by name(optional), pids of processed in them, network namespace id
  • unless you are root on the host you can't switch between network namespaces

TODO:

  • how to handle ipv6?
  • how to conjure all of this securely?
  • what provides dns?
    • in slirp4netns?
    • can i reuse avard-dns? netavard? postman?
    • resort do dnsmasq?

Interesting Reads: * https://linux-blog.anracom.com/2017/10/30/fun-with-veth-devices-in-unnamed-linux-network-namespaces-i/

It works!!!

TODO: Document Magic Script, Automate IT, PROFIT

IPv6:

Yes because i am a masochist.

Test

Test Heading

Graphical Depiction of Network Setup

Click on shapes to get to the respective part of the documentation.