This post should details the crazy journey on how i managed(or failed) to waste lots of my time and in the end got a rootless k3s cluster with bgp for loadbalancer IPs via metallb.
-Your sanity may not survive this read.
+Your sanity might not survive this read. Mine did not survive the project :P
How does k3s setup the network:
+
* k3s server is started - calls into rootlesskit https://github.com/k3s-io/k3s/blob/38a0b91c1a917d2866aee265bc7815424af3e701/pkg/rootless/rootless.go#L37
* k3s server then forks itself(where?) to handle parent(outside netns) and child(inside netns) operations.
* does not allow us to change from slirp4netns to something else like lxc-nic(easier to patch).
* where does the tap0 device comes from?
Wrote slirp4netns wrapper(crude & insecure, will need to harden):
+
* rootless wrapper:
* will write environment infos as json for rootfull process to intercept and handle
* will wait until marker file is there to wait for further startup
* helper ns <-> cluster - uplink handling
Learnings:
+
* lsns -t net is freaking awesome
* you can address network namespaces by name(optional), pids of processed in them, network namespace id
* unless you are root on the host you can't switch between network namespaces
TODO:
+
* how to handle ipv6?
* how to conjure all of this securely?