From 0f8219e4854e484e55cdd17c3aa6356783268c09 Mon Sep 17 00:00:00 2001 From: stephan48 Date: Sun, 13 Aug 2023 21:16:41 +0200 Subject: [PATCH] --- ...t-a-rootless-cluster-with-bgp-damn-it.mdwn | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn diff --git a/posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn b/posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn new file mode 100644 index 0000000..858f0b2 --- /dev/null +++ b/posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn @@ -0,0 +1,36 @@ +This post should details the crazy journey on how i managed(or failed) to waste lots of my time and in the end got a rootless k3s cluster with bgp for loadbalancer IPs via metallb. + +Your sanity may not survive this read. + +How does k3s setup the network: +* k3s server is started - calls into rootlesskit https://github.com/k3s-io/k3s/blob/38a0b91c1a917d2866aee265bc7815424af3e701/pkg/rootless/rootless.go#L37 +* k3s server then forks itself(where?) to handle parent(outside netns) and child(inside netns) operations. +* does not allow us to change from slirp4netns to something else like lxc-nic(easier to patch). +* Rootlesskit creates external(slirp4netns) process to attach tun interface - https://github.com/rootless-containers/rootlesskit/blob/master/pkg/network/slirp4netns/slirp4netns.go#L176 + * slirp4netns is called with specific options, theres no intelligent return mechanism for interface config, hence we have to reuse whats there +* Rootlesskit does child network configuration - https://github.com/rootless-containers/rootlesskit/blob/master/pkg/child/child.go#L156 + * where does the tap0 device comes from? + +Wrote slirp4netns wrapper(crude & insecure, will need to harden): +* rootless wrapper: + * will write environment infos as json for rootfull process to intercept and handle + * will wait until marker file is there to wait for further startup +* rootfull wrapper: + * takes network information from json file + * creates additional netns for support processes(i.e. attaching to lan network via dhcp) + * dns, bgp(to lan, to cluster) + * setups correct IP addresses on both sides + * network links: + * helper ns <-> host - attach do well defined bridge on host for dhcp + * helper ns <-> cluster - uplink handling + +Learnings: +* lsns -t net is freaking awesome +* you can address network namespaces by name(optional), pids of processed in them, network namespace id +* unless you are root on the host you can't switch between network namespaces + +TODO: +* how to handle ipv6? +* how to conjure all of this securely? + + -- 2.30.2