(no commit message)

diff --git a/posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn b/posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn
new file mode 100644 (file)
index 0000000..858f0b2
--- /dev/null
+++ b/posts/2023-08-13-k3s-i-want-a-rootless-cluster-with-bgp-damn-it.mdwn
@@ -0,0 +1,36 @@
+This post should details the crazy journey on how i managed(or failed) to waste lots of my time and in the end got a rootless k3s cluster with bgp for loadbalancer IPs via metallb.
+
+Your sanity may not survive this read.
+
+How does k3s setup the network:
+* k3s server is started - calls into rootlesskit https://github.com/k3s-io/k3s/blob/38a0b91c1a917d2866aee265bc7815424af3e701/pkg/rootless/rootless.go#L37
+* k3s server then forks itself(where?) to handle parent(outside netns) and child(inside netns) operations.
+* does not allow us to change from slirp4netns to something else like lxc-nic(easier to patch).
+* Rootlesskit creates external(slirp4netns) process to attach tun interface - https://github.com/rootless-containers/rootlesskit/blob/master/pkg/network/slirp4netns/slirp4netns.go#L176
+  * slirp4netns is called with specific options, theres no intelligent return mechanism for interface config, hence we have to reuse whats there
+* Rootlesskit does child network configuration - https://github.com/rootless-containers/rootlesskit/blob/master/pkg/child/child.go#L156
+  * where does the tap0 device comes from?
+
+Wrote slirp4netns wrapper(crude & insecure, will need to harden):
+* rootless wrapper:
+  * will write environment infos as json for rootfull process to intercept and handle
+  * will wait until marker file is there to wait for further startup
+* rootfull wrapper:
+  * takes network information from json file
+  * creates additional netns for support processes(i.e. attaching to lan network via dhcp)
+  * dns, bgp(to lan, to cluster)
+  * setups correct IP addresses on both sides
+  * network links:
+    * helper ns <-> host - attach do well defined bridge on host for dhcp
+    * helper ns <-> cluster - uplink handling
+
+Learnings:
+* lsns -t net is freaking awesome
+* you can address network namespaces by name(optional), pids of processed in them, network namespace id
+* unless you are root on the host you can't switch between network namespaces
+
+TODO:
+* how to handle ipv6?
+* how to conjure all of this securely?
+
+