Remove more old stuff.

2023-04-29 16:23:34 -04:00
1 parent 42465d90f4
commit c386292425
@@ -0,0 +1,678 @@
+# Design for Second Iteration of Cluster/Homelab
+
+## Context
+
+Current cluster was set up just to run CI builds as a
+trial.
+
+I'm now sold the k8s is a good approach and would like
+to move more of my services to it.
+
+This document will track my design for cluster v2.
+
+
+## Investigation
+
+### Host OS
+
+Debian:
+- on laptop
+- already on most of systems
+- stable
+- not officially tested by k3s
+- Will be using apt at work
+
+Stream:
+- Tried with k3s and had to disable systemd...
+    - On second try seemed to work even with error I saw before.
+- Cockpit is nice when managing servers.
+- Want to like RHEL
+- More stable than Fedora
+- RPMs are easier to work with
+- Using on VM host
+
+Fedora:
+- Want to like RHEL
+- Tested with k3s
+- Latest podman and frieds
+- Really fast for something stable...
+- Cockpit is nice
+- Fedora minimal can't be installed on
+  cockpit without hitting tab alot.
+
+Decision: Fedora Server
+
+
+### K3S Distro
+
+RKE2:
+   - no Debian support
+   - 4GB Minimum
+   - 2 CPU
+   - cilium and nginx not default
+
+k3s:
+   - k3d is a thing
+   - documentation online is good
+   - 512 MB of RAM
+   - 1 CPU
+   - easy installation
+
+Decision: k3s
+
+### How many clusters?
+
+Decision: Exactly two (one for "need to work" services one for CI and messing around).
+The mess with longhorn scared me... it wouldn't be that big a deal if it only effected
+CI, but it also effect Kanboard and git.
+
+### Files
+
+Decision: Host local.
+
+Files are not something I want to have to think about.
+The longhorn mess scared me.
+NFS not working with postgres is annoying.
+
+### How many nodes per cluster?
+
+The current cluster has lots of small VMs, with VMs added with more
+CPUs/RAM as the requirements grew.
+
+I'd rather limit myself to fewer more powerful VMs, and let the VM OS manage
+CPU and memory.
+
+More nodes would be useful if they were on different base hardware.
+Realistically I'm never going to pay for more than the Ingress VM...
+
+Decision:
+1 big VM per cluster.
+Both VMs hosted on current hardware.
+If we add hardware, can add an additional node at that time.
+
+### Networking
+
+Status quo is flannel with vxlan with Traefik and Klipper LB and CoreDNS.
+
+#### DNS
+
+CoreDNS is great.
+
+#### Load Balancer
+
+Klipper works fine now.
+MetalLB is the other option, is more complicated and doesn't
+seem to give much particularly with a single node cluster.
+
+Decision: Klipper
+
+#### Ingress
+
+Traefik:
+- Status Quo.
+- Works fine.
+- Outside of k8s I don't like.
+
+nginx-ingress:
+- Google
+- Used by a lot of people.
+- Nothing sexy or risky.
+- auth exposed in annotations
+
+ingress-nginx:
+- nginx upstream.
+- extra features like stream support that I'm using
+  on lightsail now.
+- full blown virtual server support.
+- maybe too complicated?
+- exposes same features as I have on lightsail through
+  annotations, which could be a thing to get keycloack to
+  work.
+- auth in annotations is behind paywall, but available through
+  a virtual server
+
+Decision: nginx-ingress
+Use LB for stuff I would use the virtual server for.
+
+#### CNI
+
+flannel vxlan
+- status quo
+- works fine
+
+cilium
+- label based network policies
+- leaning toward this plus multus though I doubt
+  I'll ever write a policy
+- I want the ability to write a policy...
+- if set up different pod cidr can do multi-cluster later
+   - cluster name and cluster id at install time
+- can do transparent encryption (not worth it...)
+
+cilium multi-cluster networking:
+- not worth the complexity
+- will manage connections with ingress/egress methods.
+
+flannel wg
+- encrypt traffic and set up overlay if I want to interact with
+  cloud machines
+- can do the same with a manual wireguard network...
+
+istio
+- I dislike side car containers
+- Traffic I'm interested in is mainly not L7.
+- blessed by Air Force
+
+multus:
+- tried on fedora and didn't get very far I think
+  because of something with k3s.
+
+Decision: cilium
+want network policies and hubble observability
+is a risk, but this is supposed to be a learning
+experience.
+
+## What goes on each cluster/VM?
+
+Lightsail:
+1. Wireguard
+2. Apt/RPM repos
+3. Main NGINX Proxy
+
+Infra Cluster:
+- On Host:
+    1. CoreDNS
+    2. Wireguard
+- On Cluster:
+    1. Keycloak
+    2. Kanboard
+    3. OneDev
+    4. Harbor
+
+Main Cluster:
+- On Host:
+    1. Wireguard
+- On Cluster:
+    1. Tekton
+    2. MQTT Broker
+    3. Squid
+    4. j7s-os-deployment
+
+## Deployments
+
+Manually kubectl apply:
+- Easy to reason about
+- running apply is fun
+- using flux has chicken and egg problem if git is also
+  deployed from flux
+
+Flux:
+- More git ops-y
+- chicken and egg problem is conquerable, in a maybe
+  confusing way
+
+Decision:
+1. Infra:
+  1. kubectl apply/helm everything
+  2. Drop keycloak image manually in k3s either using cri or
+     placing in magic place after k3s install.
+  3. Use helm with values for onedev.
+  4. Get rid of Kanboard custom image.
+     Use kubectl apply.
+2. Test:
+   1. Mostly kubectl apply for tekton.
+   2. Use flux for:
+      1. MQTT
+      2. j7s-os-deploy
+      3. squid
+
+## VM Resources
+
+Lightsail:
+- Leave alone
+
+Infra Cluster:
+- RAM 4 GiB total
+- 2 CPUs
+- 120Gib Hardrive
+
+Main Cluster:
+- RAM 4 GiB total
+- 2 CPUs
+- 120Gib Hardrive
+
+## Secrets
+
+Options:
+Mozilla Kops
+Bitnami Sealed Secrets
+
+Both work with Flux.
+Sealed Secrets seems more integrated with k8s when not using
+Flux.
+
+Decision: Bitnami Sealed Secrets
+
+## Experiments
+
+### k3s with cilium and nginx on Centos Stream 9
+
+```
+systemctl disable firewalld --now
+export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none --disable-network-policy --selinux"
+curl -sfL https://get.k3s.io | sh -s -
+```
+I see an error about selinux policies conflicting, but I'm not sure if it matters?
+
+Install cilium following instructions here:
+https://docs.cilium.io/en/v1.12/gettingstarted/k3s/
+
+Install nginx with:
+```
+helm upgrade --install ingress-nginx ingress-nginx \
+  --repo https://kubernetes.github.io/ingress-nginx \
+  --namespace ingress-nginx --create-namespace \
+  --set controller.ingressClassResource.default=true
+```
+
+### k3s with nginx on fedora server
+```
+sudo systemctl disable firewalld --now
+export INSTALL_K3S_EXEC="server --disable traefik --selinux"
+curl -sfL https://get.k3s.io | sh -s -
+sudo chown jimmy:jimmy /etc/rancher/k3s/k3s.yaml
+sudo dnf install helm
+export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
+```
+
+Import simple-ros2.
+Laptop:
+```
+podman save -o simple-ros2.tar simple-ros2:latest
+scp simple-ros2.tar 192.168.1.106:~/.
+```
+On server:
+```
+sudo ctr images import ./simple-ros2.tar
+# wait forever....
+```
+
+Test yaml:
+```
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-pod
+spec:
+  containers:
+  - name: simple-ros2
+    image: localhost/simple-ros2:latest
+    imagePullPolicy: Never
+    args: [ros2, launch, j7s-simple, j7s_publisher_launch.py]
+```
+
+### VM Host set up
+
+I **think** I ran something like this when I set up the VM host.
+I don't remember exactly, and I didn't document it...
+
+This should be carefully looked at before running.
+
+```
+nmcli connection add ifname br0 type bridge con-name br0 connection.zone trusted
+nmcli connection add type bridge-slave ifname enp4s0 master br0
+nmcli connection modify br0 bridge.stp no
+nmcli connection modify enp4s0 autoconnect no
+nmcli connection down enp4s0
+nmcli connection up id br0
+```
+
+### Kubeseal Use
+```
+apiVersion: v1
+kind: Secret
+metadata:
+  name: test-secret
+  namespace: my-namespace
+type: Opaque 
+data: 
+  username: dmFsdWUtMQ0K 
+  password: dmFsdWUtMg0KDQo=
+stringData: 
+  hostname: myapp.mydomain.com
+```
+cat secret.yaml | kubeseal --format yaml > sealedsecret.yaml
+
+# Actual Install Notes
+
+## To Do List
+
+Infra Cluster: [x]
+- On Host:
+    1. CoreDNS [x]
+    2. Wireguard [x]
+- On Cluster:
+    1. Keycloak [x]
+    2. Kanboard [x]
+    3. Gitea [x]
+    4. Harbor [x]
+
+Main Cluster:
+- On Host:
+    1. Wireguard [x]
+- On Cluster:
+    1. Tekton
+       Base install [ ]
+       Add namespace
+       Push images
+       Update tasks
+       Update jobs
+    5. Flux
+        1. MQTT Broker
+        2. Squid
+        3. j7s-os-deployment
+
+[x] Give accounts on Harbor to clusters.
+[ ] Push images to Harbor.
+[x] Hubble.
+
+## Regularly Scheduled Programming
+
+Fedora Server 37 keep defaults.
+
+Infra:
+On VM:
+```
+sudo hostnamectl set-hostname infra-cluster
+sudo systemctl disable firewalld --now
+sudo su
+export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none --disable-network-policy --cluster-cidr 10.44.0.0/16 --service-cidr 10.45.0.0/16 --cluster-dns 10.45.0.10 --selinux"
+curl -sfL https://get.k3s.io | sh -s -
+exit
+sudo cp /etc/rancher/k3s/k3s.yaml ~/infra.yaml
+sudo chown jimmy:jimmy ~/infra.yaml
+exit
+```
+
+on laptop
+```
+scp jimmy@192.168.1.112:~/infra.yaml /home/jimmy/.kube/.
+export KUBECONFIG=~/.kube/infra.yaml
+vim KUBECONFIG and fix ip.
+```
+Install cilium cli.
+
+On laptop:
+```
+cilium install
+```
+wait...
+```
+helm upgrade --debug --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
+```
+Main:
+On VM:
+```
+sudo hostnamectl set-hostname j7s-cluster
+sudo systemctl disable firewalld --now
+sudo su
+export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none --disable-network-policy --cluster-cidr 10.46.0.0/16 --service-cidr 10.47.0.0/16 --cluster-dns 10.47.0.10 --selinux --resolv-conf /run/systemd/resolve/resolv.conf"
+curl -sfL https://get.k3s.io | sh -s -
+exit
+sudo cp /etc/rancher/k3s/k3s.yaml ~/j7s-cluster.yaml
+sudo chown jimmy:jimmy ~/j7s-cluster.yaml
+exit
+```
+
+on laptop
+```
+scp jimmy@192.168.1.103:~/j7s-cluster.yaml /home/jimmy/.kube/.
+export KUBECONFIG=~/.kube/j7s-cluster.yaml
+vim KUBECONFIG and fix ip.
+```
+On laptop:
+```
+cilium install
+```
+wait...
+```
+helm upgrade --debug --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace
+```
+
+Install Sealed Secrets:
+
+Main:
+```
+export KUBECONFIG=~/.kube/j7s-cluster.yaml
+wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.5/controller.yaml
+kubectl apply -f controller.yaml
+```
+Infra:
+```
+export KUBECONFIG=~/.kube/infra.yaml
+kubectl apply -f controller.yaml
+rm controller.yaml
+```
+
+Install kubeseal.
+
+Merge kube config files:
+
+1. Manually modify each config file and get rid of all the defaults
+   to something unique for that file.
+   ( I have k3s for the original cluster, j7s for the new main cluster, and infra
+     for the new infra cluster. )
+2. Do some magic.
+```
+cp config.yaml config.yaml.back.<date>
+export KUBECONFIG=~/.kube/config:~/.kube/infra.yaml:~/.kube/j7s-cluster.yaml
+kubectl config view --flatten > new-config
+mv new-confg config
+export KUBECONFIG=~/.kube/config
+chmod 600 ~/.kube/config
+```
+
+Use kubeseal to encrypt secrets for harbor.
+
+Install harbor.
+```
+cd infra-cluster/harbor
+kubectl apply -f namespace
+kubectl apply -f secrets
+cd helm
+./install.bash
+```
+
+Build coredns rpm following instructions in coredns folder.
+scp to infra:
+```
+scp redhat/RPMS/x86_64/coredns-1.8.4-1.fc37.x86_64.rpm jimmy@192.168.1.112:~/.
+ssh jimmy@192.168.1.112
+sudo dnf install ./coredns-1.8.4-1.fc37.x86_64.rpm
+exit
+```
+Copy over corefile from coredns folder.
+```
+scp Corefile jimmy@192.168.1.112:~/.
+ssh jimmy@192.168.1.112
+sudo cp Corefile /etc/coredns/Corefile
+sudo systemctl start coredns
+sudo systemctl enable coredns
+
+sudo dnf install policycoreutils-devel rpm-build
+sepolicy generate --application /bin/coredns
+./coredns.sh
+# Until it works....
+sudo su
+ausearch -c '(coredns)' --raw | audit2allow -M my-coredns
+semodule -i my-coredns.pp
+# Also:
+sudo setsebool -P domain_can_mmap_files 1
+# Turn of resolver.
+sudo vim /etc/systemd/resolved.conf
+DNSStubListener=no
+```
+
+Wound up turning off SELinux...
+```
+sudo vi /etc/selinux/config
+# SELINUX=permissive
+sudo grubby --update-kernel ALL --args selinux=0
+```
+
+Wound up reverting back.
+
+Add:
+```
+CapabilityBoundingSet=CAP_NET_BIND_SERVICE
+AmbientCapabilities=CAP_NET_BIND_SERVICE
+```
+
+under `[Service]` in
+```
+sudo vim /usr/lib/systemd/system/coredns.service
+```
+
+Wireguard:
+
+```
+sudo dnf install wireguard-tools
+wg genkey | tee wg.key | wg pubkey > wg.pub
+vim wg0.conf
+<<<
+[Interface]
+Address = 10.100.100.?/24
+PrivateKey = <Contents from file.>
+
+[Peer]
+PublicKey = zgcRWY3MAwKGokyRs9dR4E5smoeFy1Hh4MfDcDM3iSc=
+AllowedIPs = 10.100.100.0/24
+Endpoint = vpn.jpace121.net:51902
+PersistentKeepAlive = 25
+<<<<
+```
+Add to server:
+```
+# Infra k3s node
+[Peer]
+PublicKey = <>
+AllowedIPs = 10.100.100.7/32
+
+# Add to systemd 
+sudo systemctl enable wg-quick@wg0.service 
+sudo systemctl daemon-reload 
+sudo systemctl start wg-quick@wg0 
+ ```
+
+Tried using nm below, moved to wg-quick for consistency.
+```
+nmcli con import type wireguard file /etc/wireguard/wg0.conf
+```
+
+ Better:
+ ```
+sudo cp wg0.conf /etc/wireguard/wg0.conf
+sudo chown root:root /etc/wireguard/wg0.conf
+wg-quick up wg0
+```
+
+Harbor Login:
+
+```
+scp harbor_tls.crt jimmy@10.100.100.7:.
+ssh jimmy@10.100.100.7
+sudo cp harbor_tls.crt /etc/rancher/k3s/.
+```
+
+`/etc/rancher/k3s/registries.yaml`
+```
+configs:
+  "harbor.internal.jpace121.net":
+    auth:
+      username: robot$k8s+infra-cluster
+      password: <from harbor>
+    tls:
+      ca_file: /etc/rancher/k3s/harbor_tls.crt
+```
+
+Kanboard:
+
+Get PV Name:
+```
+kubectl describe pvc kanboard-pvc --context k3s
+```
+Use PV name to locate directory:
+```
+kubectl describe pv pvc-89a4265c-b39c-4628-9e6b-df091fae4fd8 --context k3s
+```
+
+Can tell on `k3s-node1` at `/var/lib/rancher/k3s/storage/pvc-89a4265c-b39c-4628-9e6b-df091fae4fd8_default_kanboard-pvc`
+
+
+```
+ssh jimmy@192.168.1.135
+sudo su
+cd /var/lib/rancher/k3s/storage/pvc-89a4265c-b39c-4628-9e6b-df091fae4fd8_default_kanboard-pvc
+tar cvpzf /home/jimmy/kanboard-pvc.tar.gz .
+exit
+cd ~
+sudo chown jimmy:jimmy kanboard-pvc.tar.gz
+exit
+scp jimmy@192.168.1.135:~/kanboard-pvc.tar.gz /tmp/kanboard-pvc.tar.gz
+```
+Apply PVC.
+Want: `volumeBindingMode: Immediate`
+```
+kubectl apply manifests --context infra
+<wait til pvc exists>
+<delete everyone but pvc>
+kubectl describe pvc kanboard-pvc --context infra --namespace kanboard
+kubectl describe pv pvc-fe710c38-52ce-495b-bb8d-bea48222a21b --namespace kanboard
+```
+
+```
+scp /tmp/kanboard-pvc.tar.gz jimmy@192.168.1.112:.
+ssh jimmy@192.168.1.112
+sudo su
+chown root:root ./kanboard-pvc.tar.gz
+cd /var/lib/rancher/k3s/storage/pvc-fe710c38-52ce-495b-bb8d-bea48222a21b_kanboard_kanboard-pvc
+rm -rf *
+tar xpvzf /home/jimmy/kanboard-pvc.tar.gz
+exit
+exit
+kubectl apply -f manifests/
+```
+Make secret:
+```
+cat kanboard-cookie.yaml | kubeseal --format yaml > kanboard-cookie-sealed.yaml
+```
+
+Where should I proxy to?
+```
+kubectl -n ingress-nginx get svc
+ngress-nginx-controller             LoadBalancer   10.45.94.103    192.168.1.112   80:31566/TCP,443:32594/TCP   23d
+```
+> 10.100.100.7:31566
+
+### Tekton
+
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
+
+### Keycloak
+kubectl describe pv pvc-4bcbb023-e686-4082-855f-d062ff418c74 --namespace keycloak
+`/var/lib/rancher/k3s/storage/pvc-4bcbb023-e686-4082-855f-d062ff418c74_keycloak_keycloak-db-pvc`
+`scp /tmp/db-backup.tar.gz jimmy@192.168.1.112:.`
+```
+sudo su
+chown root:root ./db-backup.tar.gz
+cd /var/lib/rancher/k3s/storage/pvc-4bcbb023-e686-4082-855f-d062ff418c74_keycloak_keycloak-db-pvc
+rm -rf *
+tar xpvzf /home/jimmy/db-backup.tar.gz
+sudo chown -R systemd-oom:systemd-oom *
+```
+
@@ -0,0 +1,218 @@
+# Notes
+
+Install:
+
+1. Set up wireguard.
+2. Download k3s install script from website.
+3. For master:
+   `./k3s.sh`
+4. For node:
+   `curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -`
+   "The value to use for K3S_TOKEN is stored at /var/lib/rancher/k3s/server/node-token"
+5. Install kubectl on laptop.
+6. Copy `/etc/rancher/k3s/k3s.yaml` to laptop and change localhost IP to wireguard IP.
+7. `kubectl cluster-info`
+8. Install tkn CLI.
+   `https://tekton.dev/docs/cli/`
+   I installed manually.
+4. Apply dns updates and rollout restart of codedns:
+   `kubectl rollout restart -n kube-system deployment/coredns`
+
+Install Tekton:
+```
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
+
+```
+Set up local registry on master.
+(See below.)
+
+Tell k3s about it:
+```sudo vim /etc/rancher/k3s/registries.yaml```
+```
+configs:
+  "192.168.1.128:8443":
+    auth:
+      username: k3s
+      password: password
+    tls:
+      ca_file: /home/jimmy/registry/certs/domain.crt
+
+```
+Restart k3s.
+
+Apply rest of the CRDs.
+
+# SSH Secrets
+
+1. `ssh-keygen -t ecdsa -f ./deploy_key`
+2. `ssh-keyscan packages.jpace121.net > ./deploy_known_hosts`
+3. `cat deploy-credentials.yaml`
+   ```
+    apiVersion: v1
+    kind: Secret
+    metadata:
+        name: deploy-credentials
+    type: Opaque
+    data:
+        id_ecdsa: <base64 -w 0 .. >
+        known_hosts: <base64 -w 0 ..>
+  ```
+
+# Set up Tekton Dashboard:
+```
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
+```
+Port forward locally:
+```
+kubectl port-forward -n tekton-pipelines service/tekton-dashboard 9097:9097
+```
+
+# Local Registry
+I could have done a much better job of documenting this.
+```
+mkdir registry/
+cd registry/
+mkdir certs auth data
+cd certs/
+openssl genrsa 1024 > domain.key
+chmod 400 domain.key
+vim san.cnf
+```
+san.cf
+```
+[req]                    
+default_bits  = 2048         
+distinguished_name = req_distinguished_name
+req_extensions = req_ext
+x509_extensions = v3_req
+prompt = no
+
+[req_distinguished_name]
+countryName = US
+stateOrProvinceName = PA
+localityName = Pittsburgh
+organizationName = j7s k3s CA
+commonName = j7s k3s CA
+
+[req_ext]
+subjectAltName = @alt_names
+
+[v3_req]
+subjectAltName = @alt_names
+
+[alt_names]
+IP.1 = 10.100.100.5
+IP.2 = 192.168.1.128
+```
+```
+openssl req -new -x509 -nodes -days 36500 -key domain.key -out domain.crt -config san.cnf
+ls
+cd ..
+ls
+cd auth/
+podman run   --entrypoint htpasswd   docker.io/library/httpd:2 -Bbn k3s password > htpasswd
+cd ..
+vim run.sh
+```
+
+run.sh
+```
+#!/usr/bin/env bash
+podman run -d \
+        --restart=always \
+        --name registry \
+        -v `pwd`/auth:/auth \
+        -v `pwd`/certs:/certs \
+        -v `pwd`/data:/var/lib/registry \
+        -e REGISTRY_AUTH=htpasswd \
+        -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
+        -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
+        -e REGISTRY_HTTP_ADDR=0.0.0.0:8443 \
+        -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
+        -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
+        -e REGISTRY_STORAGE_DELETE_ENABLED=true \
+        -p 8443:8443 \
+        registry:latest
+```
+
+```
+sudo firewall-cmd --permanent --add-port=8443/tcp
+sudo firewall-cmd --reload
+```
+
+# NFS
+
+Server: CentOS 9
+Set up:
+```
+sudo dnf install nfs-utils vim
+sudo mkdir /srv/nfs
+sudo chown jimmy:jimmy /srv/nfs
+sudo chmod 777 /srv/nfs/
+```
+Put into `/etc/exports`:
+```
+/srv/nfs    192.168.1.0/24(rw,root_squash)
+```
+Start everything:
+```
+systemctl enable --now rpcbind
+systemctl enable --now nfs-server
+firewall-cmd --permanent --add-service nfs
+firewall-cmd --reload
+systemctl restart nfs-server
+```
+
+Test on Debian:
+```
+sudo apt install nfs-common
+sudo mkdir -p /mnt/nfs
+sudo mount 192.168.1.149:/srv/nfs /mnt/nfs
+```
+
+On the k3s nodes:
+```
+sudo apt install nfs-common
+```
+
+Install to the cluster:
+```
+helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
+helm install --namespace nfs-subdir-external-provisioner nfs-subdir-external-provisioner \
+  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
+  --set storageClass.onDelete=delete \
+  --set nfs.server=192.168.1.149 \
+  --set nfs.path=/srv/nfs
+```
+
+# Chains
+Set up:
+```
+kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/previous/v0.14.0/release.yaml
+# Apply secret from j7s-intoto.
+# name: signing-secrets namespace: tekton-chains data: x509.pem: base64 of pem
+kubectl apply -f chains-config.yaml
+kubectl rollout restart -n tekton-chains deployment tekton-chains-controller
+```
+
+See:
+```
+export TASKRUN_UID=$(tkn pr describe --namespace j7s-ci --last -o  jsonpath='{.metadata.uid}')
+tkn pr describe --namespace j7s-ci --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-pipelinerun-$TASKRUN_UID}" > signature
+tkn pr describe --namespace j7s-ci --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/payload-pipelinerun-$TASKRUN_UID}" | base64 -d > payload
+```
+
+# Future Ideas
+
+If we later want to do this on an overlay network:
+3. For master:
+   `INSTALL_K3S_EXEC="server --node-ip '10.100.100.5' --advertise-address '10.100.100.5' --flannel-iface 'wg0'" ./k3s.sh`
+4. For node:
+   `INSTALL_K3S_EXEC="agent --server 'https://10.100.100.5:6443' --token 'K3S_TOKEN' --node-ip '10.100.100.?' --advertise-address '10.100.100.?' --flannel-iface 'wg0'" ./k3s.sh`
+
+# Bad Ideas
+
+1. Longhorn -> wonky performance issues on cluster after installing
+2. Multus -> CNI version does not seem compatible with k3s.