15 KiB

Raw Blame History

Design for Second Iteration of Cluster/Homelab

Context

Current cluster was set up just to run CI builds as a trial.

I'm now sold the k8s is a good approach and would like to move more of my services to it.

This document will track my design for cluster v2.

Investigation

Host OS

Debian:

on laptop
already on most of systems
stable
not officially tested by k3s
Will be using apt at work

Stream:

Tried with k3s and had to disable systemd...
- On second try seemed to work even with error I saw before.
Cockpit is nice when managing servers.
Want to like RHEL
More stable than Fedora
RPMs are easier to work with
Using on VM host

Fedora:

Want to like RHEL
Tested with k3s
Latest podman and frieds
Really fast for something stable...
Cockpit is nice
Fedora minimal can't be installed on cockpit without hitting tab alot.

Decision: Fedora Server

K3S Distro

RKE2:

no Debian support
4GB Minimum
2 CPU
cilium and nginx not default

k3s:

k3d is a thing
documentation online is good
512 MB of RAM
1 CPU
easy installation

Decision: k3s

How many clusters?

Decision: Exactly two (one for "need to work" services one for CI and messing around). The mess with longhorn scared me... it wouldn't be that big a deal if it only effected CI, but it also effect Kanboard and git.

Files

Decision: Host local.

Files are not something I want to have to think about. The longhorn mess scared me. NFS not working with postgres is annoying.

How many nodes per cluster?

The current cluster has lots of small VMs, with VMs added with more CPUs/RAM as the requirements grew.

I'd rather limit myself to fewer more powerful VMs, and let the VM OS manage CPU and memory.

More nodes would be useful if they were on different base hardware. Realistically I'm never going to pay for more than the Ingress VM...

Decision: 1 big VM per cluster. Both VMs hosted on current hardware. If we add hardware, can add an additional node at that time.

Networking

Status quo is flannel with vxlan with Traefik and Klipper LB and CoreDNS.

DNS

CoreDNS is great.

Load Balancer

Klipper works fine now. MetalLB is the other option, is more complicated and doesn't seem to give much particularly with a single node cluster.

Decision: Klipper

Ingress

Traefik:

Status Quo.
Works fine.
Outside of k8s I don't like.

nginx-ingress:

Google
Used by a lot of people.
Nothing sexy or risky.
auth exposed in annotations

ingress-nginx:

nginx upstream.
extra features like stream support that I'm using on lightsail now.
full blown virtual server support.
maybe too complicated?
exposes same features as I have on lightsail through annotations, which could be a thing to get keycloack to work.
auth in annotations is behind paywall, but available through a virtual server

Decision: nginx-ingress Use LB for stuff I would use the virtual server for.

CNI

flannel vxlan

status quo
works fine

cilium

label based network policies
leaning toward this plus multus though I doubt I'll ever write a policy
I want the ability to write a policy...
if set up different pod cidr can do multi-cluster later
- cluster name and cluster id at install time
can do transparent encryption (not worth it...)

cilium multi-cluster networking:

not worth the complexity
will manage connections with ingress/egress methods.

flannel wg

encrypt traffic and set up overlay if I want to interact with cloud machines
can do the same with a manual wireguard network...

istio

I dislike side car containers
Traffic I'm interested in is mainly not L7.
blessed by Air Force

multus:

tried on fedora and didn't get very far I think because of something with k3s.

Decision: cilium want network policies and hubble observability is a risk, but this is supposed to be a learning experience.

What goes on each cluster/VM?

Lightsail:

Wireguard
Apt/RPM repos
Main NGINX Proxy

Infra Cluster:

On Host:
1. CoreDNS
2. Wireguard
On Cluster:
1. Keycloak
2. Kanboard
3. OneDev
4. Harbor

Main Cluster:

On Host:
1. Wireguard
On Cluster:
1. Tekton
2. MQTT Broker
3. Squid
4. j7s-os-deployment

Deployments

Manually kubectl apply:

Easy to reason about
running apply is fun
using flux has chicken and egg problem if git is also deployed from flux

Flux:

More git ops-y
chicken and egg problem is conquerable, in a maybe confusing way

Decision:

Infra:
kubectl apply/helm everything
Drop keycloak image manually in k3s either using cri or placing in magic place after k3s install.
Use helm with values for onedev.
Get rid of Kanboard custom image. Use kubectl apply.
Test:
1. Mostly kubectl apply for tekton.
2. Use flux for:
  1. MQTT
  2. j7s-os-deploy
  3. squid

VM Resources

Lightsail:

Leave alone

Infra Cluster:

RAM 4 GiB total
2 CPUs
120Gib Hardrive

Main Cluster:

RAM 4 GiB total
2 CPUs
120Gib Hardrive

Secrets

Options: Mozilla Kops Bitnami Sealed Secrets

Both work with Flux. Sealed Secrets seems more integrated with k8s when not using Flux.

Decision: Bitnami Sealed Secrets

Experiments

k3s with cilium and nginx on Centos Stream 9

systemctl disable firewalld --now
export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none --disable-network-policy --selinux"
curl -sfL https://get.k3s.io | sh -s -

I see an error about selinux policies conflicting, but I'm not sure if it matters?

Install cilium following instructions here: https://docs.cilium.io/en/v1.12/gettingstarted/k3s/

Install nginx with:

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.ingressClassResource.default=true

k3s with nginx on fedora server

sudo systemctl disable firewalld --now
export INSTALL_K3S_EXEC="server --disable traefik --selinux"
curl -sfL https://get.k3s.io | sh -s -
sudo chown jimmy:jimmy /etc/rancher/k3s/k3s.yaml
sudo dnf install helm
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

Import simple-ros2. Laptop:

podman save -o simple-ros2.tar simple-ros2:latest
scp simple-ros2.tar 192.168.1.106:~/.

On server:

sudo ctr images import ./simple-ros2.tar
# wait forever....

Test yaml:

---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: simple-ros2
    image: localhost/simple-ros2:latest
    imagePullPolicy: Never
    args: [ros2, launch, j7s-simple, j7s_publisher_launch.py]

VM Host set up

I think I ran something like this when I set up the VM host. I don't remember exactly, and I didn't document it...

This should be carefully looked at before running.

nmcli connection add ifname br0 type bridge con-name br0 connection.zone trusted
nmcli connection add type bridge-slave ifname enp4s0 master br0
nmcli connection modify br0 bridge.stp no
nmcli connection modify enp4s0 autoconnect no
nmcli connection down enp4s0
nmcli connection up id br0

Kubeseal Use

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
  namespace: my-namespace
type: Opaque 
data: 
  username: dmFsdWUtMQ0K 
  password: dmFsdWUtMg0KDQo=
stringData: 
  hostname: myapp.mydomain.com

cat secret.yaml | kubeseal --format yaml > sealedsecret.yaml

Actual Install Notes

To Do List

Infra Cluster: [x]

On Host:
1. CoreDNS
2. Wireguard
On Cluster:
1. Keycloak
2. Kanboard
3. Gitea
4. Harbor

Main Cluster:

On Host:
1. Wireguard
On Cluster:
1. Tekton Base install Add namespace Push images Update tasks Update jobs
2. Flux
  1. MQTT Broker
  2. Squid
  3. j7s-os-deployment

[x] Give accounts on Harbor to clusters. [ ] Push images to Harbor. [x] Hubble.

Regularly Scheduled Programming

Fedora Server 37 keep defaults.

Infra: On VM:

sudo hostnamectl set-hostname infra-cluster
sudo systemctl disable firewalld --now
sudo su
export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none --disable-network-policy --cluster-cidr 10.44.0.0/16 --service-cidr 10.45.0.0/16 --cluster-dns 10.45.0.10 --selinux"
curl -sfL https://get.k3s.io | sh -s -
exit
sudo cp /etc/rancher/k3s/k3s.yaml ~/infra.yaml
sudo chown jimmy:jimmy ~/infra.yaml
exit

on laptop

scp jimmy@192.168.1.112:~/infra.yaml /home/jimmy/.kube/.
export KUBECONFIG=~/.kube/infra.yaml
vim KUBECONFIG and fix ip.

Install cilium cli.

On laptop:

cilium install

wait...

helm upgrade --debug --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

Main: On VM:

sudo hostnamectl set-hostname j7s-cluster
sudo systemctl disable firewalld --now
sudo su
export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none --disable-network-policy --cluster-cidr 10.46.0.0/16 --service-cidr 10.47.0.0/16 --cluster-dns 10.47.0.10 --selinux --resolv-conf /run/systemd/resolve/resolv.conf"
curl -sfL https://get.k3s.io | sh -s -
exit
sudo cp /etc/rancher/k3s/k3s.yaml ~/j7s-cluster.yaml
sudo chown jimmy:jimmy ~/j7s-cluster.yaml
exit

on laptop

scp jimmy@192.168.1.103:~/j7s-cluster.yaml /home/jimmy/.kube/.
export KUBECONFIG=~/.kube/j7s-cluster.yaml
vim KUBECONFIG and fix ip.

On laptop:

cilium install

wait...

helm upgrade --debug --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

Install Sealed Secrets:

Main:

export KUBECONFIG=~/.kube/j7s-cluster.yaml
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.5/controller.yaml
kubectl apply -f controller.yaml

Infra:

export KUBECONFIG=~/.kube/infra.yaml
kubectl apply -f controller.yaml
rm controller.yaml

Install kubeseal.

Merge kube config files:

Manually modify each config file and get rid of all the defaults to something unique for that file. ( I have k3s for the original cluster, j7s for the new main cluster, and infra for the new infra cluster. )
Do some magic.

cp config.yaml config.yaml.back.<date>
export KUBECONFIG=~/.kube/config:~/.kube/infra.yaml:~/.kube/j7s-cluster.yaml
kubectl config view --flatten > new-config
mv new-confg config
export KUBECONFIG=~/.kube/config
chmod 600 ~/.kube/config

Use kubeseal to encrypt secrets for harbor.

Install harbor.

cd infra-cluster/harbor
kubectl apply -f namespace
kubectl apply -f secrets
cd helm
./install.bash

Build coredns rpm following instructions in coredns folder. scp to infra:

scp redhat/RPMS/x86_64/coredns-1.8.4-1.fc37.x86_64.rpm jimmy@192.168.1.112:~/.
ssh jimmy@192.168.1.112
sudo dnf install ./coredns-1.8.4-1.fc37.x86_64.rpm
exit

Copy over corefile from coredns folder.

scp Corefile jimmy@192.168.1.112:~/.
ssh jimmy@192.168.1.112
sudo cp Corefile /etc/coredns/Corefile
sudo systemctl start coredns
sudo systemctl enable coredns

sudo dnf install policycoreutils-devel rpm-build
sepolicy generate --application /bin/coredns
./coredns.sh
# Until it works....
sudo su
ausearch -c '(coredns)' --raw | audit2allow -M my-coredns
semodule -i my-coredns.pp
# Also:
sudo setsebool -P domain_can_mmap_files 1
# Turn of resolver.
sudo vim /etc/systemd/resolved.conf
DNSStubListener=no

Wound up turning off SELinux...

sudo vi /etc/selinux/config
# SELINUX=permissive
sudo grubby --update-kernel ALL --args selinux=0

Wound up reverting back.

Add:

CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

under [Service] in

sudo vim /usr/lib/systemd/system/coredns.service

Wireguard:

sudo dnf install wireguard-tools
wg genkey | tee wg.key | wg pubkey > wg.pub
vim wg0.conf
<<<
[Interface]
Address = 10.100.100.?/24
PrivateKey = <Contents from file.>

[Peer]
PublicKey = zgcRWY3MAwKGokyRs9dR4E5smoeFy1Hh4MfDcDM3iSc=
AllowedIPs = 10.100.100.0/24
Endpoint = vpn.jpace121.net:51902
PersistentKeepAlive = 25
<<<<

Add to server:

# Infra k3s node
[Peer]
PublicKey = <>
AllowedIPs = 10.100.100.7/32

# Add to systemd 
sudo systemctl enable wg-quick@wg0.service 
sudo systemctl daemon-reload 
sudo systemctl start wg-quick@wg0

Tried using nm below, moved to wg-quick for consistency.

nmcli con import type wireguard file /etc/wireguard/wg0.conf

Better:

sudo cp wg0.conf /etc/wireguard/wg0.conf
sudo chown root:root /etc/wireguard/wg0.conf
wg-quick up wg0

Harbor Login:

scp harbor_tls.crt jimmy@10.100.100.7:.
ssh jimmy@10.100.100.7
sudo cp harbor_tls.crt /etc/rancher/k3s/.

/etc/rancher/k3s/registries.yaml

configs:
  "harbor.internal.jpace121.net":
    auth:
      username: robot$k8s+infra-cluster
      password: <from harbor>
    tls:
      ca_file: /etc/rancher/k3s/harbor_tls.crt

Kanboard:

Get PV Name:

kubectl describe pvc kanboard-pvc --context k3s

Use PV name to locate directory:

kubectl describe pv pvc-89a4265c-b39c-4628-9e6b-df091fae4fd8 --context k3s

Can tell on k3s-node1 at /var/lib/rancher/k3s/storage/pvc-89a4265c-b39c-4628-9e6b-df091fae4fd8_default_kanboard-pvc

ssh jimmy@192.168.1.135
sudo su
cd /var/lib/rancher/k3s/storage/pvc-89a4265c-b39c-4628-9e6b-df091fae4fd8_default_kanboard-pvc
tar cvpzf /home/jimmy/kanboard-pvc.tar.gz .
exit
cd ~
sudo chown jimmy:jimmy kanboard-pvc.tar.gz
exit
scp jimmy@192.168.1.135:~/kanboard-pvc.tar.gz /tmp/kanboard-pvc.tar.gz

Apply PVC. Want: volumeBindingMode: Immediate

kubectl apply manifests --context infra
<wait til pvc exists>
<delete everyone but pvc>
kubectl describe pvc kanboard-pvc --context infra --namespace kanboard
kubectl describe pv pvc-fe710c38-52ce-495b-bb8d-bea48222a21b --namespace kanboard

scp /tmp/kanboard-pvc.tar.gz jimmy@192.168.1.112:.
ssh jimmy@192.168.1.112
sudo su
chown root:root ./kanboard-pvc.tar.gz
cd /var/lib/rancher/k3s/storage/pvc-fe710c38-52ce-495b-bb8d-bea48222a21b_kanboard_kanboard-pvc
rm -rf *
tar xpvzf /home/jimmy/kanboard-pvc.tar.gz
exit
exit
kubectl apply -f manifests/

Make secret:

cat kanboard-cookie.yaml | kubeseal --format yaml > kanboard-cookie-sealed.yaml

Where should I proxy to?

kubectl -n ingress-nginx get svc
ngress-nginx-controller             LoadBalancer   10.45.94.103    192.168.1.112   80:31566/TCP,443:32594/TCP   23d

10.100.100.7:31566

Tekton

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml

Keycloak

kubectl describe pv pvc-4bcbb023-e686-4082-855f-d062ff418c74 --namespace keycloak /var/lib/rancher/k3s/storage/pvc-4bcbb023-e686-4082-855f-d062ff418c74_keycloak_keycloak-db-pvc scp /tmp/db-backup.tar.gz jimmy@192.168.1.112:.

sudo su
chown root:root ./db-backup.tar.gz
cd /var/lib/rancher/k3s/storage/pvc-4bcbb023-e686-4082-855f-d062ff418c74_keycloak_keycloak-db-pvc
rm -rf *
tar xpvzf /home/jimmy/db-backup.tar.gz
sudo chown -R systemd-oom:systemd-oom *

15 KiB Raw Blame History

Design for Second Iteration of Cluster/Homelab

Context

Investigation

Host OS

K3S Distro

How many clusters?

Files

How many nodes per cluster?

Networking

DNS

Load Balancer

Ingress

CNI

What goes on each cluster/VM?

Deployments

VM Resources

Secrets

Experiments

k3s with cilium and nginx on Centos Stream 9

k3s with nginx on fedora server

VM Host set up

Kubeseal Use

Actual Install Notes

To Do List

Regularly Scheduled Programming

Tekton

Keycloak

15 KiB

Raw Blame History