k8s-config/notes.md

Notes

Install:

1. Set up wireguard.
2. Download k3s install script from website.
3. For master: ./k3s.sh
4. For node: curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh - "The value to use for K3S_TOKEN is stored at /var/lib/rancher/k3s/server/node-token"
5. Install kubectl on laptop.
6. Copy /etc/rancher/k3s/k3s.yaml to laptop and change localhost IP to wireguard IP.
7. kubectl cluster-info
8. Install tkn CLI. https://tekton.dev/docs/cli/ I installed manually.
9. Apply dns updates and rollout restart of codedns: kubectl rollout restart -n kube-system deployment/coredns

Install Tekton:

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml

Set up local registry on master. (See below.)

Tell k3s about it: sudo vim /etc/rancher/k3s/registries.yaml

configs:
  "192.168.1.128:8443":
    auth:
      username: k3s
      password: password
    tls:
      ca_file: /home/jimmy/registry/certs/domain.crt

Restart k3s.

Apply rest of the CRDs.

SSH Secrets

1. ssh-keygen -t ecdsa -f ./deploy_key
2. ssh-keyscan packages.jpace121.net > ./deploy_known_hosts
3. cat deploy-credentials.yaml

    apiVersion: v1
    kind: Secret
    metadata:
        name: deploy-credentials
    type: Opaque
    data:
        id_ecdsa: <base64 -w 0 .. >
        known_hosts: <base64 -w 0 ..>
   

# Set up Tekton Dashboard:

kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml

Port forward locally:

kubectl port-forward -n tekton-pipelines service/tekton-dashboard 9097:9097

# Local Registry
I could have done a much better job of documenting this.

mkdir registry/ cd registry/ mkdir certs auth data cd certs/ openssl genrsa 1024 > domain.key chmod 400 domain.key vim san.cnf

san.cf

[req]
default_bits = 2048
distinguished_name = req_distinguished_name req_extensions = req_ext x509_extensions = v3_req prompt = no

[req_distinguished_name] countryName = US stateOrProvinceName = PA localityName = Pittsburgh organizationName = j7s k3s CA commonName = j7s k3s CA

[req_ext] subjectAltName = @alt_names

[v3_req] subjectAltName = @alt_names

[alt_names] IP.1 = 10.100.100.5 IP.2 = 192.168.1.128

openssl req -new -x509 -nodes -days 36500 -key domain.key -out domain.crt -config san.cnf ls cd .. ls cd auth/ podman run --entrypoint htpasswd docker.io/library/httpd:2 -Bbn k3s password > htpasswd cd .. vim run.sh

run.sh

#!/usr/bin/env bash podman run -d
--restart=always
--name registry
-v pwd/auth:/auth
-v pwd/certs:/certs
-v pwd/data:/var/lib/registry
-e REGISTRY_AUTH=htpasswd
-e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm"
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd
-e REGISTRY_HTTP_ADDR=0.0.0.0:8443
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key
-e REGISTRY_STORAGE_DELETE_ENABLED=true
-p 8443:8443
registry:latest

sudo firewall-cmd --permanent --add-port=8443/tcp sudo firewall-cmd --reload

# NFS

Server: CentOS 9
Set up:

sudo dnf install nfs-utils vim sudo mkdir /srv/nfs sudo chown jimmy:jimmy /srv/nfs sudo chmod 777 /srv/nfs/

Put into `/etc/exports`:

/srv/nfs 192.168.1.0/24(rw,root_squash)

Start everything:

systemctl enable --now rpcbind systemctl enable --now nfs-server firewall-cmd --permanent --add-service nfs firewall-cmd --reload systemctl restart nfs-server

Test on Debian:

sudo apt install nfs-common sudo mkdir -p /mnt/nfs sudo mount 192.168.1.149:/srv/nfs /mnt/nfs

On the k3s nodes:

sudo apt install nfs-common

Install to the cluster:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ helm install --namespace nfs-subdir-external-provisioner nfs-subdir-external-provisioner
nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
--set storageClass.onDelete=delete
--set nfs.server=192.168.1.149
--set nfs.path=/srv/nfs

# Chains
Set up:

kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/previous/v0.14.0/release.yaml

Apply secret from j7s-intoto.

name: signing-secrets namespace: tekton-chains data: x509.pem: base64 of pem

kubectl apply -f chains-config.yaml kubectl rollout restart -n tekton-chains deployment tekton-chains-controller

See:

export TASKRUN_UID=$(tkn pr describe --namespace j7s-ci --last -o jsonpath='{.metadata.uid}') tkn pr describe --namespace j7s-ci --last -o jsonpath="{.metadata.annotations.chains.tekton.dev/signature-pipelinerun-$TASKRUN_UID}" > signature tkn pr describe --namespace j7s-ci --last -o jsonpath="{.metadata.annotations.chains.tekton.dev/payload-pipelinerun-$TASKRUN_UID}" | base64 -d > payload

## Longhorn

Postgres did not like NFS show I'm trying Longhorn.
Added Centos Node to cluster. Disabled firewalld and selinux...

Label k3s-nfs for storage using longhorn:

kubectl label nodes k3s-nfs node.longhorn.io/create-default-disk=true

Install longhorn using helm and only putting storage on disk with that
label:

helm repo add longhorn https://charts.longhorn.io helm repo update helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.createDefaultDiskLabeledNodes=true

On rhel nfs host:

sudo dnf install libiscsi iscsi-initiator-util sudo su echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi systemctl enable iscsid systemctl start iscsid

On all nodes:

sudo apt install open-iscsi

# Future Ideas

If we later want to do this on an overlay network:
3. For master:
   `INSTALL_K3S_EXEC="server --node-ip '10.100.100.5' --advertise-address '10.100.100.5' --flannel-iface 'wg0'" ./k3s.sh`
4. For node:
   `INSTALL_K3S_EXEC="agent --server 'https://10.100.100.5:6443' --token 'K3S_TOKEN' --node-ip '10.100.100.?' --advertise-address '10.100.100.?' --flannel-iface 'wg0'" ./k3s.sh`