# Notes Install: 1. Set up wireguard. 2. Download k3s install script from website. 3. For master: `./k3s.sh` 4. For node: `curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -` "The value to use for K3S_TOKEN is stored at /var/lib/rancher/k3s/server/node-token" 5. Install kubectl on laptop. 6. Copy `/etc/rancher/k3s/k3s.yaml` to laptop and change localhost IP to wireguard IP. 7. `kubectl cluster-info` 8. Install tkn CLI. `https://tekton.dev/docs/cli/` I installed manually. 4. Apply dns updates and rollout restart of codedns: `kubectl rollout restart -n kube-system deployment/coredns` Install Tekton: ``` kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml ``` Set up local registry on master. (See below.) Tell k3s about it: ```sudo vim /etc/rancher/k3s/registries.yaml``` ``` configs: "192.168.1.128:8443": auth: username: k3s password: password tls: ca_file: /home/jimmy/registry/certs/domain.crt ``` Restart k3s. Apply rest of the CRDs. # SSH Secrets 1. `ssh-keygen -t ecdsa -f ./deploy_key` 2. `ssh-keyscan packages.jpace121.net > ./deploy_known_hosts` 3. `cat deploy-credentials.yaml` ``` apiVersion: v1 kind: Secret metadata: name: deploy-credentials type: Opaque data: id_ecdsa: known_hosts: ``` # Set up Tekton Dashboard: ``` kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml ``` Port forward locally: ``` kubectl port-forward -n tekton-pipelines service/tekton-dashboard 9097:9097 ``` # Local Registry I could have done a much better job of documenting this. ``` mkdir registry/ cd registry/ mkdir certs auth data cd certs/ openssl genrsa 1024 > domain.key chmod 400 domain.key vim san.cnf ``` san.cf ``` [req] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = req_ext x509_extensions = v3_req prompt = no [req_distinguished_name] countryName = US stateOrProvinceName = PA localityName = Pittsburgh organizationName = j7s k3s CA commonName = j7s k3s CA [req_ext] subjectAltName = @alt_names [v3_req] subjectAltName = @alt_names [alt_names] IP.1 = 10.100.100.5 IP.2 = 192.168.1.128 ``` ``` openssl req -new -x509 -nodes -days 36500 -key domain.key -out domain.crt -config san.cnf ls cd .. ls cd auth/ podman run --entrypoint htpasswd docker.io/library/httpd:2 -Bbn k3s password > htpasswd cd .. vim run.sh ``` run.sh ``` #!/usr/bin/env bash podman run -d \ --restart=always \ --name registry \ -v `pwd`/auth:/auth \ -v `pwd`/certs:/certs \ -v `pwd`/data:/var/lib/registry \ -e REGISTRY_AUTH=htpasswd \ -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \ -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \ -e REGISTRY_HTTP_ADDR=0.0.0.0:8443 \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \ -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \ -e REGISTRY_STORAGE_DELETE_ENABLED=true \ -p 8443:8443 \ registry:latest ``` ``` sudo firewall-cmd --permanent --add-port=8443/tcp sudo firewall-cmd --reload ``` # NFS Server: CentOS 9 Set up: ``` sudo dnf install nfs-utils vim sudo mkdir /srv/nfs sudo chown jimmy:jimmy /srv/nfs sudo chmod 777 /srv/nfs/ ``` Put into `/etc/exports`: ``` /srv/nfs 192.168.1.0/24(rw,root_squash) ``` Start everything: ``` systemctl enable --now rpcbind systemctl enable --now nfs-server firewall-cmd --permanent --add-service nfs firewall-cmd --reload systemctl restart nfs-server ``` Test on Debian: ``` sudo apt install nfs-common sudo mkdir -p /mnt/nfs sudo mount 192.168.1.149:/srv/nfs /mnt/nfs ``` On the k3s nodes: ``` sudo apt install nfs-common ``` Install to the cluster: ``` helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ helm install --namespace nfs-subdir-external-provisioner nfs-subdir-external-provisioner \ nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \ --set storageClass.onDelete=delete \ --set nfs.server=192.168.1.149 \ --set nfs.path=/srv/nfs ``` # Chains Set up: ``` kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/previous/v0.14.0/release.yaml # Apply secret from j7s-intoto. # name: signing-secrets namespace: tekton-chains data: x509.pem: base64 of pem kubectl apply -f chains-config.yaml kubectl rollout restart -n tekton-chains deployment tekton-chains-controller ``` See: ``` export TASKRUN_UID=$(tkn pr describe --namespace j7s-ci --last -o jsonpath='{.metadata.uid}') tkn pr describe --namespace j7s-ci --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-pipelinerun-$TASKRUN_UID}" > signature tkn pr describe --namespace j7s-ci --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/payload-pipelinerun-$TASKRUN_UID}" | base64 -d > payload ``` ## Longhorn Postgres did not like NFS show I'm trying Longhorn. Added Centos Node to cluster. Disabled firewalld and selinux... Label k3s-nfs for storage using longhorn: ``` kubectl label nodes k3s-nfs node.longhorn.io/create-default-disk=true ``` Install longhorn using helm and only putting storage on disk with that label: ``` helm repo add longhorn https://charts.longhorn.io helm repo update helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.createDefaultDiskLabeledNodes=true ``` On rhel nfs host: ``` sudo dnf install libiscsi iscsi-initiator-util sudo su echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi systemctl enable iscsid systemctl start iscsid ``` On all nodes: ``` sudo apt install open-iscsi ``` # Future Ideas If we later want to do this on an overlay network: 3. For master: `INSTALL_K3S_EXEC="server --node-ip '10.100.100.5' --advertise-address '10.100.100.5' --flannel-iface 'wg0'" ./k3s.sh` 4. For node: `INSTALL_K3S_EXEC="agent --server 'https://10.100.100.5:6443' --token 'K3S_TOKEN' --node-ip '10.100.100.?' --advertise-address '10.100.100.?' --flannel-iface 'wg0'" ./k3s.sh`