add k3s traefik writeup
This commit is contained in:
rskntroot
238
docs/source/docs/k3s_traefik_setup.md
Normal file
238
docs/source/docs/k3s_traefik_setup.md
Normal file
@@ -0,0 +1,238 @@
|
|||||||
|
# K3S Traefik Setup
|
||||||
|
|
||||||
|
## Brief
|
||||||
|
|
||||||
|
Enabling internal access to dashboard and metrics for traefik ingress controller in k3s kubernetes cluster
|
||||||
|
|
||||||
|
- by `rskntroot` on `2024-07-01`
|
||||||
|
|
||||||
|
## Assumptions
|
||||||
|
|
||||||
|
```
|
||||||
|
$ k3s --version
|
||||||
|
k3s version v1.29.5+k3s1 (4e53a323)
|
||||||
|
go version go1.21.9
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
$ kubectl version
|
||||||
|
Client Version: v1.29.5+k3s1
|
||||||
|
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
|
||||||
|
Server Version: v1.29.5+k3s1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Traefik Dashboards
|
||||||
|
|
||||||
|
*Note: `Traefik Dashboards` refers to both traefik dashboard and prometheus*
|
||||||
|
|
||||||
|
### Preparation
|
||||||
|
|
||||||
|
Enable `internal`+`.your.domain.com` in non-public DNS
|
||||||
|
- (alt) edit the `hosts` file on your admin to point the desired k3s host IP
|
||||||
|
|
||||||
|
On host with `kubectl` access:
|
||||||
|
```
|
||||||
|
export DOMAIN=your.domain.com
|
||||||
|
```
|
||||||
|
|
||||||
|
*Note: This example does not currently include authentication and exposing these dashboards is a security risk.*
|
||||||
|
|
||||||
|
### Update Manifest
|
||||||
|
|
||||||
|
Add the following to `spec.valuesContent` section of `/var/lib/rancher/k3s/server/manifests/traefik.yaml`
|
||||||
|
```
|
||||||
|
dashboard:
|
||||||
|
enabled: true
|
||||||
|
metrics:
|
||||||
|
prometheus: true
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
spec:
|
||||||
|
chart: https://%{KUBERNETES_API}%/static/charts/traefik-25.0.3+up25.0.0.tgz
|
||||||
|
set:
|
||||||
|
global.systemDefaultRegistry: ""
|
||||||
|
valuesContent: |-
|
||||||
|
deployment:
|
||||||
|
podAnnotations:
|
||||||
|
prometheus.io/port: "8082"
|
||||||
|
prometheus.io/scrape: "true"
|
||||||
|
dashboard:
|
||||||
|
enabled: true
|
||||||
|
metrics:
|
||||||
|
prometheus: true
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restart Ingress Controller
|
||||||
|
|
||||||
|
```
|
||||||
|
kubectl -n kube-system scale deployment traefik --replicas=0
|
||||||
|
kubectl -n kube-system get deployment traefik
|
||||||
|
```
|
||||||
|
```
|
||||||
|
kubectl -n kube-system scale deployment traefik --replicas=1
|
||||||
|
```
|
||||||
|
|
||||||
|
Example: (`kubectls`?! see `#shortcuts`)
|
||||||
|
```
|
||||||
|
$ kubectls scale deployment traefik --replicas=0
|
||||||
|
deployment.apps/traefik scaled
|
||||||
|
$ kubectls get deployment traefik
|
||||||
|
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||||
|
traefik 0/0 0 0 3d1h
|
||||||
|
$ kubectls scale deployment traefik --replicas=1
|
||||||
|
deployment.apps/traefik scaled
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create Resource Definition YAML
|
||||||
|
|
||||||
|
Save the following to `traefik-dashboard.yml` in your workspace.
|
||||||
|
```
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: traefik-dashboard
|
||||||
|
namespace: kube-system
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/instance: traefik
|
||||||
|
app.kubernetes.io/name: traefik-dashboard
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- name: traefik
|
||||||
|
port: 9000
|
||||||
|
targetPort: 9000
|
||||||
|
protocol: TCP
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/instance: traefik-kube-system
|
||||||
|
app.kubernetes.io/name: traefik
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: traefik-metrics
|
||||||
|
namespace: kube-system
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/instance: traefik
|
||||||
|
app.kubernetes.io/name: traefik-metrics
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- name: traefik
|
||||||
|
port: 9100
|
||||||
|
targetPort: 9100
|
||||||
|
protocol: TCP
|
||||||
|
selector:
|
||||||
|
app.kubernetes.io/instance: traefik-kube-system
|
||||||
|
app.kubernetes.io/name: traefik
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: Ingress
|
||||||
|
metadata:
|
||||||
|
name: traefik-ingress
|
||||||
|
namespace: kube-system
|
||||||
|
annotations:
|
||||||
|
spec.ingressClassName: traefik
|
||||||
|
spec:
|
||||||
|
rules:
|
||||||
|
- host: internal.${DOMAIN}
|
||||||
|
http:
|
||||||
|
paths:
|
||||||
|
- path: /
|
||||||
|
pathType: Prefix
|
||||||
|
backend:
|
||||||
|
service:
|
||||||
|
name: traefik-dashboard
|
||||||
|
port:
|
||||||
|
number: 9000
|
||||||
|
- path: /metrics
|
||||||
|
pathType: Prefix
|
||||||
|
backend:
|
||||||
|
service:
|
||||||
|
name: traefik-metrics
|
||||||
|
port:
|
||||||
|
number: 9100
|
||||||
|
```
|
||||||
|
|
||||||
|
If you don't want `traefik-metrics` simply do not include the service definition and remove the `- path: metrics` section from the ingress definition.
|
||||||
|
|
||||||
|
### Create Service & Ingress Resources
|
||||||
|
|
||||||
|
[envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html) - `envsusbst` enables code-reuse by providing environment variable substituion as demonstrated below.
|
||||||
|
|
||||||
|
```
|
||||||
|
envsubst < traefik-dashboard.yml | kubectl apply -f -
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
$ envsubst < traefik-dashboards.yml | kubectl apply -f -
|
||||||
|
service/traefik-dashboard created
|
||||||
|
service/traefik-metrics created
|
||||||
|
ingress.networking.k8s.io/traefik-ingress created
|
||||||
|
$ kubectls get svc | grep traefik-
|
||||||
|
traefik-dashboard ClusterIP 10.43.157.54 <none> 9000/TCP 25s
|
||||||
|
traefik-metrics ClusterIP 10.43.189.128 <none> 9100/TCP 25s
|
||||||
|
```
|
||||||
|
|
||||||
|
### Access Dashboards
|
||||||
|
|
||||||
|
That's it. You should now be able to access the Traefik Ingress Controller Dashboard and metrics remotely.
|
||||||
|
|
||||||
|
Don't forget to include the appropriate uri paths:
|
||||||
|
- `internal.your.domain.com/dashboard/` for `Traefik Dashboard` *(Note: just `/dashboard` will not work.)*
|
||||||
|
- `internal.your.domain.com/metrics` for `Traefik Metrics`
|
||||||
|
|
||||||
|
### Disable Dashboards
|
||||||
|
|
||||||
|
```
|
||||||
|
envsubst < traefik-dashboard.yml | kubectl delete -f -
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
envsubst < traefik-dashboards.yml | kubectl delete -f -
|
||||||
|
service "traefik-dashboard" deleted
|
||||||
|
service "traefik-metrics" deleted
|
||||||
|
ingress.networking.k8s.io "traefik-ingress" deleted
|
||||||
|
```
|
||||||
|
|
||||||
|
## Shortcuts
|
||||||
|
|
||||||
|
### alias kubectls
|
||||||
|
|
||||||
|
*Note: kubectl-completion will not work for `kubectls`*
|
||||||
|
|
||||||
|
```
|
||||||
|
echo 'alias kubectls="kubectl -n kube-system"' >> ~/.bashrc
|
||||||
|
source ~/.bashrc
|
||||||
|
```
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```
|
||||||
|
$ echo 'alias kubectls="kubectl -n kube-system"' >> ~/.bashrc
|
||||||
|
$ source ~/.bashrc
|
||||||
|
$ kubectls get deployments
|
||||||
|
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||||
|
coredns 1/1 1 1 3d2h
|
||||||
|
local-path-provisioner 1/1 1 1 3d2h
|
||||||
|
metrics-server 1/1 1 1 3d2h
|
||||||
|
traefik 1/1 1 1 3d2h
|
||||||
|
```
|
||||||
|
|
||||||
|
(alt) use `skubectl` or `kubesctl` as the alias instead. comparisons:
|
||||||
|
- `skubectl` means you can hit `[up-arrow]` `[ctrl]+[a]` `[s]` `[enter]` when you inevitably forget to include `-n kube-system`
|
||||||
|
- `kubectls` just adds `[alt]+[right-arrow]` to the above
|
||||||
|
- `kubesctl` makes sense because all of these are really kube-system-ctl, but that adds 4x `[right-arrow]`, ewww.
|
||||||
|
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [https://docs.k3s.io](https://docs.k3s.io)
|
||||||
|
- [https://k3s.rocks/traefik-dashboard/](https://k3s.rocks/traefik-dashboard/)
|
||||||
|
- [https://doc.traefik.io/traefik/v2.10/](https://doc.traefik.io/traefik/v2.10/)
|
||||||
@@ -12,4 +12,5 @@ Rskio
|
|||||||
|
|
||||||
./docs/system_on_chip.md
|
./docs/system_on_chip.md
|
||||||
./docs/network_attached_storage.md
|
./docs/network_attached_storage.md
|
||||||
|
./docs/k3s_traefik_setup.md
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user