lost/rskio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
45034f48c3027e77751280f7c5b26c5347debe70
rskio/mkdocs/docs/projects/k3s_traefik_setup.md

8.9 KiB
Raw Blame History

K3S Traefik Setup

Brief

Enabling internal access to dashboard and metrics for traefik ingress controller in k3s kubernetes cluster

  • by rskntroot on 2024-07-01

Assumptions

$ k3s --version
k3s version v1.29.5+k3s1 (4e53a323)
go version go1.21.9

$ kubectl version
Client Version: v1.29.5+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.5+k3s1

Traefik Dashboards

Traefik Dashboards refers to both traefik dashboard and prometheus metrics for traefik which are readily available, however, disabled by default in K3S.

Preparation

Enable internal+.your.domain.com in non-public DNS

  • (alt) edit the hosts file on your admin to point the desired k3s host IP

On host with kubectl access:

export DOMAIN=your.domain.com

!!! warning "This example does not include authentication. Exposing these dashboards is a security risk."

Update Manifest

Add the following to spec.valuesContent in:

vim /var/lib/rancher/k3s/server/manifests/traefik.yaml

=== "Yaml"

``` yaml
    dashboard:
      enabled: true
    metrics:
      prometheus: true
```

=== "Example"

``` yaml
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-25.0.3+up25.0.0.tgz
  set:
    global.systemDefaultRegistry: ""
  valuesContent: |-
    deployment:
      podAnnotations:
        prometheus.io/port: "8082"
        prometheus.io/scrape: "true"
    dashboard:
      enabled: true
    metrics:
      prometheus: true
```

Restart Ingress Controller

=== "Bash"

``` bash
kubectl -n kube-system scale deployment traefik --replicas=0
# wait a few seconds
kubectl -n kube-system get deployment traefik
kubectl -n kube-system scale deployment traefik --replicas=1
```

=== "Example"

``` bash
$ kubectls scale deployment traefik --replicas=0
deployment.apps/traefik scaled
$ kubectls get deployment traefik
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
traefik   0/0     0            0           3d1h
$ kubectls scale deployment traefik --replicas=1
deployment.apps/traefik scaled
```

Create Resource Definition YAML

Save the following to traefik-dashboard.yml in your workspace.

=== "Traefik Dashboard"

``` yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-dashboard
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik-dashboard
spec:
  type: ClusterIP
  ports:
  - name: traefik
    port: 9000
    targetPort: 9000
    protocol: TCP
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: kube-system
  annotations:
    spec.ingressClassName: traefik
spec:
  rules:
    - host: internal.${DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: traefik-dashboard
                port:
                  number: 9000
```

=== "Promethus Only"

``` yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik-metrics
spec:
  type: ClusterIP
  ports:
  - name: traefik
    port: 9100
    targetPort: 9100
    protocol: TCP
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: kube-system
  annotations:
    spec.ingressClassName: traefik
spec:
  rules:
    - host: internal.${DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: traefik-dashboard
                port:
                  number: 9000
          - path: /metrics
            pathType: Prefix
            backend:
              service:
                name: traefik-metrics
                port:
                  number: 9100
```

=== "Both"

``` yaml
apiVersion: v1
kind: Service
metadata:
  name: traefik-dashboard
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik-dashboard
spec:
  type: ClusterIP
  ports:
  - name: traefik
    port: 9000
    targetPort: 9000
    protocol: TCP
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik

---

apiVersion: v1
kind: Service
metadata:
  name: traefik-metrics
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik-metrics
spec:
  type: ClusterIP
  ports:
  - name: traefik
    port: 9100
    targetPort: 9100
    protocol: TCP
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: kube-system
  annotations:
    spec.ingressClassName: traefik
spec:
  rules:
    - host: internal.${DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: traefik-dashboard
                port:
                  number: 9000
          - path: /metrics
            pathType: Prefix
            backend:
              service:
                name: traefik-metrics
                port:
                  number: 9100
```

Create Service & Ingress Resources

envsubst - enables code-reuse by providing environment variable substituion as demonstrated below.

=== "Bash"

``` bash
envsubst < traefik-dashboard.yml | kubectl apply -f -
```

=== "Example"

``` bash
$ envsubst < traefik-dashboards.yml | kubectl apply -f -
service/traefik-dashboard created
service/traefik-metrics created
ingress.networking.k8s.io/traefik-ingress created
$ kubectls get svc | grep traefik-
traefik-dashboard   ClusterIP      10.43.157.54    <none>    9000/TCP    25s
traefik-metrics     ClusterIP      10.43.189.128   <none>    9100/TCP    25s
```

Access Dashboards

That's it. You should now be able to access the Traefik Ingress Controller Dashboard and metrics remotely.

Don't forget to include the appropriate uri paths:

=== "Traefik Dashboard"

```
https://internal.your.domain.com/dashboard/
```

!!! tip "When navigating to the traefik dashboard the `/` at the end is necessary. `/dashboard` will not work. "

=== "Promethus Metrics"

  ```
  https://internal.your.domain.com/metrics
  ```

Disable Dashboards

=== "Bash"

``` bash
envsubst < traefik-dashboard.yml | kubectl delete -f -
```

=== "Example"

``` bash
$ envsubst < traefik-dashboards.yml | kubectl delete -f -
service "traefik-dashboard" deleted
service "traefik-metrics" deleted
ingress.networking.k8s.io "traefik-ingress" deleted
```

Shortcuts

alias kubectls

!!! tip "When using an alias to substitute kubectl command completion will not work."

=== "Bash"

``` bash
echo 'alias kubectls="kubectl -n kube-system"' >> ~/.bashrc
source ~/.bashrc
```

=== "Example"

``` bash
$ echo 'alias kubectls="kubectl -n kube-system"' >> ~/.bashrc
$ source ~/.bashrc
$ kubectls get deployments
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
coredns                  1/1     1            1           3d2h
local-path-provisioner   1/1     1            1           3d2h
metrics-server           1/1     1            1           3d2h
traefik                  1/1     1            1           3d2h
```

Alternatives

  • skubectl means you can hit [up-arrow] [ctrl]+[a] [s] [enter] when you inevitably forget to include -n kube-system
  • kubectls just adds [alt]+[right-arrow] into the above before [s]
  • kubesctl makes sense because all of these are really kube-system-ctl, but that adds 4x [right-arrow], ewww.

References

Reference in New Issue View Git Blame Copy Permalink