Skip to content

Traefik

The cluster's ingress controller. Every HTTPS request to *.tomoda.life that hits the cluster (Argo CD UI, Grafana, the tomoda app, auth callbacks) is terminated at Traefik and routed by Traefik to the right backend Service.

Installed by k8s/envs/platform/traefik/application.yaml, configured by k8s/envs/platform/traefik/values.yaml.

Chart and source

Field Value
Helm chart traefik
Repository https://traefik.github.io/charts
Version 42.0.0 (Traefik v3.7.15, pinned in values.yaml)
Destination namespace traefik-system (created by Argo CD)
Argo CD Application traefik

The Application uses Argo CD's multi-source pattern: the chart comes from the upstream Helm repo at a pinned version, and valueFiles references $values/k8s/envs/platform/traefik/values.yaml from this repo.

Why Traefik

  • Portability. Plain Ingress manifests with no provider-specific annotations.
  • Middleware. The admin-UI auth chain (sys-oauth2-proxy-errors redirects 401/403 to Google login, sys-oauth2-proxy-auth is forwardAuth to /oauth2/auth) is two traefik.io/v1alpha1 Middleware CRs attached with a single annotation.
  • One-line opt-in. A workload sets ingressClassName: traefik; with cert-manager and External-DNS it gets HTTPS and a DNS record with no console work.

Service exposure

Traefik runs as a Deployment in traefik-system and is fronted by a Service of type LoadBalancer:

# values.yaml
service:
  type: LoadBalancer

k3s ServiceLB (klipper-lb) binds that Service to the server's public IP on ports 80/443. The node IP is what External-DNS writes into Cloudflare for every Ingress host (proxied or DNS-only per Ingress; see External-DNS).

There is one entry point into the cluster, and Traefik fans it out to every backend based on Host header.

Client IP preservation

Backends rate-limit and block by client IP, so the real source address has to survive two hops: the Service in front of Traefik, and Cloudflare in front of proxied hosts.

externalTrafficPolicy: Local

# values.yaml
service:
  spec:
    externalTrafficPolicy: Local

The Service's load-balancer IP is the node's own IP, so kube-proxy handles inbound traffic for it directly. With the default Cluster policy, kube-proxy masquerades that traffic, and Traefik (and every backend behind it) sees every client as the cni0 gateway 10.42.0.1. One block on that address then hits all clients, plus kubelet probes, which also arrive from 10.42.0.1.

Local tells kube-proxy to deliver only to endpoints on the receiving node and skip the SNAT, so Traefik sees the real peer. The usual cost of Local (nodes without a Traefik pod drop traffic) does not apply on a single node. ServiceLB honours the policy: the svclb-traefik-* pods still bind host ports 80/443, but forward to the node's NodePort instead of the ClusterIP, and the Service only advertises nodes with a ready Traefik pod.

Trusted forwarded headers

Traefik strips incoming X-Forwarded-* headers from untrusted peers and writes X-Forwarded-For itself from the TCP peer. The web and websecure entrypoints trust only Cloudflare's published edge ranges:

# values.yaml
ports:
  web:
    forwardedHeaders:
      trustedIPs: &cloudflareIPs [173.245.48.0/20, ..., 2c0f:f248::/32]
  websecure:
    forwardedHeaders:
      trustedIPs: *cloudflareIPs
Host type TCP peer Traefik sees X-Forwarded-For reaching the backend
Cloudflare-proxied Cloudflare edge IP (trusted) Cloudflare's header kept, edge IP appended; left-most entry is the client
DNS-only The client itself (untrusted) Client-supplied header dropped; Traefik writes the client IP

insecure: true (trust everyone) is deliberately off; it would let any direct client spoof its address. Backends should trust X-Forwarded-For only from Traefik (pod CIDR 10.42.0.0/16) plus Cloudflare's ranges.

Refreshing the list. Cloudflare changes its ranges rarely and announces it in advance. Compare the cloudflareIPs anchor in values.yaml against:

curl -s https://www.cloudflare.com/ips-v4; echo; curl -s https://www.cloudflare.com/ips-v6

Update the anchor (both entrypoints share it) in a PR; Argo CD rolls the Traefik Deployment with the new arguments.

Ingress class

The chart is configured to register itself as the controller for ingressClassName: traefik. From values.yaml:

additionalArguments:
  - "--providers.kubernetesingress.ingressclass=traefik"
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik-system/traefik"
  - "--providers.kubernetesingress.allowExternalNameServices=true"
  - "--providers.kubernetescrd=true"

Every Ingress in this repo sets ingressClassName: traefik explicitly. Anything that doesn't is ignored by Traefik and (since there is no other controller installed) will never be reachable.

allowExternalNameServices=true lets ingress paths target ExternalName Services, used by the oauth2-proxy-redir Services in k8s/envs/platform/manifests/external-services.yaml to forward /oauth2/* paths from other namespaces (argocd, data, tomoda-dev, tomoda-prod) to the single oauth2-proxy Deployment in sys.

kubernetescrd=true enables the Traefik CRD provider (IngressRoute, Middleware, TLSOption, etc.). The oauth2-proxy Middlewares and the prod Authenticated Origin Pulls TLSOption (see Tomoda) depend on it.

Dashboard, logs, metrics

The Traefik dashboard is enabled in values.yaml:

ingressRoute:
  dashboard:
    enabled: true

It is reachable inside the cluster as a default IngressRoute: port-forward traefik in traefik-system on port 9000 and visit /dashboard/ for a real-time view of routers, services, and middlewares.

Logs are emitted as JSON (general log + access log), so the Alloy pipeline in Loki can extract host, status, method, level and entrypoint as labels. The access log keeps query strings (accessLog.fields.queryParameters.defaultMode: keep). Traefik can only keep or drop the whole query, so sensitive params are redacted by Alloy before the line reaches Loki; the list is in the stage.replace in k8s/envs/platform/loki/values-alloy.yaml and described in Loki. Request headers are dropped by default, so Authorization is not logged.

Prometheus metrics are exposed on port 8082 (entrypoint metrics), and the chart creates a ServiceMonitor in the monitoring namespace with the release: monitoring label so kube-prometheus-stack picks it up. The Traefik Grafana dashboard combines these metrics with the access-log labels; see Dashboards.

Operational notes

  • There is one Traefik replica by default; a server reboot blacks out ingress until it comes back. Bumping replicas is a values change; ServiceLB forwards to whichever node runs a replica.
  • Routing decisions are driven purely by Ingress + Middleware resources in Git. Do not kubectl edit Traefik's runtime; Argo CD will overwrite.
  • Adding a new public hostname requires three pieces: an Ingress with ingressClassName: traefik, the cert-manager.io/cluster-issuer: letsencrypt-prod annotation, and a route in External-DNS's domainFilters. The first two go in the workload's manifests; the third (tomoda.life) is already configured.