Skip to main content

Kubernetes

In Kubernetes, DeaconGuard has three parts. You need the server, and one or both of the others:

PartWhat it checksRuns asTokenOn Hosts
The serverNothing itself: the dashboard and the agent APIA StatefulSet with one pod and a volume for its database, under the restricted Pod Security Standard
The node agent (agent.enabled)Each node, as a machine: its packages, libraries, configuration, hardening, malware, and the containers of its podsA DaemonSet, one pod per node, privileged (it reads the node's filesystem)Reusable, one for every nodeOne host per node
The cluster agent (cluster.enabled)Kubernetes itself, through its API: workloads, RBAC, namespaces, network policies, the version and its CVEs; and the server checks the images the pods run (1.20.0 and later). See Kubernetes clustersOne unprivileged pod with a read-only role that can't read secrets, under the restricted standardAny token; it is used onceOne host, of type Kubernetes cluster

The node agent reads each node's root filesystem, read-only, and runs the same fixed, read-only checks there with the node's own tools, so its results match an agent installed on the node. The cluster agent never looks at a node or inside a container: it reads what the API shows.

You can run any of them alone: just the node agent or the cluster agent, with the server elsewhere, or just the server, with agents on machines outside the cluster. Who checks what in your pods sums up which part finds what.

With Helm​

The chart is published as an OCI artifact, signed like the image:

oci://ghcr.io/cloudopsshell/charts/deaconguard

1. The server​

kubectl create namespace deaconguard --save-config
kubectl label namespace deaconguard pod-security.kubernetes.io/enforce=restricted
kubectl -n deaconguard create secret generic deaconguard-admin --from-literal=password="$(openssl rand -base64 18)"
helm install deaconguard oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$(curl -fsSL https://get.deaconguard.io/stable.txt)" \
-n deaconguard --set server.admin.existingSecret=deaconguard-admin

The server creates the account admin with that password on its first start, and ignores the secret once any account exists. Read the password with kubectl -n deaconguard get secret deaconguard-admin -o jsonpath='{.data.password}' | base64 -d, and change it under My account after signing in.

If you install the chart before creating the secret, the server waits for it rather than failing: its page and its log say which secret to create, and it starts within a minute or so of the secret appearing. If it hasn't after two minutes, delete the pod (kubectl -n deaconguard delete pod deaconguard-0) so it mounts the secret.

Open the dashboard:

kubectl -n deaconguard port-forward svc/deaconguard 8443:8443

and go to https://127.0.0.1:8443. The certificate is self-signed until you give it one.

2. One token for the agents​

Every agent the chart runs enrolls with the same token: the node agent on each node, and the cluster agent. On the Agents page, choose Enroll a machine, then Kubernetes (DeaconGuard 1.20.2 and later). It makes the right token for this:

  • Reusable, since several agents use it: one per node, plus the cluster agent. A one-time token enrolls only the first agent to start; the others are refused with "already used".

  • For the server's address inside the cluster, which pods reach, filled in from the chart:

    https://deaconguard.deaconguard.svc:8444

    Not the dashboard's public name, which pods often can't reach on port 8444 (the agent load balancer admits only its loadBalancerSourceRanges), and not https://127.0.0.1:8444, which in a pod is the pod itself.

The dialog then shows the command for the secret and the values that use it. Or create the token from the command line:

kubectl -n deaconguard exec deaconguard-0 -- deaconguard token create \
--server-url https://deaconguard.deaconguard.svc:8444 --reusable --days 90 --name production --quiet

Each agent uses the token once, to enroll, and keeps its own identity afterwards: the token can expire, or be revoked, without stopping them. New nodes need a token that is still valid.

3. The agents​

Put the token in a secret, in the namespace of the release that runs the agents, replacing deaconguard1.… with the whole token the dialog shows, and install the chart there with both agents:

kubectl create namespace deaconguard-agent --save-config
kubectl label namespace deaconguard-agent pod-security.kubernetes.io/enforce=privileged
kubectl -n deaconguard-agent create secret generic deaconguard-enrollment --from-literal=token='deaconguard1.…'
helm install deaconguard-agent oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$(curl -fsSL https://get.deaconguard.io/stable.txt)" -n deaconguard-agent \
--set server.enabled=false --set enrollment.existingSecret=deaconguard-enrollment \
--set agent.enabled=true --set cluster.enabled=true --set cluster.name=production

enrollment.existingSecret gives the token to every agent of the release. Leave out agent.enabled or cluster.enabled to run only the other.

  • The node agent (agent.enabled): each node appears on the Agents page within a minute, named after the node, and the token's row counts them. Scan or audit them like any other host.
  • The cluster agent (cluster.enabled): see the cluster itself.

helm install prints which secret each agent reads, and whether a token is still missing.

4. The cluster itself​

The node agents check the nodes; the cluster agent checks the cluster (DeaconGuard 1.5.0 and later): its workloads, RBAC, namespaces, Kubernetes version, and the images its pods run. The cluster appears on Hosts as production (cluster.name), a Kubernetes cluster, and runs the Kubernetes check only: from Scan now, or on a schedule with the Kubernetes check ticked. It keeps its identity on a small volume (16 MiB), so the cluster needs a default storage class, or cluster.persistence.storageClass.

It is unprivileged, so it can run in the server's release instead of the node agents'. Then put the secret in the server's namespace and turn it on there:

kubectl -n deaconguard create secret generic deaconguard-enrollment --from-literal=token='deaconguard1.…'
helm upgrade deaconguard oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$(curl -fsSL https://get.deaconguard.io/stable.txt)" -n deaconguard --reset-then-reuse-values \
--set enrollment.existingSecret=deaconguard-enrollment --set cluster.enabled=true --set cluster.name=production

The upgrade leaves the server's pod as it is.

With a values file​

The same settings in values files, for helm install or helm upgrade with -f. The server's release, under the restricted Pod Security Standard:

# values.yaml: release deaconguard, namespace deaconguard
server:
admin:
existingSecret: deaconguard-admin

The agents' release, in a namespace labelled privileged, as in step 3:

# agent-values.yaml: release deaconguard-agent, namespace deaconguard-agent
server:
enabled: false
enrollment:
existingSecret: deaconguard-enrollment # one reusable token for every agent below
agent:
enabled: true # every node, and its pods' containers
cluster:
enabled: true # the cluster itself, and its images
name: production # its name on Hosts
helm upgrade --install deaconguard oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$(curl -fsSL https://get.deaconguard.io/stable.txt)" -n deaconguard -f values.yaml
helm upgrade --install deaconguard-agent oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$(curl -fsSL https://get.deaconguard.io/stable.txt)" -n deaconguard-agent -f agent-values.yaml

Everything in one release. On a cluster whose namespaces don't enforce a Pod Security Standard (k3s doesn't by default), one release can run the server and both agents: put enrollment, agent and cluster from agent-values.yaml into values.yaml, without server.enabled: false, and create the secret in deaconguard. The server's namespace then also runs the node agent's privileged pods, so whoever may manage pods in it can read every node; keep separate releases when others administer that namespace.

With a values file, Helm uses the chart's defaults plus the file on each upgrade, so keep every setting you use in it.

One token or two. agent.enrollment and cluster.enrollment take the same settings as enrollment, to give one agent a token of its own, such as a one-time token for the cluster agent, or to revoke the nodes' token without touching the cluster agent. Left empty, as they are by default, both use enrollment. Releases from before 1.20.2 that set them keep working unchanged.

Who checks what in your pods​

In your podsChecked byHow
Vulnerable packages and libraries in running containersThe node agent, with the Running containers checkIt lists each node's containers and reads their files, read-only. See Running containers
Vulnerable images, the builds the pods runThe cluster agent, with the serverThe server takes each build from the node agents' Running containers scans, without downloading it, and reads only the others from their registries (1.20.1 and later). See Running images
Risky configuration: privileged containers, running as root, host folders, secrets in environment variables; and RBAC, namespaces, network policiesThe cluster agentIt reads the objects through the API. See Kubernetes clusters
Malware: webshells, miners, dropped toolsThe node agent, with the malware and antivirus checksIn the node's own folders, and what each running container wrote since it started (1.19.0 and later). See Malware in containers
What processes do as they runNothingDeaconGuard checks at scan time; it doesn't watch processes as they run

Values​

ValueDefaultUse
image.repository, image.tagghcr.io/cloudopsshell/deaconguard, the chart's versionAnother registry, such as a mirror
imagePullSecrets[]For a private mirror
server.enabledtrueInstall the server
server.admin.usernameadminThe first account
server.admin.existingSecret, server.admin.secretKey"", passwordThe secret with its password
server.admin.password""Or the password itself, which the chart puts in a secret (it is then also in Helm's release data)
server.trustedProxies[]Networks of the proxies in front of the dashboard, such as the ingress controller's pods; see Behind a reverse proxy
server.accessLogfalseA line in kubectl logs for each request: method, path, status, duration, account and client address. Agents' and the dashboard's routine polls, and the chart's own health checks, are left out.
server.clusterImageRegistrytrueRead the images clusters run from their registries when no node agent read them; false checks only what node agents read (1.20.1 and later). See Running images
server.updateChecktrueOnce a day, read the newest release's number from get.deaconguard.io, to tell admins when the server is behind (1.34.0 and later). Nothing about the server is sent. false stops it, whatever Settings → New versions says
server.feedCacheMB400Memory, in MB, for advisory feeds kept parsed between scans, so that scans of the same release don't read them again; 0 reads them for every scan. Keep it well under server.resources.limits.memory. See Parsed feeds.
server.tls.existingSecret""A kubernetes.io/tls secret to serve instead of the self-signed certificate
server.persistence.size, .storageClass, .existingClaim5Gi, the default class, noneThe database's volume
server.service.type, .annotations, .dashboardPort, .agentPortClusterIP, none, 8443, 8444The Service
server.agentService.enabled, .type, .annotations, .loadBalancerSourceRanges, .portoff, LoadBalancer, none, none, 8444A Service with only the agent port, for agents outside the cluster
server.agentService.externalTrafficPolicy, .loadBalancerClass, .nodePortthe cluster's defaultsLocal keeps each agent's own address in the server's logs; the load balancer implementation to use, when there are several; a fixed node port (30000–32767) instead of a random one
server.ingress.enabled, .className, .annotations, .hosts, .tlsoffAn Ingress for the dashboard
server.ingress.traefik.enabled, .allowSourceRanges, .middlewaresoff, none, noneWith Traefik (chart 1.11.0 and later): the ServersTransport and Service annotations Traefik needs to reach the dashboard, an IP allow list, and other middlewares; see With Traefik
server.agentRoute.enabled, .host, .entryPointsoff, none, [websecure]Agents outside the cluster through Traefik on port 443, with TLS passed through (chart 1.11.0 and later); see With Traefik
server.resources100m CPU, 256 MiB requested; 1 GiB limitThe server needs about 450 MB while it checks an Ubuntu machine's packages
agent.enabledfalseInstall the agent DaemonSet
enrollment.existingSecret, .secretKey"", tokenThe secret with the reusable token every agent of the release uses: the node agents and the cluster agent (1.20.2 and later)
enrollment.token""Or the token itself, which the chart puts in a secret (it is then also in Helm's release data)
agent.enrollment.existingSecret, .secretKey, .token"", token, ""The node agents' own token instead of enrollment's; must be reusable
agent.identityPath/var/lib/deaconguard-agentWhere each node keeps its agent's identity
agent.resources50m CPU, 128 MiB requested; 2.5 GiB limitClamAV, when the node has it, runs inside this limit and needs about 1.5 GB
agent.tolerationsevery taintWhich nodes get an agent; with agent.nodeSelector and agent.affinity
agent.priorityClassName""
cluster.enabledfalseInstall the cluster agent, which checks the cluster through its API
cluster.namekubernetesThe cluster's name on the server
cluster.enrollment.existingSecret, .secretKey, .token"", token, ""The cluster agent's own token instead of enrollment's; a one-time token is enough for it alone
cluster.persistence.size, .storageClass16Mi, the default classThe volume that keeps its identity
cluster.resources20m CPU, 64 MiB requested; 256 MiB limit
cluster.admission.enabled, .severityfalse, mediumWarnings as workloads are applied: a validating webhook in warn mode, with a certificate the chart makes
server.nodeSelector, .tolerations, .affinity, .podAnnotations, and the same for agent and clusterScheduling and annotations

With Flux, Argo CD or a pipeline​

The chart works the same from GitOps tools and CI/CD pipelines. Keep the values in Git, and keep secrets out of it:

  • The admin password and the agents' enrollment token go in Kubernetes secrets that your secret tool creates, such as Sealed Secrets, SOPS or External Secrets. The values only name them (server.admin.existingSecret, enrollment.existingSecret). If the chart is applied before the secret exists, the server waits for it and creates the account once the secret is mounted.
  • To upgrade, change the chart's version in Git, after reading the upgrade notes. These tools render the chart from the new version's defaults plus your values every time, so there is no --reuse-values to think about.

The server's values, as on this page:

server:
admin:
existingSecret: deaconguard-admin

Flux​

Flux takes the chart from GHCR with an OCIRepository, and installs it with a HelmRelease:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: deaconguard
namespace: deaconguard
spec:
interval: 1h
url: oci://ghcr.io/cloudopsshell/charts/deaconguard
ref:
semver: "1.35.x" # patch releases come by themselves; or tag: "1.35.0"
layerSelector:
mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
operation: copy
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: deaconguard
namespace: deaconguard
spec:
interval: 10m
chartRef:
kind: OCIRepository
name: deaconguard
values:
server:
admin:
existingSecret: deaconguard-admin

To move to a new minor release, change semver (or tag) in Git. The agent DaemonSet is a second OCIRepository and HelmRelease in the agent's namespace, with the values from the agents (server.enabled: false, enrollment.existingSecret, agent.enabled: true, and cluster.enabled: true for the cluster agent).

Argo CD​

Argo CD needs GHCR declared as a Helm repository with OCI, once, then an Application:

apiVersion: v1
kind: Secret
metadata:
name: deaconguard-charts
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
name: deaconguard
type: helm
url: ghcr.io/cloudopsshell/charts
enableOCI: "true"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: deaconguard
namespace: argocd
spec:
project: default
source:
repoURL: ghcr.io/cloudopsshell/charts
chart: deaconguard
targetRevision: "1.35.*" # patch releases come by themselves; or "1.35.0"
helm:
valuesObject:
server:
admin:
existingSecret: deaconguard-admin
destination:
server: https://kubernetes.default.svc
namespace: deaconguard
syncPolicy:
automated:
prune: true
selfHeal: true

The chart is public, so the repository needs no credentials. Argo CD renders the chart with helm template and applies it, so helm list doesn't show the release; Argo CD does. The agent DaemonSet is a second Application with the agents' values.

A pipeline​

A CI/CD job (GitHub Actions, GitLab CI, Jenkins…) keeps the values in a file in its repository, and runs one command for the first install and every upgrade:

helm upgrade --install deaconguard oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$VERSION" -n deaconguard --create-namespace -f deaconguard-values.yaml --wait

-f applies your file over the new chart's defaults each time, so the job needs neither --reuse-values nor --reset-then-reuse-values. Pin VERSION in the job, or read it from https://get.deaconguard.io/stable.txt.

With Kustomize​

The same server and agent are in deploy/kustomize, with the namespaces and their Pod Security labels:

kubectl create namespace deaconguard --save-config
kubectl -n deaconguard create secret generic deaconguard-admin --from-literal=password='a-long-password'
kubectl apply -k "https://github.com/Cloudopsshell/deaconguard/deploy/kustomize/server?ref=v$(curl -fsSL https://get.deaconguard.io/stable.txt)"

# after creating a reusable token for https://deaconguard.deaconguard.svc:8444
kubectl create namespace deaconguard-agent --save-config
kubectl -n deaconguard-agent create secret generic deaconguard-enrollment --from-literal=token='deaconguard1.…'
kubectl apply -k "https://github.com/Cloudopsshell/deaconguard/deploy/kustomize/agent?ref=v$(curl -fsSL https://get.deaconguard.io/stable.txt)"

To change them, use the directories as the base of your own overlay, with patches for the storage class, resources or node selection. examples/ingress-nginx.yaml is an Ingress for ingress-nginx and cert-manager.

How the agent reads its node​

The DaemonSet's pods:

  • mount the node's / read-only at /host, and its /tmp writable at /host/tmp for the advanced malware scan's rules;
  • set DEACONGUARD_HOST_ROOT=/host, so every check runs in the node's root with the node's own tools. A check whose tool the node lacks reports that, as it would on the node itself;
  • use the node's process list (hostPID) and network (hostNetwork, with ClusterFirstWithHostNet to resolve the Service);
  • run as root, with every capability dropped except those the checks need: SYS_CHROOT, DAC_READ_SEARCH, DAC_OVERRIDE, NET_ADMIN, AUDIT_CONTROL and SYS_PTRACE, and BPF and PERFMON to load live detection's watchers into the node's kernel (1.27.0 and later); no privilege escalation, a read-only root filesystem, the runtime's default seccomp profile, and no service account token;
  • report the node's name, from DEACONGUARD_HOSTNAME, and keep their identity in /var/lib/deaconguard-agent on the node, so a restarted or upgraded pod is the same host on the server.

A node's first pod enrolls with the token. If the server is replaced, say reinstalled with a new certificate, update the secret with a token from the new server and restart the DaemonSet: each pod enrolls with it. A pod also enrolls again when the server no longer knows it, so removing a node's host on the server while its pod runs brings it back; to stop scanning a node, keep the DaemonSet off it with agent.nodeSelector or agent.affinity, or revoke the token first.

Don't run the DaemonSet on nodes that already have the agent installed as a package: each would enroll as its own host.

Agents outside the cluster​

Machines outside the cluster, such as virtual machines with the install script's agent, need a network path to the agent port. The server's Service is ClusterIP, inside the cluster only, so give agents a way in of their own. The dashboard stays inside either way:

Agent Service (any cluster)Agent route (Traefik, chart 1.11.0 and later)
Agents connect tohttps://ADDRESS:8444, a load balancer of its ownhttps://agents.example.com, port 443, next to your other sites
Setserver.agentServiceserver.agentRoute: see With Traefik
Who may connectloadBalancerSourceRanges, and your firewallyour firewall
The Hosts page shows each agentfrom its own address, with externalTrafficPolicy: Localfrom Traefik's address

Agents inside the cluster keep using https://deaconguard.deaconguard.svc:8444.

With the agent Service​

# agent-service.yaml
server:
agentService:
enabled: true
externalTrafficPolicy: Local # record each agent's own address
loadBalancerSourceRanges: # the agents' addresses; empty allows everyone
- 203.0.113.10/32
helm upgrade deaconguard oci://ghcr.io/cloudopsshell/charts/deaconguard \
--version "$(curl -fsSL https://get.deaconguard.io/stable.txt)" -n deaconguard --reset-then-reuse-values \
-f agent-service.yaml
kubectl -n deaconguard get svc deaconguard-agents

That adds the Service deaconguard-agents, a LoadBalancer with only port 8444. Its EXTERNAL-IP is the address for the machines' tokens, such as https://203.0.113.50:8444, or point a DNS name at it.

  • loadBalancerSourceRanges is the list of addresses allowed to connect: your agents' public addresses, or your private network's range. A machine whose address isn't on it can't enroll (context deadline exceeded). Find a machine's public address with curl -s https://api.ipify.org on it. Limit 8444 in your provider's firewall too, and keep the NodePort range (30000–32767) closed there: loadBalancerSourceRanges doesn't cover it.
  • externalTrafficPolicy: Local keeps each agent's own address. Without it, the cluster may rewrite connections to a node's address, and every host shows as from 10.42.0.1 or another cluster address. It is the agent Service's own setting, separate from your ingress controller's, which covers the dashboard. Set it with one node, MetalLB or a cloud load balancer. With k3s's ServiceLB on several nodes, every node still accepts agents, but only those that connect to the node running the server pod are recorded with their own address; the others show ServiceLB's address on their node. Point the agents' address (or its DNS name) at the server's node for their own addresses. After changing it, restart the agents that were already connected (sudo systemctl restart deaconguard-agent): a connection opened before keeps the old address.
  • Keep it private. Each provider has its own annotation for an internal load balancer, and its own firewall: Reaching the agent port has the values for your own data centre (k3s ServiceLB, MetalLB), Hetzner, AWS, Azure, Google Cloud, DigitalOcean, Akamai/Linode and Oracle Cloud, and what to do when the machines are in another data centre or only reachable over the internet.

Other ways in​

  • Through Traefik on port 443: server.agentRoute; see With Traefik. Traefik passes TLS through and connects to the server itself, so the Hosts page shows Traefik's address for those agents.
  • Through another Ingress: possible, but needs care. Agents check the server's certificate against the key in their token, or accept one their machine trusts for the server's name. So either pass the TLS connection through unchanged (ingress-nginx's nginx.ingress.kubernetes.io/ssl-passthrough: "true", with passthrough turned on in the controller), or end TLS at the Ingress with a certificate the machines trust for its name. For the latter, allow request bodies of 80 MB (an agent's results can be that large; ingress-nginx allows 1 MB by default: nginx.ingress.kubernetes.io/proxy-body-size: 80m), and set server.trustedProxies to the ingress controller's network.
  • Without a load balancer: server.agentService.type=NodePort and, from chart 1.10.0, a fixed server.agentService.nodePort, such as 30444, open the agent port on every node. Enroll with a node's address and that port. loadBalancerSourceRanges doesn't apply to a node port, so your firewall is the only limit. Fine for a test cluster; in production, prefer a load balancer.

Certificates and ingress​

The server serves HTTPS itself. By default it creates a self-signed certificate on its first start, kept on its volume; agents trust it through the fingerprint in their token, and browsers warn about it once.

  • An Ingress for the dashboard: set server.ingress.*, or start from the Kustomize example. The backend speaks HTTPS, so ingress-nginx needs nginx.ingress.kubernetes.io/backend-protocol: HTTPS. Set server.trustedProxies to the ingress controller's pod network, such as 10.42.0.0/16, and limit who can reach it, for example with nginx.ingress.kubernetes.io/whitelist-source-range.
  • Your own certificate on the server: put it in a kubernetes.io/tls secret and set server.tls.existingSecret. Agents accept the certificate whose key's fingerprint is in their token, or one their node trusts for the server's name. cert-manager gives a certificate a new key at each renewal by default, which agents enrolled before then don't know: set privateKey.rotationPolicy: Never on the Certificate, or keep the self-signed certificate on the server and put the trusted one on the Ingress.
  • The dashboard's Ingress is for people. Agents use port 8444: inside the cluster through the Service, outside through the agent load balancer or, with Traefik, the agent route.

With Traefik​

Traefik needs two things to reach the dashboard, which serves HTTPS itself: a ServersTransport and annotations on the Service. Without them it answers 404. From chart 1.11.0, server.ingress.traefik.enabled creates them, and can limit the dashboard to your addresses; server.agentRoute brings agents outside the cluster in on port 443 too, instead of a load balancer on 8444. This works with any Traefik 3 that has its CRDs (traefik.io). One step differs by how Traefik is installed, keeping client addresses; the rest is the same everywhere:

server:
trustedProxies:
- 10.42.0.0/16 # your pod network: see the table below
ingress:
enabled: true
className: traefik
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod # if you use cert-manager
traefik.ingress.kubernetes.io/router.entrypoints: websecure
hosts:
- host: deaconguard.example.com
tls:
- secretName: deaconguard-dashboard-tls
hosts: [deaconguard.example.com]
traefik:
enabled: true
allowSourceRanges: # who may open the dashboard; empty allows everyone
- 203.0.113.10/32
agentRoute: # optional: agents on https://agents.example.com
enabled: true
host: agents.example.com
entryPoints: [websecure] # your Traefik's HTTPS entry point
  • server.ingress.traefik.enabled creates a ServersTransport (<release>-tls) and points the Service at it, so Traefik connects to the dashboard over HTTPS. It skips checking the server's certificate, which is its own; the connection stays in the cluster. Don't add these annotations by hand as well.
  • allowSourceRanges adds an ipAllowList middleware (<release>-allow). Anyone else gets 403. middlewares adds others of your own, as NAMESPACE-NAME@kubernetescrd; don't name the allow list in a router.middlewares annotation yourself.
  • trustedProxies is the network of Traefik's pods, so the server believes the client address Traefik passes on.
  • agentRoute adds an IngressRouteTCP that passes TLS through unchanged, matched by the name agents connect to, so they still check the server's own key. Point a DNS name at Traefik and create tokens with https://agents.example.com, on port 443. Turn server.agentService off if you used it. With Cloudflare, keep the record DNS only: its proxy ends TLS, which the route can't pass through.

Traefik must see client addresses​

Unless Traefik sees each client's own address, every request seems to come from a node: the allow list blocks everyone (403), sign-in limits apply to everyone at once, and the audit log shows one address. How to keep the address depends on how Traefik is installed:

Your TraefikWhat to changetrustedProxiesEntry point
k3s or RKE2's ownA HelmChartConfig in kube-system10.42.0.0/16websecure
Installed from Traefik's Helm chart (on-premise, MetalLB, cloud)Your Traefik values: service.spec.externalTrafficPolicy: LocalYour pod network, such as 10.244.0.0/16 (Flannel) or 192.168.0.0/16 (Calico)websecure, or yours
Behind another load balancer (HAProxy, F5, a cloud NLB)PROXY protocol on the load balancer and TraefikYour pod networkYours

Check it: sign in, then open Audit log. Your sign-in should show your own address, not a 10.x or other cluster address.

This is about Traefik's Service, which carries the dashboard and your other sites. Agents on the agent Service (port 8444) don't pass through Traefik: that Service has its own setting, server.agentService.externalTrafficPolicy. Set both if you use both.

k3s or RKE2​

k3s installs Traefik itself; change its Service with a HelmChartConfig:

# traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
service:
spec:
externalTrafficPolicy: Local
kubectl apply -f traefik-config.yaml
kubectl -n kube-system get svc traefik -o jsonpath='{.spec.externalTrafficPolicy}{"\n"}'

k3s redeploys Traefik within a minute; the second command then prints Local. If you already have a HelmChartConfig named traefik, add externalTrafficPolicy: Local to its valuesContent instead. It applies to all your sites, which then see their visitors' addresses too.

Traefik's own Helm chart​

When you installed Traefik yourself, with its chart (oci://ghcr.io/traefik/helm/traefik), set it in your Traefik values and upgrade that release:

# your Traefik values
service:
spec:
externalTrafficPolicy: Local
helm upgrade traefik oci://ghcr.io/traefik/helm/traefik -n traefik --reuse-values -f traefik-values.yaml

With Local, only nodes that run a Traefik pod take the traffic: MetalLB announces the address from those nodes, and cloud load balancers send to those. Run Traefik on more than one node (replicas: 2, or deployment.kind: DaemonSet) so a node going down doesn't take the dashboard with it. Traefik on the host network (hostNetwork: true) sees client addresses already and needs no change.

Behind another load balancer​

A TCP load balancer in front of Traefik (HAProxy, F5, a cloud network load balancer) hides the address itself. Turn on PROXY protocol (version 2) towards Traefik on the load balancer, and tell Traefik to trust it, in your Traefik values:

ports:
websecure:
proxyProtocol:
trustedIPs:
- 10.0.0.5/32 # your load balancer's addresses

On k3s, put the same ports: block in the HelmChartConfig's valuesContent. Turn it on at both ends together: while the load balancer sends PROXY headers and Traefik doesn't expect them, every connection through it fails. Traefik reads the header only from the addresses in trustedIPs; connections from elsewhere work as before. On AWS, the load balancer controller turns PROXY protocol on with service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" on Traefik's Service.

Troubleshooting​

You seeCauseFix
404 on the dashboardserver.ingress.traefik.enabled is off, or a router.middlewares annotation names a middleware that doesn't existTurn it on; remove hand-written Traefik annotations from server.service and server.ingress
403 on the dashboard, for everyone, you includedTraefik doesn't see client addressesKeep client addresses
The Audit log shows 10.x addresses for sign-insThe sameThe same
403 from your address onlyIt isn't in allowSourceRangesAdd it; find it with curl -s https://api.ipify.org
An agent in the cluster fails with HTTP 401 and "already used"Several agents share a one-time token: the first to start used itCreate a reusable token with Enroll a machine > Kubernetes, replace the secret's token, and delete the waiting pods
An agent in the cluster fails with i/o timeout enrolling with the dashboard's nameThe token has the server's public address, which pods can't reach on 8444Create the token with the in-cluster address, https://deaconguard.deaconguard.svc:8444, as Enroll a machine > Kubernetes does
An agent's enrollment fails with context deadline exceededThe machine's address isn't in server.agentService.loadBalancerSourceRanges, or a firewall blocks 8444Add its public address to both. On the machine, curl -sk -o /dev/null -w '%{http_code}\n' https://ADDRESS:8444/agent/v1/job prints 401 once it gets through; run the install command again
A host shows from 10.42.0.1 (or another cluster address)The agent Service rewrites addresses, or the agent's connection is older than the changeSet server.agentService.externalTrafficPolicy: Local, then sudo systemctl restart deaconguard-agent on the machine
A host shows Traefik's address, with agentRouteExpected: Traefik passes TLS through and connects to the server itselfUse the agent Service if you need each agent's address
Agents can't connect through agentRouteA proxied (orange) Cloudflare record, or a wrong entry pointMake the record DNS only; set agentRoute.entryPoints to your HTTPS entry point

For a 404, Traefik's log names the missing object: kubectl -n kube-system logs deploy/traefik on k3s, or with your Traefik release's namespace.

Commands​

Server commands run in its pod, as its user:

kubectl -n deaconguard exec deaconguard-0 -- deaconguard user list
kubectl -n deaconguard exec -it deaconguard-0 -- deaconguard user passwd admin
kubectl -n deaconguard-agent logs daemonset/deaconguard-agent

Upgrade, back up and remove it as described in Upgrade, Back up and restore and Uninstall.