Dashboards¶
Every Grafana dashboard at https://grafana.tomoda.life is a hand-written JSON file in git under k8s/apps/grafana-dashboards/<folder>/. Kustomize packs each folder into one ConfigMap (dashboards-<folder>), deployed by the grafana-dashboards Argo CD Application. The chart's bundled Kubernetes dashboards are off (defaultDashboardsEnabled: false) and nothing is imported from grafana.com. Home is Grafana's home page (grafana.ini dashboards.default_home_dashboard_path: /tmp/dashboards/Overview/home.json).
Inventory¶
| Folder | Dashboard (UID) | File | Purpose and key panels |
|---|---|---|---|
| Overview | Home (tomoda-home) |
overview/home.json |
Entry point. Health tiles: prod up and dev up (always both), then data stores up, API 5xx ratio and p95 latency for the picked env, and cluster-wide failing probes, firing alerts and monthly cost. Data health tiles for the picked env (Postgres up, Redis up, last base backup, Postgres connections vs max, Redis memory vs maxmemory) and third-party usage over 30 days (DeepSeek tokens, Serper credits, Places estimated cost), each linking to its full dashboard. One dashboard list per folder (by tag). Links to the most-used panels |
| Overview | Platform overview (tomoda-platform-overview) |
overview/platform-overview.json |
Node CPU, memory, root-disk headroom with a 14-day projection, load, RAID1 disk I/O, public uplink; CPU and memory requests / limits / usage vs allocatable; CPU and memory by namespace; restarts in the last hour; crash-looping containers; firing alerts; Argo CD apps that are not Healthy + Synced; Deployments with fewer available replicas than desired |
| Overview | Workloads (tomoda-workloads) |
overview/workloads.json |
Drilldown by namespace, workload and pod variables: CPU and memory usage vs requests and limits, CPU throttling, restarts, readiness, last termination reason (OOMKilled etc.), network |
| Apps | Tomoda API (tomoda-api) |
apps/tomoda-api.json |
HTTP RED per env (rate, 5xx ratio, p50/p95/p99), top routes, p95 by route, error routes; WebSocket connections, messages, drops; Xavia OTA traffic and latency from Traefik |
| Apps | Tomoda workers (tomoda-workers) |
apps/tomoda-workers.json |
Async task rate, failure ratio (status="failure") and p95 by type; LLM requests, latency and tokens; search enrichment; location provider calls by escalation reason |
| Apps | Tomoda business (tomoda-business) |
apps/tomoda-business.json |
User counts, registrations, logins, OTPs, chat, events, friends, travel log, payments; users by country; activity hotspots; app_event log lines |
| Apps | Tomoda user debug (tomoda-user-debug) |
apps/tomoda-user-debug.json |
Loki: every backend log line, errors and 4xx/5xx for one user_id |
| Data | Postgres (tomoda-postgres) |
data/postgres.json |
CNPG collector up, connections vs max_connections, last base backup, last WAL archived, archive failures, WAL waiting; backends, transactions, cache hit ratio, rows, database size, WAL size, deadlocks, XID age; longest open transaction, lock waits, data on disk vs PVC with a 24h projection; container CPU vs request, CPU throttling, memory vs limit |
| Data | Redis (tomoda-redis) |
data/redis.json |
Memory vs maxmemory, clients, keys, evictions, OOM-rejected writes; commands/s, hit ratio, errors by type; clients over time, persistence status, AOF vs PVC; container CPU vs request, CPU throttling, memory vs limit |
| Data | NATS (tomoda-nats) |
data/nats.json |
Server-wide connections, subscriptions, slow consumers, message and byte rates; per-env (account) connections, subscriptions, messages and slow consumers |
| Platform | Traefik (tomoda-traefik) |
platform/traefik.json |
Requests, non-2xx and p95 per router; entrypoint codes; from access logs: top client IPs, top hosts, 4xx/5xx by host |
| Platform | Platform services (tomoda-platform-services) |
platform/platform-services.json |
Certificate expiry and readiness; CNPG base backup age, OTA DB backup CronJobs, failed Jobs; Loki and Tempo ingestion; blackbox probes (Photon, Loki, Tempo); ARC runner pods; platform namespace CPU and memory |
| Costs | Tomoda costs (tomoda-costs) |
costs/tomoda-costs.json |
See Cost monitoring |
Conventions¶
- One
envpicker. Home, every Apps and Data dashboard, and the Costs dashboard's Third-party usage row share the sameenvvariable:prod(default),devor All. See Filtering by env. - Datasource by variable. Panels reference
${datasource}(Prometheus) and${loki}rather than a hard-coded UID; both default to the provisioned UIDsprometheusandloki. - Stable UIDs. Each dashboard sets a fixed
uid(table above), so links survive edits. - Folder tags. Each dashboard carries the
tomodatag (the top-right Dashboards dropdown) plus its folder tag:overview,apps,data,platformorcost. Home lists dashboards by these tags, so a new dashboard shows up there once it has the tag. - Zero, not "No data". Counters that can legitimately be absent (a labelled counter only appears after its first increment) use
or vector(0)on stat panels andnoValueon time series and tables, so "no errors" reads 0 or None instead of an empty panel. Latency panels say "No calls" rather than inventing a 0. - Aggregate gauges with
max. Cron-set business gauges (tomoda_users_*,tomoda_friends_friendships_*, by-country and hotspot gauges) are exported by every backend pod, worker pods with the value and API pods with 0. Usemax, neversumoravg. - Single-node scope. No per-node breakdowns, no kube-scheduler / controller-manager / etcd / kube-proxy panels (k3s runs those in one process and the chart's scrapes are off), no replication-lag panels (one Postgres instance per env).
- Disk headroom is the root filesystem. Local-path PVs are host directories, so the kubelet reports no per-PVC usage (
kubelet_volume_stats_*is absent). The overview's root-disk panels are the shared headroom for every PV; the Postgres and Redis PVC panels compare the store's own data size (recording rulespod:cnpg_data_bytes:sum,pod:pvc_request_bytes:maxinpostgres-alerts.yaml) against the PVC request.
Home links to panels rather than embedding copies of them. Library panels live in the Grafana database and can't be provisioned from sidecar files, and the -- Dashboard -- datasource only reuses queries within one dashboard, so a link (/d/<uid>?viewPanel=panel-<id>) is the way to reuse a panel. Home's summary tiles (data health, usage) repeat their source panel's query and link to it; change both together.
The env and datasource variable blocks are repeated in each dashboard's templating. Provisioned Grafana has no include mechanism and Kustomize can't patch inside a JSON file, so a shared copy would need a dashboard generator; copy the block from an existing dashboard.
Filtering by env¶
Pick prod, dev or All at the top of Home, any Apps or Data dashboard, or Costs (where it scopes only the Third-party usage row). The Dashboards dropdown, Home's dashboard lists and Home's panel links carry the choice (includeVars, ${env:queryparam}), so it follows you between dashboards.
All is allValue: .*, so every query uses a regex matcher. The env lives on a different label per metric family:
| Family | Matcher |
|---|---|
Backend metrics (tomoda_*, llm_*, places_*, serper_*, search and location provider calls) |
env=~"$env" (set by the ServiceMonitor / PodMonitor, see Prometheus) |
| Postgres | pod=~"postgres-$env-.*" |
| Redis | job=~"redis-$env-metrics" |
| Backend logs (Loki) | namespace=~"tomoda-$env" |
| OTA traffic | router=~"websecure-platform-xavia-ota-${env}-.*" |
| NATS per env | env=~"$env" on gnatsd_accstatz_* (the nats ServiceMonitor maps account DEV/PROD to env) |
Cluster-wide panels ignore the picker and say so in their description: NATS server totals, Home's probes, alerts and monthly cost tiles. Home's "Prod up" and "Dev up" tiles always show both envs. Platform overview, Workloads, Platform services, Traefik and Costs have no picker (cluster-wide; Workloads filters by namespace).
When adding an env-specific panel, use =~ with the matcher above, never ="$env", or All matches nothing.
How loading works¶
k8s/apps/grafana-dashboards/<folder>/<name>.json
└─► kustomize configMapGenerator: ConfigMap dashboards-<folder>
(label grafana_dashboard=1, annotation grafana_folder=<Folder>)
└─► Grafana sidecar (watches all namespaces)
└─► writes /tmp/dashboards/<Folder>/<name>.json
└─► file provider (foldersFromFilesStructure) creates the folder and loads it
sidecar.dashboards.folderAnnotation: grafana_folder in k8s/envs/platform/monitoring/values.yaml turns the annotation into the folder. The grafana-dashboards Application (k8s/envs/platform/grafana-dashboards/application.yaml) builds k8s/apps/grafana-dashboards/ with Kustomize and syncs it server-side (a folder's ConfigMap can outgrow the 256 KiB client-side apply annotation). A changed dashboard loads within a sync cycle without a Grafana restart.
Adding or changing a dashboard¶
- Build or edit the dashboard in the Grafana UI.
- Share → Export → Export as JSON (leave "Export for sharing externally" off).
- Save it as
k8s/apps/grafana-dashboards/<folder>/<name>.json. Keep theuidand the${datasource}/${loki}references. A new file also goes in that folder'sfileslist inkustomization.yaml; a new folder gets its own generator entry with itsgrafana_folderannotation. - Validate:
jq -e .uid <file>andkustomize build k8s/apps/grafana-dashboards. - Open a PR. Argo CD syncs the ConfigMap after merge.
Grafana refuses to save UI edits to a provisioned dashboard (allowUiUpdates: false); export the JSON and commit it instead.
Verification¶
kubectl get cm -n monitoring -l grafana_dashboard=1
kubectl logs -n monitoring deploy/monitoring-grafana -c grafana-sc-dashboard | tail -20
If a dashboard is missing, check the grafana_dashboard: "1" label, validate the JSON, and look for provisioning errors in kubectl logs -n monitoring deploy/monitoring-grafana -c grafana | grep provisioning.
Related docs¶
- Prometheus, Loki, Tempo, Alerting: the data these dashboards query
- Backend metric definitions:
backend/internal/observability/in thetomodarepo