Debugging¶
A field guide for common production issues. Start with kubectl describe, escalate to logs, then dashboards.
First moves¶
# What's broken?
kubectl get pods -A | grep -v Running
# Recent events across the namespace
kubectl get events -n tomoda-prod --sort-by='.lastTimestamp' | tail -30
# Argo CD app health
argocd app list
argocd app get tomoda-prod
Pod-level diagnosis¶
describe first¶
kubectl describe pod <pod-name> -n tomoda-prod
Look for:
- Events at the bottom:
FailedScheduling,BackOff,Unhealthy, OOM kills - Conditions:
Ready: Falsewith a reason - Container statuses: restart counts, last termination reason
Logs¶
# Current container
kubectl logs <pod> -n tomoda-prod -c backend
# Previous (after a crash)
kubectl logs <pod> -n tomoda-prod -c backend --previous
# Last 5 minutes, follow
kubectl logs -f <pod> -n tomoda-prod --since=5m
Exec into the pod¶
kubectl exec -it <pod> -n tomoda-prod -- /bin/sh
# Inside the container
env | grep -v -i password # confirm env vars are loaded
nslookup postgres-postgresql.data.svc.cluster.local
curl -v http://localhost:8080/health
Logs at scale (Loki)¶
For multi-pod or historical queries, use Grafana's Explore tab against the Loki datasource:
# All errors in prod backend in the last hour
{namespace="tomoda-prod", app="tomoda-api"} |= "error" | json
# Just 5xx responses
{namespace="tomoda-prod", app="tomoda-api"} | json | status_code >= 500
# Slow requests
{namespace="tomoda-prod", app="tomoda-api"} | json | duration_ms > 1000
Loki labels available: namespace, pod, app, container.
Metrics and alerts¶
Open Grafana and load the relevant dashboard (full list in Dashboards):
- Platform overview: node CPU, memory, disk, requests vs allocatable, restarts, firing alerts, Argo CD health
- Workloads: per namespace / workload / pod CPU, memory, throttling, restarts, OOM kills
- Tomoda API / Tomoda workers: request rate, latency, 5xx by route; async task failures
- Postgres: connections, backups, WAL archiving, cache hit ratio
- Redis: memory vs maxmemory, evictions, OOM-rejected writes
Rules and what each one means: Alerting and the Alert response runbook. Check active alerts:
kubectl port-forward svc/alertmanager-operated -n monitoring 9093:9093
# Open http://localhost:9093
Argo CD sync failures¶
# What's out of sync?
argocd app diff tomoda-prod
# Recent sync history
argocd app history tomoda-prod
# Events on the application
kubectl describe application tomoda-prod -n argocd
Common causes:
ComparisonError: manifest syntax error in the latest commit. Checkargocd app getfor the line.OutOfSyncwith no auto-sync: auto-sync was disabled (Option 3 in Rollback). Re-enable:argocd app set tomoda-prod --sync-policy automated.SyncFailed: Argo CD applied the manifest but the resource controller rejected it (PDB violation, ResourceQuota, immutable field). The error message is in the sync details.
External Secrets¶
If a pod's env vars are stale or missing, check the ExternalSecret chain:
# Is it syncing?
kubectl get externalsecret -n tomoda-prod
# STATUS should be SecretSynced; READY should be True
# Why isn't it syncing?
kubectl describe externalsecret backend-secrets-prod -n tomoda-prod
# Look at Status.Conditions for the failure reason
# Are the secret stores healthy?
kubectl get clustersecretstore
kubectl describe clustersecretstore gsm-tomoda
Common causes:
- Secret Manager key broken:
external-secrets/gcpsm-credentialsis missing, or its service account lostsecretmanager.secretAccessoror had the key deleted. Re-create it per Bootstrap Step 6. - GCP SM secret missing: the
remoteRef.keydoesn't exist. Create it viascripts/setup-gcp-secrets.sh.
Force a refresh (instead of waiting for the 1h interval):
kubectl delete externalsecret backend-secrets-prod -n tomoda-prod
# Argo CD recreates it within seconds
Photon /status health check¶
Photon is the geocoder. If localized lookups stop working:
kubectl port-forward -n platform svc/photon 2322:2322 &
curl -s http://localhost:2322/status | jq
# Expect: { "status": "Ok", ... }
If status is anything else, check the pod logs; most often it is mid-download of a new index from R2 (photon-index.tomoda.life).
kubectl logs -n platform deployment/photon --tail=100
See Photon indexing for the index refresh procedure.
Postgres¶
For DB-specific diagnostics (cluster status, replication, backup health, slow queries), see Postgres Operations.