Skip to content

Debugging

A field guide for common production issues. Start with kubectl describe, escalate to logs, then dashboards.

First moves

# What's broken?
kubectl get pods -A | grep -v Running

# Recent events across the namespace
kubectl get events -n tomoda-prod --sort-by='.lastTimestamp' | tail -30

# Argo CD app health
argocd app list
argocd app get tomoda-prod

Pod-level diagnosis

describe first

kubectl describe pod <pod-name> -n tomoda-prod

Look for:

  • Events at the bottom: FailedScheduling, BackOff, Unhealthy, OOM kills
  • Conditions: Ready: False with a reason
  • Container statuses: restart counts, last termination reason

Logs

# Current container
kubectl logs <pod> -n tomoda-prod -c backend

# Previous (after a crash)
kubectl logs <pod> -n tomoda-prod -c backend --previous

# Last 5 minutes, follow
kubectl logs -f <pod> -n tomoda-prod --since=5m

Exec into the pod

kubectl exec -it <pod> -n tomoda-prod -- /bin/sh

# Inside the container
env | grep -v -i password   # confirm env vars are loaded
nslookup postgres-postgresql.data.svc.cluster.local
curl -v http://localhost:8080/health

Logs at scale (Loki)

For multi-pod or historical queries, use Grafana's Explore tab against the Loki datasource:

# All errors in prod backend in the last hour
{namespace="tomoda-prod", app="tomoda-api"} |= "error" | json

# Just 5xx responses
{namespace="tomoda-prod", app="tomoda-api"} | json | status_code >= 500

# Slow requests
{namespace="tomoda-prod", app="tomoda-api"} | json | duration_ms > 1000

Loki labels available: namespace, pod, app, container.

Metrics and alerts

Open Grafana and load the relevant dashboard (full list in Dashboards):

  • Platform overview: node CPU, memory, disk, requests vs allocatable, restarts, firing alerts, Argo CD health
  • Workloads: per namespace / workload / pod CPU, memory, throttling, restarts, OOM kills
  • Tomoda API / Tomoda workers: request rate, latency, 5xx by route; async task failures
  • Postgres: connections, backups, WAL archiving, cache hit ratio
  • Redis: memory vs maxmemory, evictions, OOM-rejected writes

Rules and what each one means: Alerting and the Alert response runbook. Check active alerts:

kubectl port-forward svc/alertmanager-operated -n monitoring 9093:9093
# Open http://localhost:9093

Argo CD sync failures

# What's out of sync?
argocd app diff tomoda-prod

# Recent sync history
argocd app history tomoda-prod

# Events on the application
kubectl describe application tomoda-prod -n argocd

Common causes:

  • ComparisonError: manifest syntax error in the latest commit. Check argocd app get for the line.
  • OutOfSync with no auto-sync: auto-sync was disabled (Option 3 in Rollback). Re-enable: argocd app set tomoda-prod --sync-policy automated.
  • SyncFailed: Argo CD applied the manifest but the resource controller rejected it (PDB violation, ResourceQuota, immutable field). The error message is in the sync details.

External Secrets

If a pod's env vars are stale or missing, check the ExternalSecret chain:

# Is it syncing?
kubectl get externalsecret -n tomoda-prod
# STATUS should be SecretSynced; READY should be True

# Why isn't it syncing?
kubectl describe externalsecret backend-secrets-prod -n tomoda-prod
# Look at Status.Conditions for the failure reason

# Are the secret stores healthy?
kubectl get clustersecretstore
kubectl describe clustersecretstore gsm-tomoda

Common causes:

  • Secret Manager key broken: external-secrets/gcpsm-credentials is missing, or its service account lost secretmanager.secretAccessor or had the key deleted. Re-create it per Bootstrap Step 6.
  • GCP SM secret missing: the remoteRef.key doesn't exist. Create it via scripts/setup-gcp-secrets.sh.

Force a refresh (instead of waiting for the 1h interval):

kubectl delete externalsecret backend-secrets-prod -n tomoda-prod
# Argo CD recreates it within seconds

Photon /status health check

Photon is the geocoder. If localized lookups stop working:

kubectl port-forward -n platform svc/photon 2322:2322 &
curl -s http://localhost:2322/status | jq
# Expect: { "status": "Ok", ... }

If status is anything else, check the pod logs; most often it is mid-download of a new index from R2 (photon-index.tomoda.life).

kubectl logs -n platform deployment/photon --tail=100

See Photon indexing for the index refresh procedure.

Postgres

For DB-specific diagnostics (cluster status, replication, backup health, slow queries), see Postgres Operations.