Skip to content

Daily ops runbook

Day-to-day commands: connect to the cluster, tail logs, deploy and roll back, run psql.

When an alert fires, use the Alert response runbook instead.

Prerequisites

  • Tailscale connected and a server login (Server Access)
  • Tools and logins in place: task setup and task auth, then task setup:doctor

task shortcuts

The repo's tasks wrap most of the commands below; the full list is the Task Catalog. The long forms below are for when a task isn't expressive enough.

Manual command Task shortcut
fetch the k3s kubeconfig over SSH task kube:auth
everything at a glance task kube:status / task server:status
kubectl get pods -A -o wide task kube:pods
kubectl logs … deploy/tomoda-api (or tomoda-async) task kube:logs -- prod
kubectl rollout restart deployment/tomoda-api (or tomoda-async) task kube:rollout -- tomoda-api prod
Argo CD app sync and health task argo:apps / task argo:health
argocd app sync <name> task argo:sync -- <name>
terraform plan in infrastructure/cloudflare/ task tf:plan
list GHCR tags task images:list -- tomoda-backend
kubectl exec … psql … task pg:console (dev) or task pg:console -- prod
kubectl exec … redis-cli task redis:cli (dev) or task redis:cli -- prod
./scripts/disaster-recovery.sh --drill task dr:drill
./scripts/disaster-recovery.sh --env … --mode … task dr:latest -- prod / task dr:pitr -- prod <timestamp> / task dr:cutover -- prod <cluster>

Connecting to the cluster

task kube:auth
export KUBECONFIG=~/.kube/tomoda-ovh.yaml   # tasks use this path by default

# Verify
kubectl get nodes

One k3s cluster on the OVH server hosts both dev and prod, separated by namespace.

Tailing logs

The backend is split into two pools: tomoda-api (HTTP + WS) and tomoda-async (worker + scheduler). Pick the right one based on what you're debugging.

# API pool (dev / prod)
kubectl logs -f deployment/tomoda-api -n tomoda-dev
kubectl logs -f deployment/tomoda-api -n tomoda-prod

# Async pool (worker + scheduler)
kubectl logs -f deployment/tomoda-async -n tomoda-prod

# Last 200 lines of a specific pod
kubectl logs --tail=200 <pod-name> -n <ns>

# Previous container after a crash
kubectl logs <pod-name> -n <ns> --previous

For longer-range queries, use Loki via Grafana. The Loki datasource is wired into the in-cluster Grafana and indexes all pod logs with namespace, pod, container and app labels. Use the Explore tab with LogQL:

{namespace="tomoda-prod", app="tomoda-api"} |= "error"
{namespace="tomoda-prod", app="tomoda-async"} |= "error"

Checking Argo CD sync status

UI: https://argo-app.tomoda.life (Google SSO).

CLI:

argocd login argo-app.tomoda.life --sso

argocd app list
argocd app get tomoda-dev
argocd app get tomoda-prod

A healthy app shows Sync Status: Synced and Health Status: Healthy.

Forcing a sync or refresh

# Pull latest manifests from Git and reconcile
argocd app sync tomoda-dev

# Just re-evaluate without pulling Git (refresh)
argocd app get tomoda-dev --refresh

# Hard refresh (re-clone Git repo)
argocd app get tomoda-dev --hard-refresh

From the UI: open the app, click Sync (top right). Use Hard Refresh under the kebab menu if you suspect Argo CD has a stale Git cache.

Restarting a deployment

# Rolls pods one-by-one respecting PDB
kubectl rollout restart deployment/tomoda-api -n tomoda-prod
kubectl rollout status deployment/tomoda-api -n tomoda-prod

# Async pool
kubectl rollout restart deployment/tomoda-async -n tomoda-prod
kubectl rollout status deployment/tomoda-async -n tomoda-prod

# Frontend
kubectl rollout restart deployment/frontend -n tomoda-prod

For HPA inspection and scaling-bound changes, see Scaling.

Use this after rotating a secret (so the pod re-reads env vars) or after a manual ConfigMap change.

Inspecting secrets

Never kubectl get secret ... -o yaml in front of others. To verify a secret exists and is synced without revealing the value:

# Are all ExternalSecrets in SecretSynced state?
kubectl get externalsecret -A

# Inspect a specific ExternalSecret resource (safe: no payload)
kubectl describe externalsecret backend-secrets-prod -n tomoda-prod

# Confirm the synced K8s Secret exists, see only key names
kubectl get secret backend-secrets-prod -n tomoda-prod -o jsonpath='{.data}' | jq 'keys'

If STATUS is anything but SecretSynced, see Debugging.

Image build logs

Image builds run in GitHub Actions (images.yml in the tomoda and devops repos):

gh run list --repo tomoda-labs/tomoda --workflow images.yml --limit 10
gh run view <run-id> --repo tomoda-labs/tomoda --log

Connecting to Postgres

The CNPG clusters live in the data namespace. Use port-forward for ad-hoc psql access; never expose them via LoadBalancer.

# Dev primary
kubectl port-forward -n data svc/postgres-dev-rw 5432:5432 &

# Get the password from the K8s secret
PGPASSWORD=$(kubectl get secret postgres-dev-credentials -n data \
  -o jsonpath='{.data.password}' | base64 -d)

PGPASSWORD="$PGPASSWORD" psql -h localhost -U tomoda_dev_user -d tomoda_dev

For prod, substitute postgres-prod-rw and tomoda_prod_user / tomoda_prod. Or exec directly into the pod:

kubectl exec -it postgres-dev-1 -n data -- psql -U tomoda_dev_user -d tomoda_dev

Connecting to Redis

# Dev
kubectl port-forward -n data svc/redis-dev-master 6379:6379 &
REDIS_PW=$(kubectl get secret backend-secrets-dev -n tomoda-dev \
  -o jsonpath='{.data.REDIS_PASSWORD}' | base64 -d)
redis-cli -h localhost -a "$REDIS_PW"

# Prod
kubectl port-forward -n data svc/redis-prod-master 6379:6379 &

Switching between dev and prod namespaces

There is one k3s cluster. Dev and prod are separated by namespace, not context. Set a default namespace per shell:

# Work in dev
kubectl config set-context --current --namespace=tomoda-dev

# Work in prod
kubectl config set-context --current --namespace=tomoda-prod

# Verify
kubectl config view --minify | grep namespace

Prefer explicit -n flags in scripts. The data namespace holds shared middleware (Postgres, Redis) that both environments reference by service DNS.