Daily ops runbook¶
Day-to-day commands: connect to the cluster, tail logs, deploy and roll back, run psql.
When an alert fires, use the Alert response runbook instead.
Prerequisites
- Tailscale connected and a server login (Server Access)
- Tools and logins in place:
task setupandtask auth, thentask setup:doctor
task shortcuts
The repo's tasks wrap most of the commands below; the full list is the Task Catalog. The long forms below are for when a task isn't expressive enough.
| Manual command | Task shortcut |
|---|---|
| fetch the k3s kubeconfig over SSH | task kube:auth |
| everything at a glance | task kube:status / task server:status |
kubectl get pods -A -o wide |
task kube:pods |
kubectl logs … deploy/tomoda-api (or tomoda-async) |
task kube:logs -- prod |
kubectl rollout restart deployment/tomoda-api (or tomoda-async) |
task kube:rollout -- tomoda-api prod |
| Argo CD app sync and health | task argo:apps / task argo:health |
argocd app sync <name> |
task argo:sync -- <name> |
terraform plan in infrastructure/cloudflare/ |
task tf:plan |
| list GHCR tags | task images:list -- tomoda-backend |
kubectl exec … psql … |
task pg:console (dev) or task pg:console -- prod |
kubectl exec … redis-cli |
task redis:cli (dev) or task redis:cli -- prod |
./scripts/disaster-recovery.sh --drill |
task dr:drill |
./scripts/disaster-recovery.sh --env … --mode … |
task dr:latest -- prod / task dr:pitr -- prod <timestamp> / task dr:cutover -- prod <cluster> |
Connecting to the cluster¶
task kube:auth
export KUBECONFIG=~/.kube/tomoda-ovh.yaml # tasks use this path by default
# Verify
kubectl get nodes
One k3s cluster on the OVH server hosts both dev and prod, separated by namespace.
Tailing logs¶
The backend is split into two pools: tomoda-api (HTTP + WS) and
tomoda-async (worker + scheduler). Pick the right one based on what
you're debugging.
# API pool (dev / prod)
kubectl logs -f deployment/tomoda-api -n tomoda-dev
kubectl logs -f deployment/tomoda-api -n tomoda-prod
# Async pool (worker + scheduler)
kubectl logs -f deployment/tomoda-async -n tomoda-prod
# Last 200 lines of a specific pod
kubectl logs --tail=200 <pod-name> -n <ns>
# Previous container after a crash
kubectl logs <pod-name> -n <ns> --previous
For longer-range queries, use Loki via Grafana. The Loki datasource is
wired into the in-cluster Grafana and indexes all pod logs with namespace,
pod, container and app labels. Use the Explore tab with LogQL:
{namespace="tomoda-prod", app="tomoda-api"} |= "error"
{namespace="tomoda-prod", app="tomoda-async"} |= "error"
Checking Argo CD sync status¶
UI: https://argo-app.tomoda.life (Google SSO).
CLI:
argocd login argo-app.tomoda.life --sso
argocd app list
argocd app get tomoda-dev
argocd app get tomoda-prod
A healthy app shows Sync Status: Synced and Health Status: Healthy.
Forcing a sync or refresh¶
# Pull latest manifests from Git and reconcile
argocd app sync tomoda-dev
# Just re-evaluate without pulling Git (refresh)
argocd app get tomoda-dev --refresh
# Hard refresh (re-clone Git repo)
argocd app get tomoda-dev --hard-refresh
From the UI: open the app, click Sync (top right). Use Hard Refresh under the kebab menu if you suspect Argo CD has a stale Git cache.
Restarting a deployment¶
# Rolls pods one-by-one respecting PDB
kubectl rollout restart deployment/tomoda-api -n tomoda-prod
kubectl rollout status deployment/tomoda-api -n tomoda-prod
# Async pool
kubectl rollout restart deployment/tomoda-async -n tomoda-prod
kubectl rollout status deployment/tomoda-async -n tomoda-prod
# Frontend
kubectl rollout restart deployment/frontend -n tomoda-prod
For HPA inspection and scaling-bound changes, see Scaling.
Use this after rotating a secret (so the pod re-reads env vars) or after a manual ConfigMap change.
Inspecting secrets¶
Never kubectl get secret ... -o yaml in front of others. To verify a secret exists and is synced without revealing the value:
# Are all ExternalSecrets in SecretSynced state?
kubectl get externalsecret -A
# Inspect a specific ExternalSecret resource (safe: no payload)
kubectl describe externalsecret backend-secrets-prod -n tomoda-prod
# Confirm the synced K8s Secret exists, see only key names
kubectl get secret backend-secrets-prod -n tomoda-prod -o jsonpath='{.data}' | jq 'keys'
If STATUS is anything but SecretSynced, see Debugging.
Image build logs¶
Image builds run in GitHub Actions (images.yml in the tomoda and devops repos):
gh run list --repo tomoda-labs/tomoda --workflow images.yml --limit 10
gh run view <run-id> --repo tomoda-labs/tomoda --log
Connecting to Postgres¶
The CNPG clusters live in the data namespace. Use port-forward for ad-hoc psql access; never expose them via LoadBalancer.
# Dev primary
kubectl port-forward -n data svc/postgres-dev-rw 5432:5432 &
# Get the password from the K8s secret
PGPASSWORD=$(kubectl get secret postgres-dev-credentials -n data \
-o jsonpath='{.data.password}' | base64 -d)
PGPASSWORD="$PGPASSWORD" psql -h localhost -U tomoda_dev_user -d tomoda_dev
For prod, substitute postgres-prod-rw and tomoda_prod_user / tomoda_prod. Or exec directly into the pod:
kubectl exec -it postgres-dev-1 -n data -- psql -U tomoda_dev_user -d tomoda_dev
Connecting to Redis¶
# Dev
kubectl port-forward -n data svc/redis-dev-master 6379:6379 &
REDIS_PW=$(kubectl get secret backend-secrets-dev -n tomoda-dev \
-o jsonpath='{.data.REDIS_PASSWORD}' | base64 -d)
redis-cli -h localhost -a "$REDIS_PW"
# Prod
kubectl port-forward -n data svc/redis-prod-master 6379:6379 &
Switching between dev and prod namespaces¶
There is one k3s cluster. Dev and prod are separated by namespace, not context. Set a default namespace per shell:
# Work in dev
kubectl config set-context --current --namespace=tomoda-dev
# Work in prod
kubectl config set-context --current --namespace=tomoda-prod
# Verify
kubectl config view --minify | grep namespace
Prefer explicit -n flags in scripts. The data namespace holds shared middleware (Postgres, Redis) that both environments reference by service DNS.