Skip to content

Synthetic monitoring

External uptime and health probes from a Cloudflare Worker, every minute from Cloudflare's edge. Catches what in-cluster Prometheus can't see when the cluster itself is down: dead server, broken DNS, a public hostname that stopped resolving.

Architecture

Cloudflare Worker (tomoda-synthetic)         Cloudflare Tunnel + Access
runs every 1 min via cron                    (loki-push.tomoda.life)
   │                                                │
   ├─► probe N URLs in parallel                     │
   │      (fetch with 5-8s timeout)                 │
   │                                                │
   ├─► reconcile against Workers KV state           │
   │   (healthy ↔ failed transitions)               │
   │                                                │
   ├─► state-change notifications ───► Discord #monitoring
   │                                    (via Tomodachi webhook)
   │                                                │
   └─► every probe result ──── HTTPS POST ──────────┤
                                                    │
                                                    ▼
                                    Cloudflare Access validates
                                    CF-Access-Client-Id / -Secret
                                                    │
                                                    ▼
                                    cloudflared (monitoring ns)
                                                    │
                                                    ▼
                                    loki.monitoring.svc.cluster.local:3100
                                                    │
                                                    ▼
                                    Loki ingests log line per probe

Probes

Defined in synthetic/worker.js:

Probe URL Allowed status Timeout
api https://api.tomoda.life/health 200 5s
api-dev https://api-dev.tomoda.life/health 200 5s
app https://tomoda.life (the canonical prod web host) 200 5s
app-dev https://app-dev.tomoda.life 200 5s
assets https://assets.tomoda.life (the Worker answers 404 at /) 404 5s
assets-dev https://assets-dev.tomoda.life 404 5s
argocd https://argo-app.tomoda.life 200, 302 8s
docs https://docs.tomoda.life 200, 302 5s
login https://api.tomoda.life/api/v1/auth/login (POST, real password) 200 10s
login-bad-password same URL (POST, deliberate wrong password), every 5 minutes 401 10s
auth-middleware https://api.tomoda.life/api/v1/auth/profile (GET, no auth header) 401 5s

app probes the apex because app.tomoda.life answers 301 to it at the edge (infrastructure/cloudflare/redirects.tf), so the redirect itself proves nothing about the web app.

Every probe runs each minute unless it sets everyMinutes, in which case it runs only when the scheduled minute is a multiple of that value. A skipped run reads and writes nothing in KV, pushes nothing to Loki and does not count toward failThreshold.

302 is healthy for argocd and docs: they're gated by oauth2-proxy and Cloudflare Access, so a redirect to the auth flow is the expected answer.

Auth probes

  • login POSTs the real credentials, exercising handler, JWT signing and DB read. The Worker gets LOGIN_PROBE_EMAIL (Terraform variable synthetic_probe_email, default synthetic+probe@tomoda.life) and LOGIN_PROBE_PASSWORD (synthetic_probe_password, from GCP SM tomoda-synthetic-probe-password at apply time). The backend seeds the same user at boot from that secret (Manual setup). A 401 means the user is missing, inactive, or the password was rotated without restarting the backend.
  • login-bad-password sends the real email with a wrong password hardcoded in worker.js and expects 401. Anything else may mean any password is accepted. It runs every 5 minutes, not every minute: the backend locks an account after 5 failed logins in 15 minutes, and a lockout turns both login probes into 429. Every 5 minutes is at most 3 or 4 failures in any 15-minute window, and successful login runs do not count. With the default failThreshold of 2, this probe posts DOWN after two failed runs, about 5 minutes apart.

While an account is locked, every login attempt, including a correct one, is recorded as another failure. So a lockout of the probe user does not clear while the login probe keeps running each minute. See the runbook to clear it. - auth-middleware GETs /api/v1/auth/profile with no Authorization header and expects 401. A 200 means protected endpoints are public; a 404 means the route moved and the probe URL needs updating.

Treat a failure of either negative probe as a security incident until proven otherwise (Alert runbook).

Alert behaviour: state-change only

Each probe has one JSON record in Workers KV under probe/<name>: {state, failedSince, lastReminder, fails}, where state is healthy, pending or failed. A missing record counts as healthy.

Transition Discord behaviour
healthy → pending Silent: first failed run
pending → failed 🚨 DOWN message with the probe name + failure detail
pending → healthy Silent: a one-run blip
failed → healthy ✅ RECOVERED message with downtime duration
failed → failed Silent, except a STILL-DOWN reminder every 60 minutes
healthy → healthy Silent (the common case)

Debounce

A probe goes DOWN only after failThreshold consecutive failed runs (default 2, settable per probe in PROBES). With the default, DOWN posts on the second failed minute, and the reported downtime counts from the first failed run. The failure count is kept only while a probe is pending; any passing run resets it.

A 6-hour outage posts about 7 messages per probe (DOWN, 5 reminders, RECOVERED), not 360.

KV write budget

The free plan allows 1000 KV writes and 1000 deletes per day per account. KV is written only when a record changes:

Event Writes Deletes
Healthy run, or failed run between reminders 0 0
One-run blip (pending, then healthy) 2 0
Incident (pending, failed, healthy) 3 0
Each STILL-DOWN reminder 1 0

So an incident of H hours costs 3 + floor(H) writes per probe, and a probe flapping one run at a time costs 2 writes per flap with no alerts. Reads are one per probe per run (about 15k a day), well inside the 100k free limit.

Records written before the single-record layout used state/<probe>, failedSince/<probe> and lastReminder/<probe>. While probe/<name> is missing, the Worker reads those keys, writes the new record (keeping an open incident) and deletes them, so the move happens once per probe.

Loki integration

In parallel with Discord, every probe result is pushed to Loki via the tunneled loki-push.tomoda.life endpoint:

  • Stream labels: job="synthetic", probe=<name>, status=healthy|failed
  • Log body: JSON with probe, url, ok, status, duration_ms, error

Query in Grafana → Explore → Loki:

# All failed probes in the last hour
{job="synthetic", status="failed"} | json

# Specific probe latency over time
{job="synthetic", probe="api"} | json | line_format "{{.duration_ms}}ms {{.status}}"

Push path to Loki

  1. Worker: deployed by the cloudflare Terraform stack.
  2. Tunnel tomoda-prod-tunnel: loki-push.tomoda.life → in-cluster cloudflared (k8s/envs/platform/cloudflared/, runs cloudflared tunnel run --token) → Loki. Transport only.
  3. Access: gates loki-push.tomoda.life before traffic enters the tunnel; only requests with the service token's CF-Access-Client-Id / CF-Access-Client-Secret pass, so nobody else can push fake logs.

Secrets

Secret Consumer Path
tomoda-alert-webhook-primary Worker → Discord Read at apply time, bound as Worker secret DISCORD_WEBHOOK_URL
tomoda-synthetic-probe-password login probes Read at apply time, bound as LOGIN_PROBE_PASSWORD
tomoda-worker-auth-secret API probes Read at apply time, bound as WORKER_AUTH_SECRET; signs api and api-dev requests with X-Tomoda-Worker-Auth (Cloudflare)
tomoda-cloudflare-tunnel-token cloudflared ESO → monitoring/cloudflared-tunnel-credentials → TUNNEL_TOKEN
Access service token Worker → Loki Generated by cloudflare_zero_trust_access_service_token.synthetic_worker, bound as LOKI_ACCESS_CLIENT_ID / LOKI_ACCESS_CLIENT_SECRET; lives only in Terraform state

Deployment

Code: synthetic/worker.js (single file, tested by node --test synthetic/worker.test.mjs in the Infra Lint workflow) and the Worker, KV namespace, cron trigger, tunnel, Access and DNS resources in infrastructure/cloudflare/. A merged PR touching either is applied by Terraform CI/CD. After the tunnel is first created, copy its token to Secret Manager (Manual setup).

Editing the probe list

Edit the PROBES array at the top of synthetic/worker.js and open a PR; the plan shows one change to cloudflare_workers_script.synthetic. The upload is atomic and the next cron run uses the new list.

Verification

kubectl get pods -n monitoring -l app=cloudflared          # Running
curl -s -o /dev/null -w '%{http_code}\n' https://loki-push.tomoda.life/ready   # 403 without Access headers: the gate works
  • KV: Cloudflare dashboard → Workers & Pages → tomoda-synthetic → KV tomoda-synthetic-probe-state has one probe/<name> record per probe.
  • Loki: Grafana Explore, {job="synthetic"} | json shows one line per probe per minute.

Rotation

The two -replace rotations are intentional exceptions to apply-on-merge: CI cannot force a replacement, so they run locally against a reviewed plan.

Secret When How
Discord webhook If leaked New webhook, update GCP SM, re-run the cloudflare apply (gh workflow run terraform-apply.yml)
Access service token Yearly (expires after 1 year) terraform apply -replace=cloudflare_zero_trust_access_service_token.synthetic_worker (local, reviewed): new pair, rebound to the Worker
Tunnel token If leaked terraform apply -replace=cloudflare_zero_trust_tunnel_cloudflared.loki_push (local, reviewed), copy the new token to GCP SM, restart cloudflared
  • Alerting: the Alertmanager + Discord pipeline that the synthetic probe shares (both post to #monitoring)
  • Manual setup: the hand-set values this stack depends on
  • Source code: synthetic/worker.js and infrastructure/cloudflare/