Synthetic monitoring¶
External uptime and health probes from a Cloudflare Worker, every minute from Cloudflare's edge. Catches what in-cluster Prometheus can't see when the cluster itself is down: dead server, broken DNS, a public hostname that stopped resolving.
Architecture¶
Cloudflare Worker (tomoda-synthetic) Cloudflare Tunnel + Access
runs every 1 min via cron (loki-push.tomoda.life)
│ │
├─► probe N URLs in parallel │
│ (fetch with 5-8s timeout) │
│ │
├─► reconcile against Workers KV state │
│ (healthy ↔ failed transitions) │
│ │
├─► state-change notifications ───► Discord #monitoring
│ (via Tomodachi webhook)
│ │
└─► every probe result ──── HTTPS POST ──────────┤
│
▼
Cloudflare Access validates
CF-Access-Client-Id / -Secret
│
▼
cloudflared (monitoring ns)
│
▼
loki.monitoring.svc.cluster.local:3100
│
▼
Loki ingests log line per probe
Probes¶
Defined in synthetic/worker.js:
| Probe | URL | Allowed status | Timeout |
|---|---|---|---|
api |
https://api.tomoda.life/health |
200 | 5s |
api-dev |
https://api-dev.tomoda.life/health |
200 | 5s |
app |
https://tomoda.life (the canonical prod web host) |
200 | 5s |
app-dev |
https://app-dev.tomoda.life |
200 | 5s |
assets |
https://assets.tomoda.life (the Worker answers 404 at /) |
404 | 5s |
assets-dev |
https://assets-dev.tomoda.life |
404 | 5s |
argocd |
https://argo-app.tomoda.life |
200, 302 | 8s |
docs |
https://docs.tomoda.life |
200, 302 | 5s |
login |
https://api.tomoda.life/api/v1/auth/login (POST, real password) |
200 | 10s |
login-bad-password |
same URL (POST, deliberate wrong password), every 5 minutes | 401 | 10s |
auth-middleware |
https://api.tomoda.life/api/v1/auth/profile (GET, no auth header) |
401 | 5s |
app probes the apex because app.tomoda.life answers 301 to it at the edge (infrastructure/cloudflare/redirects.tf), so the redirect itself proves nothing about the web app.
Every probe runs each minute unless it sets everyMinutes, in which case it runs only when the scheduled minute is a multiple of that value. A skipped run reads and writes nothing in KV, pushes nothing to Loki and does not count toward failThreshold.
302 is healthy for argocd and docs: they're gated by oauth2-proxy and Cloudflare Access, so a redirect to the auth flow is the expected answer.
Auth probes¶
loginPOSTs the real credentials, exercising handler, JWT signing and DB read. The Worker getsLOGIN_PROBE_EMAIL(Terraform variablesynthetic_probe_email, defaultsynthetic+probe@tomoda.life) andLOGIN_PROBE_PASSWORD(synthetic_probe_password, from GCP SMtomoda-synthetic-probe-passwordat apply time). The backend seeds the same user at boot from that secret (Manual setup). A 401 means the user is missing, inactive, or the password was rotated without restarting the backend.login-bad-passwordsends the real email with a wrong password hardcoded inworker.jsand expects 401. Anything else may mean any password is accepted. It runs every 5 minutes, not every minute: the backend locks an account after 5 failed logins in 15 minutes, and a lockout turns both login probes into 429. Every 5 minutes is at most 3 or 4 failures in any 15-minute window, and successfulloginruns do not count. With the defaultfailThresholdof 2, this probe posts DOWN after two failed runs, about 5 minutes apart.
While an account is locked, every login attempt, including a correct one, is recorded as another failure. So a lockout of the probe user does not clear while the login probe keeps running each minute. See the runbook to clear it.
- auth-middleware GETs /api/v1/auth/profile with no Authorization header and expects 401. A 200 means protected endpoints are public; a 404 means the route moved and the probe URL needs updating.
Treat a failure of either negative probe as a security incident until proven otherwise (Alert runbook).
Alert behaviour: state-change only¶
Each probe has one JSON record in Workers KV under probe/<name>: {state, failedSince, lastReminder, fails}, where state is healthy, pending or failed. A missing record counts as healthy.
| Transition | Discord behaviour |
|---|---|
healthy → pending |
Silent: first failed run |
pending → failed |
🚨 DOWN message with the probe name + failure detail |
pending → healthy |
Silent: a one-run blip |
failed → healthy |
✅ RECOVERED message with downtime duration |
failed → failed |
Silent, except a STILL-DOWN reminder every 60 minutes |
healthy → healthy |
Silent (the common case) |
Debounce¶
A probe goes DOWN only after failThreshold consecutive failed runs (default 2, settable per probe in PROBES). With the default, DOWN posts on the second failed minute, and the reported downtime counts from the first failed run. The failure count is kept only while a probe is pending; any passing run resets it.
A 6-hour outage posts about 7 messages per probe (DOWN, 5 reminders, RECOVERED), not 360.
KV write budget¶
The free plan allows 1000 KV writes and 1000 deletes per day per account. KV is written only when a record changes:
| Event | Writes | Deletes |
|---|---|---|
| Healthy run, or failed run between reminders | 0 | 0 |
One-run blip (pending, then healthy) |
2 | 0 |
Incident (pending, failed, healthy) |
3 | 0 |
| Each STILL-DOWN reminder | 1 | 0 |
So an incident of H hours costs 3 + floor(H) writes per probe, and a probe flapping one run at a time costs 2 writes per flap with no alerts. Reads are one per probe per run (about 15k a day), well inside the 100k free limit.
Records written before the single-record layout used state/<probe>, failedSince/<probe> and lastReminder/<probe>. While probe/<name> is missing, the Worker reads those keys, writes the new record (keeping an open incident) and deletes them, so the move happens once per probe.
Loki integration¶
In parallel with Discord, every probe result is pushed to Loki via the tunneled loki-push.tomoda.life endpoint:
- Stream labels:
job="synthetic",probe=<name>,status=healthy|failed - Log body: JSON with
probe,url,ok,status,duration_ms,error
Query in Grafana → Explore → Loki:
# All failed probes in the last hour
{job="synthetic", status="failed"} | json
# Specific probe latency over time
{job="synthetic", probe="api"} | json | line_format "{{.duration_ms}}ms {{.status}}"
Push path to Loki¶
- Worker: deployed by the cloudflare Terraform stack.
- Tunnel
tomoda-prod-tunnel:loki-push.tomoda.life→ in-clustercloudflared(k8s/envs/platform/cloudflared/, runscloudflared tunnel run --token) → Loki. Transport only. - Access: gates
loki-push.tomoda.lifebefore traffic enters the tunnel; only requests with the service token'sCF-Access-Client-Id/CF-Access-Client-Secretpass, so nobody else can push fake logs.
Secrets¶
| Secret | Consumer | Path |
|---|---|---|
tomoda-alert-webhook-primary |
Worker → Discord | Read at apply time, bound as Worker secret DISCORD_WEBHOOK_URL |
tomoda-synthetic-probe-password |
login probes |
Read at apply time, bound as LOGIN_PROBE_PASSWORD |
tomoda-worker-auth-secret |
API probes | Read at apply time, bound as WORKER_AUTH_SECRET; signs api and api-dev requests with X-Tomoda-Worker-Auth (Cloudflare) |
tomoda-cloudflare-tunnel-token |
cloudflared | ESO → monitoring/cloudflared-tunnel-credentials → TUNNEL_TOKEN |
| Access service token | Worker → Loki | Generated by cloudflare_zero_trust_access_service_token.synthetic_worker, bound as LOKI_ACCESS_CLIENT_ID / LOKI_ACCESS_CLIENT_SECRET; lives only in Terraform state |
Deployment¶
Code: synthetic/worker.js (single file, tested by node --test synthetic/worker.test.mjs in the Infra Lint workflow) and the Worker, KV namespace, cron trigger, tunnel, Access and DNS resources in infrastructure/cloudflare/. A merged PR touching either is applied by Terraform CI/CD. After the tunnel is first created, copy its token to Secret Manager (Manual setup).
Editing the probe list¶
Edit the PROBES array at the top of synthetic/worker.js and open a PR; the plan shows one change to cloudflare_workers_script.synthetic. The upload is atomic and the next cron run uses the new list.
Verification¶
kubectl get pods -n monitoring -l app=cloudflared # Running
curl -s -o /dev/null -w '%{http_code}\n' https://loki-push.tomoda.life/ready # 403 without Access headers: the gate works
- KV: Cloudflare dashboard → Workers & Pages →
tomoda-synthetic→ KVtomoda-synthetic-probe-statehas oneprobe/<name>record per probe. - Loki: Grafana Explore,
{job="synthetic"} | jsonshows one line per probe per minute.
Rotation¶
The two -replace rotations are intentional exceptions to apply-on-merge: CI cannot force a replacement, so they run locally against a reviewed plan.
| Secret | When | How |
|---|---|---|
| Discord webhook | If leaked | New webhook, update GCP SM, re-run the cloudflare apply (gh workflow run terraform-apply.yml) |
| Access service token | Yearly (expires after 1 year) | terraform apply -replace=cloudflare_zero_trust_access_service_token.synthetic_worker (local, reviewed): new pair, rebound to the Worker |
| Tunnel token | If leaked | terraform apply -replace=cloudflare_zero_trust_tunnel_cloudflared.loki_push (local, reviewed), copy the new token to GCP SM, restart cloudflared |
Related¶
- Alerting: the Alertmanager + Discord pipeline that the synthetic probe shares (both post to
#monitoring) - Manual setup: the hand-set values this stack depends on
- Source code:
synthetic/worker.jsandinfrastructure/cloudflare/