IP Blocker¶
The IP blocker is the app layer of abuse protection. The Cloudflare edge does the heavy lifting first (managed WAF ruleset, scanner-path rules, an auth rate limit; see the DevOps network-ingress threat model). What reaches the API meets proportionate controls: an attack-shaped request is rejected on its own, and a client key is blocked only after repeated violations, for an escalating but bounded time. Shared and infrastructure addresses are never blocked. Every violation and block is logged and counted, and every block heals on its own when its TTL expires; there is no admin unblock path.
Per-account defenses (login lockout, OTP attempt limits) are separate; see Auth → Account lockout. Per-route request limits are on Rate Limiting.
Source files:
backend/internal/middleware/ip_blocker.go(IPBlocker, the two middlewares)backend/internal/middleware/abuse_patterns.go(pattern classes)backend/internal/middleware/exempt.go(never-block check)backend/internal/middleware/worker_auth.go(signed first-party Workers)backend/internal/middleware/abuse_metrics.go(Prometheus metrics)backend/config/abuse.go(thresholds)backend/internal/api/context.go(ClientIP,SetTrustedProxies)
Where it runs¶
SetupRouter adds three middlewares to the global chi router, after CORS and SecurityHeaders:
r.Use(middleware.WorkerAuth(cfg.Abuse.WorkerAuthSecret))
r.Use(app.IPBlocker.BlockMiddleware())
r.Use(app.IPBlocker.SuspiciousActivityMiddleware())
They cover every route in every process mode (API, WS, async). /health, /ready, and /metrics are served by a bare root mux ahead of this router, so probes and scrapes never reach the blocker.
Request flow¶
WorkerAuthchecks for a valid first-party Worker signature. A signed request skips the next two steps.BlockMiddlewarerefuses the request with403 CLIENT_BLOCKEDif its client key is on the block list.SuspiciousActivityMiddlewarematches the request against the attack patterns. A match gets400 REQUEST_REJECTEDand records one violation; the request never reaches a handler.
The client key is ip:<canonical client IP>, from ClientIP.
Attack patterns¶
Matching is case-insensitive and never reads the request body.
| Class | Matches |
|---|---|
traversal |
../, ..\, and their percent-encoded forms in the decoded or raw URL path |
sqli |
' or '1'='1, ' or 1=1, '; drop table, union select in any query value |
xss |
<script, javascript: in any query value |
long_query |
raw query string over 2000 characters |
A match rejects only that request:
{ "error": { "code": "REQUEST_REJECTED", "message": "Request rejected" } }
The response does not echo the payload. A user who types ' OR 1=1 into a search box gets one 400 and nothing else.
Progressive blocking¶
Each violation increments abuse:violations:<key>, a fixed-window counter. When it reaches the threshold within the window, the key is blocked. Each block adds a strike, and the strike number picks the block length; repeat offenders escalate and stay at the cap.
| Setting | Default | YAML key | Env override |
|---|---|---|---|
| Violations before a block | 5 | abuse.violation_threshold |
ABUSE_VIOLATION_THRESHOLD |
| Violation window | 10 minutes | abuse.violation_window |
ABUSE_VIOLATION_WINDOW |
| Block length per strike | 15m, 1h, 24h (cap) | abuse.block_durations |
ABUSE_BLOCK_DURATIONS (comma-separated) |
| Strike memory | 7 days, renewed on each block | abuse.strike_memory |
none |
| Extra never-block CIDRs | none | abuse.allowlist |
ABUSE_ALLOWLIST (comma-separated) |
config.dev.yaml and config.prod.yaml set the defaults explicitly. Validation fails the boot on a threshold below 1, a non-positive duration, or an allowlist entry that is not a CIDR.
The counter reuses the rate limiter's atomic INCR script. A request that pushes it to the threshold writes the block unless one already exists, and the counter is cleared when a block is written. A 429 from a rate limiter is not a violation: rate limits shed load on their own, and turning throttles into blocks would punish users behind a shared carrier NAT.
Blocked response¶
HTTP/1.1 403 Forbidden
Retry-After: 901
{
"error": {
"code": "CLIENT_BLOCKED",
"message": "Too many rejected requests from your network. Try again later.",
"details": { "retry_after_seconds": 901 }
}
}
The reason is not returned to the client; it is in the logs.
Addresses that are never blocked¶
A client IP is exempt when it is:
- inside
server.trusted_proxies: the k3s pod CIDR and Cloudflare's published ranges (www.cloudflare.com/ips-v4,ips-v6). Those are shared by every Cloudflare customer. Workers subrequests to another zone arrive from2a06:98c0:3600::103(inside2a06:98c0::/29), and Cloudflare's IPv4 egress is172.64.0.0/13; both are in the list. - private (RFC 1918, IPv6 ULA), loopback, link-local, or unparseable: cluster infrastructure such as Traefik and the CNI gateway
10.42.0.1. - inside
abuse.allowlist: partners or monitors behind a fixed address.
An exempt request that matches a pattern still gets the 400, and the violation is logged and counted with scope="exempt", but nothing is written to Redis, so the address can never be blocked. server.trusted_proxies in the backend YAML is the only copy of Cloudflare's ranges in this repo, used for both client IP resolution and this check; when Cloudflare changes its ranges, update it in both backend YAML files and the Traefik values together.
Signed first-party Workers¶
Our Cloudflare Workers (the synthetic probes and the OG preview Worker) share Cloudflare's egress addresses with every other Cloudflare customer. They identify themselves with an HMAC signature instead:
X-Tomoda-Worker-Auth: t=<unix seconds>,w=<worker name>,v1=<hex HMAC-SHA256>
v1 is HMAC-SHA256 over <t>.<w>.<METHOD>.<escaped path> (the query string is not signed) with the key in WORKER_AUTH_SECRET. The backend accepts a timestamp within 5 minutes of its clock and compares signatures in constant time (hmac.Equal). A captured signature is good only for that method and path, and only until it ages out.
A valid request skips BlockMiddleware and SuspiciousActivityMiddleware, and skips the per-IP rate limits; account limits and the login lockout still apply (see Rate Limiting). A missing or invalid header is not an error: the request carries on as an ordinary unsigned one. With WORKER_AUTH_SECRET unset (local dev, or before the secret is provisioned), every request is unsigned. Either side can therefore roll out first.
WORKER_AUTH_SECRET must be at least 32 bytes. It comes from Secret Manager through an ExternalSecret; the Workers get the same value through their secret binding.
Observability¶
Every violation logs abuse violation: request rejected (info) and every block logs abuse block: client key blocked (warn), with:
| Field | Value |
|---|---|
client_ip |
resolved client IP |
key |
client key, e.g. ip:203.0.113.7 |
reason |
pattern class (traversal, sqli, xss, long_query), never the raw payload |
scope |
public or exempt |
method, path |
request method and URL path (first 200 characters, no query string) |
strike, duration |
block lines only |
Metrics (labels are bounded; no IPs):
| Metric | Labels | Meaning |
|---|---|---|
tomoda_abuse_violations_total |
reason, scope |
Requests rejected for a pattern |
tomoda_abuse_blocks_total |
reason, strike |
Keys put on the block list |
tomoda_abuse_active_blocks |
none | Keys currently blocked (read from Redis at most every 30s, so every pod reports the same value; aggregate with max) |
tomoda_abuse_blocked_requests_total |
none | Requests refused because the key is blocked |
tomoda_abuse_store_errors_total |
op |
Redis errors (lookup, record, scan) |
tomoda_abuse_worker_auth_total |
result |
Signed-header checks: valid, malformed, expired, bad_signature, disabled |
Signals worth watching:
tomoda_abuse_violations_total{scope="exempt"}rising: an exploit scan routed through another Cloudflare customer's Worker, one of ours misbehaving, or a client IP resolution regression.tomoda_abuse_worker_auth_total{result!="valid"}after the Workers send the header: clock skew or a mismatched secret.tomoda_abuse_store_errors_total{op="lookup"}: blocks are not being enforced.
Redis failure mode¶
The blocker fails open. If Redis cannot answer the block lookup, the request proceeds; if a violation cannot be recorded, the request still gets its 400. Availability wins because the edge WAF still filters exploit traffic and a Redis outage must not take the API down. Every failure increments tomoda_abuse_store_errors_total{op} and logs abuse blocker store unavailable; failing open at most once a minute per pod. The credential rate limiters are separate and fail closed; see Rate Limiting.
Redis keys¶
| Key | Value | TTL |
|---|---|---|
abuse:block:<key> |
JSON {key, reason, strike, since, until} |
the block length |
abuse:violations:<key> |
integer | violation window, from the first violation |
abuse:strikes:<key> |
integer | strike memory, renewed on each block |
Keys live in the default database of the environment's Redis (redis-dev or redis-prod in the data namespace). The older blocked_ip:<ip> and violations:<ip> keys are no longer read and expire on their own.
Emergency unblock¶
Blocks heal through their TTL alone (15m, 1h, 24h at most). To lift one early, open task redis:cli -- prod from the devops repo and run DEL abuse:block:ip:<ip> abuse:strikes:ip:<ip>.
Client IP resolution¶
Every IP-keyed defense (this blocker, the rate limiter, audit attribution) keys on api.ClientIP. With server.trusted_proxies set (dev and prod both set it), the rules are:
- If the transport peer is outside every trusted CIDR, it is the client and forwarding headers are ignored.
- Otherwise
X-Forwarded-Foris walked right to left, skipping trusted addresses. The first untrusted entry is the client. - If every entry is trusted, the leftmost one wins. If there is no
X-Forwarded-Forat all,X-Real-IPis used, then the peer.
trusted_proxy_count (hop counting from the right) applies only when trusted_proxies is empty. CF-Connecting-IP and True-Client-IP are not read.
config.dev.yaml and config.prod.yaml trust the k3s pod CIDR 10.42.0.0/16 (Traefik and the CNI gateway) plus Cloudflare's published edge ranges. TRUSTED_PROXIES (comma-separated CIDRs) replaces the YAML list. On the cluster side, Traefik's Service uses externalTrafficPolicy: Local so the real source IP survives, and forwardedHeaders.trustedIPs lists Cloudflare's ranges so Traefik keeps the edge's X-Forwarded-For and appends its own peer (see the DevOps Traefik page).
| Path | Chain seen by the pod (peer is Traefik) | Resolved client |
|---|---|---|
| Cloudflare-proxied host | client, cf-edge |
client |
| Unproxied host | client |
client |
| Worker subrequest from another zone | 2a06:98c0:3600::103, cf-edge |
2a06:98c0:3600::103 (trusted proxy, never blocked) |
| Kubelet probe | no header, peer 10.42.0.1 |
10.42.0.1 (internal, never blocked) |
Because every Cloudflare address is trusted, a Worker subrequest resolves to the Workers egress address itself, which is shared. That is why first-party Workers sign their requests instead of relying on their IP.