Skip to content

IP Blocker

The IP blocker is the app layer of abuse protection. The Cloudflare edge does the heavy lifting first (managed WAF ruleset, scanner-path rules, an auth rate limit; see the DevOps network-ingress threat model). What reaches the API meets proportionate controls: an attack-shaped request is rejected on its own, and a client key is blocked only after repeated violations, for an escalating but bounded time. Shared and infrastructure addresses are never blocked. Every violation and block is logged and counted, and every block heals on its own when its TTL expires; there is no admin unblock path.

Per-account defenses (login lockout, OTP attempt limits) are separate; see Auth → Account lockout. Per-route request limits are on Rate Limiting.

Source files:

Where it runs

SetupRouter adds three middlewares to the global chi router, after CORS and SecurityHeaders:

r.Use(middleware.WorkerAuth(cfg.Abuse.WorkerAuthSecret))
r.Use(app.IPBlocker.BlockMiddleware())
r.Use(app.IPBlocker.SuspiciousActivityMiddleware())

They cover every route in every process mode (API, WS, async). /health, /ready, and /metrics are served by a bare root mux ahead of this router, so probes and scrapes never reach the blocker.

Request flow

  1. WorkerAuth checks for a valid first-party Worker signature. A signed request skips the next two steps.
  2. BlockMiddleware refuses the request with 403 CLIENT_BLOCKED if its client key is on the block list.
  3. SuspiciousActivityMiddleware matches the request against the attack patterns. A match gets 400 REQUEST_REJECTED and records one violation; the request never reaches a handler.

The client key is ip:<canonical client IP>, from ClientIP.

Attack patterns

Matching is case-insensitive and never reads the request body.

Class Matches
traversal ../, ..\, and their percent-encoded forms in the decoded or raw URL path
sqli ' or '1'='1, ' or 1=1, '; drop table, union select in any query value
xss <script, javascript: in any query value
long_query raw query string over 2000 characters

A match rejects only that request:

{ "error": { "code": "REQUEST_REJECTED", "message": "Request rejected" } }

The response does not echo the payload. A user who types ' OR 1=1 into a search box gets one 400 and nothing else.

Progressive blocking

Each violation increments abuse:violations:<key>, a fixed-window counter. When it reaches the threshold within the window, the key is blocked. Each block adds a strike, and the strike number picks the block length; repeat offenders escalate and stay at the cap.

Setting Default YAML key Env override
Violations before a block 5 abuse.violation_threshold ABUSE_VIOLATION_THRESHOLD
Violation window 10 minutes abuse.violation_window ABUSE_VIOLATION_WINDOW
Block length per strike 15m, 1h, 24h (cap) abuse.block_durations ABUSE_BLOCK_DURATIONS (comma-separated)
Strike memory 7 days, renewed on each block abuse.strike_memory none
Extra never-block CIDRs none abuse.allowlist ABUSE_ALLOWLIST (comma-separated)

config.dev.yaml and config.prod.yaml set the defaults explicitly. Validation fails the boot on a threshold below 1, a non-positive duration, or an allowlist entry that is not a CIDR.

The counter reuses the rate limiter's atomic INCR script. A request that pushes it to the threshold writes the block unless one already exists, and the counter is cleared when a block is written. A 429 from a rate limiter is not a violation: rate limits shed load on their own, and turning throttles into blocks would punish users behind a shared carrier NAT.

Blocked response

HTTP/1.1 403 Forbidden
Retry-After: 901

{
  "error": {
    "code": "CLIENT_BLOCKED",
    "message": "Too many rejected requests from your network. Try again later.",
    "details": { "retry_after_seconds": 901 }
  }
}

The reason is not returned to the client; it is in the logs.

Addresses that are never blocked

A client IP is exempt when it is:

  • inside server.trusted_proxies: the k3s pod CIDR and Cloudflare's published ranges (www.cloudflare.com/ips-v4, ips-v6). Those are shared by every Cloudflare customer. Workers subrequests to another zone arrive from 2a06:98c0:3600::103 (inside 2a06:98c0::/29), and Cloudflare's IPv4 egress is 172.64.0.0/13; both are in the list.
  • private (RFC 1918, IPv6 ULA), loopback, link-local, or unparseable: cluster infrastructure such as Traefik and the CNI gateway 10.42.0.1.
  • inside abuse.allowlist: partners or monitors behind a fixed address.

An exempt request that matches a pattern still gets the 400, and the violation is logged and counted with scope="exempt", but nothing is written to Redis, so the address can never be blocked. server.trusted_proxies in the backend YAML is the only copy of Cloudflare's ranges in this repo, used for both client IP resolution and this check; when Cloudflare changes its ranges, update it in both backend YAML files and the Traefik values together.

Signed first-party Workers

Our Cloudflare Workers (the synthetic probes and the OG preview Worker) share Cloudflare's egress addresses with every other Cloudflare customer. They identify themselves with an HMAC signature instead:

X-Tomoda-Worker-Auth: t=<unix seconds>,w=<worker name>,v1=<hex HMAC-SHA256>

v1 is HMAC-SHA256 over <t>.<w>.<METHOD>.<escaped path> (the query string is not signed) with the key in WORKER_AUTH_SECRET. The backend accepts a timestamp within 5 minutes of its clock and compares signatures in constant time (hmac.Equal). A captured signature is good only for that method and path, and only until it ages out.

A valid request skips BlockMiddleware and SuspiciousActivityMiddleware, and skips the per-IP rate limits; account limits and the login lockout still apply (see Rate Limiting). A missing or invalid header is not an error: the request carries on as an ordinary unsigned one. With WORKER_AUTH_SECRET unset (local dev, or before the secret is provisioned), every request is unsigned. Either side can therefore roll out first.

WORKER_AUTH_SECRET must be at least 32 bytes. It comes from Secret Manager through an ExternalSecret; the Workers get the same value through their secret binding.

Observability

Every violation logs abuse violation: request rejected (info) and every block logs abuse block: client key blocked (warn), with:

Field Value
client_ip resolved client IP
key client key, e.g. ip:203.0.113.7
reason pattern class (traversal, sqli, xss, long_query), never the raw payload
scope public or exempt
method, path request method and URL path (first 200 characters, no query string)
strike, duration block lines only

Metrics (labels are bounded; no IPs):

Metric Labels Meaning
tomoda_abuse_violations_total reason, scope Requests rejected for a pattern
tomoda_abuse_blocks_total reason, strike Keys put on the block list
tomoda_abuse_active_blocks none Keys currently blocked (read from Redis at most every 30s, so every pod reports the same value; aggregate with max)
tomoda_abuse_blocked_requests_total none Requests refused because the key is blocked
tomoda_abuse_store_errors_total op Redis errors (lookup, record, scan)
tomoda_abuse_worker_auth_total result Signed-header checks: valid, malformed, expired, bad_signature, disabled

Signals worth watching:

  • tomoda_abuse_violations_total{scope="exempt"} rising: an exploit scan routed through another Cloudflare customer's Worker, one of ours misbehaving, or a client IP resolution regression.
  • tomoda_abuse_worker_auth_total{result!="valid"} after the Workers send the header: clock skew or a mismatched secret.
  • tomoda_abuse_store_errors_total{op="lookup"}: blocks are not being enforced.

Redis failure mode

The blocker fails open. If Redis cannot answer the block lookup, the request proceeds; if a violation cannot be recorded, the request still gets its 400. Availability wins because the edge WAF still filters exploit traffic and a Redis outage must not take the API down. Every failure increments tomoda_abuse_store_errors_total{op} and logs abuse blocker store unavailable; failing open at most once a minute per pod. The credential rate limiters are separate and fail closed; see Rate Limiting.

Redis keys

Key Value TTL
abuse:block:<key> JSON {key, reason, strike, since, until} the block length
abuse:violations:<key> integer violation window, from the first violation
abuse:strikes:<key> integer strike memory, renewed on each block

Keys live in the default database of the environment's Redis (redis-dev or redis-prod in the data namespace). The older blocked_ip:<ip> and violations:<ip> keys are no longer read and expire on their own.

Emergency unblock

Blocks heal through their TTL alone (15m, 1h, 24h at most). To lift one early, open task redis:cli -- prod from the devops repo and run DEL abuse:block:ip:<ip> abuse:strikes:ip:<ip>.

Client IP resolution

Every IP-keyed defense (this blocker, the rate limiter, audit attribution) keys on api.ClientIP. With server.trusted_proxies set (dev and prod both set it), the rules are:

  1. If the transport peer is outside every trusted CIDR, it is the client and forwarding headers are ignored.
  2. Otherwise X-Forwarded-For is walked right to left, skipping trusted addresses. The first untrusted entry is the client.
  3. If every entry is trusted, the leftmost one wins. If there is no X-Forwarded-For at all, X-Real-IP is used, then the peer.

trusted_proxy_count (hop counting from the right) applies only when trusted_proxies is empty. CF-Connecting-IP and True-Client-IP are not read.

config.dev.yaml and config.prod.yaml trust the k3s pod CIDR 10.42.0.0/16 (Traefik and the CNI gateway) plus Cloudflare's published edge ranges. TRUSTED_PROXIES (comma-separated CIDRs) replaces the YAML list. On the cluster side, Traefik's Service uses externalTrafficPolicy: Local so the real source IP survives, and forwardedHeaders.trustedIPs lists Cloudflare's ranges so Traefik keeps the edge's X-Forwarded-For and appends its own peer (see the DevOps Traefik page).

Path Chain seen by the pod (peer is Traefik) Resolved client
Cloudflare-proxied host client, cf-edge client
Unproxied host client client
Worker subrequest from another zone 2a06:98c0:3600::103, cf-edge 2a06:98c0:3600::103 (trusted proxy, never blocked)
Kubelet probe no header, peer 10.42.0.1 10.42.0.1 (internal, never blocked)

Because every Cloudflare address is trusted, a Worker subrequest resolves to the Workers egress address itself, which is shared. That is why first-party Workers sign their requests instead of relying on their IP.