Skip to content

Capture-picker harvesting

Part of the Locations domain: how the picker's paid harvest buys, caches, bounds and writes place data. Ingest, dedup and promotion are on the Locations page.

The check-in and moment picker lists every place within the ring, nearest first. Where the catalog is thin, the first person to open the picker in an area triggers one harvest from Serper /maps. Every returned place is minted as a permanent harvested row, so the next person there is served from the DB with no provider call.

Flow. GET /nearby paints the DB + Photon list instantly and reports fill_needed. The client then calls POST /nearby/fill, which harvest.Engine.Fill runs in this order:

  1. Read the location_seed_tiles ledger for each query of the fill: one tile per (cell, query, page), keyed q:{query} (q:{query}:{page} from page 2). A tile inside its TTL means that query is fresh; only cold queries are bought, so a chip never re-buys a query the sweep bought and the reverse. A Redis verdict cache sits in front of it; a ledger read error counts as warm, so a database blip never spends. A tile records how many places its query returned, the page's full-page count.
  2. Take a per-cell Redis lock (harvest:lock:{cell}:fill, released only by its owner's token). Requests for one cell in a process share a single run, and a concurrent request elsewhere waits for the first one's result instead of paying again.
  3. Resolve the admin area once (at the user's point for their own cell, at a neighbouring cell's centre for the others): boundary polygons, then one Photon reverse lookup, then the device's ISO country_code, which only selects a row in our countries table. Without a country nothing is minted (check-in rejects a place without one): the fill returns no_area, spends nothing, and leaves the tile cold. The area is resolved before any budget is reserved.
  4. Reserve credits from the Governor (below), then take a cluster-wide provider slot.
  5. Run the queries in parallel on a detached context with their own timeout. The request waits only up to burst_fill_budget_ms; past it the client gets deferred, and the continuation still writes the paid answer and stamps the tile. A request that is cancelled does not cancel the purchase.
  6. Store the fetched answer in the Redis outbox (harvest:outbox:{id}, 72 h), write the places through Ingest (the harvestSink), stamp the tile with its novelty (new / returned) and delete the outbox entry. If the write fails, or the process is draining at shutdown, the answer is handed to location:burst_seed instead, so a paid answer is never bought twice.

On shutdown the HTTP server gets 5 s and the harvest drain at most 20 s, which with the 4 s metrics flush fits Kubernetes' default 30 s terminationGracePeriodSeconds; no larger grace period is needed because answers still in flight are in the outbox.

A panic in a provider call or a continuation is recovered, counted (tomoda_locations_harvest_panics_total) and billed as a failed call; the cell lock is released by the deferred unlock. At most MaxInflight (64) continuations run per process, past which a paid request is refused as busy.

Provider failures never look like empty areas. MapsNearbyE reports a failed call as a *places.ProviderError (401, 402, 429, 5xx, timeout). A tile is stamped empty only when every query answered 200 with zero places. A failed fill leaves the tile cold and counts a failure on it; after 3 consecutive failures the tile is stamped error for 1 hour (tomoda_locations_harvest_error_tiles_total) and the picker stops asking. A partial batch (some queries failed) writes the places that arrived but does not settle the tile, so the missing query is retried.

Focus. Each returned place is classified from its provider types as core (places people post from: food and drink, nightlife, sights, culture, outdoors, big venues, shopping destinations, event venues, stays, travel hubs), secondary (gyms, spas, salons, campuses, hospitals, convenience stores, minor stations, and any type we do not recognize) or services (banks, parking, offices, stop-level transit). The taxonomy leaf the type resolves to decides first, because the leaf is the same in every language (a Japanese or Chinese type classifies like its English twin); only a type that resolves to no leaf falls back to English type fragments, the most specific winning, so "event planner" is a service though "event" is core. Every place inside the area is stored with its tier in harvest_tier. A service place from an area harvest (sweep or chip) is stored inactive, out of search and nearby, and goes live when someone looks it up by name or engages with it; a typed search-more lookup stores everything live. Secondary rows are listed behind core ones, flagged secondary, and the picker shows them under an "Other nearby" label. Only results beyond max(1 km, 3 x the call radius) are dropped.

Freshness. A harvested row carries last_seen_at (stamped on create and on every harvest that returns it) and closure_reported_at. A re-harvest refreshes the provider signals, hours and site of unpromoted rows. A place the provider reports permanently closed closes at once when it is still a harvested row; a curated or promoted row records the report and closes on a second one at least a week later. A harvest that returns a place open again reopens it, unless the row is soft-deleted. A place a harvest does not return is left as it is: absence changes nothing.

Governor. Spend is metered in Serper credits and bounded by safety ceilings, not throttles: normal use never meets them. A reservation uses the configured per-request estimate (config.SerperCreditsPerRequest); each response's credits field is the charge, the difference is refunded or added, and a response without the field is charged the estimate and counted in tomoda_locations_provider_credits_estimated_total. Counters live in Redis per UTC day (budgets reset at 00:00 UTC, not local midnight) under one Cluster hash tag ({gov}), so the multi-key scripts stay valid on a cluster. The Postgres location_provider_spend log is the truth and re-seeds the global, per-trigger and per-user counters after a Redis flush, so a flush cannot reopen a user's day. A reported cost is clamped to a sane range (ClampCredits) wherever it is read.

Level Key Default ceiling
Global gov:{gov}:spent:{day}:global burst_daily_cap credits. A cap of 0 refuses every paid call in every environment unless burst_uncapped is set; production refuses to boot with a Serper key or harvesting on and no cap, and never accepts burst_uncapped
Per trigger gov:{gov}:spent:{day}:trigger:{t} share of the global: fill 40%, auto 20%, pages 10%, enrich 30%. These are overlapping ceilings, not slices: the defaults sum past 100% on purpose, each share must be within 0 to 100 (checked at boot), and the global cap and the explicit reserve bound the total
Explicit taps gov:{gov}:spent:{day}:trigger:explicit 20% reserved; may spill into unused general headroom, nothing spills into it. A tap has no per-user credit cap: the abuse gates (90/h per account, 120/h per device, 1500/h per IP) and the global ceiling bound it, and it is the last call to stop when the ceiling nears
Per user gov:{gov}:spent:{day}:user:fill:{id}, :user:text:{id}, :user:auto:{id} Automatic fills: 300 credits a day (about 75 cold cells). Automatic text lookups: 150 a day, enough that a heavy user never meets it. Explicit taps count in :user:text for review but are never refused on it. Accounts under 7 days old get half of the automatic pools

Caps are safety ceilings against bugs and abuse, never a reason a user cannot find a place. Every refusal returns the free results with a limited or paused status and the app shows nothing more: an automatic lookup that is refused just leaves the free list, and the keyboard's Search key (an explicit lookup) or a chip or widen tap bypasses the per-user pools. At 95% of the global cap explicit taps still answer while automatic calls are refused. The daily purge cron logs the ten users who spent the most credits in the last day. Alert on tomoda_locations_harvest_budget_used_ratio{trigger="global"} at 0.5 (info), 0.8 and 1.

Any failure to verify spend (Redis down, the controls row unreadable or slower than 2 s) refuses the paid call with paused; the free list is unaffected. A paid call with no signed-in user is refused too: only the system's enrich trigger answers to the global cap alone. The enrich trigger meters the resolve and enrichment calls, both /maps and /search under one reservation, and works whether or not picker harvesting is enabled. When the provider is enabled but the governor cannot run (no Redis scripting or no ledger), enrichment skips its paid calls, and production refuses to boot.

  • Kill switch and cap override. location_harvest_controls is a single-row table read through a 5 s cache. UPDATE location_harvest_controls SET enabled = false pauses all paid harvesting at once, and global_cap_override replaces burst_daily_cap (0 is a cap that refuses everything), both without a redeploy. A missing row means the config defaults. They are in Postgres so a Redis flush cannot re-enable spend.
  • Circuit breaker (Redis, global). A 402 opens it for 15 minutes, then one probe call is let through. More than half of the last 20 calls failing with 5xx or timeouts in a minute opens it for 2 minutes. While open, fills return paused.
  • Concurrency. gov:slots is a cluster-wide lease set bounding in-flight Serper calls, one slot per call (burst_slot_limit, default 8).
  • Ambiguous failures. A timeout, transport or parse failure may have been billed, so it stays charged at the estimate; only a definite 4xx rejection is refunded. A partial harvest stamps the queries that answered, so a retry buys only the rest.
  • Abuse limits. POST /nearby/fill, /nearby/search-more and /search-more carry RateLimiter.HarvestLimit: 90 paid reservations per hour per account, 120 per device (the X-Device-ID header, when sent) and 1500 per IP, a coarse backstop sized for carrier-grade NAT. The count is taken just before a paid reservation, so requests served from the ledger are free; a limiter outage refuses the paid call only. Only a signed first-party worker bypasses it; an internal address skips the IP ceiling but not the account and device ones. Production refuses to boot with the routes on and no trusted-proxy policy. There is no per-country cap.

Worker handoff and outbox. location:burst_seed carries a harvest.Handoff: the fetched places, the cell, key and area, never the query. The worker takes the same lock, skips a tile that is already fresh, writes the places and stamps the tile. It never calls the provider, so a retry cannot buy twice; an undecodable payload is SkipRetry, a write error retries up to 12 times. Every paid answer is also kept in the Redis outbox until it is committed (the stored form, also used for the queued task, has no user and its origin is the centre of the cell, never the request point; the text key is kept), and cron:harvest_outbox_sweep (every 5 min) replays entries older than 3 minutes through the same path, so a crash, a deploy or an archived task (tomoda_locations_harvest_handoffs_archived_total) loses nothing. The server's shutdown drains in-flight continuations (StopHarvest) and counts any still running as tomoda_locations_harvest_continuations_lost_total.

Tiles are H3 cells. The ledger keys on H3 hexagons computed inside Postgres (harvest.PGGrid over the h3-pg functions behind the harvest.CellGrid interface: cell for a point, cell centre, k-ring neighbours, parent). The cell id is the cell column and the hex string in the Redis harvest:* keys. A call is centred on the cell centre, so every user in a cell shares one call and one cached answer, and the focus and distance gates run relative to that centre with a radius that reaches the cell's far edge. The zoom is chosen for that reach (an estimate from the cell size, not re-measured against the provider):

Use H3 resolution Cell edge Call reach Zoom
picker fills (any ring) 9 ~175 m 200 m 18
nearby search-more 8 ~460 m 530 m 17
area search-more cache 7 ~1.2 km 1400 m 17

A wider picker ring is covered by reaching several res 9 cells, not by a coarser tile. locations.cell9 (a trigger-maintained res 9 cell, btree-indexed) lets the picker's nearby read narrow by cell before the exact distance test, and that read selects only the columns a candidate carries.

Reach. A request fills the user's own cell and every other cell of its disk the ring covers by 20% or more (area, computed in SQL), seven at most (the k=1 disk, which holds the widest 500 m picker ring), in parallel. Each is a full fill with its own lock and reservation under the user's fill budget, and tiles stay tied to their cell centre. fill_needed is true when any of those cells is cold, answered by one SQL statement (PGGrid.Reach) that returns the disk's cells with their coverage, centre and tile freshness (a warm verdict is also remembered for a few seconds). The list read and the reach check run side by side. One request holds at most RequestCalls provider calls in flight across its cells. The cell centre only positions the provider call.

Queries. A sweep (no chip) runs places, restaurants, cafes, stores: generalist queries that barely overlap, measured on a Taipei field test. A batch that found under 20% new places marks its tiles saturated, trusted for twice the TTL. Paging is decided per query: page 2 and 3 of a query are asked only when its previous page came back full (about 20 places), a dense sweep pages in the background and a chip pages when the client asks.

Chips. Seven chips (restaurants, cafes, bars, shops, attractions, entertainment, hotels; BurstIntent) each hold an explicit set of taxonomy leaves in intentCategories. The membership is explicit because the taxonomy's buckets overlap: a Cafe and a Pub both carry food. A unit test keeps chips disjoint and every leaf real. Each chip runs its own queries (bars runs bars and nightclubs) under the same per-query tiles as the sweep. intent_counts lets the client order chips by what is actually nearby.

  • Typed categories. intentFromText runs typed text through the multilingual taxonomy parser: a parsed leaf first, then a parsed tag (tagIntent). Text with leftover words ("Blue Bottle coffee") is a name search and resolves to nothing.
  • Search more. When typing settles on weak results, any place search runs one paid text lookup, whose rows join the same list (POST /locations/search-more, Paid fallback). The catalog is read first: a listed place whose name, or any localized name, equals the text, is a typo of it, or contains all its words answers the search without a call (fresh). Text that normalises to nothing is refused before any spend. Engine.SearchMore runs the exact text through Serper once and mints the matches, caching their ids in location_seed_texts by (scope, cell, normalized text): 30 days for a hit, 7 days for a proven miss (an empty list is cached only when the call answered 200), so the same search in that area is free, even after a Redis flush and whatever language asked. A concurrent identical search waits for the first one's answer. The scope fixes the cell, the call zoom and the answer radius:
Scope Cell Call Answer kept within
nearby res 8, centre of the cell zoom 16 1 km
area res 7, centre of the cell zoom burst_zoom_area (14) 8 km
global g:{country hint}, no place no ll at all, only gl 100 km around the top result

A global answer stamps one area, resolved at its top result, on every kept place; with no country known nothing is minted and nothing is cached. Its spend rows record no cell and its stored copy carries no origin. Explicit lookups (the keyboard's Search key on weak results, a chip, a widened ring) draw on the reserved pool with no per-user daily cap; automatic lookups ("auto": true) draw on the per-user automatic pool (burst_user_daily_auto). A refusal returns the free matches with the status. The hourly per-area allowance (burst_search_more_cell_per_hour, default 120) is a spend gate: it counts only calls about to be paid, never a cached answer. - Depth on demand. A chip pages deeper (page 2, then 3) as the user reaches the end of its list. pageWorth allows the next page only while the previous page added at least 5 new places; otherwise the fill returns exhausted with no provider call. - Dense cells page in the background. The client never pages a sweep. When a sweep page returns at least 30 places and at least 70% are new to the catalog (SweepDeeper), the continuation fetches the next page (up to 3) under the pages trigger, one budgeted page at a time. Busy blocks fill out without the user waiting, and a cell we already know well stops after one page.

Metrics.

Metric Labels Reads as
tomoda_locations_provider_spend_credits_total trigger, outcome = ok or the failure class Serper credits by what triggered them
tomoda_locations_burst_requests_total op = fill / search_more, status Outcomes; fresh vs filled is the tile hit rate, limited / paused show ceilings and outages
tomoda_locations_search_requests_total surface, fallback Typed place searches and whether the free results left room for a paid lookup
tomoda_locations_search_more_total surface, trigger = auto / explicit, scope, outcome = hit / miss / fresh / refused Paid lookups; hit / (hit + miss) is the weak rule's precision
tomoda_locations_search_more_picked_total surface Picks from a paid lookup's results
tomoda_locations_search_duration_seconds tier = free / paid Search latency
tomoda_locations_burst_places_total kind = returned / new / core / secondary / service returned and new count stored places, so new / returned is coverage novelty and falls as an area fills in; core / secondary / service split every provider place by focus
tomoda_locations_provider_credits_estimated_total none Responses that omitted credits; a rise means the format changed
tomoda_locations_harvest_error_tiles_total none Cells put to rest after repeated failures
tomoda_locations_harvest_continuations_lost_total none Paid continuations abandoned at shutdown
tomoda_locations_harvest_panics_total none Recovered panics in harvest goroutines; alert on any
tomoda_locations_harvest_spend_log_failures_total none Paid calls whose spend row was not written; alert on any
tomoda_locations_harvest_breaker_state none Gauge: 0 closed, 1 probing, 2 open
tomoda_locations_harvest_budget_used_ratio trigger (global too) Today's credits over the ceiling; alert near 1
tomoda_locations_harvest_handoffs_archived_total none Worker tasks that exhausted their retries; the answer stays in the outbox
tomoda_locations_harvest_outbox_pending / _replays_total{outcome} none / ok, failed Stored answers awaiting commit and the sweeper's results

Alert on provider_credits_estimated_total too: the provider stopped reporting costs.

Spend log retention. Raw location_provider_spend rows (user, a cell coarsened to its res 7 parent, about 1.2 km, on user-triggered rows, a text hash) are kept 30 days. cron:provider_spend_purge (daily) then folds older rows into location_provider_spend_daily per day, provider, trigger, endpoint and outcome, which carries no user, cell or text hash and is kept 400 days. Expired tiles and texts are purged too, and so are settled tiles with no expiry once older than twice the tile TTL. Ledger rows, the spend log and the daily totals carry the provider's name.

Save-time quality. harvestCandidate drops permanently closed places, places with no provider id, and strays beyond max(1 km, 3x radius). Provider types map to taxonomy leaves through harvestTypeCategory, then an exact leaf name, then the taxonomy parser; unknown types become Other Location. Provider ratings, hours, and website go into signals. They never touch average_rating / total_ratings, which hold Tomoda ratings only. enrichment_at is stamped so the self-heal sweep leaves the row alone until promotion.

Photos. The thumbnail URL is not stored. After a harvest or a search-more mint ingests its rows, one location:rehost_photos task carries the (location id, thumbnail URL) pairs. The worker (RehostPlacePhotos) loads each row, skips one that already has a photo, and otherwise reuses rehostPlacePhoto (see Place photos). Every download counts against the global anomaly ceiling burst:photos:{yyyymmdd} (burst_photo_daily_ceiling, default 50000), which is not a throttle. Hitting it logs a warning, increments tomoda_locations_photo_rehost_total{outcome="ceiling"} (alert on it) and drops the rest of the task. Failures are dropped too, and the place keeps its category cover.

Config (ResolutionConfig): enable_burst_seed (off by default), burst_tile_ttl_days, burst_daily_cap, burst_uncapped (escape hatch, never in production), burst_user_daily_fills, burst_user_daily_auto, burst_share_* (fill, explicit, auto, pages, enrich), burst_slot_limit, burst_photo_daily_ceiling, burst_fill_budget_ms, burst_zoom_150 / _300 / _500, burst_zoom_area, burst_search_more_cell_per_hour. See Configuration.

Environment defaults

Env burst_daily_cap enable_burst_seed
local 200 off
dev 1000 on
prod 20000 (about $20 a day) off, turned on deliberately

The values live in config.<env>.yaml; no devops env override sets them.

Local development

  • Tests and the default local run use the fake provider and Photon only: harvesting is off (enable_burst_seed: false) and no test calls Serper.
  • task dev pulls the real Serper key, so config.local.yaml carries burst_daily_cap: 200. Enrichment calls with that key are metered and stop at 200 credits a day.
  • To try harvesting locally, set enable_burst_seed: true (or ENABLE_BURST_SEED=true). It uses the same real key and the same 200-credit ceiling. Raise burst_daily_cap only on purpose.
  • A cap of 0 means no paid calls, so a missing setting never becomes an open wallet. burst_uncapped: true (BURST_UNCAPPED=true) lifts it for a test harness and is rejected in production.