Capture-picker harvesting¶
Part of the Locations domain: how the picker's paid harvest buys, caches, bounds and writes place data. Ingest, dedup and promotion are on the Locations page.
The check-in and moment picker lists every place within the ring, nearest
first. Where the catalog is thin, the first person to open the picker in an area
triggers one harvest from Serper /maps. Every returned place is minted as a
permanent harvested row, so the next person there is served from the DB with no
provider call.
Flow. GET /nearby paints the DB + Photon list instantly and reports
fill_needed. The client then calls POST /nearby/fill, which harvest.Engine.Fill
runs in this order:
- Read the
location_seed_tilesledger for each query of the fill: one tile per (cell, query, page), keyedq:{query}(q:{query}:{page}from page 2). A tile inside its TTL means that query isfresh; only cold queries are bought, so a chip never re-buys a query the sweep bought and the reverse. A Redis verdict cache sits in front of it; a ledger read error counts as warm, so a database blip never spends. A tile records how many places its query returned, the page's full-page count. - Take a per-cell Redis lock (
harvest:lock:{cell}:fill, released only by its owner's token). Requests for one cell in a process share a single run, and a concurrent request elsewhere waits for the first one's result instead of paying again. - Resolve the admin area once (at the user's point for their own cell, at a
neighbouring cell's centre for the others): boundary polygons, then one Photon reverse lookup,
then the device's ISO
country_code, which only selects a row in ourcountriestable. Without a country nothing is minted (check-in rejects a place without one): the fill returnsno_area, spends nothing, and leaves the tile cold. The area is resolved before any budget is reserved. - Reserve credits from the Governor (below), then take a cluster-wide provider slot.
- Run the queries in parallel on a detached context with their own timeout. The
request waits only up to
burst_fill_budget_ms; past it the client getsdeferred, and the continuation still writes the paid answer and stamps the tile. A request that is cancelled does not cancel the purchase. - Store the fetched answer in the Redis outbox (
harvest:outbox:{id}, 72 h), write the places throughIngest(theharvestSink), stamp the tile with its novelty (new / returned) and delete the outbox entry. If the write fails, or the process is draining at shutdown, the answer is handed tolocation:burst_seedinstead, so a paid answer is never bought twice.
On shutdown the HTTP server gets 5 s and the harvest drain at most 20 s, which with the
4 s metrics flush fits Kubernetes' default 30 s terminationGracePeriodSeconds; no larger grace period is needed because answers still in flight are in the outbox.
A panic in a provider call or a continuation is recovered, counted
(tomoda_locations_harvest_panics_total) and billed as a failed call; the cell lock is
released by the deferred unlock. At most MaxInflight (64) continuations run per
process, past which a paid request is refused as busy.
Provider failures never look like empty areas. MapsNearbyE reports a failed
call as a *places.ProviderError (401, 402, 429, 5xx, timeout). A tile is stamped
empty only when every query answered 200 with zero places. A failed fill leaves
the tile cold and counts a failure on it; after 3 consecutive failures the tile is
stamped error for 1 hour (tomoda_locations_harvest_error_tiles_total) and the
picker stops asking. A partial batch (some queries failed) writes the places that
arrived but does not settle the tile, so the missing query is retried.
Focus. Each returned place is classified from its provider types as core
(places people post from: food and drink, nightlife, sights, culture, outdoors,
big venues, shopping destinations, event venues, stays, travel hubs),
secondary (gyms, spas, salons, campuses, hospitals, convenience stores, minor
stations, and any type we do not recognize) or services (banks, parking, offices,
stop-level transit). The taxonomy leaf the type resolves to decides first, because
the leaf is the same in every language (a Japanese or Chinese type classifies like
its English twin); only a type that resolves to no leaf falls back to English type
fragments, the most specific winning, so "event planner" is a service though
"event" is core. Every place inside the area is stored with its tier in
harvest_tier. A service place from an area harvest (sweep or chip) is stored inactive, out of search and nearby, and goes live when
someone looks it up by name or engages with it; a typed search-more lookup stores
everything live. Secondary rows are listed behind core ones, flagged secondary,
and the picker shows them under an "Other nearby" label. Only results beyond
max(1 km, 3 x the call radius) are dropped.
Freshness. A harvested row carries last_seen_at (stamped on create and on
every harvest that returns it) and closure_reported_at. A re-harvest refreshes the
provider signals, hours and site of unpromoted rows. A place the provider reports
permanently closed closes at once when it is still a harvested row; a curated or
promoted row records the report and closes on a second one at least a week later.
A harvest that returns a place open again reopens it, unless the row is soft-deleted.
A place a harvest does not return is left as it is: absence changes nothing.
Governor. Spend is metered in Serper credits and bounded by safety ceilings,
not throttles: normal use never meets them. A reservation uses the configured
per-request estimate (config.SerperCreditsPerRequest); each response's credits
field is the charge, the difference is refunded or added, and a response without
the field is charged the estimate and counted in
tomoda_locations_provider_credits_estimated_total. Counters live in Redis per UTC day
(budgets reset at 00:00 UTC, not local midnight) under one Cluster hash tag ({gov}),
so the multi-key scripts stay valid on a cluster. The Postgres location_provider_spend
log is the truth and re-seeds the global, per-trigger and per-user counters after a
Redis flush, so a flush cannot reopen a user's day. A reported cost is clamped to a
sane range (ClampCredits) wherever it is read.
| Level | Key | Default ceiling |
|---|---|---|
| Global | gov:{gov}:spent:{day}:global |
burst_daily_cap credits. A cap of 0 refuses every paid call in every environment unless burst_uncapped is set; production refuses to boot with a Serper key or harvesting on and no cap, and never accepts burst_uncapped |
| Per trigger | gov:{gov}:spent:{day}:trigger:{t} |
share of the global: fill 40%, auto 20%, pages 10%, enrich 30%. These are overlapping ceilings, not slices: the defaults sum past 100% on purpose, each share must be within 0 to 100 (checked at boot), and the global cap and the explicit reserve bound the total |
| Explicit taps | gov:{gov}:spent:{day}:trigger:explicit |
20% reserved; may spill into unused general headroom, nothing spills into it. A tap has no per-user credit cap: the abuse gates (90/h per account, 120/h per device, 1500/h per IP) and the global ceiling bound it, and it is the last call to stop when the ceiling nears |
| Per user | gov:{gov}:spent:{day}:user:fill:{id}, :user:text:{id}, :user:auto:{id} |
Automatic fills: 300 credits a day (about 75 cold cells). Automatic text lookups: 150 a day, enough that a heavy user never meets it. Explicit taps count in :user:text for review but are never refused on it. Accounts under 7 days old get half of the automatic pools |
Caps are safety ceilings against bugs and abuse, never a reason a user cannot find a place. Every refusal returns the free results with a limited or paused status and the app shows nothing more: an automatic lookup that is refused just leaves the free list, and the keyboard's Search key (an explicit lookup) or a chip or widen tap bypasses the per-user pools. At 95% of the global cap explicit taps still answer while automatic calls are refused. The daily purge cron logs the ten users who spent the most credits in the last day. Alert on tomoda_locations_harvest_budget_used_ratio{trigger="global"} at 0.5 (info), 0.8 and 1.
Any failure to verify spend (Redis down, the controls row unreadable or slower than 2 s)
refuses the paid call with paused; the free list is unaffected. A paid call with no
signed-in user is refused too: only the system's enrich trigger answers to the global
cap alone. The enrich trigger meters the resolve and enrichment calls, both /maps
and /search under one reservation, and works whether or not picker harvesting is
enabled. When the provider is enabled but the governor cannot run (no Redis scripting or
no ledger), enrichment skips its paid calls, and production refuses to boot.
- Kill switch and cap override.
location_harvest_controlsis a single-row table read through a 5 s cache.UPDATE location_harvest_controls SET enabled = falsepauses all paid harvesting at once, andglobal_cap_overridereplacesburst_daily_cap(0 is a cap that refuses everything), both without a redeploy. A missing row means the config defaults. They are in Postgres so a Redis flush cannot re-enable spend. - Circuit breaker (Redis, global). A 402 opens it for 15 minutes, then one probe
call is let through. More than half of the last 20 calls failing with 5xx or
timeouts in a minute opens it for 2 minutes. While open, fills return
paused. - Concurrency.
gov:slotsis a cluster-wide lease set bounding in-flight Serper calls, one slot per call (burst_slot_limit, default 8). - Ambiguous failures. A timeout, transport or parse failure may have been billed, so it stays charged at the estimate; only a definite 4xx rejection is refunded. A partial harvest stamps the queries that answered, so a retry buys only the rest.
- Abuse limits.
POST /nearby/fill,/nearby/search-moreand/search-morecarryRateLimiter.HarvestLimit: 90 paid reservations per hour per account, 120 per device (theX-Device-IDheader, when sent) and 1500 per IP, a coarse backstop sized for carrier-grade NAT. The count is taken just before a paid reservation, so requests served from the ledger are free; a limiter outage refuses the paid call only. Only a signed first-party worker bypasses it; an internal address skips the IP ceiling but not the account and device ones. Production refuses to boot with the routes on and no trusted-proxy policy. There is no per-country cap.
Worker handoff and outbox. location:burst_seed carries a harvest.Handoff: the
fetched places, the cell, key and area, never the query. The worker takes the same
lock, skips a tile that is already fresh, writes the places and stamps the tile. It
never calls the provider, so a retry cannot buy twice; an undecodable payload is
SkipRetry, a write error retries up to 12 times. Every paid answer is also kept in
the Redis outbox until it is committed (the stored form, also used for the queued task, has no user
and its origin is the centre of the cell, never the request point; the text key is kept), and cron:harvest_outbox_sweep (every 5 min)
replays entries older than 3 minutes through the same path, so a crash, a deploy or an
archived task (tomoda_locations_harvest_handoffs_archived_total) loses nothing. The server's shutdown drains in-flight
continuations (StopHarvest) and counts any still running as
tomoda_locations_harvest_continuations_lost_total.
Tiles are H3 cells. The ledger keys on H3 hexagons computed inside Postgres
(harvest.PGGrid over the h3-pg functions behind the harvest.CellGrid interface:
cell for a point, cell centre, k-ring neighbours, parent). The cell id is the
cell column and the hex string in the Redis harvest:* keys. A call is centred
on the cell centre, so every user in a cell shares one call and one cached answer,
and the focus and distance gates run relative to that centre with a radius that
reaches the cell's far edge. The zoom is chosen for that reach (an estimate from
the cell size, not re-measured against the provider):
| Use | H3 resolution | Cell edge | Call reach | Zoom |
|---|---|---|---|---|
| picker fills (any ring) | 9 | ~175 m | 200 m | 18 |
| nearby search-more | 8 | ~460 m | 530 m | 17 |
| area search-more cache | 7 | ~1.2 km | 1400 m | 17 |
A wider picker ring is covered by reaching several res 9 cells, not by a coarser tile.
locations.cell9 (a trigger-maintained res 9 cell, btree-indexed) lets the picker's
nearby read narrow by cell before the exact distance test, and that read selects only the columns a candidate
carries.
Reach. A request fills the user's own cell and every other cell of its disk the ring
covers by 20% or more (area, computed in SQL), seven at most (the k=1 disk, which holds the widest 500 m picker ring), in parallel. Each
is a full fill with its own lock and reservation under the user's fill budget, and
tiles stay tied to their cell centre. fill_needed is true when any of those
cells is cold, answered by one SQL statement (PGGrid.Reach) that returns the
disk's cells with their coverage, centre and tile freshness (a warm verdict is also
remembered for a few seconds). The list read and the reach check run side by side.
One request holds at most RequestCalls provider calls in flight across its cells.
The cell centre only positions the provider call.
Queries. A sweep (no chip) runs places, restaurants, cafes, stores:
generalist queries that barely overlap, measured on a Taipei field test. A batch
that found under 20% new places marks its tiles saturated, trusted for twice the
TTL. Paging is decided per query: page 2 and 3 of a query are asked only when its
previous page came back full (about 20 places), a dense sweep pages in the
background and a chip pages when the client asks.
Chips. Seven chips (restaurants, cafes, bars, shops, attractions,
entertainment, hotels; BurstIntent) each hold an explicit set of taxonomy
leaves in intentCategories. The membership is explicit because the taxonomy's
buckets overlap: a Cafe and a Pub both carry food. A unit test keeps chips
disjoint and every leaf real. Each chip runs its own queries (bars runs bars and
nightclubs) under the same per-query tiles as the sweep. intent_counts lets the client order chips
by what is actually nearby.
- Typed categories.
intentFromTextruns typed text through the multilingual taxonomy parser: a parsed leaf first, then a parsed tag (tagIntent). Text with leftover words ("Blue Bottle coffee") is a name search and resolves to nothing. - Search more. When typing settles on weak results, any place search runs one
paid text lookup, whose rows join the same list (
POST /locations/search-more, Paid fallback). The catalog is read first: a listed place whose name, or any localized name, equals the text, is a typo of it, or contains all its words answers the search without a call (fresh). Text that normalises to nothing is refused before any spend.Engine.SearchMoreruns the exact text through Serper once and mints the matches, caching their ids inlocation_seed_textsby (scope, cell, normalized text): 30 days for a hit, 7 days for a proven miss (an empty list is cached only when the call answered 200), so the same search in that area is free, even after a Redis flush and whatever language asked. A concurrent identical search waits for the first one's answer. The scope fixes the cell, the call zoom and the answer radius:
| Scope | Cell | Call | Answer kept within |
|---|---|---|---|
nearby |
res 8, centre of the cell | zoom 16 | 1 km |
area |
res 7, centre of the cell | zoom burst_zoom_area (14) |
8 km |
global |
g:{country hint}, no place |
no ll at all, only gl |
100 km around the top result |
A global answer stamps one area, resolved at its top result, on every kept place; with
no country known nothing is minted and nothing is cached. Its spend rows record no cell
and its stored copy carries no origin. Explicit lookups (the keyboard's Search key on weak results, a chip, a widened ring) draw on the reserved pool with no
per-user daily cap; automatic lookups ("auto": true) draw on the per-user automatic
pool (burst_user_daily_auto). A refusal returns the free matches with the status.
The hourly per-area allowance (burst_search_more_cell_per_hour, default 120) is a
spend gate: it counts only calls about to be paid, never a cached answer.
- Depth on demand. A chip pages deeper (page 2, then 3) as the user reaches
the end of its list. pageWorth allows the next page only while the previous
page added at least 5 new places; otherwise the fill returns exhausted with no
provider call.
- Dense cells page in the background. The client never pages a sweep. When a
sweep page returns at least 30 places and at least 70% are new to the catalog
(SweepDeeper), the continuation fetches the next page (up to 3) under the
pages trigger, one budgeted page at a time. Busy blocks fill out without the
user waiting, and a cell we already know well stops after one page.
Metrics.
| Metric | Labels | Reads as |
|---|---|---|
tomoda_locations_provider_spend_credits_total |
trigger, outcome = ok or the failure class |
Serper credits by what triggered them |
tomoda_locations_burst_requests_total |
op = fill / search_more, status |
Outcomes; fresh vs filled is the tile hit rate, limited / paused show ceilings and outages |
tomoda_locations_search_requests_total |
surface, fallback |
Typed place searches and whether the free results left room for a paid lookup |
tomoda_locations_search_more_total |
surface, trigger = auto / explicit, scope, outcome = hit / miss / fresh / refused |
Paid lookups; hit / (hit + miss) is the weak rule's precision |
tomoda_locations_search_more_picked_total |
surface |
Picks from a paid lookup's results |
tomoda_locations_search_duration_seconds |
tier = free / paid |
Search latency |
tomoda_locations_burst_places_total |
kind = returned / new / core / secondary / service |
returned and new count stored places, so new / returned is coverage novelty and falls as an area fills in; core / secondary / service split every provider place by focus |
tomoda_locations_provider_credits_estimated_total |
none | Responses that omitted credits; a rise means the format changed |
tomoda_locations_harvest_error_tiles_total |
none | Cells put to rest after repeated failures |
tomoda_locations_harvest_continuations_lost_total |
none | Paid continuations abandoned at shutdown |
tomoda_locations_harvest_panics_total |
none | Recovered panics in harvest goroutines; alert on any |
tomoda_locations_harvest_spend_log_failures_total |
none | Paid calls whose spend row was not written; alert on any |
tomoda_locations_harvest_breaker_state |
none | Gauge: 0 closed, 1 probing, 2 open |
tomoda_locations_harvest_budget_used_ratio |
trigger (global too) |
Today's credits over the ceiling; alert near 1 |
tomoda_locations_harvest_handoffs_archived_total |
none | Worker tasks that exhausted their retries; the answer stays in the outbox |
tomoda_locations_harvest_outbox_pending / _replays_total{outcome} |
none / ok, failed |
Stored answers awaiting commit and the sweeper's results |
Alert on provider_credits_estimated_total too: the provider stopped reporting costs.
Spend log retention. Raw location_provider_spend rows (user, a cell coarsened to its res 7 parent,
about 1.2 km, on user-triggered rows, a text hash) are kept 30 days. cron:provider_spend_purge
(daily) then folds older rows into location_provider_spend_daily per day, provider,
trigger, endpoint and outcome, which carries no user, cell or text hash and is kept 400
days. Expired tiles and texts are purged too, and so are settled tiles with no expiry once
older than twice the tile TTL. Ledger rows, the spend log and the daily totals carry the
provider's name.
Save-time quality. harvestCandidate drops permanently closed places, places
with no provider id, and strays beyond max(1 km, 3x radius). Provider types map to
taxonomy leaves through harvestTypeCategory, then an exact leaf name, then the
taxonomy parser; unknown types become Other Location. Provider ratings, hours,
and website go into signals. They never touch
average_rating / total_ratings, which hold Tomoda ratings only. enrichment_at
is stamped so the self-heal sweep leaves the row alone until promotion.
Photos. The thumbnail URL is not stored. After a harvest or a search-more mint
ingests its rows, one location:rehost_photos task carries the (location id,
thumbnail URL) pairs. The worker (RehostPlacePhotos) loads each row, skips one that
already has a photo, and otherwise reuses rehostPlacePhoto (see
Place photos). Every download counts against the global anomaly ceiling
burst:photos:{yyyymmdd} (burst_photo_daily_ceiling, default 50000), which is not a
throttle. Hitting it logs a warning, increments
tomoda_locations_photo_rehost_total{outcome="ceiling"} (alert on it) and drops the rest of the task. Failures are dropped too, and the place keeps its
category cover.
Config (ResolutionConfig): enable_burst_seed (off by default),
burst_tile_ttl_days, burst_daily_cap, burst_uncapped (escape hatch, never in production), burst_user_daily_fills,
burst_user_daily_auto, burst_share_* (fill, explicit, auto, pages, enrich),
burst_slot_limit, burst_photo_daily_ceiling, burst_fill_budget_ms,
burst_zoom_150 / _300 / _500, burst_zoom_area, burst_search_more_cell_per_hour. See Configuration.
Environment defaults¶
| Env | burst_daily_cap |
enable_burst_seed |
|---|---|---|
| local | 200 | off |
| dev | 1000 | on |
| prod | 20000 (about $20 a day) | off, turned on deliberately |
The values live in config.<env>.yaml; no devops env override sets them.
Local development¶
- Tests and the default local run use the fake provider and Photon only: harvesting is off
(
enable_burst_seed: false) and no test calls Serper. task devpulls the real Serper key, soconfig.local.yamlcarriesburst_daily_cap: 200. Enrichment calls with that key are metered and stop at 200 credits a day.- To try harvesting locally, set
enable_burst_seed: true(orENABLE_BURST_SEED=true). It uses the same real key and the same 200-credit ceiling. Raiseburst_daily_caponly on purpose. - A cap of 0 means no paid calls, so a missing setting never becomes an open wallet.
burst_uncapped: true(BURST_UNCAPPED=true) lifts it for a test harness and is rejected in production.