Location search and ranking¶
Part of the Locations domain: how search, autocomplete and nearby results are recalled, ranked and displayed.
Search & ranking¶
Every place-search surface (the main search bar, the place/area picker, the
typeahead, the event and plan pickers) funnels through one core,
searchCore in location_search.go. Discovery, pickers, and the typeahead only
shape and enrich the response, they never re-rank. The core is:
- Taxonomy-aware —
taxonomy.ParseQueryruns inside the core, splitting the query into(category, tags, free-text), so the typeahead is as category-aware as the main search. It is a multilingual dictionary matcher (28-language aliases); the semantic layer augments it, it is not replaced. - Area-aware (near vs a named area) —
detectArea(location_area.go) spots a gazetteer area in the query, by connective ("cafes in Taipei") or by a trailing place name ("Starbucks Shibuya"), resolves it to a centroid, and recenters the search there, searching only the remaining place text within a radius scaled by the area's prominence. Otherwise the viewer's location biases it; with neither, the search is global. Area lookups are Redis-cached, and a token span is only treated as an area when a settlement strongly resolves for it, so "Shibuya 109" the building still searches as a whole. - Source-selectable — the
sourceparam (all·tomoda·raw·places) picks which providers to draw from.all(default) is our catalog plus the Photon fallback;tomodais catalog-only (the Find-tab browse);rawis Photon only.places(Google) is never queried on the read path and is reserved for a future paid fallback. Codenames keep third-party product names off the wire.
Under that, the same two-stage split: PGroonga provides multilingual recall, a Go composite scorer provides the ordering.
- Recall —
SearchLexicalmatches the trigger-maintainedsearch_textcolumn (base name plus every localized name, lowercased and accent-folded) via a PGroonga index normalized withNormalizerNFKC150(width/kana folding), so a query in any language, script, or width (東京,서울,café,Starbucks) hits. It uses&@~withpgroonga_query_escape, so a multi-word query ANDs its tokens ("Starbucks Shibuya" finds the branch whose name has "starbucks" and whose city has "shibuya") and is injection-safe. Category and tag filters are pushed into the query. - Typo-tolerant recall (cost-gated) — when the exact pass returns fewer than
the limit,
SearchLexicalFuzzyruns a second PGroonga pass withpgroonga_condition(..., fuzzy_max_distance_ratio => 0.3), an edit-distance budget (a ratio of term length) that recalls misspellings ("stabucks", "shibya") across English, CJK, and romanized scripts. Results merge into the lexical pool by id, so an exact hit is never displaced. The common exact-name case fills the page from the first pass and never runs the fuzzy one. - Semantic recall (cost-gated) — when lexical comes up short (a conceptual
query like "quiet place to work", a synonym, a typo), the query is embedded
(Qwen3-Embedding-0.6B via a TEI server, Redis-cached) and
SearchSemanticpulls the nearest rows over theembeddinghalfvec, merged into the lexical pool. The common exact-name case fills the page from lexical and never pays for the embed. Vectors are computed on write by the equalizer (embedRow), gated byshouldEmbed(see Embedding gate), so the cold long tail stays lexical-only. Search runs on the first 512 components of the vector (see Vector index). Disabled → lexical-only. -
Ranking —
RankCandidates(ranking/scoring.go) scores each candidate with a name-dominant blend of six components, each normalized to a clean0..1so a weight reads as a genuine share of the decision (weightName 0.55means an exact name match is 55% of the top score). Weights are tuned offline against the golden set (see below), not hand-guessed:Component Weight Normalized 0..1shapeName relevance 0.55Graded match tier across the base name and every localized name: exact 1.0, then whole-string prefix, word prefix, substring, token overlap. Each tier interpolates within a band by how much of the name the query covers, so two prefix matches of different depth do not tie, and the bands never cross (the weakest exact still beats the strongest prefix).Proximity 0.12Gaussian decay exp(-d² / 2σ²)to the effective center,σ = 2 km(relevance halves at 2 km). Dropped (1.0) for unbiased queries.Semantic 0.11Vector cosine similarity, so a meaning-match with no name hit still surfaces. Source trust 0.101.0for curated or engaged catalog rows,0.75for harvested rows nobody has engaged with yet,0.6for Photon.Prominence 0.08Gazetteer prominence (population, heritage), clamped 0..1.Popularity 0.04log1p(saves)capped at a 100-save saturation, blended0.6with external rating0.4(rating counts only past 5 ratings).
A recall hit that matched only via non-name text (a city, a tag) floors its
name score at 0.35 (lexicalFloor) so it is never zeroed out. So an exact-name
match a few hundred metres away outranks a nearer place that merely contains
the term, and the city "Ottawa" beats "Ottawa Street".
Personalization (viewer-aware re-rank). When Search receives a
UserID (SearchOptions.UserID, set by discovery search), it layers a
personalization pass on the ranked set: personalization.Service builds a
cached UserContext (category affinities from user_affinity, saved/dismissed
places, interaction edges), and ranking.SortCandidates re-sorts after a
per-candidate delta (ranking.PersonalAdjust in the spine) — category affinity
tilts the order, saved places rise, dismissed places sink. It is a no-op for an
anonymous viewer or when the personalizer is nil, so the base ordering (and the
golden eval) are unchanged on those paths. Weights are provisional hand-set
values (defaultPersonalWeights), to be tuned on selection logs. The pass runs
on the returned set today; moving it before the result cap is a later refinement.
The ranking spine is shared, entity-agnostic infrastructure
The generic engine lives in internal/platform/ranking: the schema-keyed
weighted blend (Blend, Vector, Weights), the text-match primitives
(NameTier, NormalizeForMatch), the geo/popularity math
(GaussianProximity, LogSaturate), the NDCG/MRR metrics, and the offline
evaluator + coordinate-ascent optimizer (EvalSet, Tune). It imports no
domain models. The location/ranking package is the location adapter: it
owns the six-signal schema, the weights, the LocationCandidate feature
extractor, and the DB-over-Photon tiebreak, delegating the rest to the spine.
A new entity (event, moment, user) ranks by supplying its own extractor + a
weight schema, not by copying this code.
PGroonga supplies recall, not ranking: its relevance score reflects term frequency, so on its own it cannot tell the city "Tokyo" from a "…Tokyo…" coffee shop. The name-tier scorer is what makes the exact match win.
The Photon fallback carries a gentle, config-tunable location bias so nearby
results nudge up without overwhelming an exact-name match far away. platform/photon
passes the viewer's coords as Photon's lat/lon/zoom bias with a scale factor,
both tunable per environment via PHOTON_ZOOM and PHOTON_BIAS_SCALE
(config.Photon.Zoom / config.Photon.BiasScale, with defaults applied in
config/config.go). This shapes Photon's own ordering; the Go composite scorer still
re-ranks the merged pool, so the bias is a recall nudge, not the final say.
Embedding gate — which rows carry a vector¶
Embedding the whole catalog does not scale: a 100M+ row POI backbone would need
100M+ vectors and an ANN index over all of them, at real cost, for rows almost no
conceptual query reaches. So the equalizer embeds a row only when it earns one.
shouldEmbed (location_embedding.go) embeds a row when any of these hold; signals and metadata blobs never qualify a row:
| Signal | Rule |
|---|---|
| Prominent | prominence >= embedProminenceFloor (0.35), the log-scaled prior where a place is notable enough (a few hundred sitelinks) to be worth a conceptual match. GeoNames settlements (geonames_seed) use the higher embedCityProminenceFloor (0.57, population about 15,000), so small towns do not take a vector. Countries and admin areas are not locations rows and are not gated |
| Curated / prestige | provenance is unesco_seed or wikidata_seed (hand-curated or notable catalogs). Natural features (natural_seed) are not curated: they embed only on prominence or engagement |
| Engagement-promoted | provenance is promoted (a backbone row upgraded on engagement, see below) |
| Engaged | save_count, moment_count, total_visits, review_count, or total_ratings > 0 |
A harvested row never embeds: prominence and engagement counters do not
qualify it, and promotion flips it to promoted, which does. Only the vector is gated: search_text (lexical) and localizations
are maintained for every row by the write-time trigger and the equalizer, so the
long tail stays fully findable lexically, spatially, and by category. It just
does not answer a pure meaning query until it earns a vector.
Demand-driven promotion. A cold harvested backbone row stays cheap
(base data + Photon, no paid enrichment, no vector) until a user meaningfully
engages with it, then it earns full enrichment. LocationService.PromoteOnEngagement
is the one coherent promote path: via LocationStore.PromoteHarvested it clears the
row's harvested provenance (sets it to promoted, so isHarvested is false)
and nulls enrichment_at in the same update, then enqueues enrichment with reason
EnrichReasonEngaged. Nulling enrichment_at matters: a harvested row's create-time
Photon pass stamps it, and EnrichLocation's 7-day min-age guard would otherwise
skip the promotion outright. With it nil the guard passes, so the enricher does the
rest on the promoted row: paid commercial enrichment fires (the row is rich-detail
and no longer harvested) and embedRow fires (shouldEmbed is true for promoted). The
hasFreshGoogleFields guard still keeps a redundant re-promote idempotent. Promotion
spends paid enrichment and embeddings only on the active subset users touch, never
the 100M backbone.
Trigger on meaningful engagement only (select-from-search, open-detail, save,
visit, check-in), never a mere search-result impression, or one search would
promote every backbone row it returned. The engagement write paths live in other
domains, so PromoteOnEngagement is the location-side entry they call. Every
committed use promotes, and a pick or a resolve never does:
| Committed use | Where it promotes |
|---|---|
| Moment | moment.Handler, after the moment is stored |
| Check-in | presence.checkinService.CreateCheckin |
| Save | maps.Service.SaveLocation, on a first-time save only |
| Plan add | plan.itemService, when a place lands on a plan or in the stash |
| Event venue | event.eventService, on create, a venue change or an itinerary item |
Travel-log entries are written only by moments and check-ins, so they add no path of
their own. The save and event paths cannot hold the location service (it depends on
the personalization service, which depends on the maps service), so they enqueue a
location:promote task through the maps.EngagementPromoter port, which the worker
runs; it is deduped per place for an hour.
Per-city highlight promotion. Demand-driven promotion is reactive: a place
only earns enrichment once a user touches it, so a mid-size or small city's
marquee sights stay cold until someone happens to engage. LocationService.PromoteCityHighlights
closes that gap at load time. It reuses the same promotion mechanic, applied in
bulk to each city's top attractions: for every city (grouped by the city_id
self-FK the seed loader's set-based nearest-city pass assigns, so it works even
before region_id resolves) it takes the top-N attractions by prominence and
promotes those not already promoted. TopCityHighlightCandidates (queries.sql)
is the set-based scan, one window function partitioned by city_id served by the
partial idx_locations_city_highlights index (so it is an index scan, not a per-city
seq scan). The candidate set is any active attraction with a resolved city_id,
independent of provenance, so a curated Wikidata/prestige seed promotes just like a
harvested row (a curated-only load, with no harvested rows, still lights
up each city's marquee sights). PromoteToHighlight flips the chosen rows to
promoted regardless of source. The allowlist is derived from the taxonomy, not
hardcoded: every leaf under the attraction buckets (culture, outdoors,
entertainment, music, religion) plus destination shopping/food leaves
(Market, Food Hall, Mall, …), so the generic long tail (restaurants, cafes,
bars, grocery/convenience/supermarket, services, transit, accommodation) is never
promoted. The ranking window spans every provenance including already-promoted
rows, so the top-N set is stable across re-runs (an already-eager row keeps its rank
rather than freeing a slot for the next-deepest candidate), and PromoteToHighlight
skips anything already promoted, so re-runs do not thrash the enrichment queue. It
runs from the catalog seeder's load path: catalogseeder load --city-highlights N
runs it after the dataset is loaded (the DevOps load pipeline invokes it once the
dataset is in place).
Vector index, first 512 components¶
embedding stores the full 1024-dimension vector, but the ANN index and every
query use only its first 512 components (the model is trained so a leading slice
stays meaningful). idx_locations_embedding_hnsw is an HNSW over
subvector(embedding, 1, 512)::halfvec(512) with halfvec_cosine_ops.
SearchSemantic orders by exactly that expression
(subvector(embedding, 1, 512)::halfvec(512) <=> $1) so the planner uses the
index, and the query vector is cut to 512 client-side (embeddings.Truncate;
cosine distance does not depend on length, so no renormalization). Moving search
to the full width later is an index rebuild, not a re-embed. HNSW skips NULL
embeddings, so the async-populated column and the gated long tail are both fine.
Next lever: DiskANN, only if the gated set outgrows HNSW
The embedding gate keeps the vector set to high-value + engaged rows, which HNSW handles well. If that set itself grows past what an in-memory HNSW serves comfortably, the next step is a disk-backed ANN (pgvectorscale / DiskANN). It is deliberately not adopted now: it is a new extension.
Tuning the weights — golden set + offline optimizer¶
The weights are validated offline, not hand-guessed. A committed golden set
(internal/services/location/testdata/ranking_golden.json) is a set of
hand-authored, network-free cases: a query, the geo-bias flag, a candidate pool
carrying each candidate's raw ranking-signal inputs, and a graded relevance
label per candidate (0 irrelevant .. 3 perfect). It covers multilingual
queries, typo-recall hits, proximity-vs-name tension, and semantic-only hits.
Many cases are deliberately adversarial (a mega-popular loose match against an
exact-name row, a much-nearer street against a distant city), so a naive
weighting orders them wrong and the objective has a real gradient. The shipped
weights score NDCG@5 ~0.97 / MRR ~0.94 on the set, not a degenerate 1.0.
The shipped weights are a provisional prior
They are tuned to this hand-authored bootstrap set, not yet validated on
real traffic. Treat them as a reasonable starting point, and re-tune with
cmd/rankertune once the selection logs (below) have grown the golden set
from real queries.
- Regression guard — a Go test (
location_ranking_test.go) scores the live ranker's ordering against the labels with NDCG@5 and MRR and asserts both stay above a floor (baseline minus ~0.03), so a weight or scorer change that hurts ranking fails CI. A companion test perturbs each weight by±0.1and asserts NDCG measurably moves, proving the set is discriminating (not a flat plateau where any weighting ties). - Offline optimizer —
cmd/rankertunereads the golden set and searches (name-dominant coordinate ascent with seeded random restarts, every secondary weight floored at0.03so no signal is zeroed out) for the weight vector that maximizes NDCG, printing before/after metrics and the tuned weights. Deterministic, no network, no database. Run it, and if it beats the shipped weights, update the consts inlocation_ranking.go(round to clean shares; the raw output is false precision on a small synthetic set):
cd backend && go run ./cmd/rankertune # NDCG@5 over the default golden set
go run ./cmd/rankertune -k 10 -golden <path> # different cutoff / label file
- Guardrailed traffic retune —
cmd/rankertune -harvestcloses the loop from real selections. It reads the durable label sink (ranking_impressionjoined toranking_selectionover a window), turns each selection into a labeled case reusing the recorded feature sub-scores (no re-scoring), corrects for click position bias with inverse-propensity weighting (1/rank^eta, defaulteta = 1.0, so a pick at rank 4 counts 4x a rank-1 pick that may just be position), then runs the same coordinate-ascent optimizer on a deterministic train split (hash ofranking_query_id, no randomness). It gates the proposal behind three guardrails and only recommends applying when all pass: at least200harvested labels, a held-out NDCG lift over0.01, and no regression past0.005on the hand golden set (a permanent regression floor). It prints the shipped-vs-proposedRANK_WEIGHT_*lines and an APPLY / HOLD verdict; it never writes config or a DB row. Adopting an approved vector stays human-gated, but no longer needs a redeploy: apply it to the activeranking_weightsrow and the ranker hot-refreshes it in (see Config hot-refresh below).
cd backend && go run ./cmd/rankertune -harvest # last 30 days
go run ./cmd/rankertune -harvest -days 14 -eta 1.5 # shorter window, steeper bias correction
The harvest/IPW/guardrail logic is a set of pure functions over
[]models.RankingLabeledCase in location_ranking_harvest.go, unit-tested
without Postgres.
Config hot-refresh — applying weights without a redeploy¶
The composite weights resolve active ranking_weights DB row > env config
(RANK_WEIGHT_*) > shipped default. The ranking_weights table holds every
applied vector with exactly one active at a time, enforced by a partial unique
index (... ON ranking_weights (active) WHERE active).
- Boot —
NewLocationServiceseeds the env-config vector (so the wire graph builds without a live pool), thenStartWeightRefreshdoes one immediate best-effort DB read to apply DB precedence before the first search. An empty table keeps the config weights byte-for-byte; a read failure keeps the current vector and logs. The read never blocks startup. - Hot-refresh — the API ranker re-reads the active row every
RANK_WEIGHTS_REFRESH_INTERVAL(default2m,0disables) and atomically swaps it into the live blend, held behind anatomic.Pointer[rankWeights]so concurrent searches always read a consistent set. A failed refresh keeps the current weights. The loop runs on the API pods only, cancelled on shutdown. - Apply:
POST /backstage/locations/ranking-weights(gated by the freshbackstage.catalog.action.rankingTomoda accounts capability) takes the six weights plus an optionalnote, validates each is in[0,1]and not all-zero at the boundary, persists it as the new active row (deactivating the prior one in one transaction), and swaps it in immediately so the change takes effect without waiting for the next refresh tick. This is how an approvedcmd/rankertune -harvestrecommendation is adopted.
Ranking observability — closing the loop¶
Two structured logs turn real search traffic into training data for
cmd/rankertune without any schema change (location_ranking_log.go). Harvest
both from Loki/Grafana and join on ranking_query_id to build labelled cases.
Event (app_event) |
When | Sampling | Fields |
|---|---|---|---|
location_search_ranked |
Every ranked search, in searchCore |
5% head sample; ≤10 candidates per event | ranking_query_id, query, biased, returned, and a candidates array of {rank, location_id, source, score, f_name, f_proximity, f_popularity, f_prominence, f_source, f_semantic} (the normalized feature vector the blend combined) |
location_selected |
A user resolves a result, in Handler.Resolve |
Unsampled (each is a label) | ranking_query_id, query, location_id, selected_rank |
The selection event reads its q, rank, and ranking_query_id from optional
query params the client echoes from the search that produced the candidate; all
are best-effort, so a selection still logs (with an empty correlation id) when
the client omits them.
Localized display¶
Results are localized to the viewer's language, which the search core resolves
from ?lang= → Accept-Language → en. Two shapes:
- The autocomplete/detail payload (
LocationCandidate) ships the localizations map (trimmed to the language's fallback chain), and the client picks the field vialocalizedName/localizedField. - The main search payload (
SearchLocation) omits the map to stay small, so itsname/cityare localized server-side vialocation.LocalizedFieldbefore serializing.
Because the enrichment equalizer denormalizes 28-language name + city + country
onto each row, display reads from the row's own localizations; the city_id
geo-FK is for relational grouping/filtering, not needed for localized display.
Paid fallback and language¶
Free search never spends. When it leaves the typed place unanswered, the search runs one lookup on its own once typing settles (harvest text path); its places join the same list, with no separate tier. A surface with no bias point looks globally.
Weak rule. AssessFallback (location_fallback.go) says fallback: "eligible" when
none of the top 5 ranked results answers the text a name (base or any localization) at
word-prefix tier or better (NameTier at least 0.58), or every query word prefixing a name
word. It is never eligible for a pure category or tag query, an address-shaped query, a
settlement-only kinds, or text under the minimum length: 2 characters in a dense script (Han,
kana, Hangul, Thai, Lao, Khmer, Myanmar), 3 otherwise. The discovery search uses the same gates with its own strong-match set.
Server checks. Every paid request re-reads the catalog and refuses a text a listed place already answers. An automatic request must also pass the length and address gates.
Ranking. A nearby lookup lists nearest first; area and global lookups rank by the one composite ranker over paid and catalog rows together.
Language.
- Free recall matches any localization, because search_text folds every localized name
and the PGroonga index uses NormalizerNFKC150. Integration tests cover ja, zh-Hans,
zh-Hant, ko, th, ar, he, hi, ru and el.
- NormalizeForMatch strips only Latin, Greek and Cyrillic diacritics and Arabic and Hebrew
points. Kana voicing marks and Thai or Indic vowel and tone marks stay, so ガ and カ
or two different Thai words never collapse into one match key or one cached lookup.
- The provider receives hl as the base language (ja-JP becomes ja; zh-CN, zh-TW,
pt-BR and pt-PT stay) and gl as the resolved area's country (the device country
hint for a global lookup).
- A harvested place stores the provider's name under the request language's localization
key (zh-CN becomes zh-Hans). A later lookup in another language that returns the
same place unions its name in, so each language pays at most once per area and text.