Skip to content

Location search and ranking

Part of the Locations domain: how search, autocomplete and nearby results are recalled, ranked and displayed.

Search & ranking

Every place-search surface (the main search bar, the place/area picker, the typeahead, the event and plan pickers) funnels through one core, searchCore in location_search.go. Discovery, pickers, and the typeahead only shape and enrich the response, they never re-rank. The core is:

  • Taxonomy-aware — taxonomy.ParseQuery runs inside the core, splitting the query into (category, tags, free-text), so the typeahead is as category-aware as the main search. It is a multilingual dictionary matcher (28-language aliases); the semantic layer augments it, it is not replaced.
  • Area-aware (near vs a named area) — detectArea (location_area.go) spots a gazetteer area in the query, by connective ("cafes in Taipei") or by a trailing place name ("Starbucks Shibuya"), resolves it to a centroid, and recenters the search there, searching only the remaining place text within a radius scaled by the area's prominence. Otherwise the viewer's location biases it; with neither, the search is global. Area lookups are Redis-cached, and a token span is only treated as an area when a settlement strongly resolves for it, so "Shibuya 109" the building still searches as a whole.
  • Source-selectable — the source param (all · tomoda · raw · places) picks which providers to draw from. all (default) is our catalog plus the Photon fallback; tomoda is catalog-only (the Find-tab browse); raw is Photon only. places (Google) is never queried on the read path and is reserved for a future paid fallback. Codenames keep third-party product names off the wire.

Under that, the same two-stage split: PGroonga provides multilingual recall, a Go composite scorer provides the ordering.

  • Recall — SearchLexical matches the trigger-maintained search_text column (base name plus every localized name, lowercased and accent-folded) via a PGroonga index normalized with NormalizerNFKC150 (width/kana folding), so a query in any language, script, or width (東京, 서울, café, Starbucks) hits. It uses &@~ with pgroonga_query_escape, so a multi-word query ANDs its tokens ("Starbucks Shibuya" finds the branch whose name has "starbucks" and whose city has "shibuya") and is injection-safe. Category and tag filters are pushed into the query.
  • Typo-tolerant recall (cost-gated) — when the exact pass returns fewer than the limit, SearchLexicalFuzzy runs a second PGroonga pass with pgroonga_condition(..., fuzzy_max_distance_ratio => 0.3), an edit-distance budget (a ratio of term length) that recalls misspellings ("stabucks", "shibya") across English, CJK, and romanized scripts. Results merge into the lexical pool by id, so an exact hit is never displaced. The common exact-name case fills the page from the first pass and never runs the fuzzy one.
  • Semantic recall (cost-gated) — when lexical comes up short (a conceptual query like "quiet place to work", a synonym, a typo), the query is embedded (Qwen3-Embedding-0.6B via a TEI server, Redis-cached) and SearchSemantic pulls the nearest rows over the embedding halfvec, merged into the lexical pool. The common exact-name case fills the page from lexical and never pays for the embed. Vectors are computed on write by the equalizer (embedRow), gated by shouldEmbed (see Embedding gate), so the cold long tail stays lexical-only. Search runs on the first 512 components of the vector (see Vector index). Disabled → lexical-only.
  • Ranking — RankCandidates (ranking/scoring.go) scores each candidate with a name-dominant blend of six components, each normalized to a clean 0..1 so a weight reads as a genuine share of the decision (weightName 0.55 means an exact name match is 55% of the top score). Weights are tuned offline against the golden set (see below), not hand-guessed:

    Component Weight Normalized 0..1 shape
    Name relevance 0.55 Graded match tier across the base name and every localized name: exact 1.0, then whole-string prefix, word prefix, substring, token overlap. Each tier interpolates within a band by how much of the name the query covers, so two prefix matches of different depth do not tie, and the bands never cross (the weakest exact still beats the strongest prefix).
    Proximity 0.12 Gaussian decay exp(-d² / 2σ²) to the effective center, σ = 2 km (relevance halves at 2 km). Dropped (1.0) for unbiased queries.
    Semantic 0.11 Vector cosine similarity, so a meaning-match with no name hit still surfaces.
    Source trust 0.10 1.0 for curated or engaged catalog rows, 0.75 for harvested rows nobody has engaged with yet, 0.6 for Photon.
    Prominence 0.08 Gazetteer prominence (population, heritage), clamped 0..1.
    Popularity 0.04 log1p(saves) capped at a 100-save saturation, blended 0.6 with external rating 0.4 (rating counts only past 5 ratings).

A recall hit that matched only via non-name text (a city, a tag) floors its name score at 0.35 (lexicalFloor) so it is never zeroed out. So an exact-name match a few hundred metres away outranks a nearer place that merely contains the term, and the city "Ottawa" beats "Ottawa Street".

Personalization (viewer-aware re-rank). When Search receives a UserID (SearchOptions.UserID, set by discovery search), it layers a personalization pass on the ranked set: personalization.Service builds a cached UserContext (category affinities from user_affinity, saved/dismissed places, interaction edges), and ranking.SortCandidates re-sorts after a per-candidate delta (ranking.PersonalAdjust in the spine) — category affinity tilts the order, saved places rise, dismissed places sink. It is a no-op for an anonymous viewer or when the personalizer is nil, so the base ordering (and the golden eval) are unchanged on those paths. Weights are provisional hand-set values (defaultPersonalWeights), to be tuned on selection logs. The pass runs on the returned set today; moving it before the result cap is a later refinement.

The ranking spine is shared, entity-agnostic infrastructure

The generic engine lives in internal/platform/ranking: the schema-keyed weighted blend (Blend, Vector, Weights), the text-match primitives (NameTier, NormalizeForMatch), the geo/popularity math (GaussianProximity, LogSaturate), the NDCG/MRR metrics, and the offline evaluator + coordinate-ascent optimizer (EvalSet, Tune). It imports no domain models. The location/ranking package is the location adapter: it owns the six-signal schema, the weights, the LocationCandidate feature extractor, and the DB-over-Photon tiebreak, delegating the rest to the spine. A new entity (event, moment, user) ranks by supplying its own extractor + a weight schema, not by copying this code.

PGroonga supplies recall, not ranking: its relevance score reflects term frequency, so on its own it cannot tell the city "Tokyo" from a "…Tokyo…" coffee shop. The name-tier scorer is what makes the exact match win.

The Photon fallback carries a gentle, config-tunable location bias so nearby results nudge up without overwhelming an exact-name match far away. platform/photon passes the viewer's coords as Photon's lat/lon/zoom bias with a scale factor, both tunable per environment via PHOTON_ZOOM and PHOTON_BIAS_SCALE (config.Photon.Zoom / config.Photon.BiasScale, with defaults applied in config/config.go). This shapes Photon's own ordering; the Go composite scorer still re-ranks the merged pool, so the bias is a recall nudge, not the final say.

Embedding gate — which rows carry a vector

Embedding the whole catalog does not scale: a 100M+ row POI backbone would need 100M+ vectors and an ANN index over all of them, at real cost, for rows almost no conceptual query reaches. So the equalizer embeds a row only when it earns one. shouldEmbed (location_embedding.go) embeds a row when any of these hold; signals and metadata blobs never qualify a row:

Signal Rule
Prominent prominence >= embedProminenceFloor (0.35), the log-scaled prior where a place is notable enough (a few hundred sitelinks) to be worth a conceptual match. GeoNames settlements (geonames_seed) use the higher embedCityProminenceFloor (0.57, population about 15,000), so small towns do not take a vector. Countries and admin areas are not locations rows and are not gated
Curated / prestige provenance is unesco_seed or wikidata_seed (hand-curated or notable catalogs). Natural features (natural_seed) are not curated: they embed only on prominence or engagement
Engagement-promoted provenance is promoted (a backbone row upgraded on engagement, see below)
Engaged save_count, moment_count, total_visits, review_count, or total_ratings > 0

A harvested row never embeds: prominence and engagement counters do not qualify it, and promotion flips it to promoted, which does. Only the vector is gated: search_text (lexical) and localizations are maintained for every row by the write-time trigger and the equalizer, so the long tail stays fully findable lexically, spatially, and by category. It just does not answer a pure meaning query until it earns a vector.

Demand-driven promotion. A cold harvested backbone row stays cheap (base data + Photon, no paid enrichment, no vector) until a user meaningfully engages with it, then it earns full enrichment. LocationService.PromoteOnEngagement is the one coherent promote path: via LocationStore.PromoteHarvested it clears the row's harvested provenance (sets it to promoted, so isHarvested is false) and nulls enrichment_at in the same update, then enqueues enrichment with reason EnrichReasonEngaged. Nulling enrichment_at matters: a harvested row's create-time Photon pass stamps it, and EnrichLocation's 7-day min-age guard would otherwise skip the promotion outright. With it nil the guard passes, so the enricher does the rest on the promoted row: paid commercial enrichment fires (the row is rich-detail and no longer harvested) and embedRow fires (shouldEmbed is true for promoted). The hasFreshGoogleFields guard still keeps a redundant re-promote idempotent. Promotion spends paid enrichment and embeddings only on the active subset users touch, never the 100M backbone.

Trigger on meaningful engagement only (select-from-search, open-detail, save, visit, check-in), never a mere search-result impression, or one search would promote every backbone row it returned. The engagement write paths live in other domains, so PromoteOnEngagement is the location-side entry they call. Every committed use promotes, and a pick or a resolve never does:

Committed use Where it promotes
Moment moment.Handler, after the moment is stored
Check-in presence.checkinService.CreateCheckin
Save maps.Service.SaveLocation, on a first-time save only
Plan add plan.itemService, when a place lands on a plan or in the stash
Event venue event.eventService, on create, a venue change or an itinerary item

Travel-log entries are written only by moments and check-ins, so they add no path of their own. The save and event paths cannot hold the location service (it depends on the personalization service, which depends on the maps service), so they enqueue a location:promote task through the maps.EngagementPromoter port, which the worker runs; it is deduped per place for an hour.

Per-city highlight promotion. Demand-driven promotion is reactive: a place only earns enrichment once a user touches it, so a mid-size or small city's marquee sights stay cold until someone happens to engage. LocationService.PromoteCityHighlights closes that gap at load time. It reuses the same promotion mechanic, applied in bulk to each city's top attractions: for every city (grouped by the city_id self-FK the seed loader's set-based nearest-city pass assigns, so it works even before region_id resolves) it takes the top-N attractions by prominence and promotes those not already promoted. TopCityHighlightCandidates (queries.sql) is the set-based scan, one window function partitioned by city_id served by the partial idx_locations_city_highlights index (so it is an index scan, not a per-city seq scan). The candidate set is any active attraction with a resolved city_id, independent of provenance, so a curated Wikidata/prestige seed promotes just like a harvested row (a curated-only load, with no harvested rows, still lights up each city's marquee sights). PromoteToHighlight flips the chosen rows to promoted regardless of source. The allowlist is derived from the taxonomy, not hardcoded: every leaf under the attraction buckets (culture, outdoors, entertainment, music, religion) plus destination shopping/food leaves (Market, Food Hall, Mall, …), so the generic long tail (restaurants, cafes, bars, grocery/convenience/supermarket, services, transit, accommodation) is never promoted. The ranking window spans every provenance including already-promoted rows, so the top-N set is stable across re-runs (an already-eager row keeps its rank rather than freeing a slot for the next-deepest candidate), and PromoteToHighlight skips anything already promoted, so re-runs do not thrash the enrichment queue. It runs from the catalog seeder's load path: catalogseeder load --city-highlights N runs it after the dataset is loaded (the DevOps load pipeline invokes it once the dataset is in place).

Vector index, first 512 components

embedding stores the full 1024-dimension vector, but the ANN index and every query use only its first 512 components (the model is trained so a leading slice stays meaningful). idx_locations_embedding_hnsw is an HNSW over subvector(embedding, 1, 512)::halfvec(512) with halfvec_cosine_ops. SearchSemantic orders by exactly that expression (subvector(embedding, 1, 512)::halfvec(512) <=> $1) so the planner uses the index, and the query vector is cut to 512 client-side (embeddings.Truncate; cosine distance does not depend on length, so no renormalization). Moving search to the full width later is an index rebuild, not a re-embed. HNSW skips NULL embeddings, so the async-populated column and the gated long tail are both fine.

Next lever: DiskANN, only if the gated set outgrows HNSW

The embedding gate keeps the vector set to high-value + engaged rows, which HNSW handles well. If that set itself grows past what an in-memory HNSW serves comfortably, the next step is a disk-backed ANN (pgvectorscale / DiskANN). It is deliberately not adopted now: it is a new extension.

Tuning the weights — golden set + offline optimizer

The weights are validated offline, not hand-guessed. A committed golden set (internal/services/location/testdata/ranking_golden.json) is a set of hand-authored, network-free cases: a query, the geo-bias flag, a candidate pool carrying each candidate's raw ranking-signal inputs, and a graded relevance label per candidate (0 irrelevant .. 3 perfect). It covers multilingual queries, typo-recall hits, proximity-vs-name tension, and semantic-only hits. Many cases are deliberately adversarial (a mega-popular loose match against an exact-name row, a much-nearer street against a distant city), so a naive weighting orders them wrong and the objective has a real gradient. The shipped weights score NDCG@5 ~0.97 / MRR ~0.94 on the set, not a degenerate 1.0.

The shipped weights are a provisional prior

They are tuned to this hand-authored bootstrap set, not yet validated on real traffic. Treat them as a reasonable starting point, and re-tune with cmd/rankertune once the selection logs (below) have grown the golden set from real queries.

  • Regression guard — a Go test (location_ranking_test.go) scores the live ranker's ordering against the labels with NDCG@5 and MRR and asserts both stay above a floor (baseline minus ~0.03), so a weight or scorer change that hurts ranking fails CI. A companion test perturbs each weight by ±0.1 and asserts NDCG measurably moves, proving the set is discriminating (not a flat plateau where any weighting ties).
  • Offline optimizer — cmd/rankertune reads the golden set and searches (name-dominant coordinate ascent with seeded random restarts, every secondary weight floored at 0.03 so no signal is zeroed out) for the weight vector that maximizes NDCG, printing before/after metrics and the tuned weights. Deterministic, no network, no database. Run it, and if it beats the shipped weights, update the consts in location_ranking.go (round to clean shares; the raw output is false precision on a small synthetic set):
cd backend && go run ./cmd/rankertune          # NDCG@5 over the default golden set
go run ./cmd/rankertune -k 10 -golden <path>   # different cutoff / label file
  • Guardrailed traffic retune — cmd/rankertune -harvest closes the loop from real selections. It reads the durable label sink (ranking_impression joined to ranking_selection over a window), turns each selection into a labeled case reusing the recorded feature sub-scores (no re-scoring), corrects for click position bias with inverse-propensity weighting (1/rank^eta, default eta = 1.0, so a pick at rank 4 counts 4x a rank-1 pick that may just be position), then runs the same coordinate-ascent optimizer on a deterministic train split (hash of ranking_query_id, no randomness). It gates the proposal behind three guardrails and only recommends applying when all pass: at least 200 harvested labels, a held-out NDCG lift over 0.01, and no regression past 0.005 on the hand golden set (a permanent regression floor). It prints the shipped-vs-proposed RANK_WEIGHT_* lines and an APPLY / HOLD verdict; it never writes config or a DB row. Adopting an approved vector stays human-gated, but no longer needs a redeploy: apply it to the active ranking_weights row and the ranker hot-refreshes it in (see Config hot-refresh below).
cd backend && go run ./cmd/rankertune -harvest            # last 30 days
go run ./cmd/rankertune -harvest -days 14 -eta 1.5        # shorter window, steeper bias correction

The harvest/IPW/guardrail logic is a set of pure functions over []models.RankingLabeledCase in location_ranking_harvest.go, unit-tested without Postgres.

Config hot-refresh — applying weights without a redeploy

The composite weights resolve active ranking_weights DB row > env config (RANK_WEIGHT_*) > shipped default. The ranking_weights table holds every applied vector with exactly one active at a time, enforced by a partial unique index (... ON ranking_weights (active) WHERE active).

  • Boot — NewLocationService seeds the env-config vector (so the wire graph builds without a live pool), then StartWeightRefresh does one immediate best-effort DB read to apply DB precedence before the first search. An empty table keeps the config weights byte-for-byte; a read failure keeps the current vector and logs. The read never blocks startup.
  • Hot-refresh — the API ranker re-reads the active row every RANK_WEIGHTS_REFRESH_INTERVAL (default 2m, 0 disables) and atomically swaps it into the live blend, held behind an atomic.Pointer[rankWeights] so concurrent searches always read a consistent set. A failed refresh keeps the current weights. The loop runs on the API pods only, cancelled on shutdown.
  • Apply: POST /backstage/locations/ranking-weights (gated by the fresh backstage.catalog.action.ranking Tomoda accounts capability) takes the six weights plus an optional note, validates each is in [0,1] and not all-zero at the boundary, persists it as the new active row (deactivating the prior one in one transaction), and swaps it in immediately so the change takes effect without waiting for the next refresh tick. This is how an approved cmd/rankertune -harvest recommendation is adopted.

Ranking observability — closing the loop

Two structured logs turn real search traffic into training data for cmd/rankertune without any schema change (location_ranking_log.go). Harvest both from Loki/Grafana and join on ranking_query_id to build labelled cases.

Event (app_event) When Sampling Fields
location_search_ranked Every ranked search, in searchCore 5% head sample; ≤10 candidates per event ranking_query_id, query, biased, returned, and a candidates array of {rank, location_id, source, score, f_name, f_proximity, f_popularity, f_prominence, f_source, f_semantic} (the normalized feature vector the blend combined)
location_selected A user resolves a result, in Handler.Resolve Unsampled (each is a label) ranking_query_id, query, location_id, selected_rank

The selection event reads its q, rank, and ranking_query_id from optional query params the client echoes from the search that produced the candidate; all are best-effort, so a selection still logs (with an empty correlation id) when the client omits them.

Localized display

Results are localized to the viewer's language, which the search core resolves from ?lang= → Accept-Language → en. Two shapes:

  • The autocomplete/detail payload (LocationCandidate) ships the localizations map (trimmed to the language's fallback chain), and the client picks the field via localizedName / localizedField.
  • The main search payload (SearchLocation) omits the map to stay small, so its name / city are localized server-side via location.LocalizedField before serializing.

Because the enrichment equalizer denormalizes 28-language name + city + country onto each row, display reads from the row's own localizations; the city_id geo-FK is for relational grouping/filtering, not needed for localized display.

Free search never spends. When it leaves the typed place unanswered, the search runs one lookup on its own once typing settles (harvest text path); its places join the same list, with no separate tier. A surface with no bias point looks globally.

Weak rule. AssessFallback (location_fallback.go) says fallback: "eligible" when none of the top 5 ranked results answers the text a name (base or any localization) at word-prefix tier or better (NameTier at least 0.58), or every query word prefixing a name word. It is never eligible for a pure category or tag query, an address-shaped query, a settlement-only kinds, or text under the minimum length: 2 characters in a dense script (Han, kana, Hangul, Thai, Lao, Khmer, Myanmar), 3 otherwise. The discovery search uses the same gates with its own strong-match set.

Server checks. Every paid request re-reads the catalog and refuses a text a listed place already answers. An automatic request must also pass the length and address gates.

Ranking. A nearby lookup lists nearest first; area and global lookups rank by the one composite ranker over paid and catalog rows together.

Language. - Free recall matches any localization, because search_text folds every localized name and the PGroonga index uses NormalizerNFKC150. Integration tests cover ja, zh-Hans, zh-Hant, ko, th, ar, he, hi, ru and el. - NormalizeForMatch strips only Latin, Greek and Cyrillic diacritics and Arabic and Hebrew points. Kana voicing marks and Thai or Indic vowel and tone marks stay, so ガ and カ or two different Thai words never collapse into one match key or one cached lookup. - The provider receives hl as the base language (ja-JP becomes ja; zh-CN, zh-TW, pt-BR and pt-PT stay) and gl as the resolved area's country (the device country hint for a global lookup). - A harvested place stores the provider's name under the request language's localization key (zh-CN becomes zh-Hans). A later lookup in another language that returns the same place unions its name in, so each language pays at most once per area and text.