Location resolve and dedup¶
Part of the Locations domain: resolving a picked place into a Location and how duplicates are found, merged and kept apart.
Resolve flow¶
There is one entry point: Resolve. It walks the lookup ladder and persists a row from the first provider that knows the place; if neither does, it returns (nil, nil) and no row is written.
flowchart TD
Req["ResolveRequest"] --> P1{Provider + ProviderID set?}
P1 -- yes --> Cache["Redis lookup<br/>location:provider:{provider}:{id}"]
Cache -- hit --> Return["return Location"]
Cache -- miss --> DBProv["DB lookup by<br/>(provider, provider_id)"]
DBProv -- hit --> Return
DBProv -- miss --> Spatial
P1 -- no --> Spatial["Spatial + name fuzzy<br/>(50m radius)"]
Spatial -- hit --> Return
Spatial -- miss --> Policy{Provider policy}
Policy -- "caller hint = google_places" --> Google["Google: DetailsByID or NearbySearch"]
Policy -- otherwise --> Photon["Photon ReverseGeo"]
Photon -- match --> CreatePhoton["Create row<br/>provider=photon"]
Photon -- empty --> Nil["return (nil, nil)"]
Google -- match --> CreateGoogle["Create row<br/>provider=google_places<br/>enrichment_at=now"]
Google -- empty --> Nil
CreatePhoton --> Enqueue["Enqueue location:enrich"]
CreateGoogle --> Return
Enqueue --> Return
Nil --> Return
Write paths today:
- Frontend autocomplete picks call
POST /locations/resolveat moment/event submit time (not on the pick itself) to convert a candidate into a concretelocation_id. A 404 means the caller should submit withoutlocation_idand rely on its off-grid label. - Moment/event create handlers keep a defensive
Resolvecall as a safety net for callers that didn't pre-resolve (map-pin moments with no autocomplete pick, scripted clients). If Photon doesn't know the place, the moment/event keeps its coords on its own row.
Google is consulted on the synchronous path in exactly one case: when the caller hints Provider = ProviderGoogle (they already know it's a Google place). The "Photon hit + rich-detail" case picks up Google rich detail via the background enrichment worker.
Allowlist gate — uncategorizable rows never persist¶
After Photon returns a match, the mapper (mapOsmCategory in backend/internal/platform/photon/photon_osm_mapping.go) runs the OSM tag through a curated table. The match must produce a non-empty Category from place_taxonomy.json for a row to be created. If the mapper returns blank — because the OSM feature is residential, settlement, utility infrastructure, or otherwise not a Tomoda POI — resolveViaProvider returns (nil, nil) and no row is written. The moment/event handler then falls through to its LocationLabel + own Spatial off-grid path.
Defensive denylist entries in the mapper explicitly block: building:residential / apartments / house / yes / dormitory / garage / industrial / commercial / warehouse, all landuse:residential / commercial / industrial / forest / cemetery, all place:* settlements, all boundary:* administrative, all waterway:* flow features, micro-infrastructure (amenity:vending_machine, bench, toilets, recycling, waste_basket, etc.), and stop-level transit plus street utilities (highway:bus_stop, public_transport:stop_position / platform, railway:platform / subway_entrance / tram_stop, amenity:bicycle_rental / taxi / parking / charging_station / atm). Those last ones are dense in cities and crowd the capture picker, while nobody checks in at a bus stop; real stations (railway:station, public_transport:station) stay. Even if a future per-key fallback gets added, these stay blocked.
Extras-aware disambiguation¶
The mapper takes properties.extra.tags (Photon's secondary OSM tags) as a third argument and uses them to resolve ambiguous primary tags:
amenity:place_of_worship+religion=buddhist/hindu/shinto→ Temple (the default falls through to Church)amenity:place_of_worship+religion=muslim→ Mosque; +religion=jewish→ Synagogueamenity:place_of_worship+building=cathedral→ Cathedralamenity:marketplace+produce=*→ Farmers Market; +second_hand=yes→ Flea Market; default → Marketleisure:fitness_centre+sport=yoga→ Yoga Studio; +sport=climbing→ Climbing Gym; default → Gymamenity:bar+live_music=yes→ Music Venueamenity:theatre+theatre:type=comedy→ Comedy Clubtourism:resort+resort=ski→ Ski Resort
The same extras pass feeds extractFacetTagsFromExtras which adds facet tags directly onto the candidate's Tags (cuisine values → italian/japanese/…, diet:vegan=yes → vegan, outdoor_seating=yes → outdoor-seating, wifi=yes → wifi, wheelchair=yes/limited → wheelchair-accessible, etc.). This is free enrichment — Photon-rooted rows now carry rich facets without waiting for the Google enrichment worker.
Dedup story¶
The "same physical place exists as multiple rows" failure mode is what the service is built to prevent. Without dedup, the travel log fragments, the game engine misses stamps, and the discovery map gets duplicate pins.
Two layers catch most cases at write time. Both are binary by design — there is no confidence score. The two rails below are high-confidence by construction; anything they don't catch goes to the admin queue for human review rather than being auto-merged at a threshold we'd have to tune.
| Layer | Catches | Where |
|---|---|---|
| Provider-ID match | Same provider returned the same place | (provider, provider_id) unique index + Redis cache |
| Spatial + name similarity | Different providers, lat/lng jitter, typo'd or multilingual names | FindByNameSimilarity(radius, name) PGroonga &@* recall over the NFKC-folded name, gated in Go by sameNameForMerge — radius is per-category (see below) |
The similarity rail is two steps. PGroonga's &@* operator does a loose recall over
idx_locations_name_pgroonga (the normalized name index, immutable_unaccent(lower(name)) with
NormalizerNFKC150), returning the single best nearby candidate within the radius. Loose recall can
pull in a distinct place that merely shares a token, so sameNameForMerge (location_dedup.go)
re-checks the candidate in Go before any merge: an exact normalized match, a typo-level
match (one edit on a word of five letters or more, or a Damerau-Levenshtein ratio at or above
0.85), or one name's words contained in the other with at most two extra words ("Starbucks"
vs "Starbucks Coffee"). It keeps names apart on a clear difference:
- the words containing digits differ ("Gate 2" vs "Gate 3", "88A" vs "88B", "1F" vs "B1");
- the names read alike but one word is replaced by an unrelated one that is neither a prefix, a compound nor a typo of it ("Stolperstein dedicated to Rosel Gerson" vs "Lisa Gerson", "Church of St Peter" vs "Church of St Paul").
The funnel adds the categories (sameVenue): the two taxonomy categories must share a bucket,
with food, drinks and nightlife counting as one family and entertainment, sports, fitness and
wellness as another, so a church never merges with a headstone beside it and a park never with
a restaurant. A match by containment also needs the same category. An empty category is
compatible with anything. A "Coffee" vs "Cafe" word swap and two different
brands sharing "Coffee" both fall below the gate and stay as separate rows for admin review. Tight
radius + LIMIT 1 + this gate is what makes the rail high-confidence with no false merges.
A settlement row (City, Town, Neighborhood, Place) is never a fuzzy target for a venue:
token containment would otherwise fold "Essen Minster" or "War memorial of Viriville" into the
city named in the venue's title. Settlements still dedup by provider id.
Per-category dedup radius¶
The spatial dedup radius isn't a single flat number. A 50m window is too tight for a 400m park (entries on opposite gates look like different places) and too wide for a city-block-dense restaurant row (next-door cafés get falsely merged). Each category in backend/data/place_taxonomy.json carries an optional dedup_radius_m field; LocationService.dedupRadiusFor(req.Hint) reads it via TaxonomyService.DedupRadiusFor(category). Categories without the field fall back to a 50m flat default.
| Category bucket | Radius | Why |
|---|---|---|
| Restaurants, cafés, bars, bakeries, nightclubs, fast food | 30 m | Dense urban; tight to avoid collapsing next-door spots |
| Most categories (no override) | 50 m | Default fallback |
| Stations, malls, hospitals | 100–150 m | Big footprints with multiple labeled entries |
| Stadiums, arenas, universities, parks, beaches, hiking trails, botanical gardens | 150–200 m | Large polygons; two entries on opposite gates are the same place |
| Airports, theme parks, zoos, lakes | 200–300 m | Very large footprints |
| Mountains, forests, national parks | 500 m | Fuzzy, range-shaped boundaries |
The list is open-ended: add dedup_radius_m to any category that produces noticeable false-merges or false-non-merges in prod. No deploy step beyond shipping the JSON change — the taxonomy reloads at startup.
Search-time same-name collapse¶
The taxonomy radii above govern the write-time merge. The read-time
autocomplete/nearby path has its own pass, dedupeNearbyByName
(location_service.go), that folds duplicate same-name rows out of a single
result set (DB + Photon candidates combined) before they reach the client. It
picks the radius from the pair's kind, classified by candRank (POI = 2,
settlement = 1, blank/address = 0; settlementCategories is City, Town,
Neighborhood, Place):
| Pair | Radius | Why |
|---|---|---|
Same-name settlements/places (candRank ≤ 1 on both) |
25 km | A city has several OSM representations (relation + boundary node) 10 km+ apart; a real POI (rank 2) never merges this way |
| Both blank-category (street segments) | 5 km | One street returns many same-named OSM segments |
| Anything else (POIs) | 120 m | Tight, so next-door same-name venues stay distinct |
Survivor selection prefers the more specific kind (POI > settlement > blank), and a DB canonical only breaks a same-kind tie, so a curated POI wins over a Photon duplicate but a gazetteer locality never suppresses a landmark of the same name.
Admin merge — for what auto-dedup misses¶
When auto-dedup misses (e.g. "Apple Store - Ginza" vs "Apple Ginza": different name, both Photon nodes, past the Levenshtein threshold), Tomoda accounts can fold the duplicate into the survivor via POST /backstage/locations/merge (see the Backstage docs). MergeInto is admin-triggered post-hoc cleanup; it is never called from any automatic path. The two rails above are the only automatic dedup.
MergeInto mechanics¶
The merge handler runs in one transaction:
- Rewrite every cross-reference from source to target. Driven by a private
locationFKRewritesmanifest inbackend/internal/services/location/location_store.go, one row per(table, column)pointing atlocations(id). - Cross-link providers — append source's
(provider, provider_id)to target'ssecondary_providers(plus any of source's own secondaries), deduped. Target stays reachable by either id from now on. - Hard-delete source via
tx.Unscoped().Delete(). Nothing alive references it any more (step 1 rewrote everything); the forensic trail lives entirely inaudit_logs. - Invalidate Redis — provider-pair cache entries on the surviving row are cleared so callers stop hitting stale pointers.
There is no merged_into redirect column. Hard-delete + the audit log is the entire story; stale client-side caches that hold a source id get a clean 404 on next fetch.
Adding a new table with a location_id FK¶
Every Postgres FK pointing at locations(id) MUST appear in the locationFKRewrites manifest. The store exposes VerifyMergeCoverage(ctx), which queries information_schema and fails if any FK isn't registered. It runs at server startup after Migrate (called from backend/internal/migrate/manual.go), so a missing registration fails the boot rather than silently leaving dangling references after the next admin merge.
When you ship a new model with location_id:
- Add
{"your_table", "location_id"}tolocationFKRewritesinbackend/internal/services/location/location_store.go. Keep alphabetical unless a unique constraint forces a specific rewrite order. - If your table has a unique constraint involving
location_id(e.g.UNIQUE (user_id, location_id)), the naïveUPDATEcollides when a single user has rows at both source and target. Add a pre-rewrite step that folds the source-side row into the target-side row first. - The startup verifier will flag the missing entry on local boot; the same guard prevents a half-wired merge from shipping to prod.
Cost of the fuzzy recall¶
The ingest funnel's stored-row recall (Neighbors) is spatial first: one query per batch
joins the batch's points to the GiST index on coordinates with a box a little wider than
the radius and returns the live rows inside it. The exact 150 m distance, the settlement
exclusion, sameNameForMerge and the Google-id conflict check then run in Go over those
few rows (the conflict check is one more batched query for the survivors). The nearest
exact-name match wins. Nothing depends on PGroonga's similarity threshold, which also
varied with the index's term statistics.
Measured on 41,648 notable places loaded into a table of 930,000 rows: the lookup costs
about 0.15 ms per probe, against 1.5 ms for the earlier PGroonga-then-spatial form, which
grew with the number of rows worldwide sharing a name. The deleted_at IS NULL filter is
applied to the returned rows, never as a predicate: as one, the planner ANDs the
deleted_at index into every probe and scans the whole table.