Skip to content

Location resolve and dedup

Part of the Locations domain: resolving a picked place into a Location and how duplicates are found, merged and kept apart.

Resolve flow

There is one entry point: Resolve. It walks the lookup ladder and persists a row from the first provider that knows the place; if neither does, it returns (nil, nil) and no row is written.

flowchart TD
  Req["ResolveRequest"] --> P1{Provider + ProviderID set?}
  P1 -- yes --> Cache["Redis lookup<br/>location:provider:{provider}:{id}"]
  Cache -- hit --> Return["return Location"]
  Cache -- miss --> DBProv["DB lookup by<br/>(provider, provider_id)"]
  DBProv -- hit --> Return
  DBProv -- miss --> Spatial
  P1 -- no --> Spatial["Spatial + name fuzzy<br/>(50m radius)"]
  Spatial -- hit --> Return
  Spatial -- miss --> Policy{Provider policy}
  Policy -- "caller hint = google_places" --> Google["Google: DetailsByID or NearbySearch"]
  Policy -- otherwise --> Photon["Photon ReverseGeo"]
  Photon -- match --> CreatePhoton["Create row<br/>provider=photon"]
  Photon -- empty --> Nil["return (nil, nil)"]
  Google -- match --> CreateGoogle["Create row<br/>provider=google_places<br/>enrichment_at=now"]
  Google -- empty --> Nil
  CreatePhoton --> Enqueue["Enqueue location:enrich"]
  CreateGoogle --> Return
  Enqueue --> Return
  Nil --> Return

Write paths today:

  • Frontend autocomplete picks call POST /locations/resolve at moment/event submit time (not on the pick itself) to convert a candidate into a concrete location_id. A 404 means the caller should submit without location_id and rely on its off-grid label.
  • Moment/event create handlers keep a defensive Resolve call as a safety net for callers that didn't pre-resolve (map-pin moments with no autocomplete pick, scripted clients). If Photon doesn't know the place, the moment/event keeps its coords on its own row.

Google is consulted on the synchronous path in exactly one case: when the caller hints Provider = ProviderGoogle (they already know it's a Google place). The "Photon hit + rich-detail" case picks up Google rich detail via the background enrichment worker.

Allowlist gate — uncategorizable rows never persist

After Photon returns a match, the mapper (mapOsmCategory in backend/internal/platform/photon/photon_osm_mapping.go) runs the OSM tag through a curated table. The match must produce a non-empty Category from place_taxonomy.json for a row to be created. If the mapper returns blank — because the OSM feature is residential, settlement, utility infrastructure, or otherwise not a Tomoda POI — resolveViaProvider returns (nil, nil) and no row is written. The moment/event handler then falls through to its LocationLabel + own Spatial off-grid path.

Defensive denylist entries in the mapper explicitly block: building:residential / apartments / house / yes / dormitory / garage / industrial / commercial / warehouse, all landuse:residential / commercial / industrial / forest / cemetery, all place:* settlements, all boundary:* administrative, all waterway:* flow features, micro-infrastructure (amenity:vending_machine, bench, toilets, recycling, waste_basket, etc.), and stop-level transit plus street utilities (highway:bus_stop, public_transport:stop_position / platform, railway:platform / subway_entrance / tram_stop, amenity:bicycle_rental / taxi / parking / charging_station / atm). Those last ones are dense in cities and crowd the capture picker, while nobody checks in at a bus stop; real stations (railway:station, public_transport:station) stay. Even if a future per-key fallback gets added, these stay blocked.

Extras-aware disambiguation

The mapper takes properties.extra.tags (Photon's secondary OSM tags) as a third argument and uses them to resolve ambiguous primary tags:

  • amenity:place_of_worship + religion=buddhist/hindu/shinto → Temple (the default falls through to Church)
  • amenity:place_of_worship + religion=muslim → Mosque; + religion=jewish → Synagogue
  • amenity:place_of_worship + building=cathedral → Cathedral
  • amenity:marketplace + produce=* → Farmers Market; + second_hand=yes → Flea Market; default → Market
  • leisure:fitness_centre + sport=yoga → Yoga Studio; + sport=climbing → Climbing Gym; default → Gym
  • amenity:bar + live_music=yes → Music Venue
  • amenity:theatre + theatre:type=comedy → Comedy Club
  • tourism:resort + resort=ski → Ski Resort

The same extras pass feeds extractFacetTagsFromExtras which adds facet tags directly onto the candidate's Tags (cuisine values → italian/japanese/…, diet:vegan=yes → vegan, outdoor_seating=yes → outdoor-seating, wifi=yes → wifi, wheelchair=yes/limited → wheelchair-accessible, etc.). This is free enrichment — Photon-rooted rows now carry rich facets without waiting for the Google enrichment worker.

Dedup story

The "same physical place exists as multiple rows" failure mode is what the service is built to prevent. Without dedup, the travel log fragments, the game engine misses stamps, and the discovery map gets duplicate pins.

Two layers catch most cases at write time. Both are binary by design — there is no confidence score. The two rails below are high-confidence by construction; anything they don't catch goes to the admin queue for human review rather than being auto-merged at a threshold we'd have to tune.

Layer Catches Where
Provider-ID match Same provider returned the same place (provider, provider_id) unique index + Redis cache
Spatial + name similarity Different providers, lat/lng jitter, typo'd or multilingual names FindByNameSimilarity(radius, name) PGroonga &@* recall over the NFKC-folded name, gated in Go by sameNameForMerge — radius is per-category (see below)

The similarity rail is two steps. PGroonga's &@* operator does a loose recall over idx_locations_name_pgroonga (the normalized name index, immutable_unaccent(lower(name)) with NormalizerNFKC150), returning the single best nearby candidate within the radius. Loose recall can pull in a distinct place that merely shares a token, so sameNameForMerge (location_dedup.go) re-checks the candidate in Go before any merge: an exact normalized match, a typo-level match (one edit on a word of five letters or more, or a Damerau-Levenshtein ratio at or above 0.85), or one name's words contained in the other with at most two extra words ("Starbucks" vs "Starbucks Coffee"). It keeps names apart on a clear difference:

  • the words containing digits differ ("Gate 2" vs "Gate 3", "88A" vs "88B", "1F" vs "B1");
  • the names read alike but one word is replaced by an unrelated one that is neither a prefix, a compound nor a typo of it ("Stolperstein dedicated to Rosel Gerson" vs "Lisa Gerson", "Church of St Peter" vs "Church of St Paul").

The funnel adds the categories (sameVenue): the two taxonomy categories must share a bucket, with food, drinks and nightlife counting as one family and entertainment, sports, fitness and wellness as another, so a church never merges with a headstone beside it and a park never with a restaurant. A match by containment also needs the same category. An empty category is compatible with anything. A "Coffee" vs "Cafe" word swap and two different brands sharing "Coffee" both fall below the gate and stay as separate rows for admin review. Tight radius + LIMIT 1 + this gate is what makes the rail high-confidence with no false merges.

A settlement row (City, Town, Neighborhood, Place) is never a fuzzy target for a venue: token containment would otherwise fold "Essen Minster" or "War memorial of Viriville" into the city named in the venue's title. Settlements still dedup by provider id.

Per-category dedup radius

The spatial dedup radius isn't a single flat number. A 50m window is too tight for a 400m park (entries on opposite gates look like different places) and too wide for a city-block-dense restaurant row (next-door cafés get falsely merged). Each category in backend/data/place_taxonomy.json carries an optional dedup_radius_m field; LocationService.dedupRadiusFor(req.Hint) reads it via TaxonomyService.DedupRadiusFor(category). Categories without the field fall back to a 50m flat default.

Category bucket Radius Why
Restaurants, cafés, bars, bakeries, nightclubs, fast food 30 m Dense urban; tight to avoid collapsing next-door spots
Most categories (no override) 50 m Default fallback
Stations, malls, hospitals 100–150 m Big footprints with multiple labeled entries
Stadiums, arenas, universities, parks, beaches, hiking trails, botanical gardens 150–200 m Large polygons; two entries on opposite gates are the same place
Airports, theme parks, zoos, lakes 200–300 m Very large footprints
Mountains, forests, national parks 500 m Fuzzy, range-shaped boundaries

The list is open-ended: add dedup_radius_m to any category that produces noticeable false-merges or false-non-merges in prod. No deploy step beyond shipping the JSON change — the taxonomy reloads at startup.

Search-time same-name collapse

The taxonomy radii above govern the write-time merge. The read-time autocomplete/nearby path has its own pass, dedupeNearbyByName (location_service.go), that folds duplicate same-name rows out of a single result set (DB + Photon candidates combined) before they reach the client. It picks the radius from the pair's kind, classified by candRank (POI = 2, settlement = 1, blank/address = 0; settlementCategories is City, Town, Neighborhood, Place):

Pair Radius Why
Same-name settlements/places (candRank ≤ 1 on both) 25 km A city has several OSM representations (relation + boundary node) 10 km+ apart; a real POI (rank 2) never merges this way
Both blank-category (street segments) 5 km One street returns many same-named OSM segments
Anything else (POIs) 120 m Tight, so next-door same-name venues stay distinct

Survivor selection prefers the more specific kind (POI > settlement > blank), and a DB canonical only breaks a same-kind tie, so a curated POI wins over a Photon duplicate but a gazetteer locality never suppresses a landmark of the same name.

Admin merge — for what auto-dedup misses

When auto-dedup misses (e.g. "Apple Store - Ginza" vs "Apple Ginza": different name, both Photon nodes, past the Levenshtein threshold), Tomoda accounts can fold the duplicate into the survivor via POST /backstage/locations/merge (see the Backstage docs). MergeInto is admin-triggered post-hoc cleanup; it is never called from any automatic path. The two rails above are the only automatic dedup.

MergeInto mechanics

The merge handler runs in one transaction:

  1. Rewrite every cross-reference from source to target. Driven by a private locationFKRewrites manifest in backend/internal/services/location/location_store.go, one row per (table, column) pointing at locations(id).
  2. Cross-link providers — append source's (provider, provider_id) to target's secondary_providers (plus any of source's own secondaries), deduped. Target stays reachable by either id from now on.
  3. Hard-delete source via tx.Unscoped().Delete(). Nothing alive references it any more (step 1 rewrote everything); the forensic trail lives entirely in audit_logs.
  4. Invalidate Redis — provider-pair cache entries on the surviving row are cleared so callers stop hitting stale pointers.

There is no merged_into redirect column. Hard-delete + the audit log is the entire story; stale client-side caches that hold a source id get a clean 404 on next fetch.

Adding a new table with a location_id FK

Every Postgres FK pointing at locations(id) MUST appear in the locationFKRewrites manifest. The store exposes VerifyMergeCoverage(ctx), which queries information_schema and fails if any FK isn't registered. It runs at server startup after Migrate (called from backend/internal/migrate/manual.go), so a missing registration fails the boot rather than silently leaving dangling references after the next admin merge.

When you ship a new model with location_id:

  1. Add {"your_table", "location_id"} to locationFKRewrites in backend/internal/services/location/location_store.go. Keep alphabetical unless a unique constraint forces a specific rewrite order.
  2. If your table has a unique constraint involving location_id (e.g. UNIQUE (user_id, location_id)), the naïve UPDATE collides when a single user has rows at both source and target. Add a pre-rewrite step that folds the source-side row into the target-side row first.
  3. The startup verifier will flag the missing entry on local boot; the same guard prevents a half-wired merge from shipping to prod.

Cost of the fuzzy recall

The ingest funnel's stored-row recall (Neighbors) is spatial first: one query per batch joins the batch's points to the GiST index on coordinates with a box a little wider than the radius and returns the live rows inside it. The exact 150 m distance, the settlement exclusion, sameNameForMerge and the Google-id conflict check then run in Go over those few rows (the conflict check is one more batched query for the survivors). The nearest exact-name match wins. Nothing depends on PGroonga's similarity threshold, which also varied with the index's term statistics.

Measured on 41,648 notable places loaded into a table of 930,000 rows: the lookup costs about 0.15 ms per probe, against 1.5 ms for the earlier PGroonga-then-spatial form, which grew with the number of rows worldwide sharing a name. The deleted_at IS NULL filter is applied to the returned rows, never as a predicate: as one, the planner ANDs the deleted_at index into every probe and scans the whole table.