Skip to content

Link Capture & Enrichment

How a shared link (a Reddit post, a Google Maps share, an Instagram reel) becomes a board item with a resolved place and a set of "found online" links, and how that fills in live on the frontend. This is the engineer's map of the capture pipeline. For the small-LLM resolver internals, see Semantic Resolver.

Mental model

A capture is created immediately and filled in by two async passes, so the UI is never blocked on a network round-trip. The client polls until both land. Inside each pass, an item's links are independent and fan out concurrently (bounded at 3); within one link's enrichment, the Serper /search and /maps calls run in parallel, and the place resolve overlaps the thumbnail rehost. Both tasks carry a 3-minute Asynq timeout, and link:enrich holds a per-item Redis lock so a double fire can't double-spend.

POST /itemsshare / paste / note
→
ItemService.Createpersists item + links, status pending_parse
↓
Asynq "low" queue
Pass 1: item:parsedeterministic parse
↓ status → parsed
Pass 2: link:enrichplace + discovered links
→
Ollama / LLMextract + select hooks
Serper/search + /maps
Redisplace enrichment cache
Postgresitem_links.metadata
The client polls GET /items/:id every ~2.5s while either pass is in flight, then stops.

Both passes run in the async worker, scheduled on the Asynq low queue from the create path (ItemService.Create → EnqueueItemParse → on completion → EnqueueLinkEnrich). With no enqueuer wired (seeder / tests) they run synchronously.

Pass Worker task Entry point Writes
1. Parse item:parse ItemService.ParseItemLinks → captureFetcher.parse link title/description/thumbnail, structured facts, status parsed
2. Enrich link:enrich ItemService.EnrichItemLinks → enrichLink resolved place (resolved_location_id), discovered_links, serper_raw

Stage 1: deterministic parse (no AI)

captureFetcher.parse (capture_parsers.go) turns a URL into a parseResult. The flow is resolveAndCanonicalize (follow redirects, strip tracking params) → detectPlatform → a per-platform parser:

Platform Parser Gets
Google Maps parseMaps place name + coordinates (resolves a Location directly)
Reddit parseReddit title + selftext via the .json API; oEmbed fallback (title/author only) when blocked
YouTube / TikTok parseOEmbed title + author + thumbnail
Instagram / Facebook / other parseOpenGraph og:title / og:description / og:image, then JSON-LD via applyJSONLD

applyJSONLD mines schema.org for a physical venue (geocoded through LocationService, Photon-first) and structured LinkFacts (event date/price, restaurant rating/cuisine/phone). Whatever the deterministic parse resolves (a Maps share's coordinates, a JSON-LD venue) is kept; the captured link's own description is never overwritten by a later search snippet.

Reddit captions

The post body (selftext) only comes from Reddit's .json endpoint, which is blocked from datacenter IPs. From the cloud the caption is often lost and the candidate falls back to the post title. This is a known limitation, not a bug.

enrichLink (item_service.go) is the enrichment entry per link. It runs three sub-stages around two Serper calls; two of the three use the LLM, the rest is deterministic and is the safety floor.

candidate (link title)
  │
  ├─ known-place match (free, DB)  ── hit → reuse cached enrichment, done
  │
  ▼  SearchEnricher.Enrich (search_enrichment.go)
  (1) extract   LLM ExtractEntity → a focused query ("Saffron", not "best Persian food")
  (2) harvest   Serper /search (discovered links) + /maps (place candidates)
  (3) select    LLM SelectPlace → pick one, then the deterministic floor:
                  • caption-match: adopt the /maps result the post NAMES
                  • G1 name guard: reject a confident-wrong venue
                  • G2 coords bias: only true coords bias /maps
  │
  ▼  resolve place_id → LocationService.ResolveExternalPlace (dedup) → resolved_location_id
  ▼  curateDiscoveredLinks → official / menu / social / mention / review / map
  ▼  mergeEnrichMetadata (additive) → item_links.metadata
  ▼  cache by place: enrich:place:<location_id>  (Redis, 14d)

The two LLM hooks

The small LLM only refines which place resolves; it can never veto a clear deterministic match. Both hooks are best-effort: a disabled / unkeyed / timed-out / malformed answer falls back to the deterministic path. See Semantic Resolver for prompts, parsing, and the provider interface.

  • Extract (SemanticResolver.ExtractEntity) distills the noisy caption into a short search query plus the physical venue to geo-resolve. An is_place_like: false with no query vetoes the billable /maps spend on a clip that names no place. A non-empty query always runs /maps, even if the model also flags is_place_like: false (it mislabels real places often enough that the query wins).
  • Select (SemanticResolver.SelectPlace) picks the right /maps candidate from the list, or returns "none". A confident pick is committed; anything else falls through to the deterministic floor.

The deterministic floor (choosePlace)

Runs when the model is off, misses, or is unsure. It is eager by design: it adopts the /maps result whose distinctive name appears in the raw caption (a generic LLM query like "authentic Persian food" still returns a list; the caption names "Saffron", so we pick Saffron Kabab from that list). Guards keep it honest:

  • G1 name match (acceptPlace): the place's distinctive name tokens must appear in the caption. Generic food/category words (cuisine, kebab, grill, …) are not distinctive, so a same-cuisine neighbor can't be mistaken for the post's place.
  • G2 coords bias (maps): only real place coordinates bias /maps, never a map viewport centre.
  • Locality reject (looksLikeLocality): a /maps result that is really a city (no reviews, no category, name == its own address locality) is dropped.

A non-Maps adopt always sets needs_confirmation, so the UI shows "is this X?" and a wrong pick is one tap to dismiss. This is why the floor can afford to be eager.

curateDiscoveredLinks classifies the /search organic results into typed, snippet-bearing DiscoveredLinks, anchored on the resolved place's own domain so a third-party directory can't masquerade as the official site:

official · menu · social (the place's profiles) · mention (third-party posts about it, e.g. another reddit thread, an IG reel) · review · map (the Google Maps place). Aggregators / directories / unrecognized hosts are dropped. The classifier is a discrete deterministic step; an LLM link-curator can replace classifyDiscoveredLink later without reshaping the data.

City bias

Queries are biased to a city so "Canoe" resolves the Toronto restaurant, not a random one. EnrichInput.City is sourced in captureCity from the item's resolved place, else its plan's anchor place, else the launch-market default (enrichCity). withCity appends the city only when the candidate doesn't already name it.

Place enrichment cache (cross-user reuse)

Enrichment is keyed by the resolved place, not the link, so it is shared. After a place resolves, its curated output is cached at enrich:place:<location_id> (Redis, 14d TTL): { discovered_links, place_cid, place_fid, thumbnail }. A known-place match, a re-capture, or another user capturing a different link to the same place reuses it with no Serper spend. A reparse (force) bypasses the cache and refreshes it. mergeEnrichMetadata is additive: a pass that produced nothing (an LLM miss, a quota stop) never wipes good prior data.

Places lookup

Resolution layers, cheapest first:

  1. Known-place match (LocationService.MatchKnown → pickKnownPlaceMatch): substring-matches the caption against names of Locations we already have. Free, no external call. Only adopts a name with a distinctive (non-generic) token.
  2. Serper /maps (Google): the primary POI lookup. Strong coverage for restaurants/venues.
  3. Photon (self-hosted OSM geocoder): used by LocationService for the manual place picker and to geocode a JSON-LD venue name (Photon-first, Google fallback). Not used for free-text caption candidates.

A resolved Google place is funneled through LocationService.ResolveExternalPlace, which find-or-creates a canonical Location (dedup by provider id), so the same place captured by many users maps to one Location and one cache entry. Rows minted this way carry provenance = serper_maps (sparse, no Places API call); the background worker upgrades them to full Places detail under a daily budget — see Locations → Provenance.

Frontend

The shape the FE consumes is ItemLinkSignals (decoded from ItemLink.metadata) in services/itemService.ts. The captured link is the user's item.links[]; the enrichment's discovered_links are distinct ("found online", not user-entered).

  • Progress — itemParseState(item) returns parsing (status pending_parse), enriching (parsed, recent, no place/links yet), or ready. It is the single source of truth shared by both screens.
  • Polling — the item sheet (ItemSheet.tsx) and the plan canvas (app/(social)/plans/[id].tsx) poll GET /items/:id (or list) every 2.5s while anything is parsing/enriching, then stop. So a place + its links appear live, and the indicators clear themselves.
  • Display — the item sheet shows the captured link, then a "Found online" section of discovered links (icon + title + snippet + source). The canvas place card shows actionable pills (Menu / Official / Google Maps) + cuisine.
  • Reparse — the refresh button enqueues a force re-enrich (off the request, so the slow LLM/Serper pass doesn't time out the call) and the sheet polls until the refreshed place/links land. POST /items/:id/reparse.

Configuration

Key Where Default Purpose
SERPER_API_KEY env (backend/.env.local) — Serper search key. No key → no place/discovered-links enrichment.
ENABLE_SEARCH_ENRICHMENT env false Master switch for the Serper pass.
CAPTURE_DAILY_PARSE_LIMIT env 0 (unbounded) Per-user/day Serper budget; dev accounts bypass.
SEMANTIC_ENABLED config.local.yaml (env overrides) true (local) Small-LLM resolver master switch.
SEMANTIC_BASE_URL config / env http://localhost:11434 LLM server root (OpenAI-compatible).
SEMANTIC_MODEL config / env qwen2.5:3b-instruct Model id.
SEMANTIC_API_KEY config / env — Empty for keyless self-host; set for a hosted provider.
SEMANTIC_TIMEOUT config / env 30s (local) Per-call cap; CPU inference needs ≥ ~10s.

See docs/reference/env-vars.md and configuration.

Setup

One command from a fresh checkout:

./scripts/setup-capture.sh

It checks host prerequisites (Docker, Go, Task), installs Go deps, brings up the local stack + the Ollama LLM server, pulls the resolver model, and writes the capture env keys into backend/.env.local. Then add your SERPER_API_KEY and run task dev. Manual equivalent:

docker compose -f docker-compose.dev.yml up -d postgres redis minio photon
task semantic:up && task semantic:pull        # Ollama + pull qwen2.5:3b-instruct
# backend/.env.local: SERPER_API_KEY=...  ENABLE_SEARCH_ENRICHMENT=true
task dev

CPU inference is fine for local dev and low volume. For scale, point SEMANTIC_BASE_URL at a GPU-backed server or a hosted provider (below), no code change.

Swapping the LLM provider

The resolver depends on a transport interface, so self-hosted vs hosted is a config swap for any OpenAI-compatible provider (Ollama, OpenAI, Together, Groq, vLLM, llama.cpp). openAICompatClient (llm_client.go) calls POST {SEMANTIC_BASE_URL}/v1/chat/completions.

OpenAI-compatible provider (env only, no code):

SEMANTIC_BASE_URL=https://api.openai.com
SEMANTIC_MODEL=gpt-4o-mini
SEMANTIC_API_KEY=sk-...
# SEMANTIC_JSON_MODE controls response_format: json_object; keep it on if supported.

Hosted models are far faster than local CPU, so SEMANTIC_TIMEOUT can drop back toward a few seconds.

A non-compatible provider (a bespoke HTTP shape) needs a second LLMClient implementation:

  1. Implement LLMClient (one method: Complete(ctx, LLMRequest) (string, error)) in a new file next to llm_client.go.
  2. Construct it in NewSemanticResolver (semantic_resolver.go) behind a config discriminator, or add a ProvideSemanticResolver branch in wiring/providers.go.

Nothing else changes: the resolver, the enricher, and the prompts are transport-agnostic. The model only needs to return constrained JSON; the resolver brace-scans the first balanced object and validates, so prose-wrapped output is tolerated.

Where to look

Testing

  • Resolver parse paths + the deterministic guards: semantic_resolver_test.go, search_enrichment_test.go (TestAcceptPlace, TestClassifyDiscoveredLink, TestWithCity), semantic_enrich_test.go (the place-selection floor, the generic-query-list case, additive merge).
  • Live Serper sanity (skipped without SERPER_API_KEY): TestLiveEnrich.
  • Run the suite: from backend/, go test ./internal/services/....