Skip to content

Semantic Resolver

This is the small-LLM layer inside the capture pipeline. For the whole pipeline (parse, place resolution, discovered links, frontend, setup), start at Link Capture & Enrichment.

Purpose

The semantic resolver is an optional small-LLM layer that improves link-capture resolution. It sits in front of the Serper search enrichment (backend/internal/platform/llm/search_enrichment.go) and fixes the two ways a caption-driven query goes wrong: it distills a clean entity query from a noisy social caption, and it picks the right place candidate (or none) instead of letting a confidently-wrong venue through. It is best-effort and off by default: with no model server, the pipeline runs exactly as it did before.

Mental model

Resolution runs in the async enrich worker (EnrichItemLinks). The resolver adds two hooks around the existing Serper calls; the deterministic guards (G1 name match, G2 true-coords bias) stay as the safety floor.

enrichLinkbuilds SemanticSignals
→
SearchEnricher.Enrich
Hook A: ExtractEntityLLM: caption → query
→
Serper /search + /maps
→
Hook B: SelectPlaceLLM: pick one or none
on miss / low confidence / disabled
→
deterministic G1 / G2 guard
The LLM augments the guards, it never removes them.
  • Hook A (entity extraction). Turns the capture signals (caption, hashtags, author, platform, secondary text) into a short search query, a physical venue to geo-resolve (which differs from the query for events: /search keeps the event name, /maps resolves the arena), and a typed guess. An is_place_like: false answer vetoes the billable /maps spend on a clip that names no place. Skipped for a trusted Maps share (already has a clean name) and for a bare share with no signals to extract from.
  • Hook B (place selection). Given the post context and the harvested /maps candidates, the model returns the index of the right one, or -1 for "none of these" (including roundup posts that list several places). Because it answers with an index from an enumerated list, it cannot invent a venue. Runs for non-trusted shares only. The model does not write card copy. The card's "Found online" links (official, menu, socials, related posts, reviews, each with its search snippet) are classified deterministically in the enricher (classifyDiscoveredLink + curateDiscoveredLinks), anchored on the resolved place's own domain. That classifier is a discrete step a future LLM link-curator can wrap for the long tail.

Service surface

Two layers, mirroring the location-provider pattern (provider_adapter.go): a domain interface the pipeline depends on, and a transport interface that makes self-hosted vs hosted a config swap.

SemanticResolver (backend/internal/platform/llm/semantic_resolver.go):

ExtractEntity(ctx, in SemanticSignals) (EntityExtraction, bool)
SelectPlace(ctx, in SemanticSignals, cands []PlaceCandidate) (PlaceSelection, bool)
Enabled() bool

The bool second return is the best-effort signal: false means "no usable answer, fall back to the deterministic path". A disabled resolver returns a no-op for every hook.

LLMClient (backend/internal/platform/llm/client.go) is the transport, and SEMANTIC_PROVIDER selects the implementation:

Provider Transport Notes
openai_compat (default) POST {base_url}/v1/chat/completions Self-hosted runtimes (Ollama, llama.cpp, vLLM) and most hosted OpenAI-style APIs. Retries a 429/5xx once, honoring Retry-After (capped at 5 s); error bodies surface in logs.

An unknown provider value degrades to the no-op resolver rather than speaking the wrong wire protocol. A future non-OpenAI-style provider slots in as a new LLMClient implementation plus a case in the NewSemanticResolver switch.

Positive answers are memoized in Redis for 14 days, keyed by a hash of the full rendered prompt (semantic:extract:* / semantic:select:*): a re-capture or reparse of byte-identical content skips both model calls, while any change to the caption, candidate list, or prompt template is a distinct key.

Reliability of structured output

Small quantized models are unreliable free-form but good at constrained JSON. The resolver stacks guardrails: response_format: json_object when the server supports it, a tight schema with few-shot examples, temperature 0, and a defensive parser (extractJSONObject brace-scans the first balanced object, then validates). An over-long query (the caption echoed back), an out-of-range chosen_index, or any unparseable output is treated as a miss and falls back to the guard.

Confidence

A semantic place pick carries the model's confidence, surfaced as the item's resolution_confidence for display. It is display-only: a place resolved from a caption (any non-Maps share) always routes through the "is this X?" confirmation prompt regardless of how confident the model is (needsConfirmation), so the model can improve which place resolves but never silently commits one. Only an explicit Maps share auto-accepts. A positive pick below semanticAcceptMin (0.5) is discarded in favor of the G1 guard.

The card surfaces what search enrichment found about the place as a "Found online" list of typed DiscoveredLinks, each with the result's title + snippet: official, menu, social (the place's own profiles), mention (third-party posts, e.g. another reddit thread or an Instagram reel), review, map. They ride item_links.metadata (alongside serper_raw) and are distinct from the user's captured link. Classification is deterministic (classifyDiscoveredLink): the place's website domain (from the /maps POI) anchors official/menu, social hosts split into profile-vs-post by path shape, and aggregators / directories / unknown hosts are dropped. The captured link keeps its own caption description; a search snippet never overwrites it.

Resolution guards (deterministic)

Independent of the model, two guards protect place resolution: a share with no name and no true coords resolves nothing (skips /maps), and a /maps result that looks like an administrative area (no reviews, no category, name == its own address locality, e.g. "Ottawa") is rejected as a venue.

Configuration

Same OpenAI-compatible transport (openAICompatClient) covers hosted and self-host — the choice is a base URL + API key, never a code path. Prod runs against DeepSeek's hosted endpoint; local dev can point at Ollama for offline work.

Providers

Hosted DeepSeek (default in config.prod.yaml / config.dev.yaml). api_key mounts from GCP Secret Manager via ExternalSecrets:

semantic:
  enabled: true
  base_url: "https://api.deepseek.com"
  model: "deepseek-v4-flash"
  timeout: 12s
  json_mode: true
  allow_private: false
  • The client posts to {base_url}/v1/chat/completions (path appended by openAICompatClient), so use the API root — no /v1 suffix in the config.
  • deepseek-v4-flash is the non-thinking, JSON-mode-friendly variant. deepseek-v4-pro (thinking) costs ~3× and we don't need reasoning for structured extraction.
  • Prompt caching is server-side and automatic when the prefix bytes match. Both extract and select reuse a static system prompt, so cache-hit rate is high in practice. Do not add dynamic content (timestamps, request IDs) to the system prompt or the cache slot invalidates.

Self-hosted Ollama (opt-in, local dev):

semantic:
  enabled: true
  base_url: "http://semantic:11434"
  model: "qwen2.5:3b-instruct"
  timeout: 8s
  json_mode: true
  allow_private: true
  • api_key stays empty — the client omits the Authorization header entirely.
  • allow_private: true opts out of httpx.SafeClient's SSRF guard so the docker-network host resolves.
  • Bring the container up with task semantic:up && task semantic:pull; see docker-compose.

Env-var overrides

Every YAML field has a matching env var, checked at boot; env wins when set.

Key Default Purpose
SEMANTIC_ENABLED from YAML Master switch.
SEMANTIC_PROVIDER openai_compat Transport; openai_compat is the only value today.
SEMANTIC_BASE_URL from YAML Model server root (OpenAI-compatible).
SEMANTIC_MODEL from YAML Model id.
SEMANTIC_API_KEY — From secret in prod (tomoda-deepseek-api-key/api_key); empty for keyless Ollama.
SEMANTIC_TIMEOUT from YAML (fallback 8s) Per-call cap; runs in the async enrich worker.
SEMANTIC_JSON_MODE from YAML Request response_format: json_object.
SEMANTIC_ALLOW_PRIVATE from YAML true allows loopback / RFC1918 hosts (dev Ollama).

Metrics + tracing + cost

Every LLM call and every search-enrichment provider call emits Prometheus metrics + an OTel span. Spans nest under the enclosing SearchEnricher.Enrich span, so a single trace shows extract → search → maps → select in one waterfall.

LLM (openAICompatClient):

Metric Labels Purpose
llm_requests_total model, kind, status Volume + error rate (status = ok, err_429, err_5xx, err_auth, err_4xx, err_transport, err_encode, err_read, err_parse, err_empty).
llm_request_duration_seconds model, kind Latency histogram (0.25s–20s buckets).
llm_tokens_total model, kind, direction (input/output), cache (hit/miss/na) Input tokens split into prompt_cache_hit_tokens / prompt_cache_miss_tokens when the provider reports them. Providers that omit these get all input tokens accounted as miss (conservative).

Spans: llm.<kind> (e.g. llm.extract, llm.select) with attributes llm.model, llm.kind, llm.json_mode, and (on success) llm.tokens.prompt, llm.tokens.completion, llm.tokens.cache_hit — useful for spotting which specific request caused a cost spike.

Cost per call = input_hit × $0.0028/M + input_miss × $0.14/M + output × $0.28/M (deepseek-v4-flash off-peak, as of the pricing page). Peak-window calls double when the provider's peak-valley schedule applies.

Search enrichment (provider-agnostic):

Metric Labels Purpose
search_enrichment_requests_total provider, endpoint (search/maps/other), status Volume + error rate. Same status bucketing as LLM.
search_enrichment_request_duration_seconds provider, endpoint Latency histogram (0.1s–8s buckets).

Spans: search_enrichment.<endpoint> with attributes search.provider and search.endpoint. Errored calls record the error on the span; 4xx/5xx sets http.status_code. The provider label lets us swap the backend (or run two in parallel during a migration) without renaming metrics or rewiring alerts. Provider-specific dollar cost lives on the vendor's billing page — watch search_enrichment_requests_total for volume and adjust the parse quota (CAPTURE_DAILY_PARSE_LIMIT) as budget requires.

See Outbound HTTP for the SafeClient dial-time SSRF guard that wraps every outbound call.

Force-reparse is quota-gated via CAPTURE_DAILY_PARSE_LIMIT (see parseQuotaOK in item_service.go) so a runaway client can't blow through credits. Deterministic Serper heuristic is the fallback the moment the quota tops out or the LLM errors — the pipeline never blocks on the model.

Where to look