Skip to content

Semantic Resolver

This is the small-LLM layer inside the capture pipeline. For the whole pipeline (parse, place resolution, discovered links, frontend, setup), start at Link Capture & Enrichment.

Purpose

The semantic resolver is an optional small-LLM layer that improves link-capture resolution. It sits in front of the Serper search enrichment (backend/internal/platform/llm/search_enrichment.go) and fixes the two ways a caption-driven query goes wrong: it distills a clean entity query from a noisy social caption, and it picks the right place candidate (or none) instead of letting a confidently-wrong venue through. It is best-effort and off by default: with no model server, the pipeline runs exactly as it did before.

Mental model

Resolution runs in the async enrich worker (EnrichItemLinks). The resolver adds two hooks around the existing Serper calls; the deterministic guards (G1 name match, G2 true-coords bias) stay as the safety floor.

enrichLinkbuilds SemanticSignals
→
SearchEnricher.Enrich
Hook A: ExtractEntityLLM: caption → query
→
Serper /search + /maps
→
Hook B: SelectPlaceLLM: pick one or none
on miss / low confidence / disabled
→
deterministic G1 / G2 guard
The LLM augments the guards, it never removes them.
  • Hook A (entity extraction). Turns the capture signals (caption, hashtags, author, platform, secondary text) into a short search query, a physical venue to geo-resolve (which differs from the query for events: /search keeps the event name, /maps resolves the arena), and a typed guess. An is_place_like: false answer vetoes the billable /maps spend on a clip that names no place. Skipped for a trusted Maps share (already has a clean name) and for a bare share with no signals to extract from.
  • Hook B (place selection). Given the post context and the harvested /maps candidates, the model returns the index of the right one, or -1 for "none of these" (including roundup posts that list several places). Because it answers with an index from an enumerated list, it cannot invent a venue. Runs for non-trusted shares only. The model does not write card copy. The card's "Found online" links (official, menu, socials, related posts, reviews, each with its search snippet) are classified deterministically in the enricher (classifyDiscoveredLink + curateDiscoveredLinks), anchored on the resolved place's own domain. That classifier is a discrete step a future LLM link-curator can wrap for the long tail.

Service surface

The two hooks are registered assist tasks, location_resolve and location_select (see Assist service). The places package keeps its own port, so the pipeline does not depend on assist directly.

SemanticResolver (backend/internal/platform/places/semantic_resolver.go):

ExtractEntity(ctx, in SemanticSignals) (EntityExtraction, bool)
SelectPlace(ctx, in SemanticSignals, cands []PlaceCandidate) (PlaceSelection, bool)
Enabled() bool

The bool second return is the best-effort signal: false means "no usable answer, fall back to the deterministic path". With no small model configured, or the assist feature switch off, every hook returns it.

The transport is the platform chat client (platform/llm, OpenAI-compatible POST {base_url}/v1/chat/completions), built from the semantic.* settings as the assist engine's small model. It always asks for a JSON object at temperature 0, so SEMANTIC_JSON_MODE no longer changes the request.

Positive answers are memoised in Redis for 14 days, keyed by the task, its prompt version and a hash of the rendered prompt: a re-capture or reparse of byte-identical content skips both model calls, while any change to the caption, candidate list or prompt version is a distinct key.

Reliability of structured output

Small quantized models are unreliable free-form but good at constrained JSON. The resolver stacks guardrails: response_format: json_object when the server supports it, a tight schema with few-shot examples, temperature 0, and a defensive parser (extractJSONObject brace-scans the first balanced object, then validates). An over-long query (the caption echoed back), an out-of-range chosen_index, or any unparseable output is treated as a miss and falls back to the guard.

Confidence

A semantic place pick carries the model's confidence, surfaced as the item's resolution_confidence for display. It is display-only: a place resolved from a caption (any non-Maps share) always routes through the "is this X?" confirmation prompt regardless of how confident the model is (needsConfirmation), so the model can improve which place resolves but never silently commits one. Only an explicit Maps share auto-accepts. A positive pick below semanticAcceptMin (0.5) is discarded in favor of the G1 guard.

The card surfaces what search enrichment found about the place as a "Found online" list of typed DiscoveredLinks, each with the result's title + snippet: official, menu, social (the place's own profiles), mention (third-party posts, e.g. another reddit thread or an Instagram reel), review, map. They ride item_links.metadata (alongside serper_raw) and are distinct from the user's captured link. Classification is deterministic (classifyDiscoveredLink): the place's website domain (from the /maps POI) anchors official/menu, social hosts split into profile-vs-post by path shape, and aggregators / directories / unknown hosts are dropped. The captured link keeps its own caption description; a search snippet never overwrites it.

Resolution guards (deterministic)

Independent of the model, two guards protect place resolution: a share with no name and no true coords resolves nothing (skips /maps), and a /maps result that looks like an administrative area (no reviews, no category, name == its own address locality, e.g. "Ottawa") is rejected as a venue.

Configuration

Same OpenAI-compatible transport (openAICompatClient) covers hosted and self-host — the choice is a base URL + API key, never a code path. Prod runs against DeepSeek's hosted endpoint; local dev can point at Ollama for offline work.

Providers

Hosted DeepSeek (default in config.prod.yaml / config.dev.yaml). api_key mounts from GCP Secret Manager via ExternalSecrets:

semantic:
  enabled: true
  base_url: "https://api.deepseek.com"
  model: "deepseek-v4-flash"
  timeout: 12s
  json_mode: true
  allow_private: false
  • The client posts to {base_url}/v1/chat/completions (path appended by openAICompatClient), so use the API root — no /v1 suffix in the config.
  • deepseek-v4-flash is the non-thinking, JSON-mode-friendly variant. deepseek-v4-pro (thinking) costs ~3× and we don't need reasoning for structured extraction.
  • Prompt caching is server-side and automatic when the prefix bytes match. Both extract and select reuse a static system prompt, so cache-hit rate is high in practice. Do not add dynamic content (timestamps, request IDs) to the system prompt or the cache slot invalidates.

Self-hosted Ollama (opt-in, local dev):

semantic:
  enabled: true
  base_url: "http://semantic:11434"
  model: "qwen2.5:3b-instruct"
  timeout: 8s
  json_mode: true
  allow_private: true
  • api_key stays empty — the client omits the Authorization header entirely.
  • allow_private: true opts out of httpx.SafeClient's SSRF guard so the docker-network host resolves.
  • Bring the container up with task semantic:up && task semantic:pull; see docker-compose.

Env-var overrides

Every YAML field has a matching env var, checked at boot; env wins when set.

Key Default Purpose
SEMANTIC_ENABLED from YAML Master switch.
SEMANTIC_PROVIDER openai_compat Transport; openai_compat is the only value today.
SEMANTIC_BASE_URL from YAML Model server root (OpenAI-compatible).
SEMANTIC_MODEL from YAML Model id.
SEMANTIC_API_KEY — From secret in prod (tomoda-deepseek-api-key/api_key); empty for keyless Ollama.
SEMANTIC_TIMEOUT from YAML (fallback 8s) Per-call cap; runs in the async enrich worker.
SEMANTIC_JSON_MODE from YAML Request response_format: json_object.
SEMANTIC_ALLOW_PRIVATE from YAML true allows loopback / RFC1918 hosts (dev Ollama).

Metrics + tracing + cost

Every LLM call and every search-enrichment provider call emits Prometheus metrics + an OTel span. Spans nest under the enclosing SearchEnricher.Enrich span, so a single trace shows extract → search → maps → select in one waterfall.

Both families register on the /metrics registry via the observability.New* helpers. llm.PrimeMetrics zero-initialises the extract and select request and token series when the resolver is built, and NewSearchEnricher does the same for the Serper endpoints, so panels read 0 before the first call.

LLM (openAICompatClient):

Metric Labels Purpose
llm_requests_total model, kind, status Volume + error rate (status = ok, err_429, err_5xx, err_auth, err_4xx, err_timeout, err_transport, err_encode, err_read, err_parse, err_empty).
llm_request_duration_seconds model, kind Latency histogram (0.25s–20s buckets).
llm_tokens_total model, kind, direction (input/output), cache (hit/miss/na) Input tokens split into prompt_cache_hit_tokens / prompt_cache_miss_tokens when the provider reports them. Providers that omit these get all input tokens accounted as miss (conservative).
deepseek_balance_usd currency DeepSeek prepaid total_balance per currency, refreshed hourly by cron:deepseek_balance on the async pods. A failed check keeps the last-good value. Feeds the DeepSeekBalanceLow alert.
deepseek_balance_check_errors_total none Failed balance checks (transport, non-200, parse).
deepseek_balance_last_success_timestamp_seconds none Unix time of the last successful balance check.

Spans: llm.<kind> (e.g. llm.extract, llm.select) with attributes llm.model, llm.kind, llm.json_mode, and (on success) llm.tokens.prompt, llm.tokens.completion, llm.tokens.cache_hit — useful for spotting which specific request caused a cost spike.

Cost per call = input_hit × $0.0028/M + input_miss × $0.14/M + output × $0.28/M (deepseek-v4-flash off-peak, as of the pricing page). Peak-window calls double when the provider's peak-valley schedule applies.

Search enrichment (provider-agnostic):

Metric Labels Purpose
search_enrichment_requests_total provider, endpoint (search/maps/other), status Volume + error rate. Same status bucketing as LLM.
search_enrichment_request_duration_seconds provider, endpoint Latency histogram (0.1s–8s buckets).
serper_credits_used_total operation (search/maps) Serper credits spent, read from each response's credits field.

Spans: search_enrichment.<endpoint> with attributes search.provider and search.endpoint. Errored calls record the error on the span; 4xx/5xx sets http.status_code. The provider label lets us swap the backend (or run two in parallel during a migration) without renaming metrics or rewiring alerts. Watch serper_credits_used_total for spend and adjust the parse quota (CAPTURE_DAILY_PARSE_LIMIT) as budget requires. The scrape target carries its own endpoint label, so this metric's endpoint reaches Prometheus as exported_endpoint. See Third-party usage.

See Outbound HTTP for the SafeClient dial-time SSRF guard that wraps every outbound call.

Force-reparse is quota-gated via CAPTURE_DAILY_PARSE_LIMIT (see parseQuotaOK in item_service.go) so a runaway client can't blow through credits. Deterministic Serper heuristic is the fallback the moment the quota tops out or the LLM errors — the pipeline never blocks on the model.

Where to look