Semantic Resolver¶
This is the small-LLM layer inside the capture pipeline. For the whole pipeline (parse, place resolution, discovered links, frontend, setup), start at Link Capture & Enrichment.
Purpose¶
The semantic resolver is an optional small-LLM layer that improves link-capture
resolution. It sits in front of the Serper search enrichment
(backend/internal/platform/llm/search_enrichment.go) and fixes the two ways a
caption-driven query goes wrong: it distills a clean entity query from
a noisy social caption, and it picks the right place candidate (or none) instead
of letting a confidently-wrong venue through. It is best-effort and off by
default: with no model server, the pipeline runs exactly as it did before.
Mental model¶
Resolution runs in the async enrich worker (EnrichItemLinks). The resolver adds
two hooks around the existing Serper calls; the deterministic guards (G1 name
match, G2 true-coords bias) stay as the safety floor.
- Hook A (entity extraction). Turns the capture signals (caption, hashtags,
author, platform, secondary text) into a short search
query, a physicalvenueto geo-resolve (which differs from the query for events:/searchkeeps the event name,/mapsresolves the arena), and a typed guess. Anis_place_like: falseanswer vetoes the billable/mapsspend on a clip that names no place. Skipped for a trusted Maps share (already has a clean name) and for a bare share with no signals to extract from. - Hook B (place selection). Given the post context and the harvested
/mapscandidates, the model returns the index of the right one, or-1for "none of these" (including roundup posts that list several places). Because it answers with an index from an enumerated list, it cannot invent a venue. Runs for non-trusted shares only. The model does not write card copy. The card's "Found online" links (official, menu, socials, related posts, reviews, each with its search snippet) are classified deterministically in the enricher (classifyDiscoveredLink+curateDiscoveredLinks), anchored on the resolved place's own domain. That classifier is a discrete step a future LLM link-curator can wrap for the long tail.
Service surface¶
The two hooks are registered assist tasks, location_resolve and location_select
(see Assist service). The places package keeps its own port, so the
pipeline does not depend on assist directly.
SemanticResolver (backend/internal/platform/places/semantic_resolver.go):
ExtractEntity(ctx, in SemanticSignals) (EntityExtraction, bool)
SelectPlace(ctx, in SemanticSignals, cands []PlaceCandidate) (PlaceSelection, bool)
Enabled() bool
The bool second return is the best-effort signal: false means "no usable
answer, fall back to the deterministic path". With no small model configured, or
the assist feature switch off, every hook returns it.
The transport is the platform chat client (platform/llm, OpenAI-compatible
POST {base_url}/v1/chat/completions), built from the semantic.* settings as
the assist engine's small model. It always asks for a JSON object at temperature
0, so SEMANTIC_JSON_MODE no longer changes the request.
Positive answers are memoised in Redis for 14 days, keyed by the task, its prompt version and a hash of the rendered prompt: a re-capture or reparse of byte-identical content skips both model calls, while any change to the caption, candidate list or prompt version is a distinct key.
Reliability of structured output¶
Small quantized models are unreliable free-form but good at constrained JSON.
The resolver stacks guardrails: response_format: json_object when the server
supports it, a tight schema with few-shot examples, temperature 0, and a
defensive parser (extractJSONObject brace-scans the first balanced object, then
validates). An over-long query (the caption echoed back), an out-of-range
chosen_index, or any unparseable output is treated as a miss and falls back to
the guard.
Confidence¶
A semantic place pick carries the model's confidence, surfaced as the item's
resolution_confidence for display. It is display-only: a place resolved from
a caption (any non-Maps share) always routes through the "is this X?"
confirmation prompt regardless of how confident the model is (needsConfirmation),
so the model can improve which place resolves but never silently commits one.
Only an explicit Maps share auto-accepts. A positive pick below semanticAcceptMin
(0.5) is discarded in favor of the G1 guard.
Discovered links¶
The card surfaces what search enrichment found about the place as a "Found
online" list of typed DiscoveredLinks, each with the result's title + snippet:
official, menu, social (the place's own profiles), mention (third-party
posts, e.g. another reddit thread or an Instagram reel), review, map. They ride
item_links.metadata (alongside serper_raw) and are distinct from the user's
captured link. Classification is deterministic (classifyDiscoveredLink): the
place's website domain (from the /maps POI) anchors official/menu, social
hosts split into profile-vs-post by path shape, and aggregators / directories /
unknown hosts are dropped. The captured link keeps its own caption description; a
search snippet never overwrites it.
Resolution guards (deterministic)¶
Independent of the model, two guards protect place resolution: a share with no
name and no true coords resolves nothing (skips /maps), and a /maps result
that looks like an administrative area (no reviews, no category, name == its own
address locality, e.g. "Ottawa") is rejected as a venue.
Configuration¶
Same OpenAI-compatible transport (openAICompatClient) covers hosted and
self-host — the choice is a base URL + API key, never a code path. Prod runs
against DeepSeek's hosted endpoint; local dev can point at Ollama for offline
work.
Providers¶
Hosted DeepSeek (default in config.prod.yaml / config.dev.yaml).
api_key mounts from GCP Secret Manager via ExternalSecrets:
semantic:
enabled: true
base_url: "https://api.deepseek.com"
model: "deepseek-v4-flash"
timeout: 12s
json_mode: true
allow_private: false
- The client posts to
{base_url}/v1/chat/completions(path appended byopenAICompatClient), so use the API root — no/v1suffix in the config. deepseek-v4-flashis the non-thinking, JSON-mode-friendly variant.deepseek-v4-pro(thinking) costs ~3× and we don't need reasoning for structured extraction.- Prompt caching is server-side and automatic when the prefix bytes match.
Both
extractandselectreuse a static system prompt, so cache-hit rate is high in practice. Do not add dynamic content (timestamps, request IDs) to the system prompt or the cache slot invalidates.
Self-hosted Ollama (opt-in, local dev):
semantic:
enabled: true
base_url: "http://semantic:11434"
model: "qwen2.5:3b-instruct"
timeout: 8s
json_mode: true
allow_private: true
api_keystays empty — the client omits theAuthorizationheader entirely.allow_private: trueopts out ofhttpx.SafeClient's SSRF guard so the docker-network host resolves.- Bring the container up with
task semantic:up && task semantic:pull; see docker-compose.
Env-var overrides¶
Every YAML field has a matching env var, checked at boot; env wins when set.
| Key | Default | Purpose |
|---|---|---|
SEMANTIC_ENABLED |
from YAML | Master switch. |
SEMANTIC_PROVIDER |
openai_compat |
Transport; openai_compat is the only value today. |
SEMANTIC_BASE_URL |
from YAML | Model server root (OpenAI-compatible). |
SEMANTIC_MODEL |
from YAML | Model id. |
SEMANTIC_API_KEY |
— | From secret in prod (tomoda-deepseek-api-key/api_key); empty for keyless Ollama. |
SEMANTIC_TIMEOUT |
from YAML (fallback 8s) | Per-call cap; runs in the async enrich worker. |
SEMANTIC_JSON_MODE |
from YAML | Request response_format: json_object. |
SEMANTIC_ALLOW_PRIVATE |
from YAML | true allows loopback / RFC1918 hosts (dev Ollama). |
Metrics + tracing + cost¶
Every LLM call and every search-enrichment provider call emits Prometheus
metrics + an OTel span. Spans nest under the enclosing
SearchEnricher.Enrich span, so a single trace shows extract → search →
maps → select in one waterfall.
Both families register on the /metrics registry via the
observability.New* helpers. llm.PrimeMetrics zero-initialises the
extract and select request and token series when the resolver is built,
and NewSearchEnricher does the same for the Serper endpoints, so panels
read 0 before the first call.
LLM (openAICompatClient):
| Metric | Labels | Purpose |
|---|---|---|
llm_requests_total |
model, kind, status |
Volume + error rate (status = ok, err_429, err_5xx, err_auth, err_4xx, err_timeout, err_transport, err_encode, err_read, err_parse, err_empty). |
llm_request_duration_seconds |
model, kind |
Latency histogram (0.25s–20s buckets). |
llm_tokens_total |
model, kind, direction (input/output), cache (hit/miss/na) |
Input tokens split into prompt_cache_hit_tokens / prompt_cache_miss_tokens when the provider reports them. Providers that omit these get all input tokens accounted as miss (conservative). |
deepseek_balance_usd |
currency |
DeepSeek prepaid total_balance per currency, refreshed hourly by cron:deepseek_balance on the async pods. A failed check keeps the last-good value. Feeds the DeepSeekBalanceLow alert. |
deepseek_balance_check_errors_total |
none | Failed balance checks (transport, non-200, parse). |
deepseek_balance_last_success_timestamp_seconds |
none | Unix time of the last successful balance check. |
Spans: llm.<kind> (e.g. llm.extract, llm.select) with attributes
llm.model, llm.kind, llm.json_mode, and (on success) llm.tokens.prompt,
llm.tokens.completion, llm.tokens.cache_hit — useful for spotting which
specific request caused a cost spike.
Cost per call = input_hit × $0.0028/M + input_miss × $0.14/M + output × $0.28/M
(deepseek-v4-flash off-peak, as of the pricing page). Peak-window calls
double when the provider's peak-valley schedule applies.
Search enrichment (provider-agnostic):
| Metric | Labels | Purpose |
|---|---|---|
search_enrichment_requests_total |
provider, endpoint (search/maps/other), status |
Volume + error rate. Same status bucketing as LLM. |
search_enrichment_request_duration_seconds |
provider, endpoint |
Latency histogram (0.1s–8s buckets). |
serper_credits_used_total |
operation (search/maps) |
Serper credits spent, read from each response's credits field. |
Spans: search_enrichment.<endpoint> with attributes search.provider and
search.endpoint. Errored calls record the error on the span; 4xx/5xx sets
http.status_code. The provider label lets us swap the backend (or run
two in parallel during a migration) without renaming metrics or rewiring
alerts. Watch serper_credits_used_total for spend and adjust the parse quota
(CAPTURE_DAILY_PARSE_LIMIT) as budget requires. The scrape target carries
its own endpoint label, so this metric's endpoint reaches Prometheus as
exported_endpoint. See Third-party usage.
See Outbound HTTP for the SafeClient dial-time SSRF guard that wraps every outbound call.
Force-reparse is quota-gated via CAPTURE_DAILY_PARSE_LIMIT (see
parseQuotaOK in item_service.go) so a runaway client can't blow through
credits. Deterministic Serper heuristic is the fallback the moment the quota
tops out or the LLM errors — the pipeline never blocks on the model.
Where to look¶
backend/internal/platform/llm/semantic_resolver.go— interface, DTOs, prompts, parsers, memoization.backend/internal/platform/llm/client.go— OpenAI-compatible transport (+ retry).backend/internal/platform/llm/search_enrichment.go—Enrichhooks A + B,choosePlace, and the deterministicclassifyDiscoveredLink+curateDiscoveredLinks.backend/internal/services/plan/item_service.go—enrichLinkbuildsSemanticSignals;mergeEnrichMetadatafolds the discovered links into the snapshot.backend/internal/wiring/providers.go—ProvideSemanticResolver.