Assist service¶
Purpose¶
backend/internal/platform/assist is the semantic layer behind every model-assisted feature, built on the platform/llm transport. It owns the task registry, the engine that runs a task, strict output validation, caching and the stored results. It imports no domain. See Assist layer for the system view.
Layout¶
| Path | Holds |
|---|---|
assist/ |
Task, Spec, Handler, the Engine, schema validation, confidence, the ai_reviews store, the eval runner and the fake model |
assist/tasks/app/ |
The member app's tasks, one file each, with their eval fixtures |
assist/tasks/backstage/ |
Backstage's tasks, one file each, with their eval fixtures |
assist/tasks/registry.go |
The single registry, listing the two groups separately |
assist/assistdb/ |
sqlc queries for ai_reviews |
Tasks¶
| Task | Audience | Run from | Model | What it returns |
|---|---|---|---|---|
location_resolve |
app | The link-capture enricher (platform/places) |
small | A clean search query, venue and kind for a captured post |
location_select |
app | The link-capture enricher | small | The index of the right place candidate, or none |
report_triage |
backstage | Job queued when a report is filed | review | Outcome (dismiss, action, review), kind, policy category, rationale, confidence and indicators |
photo_review |
backstage | Job queued when a cover photo is submitted | review | Accept, reject or review, with relevance, people, quality, duplicate, rationale and indicators |
translate |
backstage | Job queued when an announcement is saved | review | Title and message in each locale staff have not written |
Every task is run at temperature 0. translate works in batches of five languages so a long message stays inside the output cap.
Gaps¶
- There is no place review task. No member-created place or place change reaches a review queue today, so a task would have no trigger. Add one with the review path.
search_intentdoes not exist in the code, so it has no task.photo_reviewcannot see pixels. It works from the place, the caption of the source moment, the platform checks and the member's earlier rejection count.
Running a task¶
A domain holds a Runner (one Do method) scoped to its audience and calls assist.Run(ctx, runner, task, subject, input). The engine then:
- Refuses when the assist switch is off (
ErrOff) or the task belongs to the other audience. - Returns the stored result when this subject already has one for the same input hash and prompt version, or the memoised one for a call with no subject.
- Applies the task's
Prerules. A rule can decide the result or withhold the item without a model call. - Calls the model with the task's prompt, validates the JSON against the closed schema, and asks once more if it is invalid. A second failure stores
no_suggestion. - Applies the task's
Constrainrules, which can override the model, then derives the confidence bucket. - Stores the record on its subject (
ai_reviews) or in the memo.
Confidence starts from the model's own number (0.8 and up is high, 0.5 and up is medium). A conflict with the rule signals lowers it one bucket, so does needing the retry, and a rule that overrode the model caps it at medium.
Results on items¶
ai_reviews keeps one row per run, keyed by subject_type, subject_id and task, newest first. Backstage reads them back as ai_review on the report group and cover submission detail, as ai_reviews by id on the report and cover queue pages and as suggested_texts on each announcement. A translation batch is stored under its own subject, derived from the announcement id, so batches never replace each other.
Triggers¶
| Event | Hook | Job |
|---|---|---|
| A report is filed on a group (new report only) | report.ReviewTrigger |
assist:review kind report_group |
| A cover photo is submitted | geocover.ReviewTrigger |
kind geo_cover |
| An announcement is created or edited | backstage.Service |
kind announcement |
The two domain ports are satisfied by backstage.ReviewTriggers, which only enqueues and logs a failure. assist:review runs on the low queue, starts 20 seconds after the change that queued it so a burst of reports runs once after the last of them, reads the item as it is when it executes, and skips retry when assist is off, has no model or is out of budget. When an item's inputs change the next run sees a new input hash and calls the model again.
Configuration¶
| Setting | Meaning |
|---|---|
assist.base_url, assist.model, assist.timeout, assist.allow_private, ASSIST_API_KEY |
The review model. Empty base URL or model leaves Backstage tasks without a model |
assist.daily_token_budget, ASSIST_DAILY_TOKEN_BUDGET |
Review-model tokens per UTC day across all tasks, default 5,000,000 |
semantic.* |
The small model for the place resolution tasks, as before |
semantic.daily_token_budget, SEMANTIC_DAILY_TOKEN_BUDGET |
Small-model tokens per UTC day, counted apart from the review model's budget so place resolution cannot starve staff review. Default 2,000,000 |
assist feature switch |
Turns every task off at once, from Backstage with a reason, audited |
Metrics carry the audience. assist_runs_total counts runs by audience, task and outcome (model, cache, rule, invalid, off, budget, error), and the llm_* series label each call <audience>.<task>.
Retention¶
ai_reviews keeps results for 90 days and the newest three of each task on an item. A daily job (cron:assist_review_prune) applies both. Database triggers delete an item's results when the item is deleted and when the member it is about is erased, because the table has no foreign key. See Privacy.
Evals¶
Each task has a fixture of labelled cases next to its group (assist/tasks/<audience>/fixtures/). A case has an input, the labels the final output must carry after the rules ran, and a canned reply for the fake model. Cases flagged exemplar are baked into the prompt as few-shot examples.
- In tests.
go test ./internal/platform/assist/...runs every case through the real engine against the fake model. It covers parsing, closed vocabularies, rule overrides, confidence, caching and determinism.report_triagehas more than forty cases including benign content, sarcasm and other languages,photo_reviewmore than twenty, andtranslatechecks placeholder and length preservation.task assist:evalruns the same check from the command line. - Against the real model.
task assist:eval:realsends every case to the model named byASSIST_EVAL_BASE_URL,ASSIST_EVAL_MODELandASSIST_EVAL_API_KEYand prints which labels differ. It asks for confirmation, costs one call per case and never runs in CI.
Adding a task¶
- Pick the audience.
appfor what a member's own use case calls.backstagefor what staff review or translate. The audience sets the folder (assist/tasks/app/orassist/tasks/backstage/), the runner a domain needs and who may trigger it. - Write the input type and the closed schema. One file per task. Outputs are enums, closed lists, numbers from 0 to 1 and one short rationale. Anything else is rejected.
- Put rules first. Write
Prefor what a rule can decide or withhold (child safety, banned terms, exact duplicates),Constrainfor what the model may never do, andAgreefor how rule signals corroborate or conflict.Rendermasks every member-written string withllm.Untrusted. - Write the prompt and bump its version. Exemplars come from fixture cases flagged
exemplar. The prompt lock test hashes the prompt, the fields and the behavior of every rule over the fixture cases, and fails until the version and hash are updated together. - Add the fixture. Cases with labels and fake replies, including benign, ambiguous and rule-override cases. Backstage review tasks need enough cases to be meaningful (dozens, not a handful).
- Pick the model kind and budget class.
smallorreview, andtiny,small,mediumorlargefor the output cap. - Register it in the group's
All(). - Add the trigger. For an app task, the domain calls it behind its own port in
deps.go. For a Backstage task, the domain gets aReviewTriggerport, Backstage satisfies it by enqueuingassist:review, and the job body lives inbackstage/reviews.go. Never add an endpoint that runs a task. - Show it with the item. Return the stored result as part of the item's response.
Invariants¶
- No endpoint runs a task. A test asserts no Backstage route mentions assist, suggest, draft or translate.
- A Backstage task is imported only by Backstage code, an app task never by Backstage code, and the assist package imports no service or model. Tests walk the imports.
- A child safety report reaches neither the model nor the input hash.
- Staff text from a translation is never published by assist. Saved translations come from the editor.