AI applications: retrieval, safe output and evaluation
Outcome and deterministic baseline
A useful assistant must solve an authorized task more reliably than the simpler alternative. Begin with a baseline and a dataset, not a model selection. The fixture answers from deterministic lexical retrieval so its permission and rendering boundaries are inspectable without API keys or paid calls. It is not a claim of production LLM quality.
Retrieval architecture
Filter by current tenant/permission and deletion state before ranking. Ranking first and hiding unauthorized citations afterward can still leak content into generated text or traces. Preserve document ID, version, source and retrieval time so the answer can explain its evidence. In a real index, test deletion propagation and permissions updates independently of ingestion success.
Chunking trades local context against retrieval precision and index cost. Evaluate lexical, semantic and hybrid retrieval on actual questions, including no-answer and ambiguous cases. Reranking can improve relevance while adding latency and another data-processing boundary. Measure authorized recall and grounded answer quality separately.
Safe output contract
Treat model and tool output as untrusted data. Decode structured output against a versioned schema and render only registered components with narrow props. Text can be escaped and still contain misleading instructions; escaping prevents execution, not persuasion. Retrieved instructions must never change server authorization or tool policy.
Keep citations tied to actual retrieved records. An unsupported answer should say it lacks evidence rather than invent a source. Attachments require limits, content validation, storage access policy and retention. A generated URL needs scheme and destination checks before navigation or server fetching.
The workbench also includes a server-side model gateway and view schema. Its injected-provider tests ensure unauthorized evidence is never supplied, unsupported citations fail, empty evidence abstains without a call, and oversized output is rejected. This gateway defines a narrow JSON contract; integrating a vendor requires an explicit adapter and real evaluation.
Evaluation and provider changes
Create fixtures for answerable questions, missing evidence, revoked access, deleted documents, malicious instructions and conflicting sources. Score task correctness, source support, abstention, leakage and latency/cost per successful task. Keep a held-out set and inspect error categories; a single aggregate score can hide a permission regression.
A provider adapter should expose cancellation, bounded output, schema validation and usage metadata. Keep secrets on the server. Retries can duplicate cost and work; distinguish safe transport retry from continuing an already-started generation. Model/provider changes require the same eval suite and a deterministic fallback.
The current lab's lexical fixture does not establish that an LLM beats the baseline. That comparison needs a selected provider, recorded model/configuration and actual evaluation runs. Do not fabricate a quality score to complete a checklist.
Framework integration and migration
AI SDK UI and a domain event stream are different protocols; use an explicit adapter rather than sending arbitrary SSE into a framework message parser. AI SDK RSC is experimental and its documentation recommends AI SDK UI for production. This warning concerns that package, not ordinary React Server Components.
Start with safe text and citations, then add validated rich components and optional generation behind an evaluated boundary. Preserve deterministic search when generation is unavailable. The exercise passes when unauthorized/deleted content never reaches the answer, malicious fixture text remains inert, and every quality claim names its evidence.
Executable reference and sources
Implementation · Behavioral evidence · Primary source.
Run the projects and assessment. Human review is pending.