The hot path is the visitor-message → first-token pipeline. It has a hard 1 second p95 contract — beyond that, the perceived "is this thing alive?" tension breaks. Everything that doesn't have to happen on the hot path is pushed off it.

Latency budget

Target breakdown for first-token at p95:

Headroom for the rest is < 150ms. Anything beyond first-token streams out incrementally — full-response time is bounded by token rate, not the budget.

Hard rules

These are enforced by code review and by tests:

1. No DB writes on the hot path. Persistence is async after the stream ends.
2. No synchronous webhooks. Outgoing webhooks are dispatched as queue jobs.
3. No retries. If a provider fails mid-stream, the user sees a graceful error and the widget auto-retries client-side. Server doesn't loop.
4. No N+1 queries. All reads are batched. Recent history comes from Redis (conv:{id}:history), not Postgres.
5. One LLM call per turn. No multi-step agent reasoning that fans out into multiple model calls.
6. Short-circuit DB writes ride below emit('done'). The human-takeover, human-pending, and human-intent short-circuits used to persist the visitor's Message + stamp conversation.attribution before the SSE close event reached the client. Audit 2026-05-30 moved these below emit('done') via persistShortCircuitVisitorTurn() so the stream closes immediately and the writes run after. The escalation_offered_at stamp emitted by the main LLM path is deferred to the same post-emit slot — next turn's tool loop re-reads the row from DB so the lag doesn't change tool gating semantics.

What's off the hot path

Everything below is dispatched after the stream completes. None of it blocks the visitor:

• PersistTurnJob — save user + assistant messages.
• IncrementUsageJob — bump the workspace's monthly conversation counter.
• DetectGapJob — cluster low-confidence questions for the gap report.
• DispatchWebhookJob — fan out to subscribed customer endpoints (workflow side; the lead-captured webhook fires inline via SignedDispatcher when a visitor submits the lead form).
• AutoIndexPageVisit — synchronous service triggered at /init time that queues a CrawlPageJob for the visited URL when auto-indexing is enabled. Not a hot-path job, but worth knowing where auto-index runs.

Caching

Two caches keep the hot path tight:

• Retrieval cache — Redis, rag:retrieve:{agentId}:{hash(query|currentPageUrl)}, 30-minute TTL. Same question on the same page hits cache. Invalidated when sources change.
• Conversation history cache — Redis, conv:{convId}:history, 2-hour TTL, capped at 12 messages (6 turns). Reads from this on every turn instead of Postgres.

Streaming mechanics

SSE is dead simple — keep-alive HTTP, write data: {...}\n\n per token, flush. The widget reads via EventSource (or fetch + reader for older browsers without EventSource on POST).

Critically, the SSE response is constructed before any RAG work runs. We start writing headers immediately on request receipt so any proxy in front of us (Cloudflare, load balancer) commits to streaming early. By the time tokens arrive, the connection is already open.

Where the spans live

OpenTelemetry spans wrap each phase:

• widget.message.receive
• rag.curated.match
• rag.embed
• rag.vector.search
• rag.rerank
• rag.prompt.assemble
• rag.llm.first_token
• rag.llm.stream
• rag.persist.async

Honeycomb / Grafana shows the p95 of each. When the budget breaks, the span heatmap usually points right at the offender.

Failure modes

The principle: the visitor always gets a response, even if it's "I'm not sure". The agent is allowed to be ignorant; it isn't allowed to silently break.