Conversational Product Copilot

The problem

Two conversational features, superficially similar and architecturally opposite.

The first is an in-product assistant: it should navigate the user to the right screen, explain what a feature does, create simple records, and hand off to a human when it cannot help. To be useful it has to act, which means tools — and the tools it needs are the ones the current screen can actually perform, which the backend does not know.

The second answers questions about one specific document, and must not answer anything else. Its failure mode is not being unhelpful; it is confidently supplying general knowledge about contract law that is not in the document the user is looking at.

The approach

Two LangGraph agents in one runtime, each with its own graph and its own AG-UI endpoint, sharing a Postgres checkpointer and the surrounding infrastructure. Conversation history lives in checkpoints keyed by thread, streamed to the browser over server-sent events.

The navigation agent binds tools that arrive in its state, declared by the frontend. Five tools: canonical navigation over a vocabulary of screens with aliases, capability description, concept explanation, permission lookup by role, and opening a human support conversation.

The document agent has no tools and no system message at all.

Decisions worth defending

The document agent has no system prompt, and that is the design. Vertex context caching is mutually exclusive with both tools and a system message. Sending the document again on every turn would have been the obvious alternative and would have kept the persona where a persona belongs — at a per-turn input token cost roughly an order of magnitude above reading from cache, on every message of every conversation.

So the persona is baked into the cached content itself: the instructions live inside the cache alongside the document. The cost is real. The agent’s behaviour is now defined by something minted at cache creation time rather than by code, which makes it harder to change and harder to read. Two things make that acceptable: a degraded inline mode exists for when caching is unavailable, and the cache lifecycle is persisted per thread so a stale cache is re-minted rather than silently reused. Given a long document and a chatty conversation, the economics were not close.

Threads are created lazily. A conversation row is written on the first real user message, not on connect. Connection and polling events would otherwise manufacture empty threads faster than anyone could clean them up, and a history list full of ghosts makes the feature look broken.

Prompt injection is defended by double anchoring. Identity is fixed at the top of the system prompt and restated as its last line, with an explicit instruction to ignore override attempts — including ones arriving from the frontend, not just from the user. The frontend is a legitimate participant that declares tools; treating it as automatically trusted would mean an XSS anywhere in the product becomes agent control.

Tool routing is deterministic, with declared precedence. Exact alias beats generic alias; explanatory intent beats navigation; the tool is called once with the canonical key rather than with the user’s phrasing. Letting the model choose freely produced navigation to plausible-sounding screens that did not exist, and passing raw user phrases as keys made the tool’s behaviour depend on how someone happened to word a request.

Escalation to a human is a first-class rule, not a fallback. It has its own policy that differs by reason, plus an explicit prohibition on escalating abusive requests — because the first version escalated them politely and helpfully, straight to a person.

Scale

• ~2,590 lines of source and ~2,650 lines of tests — a ratio above 1:1
• 2 graphs, 5 tools, 4 REST endpoints for conversation management
• Integration tests behind an opt-in marker so continuous integration does not burn model quota