We Built an AI Agent That Knows When It Doesn’t Know

Read time:
3 minutes

Patrick Farren

We recently completed a project for Joulen that moved support investigations from hours of manual work down to about 30 seconds. By building an agentic system that sits directly on top of their data in KDB-X, we turned a slow, manual escalation process into an automated diagnostic tool.

The Support-to-Dev Bottleneck

The workflow we addressed is one many engineering teams recognise. A support ticket comes in about battery telemetry: “Why is Site 133 not charging?” The support team doesn’t have direct database access or the technical depth to diagnose the root cause, so the ticket is escalated to the dev team.

A developer then has to stop feature work, context-switch, query the database, and dig through historical records to find similar past incidents. As the number of sites grew for Joulen, this investigation cost was becoming a significant drain on engineering resources.

Agentic Reasoning, Not Just a Chatbot

We avoided building a simple chatbot wrapper. Instead, we built an agentic system that reasons through a problem and iterates until it has evidence to support its conclusion.

The core is an orchestration loop. When a query comes in, the system decides which tools it needs to call via Model Context Protocol (MCP). It might take two iterations or ten, but the logic is enforced in the code rather than relying solely on a prompt.

The architecture relies on two key components:

KDB-X MCP: Queries time-series telemetry data and runs RAG (Retrieval-Augmented Generation) semantic searches over historical tickets and documentation.
Postgres MCP: Resolves site IDs to device identifiers. This ensures the agent correctly identifies the battery and meter IDs associated with a site before it attempts to query the telemetry.

KDB-X: The One-Stop Shop

A central part of this architecture was using KDB-X for both structured telemetry and unstructured vector embeddings.

We bypassed the need for a separate vector database. Instead, a nightly indexer processes documentation and stores 768-dimensional embeddings directly in KDB-X. This allows cosine similarity searches to run natively in q.

By keeping the telemetry and the historical context in one place, we kept the infrastructure footprint small and the query performance high. Proper optimisation of the HDB partitions made a significant difference in execution speed, ensuring the agent provides answers in near real-time.

Security and Local Deployment

The system supports OpenAI, Anthropic, and local models (such as qwen3 or mistral) via Ollama. This flexibility is essential in regulated environments. Running local models means sensitive production data never leaves the internal infrastructure, satisfying compliance requirements that often block external API calls.

We ensured the architecture remains robust regardless of the provider by strictly enforcing tool calling capabilities. If a model can’t interact with the databases accurately, the agentic workflow fails, so we built the system to validate these interactions at every step.

The Confidence Calculator

LLMs can sound authoritative even when they are incorrect. In a production support context, a hallucinated answer about telemetry doesn’t just waste time, it leads to wrong decisions. We needed the system to be honest about its own uncertainty.

We built a five component scoring system to evaluate every response that production support can examine:

Tool Execution (30%): Did the database queries actually return data?
Error Recovery (25%): Did the agent struggle or fail repeatedly to get a result?
Iteration Efficiency (20%): Did the agent find the answer directly, or did it “thrash” through too many steps?
Data Quality (15%): How strong was the similarity score for historical matches?
Language Analysis (10%): Is the model using hedging language like “possibly” or “unclear”?

The output is a simple score: HIGH (75%+), MEDIUM (50-74%), LOW (below 50%). The support agent sees this and knows instantly whether to trust the diagnosis or escalate the ticket to a developer.

Early Feedback from the Floor

Testing is still in its early stages, but the response from Joulen’s support team has been very promising.

On how the agent presents its work:

“I like the way it clearly details each action completed as part of the work it did, as this helps me trust the response that follows.“

On a scenario with a known root cause (missing telemetry data):

“The Agent correctly identified the root cause of the issue, described it well, and provided sensible troubleshooting options to follow up on.“

On a harder case with no clean solution:

“The actionable insights clearly demonstrated an understanding of the problem and the related business logic. Follow-up actions offered by the AI were sensible, correct, and closely matched the first steps usually taken by L2 support.“

Transferring the Pattern to Capital Markets

While this specific implementation solved a renewable energy challenge, the pattern is domain-agnostic. The agentic logic, MCP orchestration, and confidence scoring are not tied to any one industry.

In Capital Markets production support, the same architecture applies:

The KDB-X MCP queries trade data, order books, and risk metrics instead of battery telemetry.
RAG searches over incident reports and post-mortems instead of energy tickets.
The Postgres MCP resolves application names to service registries or deployment metadata.

The query changes from “Why is Site 133 not charging?” to “Why are FX order rejections spiking since the latest release?” The underlying reasoning engine remains the same.