Lesson 1 of 8 · 8 min
Why Data Agents Fail Outside the Demo
A good demonstration usually starts with a question the builder already knows will work. It is the equivalent of testing a navigation app on the one route you drive every day.
A real user takes the unmarked side road. They shorten terms, mix business language with technical language, omit important filters, ask for a number that has three competing definitions, and expect the system to understand what they meant.
That gap is why many data agents look excellent in a demo and disappoint in production.
Three systems that are often confused
Before discussing reliability, separate three different products people casually call an agent.
A chatbot
A chatbot manages a conversation. It may answer from instructions or retrieved documents, but conversation alone does not make it a data agent.
A data agent
A data agent answers questions from governed data. In the generally available standard runtime, a Fabric Data Agent can generate T-SQL, KQL, or DAX for supported structured Fabric sources such as Warehouses, Lakehouse SQL analytics endpoints, KQL databases, Power BI semantic models, SQL databases, and mirrored databases.
Preview source options can include Graph Models, ontologies, Azure AI Search, and other evolving integrations. Those sources do not all use the same query path, so check the current documentation before designing around them.
The defining behavior is not simply that the agent can talk. For structured analytical sources, it translates a question into a source-native query, runs that query under the user’s access context, and grounds the response in the result.
An orchestration agent
An orchestration agent coordinates a broader task. A Microsoft Foundry agent can decide which tool to call, combine multiple knowledge sources, invoke APIs, and manage a multi-step workflow.
The distinction matters because each system fails differently.
- A chatbot fails when retrieval or instructions are weak.
- A data agent fails when business meaning, query generation, permissions, or source data are weak.
- An orchestration agent fails when planning, tool selection, state, or action controls are weak.
What reliable means
For this course, a reliable data agent must satisfy five tests. Think of them as the legs of a table. If one is missing, the experience may still look polished, but it will not stand up under real use.
- Correctness: the answer matches the governed source.
- Consistency: equivalent questions produce equivalent results.
- Explainability: you can inspect how the answer was produced.
- Security: the agent cannot reveal data the user cannot access directly.
- Boundaries: the agent recognizes questions it cannot answer safely.
Notice what is missing from this list: eloquence.
A polished explanation of the wrong number is still wrong. Reliability begins before wording.
The most common production failures
Ambiguous business language
“Active customer” might mean a customer with an open contract, a purchase in the last 90 days, or a nonzero balance. If the definition is not encoded in the source or instructions, the agent has to guess.
Weak schemas
Cryptic names such as cust_st_cd and amt_01 make query generation harder. A human analyst may know what they mean. An agent only sees the available schema and descriptions.
Missing joins
The agent can select the correct tables and still produce the wrong result if the join path is unclear or many-to-many relationships are not handled correctly.
No evaluation set
Teams often test by chatting until several answers look reasonable. That is exploration, not evaluation. Without expected answers, there is no objective definition of improvement.
Security tested too late
An agent may work perfectly for its builder and fail for users with restricted access. Reliability has to include role-level testing from the beginning.
Asking analytics to perform causality
A query can show where a metric changed and which factors correlate with it. That does not automatically prove why the change occurred. Root-cause and causal questions often require additional analysis, experiments, or domain evidence.
Define the job before the agent
Choose one business workflow. Keep it narrow enough that you can name the users, decisions, and source of truth without writing a strategy document.
Write down:
- The people who will use the agent
- The decisions they are trying to make
- Ten questions it must answer
- Five questions it must refuse or redirect
- The source of truth for every expected answer
- The permissions each user role should have
This is the first deliverable of the course. A useful contract should fit on one page. If it cannot, the scope is probably still too broad.
Do not start by asking, “What can the agent do?”
Start with, “Which decisions must become faster without becoming less trustworthy?”
Exercise: write the reliability contract
Create a one-page contract with these headings. Use real roles and real questions wherever possible, even if you are working with sample data.
| Area | Your definition |
|---|---|
| Primary users | Who will ask questions? |
| Supported decisions | What will answers help them decide? |
| In-scope data | Which governed sources can be used? |
| Correctness target | How will you determine that an answer is correct? |
| Security roles | Which users should see which data? |
| Refusal boundary | Which questions should not be answered? |
| Evidence | What query, source, or citation should be visible? |
When you finish, ask a data owner and an intended user to read it. If they interpret a supported decision or refusal boundary differently, resolve that disagreement now.
You will use this contract throughout the course. It is the guardrail that prevents the project from becoming an endless attempt to make the agent answer everything.
Carry this forward
A Fabric Data Agent is not a replacement for a well-designed data platform or a skilled analyst. It is a new interface over that foundation.
If the foundation is ambiguous, the agent exposes the ambiguity faster.
In the next lesson, we will assess whether the data is ready to support the reliability contract you just created.