Inspecting and Evaluating Answers

The final sentence is the least useful place to begin debugging a data agent. The answer is the receipt. The generated query is the itemized bill that shows how the total was produced.

Start with the query.

A Fabric Data Agent routes the question to a source, uses the applicable query or retrieval path, executes it under the available access context, and converts the result into a response.

The source determines the language.

• Warehouse, SQL database, mirrored database, and Lakehouse SQL analytics endpoint sources use T-SQL.
• Eventhouse and KQL databases use KQL.
• Power BI semantic models execute DAX through XMLA.

Preview sources use their documented paths. For example, Graph Models can use GQL, while Azure AI Search uses retrieval rather than the SQL, KQL, or DAX flow described in this lesson.

Understanding this path turns agent testing from subjective conversation review into data engineering.

Inspect SQL

For a SQL answer, review:

Selected tables

Did the agent choose the trusted gold table or an implementation table with duplicate history?

Join path

Did it join at the correct grain? A student joined to multiple risk assessments without a current-record filter may be counted several times.

Filters

Did it apply term, status, test-record, and date filters correctly?

Aggregation

Should the query use COUNT(*), COUNT(DISTINCT StudentId), SUM, or a precomputed measure?

Null handling

Did missing values silently disappear or change the denominator?

Inspect KQL

For KQL, review:

• Time range
• Timestamp field
• Event type filters
• Entity identifiers
• summarize grain
• Windowing
• Ordering and limits

A correct KQL pattern with the wrong time field can return a confident but irrelevant answer.

Inspect DAX

For DAX, review:

• Measure selection
• Filter context
• Date table usage
• Relationship behavior
• Time-intelligence assumptions
• Security context

If a trusted measure already exists, the generated DAX should usually rely on it rather than reconstructing business logic from columns.

Use the run details and diagnostics

The authoring experience exposes generated code and intermediate execution information. Use the run-step view to understand which configuration or examples influenced the answer.

Diagnostics can export configuration and execution details for deeper investigation.

This evidence helps distinguish:

• A data problem
• A schema-description problem
• An instruction problem
• An example-query problem
• A query-generation limitation
• A permissions problem

Build the ground-truth set

A useful evaluation dataset contains:

Field	Purpose
Question	Exact user wording
Expected answer	Trusted result or acceptable range
Expected source	Source the agent should use
Critical filters	Filters that must appear
Allowed variation	Wording or rounding differences that are acceptable
User role	Security context for the test
Category	Direct, paraphrase, ambiguous, restricted, unsupported

Start with 20 to 30 questions.

Use a balanced set:

• 8 direct metric questions
• 6 paraphrases
• 4 multi-filter questions
• 4 ambiguous questions
• 4 restricted or unsupported questions

Evaluate systematically

The preview Fabric Data Agent SDK provides evaluation capabilities for comparing agent answers with expected answers. You can use a question-and-expected-answer dataset, review summary and detail results, and refine the evaluation criteria.

The SDK and evaluation capabilities are preview and their API paths can change. Verify the current documentation before making this pipeline a production dependency.

Whether you automate immediately or begin manually, keep the process identical:

1. Freeze the test set.
2. Run every question.
3. Capture generated queries and answers.
4. Mark failures by category.
5. Change one configuration area.
6. Run the full set again.

Diagnose instead of prompt-tuning blindly

Use failure categories.

Wrong source

Reduce source overlap or clarify source-selection instructions.

Wrong field or measure

Improve descriptions, AI schema, or measure names.

Wrong join

Add an approved query example or simplify the exposed schema.

Missing filter

Encode the business rule explicitly and add a regression question.

Unsupported reasoning

Change the boundary. Do not force a query agent to perform causal inference or advanced forecasting it was not designed to perform.

Security mismatch

Correct permissions or role configuration before changing wording.

Exercise: run the first regression

Create a baseline report:

Metric	Result
Questions tested
Correct answers
Correct source selected
Correct query pattern
Appropriate refusals
Security tests passed

Record the three highest-frequency failure categories and assign one improvement to each.

Then rerun the entire set. Change one major configuration area at a time. If you change the schema, instructions, examples, and measures together, you will not know which change improved the result or introduced the next failure.

The goal is not a perfect score after one iteration. The goal is a test process that makes progress visible and prevents old failures from returning.

Carry this forward

Once the agent can answer bounded analytical questions reliably, users will ask it to cross domains, explain relationships, and recommend actions.

That is where the distinction between semantic models and ontologies becomes important.

Microsoft references

• Evaluate a Fabric data agent
• Develop an iterative improvement process
• Fabric data agent runtime