Lesson 8 of 8 · 12 min
The Production Playbook
A production agent is not simply a published agent. Publishing is opening the front door. Production readiness is making sure the building has locks, alarms, maintenance, owners, and an evacuation plan.
It is a managed system with owners, tests, security boundaries, telemetry, release discipline, and a way to learn from user conversations.
This final lesson brings those parts together.
Governance before access
Document:
- Intended users
- Permitted data sources
- Sensitive fields
- Allowed decisions
- Prohibited uses
- Human-approval requirements
- Retention requirements
- Owners and escalation paths
Microsoft Purview can support auditing, data security posture management, retention, eDiscovery, and policy controls for Copilot and AI interactions.
Fabric Data Agent interaction auditing is currently preview. It requires the applicable Purview Audit capabilities, the DSPM for AI capture policy, and the Fabric tenant setting that allows Purview to secure AI interactions. When configured, it can capture prompts, responses, metadata, and generated query or code.
Purview features do not all share the same release status. For example, Warehouse DLP is generally available, while some access-restriction policies remain preview. Confirm licensing, tenant settings, source coverage, and feature status before treating any of them as a production control.
Security test matrix
Test at least four roles:
| Role | Expected behavior |
|---|---|
| Agent owner | Can configure and evaluate the draft |
| Standard advisor | Sees only permitted student records |
| Restricted user | Cannot access protected rows or columns |
| No-source-access user | Cannot retrieve underlying data |
For every role, test direct questions and indirect attempts to obtain restricted information.
Do not assume a refusal in one wording proves the boundary. Test paraphrases and aggregation requests.
Conversation persistence is an architecture choice
Fabric Data Agent conversation history can persist across user sessions for up to 28 days, depending on geography and tenant settings. Clearing the chat permanently deletes that history.
Even when history is available, do not treat the in-product chat as your durable analytics store. It is a user feature, not a governed product-analytics dataset.
Choose the persistence path based on the objective.
Copilot Studio and Dataverse
Use this when you need ongoing conversation analytics and the supported channel and environment fit your solution. Transcript saving must be enabled in a supported Dataverse environment, and access requires the appropriate transcript roles.
The default transcript retention period is typically 30 days. For longer-term analytics, export or copy the permitted data into a governed store.
Parse conversation transcripts into:
- Session
- Turn
- Prompt
- Response
- Outcome
- Feedback
- Topic
- Latency
- Error category
Purview
Use this primarily for audit, retention, investigation, and compliance workflows. eDiscovery may be required when complete content must be searched or exported.
Programmatic client
A published Fabric Data Agent can expose a preview MCP server endpoint for supported programmatic clients. The management Python SDK is also preview. Check the current interface, authentication model, and support status before choosing an integration.
When your own application calls the agent, log the permitted exchange directly into your governed store.
Capture only what policy allows.
Production observability
Track four layers. No single dashboard provides all of them. Plan to combine Fabric capacity metrics, query diagnostics, evaluation results, application logs, Foundry traces, and your own conversation store.
Platform health
- Availability
- Request count
- Latency
- Capacity pressure
- Tool failures
Query health
- SQL, KQL, or DAX generation success
- Query execution failure
- Empty results
- Timeouts
- Source-selection errors
Answer quality
- Evaluation pass rate
- Unsupported questions
- Appropriate refusals
- User corrections
- Negative feedback
Business value
- Active users
- Repeat usage
- Time saved
- Decisions supported
- Follow-up completion
- Escalations avoided
Foundry traces can help with operational spans, identifiers, status, duration, and tool failures. They do not automatically provide capacity pressure, complete conversation content, or answer quality. Those signals must come from the appropriate platform and application sources.
Token usage and latency do not tell you whether the agent is useful.
Release process
Use this sequence for every significant change:
- Update schema, measures, instructions, examples, or ontology.
- Run the frozen regression set.
- Review changed queries, not only changed wording.
- Test representative security roles.
- Record preview dependencies.
- Publish the new version.
- Monitor the first production sessions.
- Roll back or correct quickly if a critical regression appears.
Production readiness review
Before opening access beyond the implementation team, run a formal readiness review. Treat it like a preflight check. The goal is not to prove that the agent can take off once. It is to show that the complete system can be operated safely, monitored in flight, and landed when something goes wrong.
Bring the following evidence:
Reliability contract
Document the supported users, questions, decisions, boundaries, and measurable success criteria. Be explicit about what the agent must refuse or qualify.
Agent-readiness scorecard
Review the same five categories established in lesson two: meaning, metadata, access, observability, and accountability. Any weak area should have a named owner and a dated remediation plan.
Semantic packet
Keep definitions, measures, grain, joins, instructions, synonyms, and verified examples together. This becomes the shared reference for data owners, agent builders, and evaluators.
Regression dataset
Maintain a frozen set of representative questions with expected answers, expected sources, security roles, and acceptable tolerances. Add production failures to this set before fixing them.
Query evidence
Retain inspected SQL, KQL, and DAX examples for high-value and high-risk questions. Review query behavior whenever schemas, measures, instructions, or models change.
Integration and identity diagram
Show the user, Foundry agent, Fabric Data Agent, knowledge sources, action tools, identity flow, permissions, logging, and failure boundaries.
Production scorecard
| Area | Ready? | Evidence |
|---|---|---|
| Correctness | ||
| Security | ||
| Appropriate boundaries | ||
| Regression testing | ||
| Tool orchestration | ||
| Action controls | ||
| Observability | ||
| Conversation analytics | ||
| Ownership |
Do not approve production because the average score looks good. A critical security, correctness, or ownership gap is a release blocker.
Operating cadence
A production agent needs a recurring operating rhythm.
Weekly
- Review failed and empty-result queries.
- Triage negative feedback and user corrections.
- Check latency, capacity pressure, and tool failures.
- Add confirmed defects to the regression set.
Monthly
- Review unsupported question patterns.
- Reassess instructions, examples, descriptions, and semantic definitions.
- Confirm that owners and escalation paths are still current.
- Compare usage with the business outcomes the agent was intended to support.
Before every significant release
- Re-run the complete regression and security suites.
- Review generated-query changes.
- Validate identity flow and action confirmation.
- Record preview dependencies and known limitations.
- Define rollback criteria before publishing.
Incident response
Treat incorrect high-impact answers, unauthorized data exposure, and uncontrolled actions as production incidents.
- Restrict or unpublish the affected capability.
- Preserve the prompt, response, generated query, identity context, tool trace, and source state.
- Classify whether the failure came from data, semantics, instructions, orchestration, permissions, or the client.
- Correct the root cause rather than adding a narrow wording workaround.
- Add the incident scenario to regression testing.
- Republish only after the relevant security and quality checks pass.
The fastest fix is not always the safest release. Production discipline means knowing when to stop the system, not only how to improve it.
The complete playbook
The entire course reduces to eight moves:
- Define the decision and reliability contract.
- Prepare agent-ready data.
- Build the trusted semantic layer.
- Create and publish a bounded Fabric Data Agent.
- Inspect queries and evaluate against ground truth.
- Add an ontology only when relationships, rules, and actions require it.
- Use Foundry to orchestrate Fabric analytics with knowledge and actions.
- Operate the system through governance, telemetry, and conversation learning.
The model is not the product.
The product is the full system that lets a user move from a question to a trusted answer, and from a trusted answer to a controlled outcome.
You now have the architecture and the operating method to build that system.