Lesson 6 of 8 · 11 min
Guardrails: Surge Protection, Overage, and Isolation
A well-sized capacity still has bad days. A new notebook runs against the wrong table, a quarter-end refresh lands on a busy morning, a team publishes a report that every manager opens at once. The question this lesson answers is: when demand runs ahead of supply, whose work should suffer, and what are you willing to pay to prevent it?
Fabric gives you four guardrails. They solve different problems and cost different things. They work best when you choose them deliberately rather than discover them during an incident.
Capacity-level surge protection
By default, a capacity allows background work to consume up to the full 24-hour budget before it rejects new background operations. By then, interactive users are usually already being delayed or rejected.
Capacity-level surge protection lets a capacity admin reject new background operations earlier. You set two thresholds against the capacity’s 24-hour background percentage:
- A rejection threshold. When background usage reaches it, new background operations are rejected.
- A recovery threshold. When background usage drops below it, background operations are accepted again.
Set them from evidence. On the Capacity Metrics app Compute page, compare the Background rejection chart with the Interactive rejection chart. If interactive rejection crossed 100% when background reached, say, 75%, a rejection threshold somewhere between your normal background peaks and that level protects interactive users. Microsoft’s guidance includes worked scenarios for choosing both values.
Know the limits:
- It rejects background work, which might be exactly the refresh someone needs. Make sure job owners know it exists and how to read the rejection.
- It is not an upper limit, because in-progress jobs keep running and reporting usage.
- It does not block operations billed through on-demand billing for Spark.
- If 80% or more of your utilization is background work, limiting background work might not help much. Fix the background work instead.
Workspace-level surge protection (preview)
Capacity-level protection treats all background work the same. Workspace-level surge protection lets you cap how much compute any single workspace can consume in a rolling 24-hour window, as a percentage of the capacity.
Each workspace has one of three states:
| State | Behavior |
|---|---|
| Available | Default. Subject to surge protection and can be blocked automatically when it exceeds its limit |
| Mission critical | Exempt from surge protection rules, though normal capacity throttling still applies |
| Blocked | Rejects all interactive and background operations until an admin unblocks it or the block period ends |
This is the most precise guardrail for a shared capacity. A runaway experiment in one workspace is blocked before it damages everyone else, while the executive reporting workspace is marked mission critical.
Because it is in preview, test it on a non-production capacity first, and agree with workspace owners in advance what happens when their workspace is blocked and who can unblock it.
Capacity overage
Capacity overage takes the opposite approach. Instead of rejecting work, it pays for excess usage that would otherwise throttle, so operations keep running.
The details matter:
- Overage is billed at three times the pay-as-you-go rate, on a separate meter in Azure Cost Management.
- It activates when the capacity reaches the interactive delay stage, and pays off the carryforward at that point.
- You set a rolling 24-hour threshold in CU-hours. When it is reached, overage stops and normal throttling resumes.
- The threshold is not a hard spending cap. Fabric evaluates it every five minutes and running operations continue, so charges can exceed it.
- It needs Fabric quota equal to one twenty-fourth of the threshold.
- It does not make the capacity faster. It only prevents throttling.
- Microsoft enables it by default when a capacity is created.
That last point deserves a check today. If your capacity has overage enabled and you did not choose it, you have an unreviewed spending policy. It can be the right policy, but it should be a decision.
Overage fits short, unpredictable spikes where continuity is worth a premium. It does not fit sustained demand. If overage charges appear most weeks, a larger SKU or better isolation is usually cheaper.
Isolation with separate capacities
Some workloads are efficient and still should not share a failure boundary. Executive reporting, self-service experimentation, and a heavy overnight engineering platform have different service expectations. If they share a capacity, one team’s mistake becomes another team’s incident.
Moving workspaces to separate capacities gives each its own budget, its own throttling, and its own owner. It also adds cost, because each capacity needs its own headroom, and it adds administrative work. Use it when the service-level difference is real.
When you move workspaces, remember that reassigning a workspace to a different capacity cancels its running jobs. Plan the move for a quiet window.
On-demand billing for Spark
On-demand billing for Spark, previously called Autoscale Billing for Spark, moves Spark jobs off the shared capacity onto a separately billed, serverless pay-as-you-go meter. You set a maximum CU limit and pay only for what Spark uses.
It fits capacities where bursty Spark work competes with reporting or warehouse queries. It changes the cost model, so compare it against a measured Spark baseline from the Capacity Metrics app rather than enabling it by instinct.
Choosing your guardrails
| Problem | Best first guardrail | What it costs |
|---|---|---|
| Background work starves interactive users | Capacity-level surge protection | Rejected background jobs at peak |
| One workspace can damage the whole capacity | Workspace-level surge protection | Blocked workspaces, admin attention |
| Rare, short spikes where continuity matters | Capacity overage with a modest threshold | Premium-rate charges during spikes |
| Workloads with very different service levels | Separate capacities | More capacities to fund and run |
| Spark bursts compete with everything else | On-demand billing for Spark | A separate, variable Spark bill |
These are not exclusive. A common pattern for a shared production capacity is workspace-level limits for self-service workspaces, mission-critical status for a few governed workspaces, capacity-level surge protection as a backstop, and a small overage threshold for genuine emergencies.
Exercise: write your guardrail policy
For each capacity you own, record the decision and the reason.
| Guardrail | On or off | Setting | Reason | Who is notified when it acts |
|---|---|---|---|---|
| Capacity-level surge protection | ||||
| Workspace-level surge protection | ||||
| Mission-critical workspaces | ||||
| Capacity overage and threshold | ||||
| On-demand billing for Spark and limit | ||||
| Isolation from other capacities |
If any row says “on” but nobody can explain the reason, review it this week.
Carry this forward
Guardrails decide what happens on bad days. The next lesson covers the ordinary days: matching what you pay to the shape of your usage with reservations, scheduled resizing, and pausing.