Skip to content
plainsight.pro

Autoscale Billing for Spark in Microsoft Fabric

Purpose

Autoscale Billing for Spark enables serverless, pay-as-you-go compute for Spark workloads in Microsoft Fabric. Once enabled, Spark jobs no longer consume the shared capacity, giving you the flexibility to scale Spark workloads independently and optimize costs.

Requirements

Requirement Details
Capacity Only available for Fabric F-SKUs (F2 and above). Not supported on P-SKUs or Fabric Trial capacities.
Access You must be a Fabric Capacity Administrator to configure Autoscale Billing.

Important

Enabling, disabling, or reducing the Maximum Capacity Units will cancel all active Spark jobs running under Autoscale Billing to avoid billing overlaps.

How to Configure Autoscale Billing for Spark

Follow these steps to enable and manage Autoscale Billing settings for a Fabric capacity:

  1. Navigate to the Microsoft Fabric Admin Portal

  2. Under the Governance and insights section, select Admin portal

  3. Access Capacity Settings

  4. From the left-hand menu, choose Capacity settings and go to the Fabric capacity tab

  5. Select the capacity you want to configure

  6. Enable Autoscale Billing

  7. In the Capacity Settings page, scroll to the Autoscale Billing for Fabric Spark section

  8. Enable the Autoscale Billing toggle

  9. Set Maximum Capacity Units (CU)

  10. Use the slider to set the Maximum Capacity Units (CU) you want to allocate to Spark jobs
  11. The maximum limit available is based on:
    • The quota approved for your Azure subscription
    • The type of subscription you're using
  12. You are only billed for the compute used, up to this limit

  13. To increase quota, see Request additional quotas below

  14. Save Your Settings

  15. Click Save to apply your configuration
  16. Your Spark Pools can now utilize the new CU quota set by Autoscale Billing

tip

Once Autoscale Billing is enabled, consider whether you can resize your shared Fabric capacity down (see Optional: Resize and reset capacity below), since Spark workloads are now billed separately.

Optional: Resize and Reset Capacity for Cost Optimization

After enabling Autoscale Billing, you can downsize your shared Fabric capacity if Spark workloads are no longer using it. This reduces the cost of your base capacity:

  1. Go to the Azure Portal

  2. Search for and select your Fabric capacity

  3. Reset Capacity Usage

  4. Click Pause to temporarily stop the capacity
  5. This clears any active or unsmoothed Spark usage on the shared capacity

  6. Wait 5 minutes, then click Resume to restart the capacity

  7. Resize to a Lower SKU

  8. Resize the capacity to a lower SKU that fits your remaining workloads (e.g., Power BI, Data Warehouse, Real-Time Intelligence, Databases)

Note

Only Azure administrators can resize SKUs. This change is made in the Azure portal, not within Fabric settings.

Monitor Billing and Usage

After enabling Autoscale Billing, use Azure's built-in cost management tools to track compute usage:

  1. Navigate to Azure Cost Analysis
  2. Go to the Azure portal
  3. Select the Subscription linked to your Fabric capacity

  4. In the subscription page, go to Cost Analysis

  5. Filter for Spark Usage

  6. Filter by the resource (Fabric capacity)
  7. Look for the meter: Autoscale for Spark Capacity Usage CU
  8. View real-time compute spend for Spark workloads using Autoscale Billing

Request Additional Quotas

If your data engineering or data science workloads require a higher quota than your current maximum Capacity Unit (CU) limit, you can request an increase:

  1. Navigate to Azure Quotas
  2. Go to the Azure portal and sign in

  3. In the search bar, type and select Azure Quotas

  4. Request a Quota Increase

  5. Choose Microsoft Fabric from the list of available services
  6. Select the subscription associated with your Fabric capacity
  7. Edit the quota limit by entering the new CU limit you need

  8. Submit your quota request

  9. Wait for Approval

  10. Once approved, the new CU limits will be refreshed and applied to your Fabric capacity
  11. This ensures your Autoscale Billing model can accommodate increased demand without interrupting Spark workloads