Submit feedback on
Inefficient BI Queries Driving Excessive Compute Usage
We've received your feedback.
Thanks for reaching out!
Oops! Something went wrong while submitting the form.
Close
Inefficient BI Queries Driving Excessive Compute Usage
Scott Shulman
Service Category
Compute
Cloud Provider
Databricks
Service Name
Interactive Clusters
Inefficiency Type
Inefficient Query Patterns
Explanation

Business Intelligence dashboards and ad-hoc analyst queries frequently drive Databricks compute usage — especially when: * Dashboards are auto-refreshed too frequently * Queries scan full datasets instead of leveraging filtered views or materialized tables * Inefficient joins or large broadcast operations are used * Redundant or exploratory queries are triggered during interactive exploration This often results in clusters staying active for longer than necessary, or being autoscaled up to handle inefficient workloads, leading to unnecessary DBU consumption.

Relevant Billing Model

Databricks usage is billed per DBU (Databricks Unit) per hour, depending on the cluster type and size. BI dashboards and ad-hoc SQL queries can keep interactive or all-purpose clusters running, consuming DBUs even when queries are poorly optimized or overly frequent.

Detection
  • Review query logs from SQL Analytics or interactive clusters to identify:
  • Query patterns that scan large volumes of data repeatedly
  • Dashboards with auto-refresh settings that exceed business needs
  • Repeated ad-hoc queries with similar structure but different filters
  • Long-running queries due to unoptimized joins, filters, or aggregations
  • High DBU usage attributed to BI users or SQL endpoints with low result value
Remediation
  • Refactor BI queries to limit scan scope and reduce complexity
  • Materialize frequently used intermediate results into temp or Delta tables
  • Reduce auto-refresh frequency of dashboards unless real-time data is essential
  • Educate analyst teams on query efficiency and impact on backend costs
  • Enable query result caching or switch to serverless SQL warehouses if feasible
Submit Feedback