Submit feedback on
Overuse of Photon in Non-Production Workloads
We've received your feedback.
Thanks for reaching out!
Oops! Something went wrong while submitting the form.
Close
Overuse of Photon in Non-Production Workloads
Nicole Boyd
Service Category
Compute
Cloud Provider
Databricks
Service Name
Databricks Compute
Inefficiency Type
Inefficient Configuration
Explanation

Photon is frequently enabled by default across Databricks workspaces, including for development, testing, and low-concurrency workloads. In these non-production contexts, job runtimes are typically shorter, SLAs are relaxed or nonexistent, and performance gains offer little business value.

Enabling Photon in these environments can inflate DBU costs substantially without meaningful runtime improvements. By not differentiating cluster configurations between production and non-production, organizations may pay a premium for workloads that could run just as efficiently on standard compute.

Cluster policies can be used to restrict Photon usage to explicitly tagged production workloads, helping enforce cost-conscious defaults and reduce unnecessary spend.

Relevant Billing Model

Photon-enabled compute clusters incur a higher Databricks Unit (DBU) rate compared to standard compute. While the performance boost can be significant for complex workloads, the DBU multiplier — which can reach approximately 2.9x — may not be justified in development, staging, or ad-hoc environments where performance is less critical. This results in unnecessary cost when Photon is enabled for non-production workloads that don't benefit from its capabilities.

Detection
  • Identify compute clusters or job configurations with Photon enabled in dev, test, or sandbox environments
  • Review naming conventions, tags, or workspace metadata to isolate non-production workload
  • Analyze job characteristics: short runtimes, low concurrency, small data volumes, or non-SLA workloads using Photon
  • Compare cost and runtime of similar jobs executed with and without Photon enabled
  • Review cluster policies and workspace defaults to determine whether Photon is disabled by default for non-prod
Remediation
  • Disable Photon by default in dev/test environments using workspace settings or cluster policies
  • Create separate cluster templates or policies for production and non-production workloads
  • Use tagging or automation to flag or block Photon usage in low-priority environments
  • Educate teams on the cost-performance tradeoffs of Photon and when its use is justified
  • Periodically review Photon usage by environment and update governance rules accordingly
Relevant Documentation
Submit Feedback