Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
2 answers
Scheduling Full Scans in Purview for Databricks Unity Catalog
How can a full scan be scheduled in Purview for a Databricks Unity Catalog data source every other day? The documentation indicates that only an initial full scan occurs upon scheduling, followed by incremental scans. However, the incremental scans are…
Azure Databricks
asked 2025-08-27T19:07:27.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 0%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JS
0
Reputation points
commented 2025-08-29T20:34:26.0133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
29,500
Reputation points Microsoft External Staff
Moderator
0 answers
Azure quota exceeded exception databricks
While trying to create Compute in Databricks I encountered the error Azure quota exceeded exception. How to fix this?
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 61%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
0 answers
unable to create compute in databricks
Hi Team i am not able to create compute in databricks is showing error.
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-29T18:07:12.8966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 96%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gunjan Seth
0
Reputation points
commented 2025-08-29T18:08:10.7233333+00:00
Gunjan Seth
0
Reputation points
1 answer
Getting PermissionError from workers trying to create a "faulthandler_log_file"
While running ".foreach()" on a dataframe, I keep getting the following error: Py4JJavaError: An error occurred while calling o13961.javaToPython. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4917.0…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-27T12:11:49.67+00:00
Bjarne Thorsted
0
Reputation points
commented 2025-08-29T17:30:32.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 16%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Kalyani Kondavaradala
1,015
Reputation points Microsoft External Staff
Moderator
1 answer
How to enable metastore in azure databricks.
I have made a new account with pay as you go subscription. Also added an external user with email having organisation domain not a personal email. Felt like azure entra was not letting me get in. Now after creating VPCs and launch databricks is fine. But…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-29T15:26:51.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 51%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKU%3C/text%3E%3C/svg%3E)
Kemal Uigar
0
Reputation points
commented 2025-08-29T16:50:18.09+00:00
Kemal Uigar
0
Reputation points
0 answers
JDBC batch vs TVP ingestion into Azure SQL Hyperscale
We are currently inserting and updating data into Azure SQL Database Hyperscale using JDBC batch updates. Our process is: Read data from ADLS raw layer Use JDBC batch insert/update to load data into the Hyperscale main table We have observed that…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-29T09:00:53.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 56.00000000000001%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Janice Chi
620
Reputation points
edited a comment 2025-08-29T12:16:45.34+00:00
Smaran Thoomu
29,500
Reputation points Microsoft External Staff
Moderator
0 answers
Databricks → Azure SQL Hyperscale streaming MERGE scale-out pattern
We are implementing a Kafka → Databricks → Azure SQL Hyperscale streaming ingestion pipeline and would like your guidance. Our scenario Scale: 200 Kafka topics → 200 Databricks Structured Streaming jobs (one per topic). Volume: ~200,000 events per…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-29T09:11:46.05+00:00
Janice Chi
620
Reputation points
commented 2025-08-29T11:52:30.97+00:00
Janice Chi
620
Reputation points
1 answer
Auto loader and schema changes
Hi kind friends, we are implementing auto loader to track schema changes , and schema drift to be precise. I understand , it is a best practice to implement auto loader at landing to bronze layer for capturing the data anamolies at the outset , and at…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-28T18:05:19.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
755
Reputation points
edited a comment 2025-08-29T09:47:08.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 72%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Swapnesh Panchal
5
Reputation points Microsoft External Staff
Moderator
0 answers
How to change cluster location from EASTUS to any other, Having issue with creating compute
Hi Team, Need help, Im trying to create compute but getting error as below 'loud Provider Resource Stockout: The VM size you are specifying is not available. [details] SkuNotAvailable: The requested VM size for resource 'Following SKUs have failed for…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-29T07:15:59.2166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 94%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Naveen Sequeira
0
Reputation points
commented 2025-08-29T09:28:09.5433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVP%3C/text%3E%3C/svg%3E)
VRISHABHANATH PATIL
85
Reputation points Microsoft External Staff
Moderator
0 answers
Unable to start job runs and start cluster in Azure Databricks
Every time I start a workflow/job, I get the error message: "Triggering new runs for organization [org_number] is currently disabled temporarily." Every time I start any cluster, I get the error message: "Could not start cluster…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-07-30T22:19:04.2066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 63%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kian Baula
0
Reputation points
commented 2025-08-28T16:54:21.8466667+00:00
Oury Ba-MSFT
21,111
Reputation points Microsoft Employee
Moderator
1 answer
Cost Comparison Between Databricks, ADLS (Archive/Cold), and Azure SQL Hyperscale
Databricks Serverless SQL Warehouse + ADLS Archive Tier Databricks Serverless SQL Warehouse + ADLS Cold Tier Azure SQL Database Hyperscale (Serverless) How should we do a fair, apple-to-apple compute comparison between Databricks Serverless and Azure…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-28T13:01:24.7466667+00:00
Janice Chi
620
Reputation points
answered 2025-08-28T14:03:08.9566667+00:00
Smaran Thoomu
29,500
Reputation points Microsoft External Staff
Moderator
1 answer
There is no model databricks-claude-sonnet-4 in the list of Foundation Model APIs models
I have created the Databricks workspace of Premium tier in East US region. According to documentation there are following Foundation Model APIs models available: databricks-gemma-3-12b databricks-claude-sonnet-4 databricks-claude-opus-4 …
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-07-30T13:28:51.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 7.000000000000001%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Michal
0
Reputation points
commented 2025-08-28T12:22:53.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 98%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Anish Bista
0
Reputation points
2 answers
One of the answers was accepted by the question author.
databricks not working with Geoparquet for write data
Py4JJavaError: An error occurred while calling o535.save. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 4 times, most recent failure: Lost task 0.3 in stage 17.0 (TID 45) (10.137.237.68 executor 0):…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-07T10:50:36.78+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 52%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Anil Kasar
25
Reputation points
accepted 2025-08-28T04:45:42.8433333+00:00
Anil Kasar
25
Reputation points
1 answer
One of the answers was accepted by the question author.
cluster on databricks
I am trying to deploy a cluster on databricks but Standard_DS3_v2 is not available in central india region
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-24T13:09:29.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 93%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Aditya Mishra
20
Reputation points
accepted 2025-08-28T04:17:45.2833333+00:00
Aditya Mishra
20
Reputation points
0 answers
Azure Databricks Sonnet Access
Hi, I’ve set up Databricks via Azure, but I can’t access the Anthropic models. How can I enable them or request access? I don’t have an Azure support plan.
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-25T15:22:08.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 61%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMF%3C/text%3E%3C/svg%3E)
Manuel Ferrigno
0
Reputation points
commented 2025-08-26T21:33:04.5866667+00:00
Smaran Thoomu
29,500
Reputation points Microsoft External Staff
Moderator
0 answers
can not create or close cluster
I created cluster and was using databricks for more than a month. Now when i try to start my cluster to run a notebook , it is showing error as can not create cluster. I have tried troubleshooting but no use. Hoping to find solution here.
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-04-22T20:55:26.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 18%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Ankitha
0
Reputation points
commented 2025-08-26T13:36:31.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
24,096
Reputation points
Volunteer Moderator
1 answer
Azure Devops pipeline and dataset parametrization issue
I have added parametrization for ADF pipelines but yet I don't see the parameters reflected in the override parameters list. Tried reconnecting the adf connection to git and also overwriting the live mode from source branch and published the adf…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-04T13:11:45.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 75%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDM%3C/text%3E%3C/svg%3E)
Deepa M
20
Reputation points
commented 2025-08-26T12:25:55.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari
5,815
Reputation points Microsoft External Staff
Moderator
1 answer
One of the answers was accepted by the question author.
Data Pipeline to push files from External System (Adobe) to ADLS Gen2
Hello, May someone please share detail step by step for below scenario. Adobe Campaign push Email Delivery and Email Tracking data objects (2 files) to File Storage on Bronze Layer. which component I have to use to for polling and grab the new file and…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 30%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
1 answer
DATABRICKS VS ADF for Recon
We are in the process of implementing the one-time historical migration from DB2 (80 TB total) to Azure SQL Hyperscale. Our current focus is on the CD History table (~18 TB) and many others totl 800 tables , few tables are partitioned into 30–40…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-25T09:11:44.78+00:00
Janice Chi
620
Reputation points
commented 2025-08-26T11:40:55.6366667+00:00
Venkat Reddy Navari
5,815
Reputation points Microsoft External Staff
Moderator
1 answer
Setup UC Metastore in Gov Cloud.
I am trying to setup/configure Unity Catalog in our Azure Databricks Service in gov cloud. There's no UI support to do this in gov cloud so I am attempting to do it via the CLI. To use the CLI, I will first need to authenticate via a service principal.…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,653 questions
asked 2025-08-25T18:42:05.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 44%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Ryan De Four
0
Reputation points
commented 2025-08-26T03:45:43.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 13%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Pratyush Vashistha
1,525
Reputation points Microsoft External Staff
Moderator