databricks not working with Geoparquet for write data

Question

databricks not working with Geoparquet for write data

Anil Kasar 25

Py4JJavaError: An error occurred while calling o535.save. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 4 times, most recent failure: Lost task 0.3 in stage 17.0 (TID 45) (10.137.237.68 executor 0): java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(Ljava/lang/String;Lorg/apache/hadoop/mapreduce/TaskAttemptContext;)V at org.apache.spark.sql.execution.datasources.parquet.GeoParquetFileFormat$$anon$1.newInstance(GeoParquetFileFormat.scala:162) at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:186) at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:168) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:544) at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:117)

Venkat Reddy Navari 5,815 Reputation points Microsoft External Staff Moderator

2025-08-07T11:33:38.0333333+00:00

Hi Anil Kasar That error looks like a compatibility issue between the GeoParquet library you're using and the version of Spark that comes with your Databricks Runtime. Specifically, the NoSuchMethodError around ParquetOutputWriter usually means the library is expecting a method or class that doesn’t exist in your current Spark version.

A few things you might want to check:

Databricks Runtime version – Not all versions are compatible with every library, especially ones that hook into low-level file I/O like GeoParquet. You can check the Spark version tied to your DBR here: https://docs.databricks.com/release-notes/runtime/index.html

GeoParquet version – If you're pulling the library from GitHub or Maven, make sure it's tested with your Spark/DBR version. Sometimes newer releases drop support for older Spark APIs.

Parquet format support in Databricks – This doc might help clarify how Databricks handles Parquet under the hood: https://learn.microsoft.com/en-us/azure/databricks/query/formats/parquet

Dependency conflicts – If you’re including additional JARs (e.g., for GeoParquet or Parquet libraries), it’s possible there’s a version clash with what Databricks already includes. Worth double-checking what's loaded on your cluster.

As a workaround, you can write geospatial data to standard Parquet files by converting geometries to WKT or WKB format. This avoids the need for GeoParquet and tends to be more reliable across Spark versions.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Accepted answer

1 additional answer

Your answer

Venkat Reddy Navari 5,815 Reputation points Microsoft External Staff Moderator

2025-08-07T11:33:38.0333333+00:00

Hi Anil Kasar That error looks like a compatibility issue between the GeoParquet library you're using and the version of Spark that comes with your Databricks Runtime. Specifically, the NoSuchMethodError around ParquetOutputWriter usually means the library is expecting a method or class that doesn’t exist in your current Spark version.

A few things you might want to check:

Databricks Runtime version – Not all versions are compatible with every library, especially ones that hook into low-level file I/O like GeoParquet. You can check the Spark version tied to your DBR here: https://docs.databricks.com/release-notes/runtime/index.html

GeoParquet version – If you're pulling the library from GitHub or Maven, make sure it's tested with your Spark/DBR version. Sometimes newer releases drop support for older Spark APIs.

Parquet format support in Databricks – This doc might help clarify how Databricks handles Parquet under the hood: https://learn.microsoft.com/en-us/azure/databricks/query/formats/parquet

Dependency conflicts – If you’re including additional JARs (e.g., for GeoParquet or Parquet libraries), it’s possible there’s a version clash with what Databricks already includes. Worth double-checking what's loaded on your cluster.

As a workaround, you can write geospatial data to standard Parquet files by converting geometries to WKT or WKB format. This avoids the need for GeoParquet and tends to be more reliable across Spark versions.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Answer 1

Hi Anil Kasar
Thank you for sharing the details.

The issue you experienced while writing GeoParquet files in Databricks was identified as a bug on the Databricks backend. We coordinated with the Databricks Product Group, and a fix has now been deployed.

You should now be able to write GeoParquet data successfully without any changes on your side. Since you’ve already confirmed that the issue is resolved, no further action is required.

Key points:

Symptom: Error when writing GeoParquet (NoSuchMethodError: ParquetOutputWriter).
Cause: Databricks backend bug related to Parquet output handling.
Resolution: Databricks Product Group deployed a fix.

This issue was also reported in the Apache Sedona GitHub repo here for context. However, the resolution was applied by Databricks directly in the backend, and you should now be able to write GeoParquet files without any code or library changes.

If the problem reappears or you encounter any other issues with GeoParquet, please let us know, and we’ll assist promptly.

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Anil Kasar 25

Thanks @Venkat Reddy Navari ,

Till day before it was working fine , and from yesterday its started giving problem. I have not made any changes in Library and the runtime version.

Venkat Reddy Navari 5,815 Reputation points Microsoft External Staff Moderator

2025-08-07T13:41:36.81+00:00
Anil Kasar Thanks for the update. Since you mentioned the code was working fine until recently and you haven’t made any changes to the library or Databricks runtime, it’s possible that:

A system-level change (e.g., library or dependency update at cluster level) may have occurred.

Cluster restart or auto-upgrade might have refreshed dependency resolution.

A transient issue or corruption in cached dependencies could be affecting execution.

Suggested Next Steps:

Restart the cluster if you haven’t already this often clears transient or cache-related issues.

Check the cluster libraries tab and confirm if any new libraries (JARs or wheels) were auto installed or updated.

Try running the job on a new cluster with the same runtime and settings to rule out cluster-specific problems.

If the issue persists, consider switching to a standard Parquet write by converting geometries to WKT/WKB format temporarily as a workaround.

Let me know if nothing is working out.

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Anil Kasar 25 Reputation points

2025-08-07T14:01:27.2066667+00:00

Hi @Venkat Reddy Navari ,

I restarted compute cupola of time , removed all libraries and added them back. Problem is only for write the geoparquet. This issue is not only one compute and workspace but the other compute and workspace also similar issues where earlier code was working fine but now not able to write data in geoparquet.

I spent day to check all this think and later I added comment for help.
Venkat Reddy Navari 5,815 Reputation points Microsoft External Staff Moderator

2025-08-08T17:28:33.34+00:00

Anil Kasar Could you please share the details I requested in the private message?
Venkat Reddy Navari 5,815 Reputation points Microsoft External Staff Moderator

2025-08-11T12:26:02.7266667+00:00

Anil Kasar Thanks for sharing the details SR# 2508080050000959. We’ll coordinate with our team and get back to you with an update.
Venkat Reddy Navari 5,815 Reputation points Microsoft External Staff Moderator

2025-08-26T12:21:22.4766667+00:00

Anil Kasar The bug has now been fixed by the backend Databricks team. Please try again and confirm if the issue is resolved on your side.
Anil Kasar 25 Reputation points

2025-08-26T14:07:23.18+00:00

hi @Venkat Reddy Navari , The issue has been resolved. Now I am able to write a data in Geo parquet.

Share via

databricks not working with Geoparquet for write data

1 additional answer

Your answer