Databricks Runtime 16.0 (EoS)

The following release notes provide information about Databricks Runtime 16.0, powered by Apache Spark 3.5.0.

Databricks released this version in November 2024.

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support and end-of-life history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Behavioral changes

Breaking change: JDK 17 is now the default

In Databricks Runtime 16.0 and above, the default JDK version is switched from JDK 8 to JDK 17. This change is made because of the planned deprecation and end-of-support for JDK 8. This affects the following:

  • Java code running on Azure Databricks compute must be compatible with Java 17.
  • Scala code running in notebooks or Azure Databricks compute must be compatible with Java 17.
  • Java and Scala libraries installed on compute must be compatible with Java 17.
  • Versions of the Apache Hive metastore client below 2.x. Setting the Spark configuration spark.sql.hive.metastore.version to a version lower than 2.x will cause compatibility issues with Java 17 and connection failures to the Hive metastore. Databricks recommends upgrading Hive to a version above 2.0.0.

If you need to revert to Java 8, add the following to the Spark environment variables when you configure your Azure Databricks compute:

JNAME=zulu8-ca-amd64

If you're using ARM instances, use the following:

JNAME=zulu8-ca-arm64

To learn more about specifying JDK versions with Azure Databricks compute, see Create a cluster with a specific JDK version.

For help with migrating your code from Java 8, see the following guides:

Breaking change: Hosted RStudio is end-of-life

With this release, Databricks-hosted RStudio Server is end-of-life and unavailable on any Azure Databricks workspace running Databricks Runtime 16.0 and above. To learn more and see a list of alternatives to RStudio, see Connect to a Databricks-hosted RStudio Server.

Breaking change: Removal of support for changing byte, short, int and long types to wider types

In Databricks Runtime 15.4.3 and above, the following data type changes can no longer be applied to tables with the type widening feature enabled:

  • byte, short, int and long to decimal.
  • byte, short, and int to double.

This change is made to ensure consistent behavior across Delta and Apache Iceberg tables. To learn more about type widening, see Type widening.

Correct parsing of regex patterns with negation in nested character grouping

This release includes a change to support the correct parsing of regex patterns with negation in nested character grouping. For example, [^[abc]] will be parsed as “any character that is NOT one of 'abc'”.

Additionally, Photon behavior was inconsistent with Spark for nested character classes. Regex patterns containing nested character classes will no longer use Photon, and instead will use Spark. A nested character class is any pattern containing square brackets within square brackets, such as [[a-c][1-3]].

Improve duplicate match detection in Delta Lake MERGE

In Databricks Runtime 15.4 LTS and below, MERGE operations fail if more than one row in the source table matches the same row in the target table based on the MERGE condition specified in the ON clause. In Databricks Runtime 16.0 and above, MERGE also considers conditions specified in the WHEN MATCHED clause. See Upsert into a Delta Lake table using merge.

Cluster library installation method can no longer be overridden

The Spark configurations spark.databricks.libraries.enableSparkPyPI, spark.databricks.libraries.enableMavenResolution, and spark.databricks.libraries.enableCRANResolutionnow now default to true and cannot be overridden.

Default timeout of two hours for cluster-scoped library installs

In Databricks Runtime 16.0 and above, cluster-scoped library installation has a default timeout of two hours. Library installations that take longer than this timeout will fail, and the installation is terminated. When configuring a cluster, you can change the timeout period using the Spark configuration spark.databricks.driver.clusterLibraryInstallationTimeoutSec.

Installing libraries from DBFS and setting the spark conf spark.databricks.driver.dbfsLibraryInstallationAllowed are disabled

In Databricks Runtime 16.0 and above, installing libraries from DBFS is fully disabled. This change is made to improve the security of libraries in a Databricks workspace. Additionally, in Databricks Runtime 16.0 and above, you can no longer use the Spark configuration spark.databricks.driver.dbfsLibraryInstallationAllowed.

The addArtifact() functionality is now consistent across compute types

With this release, the archive is automatically unpacked when you use addArtifact(archive = True) to add a dependency to shared or serverless Azure Databricks compute. This change makes the addArtifact(archive = True) behavior on these compute types consistent with dedicated compute (formerly single user compute), which already supports automatically unpacking archives.

Backticks rule properly enforced for principal identifiers with special characters

With this release, principal identifiers with special characters in GRANT, DENY, and REVOKE statements will now throw an error if they are not enclosed in backticks.

New features and improvements

More reliable reloading of modified Python modules with improvements to autoreload

In Databricks Runtime 16.0 and above, updates to the autoreload extension improve the safety and reliability of reloading modified Python modules imported from workspace files. With these changes, autoreload, when possible, reloads only the portion of a module that has changed instead of the entire module. Additionally, Azure Databricks now automatically suggests using the autoreload extension if the module has changed since its last import. See Autoreload for Python modules.

Avro support for recursive schema

You can now use the recursiveFieldMaxDepth option with the from_avro function and the avro data source. This option sets the maximum depth for schema recursion on the Avro data source. See Read and write streaming Avro data.

to_avro and from_avro functions

The to_avro and from_avro functions allow conversion of SQL types to Avro binary data and back.

Expanded support for Confluent Schema Registry for Avro

Azure Databricks now supports Avro schema reference with the Confluent Schema Registry. See Authenticate to an external Confluent Schema Registry.

Force reclustering on tables with liquid clustering

In Databricks Runtime 16.0 and above, you can use the OPTIMIZE FULL syntax to force the reclustering of all records in a table with liquid clustering enabled. See Force reclustering.

The Delta APIs for Python and Scala now support identity columns

You can now use the Delta APIs for Python and Scala to create tables with identity columns. See Identity columns.

Fine-grained access control on dedicated compute (formerly single user compute) is generally available

In Databricks Runtime 16.0 and above, fine-grained access control on dedicated compute is generally available. In workspaces enabled for serverless compute, if a query is run on supported compute such as dedicated compute and the query accesses any of the following objects, the compute resource passes the query to the serverless compute to run data filtering:

  • Views defined over tables on which the user does not have the SELECT privilege.
  • Dynamic views.
  • Tables with row filters or column masks applied.
  • Materialized views and streaming tables

Create liquid clustered tables during streaming writes

You can now use clusterBy to enable liquid clustering when creating new tables with Structured Streaming writes. See Enable liquid clustering.

Support for the OPTIMIZE FULL clause

Databricks Runtime 16.0 supports the OPTIMIZE FULL clause. This clause optimizes all records in a table that uses liquid clustering, including data that might have previously been clustered.

Support for WITH options specification in INSERT and table-reference

Databricks Runtime 16.0 supports an options specification for table references and table names of an INSERT statement which can be used to control the behavior of data sources.

New SQL functions

The following SQL functions are added in Databricks Runtime 16.0:

  • try_url_decode

    This function is an error-tolerant version of url_decode. This function returns NULL if the input is not a valid URL-encoded string.

  • zeroifnull

    If the input expression to the zeroifnull() function is NULL, then the function returns 0. Otherwise, the value of the input expression is returned.

  • nullifzero

    Returns NULL if the input is 0 or its input if it is not 0. If the input expression to the nullifzero() function is 0, then the function returns NULL. If the input expression is not 0, the value of the input expression is returned

Enable automatic schema evolution when merging data into a Delta table

This release adds support for the withSchemaEvolution() member of the DeltaMergeBuilder class. Use withSchemaEvolution() to enable automatic schema evolution during MERGE operations. For example, mergeBuilder.whenMatched(...).withSchemaEvolution().execute()}}.

Other changes

SparkR is now deprecated

In Databricks Runtime 16.0 and above, SparkR in Databricks is deprecated in preparation for its deprecation in the upcoming Spark 4 release. See the Apache Spark Deprecate SparkR thread.

Databricks recommends using sparklyr instead.

Databricks Runtime 16.0 is not supported with PVC

Databricks Runtime 16.0 is not supported by Databricks Private Virtual Cloud (PVC). You must use Databricks Runtime 15.4 or below with all PVC releases.

Bug fixes

Auto Loader now rescues Avro record types with empty schemas

When loading an Avro file into a Delta table using Auto Loader, record types in the file that have an empty schema are now added to the rescued data column. Because you cannot ingest empty complex data types into a Delta table, this resolves an issue with loading some Avro files. To learn more about rescued data, see What is the rescued data column?.

Fix for error writing timestamps with timezones containing a second offset.

This release fixes a bug affecting some timestamps with timezones containing a second offset. This bug causes the seconds to be omitted when writing to JSON, XML, or CSV, leading to incorrect timestamp values.

To return to the previous behavior, use the following option when writing to one of the affected formats: .option("yyyy-MM-dd'T'HH:mm:ss[.SSS][XXX]").

Library upgrades

  • Upgraded Python libraries:
    • azure-core from 1.30.2 to 1.31.0
    • azure-storage-blob from 12.19.1 to 12.23.0
    • azure-storage-file-datalake from 12.14.0 to 12.17.0
    • black from 23.3.0 to 24.4.2
    • blinker from 1.4 to 1.7.0
    • boto3 from 1.34.39 to 1.34.69
    • botocore from 1.34.39 to 1.34.69
    • certifi from 2023.7.22 to 2024.6.2
    • cffi from 1.15.1 to 1.16.0
    • click from 8.0.4 to 8.1.7
    • comm from 0.1.2 to 0.2.1
    • contourpy from 1.0.5 to 1.2.0
    • cryptography from 41.0.3 to 42.0.5
    • Cython from 0.29.32 to 3.0.11
    • databricks-sdk from 0.20.0 to 0.30.0
    • dbus-python from 1.2.18 to 1.3.2
    • filelock from 3.13.4 to 3.15.4
    • fonttools from 4.25.0 to 4.51.0
    • GitPython from 3.1.43 to 3.1.37
    • google-api-core from 2.18.0 to 2.20.0
    • google-auth from 2.31.0 to 2.35.0
    • google-cloud-storage from 2.17.0 to 2.18.2
    • google-crc32c from 1.5.0 to 1.6.0
    • google-resumable-media from 2.7.1 to 2.7.2
    • googleapis-common-protos from 1.63.2 to 1.65.0
    • httplib2 from 0.20.2 to 0.20.4
    • idna from 3.4 to 3.7
    • ipykernel from 6.25.1 to 6.28.0
    • ipython from 8.15.0 to 8.25.0
    • jedi from 0.18.1 to 0.19.1
    • jmespath from 0.10.0 to 1.0.1
    • joblib from 1.2.0 to 1.4.2
    • jupyter_client from 7.4.9 to 8.6.0
    • jupyter_core from 5.3.0 to 5.7.2
    • launchpadlib from 1.10.16 to 1.11.0
    • lazr.restfulclient from 0.14.4 to 0.14.6
    • matplotlib from 3.7.2 to 3.8.4
    • mlflow-skinny from 2.11.4 to 2.15.1
    • more-itertools from 8.10.0 to 10.3.0
    • mypy-extensions from 0.4.3 to 1.0.0
    • nest-asyncio from 1.5.6 to 1.6.0
    • numpy from 1.23.5 to 1.26.4
    • oauthlib from 3.2.0 to 3.2.2
    • packaging from 23.2 to 24.1
    • patsy from 0.5.3 to 0.5.6
    • pip from 23.2.1 to 24.2
    • plotly from 5.9.0 to 5.22.0
    • prompt-toolkit from 3.0.36 to 3.0.43
    • pyarrow from 14.0.1 to 15.0.2
    • pydantic from 1.10.6 to 2.8.2
    • PyGObject from 3.42.1 to 3.48.2
    • PyJWT from 2.3.0 to 2.7.0
    • pyodbc from 4.0.38 to 5.0.1
    • python-dateutil from 2.8.2 to 2.9.0.post0
    • python-lsp-jsonrpc from 1.1.1 to 1.1.2
    • pytz from 2022.7 to 2024.1
    • PyYAML from 6.0 to 6.0.1
    • pyzmq from 23.2.0 to 25.1.2
    • requests from 2.31.0 to 2.32.2
    • scikit-learn from 1.3.0 to 1.4.2
    • scipy from 1.11.1 to 1.13.1
    • seaborn from 0.12.2 to 0.13.2
    • setuptools from 68.0.0 to 74.0.0
    • smmap from 5.0.1 to 5.0.0
    • sqlparse from 0.5.0 to 0.5.1
    • statsmodels from 0.14.0 to 0.14.2
    • tornado from 6.3.2 to 6.4.1
    • traitlets from 5.7.1 to 5.14.3
    • typing_extensions from 4.10.0 to 4.11.0
    • ujson from 5.4.0 to 5.10.0
    • virtualenv from 20.24.2 to 20.26.2
    • wheel from 0.38.4 to 0.43.0
    • zipp from 3.11.0 to 3.17.0
  • Upgraded R libraries:
    • arrow from 14.0.0.2 to 16.1.0
    • backports from 1.4.1 to 1.5.0
    • base from 4.3.2 to 4.4.0
    • bitops from 1.0-7 to 1.0-8
    • boot from 1.3-28 to 1.3-30
    • brio from 1.1.4 to 1.1.5
    • broom from 1.0.5 to 1.0.6
    • bslib from 0.6.1 to 0.8.0
    • cachem from 1.0.8 to 1.1.0
    • callr from 3.7.3 to 3.7.6
    • cli from 3.6.2 to 3.6.3
    • clock from 0.7.0 to 0.7.1
    • cluster from 2.1.4 to 2.1.6
    • codetools from 0.2-19 to 0.2-20
    • colorspace from 2.1-0 to 2.1-1
    • compiler from 4.3.2 to 4.4.0
    • crayon from 1.5.2 to 1.5.3
    • curl from 5.2.0 to 5.2.1
    • data.table from 1.15.0 to 1.15.4
    • datasets from 4.3.2 to 4.4.0
    • DBI from 1.2.1 to 1.2.3
    • dbplyr from 2.4.0 to 2.5.0
    • digest from 0.6.34 to 0.6.36
    • downlit from 0.4.3 to 0.4.4
    • evaluate from 0.23 to 0.24.0
    • farver from 2.1.1 to 2.1.2
    • fastmap from 1.1.1 to 1.2.0
    • foreign from 0.8-85 to 0.8-86
    • fs from 1.6.3 to 1.6.4
    • future from 1.33.1 to 1.34.0
    • future.apply from 1.11.1 to 1.11.2
    • gert from 2.0.1 to 2.1.0
    • ggplot2 from 3.4.4 to 3.5.1
    • gh from 1.4.0 to 1.4.1
    • globals from 0.16.2 to 0.16.3
    • graphics from 4.3.2 to 4.4.0
    • grDevices from 4.3.2 to 4.4.0
    • grid from 4.3.2 to 4.4.0
    • gt from 0.10.1 to 0.11.0
    • gtable from 0.3.4 to 0.3.5
    • hardhat from 1.3.1 to 1.4.0
    • highr from 0.10 to 0.11
    • htmltools from 0.5.7 to 0.5.8.1
    • httpuv from 1.6.14 to 1.6.15
    • httr2 from 1.0.0 to 1.0.2
    • ipred from 0.9-14 to 0.9-15
    • KernSmooth from 2.23-21 to 2.23-22
    • knitr from 1.45 to 1.48
    • lattice from 0.21-8 to 0.22-5
    • lava from 1.7.3 to 1.8.0
    • markdown from 1.12 to 1.13
    • MASS from 7.3-60 to 7.3-60.0.1
    • Matrix from 1.5-4.1 to 1.6-5
    • methods from 4.3.2 to 4.4.0
    • mgcv from 1.8-42 to 1.9-1
    • mlflow from 2.10.0 to 2.14.1
    • munsell from 0.5.0 to 0.5.1
    • nlme from 3.1-163 to 3.1-165
    • openssl from 2.1.1 to 2.2.0
    • parallel from 4.3.2 to 4.4.0
    • parallelly from 1.36.0 to 1.38.0
    • pkgbuild from 1.4.3 to 1.4.4
    • pkgdown from 2.0.7 to 2.1.0
    • pkgload from 1.3.4 to 1.4.0
    • processx from 3.8.3 to 3.8.4
    • prodlim from 2023.08.28 to 2024.06.25
    • promises from 1.2.1 to 1.3.0
    • ps from 1.7.6 to 1.7.7
    • ragg from 1.2.7 to 1.3.2
    • Rcpp from 1.0.12 to 1.0.13
    • RcppEigen from 0.3.3.9.4 to 0.3.4.0.0
    • reactR from 0.5.0 to 0.6.0
    • recipes from 1.0.9 to 1.1.0
    • remotes from 2.4.2.1 to 2.5.0
    • reprex from 2.1.0 to 2.1.1
    • rlang from 1.1.3 to 1.1.4
    • rmarkdown from 2.25 to 2.27
    • roxygen2 from 7.3.1 to 7.3.2
    • rpart from 4.1.21 to 4.1.23
    • RSQLite from 2.3.5 to 2.3.7
    • rstudioapi from 0.15.0 to 0.16.0
    • rvest from 1.0.3 to 1.0.4
    • sass from 0.4.8 to 0.4.9
    • shape from 1.4.6 to 1.4.6.1
    • shiny from 1.8.0 to 1.9.1
    • sparklyr from 1.8.4 to 1.8.6
    • spatial from 7.3-15 to 7.3-17
    • splines from 4.3.2 to 4.4.0
    • stats from 4.3.2 to 4.4.0
    • stats4 from 4.3.2 to 4.4.0
    • stringi from 1.8.3 to 1.8.4
    • survival from 3.5-5 to 3.6-4
    • swagger from 3.33.1 to 5.17.14.1
    • systemfonts from 1.0.5 to 1.1.0
    • tcltk from 4.3.2 to 4.4.0
    • testthat from 3.2.1 to 3.2.1.1
    • textshaping from 0.3.7 to 0.4.0
    • tidyselect from 1.2.0 to 1.2.1
    • tinytex from 0.49 to 0.52
    • tools from 4.3.2 to 4.4.0
    • usethis from 2.2.2 to 3.0.0
    • utils from 4.3.2 to 4.4.0
    • uuid from 1.2-0 to 1.2-1
    • V8 from 4.4.1 to 4.4.2
    • withr from 3.0.0 to 3.0.1
    • xfun from 0.41 to 0.46
    • xopen from 1.0.0 to 1.0.1
    • yaml from 2.3.8 to 2.3.10
  • Upgraded Java libraries:
    • com.amazonaws.aws-java-sdk-autoscaling from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudformation from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudfront from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudhsm from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudsearch from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudtrail from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudwatch from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cloudwatchmetrics from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-codedeploy from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cognitoidentity from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-cognitosync from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-config from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-core from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-datapipeline from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-directconnect from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-directory from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-dynamodb from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ec2 from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ecs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-efs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elasticache from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elasticbeanstalk from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elasticloadbalancing from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-elastictranscoder from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-emr from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-glacier from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-glue from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-iam from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-importexport from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-kinesis from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-kms from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-lambda from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-logs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-machinelearning from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-opsworks from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-rds from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-redshift from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-route53 from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-s3 from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ses from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-simpledb from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-simpleworkflow from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-sns from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-sqs from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-ssm from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-storagegateway from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-sts from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-support from 1.12.610 to 1.12.638
    • com.amazonaws.aws-java-sdk-workspaces from 1.12.610 to 1.12.638
    • com.amazonaws.jmespath-java from 1.12.610 to 1.12.638
    • com.google.protobuf.protobuf-java from 2.6.1 to 3.25.1
    • io.airlift.aircompressor from 0.25 to 0.27
    • io.delta.delta-sharing-client_2.12 from 1.1.3 to 1.2.0
    • io.netty.netty-all from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-buffer from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec-http from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec-http2 from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-codec-socks from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-common from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-handler from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-handler-proxy from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-resolver from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport-classes-epoll from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport-classes-kqueue from 4.1.96.Final to 4.1.108.Final
    • io.netty.netty-transport-native-epoll from 4.1.96.Final-linux-x86_64 to 4.1.108.Final-linux-x86_64
    • io.netty.netty-transport-native-kqueue from 4.1.96.Final-osx-x86_64 to 4.1.108.Final-osx-x86_64
    • io.netty.netty-transport-native-unix-common from 4.1.96.Final to 4.1.108.Final
    • org.apache.ivy.ivy from 2.5.1 to 2.5.2
    • org.apache.zookeeper.zookeeper from 3.6.3 to 3.9.2
    • org.apache.zookeeper.zookeeper-jute from 3.6.3 to 3.9.2
    • org.rocksdb.rocksdbjni from 8.11.4 to 9.2.1
    • org.scalactic.scalactic_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-compatible from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-core_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-diagrams_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-featurespec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-flatspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-freespec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-funspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-funsuite_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-matchers-core_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-mustmatchers_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-propspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-refspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-shouldmatchers_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest-wordspec_2.12 from 3.2.15 to 3.2.16
    • org.scalatest.scalatest_2.12 from 3.2.15 to 3.2.16

Apache Spark

Databricks Runtime 16.0 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 15.4 LTS, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-49093] [DBRRM-1371] Revert “[SC-172958][sql] GROUP BY with MapType nes…
  • [SPARK-49898] [DBRRM-1282][sc-178410] Fix documentation and default for event log task metrics accumulator logging flag from SPARK-42204
  • [SPARK-49743] [ES-1260022][behave-157][SC-177475][sql] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
  • [SPARK-49816] [SC-177896][sql] Should only update out-going-ref-count for referenced outer CTE relation
  • [SPARK-48939] [SC-177022][sc-172766][AVRO] Support reading Avro with recursive schema reference
  • [SPARK-49688] [SC-177468][es-1242349][CONNECT] Fix a data race between interrupt and execute plan
  • [SPARK-49771] [SC-177466][python] Improve Pandas Scalar Iter UDF error when output rows exceed input rows
  • [SPARK-48866] [SC-170772][sql] Fix hints of valid charset in the error message of INVALID_PARAMETER_VALUE.CHARSET
  • [SPARK-48195] [FIXFORWARD][sc-177267][CORE] Save and reuse RDD/Broadcast created by SparkPlan
  • [SPARK-49585] [CONNECT] Replace executions map in SessionHolder with operationID set
  • [SPARK-49211] [SC-174257][sql] V2 Catalog can also support built-in data sources
  • [SPARK-49684] Minimise the lifetime of the session restoration lock
  • [SPARK-48059] [SPARK-48145][spark-48134][SPARK-48182][spark-48209][SPARK-48291] Structured log framework on the java side
  • [SPARK-48857] [SC-170661][sql] Restrict charsets in CSVOptions
  • [SPARK-49152] [SC-173690][sql] V2SessionCatalog should use V2Command
  • [SPARK-42846] [SC-176588][sql] Remove error condition _LEGACY_ERROR_TEMP_2011
  • [SPARK-48195] [SC-177267][core] Save and reuse RDD/Broadcast created by SparkPlan
  • [SPARK-49630] [SC-177379][ss] Add flatten option to process collection types with state data source reader
  • [SPARK-49699] [SC-177154][ss] Disable PruneFilters for streaming workloads
  • [SPARK-48781] [SC-175282][sql] Add Catalog APIs for loading stored procedures
  • [SPARK-49667] [SC-177068][sql] Disallow CS_AI collators with expressions that use StringSearch
  • [SPARK-49737] [SC-177207][sql] Disable bucketing on collated columns in complex types
  • [SPARK-48712] [SC-169794][sql] Perf Improvement for encode with empty values or UTF-8 charset
  • [SPARK-49038] [SC-173933][sql] SQLMetric should report the raw value in the accumulator update event
  • [SPARK-48541] [SC-169066][core] Add a new exit code for executors killed by TaskReaper
  • [SPARK-48774] [SC-170138][sql] Use SparkSession in SQLImplicits
  • [SPARK-49719] [SC-177139][sql] Make UUID and SHUFFLE accept integer seed
  • [SPARK-49713] [SC-177135][python][CONNECT] Make function count_min_sketch accept number arguments
  • [SPARK-47601] [SC-162499][graphx] Graphx: Migrate logs with variables to structured logging framework
  • [SPARK-49738] [SC-177219][sql] Endswith bug fix
  • [SPARK-48623] [SC-170822][core] Structured logging migrations [Part 3]
  • [SPARK-49677] [SC-177148][ss] Ensure that changelog files are written on commit and forceSnapshot flag is also reset
  • [SPARK-49684] [SC-177040][connect] Remove global locks from session and execution managers
  • [SPARK-48302] [SC-168814][python] Preserve nulls in map columns in PyArrow Tables
  • [SPARK-48601] [SC-169025][sql] Give a more user friendly error message when setting a null value for JDBC Option
  • [SPARK-48635] [SC-169263][sql] Assign classes to join type errors and as-of join error
  • [SPARK-49673] [SC-177032][connect] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE
  • [SPARK-49693] [SC-177071][python][CONNECT] Refine the string representation of timedelta
  • [SPARK-49687] [SC-176901][sql] Delay sorting in validateAndMaybeEvolveStateSchema
  • [SPARK-49718] [SC-177112][ps] Switch Scatter plot to sampled data
  • [SPARK-48472] [SC-169044][sql] Enable reflect expressions with collated strings
  • [SPARK-48484] [SC-167484][sql] Fix: V2Write use the same TaskAttemptId for different task attempts
  • [SPARK-48341] [SC-166560][connect] Allow plugins to use QueryTest in their tests
  • [SPARK-42252] [SC-168723][core] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer
  • [SPARK-48314] [SC-166565][ss] Don't double cache files for FileStreamSource using Trigger.AvailableNow
  • [SPARK-49567] [SC-176241][python] Use classic instead of vanilla from PySpark code base
  • [SPARK-48374] [SC-167596][python] Support additional PyArrow Table column types
  • [SPARK-48300] [SC-166481][sql] Codegen Support for from_xml
  • [SPARK-49412] [SC-177059][ps] Compute all box plot metrics in single job
  • [SPARK-49692] [SC-177031][python][CONNECT] Refine the string representation of literal date and datetime
  • [SPARK-49392] [ES-1130351][sc-176705][SQL] Catch errors when failing to write to external data source
  • [SPARK-48306] [SC-166241][sql] Improve UDT in error message
  • [SPARK-44924] [SC-166379][ss] Add config for FileStreamSource cached files
  • [SPARK-48176] [SC-165644][sql] Adjust name of FIELD_ALREADY_EXISTS error condition
  • [SPARK-49691] [SC-176988][python][CONNECT] Function substring should accept column names
  • [SPARK-49502] [SC-176077][core] Avoid NPE in SparkEnv.get.shuffleManager.unregisterShuffle
  • [SPARK-49244] [SC-176703][sql] Further exception improvements for parser/interpreter
  • [SPARK-48355] [SC-176684][sql] Support for CASE statement
  • [SPARK-49355] [SC-175121][sql] levenshtein should check whether the collation values of all parameter types are the same
  • [SPARK-49640] [SC-176953][ps] Apply reservoir sampling in SampledPlotBase
  • [SPARK-49678] [SC-176857][core] Support spark.test.master in SparkSubmitArguments
  • [SPARK-49680] [SC-176856][python] Limit Sphinx build parallelism to 4 by default
  • [SPARK-49396] Revert “[SC-176030][sql] Modify nullability check for CaseWhen expression”
  • [SPARK-48419] [SC-167443][sql] Foldable propagation replace foldable column shoul…
  • [SPARK-49556] [SC-176757][sql] Add SQL pipe syntax for the SELECT operator
  • [SPARK-49438] [SC-175237][sql] Fix the pretty name of the FromAvro & ToAvro expression
  • [SPARK-49659] [SC-1229924][sql] Add a nice user-facing error for scalar subqueries inside VALUES clause
  • [SPARK-49646] [SC-176778][sql] fix subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences
  • [SPARK-49354] [SC-175034][sql] split_part should check whether the collation values of all parameter types are the same
  • [SPARK-49478] [SC-175914][connect] Handle null metrics in ConnectProgressExecutionListener
  • [SPARK-48358] [SC-176374][sql] Support for REPEAT statement
  • [SPARK-49183] [SC-173680][sql] V2SessionCatalog.createTable should respect PROP_IS_MANAGED_LOCATION
  • [SPARK-49611] [SC-176791][sql] Introduce TVF collations() & remove the SHOW COLLATIONS command
  • [SPARK-49261] [SC-176589][sql] Don't replace literals in aggregate expressions with group-by expressions
  • [SPARK-49099] [SC-173229][sql] CatalogManager.setCurrentNamespace should respect custom session catalog
  • [SPARK-49594] [SC-176569][ss] Adding check on whether columnFamilies were added or removed to write StateSchemaV3 file
  • [SPARK-49578] [SC-176385][sql] Remove the ANSI config suggestion in CAST_INVALID_INPUT and CAST_OVERFLOW
  • [SPARK-48882] [SC-174256][ss] Assign names to streaming output mode related error classes
  • [SPARK-49155] [SC-176506][sql][SS] Use more appropriate parameter type to construct GenericArrayData
  • [SPARK-49519] [SC-176388][sql] Merge options of table and relation when constructing FileScanBuilder
  • [SPARK-49591] [SC-176587][sql] Add Logical Type column to variant readme
  • [SPARK-49596] [SC-176423][sql] Improve performance of FormatString
  • [SPARK-49525] [SC-176044][ss][CONNECT] Minor log improvement to Server Side Streaming Query ListenerBus Listener
  • [SPARK-49583] [SC-176272][sql] Define the error sub-condition SECONDS_FRACTION for invalid seconds fraction pattern
  • [SPARK-49536] [SC-176242] Handle error in python streaming data source record prefetching
  • [SPARK-49443] [SC-176273][sql][PYTHON] Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects
  • [SPARK-49544] [SASP-3990][sc-176557][CONNECT] Replace coarse-locking in SparkConnectExecutionManager with ConcurrentMap
  • [SPARK-49548] [SASP-3990][sc-176556][CONNECT] Replace coarse-locking in SparkConnectSessionManager with ConcurrentMap
  • [SPARK-49551] [SC-176218][ss] Improve RocksDB log for replayChangelog
  • [SPARK-49595] [SC-176396][connect][SQL] Fix DataFrame.unpivot/melt in Spark Connect Scala Client
  • [SPARK-49006] [SC-176162] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files
  • [SPARK-49600] [SC-176426][python] Remove Python 3.6 and older-related logic from try_simplify_traceback
  • [SPARK-49303] [SC-176013][ss] Implement TTL for ValueState in transformWithStateInPandas API
  • [SPARK-49191] [SC-176243][ss] Add support for reading transformWithState map state variables with state data source reader
  • [SPARK-49593] [SC-176371][ss] Throw RocksDB exception to the caller on DB close if an error is seen
  • [SPARK-49334] [SC-174803][sql] str_to_map should check whether the collation values of all parameter types are the same
  • [SPARK-42204] [SC-176126][core] Add option to disable redundant logging of TaskMetrics internal accumulators in event logs
  • [SPARK-49575] [SC-176256][ss] Add logging for lock release only if acquiredThreadInfo is not null
  • [SPARK-49539] [SC-176250][ss] Update internal col families start identifier to a different one
  • [SPARK-49205] [SC-173853][sql] KeyGroupedPartitioning should inherit HashPartitioningLike
  • [SPARK-49396] [SC-176030][sql] Modify nullability check for CaseWhen expression
  • [SPARK-49476] [SC-175700][sql] Fix nullability of base64 function
  • [SPARK-47262] [SC-174829][sql] Assign names to error conditions for parquet conversions
  • [SPARK-47247] [SC-158349][sql] Use smaller target size when coalescing partitions with exploding joins
  • [SPARK-49501] [SC-176166][sql] Fix double-escaping of table location
  • [SPARK-49083] [SC-173214][connect] Allow from_xml and from_json to natively work with json schemas
  • [SPARK-49043] [SC-174673][sql] Fix interpreted codepath group by on map containing collated strings
  • [SPARK-48986] [SC-172265][connect][SQL] Add ColumnNode Intermediate Representation
  • [SPARK-49326] [SC-176016][ss] Classify Error class for Foreach sink user function error
  • [SPARK-48348] [SC-175950][spark-48376][SQL] Introduce LEAVE and ITERATE statements
  • [SPARK-49523] [SC-175949][connect] Increase maximum wait time for connect server to come up for testing
  • [SPARK-49000] [BEHAVE-105][es-1194747][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates - Databricks Runtime version 16.x
  • [SPARK-49311] [SC-175038][sql] Make it possible for large 'interval second' values to be cast to decimal
  • [SPARK-49200] [SC-173699][sql] Fix null type non-codegen ordering exception
  • [SPARK-49467] [SC-176051][ss] Add support for state data source reader and list state
  • [SPARK-47307] [SC-170891][sql] Add a config to optionally chunk base64 strings
  • [SPARK-49391] [SC-176032][ps] Box plot select outliers by distance from fences
  • [SPARK-49445] [SC-175845][ui] Support show tooltip in the progress bar of UI
  • [SPARK-49451] [SC-175702] Allow duplicate keys in parse_json.
  • [SPARK-49275] [SC-175701][sql] Fix return type nullness of the xpath expression
  • [SPARK-49021] [SC-175578][ss] Add support for reading transformWithState value state variables with state data source reader
  • [SPARK-49474] [BEHAVE-143][sc-169253][SC-175933][ss] Classify Error class for FlatMapGroupsWithState user function error
  • [SPARK-49408] [SC-175932][sql] Use IndexedSeq in ProjectingInternalRow
  • [SPARK-49509] [SC-175853][core] Use Platform.allocateDirectBuffer instead of ByteBuffer.allocateDirect
  • [SPARK-49382] [SC-175013][ps] Make frame box plot properly render the fliers/outliers
  • [SPARK-49002] [SC-172846][sql] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
  • [SPARK-49480] [SC-175699][core] Fix NullPointerException from SparkThrowableHelper.isInternalError
  • [SPARK-49477] [SC-175828][python] Improve pandas udf invalid return type error message
  • [SPARK-48693] [SC-169492][sql] Simplify and unify toString of Invoke and StaticInvoke
  • [SPARK-49441] [SC-175716][ml] StringIndexer sort arrays in executors
  • [SPARK-49347] [SC-175004][r] Deprecate SparkR
  • [SPARK-49357] [SC-175227][connect][PYTHON] Vertically truncate deeply nested protobuf message
  • [SPARK-41982] [SC-120604][sql] Partitions of type string should not be treated as numeric types
  • [SPARK-48776] [SC-170452][behave-72] Fix timestamp formatting for json, xml and csv
  • [SPARK-49223] [SC-174800][ml] Simplify the StringIndexer.countByValue with builtin functions
  • [SPARK-49016] Revert “[SC-174663][sql] Restore the behavior thatqueries from raw CSV files are disallowed when only include corrupt record column and assign name to _LEGACY_ERROR_TEMP_1285”
  • [SPARK-49041] [SC-172392][python][CONNECT] Raise proper error for dropDuplicates when wrong subset is given
  • [SPARK-49050] [SC-175235] Enabling deleteIfExists operator in TWS with Virtual Column Families
  • [SPARK-49216] [SC-173919][core]Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off
  • [SPARK-49252] [SC-175596][core] MakeTaskSetExcludeList and HeathTracker independent
  • [SPARK-49352] [SC-174971][sql] Avoid redundant array transform for identical expression
  • [SPARK-42307] [SC-173863][sql] Assign name for error _LEGACY_ERROR_TEMP_2232
  • [SPARK-49197] [SC-173732][core] Redact Spark Command output in launcher module
  • [SPARK-48913] [SC-173934][sql] Implement IndentingXMLStreamWriter
  • [SPARK-49306] [SC-175363][python][SQL] Create SQL function aliases for 'zeroifnull' and 'nullifzero'
  • [SPARK-48344] [SQL] SQL Scripting execution (including Spark Connect)
  • [SPARK-49402] [SC-175122][python] Fix Binder integration in PySpark documentation
  • [SPARK-49017] [SC-174664][sql] Insert statement fails when multiple parameters are being used
  • [SPARK-49318] [SC-174733][sql] Preempt low priority error on LCA until end of check analysis to improve error experience
  • [SPARK-49016] [SC-174663][sql] Restore the behavior that queries from raw CSV files are disallowed when only include corrupt record column and assign name to _LEGACY_ERROR_TEMP_1285
  • [SPARK-49387] [SC-175124][python] Fix type hint for accuracy in percentile_approx and approx_percentile
  • [SPARK-49131] [SC-174666][ss] TransformWithState should properly set implicit grouping keys even with lazy iterators
  • [SPARK-49301] [SC-174795][ss] Chunk arrow data passed to Python worker
  • [SPARK-49039] [SC-174651][ui] Reset checkbox when executor metrics are loaded in the Stages tab
  • [SPARK-48428] [SC-169806][sql]: Fix IllegalStateException in NestedColumnAliasing
  • [SPARK-49353] [SC-174830][sql] Update docs related to UTF-32 encoding/decoding
  • [SPARK-48613] [SC-170966][sql] SPJ: Support auto-shuffle one side + less join keys than partition keys
  • [SPARK-47473] [SC-160450][behave-127][SQL] Fix correctness issue of converting postgres INFINITY timestamps
  • [SPARK-49142] [SC-173658][connect][PYTHON] Follow up to revert proto to string performance cost
  • [SPARK-49300] [SC-175008][core] Fix Hadoop delegation token leak when tokenRenewalInterval is not set.
  • [SPARK-49367] [SC-175012][ps] Parallelize the KDE computation for multiple columns (plotly backend)
  • [SPARK-49365] [SC-175011][ps] Simplify the bucket aggregation in hist plot
  • [SPARK-49372] [SC-175003][ss] Ensure that latestSnapshot is set to none on close to avoid subsequent use
  • [SPARK-49341] [SC-174785] Remove connector/docker in favor of Apache Spark Operator
  • [SPARK-49344] [SC-174894][ps] Support json_normalize for Pandas API on Spark
  • [SPARK-49306] [SC-174794][sql] Create new SQL functions 'zeroifnull' and 'nullifzero'
  • [SPARK-48796] [SC-174668][ss] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
  • [SPARK-49342] [SC-174899][sql] Make TO_AVRO SQL function 'jsonFormatSchema' argument optional
  • [SPARK-48628] [SC-174695][core] Add task peak on/off heap memory metrics
  • [SPARK-47407] [SC-159379][behave-126][SQL] Support java.sql.Types.NULL map to NullType
  • [SPARK-48628] [SC-173407][core] Add task peak on/off heap memory metrics
  • [SPARK-49166] [SC-173987][sql] Support OFFSET in correlated subquery
  • [SPARK-49269] [SC-174676][sql] Eagerly evaluate VALUES() list in AstBuilder
  • [SPARK-49281] [SC-174782][sql] Optimze parquet binary getBytes with getBytesUnsafe to avoid copy cost