Databricks Runtime 19

This page contains unified release notes for Databricks Runtime 19, powered by Apache Spark 4.2.0. It lists all features, behavioral changes, and fixes shipped throughout the Databricks Runtime 19 release cycle. Clusters receive updates when they restart.

For information about the Databricks Runtime support lifecycle, see Databricks support lifecycles.

Databricks Runtime 19: July 23, 2026

Databricks Runtime 19 is now generally available (GA). For lifecycle details, see Databricks Runtime support lifecycles.

Behavioral changes

• Arrow-optimized Python UDFs enabled by default: Regular Python UDFs now use Arrow-based serialization by default. This can change type-coercion behavior for some UDFs compared to previous releases. To restore the previous behavior, set spark.databricks.execution.pythonUDF.arrow.enabled to false.

New features and improvements

• TopN pushdown for fine-grained access control on dedicated compute: Fine-grained access control on dedicated compute now pushes down ORDER BY ... LIMIT (TopN) operations to the remote cluster that performs the filtering, improving query performance.
• parquet-java upgraded to 1.17.0: The apache/parquet-java dependency is upgraded to version 1.17.0. For the full list of changes, see the apache/parquet-java release notes.

Databricks Runtime 19 (Beta): June 26, 2026

Behavioral changes

• Environment variable availability restricted in standard access mode: In standard access mode, only a predefined set of environment variables is available to the Spark engine and init scripts. This set includes common configuration variables, such as proxy settings (HTTP_PROXY), cloud credentials (AWS_ACCESS_KEY_ID, AZURE_CLIENT_ID), and catalog variables (CATALOG). Other variables that you set on a cluster remain available to your user code, including UDFs, but are not available to the Spark engine or init scripts. Previously, all cluster environment variables were available to the Spark engine and init scripts. See Environment variable limitations.
• Restricted Spark configurations in standard access mode: In standard access mode, you can no longer set certain Spark configuration properties. Creating or editing a cluster that sets a restricted property (matched by prefix) fails with an error. Restricted properties include spark.driver.extraJavaOptions, spark.executor.extraJavaOptions, spark.jars, spark.files, spark.executorEnv.*, and spark.kubernetes.*, among others. Previously, you could set these properties in standard access mode. Remove them from your cluster configuration before upgrading. For the complete list, see Spark configuration limitations.

Databricks Runtime 19 (Beta): June 15, 2026

Databricks Runtime 19 is now available in Beta, powered by Apache Spark 4.2.0.

Behavioral changes

• JDK 17 removed: Databricks Runtime 19 only supports JDK 21. Previously, Databricks Runtime 18 included JDK 17 as a fallback option that could be selected using the JNAME environment variable. That fallback is no longer available in Databricks Runtime 19. Clusters that set JNAME=zulu17-ca-amd64 (or the ARM equivalent) must remove that environment variable before upgrading.
• Python packages removed: Compared to Databricks Runtime 18, approximately 90 standard runtime packages are removed in Databricks Runtime 19. Notable removals include plotly, seaborn, openai, langchain-core, langchain-openai, huggingface_hub, psycopg2, and pyodbc. The JupyterLab bundled server is split out of the base image, removing around 54 Jupyter-related packages. Workloads that depend on these packages must install them explicitly using a cluster library or init script. For the full list of packages included in Databricks Runtime 19, see Installed Python libraries. For ML runtime-specific removals, see Databricks Runtime 19 for Machine Learning.
• Reserved table property pipelines.pipelineId: Databricks Runtime 19 reserves the pipelines.pipelineId table property on all tables. Attempting to set this property manually results in an error. Remove any explicit pipelines.pipelineId assignments from your table DDL before upgrading.
• Real-time mode watermark delay shifted by +1 ms: In real-time mode (RTM) streaming queries, when the watermark advances mid-batch via real-time watermark propagation, a record whose event time equals the current watermark was previously treated as late and dropped. The event-time watermark delay is now shifted by +1 ms so the late-record check is effectively strict, and records whose event time equals the current watermark are now retained. Micro-batch mode is unaffected.
• DStream checkpoint class allowlist required for custom types: Apache Spark 4.2.0 introduces a security control for DStream checkpoint deserialization. Users who rely on custom or third-party types inside DStream closures must set spark.streaming.checkpoint.allowedClasses to a comma-separated list of fully qualified class names or wildcard patterns before attempting checkpoint recovery. The default built-in allowlist covers standard Spark types only.

Apache Spark fixes and improvements

Databricks Runtime 19 includes Apache Spark 4.2.0. This release includes all Spark fixes and improvements included in Databricks Runtime 18 LTS, as well as the following additional bug fixes and improvements made to Spark:

• is_valid_variant expression: SQL now supports is_valid_variant(expr), which returns true if the input is a well-formed VARIANT value and false otherwise. Use this to validate variant data before downstream processing.
• INSERT INTO ... REPLACE ON/USING syntax: SQL now supports INSERT INTO ... REPLACE ON <columns> and INSERT INTO ... REPLACE USING <subquery> syntax, enabling upsert-style writes that replace existing rows based on matching column values.
• DSv2 transaction management: Data Source v2 catalogs now support transaction management, enabling atomic multi-table writes and consistent reads across catalog operations.
• Batch CDC post-processing with ChangelogTable: SQL now supports reading Change Data Capture (CDC) output as a changelog and computing net changes in batch mode. The ResolveChangelogTable analyzer validates changelog schema and computes merged results from raw CDC rows.
• TABLESAMPLE SYSTEM block sampling with DSv2 pushdown: TABLESAMPLE SYSTEM is now supported with Data Source v2 tables, allowing percentage-based block-level sampling to be pushed down to the data source for improved performance.
• CREATE VIEW and ALTER VIEW for DSv2 catalogs: Data Source v2 catalogs now support CREATE VIEW and ALTER VIEW statements, bringing view DDL parity to non-Hive catalogs.
• CREATE METRIC VIEW on V2 catalogs: CREATE METRIC VIEW is now supported on Data Source v2 catalogs.
• KLL quantile functions in SQL: SQL now natively supports KLL quantile sketch functions powered by Apache DataSketches, including aggregation and CDF computation functions for approximate quantile queries.
• Tuple Sketch functions in SQL: SQL now natively supports Apache Tuple Sketch aggregation functions, enabling approximate set intersection and union operations over arbitrarily typed payloads.
• withSchemaEvolution() in DataFrame writer API: The DataFrame writer API now includes withSchemaEvolution(), which enables automatic schema evolution when writing to tables that support it.
• Dataset.zipWithIndex in Scala API: The Scala Dataset API now includes zipWithIndex, pairing each element with its index position, consistent with the existing Python and Java APIs.
• Scalar UDFs in table-valued function arguments: SQL now supports using scalar user-defined functions as arguments to table-valued functions, enabling richer parameterization.
• DataFrameGroupBy.cov in pandas API on Spark: The pandas API on Spark now supports DataFrameGroupBy.cov, computing pairwise covariance of columns within each group.
• SeriesGroupBy.describe in pandas API on Spark: The pandas API on Spark now supports SeriesGroupBy.describe, generating descriptive statistics for each group.
• Plan download links in Spark UI: The Spark UI execution page now includes download links for query plans in SVG, DOT, and TXT formats.
• pa.ChunkedArray support in createDataFrame: createDataFrame() now correctly handles pandas DataFrames with PyArrow ChunkedArray columns, preventing type errors when using Arrow-backed pandas data.
• Codegen support for array higher-order functions: Array higher-order functions such as filter, transform, and aggregate now benefit from JVM bytecode generation during query compilation, improving performance for workloads that use these functions heavily.
• Fix: exceptAll correctness after dropDuplicates with column subset: A bug is fixed where df.dropDuplicates(subset).exceptAll(other) produced incorrect results. The fix ensures duplicate-tracking logic correctly handles subset deduplication.
• Fix: SQL parser preserves statements ending with block comments: A bug is fixed where SQL statements ending with a block comment were silently dropped by the semicolon splitter. All statements are now correctly parsed regardless of trailing comment style.
• Fix: stream-stream join correctness in state format V4: A bug is fixed in stream-stream joins with time window predicates when using RocksDB state store format V4, preventing incorrect join results during stateful streaming queries.

Library upgrades

Libraries are only upgraded when there is a security risk with an older version. When a vulnerability is identified, Databricks upgrades to the earliest version that resolves the issue. If a library was upgraded during this release cycle due to a confirmed security patch, the previous version is also listed to document the transition.

Python

R