Comment
Author: Admin | 2025-04-28
Mode in Unity Catalog has the following limitations. These are in addition to the general limitations for all Unity Catalog access modes. See General limitations for Unity Catalog.Databricks Runtime ML and Spark Machine Learning Library (MLlib) are not supported.Spark-submit job tasks are not supported. Use a JAR task instead.DBUtils and other clients that directly read the data from cloud storage are only supported when you use an external location to access the storage location. See Create an external location to connect cloud storage to Databricks.In Databricks Runtime 13.3 and above, individual rows must not exceed 128MB.DBFS root and mounts do not support FUSE.Custom containers are not supported.Language support for Unity Catalog standard access modeR is not supported.Scala is supported in Databricks Runtime 13.3 and above.In Databricks Runtime 15.4 LTS and above, all Java or Scala libraries (JAR files) bundled with Databricks Runtime are available on compute in Unity Catalog access modes.For Databricks Runtime 15.3 or below on compute that uses standard access mode, set the Spark config spark.databricks.scala.kernel.fullClasspath.enabled to true.Spark API limitations and requirements for Unity Catalog standard access modeRDD APIs are not supported.Spark Context (sc),spark.sparkContext, and sqlContext are not supported for Scala in any Databricks Runtime and are not supported for Python in Databricks Runtime 14.0 and above.Databricks recommends using the spark variable to interact with the SparkSession instance.The following sc functions are also not supported: emptyRDD, range, init_batched_serializer, parallelize, pickleFile, textFile, wholeTextFiles, binaryFiles, binaryRecords, sequenceFile, newAPIHadoopFile, newAPIHadoopRDD, hadoopFile, hadoopRDD, union, runJob, setSystemProperty, uiWebUrl, stop, setJobGroup, setLocalProperty, getConf.The following Scala Dataset API operations require Databricks Runtime 15.4 LTS or above: map, mapPartitions, foreachPartition, flatMap, reduce and filter.The Spark configuration property spark.executor.extraJavaOptions is not supported.UDF limitations and requirements for Unity Catalog standard access modeUser-defined functions (UDFs) have the following limitations with standard access mode:Hive UDFs are not supported.applyInPandas and mapInPandas require Databricks Runtime 14.3 or above.PySpark UDFs cannot access Git folders, workspace files, or volumes to import modules in Databricks Runtime 14.2 and below.Scala scalar UDFs require Databricks Runtime 14.2 or above. Other Scala UDFs and UDAFs are not supported.In Databricks Runtime 14.2 and below, using a custom version of grpc, pyarrow, or protobuf in a PySpark UDF through notebook-scoped or cluster-scoped libraries is not supported because the installed version is always preferred. To find the version of installed libraries, see the System Environment section of the specific Databricks Runtime version release notes.Python scalar UDFs and Pandas UDFs require Databricks Runtime 13.3 LTS or above.Non-scalar Python and Pandas UDFs, including UDAFs, UDTFs, and Pandas on Spark, require Databricks Runtime 14.3 LTS or above.See User-defined functions (UDFs) in Unity Catalog.Streaming limitations and requirements for Unity Catalog standard access modeYou cannot use the formats statestore and state-metadata to query state information for stateful streaming queries.transformWithState, transformWithStateInPandas, and associated APIs are not supported.For Scala, foreach requires Databricks Runtime 16.1 or above. foreachBatch, and flatMapGroupsWithState require Databricks Runtime 16.2 or above.For Python, foreachBatch has the following behavior changes in Databricks Runtime 14.0 and above:print() commands write output to the driver logs.You cannot access the
Add Comment