如何解决使用 Azure SQL DB MetaException 的 Databricks 外部配置单元元存储消息:在元存储中找不到版本信息
我正在尝试使用以下初始化脚本为 Databricks 设置外部配置单元元存储:
dbutils.fs.put(
"/databricks/scripts/external-metastore.sh","""#!/bin/sh
|# Loads environment variables to determine the correct JDBC driver to use.
|source /etc/environment
|# Quoting the label (i.e. EOF) with single quotes to disable variable interpolation.
|cat << 'EOF' > /databricks/driver/conf/00-custom-spark.conf
|[driver] {
| # Hive specific configuration options.
| # spark.hadoop prefix is added to make sure these Hive specific options will propagate to the metastore client.
| # JDBC connect string for a JDBC metastore
| "spark.hadoop.javax.jdo.option.ConnectionURL" = "jdbc:sqlserver://tst.database.windows.net:1433;database=db;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net"
|
| # Username to use against metastore database
| "spark.hadoop.javax.jdo.option.ConnectionUserName" = "admin"
|
| # Password to use against metastore database
| "spark.hadoop.javax.jdo.option.ConnectionPassword" = "password"
|
| # Driver class name for a JDBC metastore
| "spark.hadoop.javax.jdo.option.ConnectionDriverName" = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
|
| # Spark specific configuration options
| "spark.sql.hive.metastore.version" = "2.3.7"
| # Skip this one if <hive-version> is 0.13.x.
| "spark.sql.hive.metastore.jars" = "builtin"
|}
|EOF
|""".stripMargin,overwrite = true)
我运行这个 Scala 代码
spark.conf.get("spark.sql.hive.metastore.version")
输出:
res0: String = 2.3.7
但是当我尝试运行时:
spark.table("diamonds").withColumnRenamed("table","table_number")
.write
.jdbc(jdbcUrl,"diamonds",connectionProperties)
我收到此错误:
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
Caused by: HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: InvocationTargetException:
Caused by: MetaException: Version information not found in metastore.
解决方法
仅来自文档 External Apache Hive metastore:
- SQL Server 不能作为底层 Metastore 数据库用于 Hive 2.0 及更高版本。
请尝试更改版本。在这里,您可以参考另一个相同的问题: Azure Databricks external Hive Metastore
HTH。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。