如何解决Spark提交:您尚未在本地或通过ConfigMap指定krb5.conf文件 提交Spark
我用来直接进行Spark提交,以将Spark应用程序提交到Kubernetes集群。使用以下命令:
提交Spark
./bin/spark-submit --master k8s://https://localhost:8443
--deploy-mode cluster --name spark-pi
--class com.xxx.Main
--conf spark.executor.instances=5
--conf spark.kubernetes.container.image=spark:latest
local:///home/SparkJob_jar spark.kubernetes.file.upload.path=/opt/xxx
但是我遇到了以下异常:
20/09/25 11:33:57 WARN Utils: Your hostname,XXXX resolves to a loopback address: 127.0.1.1; using xxx.yyy.zzz instead (on interface wlp59s0)
20/09/25 11:33:57 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/09/25 11:33:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/09/25 11:33:58 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
20/09/25 11:33:58 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
20/09/25 11:33:58 WARN WatchConnectionManager: Exec Failure
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
帖子https://issues.apache.org/jira/browse/SPARK-31800建议添加 spark.kubernetes.file.upload.path 来解决该问题,我将其添加到了Submit命令中,但问题仍然存在。感谢您为找出此问题的根本原因提供的帮助。
解决方法
您可能需要添加以下配置标志以使证书可用
--conf spark.kubernetes.authenticate.caCertFile=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
--conf spark.kubernetes.authenticate.oauthTokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。