如何解决在Spark会话中声明时,spark 2.4.4内核失败
我在使用Spark应用程序资源时确实遇到了一个奇怪的问题。我的spark spark-defaults.conf 文件如下所示。
spark.executor.memory 9486M
spark.executor.cores 4
在我的应用程序代码中,我声明没有比默认更大的内核。就像下面的
spark = SparkSession \
.builder \
.enableHiveSupport()\
.config("spark.executor.memory","8g")\
.config("spark.executor.cores","6")\
.appName(app_name)\
.getOrCreate()
火花提交看起来像
spark-submit --master yarn --deploy-mode cluster main.py
我的应用程序失败,出现以下错误
LogType:stdout
Log Upload Time:Sun Aug 16 13:02:27 +0000 2020
LogLength:22066
Log Contents:
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 1159,in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception,another exception occurred:
Traceback (most recent call last):
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 985,in send_command
response = connection.send_command(command)
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 1164,in send_command
"Error while receiving",e,proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File "main.py",line 50,in <module>
.appName(app_name)\
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/pyspark.zip/pyspark/sql/session.py",line 173,in getOrCreate
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/pyspark.zip/pyspark/context.py",line 375,line 136,in __init__
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/pyspark.zip/pyspark/context.py",line 198,in _do_init
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/pyspark.zip/pyspark/context.py",line 314,in _initialize_context
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 1525,in __call__
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/protocol.py",line 336,in get_return_value
py4j.protocol.Py4JError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:45417)
Traceback (most recent call last):
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 929,in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception,line 1067,in start
self.socket.connect((self.address,self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:45417)
Traceback (most recent call last):
File "/mnt1/yarn/usercache/hadoop/appcache/application_1592991612264_0475/container_1592991612264_0475_01_000001/py4j-0.10.7-src.zip/py4j/java_gateway.py",in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
日志文件很大,但这是相关信息。 现在,有趣的是,当我将内核数减少到等于或小于4时,它起作用了。我的印象是,在资源配置时spark优先考虑应用程序代码。 有人遇到这样的问题吗?请在这里帮助我。
我的Spark版本:
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。