如何解决许多死亡执行者EMR
我试图通过创建步骤Application Spark在AWS EMR集群上执行我的spark scala应用程序。
我的集群包含4 m3.xlarge
我使用以下命令启动我的应用程序：
spark-submit --deploy-mode cluster --class Main s3://mybucket/myjar_2.11-0.1.jar s3n://oc-mybucket/folder arg1 arg2
我的应用程序有3个参数，第一个是文件夹。
不幸的是，启动该应用程序后，我看到只有一个执行器（+主机）处于活动状态，而我有3个执行器死亡，因此所有任务仅在第一个执行。看图片
我尝试了许多方法来激活那些求职者，但是没有任何结果（“ spark.default.parallelism”，“ spark.executor.instances”和“ spark.executor.cores”）。 我应该怎么做才能使所有执行程序都处于活动状态并正在处理数据？
此外，在查看Ganglia时，我的CPU始终低于35％，是否有办法使CPU的唤醒率超过75％？
谢谢
UPDTAE
这是死掉的执行者的标准错误
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/14/__spark_libs__3671437061469038073.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/08/15 23:28:56 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 14765@ip-172-31-39-255
20/08/15 23:28:56 INFO SignalUtils: Registered signal handler for TERM
20/08/15 23:28:56 INFO SignalUtils: Registered signal handler for HUP
20/08/15 23:28:56 INFO SignalUtils: Registered signal handler for INT
20/08/15 23:28:57 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/08/15 23:28:57 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/08/15 23:28:57 INFO SecurityManager: Changing view acls groups to:
20/08/15 23:28:57 INFO SecurityManager: Changing modify acls groups to:
20/08/15 23:28:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn,hadoop); groups with view permissions: Set(); users with modify permissions: Set(yarn,hadoop); groups with modify permissions: Set()
20/08/15 23:28:58 INFO TransportClientFactory: Successfully created connection to ip-172-31-36-83.eu-west-1.compute.internal/172.31.36.83:37115 after 186 ms (0 ms spent in bootstraps)
20/08/15 23:28:58 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/08/15 23:28:58 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/08/15 23:28:58 INFO SecurityManager: Changing view acls groups to:
20/08/15 23:28:58 INFO SecurityManager: Changing modify acls groups to:
20/08/15 23:28:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn,hadoop); groups with modify permissions: Set()
20/08/15 23:28:58 INFO TransportClientFactory: Successfully created connection to ip-172-31-36-83.eu-west-1.compute.internal/172.31.36.83:37115 after 2 ms (0 ms spent in bootstraps)
20/08/15 23:28:58 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1597532473783_0002/blockmgr-d0d258ba-4345-45d1-9279-f6a97b63f81c
20/08/15 23:28:58 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1597532473783_0002/blockmgr-e7ae1e29-85fa-4df9-acf1-f9923f0664bc
20/08/15 23:28:58 INFO MemoryStore: MemoryStore started with capacity 2.6 GB
20/08/15 23:28:59 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@ip-172-31-36-83.eu-west-1.compute.internal:37115
20/08/15 23:28:59 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
20/08/15 23:28:59 INFO Executor: Starting executor ID 3 on host ip-172-31-39-255.eu-west-1.compute.internal
20/08/15 23:28:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40501.
20/08/15 23:28:59 INFO NettyBlockTransferService: Server created on ip-172-31-39-255.eu-west-1.compute.internal:40501
20/08/15 23:28:59 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/08/15 23:29:00 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(3,ip-172-31-39-255.eu-west-1.compute.internal,40501,None)
20/08/15 23:29:00 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(3,None)
20/08/15 23:29:00 INFO BlockManager: external shuffle service port = 7337
20/08/15 23:29:00 INFO BlockManager: Registering executor with local external shuffle service.
20/08/15 23:29:00 INFO TransportClientFactory: Successfully created connection to ip-172-31-39-255.eu-west-1.compute.internal/172.31.39.255:7337 after 20 ms (0 ms spent in bootstraps)
20/08/15 23:29:00 INFO BlockManager: Initialized BlockManager: BlockManagerId(3,None)
20/08/15 23:29:03 INFO CoarseGrainedExecutorBackend: eagerFSInit: Eagerly initialized FileSystem at s3://does/not/exist in 3363 ms
20/08/15 23:30:02 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
20/08/15 23:30:02 INFO DiskBlockManager: Shutdown hook called
20/08/15 23:30:02 INFO ShutdownHookManager: Shutdown hook called
此问题一定与内存有关吗？
解决方法
默认情况下，spark-submit不会使用所有执行器，您可以指定执行器--num-executors，executor-core和executor-memory的数量。
例如，增加执行程序（默认为2个）
spark-submit --num-executors N #where N is desired number of executors like 5,10,50
请参见docs here
中的示例如果它不能帮助或替代spark-submit，则可以覆盖spark.executor.instances文件或类似文件中的conf/spark-defaults.conf，因此您不必在命令行上明确指定
对于CPU利用率，您应该查看executor-core和executor-core并在spark-submit或conf中进行更改。增加cpu内核将有望增加使用率。
更新：
正如@Lamanus和我double checked所指出的，大于4.4的emr已将spark.dynamicAllocation.enabled设置为true，我建议您仔细检查数据分区，因为具有动态分配启用的执行程序实例数取决于分区数，分区数根据DAG执行阶段的不同而不同。此外，通过动态分配，您可以尝试spark.dynamicAllocation.initialExecutors，spark.dynamicAllocation.maxExecutors，spark.dynamicAllocation.maxExecutors来控制执行者。
这可能有点晚了，但是我发现这个AWS Big Data博客很有见地，可以确保充分利用我的大多数集群，并能够实现尽可能多的并行性。
更具体地说：
每个实例的执行程序数量=（每个实例的虚拟核心总数） 实例-1）/ spark.executors.cores
执行器总内存=每个实例的总RAM /执行器数 每个实例
然后您可以使用spark.default.parallelism或repartitioning控制阶段中并行任务的数量。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。