如何解决火花提交传递--files
我想在spark-submit中传递两个文件：
- key.jks
- trustore.jks
位置如下：
- /home/my_user/config/key.jks
- /home/my_user/config/trustore.jks
我执行以下命令：
spark-submit --name historization \
--class com.ConsumerLauncher \
--master yarn \
--deploy-mode cluster \
--files /home/my_user/config/key.jks,/home/my_user/config/trustore.jks \
/home/my_user/jars/app.jar
我想在我的代码中将这两个文件用于我的kafka参数：
val kafkaParams = Map[String,Object](
"bootstrap.servers" -> "myhost","group.id" -> "grp-test","key.deserializer" -> classOf[StringDeserializer],"value.deserializer" -> classOf[StringDeserializer],"auto.offset.reset" -> "earliest","enable.auto.commit" -> (false: java.lang.Boolean),"security.protocol" -> "SSL","ssl.truststore.password" -> "xxxx","ssl.key.password" -> "xxxx","ssl.keystore.password" -> "xxxx",//this two lines
"ssl.truststore.location" -> "/home/my_user/config/trustore.jks","ssl.keystore.location" -> "/home/my_user/config/key.jks"
//this two line also bug
//"ssl.truststore.location" -> getClass.getResource("/ssl/trustore.jks").getPath,//"ssl.keystore.location" -> getClass.getResource("/ssl/key.jks").getPath
)
不幸的是我得到了这个错误
Caused by: java.io.FileNotFoundException: /home/my_user/config/key.jks (No such file or directory)
你有什么主意吗？
解决方法
使用--files时，文件将上传到HDFS。 您现在可以通过以下方式访问：trustore.jks和key.jks
val kafkaParams = Map[String,Object](
"bootstrap.servers" -> "myhost","group.id" -> "grp-test","key.deserializer" -> classOf[StringDeserializer],"value.deserializer" -> classOf[StringDeserializer],"auto.offset.reset" -> "earliest","enable.auto.commit" -> (false: java.lang.Boolean),"security.protocol" -> "SSL","ssl.truststore.password" -> "xxxx","ssl.key.password" -> "xxxx","ssl.keystore.password" -> "xxxx",//this two lines
"ssl.truststore.location" -> "trustore.jks","ssl.keystore.location" -> "key.jks"
//this two line also bug
//"ssl.truststore.location" -> getClass.getResource("/ssl/trustore.jks").getPath,//"ssl.keystore.location" -> getClass.getResource("/ssl/key.jks").getPath
)
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。