当我键入以下命令让我的java spark从amazons3读取csv文件时
spark-submit --class myApp --packages org.apache.hadoop:hadoop-aws:2.7.6 --master local ./target/spark_pratice.jar
错误消息
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::
:: javax.activation#activation;1.1!activation.jar
:: asm#asm;3.2!asm.jar
:: org.apache.avro#avro;1.7.4!avro.jar
:: com.thoughtworks.paranamer#paranamer;2.3!paranamer.jar
:: org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.jar(bundle)
:: io.netty#netty;3.6.2.Final!netty.jar(bundle)
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [
download failed: javax.activation#activation;1.1!activation.jar,
download failed: asm#asm;3.2!asm.jar,
download failed: org.apache.avro#avro;1.7.4!avro.jar,
download failed: com.thoughtworks.paranamer#paranamer;2.3!paranamer.jar,
download failed: org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.jar(bundle),
download failed: io.netty#netty;3.6.2.Final!netty.jar(bundle)
]
我不知道如何解决这个问题。我需要更新什么吗?Maven?或者别的什么?
2条答案
按热度按时间k3fezbri1#
我也面临同样的问题&我已经删除了
.ivy2/cache
,ivy2/jars
以及.m2/repository/
目录和执行spark-shell --package
命令它开始工作。7ivaypg92#
最后,我尝试从maven repository maven repository下载丢失的jar文件
我把它们放到了.m2/repository的正确存储库中