我在bigdata方面比较新,我试着用apachelivy调用spark jobs。使用submit命令行工作正常。我和利维有个例外
命令行:
curl-x post--data'{“file”:“/user/romain/spark examples.jar”,“classname”:“org.apache.spark.examples.sparkpi”}'-h'内容类型:application/json'http://localhost:8998/批
livy日志: 2019-06-01 00:43:19,160 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.io.FileNotFoundException: File hdfs://localhost:9000/home/spark-2.4.3-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.3.jar does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795) .......org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2条答案
按热度按时间bzzcjhmw1#
如果它在本地计算机中,则尝试使用
'file':'local:<path/to/file>
hc2pp10m2#
与@cricket\u 007注解相关,由execute hadoop fs-copyfromlocal命令行解析