我正在尝试注册我的azure blob存储到我的spark流媒体,但得到以下代码&error:-
code:-
SparkConf sparkConf = new SparkConf().setAppName("JavaNetworkWordCount");
JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, Durations.seconds(1));
ssc.textFileStream("wasb[s]://mycontainer@rtest.blob.core.windows.net/");
ssc.start();
ssc.awaitTermination();
不确定wasb链接的路径中应该有什么
https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#address-azure存储中的文件
链接说我应该给出一个路径,但是我的容器没有任何路径。图像直接存储在容器中。
error:-
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:163)
at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:513)
at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:573)
at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:554)
at org.bnr.process_panos.JavaNetworkWordCount.main(JavaNetworkWordCount.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
1条答案
按热度按时间2ic8powd1#
可以使用相对路径或绝对路径。例如,hdinsight clusters附带的hadoop-mapreduce-examples.jar文件可以使用以下方法之一进行引用:
例1:wasb://mycontainer@myaccount.blob.core.windows.net/example/jars/hadoop-mapreduce-examples.jar
示例2:wasb:///example/jars/hadoop-mapreduce-examples.jar
示例3:/example/jars/hadoop-mapreduce-examples.jar
以下错误消息在没有输出运算符的情况下出现在dstream上,不会调用任何计算。您需要在流上调用以下任何方法。
打印()
foreachrdd(函数)
saveasobjectfiles(前缀,[后缀])
saveastextfiles(前缀,[后缀])
saveashadoopfiles(前缀,[后缀])
有关详细信息,请参阅“http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-操作”。