spark流访问azure blob

siv3szwd  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(466)

我正在尝试注册我的azure blob存储到我的spark流媒体,但得到以下代码&error:-
code:-

SparkConf sparkConf = new SparkConf().setAppName("JavaNetworkWordCount");
        JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, Durations.seconds(1));
        ssc.textFileStream("wasb[s]://mycontainer@rtest.blob.core.windows.net/");
        ssc.start();
        ssc.awaitTermination();

不确定wasb链接的路径中应该有什么
https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-use-blob-storage#address-azure存储中的文件
链接说我应该给出一个路径,但是我的容器没有任何路径。图像直接存储在容器中。
error:-

java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
    at scala.Predef$.require(Predef.scala:224)
    at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:163)
    at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:513)
    at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:573)
    at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
    at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:554)
    at org.bnr.process_panos.JavaNetworkWordCount.main(JavaNetworkWordCount.java:43)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2ic8powd

2ic8powd1#

可以使用相对路径或绝对路径。例如,hdinsight clusters附带的hadoop-mapreduce-examples.jar文件可以使用以下方法之一进行引用:
例1:wasb://mycontainer@myaccount.blob.core.windows.net/example/jars/hadoop-mapreduce-examples.jar
示例2:wasb:///example/jars/hadoop-mapreduce-examples.jar
示例3:/example/jars/hadoop-mapreduce-examples.jar
以下错误消息在没有输出运算符的情况下出现在dstream上,不会调用任何计算。您需要在流上调用以下任何方法。
打印()
foreachrdd(函数)
saveasobjectfiles(前缀,[后缀])
saveastextfiles(前缀,[后缀])
saveashadoopfiles(前缀,[后缀])
有关详细信息,请参阅“http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-操作”。

相关问题