使用byoc hdinsight cluster(linux)的adf配置单元活动失败

93ze6v8z  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(476)

尝试执行adf配置单元活动时出现以下错误。当我直接在hdinsight集群上运行配置单元查询时,它可以正常工作。但在adfHive活动中运行时,它正在失败。我做了很多尝试和错误,但问题仍然存在。有人知道会有什么问题吗?

WARNING: Use "yarn jar" to launch YARN applications.
17/02/09 06:09:45 INFO impl.TimelineClientImpl: Timeline service address: http://headnodehost:8188/ws/v1/timeline/
17/02/09 06:09:46 INFO impl.TimelineClientImpl: Timeline service address: http://headnodehost:8188/ws/v1/timeline/
17/02/09 06:09:46 WARN ipc.Client: Failed to connect to server: hn0-xxxxx.xxxxx133hn2u3kb1xxx0vlmsre.jx.internal.cloudapp.net/10.0.0.17:8050: retries get failed due to exceeded maximum allowed retries number: 0
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:649)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:744)
    at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:397)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
    at org.apache.hadoop.ipc.Client.call(Client.java:1431)
    at org.apache.hadoop.ipc.Client.call(Client.java:1392)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy19.getNewApplication(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getNewApplication(ApplicationClientProtocolPBClientImpl.java:221)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy20.getNewApplication(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:220)
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:228)
    at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:188)
    at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:231)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:153)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
    at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:1014)
    at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:135)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
    at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
17/02/09 06:09:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
17/02/09 06:09:50 INFO mapred.FileInputFormat: Total input paths to process : 1
17/02/09 06:09:51 INFO mapreduce.JobSubmitter: number of splits:1
17/02/09 06:09:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1486536013226_0707
17/02/09 06:09:51 INFO mapreduce.JobSubmitter: Kind: mapreduce.job, Service: job_1486536013226_0706, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@7a399295)
17/02/09 06:09:53 INFO impl.YarnClientImpl: Submitted application application_1486536013226_0707
17/02/09 06:09:53 INFO mapreduce.Job: The url to track the job: http://hn1-xxxxx.xxxxx133hn2u3kb1xxx0vlmsre.jx.internal.cloudapp.net:8088/proxy/application_1486536013226_0707/
17/02/09 06:09:53 INFO mapreduce.Job: Running job: job_1486536013226_0707
17/02/09 06:10:11 INFO mapreduce.Job: Job job_1486536013226_0707 running in uber mode : false
17/02/09 06:10:11 INFO mapreduce.Job:  map 0% reduce 0%
17/02/09 06:10:31 INFO mapreduce.Job: Task Id : attempt_1486536013226_0707_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
mlnl4t2r

mlnl4t2r1#

我让它工作了。实际上,.net活动在数据工厂中失败了,它使用链接服务作为hdinsight集群(我使用的是基于linux的集群)。我将链接服务更改为azure批量帐户链接服务,并且此管道按预期工作。

相关问题