aws emr s3distcp:auxservice:mapreduce_shuffle 不存在

g6ll5ycj 于 2021-05-29 发布在 Hadoop

关注(0)|答案(1)|浏览(430)

我通过ssh连接到aws emr v5.4.0示例，我想调用s3distcp。此链接演示如何设置emr步骤以调用它，但当我运行它时，会出现以下错误：

Container launch failed for container_1492469375740_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:390)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

我按照这里的说明做了，但还是不管用。

hadoop elastic-map-reduce

来源：https://stackoverflow.com/questions/43478985/aws-emr-s3distcp-the-auxservicemapreduce-shuffle-does-not-exist

1条答案

按热度按时间

fcipmucu1#

结果是我需要在配置之后重新启动yarn nodemanager服务 mapreduce_shuffle :

$ initctl list | grep yarn
hadoop-yarn-resourcemanager start/running, process 1256
hadoop-yarn-proxyserver start/running, process 702
hadoop-yarn-nodemanager start/running, process 896
$ sudo stop hadoop-yarn-nodemanager
$ sudo start hadoop-yarn-nodemanager

另外，万一有助于 yarn-site.xml 文件位于： /etc/hadoop/conf/yarn-site.xml . 它已经有了一个 yarn.nodemanager.aux-services 但mapreduce\u shuffle未配置：

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>spark_shuffle,</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
  <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

所以我加了这样一句：

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>spark_shuffle,mapreduce_shuffle</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
  <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

赞(0）回复(0）举报 2021-05-29

我来回答

aws emr s3distcp:auxservice:mapreduce_shuffle 不存在

1条答案

相关问题

热门标签

最新问答