在从属节点(yarn)上运行spark不起作用

piztneat  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(478)

我可以在主节点上运行sparkpi示例，但在从属节点上尝试相同的命令“spark submit--class sparkpi--master warn client sparkpi.jar 10”时，出现错误：

  1. 2015-05-19 14:05:44,881 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: maintainer
  2. 2015-05-19 14:05:44,886 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: maintainer
  3. 2015-05-19 14:05:44,887 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(maintainer); users with modify permissions: Set(maintainer)
  4. 2015-05-19 14:05:45,389 INFO  [sparkDriver-akka.actor.default-dispatcher-4] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
  5. 2015-05-19 14:05:45,443 INFO  [sparkDriver-akka.actor.default-dispatcher-4] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
  6. 2015-05-19 14:05:45,641 INFO  [sparkDriver-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriver@slave2.com:33055]
  7. 2015-05-19 14:05:45,644 INFO  [sparkDriver-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting now listens on addresses: [akka.tcp://sparkDriver@slave2.com:33055]
  8. 2015-05-19 14:05:45,653 INFO  [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'sparkDriver' on port 33055.
  9. 2015-05-19 14:05:45,674 INFO  [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering MapOutputTracker
  10. 2015-05-19 14:05:45,688 INFO  [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering BlockManagerMaster
  11. 2015-05-19 14:05:45,707 INFO  [main] storage.DiskBlockManager (Logging.scala:logInfo(59)) - Created local directory at /tmp/spark-local-20150519140545-c81b
  12. 2015-05-19 14:05:45,712 INFO  [main] storage.MemoryStore (Logging.scala:logInfo(59)) - MemoryStore started with capacity 265.4 MB
  13. 2015-05-19 14:05:46,205 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  14. 2015-05-19 14:05:46,408 INFO  [main] spark.HttpFileServer (Logging.scala:logInfo(59)) - HTTP File server directory is /tmp/spark-e95a2b5b-efea-41eb-93b9-0a9f7d6f6701
  15. 2015-05-19 14:05:46,413 INFO  [main] spark.HttpServer (Logging.scala:logInfo(59)) - Starting HTTP Server
  16. 2015-05-19 14:05:46,477 INFO  [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
  17. 2015-05-19 14:05:46,499 INFO  [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SocketConnector@0.0.0.0:52737
  18. 2015-05-19 14:05:46,500 INFO  [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'HTTP file server' on port 52737.
  19. 2015-05-19 14:05:46,790 INFO  [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
  20. 2015-05-19 14:05:46,805 INFO  [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector@0.0.0.0:4040
  21. 2015-05-19 14:05:46,805 INFO  [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'SparkUI' on port 4040.
  22. 2015-05-19 14:05:46,808 INFO  [main] ui.SparkUI (Logging.scala:logInfo(59)) - Started SparkUI at http://slave2.com:4040
  23. 2015-05-19 14:05:47,058 INFO  [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added JAR file:/home/maintainer/myjars/sparkpi.jar at http://[ip]:52737/jars/sparkpi.jar with timestamp 1432033547057
  24. 2015-05-19 14:05:47,190 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
  25. 2015-05-19 14:09:45,861 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
  27. **2015-05-19 14:09:47,067 INFO  [main] ipc.Client (Client.java:handleConnectionFailure(842)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  29. 2015-05-19 14:09:48,068 INFO  [main] ipc.Client (Client.java:handleConnectionFailure(842)) - Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  30. ...**
hadoopyarnapache-sparkmaster-slave

2条答案

按热度按时间
x6h2sr28

x6h2sr281#

为导出正确的值 HADOOP_CONF_DIR 解决了这个问题。

  1. export HADOOP_CONF_DIR=/your-path/hadoop/conf
赞(0）分享  回复(0）举报  2021-06-03
ux6nzvsh

ux6nzvsh2#

除了指定 yarn.resourcemanager.hostname 中的属性 yarn-site.xml ，还需要将配置文件传播到工作程序。
可能是用这条线（在运行之前）完成的 spark-submit ):

  1. export SPARK_YARN_DIST_FILES=$(ls $HADOOP_CONF_DIR* | sed 's#^#file://#g' | tr '\n' ',' | sed 's/,$//')

如果所有配置都正确，您将看到rm hostname而不是 0.0.0.0 在此行中：

  1. 2015-05-19 14:05:47,190 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032
赞(0）分享  回复(0）举报  2021-06-02
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com