如何解决“securitymanager:身份验证已禁用;Spark分布簇异常“?

i1icjdpr  于 2021-07-06  发布在  Java
关注(0)|答案(0)|浏览(890)

我在linux(ubuntu)环境中创建了一个spark分发集群来运行spark提交作业。我有三台机器(m1,s1和s2;m1是主机器,s1和s2是从机器)。
在主spark和从spark目录中 ../spark/conf/ spark-defaults.conf , spark-env.sh . 我已经配置好了一切。
spark-env.sh公司

  1. export SPARK_MASTER_HOST=198.77.8.7
  2. export SPARK_WORKER_HOST=198.77.8.12
  3. export SPARK_WORKER_HOST=198.77.8.18
  4. export SPARK_LOCAL_IP=198.77.8.7
  5. export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  6. export SPARK_WORKER_CORES=8

spark-defaults.conf格式

  1. spark.master yarn
  2. spark.executor.memory 1g
  3. spark.driver.memory 1g
  4. spark.yarn.am.memory 1g

工作指令为:
bin/spark submit--master local[*]--deploy mode client--class com.x.y.app--executor memory 15g--executor cores 2--driver memory 15g/home/user/workspace/test.jar
我正在获取输出文件,但是当我在集群模式下运行时,它会给我一个异常。命令是:
bin/spark提交--主spark://198.47.12.8:7077--部署模式集群--类com.x.y.app--执行器内存15g--执行器内核2--驱动程序内存15g/home/user/workspace/test.jar
例外情况:

  1. 21/01/08 16:51:51 INFO SecurityManager: Changing modify acls groups to:
  2. 21/01/08 16:51:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sys92); groups with view permissions: Set(); users with modify permissions: Set(sys92); groups with modify permissions: Set()
  3. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  4. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  5. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  6. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  7. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  8. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  9. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  10. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  11. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  12. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  13. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  14. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  15. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  16. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  17. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  18. 21/01/08 16:51:51 WARN Utils: Service 'Driver' could not bind on a random free port. You may check whether configuring an appropriate binding address.
  19. Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'Driver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
  20. at sun.nio.ch.Net.bind0(Native Method)
  21. at sun.nio.ch.Net.bind(Net.java:461)
  22. at sun.nio.ch.Net.bind(Net.java:453)
  23. at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
  24. at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
  25. at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
  26. at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
  27. at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
  28. at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
  29. at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
  30. at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
  31. at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
  32. at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
  33. at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
  34. at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
  35. at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
  36. at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  37. at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  38. at java.lang.Thread.run(Thread.java:748)

我检查了所有3个系统 /etc/hosts 文件(我认为这不是问题)。
主etc/主机

  1. # 127.0.0.1 localhost
  2. # 127.0.1.1 mater1
  3. 198.77.8.7 mater1
  4. 198.77.8.12 slave1
  5. 198.77.8.18 slave2

salve1等/主机

  1. # 127.0.0.1 localhost
  2. # 127.0.1.1 slave1
  3. 198.77.8.7 mater1
  4. 198.77.8.12 slave1
  5. 198.77.8.18 slave2

从属2等/主机

  1. # 127.0.0.1 localhost
  2. # 127.0.1.1 slave2
  3. 198.77.8.7 mater1
  4. 198.77.8.12 slave1
  5. 198.77.8.18 slave2

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题