ec2和超光速粒子hadoop版本

qvtsj1bj  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(283)

我尝试使用spark-ec2启动带有hadoop版本2.x的ec2集群,所以我尝试了:

  1. ./spark-ec2 -k spark -i ~/.ssh/spark.pem -s 1 --hadoop-major-version=2 launch my-spark-cluster

然后我发现在超光速粒子设置过程中有错误:

  1. Setting up tachyon
  2. RSYNC'ing /root/tachyon to slaves...
  3. ec2-52-1-147-16.compute-1.amazonaws.com
  4. ec2-52-1-147-16.compute-1.amazonaws.com: Formatting Tachyon Worker @ ip-172-31-21-86.ec2.internal
  5. ec2-52-1-147-16.compute-1.amazonaws.com: Removing local data under folder: /mnt/ramdisk/tachyonworker/
  6. Formatting Tachyon Master @ ec2-52-1-14-186.compute-1.amazonaws.com
  7. Formatting JOURNAL_FOLDER: /root/tachyon/libexec/../journal/
  8. Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
  9. at tachyon.util.CommonUtils.runtimeException(CommonUtils.java:246)
  10. at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:73)
  11. at tachyon.UnderFileSystemHdfs.getClient(UnderFileSystemHdfs.java:53)
  12. at tachyon.UnderFileSystem.get(UnderFileSystem.java:53)
  13. at tachyon.Format.main(Format.java:54)
  14. Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
  15. at org.apache.hadoop.ipc.Client.call(Client.java:1070)
  16. at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
  17. at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
  18. at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
  19. at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
  20. at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
  21. at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
  22. at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
  23. at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
  24. at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
  25. at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
  26. at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
  27. at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
  28. at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
  29. at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:69)
  30. ... 3 more

我找了一些相关的问题 Server IPC version 7 cannot communicate with client version 4 表示服务器使用Hadoop2.x,客户端使用Hadoop1.x。但是,我用hadoop2.4.0构建了spark,还用hadoop2.4.0和更高版本尝试了官方的spark预构建版本,这两个版本都导致了相同的错误。
顺便说一下,hadoop版本是通过设置 --hadoop-major-version=2Hadoop 2.0.0-cdh4.2.0 . 这有问题吗?但是我尝试在这里使用2.4或2.4.0,它们都不是有效的hadoop版本

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题