ec2和超光速粒子hadoop版本

qvtsj1bj  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(257)

我尝试使用spark-ec2启动带有hadoop版本2.x的ec2集群,所以我尝试了:

./spark-ec2 -k spark -i ~/.ssh/spark.pem -s 1 --hadoop-major-version=2 launch my-spark-cluster

然后我发现在超光速粒子设置过程中有错误:

Setting up tachyon
RSYNC'ing /root/tachyon to slaves...
ec2-52-1-147-16.compute-1.amazonaws.com
ec2-52-1-147-16.compute-1.amazonaws.com: Formatting Tachyon Worker @ ip-172-31-21-86.ec2.internal
ec2-52-1-147-16.compute-1.amazonaws.com: Removing local data under folder: /mnt/ramdisk/tachyonworker/
Formatting Tachyon Master @ ec2-52-1-14-186.compute-1.amazonaws.com
Formatting JOURNAL_FOLDER: /root/tachyon/libexec/../journal/
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
    at tachyon.util.CommonUtils.runtimeException(CommonUtils.java:246)
    at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:73)
    at tachyon.UnderFileSystemHdfs.getClient(UnderFileSystemHdfs.java:53)
    at tachyon.UnderFileSystem.get(UnderFileSystem.java:53)
    at tachyon.Format.main(Format.java:54)
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
    at tachyon.UnderFileSystemHdfs.<init>(UnderFileSystemHdfs.java:69)
    ... 3 more

我找了一些相关的问题 Server IPC version 7 cannot communicate with client version 4 表示服务器使用Hadoop2.x,客户端使用Hadoop1.x。但是,我用hadoop2.4.0构建了spark,还用hadoop2.4.0和更高版本尝试了官方的spark预构建版本,这两个版本都导致了相同的错误。
顺便说一下,hadoop版本是通过设置 --hadoop-major-version=2Hadoop 2.0.0-cdh4.2.0 . 这有问题吗?但是我尝试在这里使用2.4或2.4.0,它们都不是有效的hadoop版本

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题