尝试写入hdfs时发生invalidprotocolbufferexception

6mzjoqzu  于 2021-06-03  发布在  Hadoop
关注(0)|答案(3)|浏览(456)

这是我的密码:

Configuration conf = new Configuration();
        conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
        conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
        conf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml"));
        conf.addResource(new Path("/etc/hadoop/conf/yarn-site.xml"));
        FileSystem fs = FileSystem.get(new URI("hdfs://localhost:8020"),conf);
        Path path = new Path(hdfsDestination);
        FSDataOutputStream outputStream = fs.create(path);

hdfsdestination是/user/msknapp/insurance,下面是输出:

java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).; Host Details : local host is: "localhost.localdomain/127.0.0.1"; destination host is: "localhost":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
    at org.apache.hadoop.ipc.Client.call(Client.java:1351)
    at org.apache.hadoop.ipc.Client.call(Client.java:1300)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    at com.sun.proxy.$Proxy9.create(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy9.create(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:227)
    at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1389)
    at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1382)
    at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1307)
    at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:384)
    at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:380)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:380)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:324)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:772)
    at knapp.hdfs.importer.insurance.InsuranceIO.writeToHdfsAsAvro(InsuranceIO.java:68)
    at knapp.hdfs.importer.insurance.InsuranceIO.writeFileToHdfsAsAvro(InsuranceIO.java:51)
    at knapp.hdfs.importer.ImporterCommandLine.run(ImporterCommandLine.java:72)
    at knapp.hdfs.importer.ImporterCommandLine.main(ImporterCommandLine.java:61)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
    at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.<init>(RpcHeaderProtos.java:1398)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.<init>(RpcHeaderProtos.java:1362)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1492)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto$1.parsePartialFrom(RpcHeaderProtos.java:1487)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
    at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
    at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
    at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
    at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)

我让cdh4.8以伪分布式模式运行:

[msknapp@localhost conf]$ hadoop fs -ls /user/msknapp
Found 2 items
drwxr-xr-x   - msknapp supergroup          0 2014-01-01 10:22 /user/msknapp/input
drwxr-xr-x   - msknapp supergroup          0 2014-01-01 10:35 /user/msknapp/output23
[msknapp@localhost conf]$ hadoop fs -ls /user
Found 1 items
drwxrwxrwx   - msknapp supergroup          0 2014-01-01 10:35 /user/msknapp
[msknapp@localhost conf]$ whereis hadoop
hadoop: /usr/bin/hadoop /etc/hadoop /usr/lib/hadoop /usr/share/man/man1/hadoop.1.gz

我用的是centos 6.4。我可以从命令行将数据放到hdfs中,但是由于某些原因不能从代码中。有人能告诉我为什么我不能用我的代码写hdfs吗?

bq8i3lrv

bq8i3lrv1#

你想写在这个路径“/usr/msknapp/insurance”但是显示权限这个路径“/user/msknapp”!

fhity93d

fhity93d2#

我也遇到了同样的异常,我发现缺少了一些依赖项,我只有hadoop内核,但还需要一些。当我添加这些时,它起了作用:

<dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-core</artifactId>
                <version>2.0.0-mr1-cdh4.3.0</version>
                <scope>test</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-hdfs</artifactId>
                <version>2.0.0-cdh4.3.0</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>2.0.0-cdh4.3.0</version>
            </dependency>
r3i60tvu

r3i60tvu3#

似乎正在使用protobuf-java-x.jar的不兼容版本。请检查用于构建应用程序的protobufjar版本以及cdh4.8中使用的内容

相关问题