获取错误,因为尝试用movielens数据从hdfs加载数据时未能创建数据存储

flvtvl50  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(209)

我试图加载数据从hdfs到Pig,但我得到的错误,因为无法创建数据存储。我执行的命令是:

movies = LOAD 'hdfs://localhost:9000/Movie_Lens/ratings' USING PigStorage(':') AS (user_id, dummy1, movie_id, dummy2, movie_rating, dummy3, timestamp);

我试图在堆栈溢出中找到上述问题,但我得到的链接与hdfs和pig无关,它们与hdfs和hbase或pig和hbase有关。
日志文件的详细信息如下所述。
在日志文件的某个地方我发现了这样一个问题:原因:org.apache.hadoop.ipc.remoteexception:服务器ipc版本9无法与客户端版本4通信

清管器堆放痕迹

错误1200:未能创建数据存储

Failed to parse: Failed to create DataStorage
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:201)
    at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1707)
    at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1680)
    at org.apache.pig.PigServer.registerQuery(PigServer.java:623)
    at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1082)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:505)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:565)
    at org.apache.pig.Main.main(Main.java:177)
Caused by: java.lang.RuntimeException: Failed to create DataStorage
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:53)
    at org.apache.pig.builtin.JsonMetadata.findMetaFile(JsonMetadata.java:109)
    at org.apache.pig.builtin.JsonMetadata.getSchema(JsonMetadata.java:189)
    at org.apache.pig.builtin.PigStorage.getSchema(PigStorage.java:538)
    at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
    at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
    at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901)
    at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
    at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
    at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
    at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
    at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
    at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
    ... 10 more
Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
    at org.apache.hadoop.ipc.Client.call(Client.java:1070)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy4.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:70)
    ... 23 more

为了解决这个问题,我尝试了使用“ant”,所以当我运行命令时

bash ant -version

在antbin文件夹中,它正在工作,但当我运行命令时

bash ant clean jar-all -Dhadoopversion=23

在bin文件夹中,它不工作。在一些链接中,我发现新版本的pig没有jar all命令,所以我尝试了以下命令

bash ant clean jar -Dhadoopversion=23

而且这个命令也不起作用。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题