块池的hadoop hdfs格式化获取错误失败

b1zrtrql  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(360)

格式化hdfs后,出现以下错误:

  1. 2015-05-28 21:41:57,544 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop/dfs/datanode: namenode clusterID = CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8; datanode clusterID = CID-6c250e90-658c-4363-9346-972330ff8bf9
  2. 2015-05-28 21:41:57,545 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000. Exiting.
  3. java.io.IOException: All specified directories are failed to load.
  4. at.. org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
  5. at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1387)
  6. at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1352)
  7. at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
  8. at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228)
  9. at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:852)
  10. at java.lang.Thread.run(Thread.java:745)
  11. ...blah...
  12. SHUTDOWN_MSG:
  13. /************************************************************
  14. SHUTDOWN_MSG: Shutting down DataNode at der-Inspiron-3521/127.0.1.1
  15. ************************************************************/

以下是我所做的步骤:

  1. sbin/stop-dfs.sh
  2. hdfs namenode -format
  3. sbin/start-dfs.sh

供您参考:我的 core-site.xml文件 有体温吗 目录 具体如下:

  1. <property>
  2. <name>hadoop.tmp.dir</name>
  3. <value>/usr/local/hadoop</value>
  4. <description>A base for other temporary directories.
  5. </description>
  6. </property>

还有我的 hdfs-site.xml文件 作为 名称节点 和 数据节点 具体如下:

  1. <property>
  2. <name>dfs.namenode.name.dir</name>
  3. <value>file:/usr/local/hadoop/dfs/namenode</value>
  4. </property>
  5. <property>
  6. <name>dfs.datanode.data.dir</name>
  7. <value>file:/usr/local/hadoop/dfs/datanode</value>
  8. </property>

更新:我有一点进一步的问题,但我仍然得到相同类型的错误。我有能力运行 hdfs dfs -format 并按建议更改版本**。之后我用 hdfs dfs -ls and hdfs dfs -mkdir to create /user/der 我的登录名是什么。但是,当我运行mypig文件时,我在pig文件中获取mkdirs和chmod错误。以下是my datanode和namenode的权限:

  1. drwx------ 3 der der 4096 May 29 08:13 datanode
  2. drwxrwxrwx 4 root root 4096 May 28 11:34 name
  3. drwxrwxr-x 3 der der 4096 May 29 08:13 namenode
  4. drwxrwxr-x 3 der der 4096 May 29 08:13 namesecondary
  5. drwxr-xr-x 2 root root 4096 May 28 11:46 ww

似乎datanode只有所有者和组的权限,而没有用户的权限。
以下是我的pig脚本错误:

  1. 2015-05-29 08:37:27,152 [JobControl] INFO org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob - PigLatin:totalmiles.pig got an error while submitting
  2. ENOENT: No such file or directory
  3. at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
  4. at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
  5. at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:724)
  6. at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
  7. at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:600)
  8. at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
  9. at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:98)
  10. at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:193)
  11. at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
  12. at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
  13. at java.security.AccessController.doPrivileged(Native Method)
  14. at javax.security.auth.Subject.doAs(Subject.java:415)
  15. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  16. at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
  17. at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
  18. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  19. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl

这是我的Pig剧本:

  1. records = LOAD '1987.csv' USING PigStorage(',') AS
  2. (Year, Month, DayofMonth, DayOfWeek,
  3. DepTime, CRSDepTime, ArrTime, CRSArrTime,
  4. UniqueCarrier, FlightNum, TailNum,ActualElapsedTime,
  5. CRSElapsedTime,AirTime,ArrDelay, DepDelay,
  6. Origin, Dest, Distance:int, TaxIn,
  7. TaxiOut, Cancelled,CancellationCode, Diverted,
  8. CarrierDelay, WeatherDelay, NASDelay, SecurityDelay,
  9. lateAircraftDelay);
  10. milage_recs= GROUP records ALL;
  11. tot_miles = FOREACH milage_recs GENERATE SUM(records.Distance);
  12. STORE tot_miles INTO 'totalmiles4';

更新:顺便说一下,我在datanode上使用了chmodgo+rw(在我停止namenode服务器和datanode服务器之后)。但效果不太好。
更新5月30日:多一点细节。我将pig脚本中pig脚本的父目录更改为:

  1. records = LOAD '/user/der/1987.csv' USING PigStorage(',') AS

我也有同样的错误。在客户端,这里是错误。唯一的区别是,失败的输入读取没有hdfs://前缀。

  1. Failed to read data from "/user/der/1987.csv"
  2. Output(s):
  3. Failed to produce result in "hdfs://localhost:9000/user/der/totalmiles4"

在服务器端,这里是namenode日志,就在我从pig脚本得到无效文件请求的那一刻。日志(使用tail-f)滚动。这表示服务器正在接受pig命令的请求。

  1. 2015-05-30 07:01:28,140 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50010 is added to
  2. blk_1073741885_1061{UCState=UNDER_CONSTRUCTION,
  3. truncateBlock=null,
  4. primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c84e0e37-2726-44da-af3e-67167c1010d1:NORMAL:127.0.0.1:50010|RBW]]}
  5. size 0
  6. 2015-05-30 07:01:28,148 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile:
  7. /tmp/temp-11418443/tmp85697770/automaton-1.11-8.jar
  8. is closed by DFSClient_NONMAPREDUCE_-1939565577_1

我只需要获取pig脚本的源代码并检查它发出的extact hdfs命令。我认为我配置的hadoop hdfs服务有问题。

snvhrwxg

snvhrwxg1#

2015-05-28 21:41:57544警告org.apache.hadoop.hdfs.server.common.storage:java.io.ioexception:namenode clusterid=cid-e77ee39a-ab4de1-b1a4-9d4da78b83e8;数据节点群集ID=cid-6c250e90-658c-4363-9346-972330ff8bf9
您的namenode和datanode群集id不匹配。
打开你的 usr/local/hadoop/dfs/datanode/current/VERSION 文件和更改:

  1. clusterID=CID-6c250e90-658c-4363-9346-972330ff8bf9

  1. clusterID=CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8

注意:每当格式化namenode时,请检查namenode和datanode的版本文件。它们都应该具有相同的clusterid和namespaceid。否则,您的datanode将无法启动。

lyfkaqu1

lyfkaqu12#

同样的问题也发生在我身上。我删除了data目录的内容,然后运行hdfs datanode。这在datanode目录中创建了版本和其他文件。这解决了我的问题。

相关问题