hadoop无法设置路径：\tmp的权限\

ryhaxcpt  于 2021-06-04  发布在  Hadoop
关注(0)|答案(2)|浏览(545)

我在cygwin的windows机器上运行hadoop1.2.1。不幸的是，我的hadoop有严重的问题。当我试图在本地模式下执行简单的pig脚本时，出现以下错误。

  1. Backend error message during job submission
  2. -------------------------------------------
  3. java.io.IOException: Failed to set permissions of path: \tmp\hadoop-antonbelev\mapred\staging\antonbelev1696923409\.staging to 0700
  4.     at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
  5.     at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
  6.     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
  7.     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
  8.     at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
  9.     at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
  10.     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
  11.     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
  12.     at java.security.AccessController.doPrivileged(Native Method)
  13.     at javax.security.auth.Subject.doAs(Subject.java:415)
  14.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  15.     at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  16.     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
  17.     at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
  18.     at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
  19.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  20.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  21.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  22.     at java.lang.reflect.Method.invoke(Method.java:601)
  23.     at org.apache.pig.backend.hadoop20.PigJobControl.mainLoopAction(PigJobControl.java:157)
  24.     at org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:134)
  25.     at java.lang.Thread.run(Thread.java:722)
  26.     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
  28. Pig Stack Trace
  29. ---------------
  30. ERROR 2244: Job failed, hadoop does not return any error message
  32. org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
  33.     at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:148)
  34.     at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
  35.     at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
  36.     at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
  37.     at org.apache.pig.Main.run(Main.java:607)
  38.     at org.apache.pig.Main.main(Main.java:156)
  39.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  40.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  41.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  42.     at java.lang.reflect.Method.invoke(Method.java:601)
  43.     at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

我假设hadoop安装或配置文件有问题，但我对hadoop还不熟悉，所以这只是一个假设。有人能帮我解决这个问题吗。谢谢！：）
ps也为什么在路上 \tmp\hadoop-antonbelev\mapred\staging\antonbelev1696923409\.staging to 0700 是否使用windows反斜杠？我试图找到这个文件，但它不存在。
更新：
这里是我的配置文件：
core-site.xml：

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  4. <!-- Put site-specific property overrides in this file. -->
  6. <configuration>
  7.     <property>
  8.         <name>fs.default.name</name>
  9.         <value>localhost:9100</value>
  10.     </property>
  11. </configuration>

hdfs-site.xml：

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  4. <!-- Put site-specific property overrides in this file. -->
  6. <configuration>
  7.     <property>
  8.         <name>dfs.replication</name>
  9.         <value>1</value>
  10.     </property>
  11. </configuration>

mapred-site.xml:

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  4. <!-- Put site-specific property overrides in this file. -->
  6. <configuration>
  7.     <property>
  8.         <name>mapred.job.tracker</name>
  9.         <value>localhost:9101</value>
  10.     </property>
  11. </configuration>

hadoop-env.sh文件：

  2. # Set Hadoop-specific environment variables here.
  4. # The only required environment variable is JAVA_HOME.  All others are
  6. # optional.  When running a distributed configuration it is best to
  8. # set JAVA_HOME in this file, so that it is correctly defined on
  10. # remote nodes.
  12. # The java implementation to use.  Required.
  14.  export JAVA_HOME="C:/Program Files/Java/jdk1.7.0_07"
  16. # Extra Java CLASSPATH elements.  Optional.
  18. # export HADOOP_CLASSPATH=
  20. # The maximum amount of heap to use, in MB. Default is 1000.
  22. # export HADOOP_HEAPSIZE=2000
  24. # Extra Java runtime options.  Empty by default.
  26. # export HADOOP_OPTS=-server
  28. # Command specific options appended to HADOOP_OPTS when specified
  30. export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
  31. export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
  32. export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
  33. export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
  34. export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
  36. # export HADOOP_TASKTRACKER_OPTS=
  38. # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
  40. # export HADOOP_CLIENT_OPTS
  42. # Extra ssh options.  Empty by default.
  44. # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
  46. # Where log files are stored.  $HADOOP_HOME/logs by default.
  48. # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
  50. # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
  52. # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
  54. # host:path where hadoop code should be rsync'd from.  Unset by default.
  56. # export HADOOP_MASTER=master:/home/$USER/src/hadoop
  58. # Seconds to sleep between slave commands.  Unset by default.  This
  60. # can be useful in large clusters, where, e.g., slave rsyncs can
  62. # otherwise arrive faster than the master can service them.
  64. # export HADOOP_SLAVE_SLEEP=0.1
  66. # The directory where pid files are stored. /tmp by default.
  68. # NOTE: this should be set to a directory that can only be written to by
  70. # the users that are going to run the hadoop daemons.  Otherwise there is
  72. # the potential for a symlink attack.
  74. # export HADOOP_PID_DIR=/var/hadoop/pids
  76. # A string representing this instance of hadoop. $USER by default.
  78. # export HADOOP_IDENT_STRING=$USER
  80. # The scheduling priority for daemon processes.  See 'man nice'.
  82. # export HADOOP_NICENESS=10

我不确定其他配置文件是否相关。

hadoopwindowscygwin

2条答案

按热度按时间
uyhoqukh

uyhoqukh1#

尝试更改用作hadoop tmp文件夹的文件夹的文件权限。比如：

  1. sudo chmod a+w /app/hadoop/tmp -R
赞(0）分享  回复(0）举报  2021-06-04
b1zrtrql

b1zrtrql2#

请像这样在core-site.xml中添加此条目

  1. <property>
  2. <name>hadoop.tmp.dir</name>
  3.  <value>/tmp/hadoop-${user.name}</value>
  4.  <description>A base for other temporary directories.</description>
  5. </property>

配置的问题是hadoop正在读取的tmp文件夹位于root或/tmp下。

赞(0）分享  回复(0）举报  2021-06-04
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com