在hadoop集群中运行mrjob python脚本时出错

tkclm6bt  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(382)

嗨,我想排序的python脚本电影收视率,但我得到的错误

`[root@sandbox-hdp maria_dev]# python RatingsBreakdown.py -r hadoop --hadoop-streaming-jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar u.data
No configs found; falling back on auto-configuration
No configs specified for hadoop runner
Looking for hadoop binary in $PATH...
Found hadoop binary: /usr/bin/hadoop
Using Hadoop version 3.1.1.3.0.1.0
Creating temp directory /tmp/RatingsBreakdown.maria_dev.20190830.233300.332634
STDERR: mkdir: Permission denied: user=root, access=WRITE, inode="/user/maria_dev"                     :maria_dev:hdfs:drwxr-xr-x
Traceback (most recent call last):
File "RatingsBreakdown.py", line 19, in <module>
RatingsBreakdown.run()
File "/usr/lib/python2.7/site-packages/mrjob/job.py", line 446, in run
mr_job.execute()
File "/usr/lib/python2.7/site-packages/mrjob/job.py", line 473, in execute
super(MRJob, self).execute()
File "/usr/lib/python2.7/site-packages/mrjob/launch.py", line 202, in execute
self.run_job()
File "/usr/lib/python2.7/site-packages/mrjob/launch.py", line 247, in run_job
return self._handle(name, path, path)
File "/usr/lib/python2.7/site-packages/mrjob/fs/composite.py", line 118, in _han                     dle
return getattr(fs, name)(*args,**kwargs)
File "/usr/lib/python2.7/site-packages/mrjob/fs/hadoop.py", line 298, in mkdir
raise IOError("Could not mkdir %s" % path)
IOError: Could not mkdir hdfs:///user/maria_dev/tmp/mrjob/RatingsBreakdown.maria_d                     ev.20190830.233300.332634/files/wd`

你能描述一下这里有什么问题吗

kxxlusnw

kxxlusnw1#

请看这两个参考资料:
hdfs上的权限被拒绝
权限被拒绝:user=root,access=write,inode=“/user”:hdfs:supergroup:博士

cedebl8k

cedebl8k2#

我发现hortonworks需要很多时间来启动当我启动正确时,它工作得很好,启动大约需要1个小时

相关问题