使用ranger时，sqoop无法导入数据

toiithl6  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(659)

我使用的是hdp-2.6，ranger允许管理员访问
我在ranger中添加了一个策略，以授予对/data的完全权限（读、写、执行），并启用了“递归”
我想使用sqoop将数据从mysql导入hive。但每次我都有以下例外：

30998 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Loading data to table ods.test_table
30998 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Loading data to table ods.test_table
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=EXECUTE, inode="/data/hive/warehouse/ods/test_table/part-m-00000.gz":admin:hadoop:drwx------
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=EXECUTE, inode="/data/hive/warehouse/ods/test_table/part-m-00000.gz":admin:hadoop:drwx------
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:292)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:292)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:238)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1903)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1903)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:63)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:63)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1850)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1850)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:821)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:821)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:465)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:465)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
31222 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at java.security.AccessController.doPrivileged(Native Method)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at java.security.AccessController.doPrivileged(Native Method)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at javax.security.auth.Subject.doAs(Subject.java:422)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at javax.security.auth.Subject.doAs(Subject.java:422)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
31223 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)

起初我以为ranger权限不起作用，但在range audit日志中，它显示了以下内容：

{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.446","access":"WRITE","resource":"/data/hive/warehouse/ods/test_table/part-m-00002.gz","resType":"path","action":"write","result":1,"policy":10,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269954","seq_num":433117,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.446","access":"WRITE","resource":"/data/hive/warehouse/ods/test_table/part-m-00002.gz","resType":"path","action":"write","result":1,"policy":10,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269955","seq_num":433119,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.447","access":"READ","resource":"/data/hive/warehouse/ods/test_table","resType":"path","action":"read","result":1,"policy":10,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269956","seq_num":433121,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.447","access":"EXECUTE","resource":"/data/hive/warehouse/ods/test_table/part-m-00000.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269957","seq_num":433123,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.448","access":"EXECUTE","resource":"/data/hive/warehouse/ods/test_table/part-m-00001.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269958","seq_num":433125,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.448","access":"EXECUTE","resource":"/data/hive/warehouse/ods/test_table/part-m-00002.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269959","seq_num":433127,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.449","access":"WRITE","resource":"/data/hive/warehouse/ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":10,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269960","seq_num":433129,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.449","access":"WRITE","resource":"/data/hive/warehouse/ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":10,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269961","seq_num":433131,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}
{"repoType":1,"repo":"xstudy_hadoop","reqUser":"yarn","evtTime":"2018-03-05 17:33:18.451","access":"EXECUTE","resource":"/data/hive/warehouse/ods/test_table/part-m-00003.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/data/hive/warehouse/ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-269962","seq_num":433133,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"xstudy"}

从审计日志来看，纱只是没有执行权限，为什么？我百分之百肯定得到了许可。
我有两个问题：
为什么yarn无法获得文件夹/数据的“执行”权限？
sqoop已经导入了数据，为什么yarn要求“execute”权限？
=======添加更多信息=====
今天，我尝试在集群中的linux机器上运行sqoop命令，它是完全相同的命令，具有相同的参数，它成功地执行了，恩。。。。。

hadoopsqoopapache-ranger

1条答案

按热度按时间
yc0p9oo0

yc0p9oo01#

我认为问题是目录的所有者 /data/hive/warehouse/ods/test_table/ 是用户 admin 是谁 hadoop 组成员。和用户 yarn 没有在那里执行作业的权限，因为 drwx------ .
尝试更改对目录的权限。运行命令：

sudo -u hdfs hdfs dfs -chmod -R 775 /data

试着重新开始你的工作。

赞(0）分享  回复(0）举报  2021-06-01
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com