在s3中使用hadoop命令时需要什么策略

dy2hfwbg 于 2021-05-31 发布在 Hadoop

关注(0)|答案(1)|浏览(409)

我想把pre上的数据复制到s3上。
我试着用下面的命令。

hadoop fs -Dfs.s3a.access.key=*******-Dfs.s3a.secret.key=*******-cp -f hdfs://on-pre/cluster/mydata/dt=20200601/ s3a://some-bucket/somewhere/

运行此命令时，出现以下错误（路径都是假的）：

cp: s3a://some-bucket/somewhere/dt=20200601/000000_0.gz: getFileStatus on s3a://some-bucket/somewhere/dt=20200601/000000_0.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxx

我制定了以下政策。

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::some-bucket/somewhere/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": "s3:List*",
            "Resource": "arn:aws:s3:::some-bucket",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "somewhere/*"
                    ]
                }
            }
        }
    ]
}

我应该使用什么s3策略 hadoop fs cp ?

hadoop amazon-s3 amazon-iam

来源：https://stackoverflow.com/questions/62130777/what-policies-are-required-when-using-hadoop-command-with-s3

1条答案

按热度按时间

gkl3eglg1#

子目录本身需要一个listbucket
必要的权限可能会更改，如注解中所述。
不过，这里的代码，解决了你的参考

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket/somewhere/*"
            ]
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "somewhere",
                        "somewhere/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::9999999999:user/john"
            },
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::some-bucket"
            ]
        }
    ]
}

注意，这个iam声明在2020年6月5日使用hadoop3.2.1或更早版本。未来的hadoop版本可能会更改规则，因为它们或aws会分别更改连接器或s3的功能。

赞(0）回复(0）举报 2021-05-31

我来回答

在s3中使用hadoop命令时需要什么策略

1条答案

相关问题

热门标签

最新问答