更新:这是我愚蠢的复制/粘贴键错误。其他一些有趣的方面,我将尝试解释一下。。。
原始问题如下:
我试图编写一个spark程序,将json写入s3,但遇到了以下错误:我们计算的请求签名与您提供的签名不匹配。检查您的密钥和签名方法。
因此,我决定尝试以最简单的方式工作,使用awsjavasdk和这里描述的示例程序
在~/.aws/credentials中正确提供了我的凭证之后,上面引用的示例代码就可以正常工作,没有任何修改。
然后,我决定将~/.aws/凭证移到一边并重新运行,这样就搞错了。然后我尝试通过显式提供凭据来修复。您可以在“完全修改的程序”一节中看到稍微修改的示例程序的完整代码。
但我尝试的本质就在下面。请注意,我是通过
.withCredentials(provider)
在客户端生成器上。我还试着用下面的行运行,这两行都被注解掉并启用了:
//.withClientConfiguration(clientConfiguration)
.
public static void main(String[] args) throws IOException {
String bucketName = "cbedford-sqlshackdemocli-test-only";
String stringObjKeyName = "durango.saturday1";
String fileObjKeyName = "clean.money.txt";
String fileName = "/tmp/x";
ClientConfiguration clientConfiguration = new ClientConfiguration();
// Solution is update the Signer Version.
clientConfiguration.setSignerOverride("S3SignerType");
AWSCredentialsProvider provider = new AWSCredentialsProvider() {
@Override
public AWSCredentials getCredentials() {
return new AWSCredentials () {
@Override
public String getAWSAccessKeyId() {
// String below copied form ~/.aws/credentials
// [default]
// aws_access_key_id = somethingsomething
return "somethingsomething";
}
@Override
public String getAWSSecretKey() {
// String below copied form ~/.aws/credentials
// [default]
// aws_secret_access_key = somethingsomethingelse
return "somethingsomethingelse";
}
};
}
@Override
public void refresh() {
}
};
try {
//This code expects that you have AWS credentials set up per:
// https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.US_EAST_1)
.withCredentials(provider)
//.withClientConfiguration(clientConfiguration)
.build();
无论我尝试了什么,通过inline方法提供与.aws目录中相同的凭据的所有尝试都失败了。如果有人能发现我愚蠢的错误(我知道我一定是在犯错误!)我将不胜感激。
完全修改程序
import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.SdkClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import java.io.File;
import java.io.IOException;
public class UploadObject {
public static void main(String[] args) throws IOException {
String bucketName = "cbedford-sqlshackdemocli-test-only";
String stringObjKeyName = "durango.saturday1";
String fileObjKeyName = "clean.money.txt";
String fileName = "/tmp/x";
ClientConfiguration clientConfiguration = new ClientConfiguration();
// Solution is update the Signer Version.
clientConfiguration.setSignerOverride("S3SignerType");
AWSCredentialsProvider provider = new AWSCredentialsProvider() {
@Override
public AWSCredentials getCredentials() {
return new AWSCredentials () {
@Override
public String getAWSAccessKeyId() {
return "somethingsomething";
}
@Override
public String getAWSSecretKey() {
return "somethingsomethingelse";
}
};
}
@Override
public void refresh() {
}
};
try {
//This code expects that you have AWS credentials set up per:
// https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.US_EAST_1)
.withCredentials(provider)
//.withClientConfiguration(clientConfiguration)
.build();
// Upload a text string as a new object.
s3Client.putObject(bucketName, stringObjKeyName, "Uploaded String Object");
// Upload a file as a new object with ContentType and title specified.
PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName));
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("plain/text");
metadata.addUserMetadata("title", "someTitle");
request.setMetadata(metadata);
s3Client.putObject(request);
} catch (AmazonServiceException e) {
// The call was transmitted successfully, but Amazon S3 couldn't process
// it, so it returned an error response.
e.printStackTrace();
} catch (SdkClientException e) {
// Amazon S3 couldn't be contacted for a response, or the client
// couldn't parse the response from Amazon S3.
e.printStackTrace();
}
}
}
1条答案
按热度按时间hts6caw31#
可能与v4签名有关,您需要将请求路由到存储桶所在的特定aws区域。最近的地区(伦敦、法兰克福等)只有v4签名。
为什么不为lib dir*中的hadoop版本添加hadoop aws jar以及构建它时使用的确切aws jar,并尝试使用s3a连接器,看看它是否工作,如果不工作,会提供哪些错误。然后看看hadoop文档上的疑难解答文档,看看可能的原因是什么。