我正在尝试使用aws glue scala脚本处理awss3文件夹中的几个文件。在那之前我需要检查文件的数量和名字。为此,我编写了一个java类,它包含检查文件数量和名称的所有操作,并打包到一个jar文件中。我在glue配置中将这个jar文件作为一个依赖jars路径。当我在本地运行这个java类时,它运行得很好。但用于涂胶时,其刮刀误差如下。
ERROR ApplicationMaster: User class threw exception: java.lang.RuntimeException: Not initialized
java.lang.RuntimeException: Not initialized
at com.amazonaws.services.glue.util.JobBookmark$.preCheck(JobBookmarkUtils.scala:22)
at com.amazonaws.services.glue.util.JobBookmark$.commit(JobBookmarkUtils.scala:84)
at com.amazonaws.services.glue.util.Job$.commit(Job.scala:94)
at GlueApp$.runJob(script_2020-10-04-04-10-24.scala:117)
at GlueApp$.main(script_2020-10-04-04-10-24.scala:62)
我的java jar文件pom.xml依赖项
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.875</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.875</version>
</dependency>
</dependencies>
java类方法
public ArrayList<String> getFiles() {
BasicAWSCredentials credit = new BasicAWSCredentials("key",
"secreat");
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_EAST_2)
.withCredentials(new AWSStaticCredentialsProvider(credit)).build();
ArrayList<String> records = new ArrayList<>();
ListObjectsV2Request req = new ListObjectsV2Request().withBucketName("my-inputpath")
.withPrefix("monthly/data/");
ListObjectsV2Result listing = s3Client.listObjectsV2(req);
for (S3ObjectSummary summary : listing.getObjectSummaries()) {
records.add(summary.getKey().replace("monthly/", "/"));
}
return records;
}
我的胶水作业版本0.9
我不明白是什么问题?在日志中发现异常后,作业中止
INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException: Not initialized)
提前谢谢。。
暂无答案!
目前还没有任何答案,快来回答吧!