我试图从eclipse运行elasticmapreduce,但是做不到。
我的代码如下:
public class RunEMR {
/**
* @param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
AWSCredentials credentials = new BasicAWSCredentials("xxxx","xxxx");
AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(credentials);
StepFactory stepFactory = new StepFactory();
StepConfig enableDebugging = new StepConfig()
.withName("Enable Debugging")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newEnableDebuggingStep());
StepConfig installHive = new StepConfig()
.withName("Install Hive")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newInstallHiveStep());
StepConfig hiveScript = new StepConfig().withName("Hive Script")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newRunHiveScriptStep("s3://mywordcountbuckett/binary/WordCount.jar"));
RunJobFlowRequest request = new RunJobFlowRequest()
.withName("Hive Interactive")
.withSteps(enableDebugging, installHive)
.withLogUri("s3://mywordcountbuckett/")
.withInstances(new JobFlowInstancesConfig()
.withEc2KeyName("xxxx")
.withHadoopVersion("0.20")
.withInstanceCount(3)
.withKeepJobFlowAliveWhenNoSteps(true)
.withMasterInstanceType("m1.small")
.withSlaveInstanceType("m1.small"));
RunJobFlowResult result = emr.runJobFlow(request);
}
}
我得到的错误是:
Exception in thread "main" com.amazonaws.AmazonServiceException: InstanceProfile is required for creating cluster. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: 7a96ee32-9744-11e5-947d-65ca8f7db0a5
我试了几个小时,但没能修好。有人知道怎么做吗?
3条答案
按热度按时间li9yvcax1#
对于ec2角色(这里是jobflowrole),将在内部创建具有相同名称的示例概要文件。因此,它可以互换地使用这些名称。如果您使用boto3从头开始创建emr集群,那么还应该创建emr服务角色、一个ec2jobflow角色、一个链接到ec2jobflow角色的示例概要文件。aws文件
dw1jzc5e2#
您尝试使用的版本已弃用,需要iam角色。遵循文档中给出的示例http://docs.aws.amazon.com/elasticmapreduce/latest/managementguide/calling-emr-with-java-sdk.html.
mmvthczy3#
我也有同样的例外
InstanceProfile is required for creating cluster
.必须设置服务角色和作业流角色,如下所示
之后我就没事了。
用于emr iam角色的aws文件说
aws身份和访问管理(iam)角色为iam用户或aws服务提供了一种对资源具有特定权限和访问权限的方法。例如,这可能允许用户访问资源或其他服务以代表您行事。必须为集群指定两个iam角色:amazon emr服务的角色(service role)和amazon emr管理的ec2示例的角色(instance profile)。
所以说这个词
InstanceProfile
在异常消息中可能意味着a role for the EC2 instances (instance profile)
但我在指定JobFlowRole
. 有点奇怪。