提交作业的hadoop hdinsight.net sdk API

jtw3ybtb  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(304)

我正在使用hdinsight.net hadoop API在asp.net应用程序中提交map reduce作业。
使用microsoft.hadoop.mapreduce;
var hadoop=hadoop.connect();
var result=hadoop.mapreducejob.executejob();
//我也试过这个,但同样的例外
//var result=hadoop.mapreducejob.executejob(配置);
executejob()调用失败,并在运行时抛出异常。这个世界上有没有人能成功地运行这个呼叫。是否可以通过添加更多的输入参数或对象来自定义map()函数(除了microsoft在mapperbase类中给出的以外)?mapper和reducer方法中的逻辑可以访问缓存/数据库吗?

z4iuyo4d

z4iuyo4d1#

使用hdinsight.net sdk提交mapreduce作业的示例发布在此处:
http://www.windowsazure.com/en-us/manage/services/hdinsight/submit-hadoop-jobs-programmatically/#mapreduce-软件开发包

// Define the MapReduce job
MapReduceJobCreateParameters mrJobDefinition = new MapReduceJobCreateParameters()
{
    JarFile = "wasb:///example/jars/hadoop-examples.jar",
    ClassName = "wordcount"
};

mrJobDefinition.Arguments.Add("wasb:///example/data/gutenberg/davinci.txt");
mrJobDefinition.Arguments.Add("wasb:///example/data/WordCountOutput");

// Get the certificate object from certificate store using the friendly name to identify it
X509Store store = new X509Store();
store.Open(OpenFlags.ReadOnly);
X509Certificate2 cert = store.Certificates.Cast<X509Certificate2>().First(item => item.FriendlyName == certfrientlyname);
JobSubmissionCertificateCredential creds = new JobSubmissionCertificateCredential(new Guid(subscriptionID), cert, clusterName);

// Create a hadoop client to connect to HDInsight
var jobClient = JobSubmissionClientFactory.Connect(creds);

// Run the MapReduce job
JobCreationResults mrJobResults = jobClient.CreateMapReduceJob(mrJobDefinition);

// Wait for the job to complete
WaitForJobCompletion(mrJobResults, jobClient);

相关问题