我在hadoop集群上运行mapreduce作业时遇到了一个奇怪的异常。有趣的是,我可以访问hdfs,但无法运行作业。
UriBuilder uriBuilder = new UriBuilder("192.168.16.132");
uriBuilder.Port = 8021;//8082;
var hadoop = Hadoop.Connect(uriBuilder.Uri, "username", "password");
hadoop.StorageSystem.MakeDirectory("user/username/test"); //This works
//establish job configuration
HadoopJobConfiguration myConfig = new HadoopJobConfiguration();
myConfig.InputPath = "/user/username/input";
myConfig.OutputFolder = "/user/username/output";
try
{
//connect to cluster
MapReduceResult jobResult = hadoop.MapReduceJob.Execute<MySimpleMapper, MySimpleReducer>(myConfig); //This does not work and produces an error: The remote name could not be resolved
//write job result to console
int exitCode = jobResult.Info.ExitCode;
string exitStatus = "Failure";
if (exitCode == 0) exitStatus = "Success";
exitStatus = exitCode + " (" + exitStatus + ")";
Console.WriteLine();
Console.Write("Exit Code = " + exitStatus);
Console.Read();
}
catch (Exception exc)
{
//Error sending request.
}
我使用hortonworks沙盒进行测试,如果它有任何区别的话。确切的错误是:“无法解析远程名称:'sandbox'”。
有人能解释一下为什么会发生这种情况,以及我能做些什么来解决它吗?
编辑:我已经通过将hadoop集群的ip添加到hosts文件中修复了这个问题,但是现在我得到了以下异常:“response status code does not indicates success:500(server error)。”
1条答案
按热度按时间yvfmudvl1#
事实证明,服务器不是在windowsazure上,而是在apache实现上。协议不兼容。hdfs协议对于两种实现都是相同的,所以这可以工作。但是,不支持map/reduce框架。