类型的未处理异常 System.AggregateException
发生在 mscorlib.dll
内部异常: {"Response status code does not indicate success: 403 (Forbidden)."}
有机会得到: {"Response status code does not indicate success: 401 (Credentials required)."}
所有登录都正确。hadoop.connect()正确连接。
堆栈跟踪:
在system.threading.tasks.task.ThrowifeException(布尔includetaskcanceledexceptions)在system.threading.tasks.task.wait(int32毫秒计时,cancellationtoken cancellationtoken)位于microsoft.hadoop.webclient.webhcatclient.webhcatmapreducestreamingexecutor.execute的system.threading.tasks.task.wait(),位于microsoft.hadoop.mapreduce.execution.hadoop.streamingjobexecutorbase.executecore(类型Map器、类型还原器、类型组合器、hadoopjobconfiguration配置)
在microsoft.hadoop.mapreduce.execution.hadoop.streamingjobexecutorbase.execute(类型mappertype,类型reducertype,类型combinertype,hadoopjobconfiguration config)的microsoft.hadoop.mapreduce.execution.hadoop.streamingjobexecutorbase.execute[tmapper,在c:\asc project info\talentexchange\demo project\pwc.demo.hdpclient\pwc.demo.hdpclient\program.cs中的pwc.demo.hdpclient.program.main(string[]args)处的treducer](hadoopjobconfiguration config):system.appdomain.的第49行。\u system.appdomain.executeassembly(string assemblyfile,microsoft.visualstudio.hostingprocess.hostproc.runusersassembly()上的证据assemblysecurity,string[]args)
在system.threading.threadhelper.threadstart\u上下文(对象状态)
在system.threading.executioncontext.runinternal(executioncontext executioncontext,contextcallback callback,object state,boolean preservesyncctx)在system.threading.executioncontext.run(executioncontext executioncontext,contextcallback callback,object state,在system.threading.threadhelper.threadstart()处运行(executioncontext executioncontext,contextcallback回调,对象状态)
class Program
{
static void Main(string[] args)
{
HadoopJobConfiguration myConfig = new HadoopJobConfiguration();
myConfig.InputPath = "asv://hdsto-something-.net/data/in/reviews.txt";
myConfig.OutputFolder = "asv://hd-something-.blob.core.windows.net/data/out/";
Environment.SetEnvironmentVariable("HADOOP_HOME", @"C:\apps\dist\hadoop-2.7.1.2.3.3.1-25\bin");
Environment.SetEnvironmentVariable("JAVA_HOME", @"C:\Program Files\Java\jre1.8.0_101\bin\javaw.exe");
Uri azureCluster = new Uri("https://somename--Demo.azurehdinsight.net");
string clusterUserName = "***";
string clusterPassword = "****";
// This is the name of the account under which Hadoop will execute jobs.
// Normally this is just "Hadoop".
string hadoopUserName = "Hadoop";
// Azure Storage Information.
string azureStorageAccount = "somestore.blob.core.windows.net";
string azureStorageKey = "***==";
string azureStorageContainer = "**";
bool createContinerIfNotExist = false;
IHadoop myCluster = Hadoop.Connect(azureCluster,
clusterUserName,
hadoopUserName,
clusterPassword,
azureStorageAccount,
azureStorageKey,
azureStorageContainer,
createContinerIfNotExist);
//execute mapreduce job
MapReduceResult jobResult =
myCluster.MapReduceJob.Execute<MySimpleMapper, MySimpleReducer>(myConfig);
int exitCode = jobResult.Info.ExitCode;
string exitStatus = "Failure";
if (exitCode == 0) exitStatus = "Success";
exitStatus = exitCode + " (" + exitStatus + ")";
Console.WriteLine();
Console.Write("Exit Code = " + exitStatus);
Console.Read();
}
}
public class MySimpleMapper : MapperBase
{
public override void Map(string inputLine, MapperContext context)
{
int value = int.Parse(inputLine);
string key = (value % 2 == 0) ? "even" : "odd";
context.EmitKeyValue(key, value.ToString());
}
}
public class MySimpleReducer : ReducerCombinerBase
{
public override void Reduce(string key, IEnumerable<string> values, ReducerCombinerContext context)
{
//initialize counters
int myCount = 0;
int mySum = 0;
//count and sum incoming values
foreach (string value in values)
{
mySum += int.Parse(value);
myCount++;
}
context.EmitKeyValue(key, myCount + "t" + mySum);
}
}
错误快照:
1条答案
按热度按时间nr7wwzry1#
输入路径不应指向特定文件,而应指向目录。该目录应该只包含文件而不包含文件夹。