pipemapred waitoutputthreads子进程失败,代码为255

sbdsn5lh  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(361)

我正在azurehdinsight(hadoop)上运行一个简单的map/reduce作业。Map程序失败,出现以下错误(来自jobtracker日志)。。。

2014-03-27 16:42:50,330 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201403262112_0004_m_000000_0: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
at org.apache.hadoop.mapred.Child.main(Child.java:260)

我正在使用hadoop.connect方法连接到群集。。。

// Connect to the HDInsight cluster
var hadoop = Hadoop.Connect(hadoopClusterUri,
                            hadoopUserName,
                            localHadoopUser,
                            hadoopPassword,
                            asvAccount,
                            asvKey,
                            asvContainer,
                            createContinerIfNotExist);

由于作业已启动并记录,我假设连接成功。作为一个简单的测试,我发射了少量的数据。。。

context.EmitKeyValue("This is a key", "This is a value");

完全被难住了,比尔
更新此mapreduce将生成两个作业。第一个成功完成,第二个失败,出现以下异常。。。

Unhandled Exception: Microsoft.Hadoop.MapReduce.StreamingException: The user type could not be loaded. DLL=MapReduce.exe, Type=PrescienceRx.MapReduce.BeneficiaryMapper ---> System.BadImageFormatException: Could not load file or assembly 'file:///c:\apps\temp\hdfs\mapred\local\taskTracker\admin\jobcache\job_201403262112_0006\attempt_201403262112_0006_m_000001_0\work\MapReduce.exe' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at Microsoft.Hadoop.MapReduce.MapperMain.Main()
--- End of inner exception stack trace ---

位于microsoft.hdinsight.mapdriver.program.main(string[]args)的microsoft.hadoop.mapreduce.mappermain.main()中

zbdgwd5y

zbdgwd5y1#

看起来像 System.BadImageFormatException: Could not load file or assembly 可能是因为加载程序的.net版本与程序集不匹配。尝试将其设置回4.0,然后在nuget package manager窗口中执行以下操作 update-package -reinstall Microsoft.Hadoop.MapReduce 以确保您拥有所有正确的版本。

kcwpcxri

kcwpcxri2#

我在解决方案的配置管理器中将目标cpu设置为x64,这样就解决了问题。

相关问题