我已经花了两天的时间来缩短这个错误,甚至我尝试了一些stackoverflow帖子中建议的解决方法“-djava.util.arrays.uselegacymergesort=true”,但它也不起作用。
以下是我的命令及其返回错误的详细信息:
命令:
hadoop jar CloudBrush.jar -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -reads /Ec10k -asm Ec10k_Brush -k 21 -readlen 36
错误:
Error: java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeHi(TimSort.java:895)
at java.util.TimSort.mergeAt(TimSort.java:512)
at java.util.TimSort.mergeCollapse(TimSort.java:437)
at java.util.TimSort.sort(TimSort.java:241)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
at Brush.VerifyOverlap.run(VerifyOverlap.java:381)
at Brush.BrushAssembler.buildOverlap(BrushAssembler.java:326)
at Brush.BrushAssembler.run(BrushAssembler.java:838)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at Brush.BrushAssembler.main(BrushAssembler.java:913)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
这就是 Comparator
:
class OvelapSizeComparator implements Comparator {
public int compare(Object element1, Object element2) {
OverlapInfo obj1 = (OverlapInfo) element1;
OverlapInfo obj2 = (OverlapInfo) element2;
if ((int)(obj1.overlap_size - obj2.overlap_size) >= 0) {
return -1;
} else {
return 1;
}
}
}
3条答案
按热度按时间dzhpxtsq1#
我把错误修好了。我以为是hadoop的问题,我错了。这是java版本的问题。我们还升级了java版本和hadoop。Java7和Java8的arrays.sort()方法在要排序的元素数超过32个时使用timsort。tim-sort在比较中强制了严格的传递性。
如果((比较(x,y)>0)&((比较(y,z)>0)),则比较(x,z)应大于0。否则将抛出“java.lang.illegalargumentexception:comparison方法违反其常规约定”错误。
您应该将compare方法更改为与transitive属性一致,或者通过将“map task java opts base”或“reduce task java opts base”的“java.util.arrays.uselegacymergesort”设置为true来使用旧版本的sort,然后它应该应用于为map或reduce启动的所有jvm。
对于2.6.0-cdh5.4.2 haddop,可以通过添加
-d mapreduce.map.java.opts=“-djava.util.arrays.uselegacymergesort=true”
-d mapreduce.reduce.java.opts=“-djava.util.arrays.uselegacymergesort=true”
或通过代码
job.getconfiguration().set(“mapreduce.map.java.opts”,“-djava.util.arrays.uselegacymergesort=true”);
job.getconfiguration().set(“mapreduce.reduce.java.opts”,“-djava.util.arrays.uselegacymergesort=true”);
qvtsj1bj2#
通过在JVM7或更高版本上运行test.java代码中的main方法,可以重现此错误。总结此代码的作用。此代码尝试使用不包含传递属性的compare方法对40个人对象进行排序
//个人.java
公共阶层人士{
}
//测试.java
导入java.util.arrays;
公开课考试{
}
epggiuax3#
真正的问题是你的
OvelapSizeComparator
[sic]有缺陷。如果两个物体overlap_size
如果值相等,则返回-1
它应该什么时候回来0
. 如果它们不相等,则返回一个符号不正确的值。要解决此问题,请替换以下内容:
…用这个: