error:java.lang.illegalargumentexception:comparison方法违反了它的一般约定,甚至使用了变通方法

cigdeys3  于 2021-05-29  发布在  Hadoop
关注(0)|答案(3)|浏览(789)

我已经花了两天的时间来缩短这个错误,甚至我尝试了一些stackoverflow帖子中建议的解决方法“-djava.util.arrays.uselegacymergesort=true”,但它也不起作用。
以下是我的命令及其返回错误的详细信息:
命令:

hadoop jar CloudBrush.jar -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -reads /Ec10k -asm Ec10k_Brush -k 21 -readlen 36

错误:

Error: java.lang.IllegalArgumentException: Comparison method violates its general contract!
    at java.util.TimSort.mergeHi(TimSort.java:895)
    at java.util.TimSort.mergeAt(TimSort.java:512)
    at java.util.TimSort.mergeCollapse(TimSort.java:437)
    at java.util.TimSort.sort(TimSort.java:241)
    at java.util.Arrays.sort(Arrays.java:1512)
    at java.util.ArrayList.sort(ArrayList.java:1454)
    at java.util.Collections.sort(Collections.java:175)
    at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
    at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
    at Brush.VerifyOverlap.run(VerifyOverlap.java:381)
    at Brush.BrushAssembler.buildOverlap(BrushAssembler.java:326)
    at Brush.BrushAssembler.run(BrushAssembler.java:838)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at Brush.BrushAssembler.main(BrushAssembler.java:913)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

这就是 Comparator :

class OvelapSizeComparator implements Comparator {
        public int compare(Object element1, Object element2) {
            OverlapInfo obj1 = (OverlapInfo) element1;
            OverlapInfo obj2 = (OverlapInfo) element2;
            if ((int)(obj1.overlap_size - obj2.overlap_size) >= 0) {
                return -1;
            } else {
                return 1;
            }
        }
    }
dzhpxtsq

dzhpxtsq1#

我把错误修好了。我以为是hadoop的问题,我错了。这是java版本的问题。我们还升级了java版本和hadoop。Java7和Java8的arrays.sort()方法在要排序的元素数超过32个时使用timsort。tim-sort在比较中强制了严格的传递性。
如果((比较(x,y)>0)&((比较(y,z)>0)),则比较(x,z)应大于0。否则将抛出“java.lang.illegalargumentexception:comparison方法违反其常规约定”错误。
您应该将compare方法更改为与transitive属性一致,或者通过将“map task java opts base”或“reduce task java opts base”的“java.util.arrays.uselegacymergesort”设置为true来使用旧版本的sort,然后它应该应用于为map或reduce启动的所有jvm。
对于2.6.0-cdh5.4.2 haddop,可以通过添加
-d mapreduce.map.java.opts=“-djava.util.arrays.uselegacymergesort=true”
-d mapreduce.reduce.java.opts=“-djava.util.arrays.uselegacymergesort=true”
或通过代码
job.getconfiguration().set(“mapreduce.map.java.opts”,“-djava.util.arrays.uselegacymergesort=true”);
job.getconfiguration().set(“mapreduce.reduce.java.opts”,“-djava.util.arrays.uselegacymergesort=true”);

qvtsj1bj

qvtsj1bj2#

通过在JVM7或更高版本上运行test.java代码中的main方法,可以重现此错误。总结此代码的作用。此代码尝试使用不包含传递属性的compare方法对40个人对象进行排序
//个人.java
公共阶层人士{

public String name;
public int age ;
public int salary;

@Override
public int compareTo(Person o) {
    if(o instanceof Person){
        int ret=0;
        if(age == 25 && ((Person)o).age ==27) ret = 1;
        else if(age == 27 && ((Person)o).age ==29) ret = 1;
        else if(age == 25 && ((Person)o).age ==29) ret = -1;
        else{
                if( age < ((Person)o).age) ret = -1;
                if(age > ((Person)o).age) ret = 1;
                if(salary < ((Person)o).salary) ret = -1;
                if(salary > ((Person)o).salary) ret = 1;           
        }
        return ret; 

    }
    return 0;
}

@Override
public String toString(){
 return "name="+name+":age="+age+";";     
}

}
//测试.java
导入java.util.arrays;
公开课考试{

public static void main(String args[]) {
    Test t = new Test();
    t.sortPersons(args);        
}

public void sortPersons(String args[]) {        

    Person p1 = new Person();
    p1.age = 25;
    p1.name = "ABC";

    Person p2 = new Person();
    p2.age = 29;
    p2.name = "ABZ";

    Person p3 = new Person();
    p3.age = 27;
    p3.name = "AZ";

    Person p4 = new Person();
    p4.age = 27;
    p4.name = "AZ";

    Person p5 = new Person();
    //p5.age = 22;
    //p5.name="ZZ";

    Person[] p = new Person[40];

    p[0] = p2;
    p[1] = p3;
    p[2] = p4;
    p[3] = p1;
    p[4] = p5;
    for (int i = 1; i < 8; i++) {
        p[i * 5] = p[0];
        p[i * 5 + 1] = p[1];
        p[i * 5 + 2] = p[2];
        p[i * 5 + 3] = p[3];
        p[i * 5 + 4] = p[4];
    }

    System.out.println("\nSortingInput\n");           
    Arrays.sort(p);       
    System.out.println("\nSorting complete\n");
}

}

epggiuax

epggiuax3#

真正的问题是你的 OvelapSizeComparator [sic]有缺陷。如果两个物体 overlap_size 如果值相等,则返回 -1 它应该什么时候回来 0 . 如果它们不相等,则返回一个符号不正确的值。
要解决此问题,请替换以下内容:

if ((int)(obj1.overlap_size - obj2.overlap_size) >= 0) {
   return -1;
} else {
   return 1;
}

…用这个:

return obj1.overlap_size - obj2.overlap_size;

相关问题