com.clearspring.analytics.stream.cardinality.HyperLogLog.merge()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(2.6k)|赞(0)|评价(0)|浏览(158)

本文整理了Java中com.clearspring.analytics.stream.cardinality.HyperLogLog.merge()方法的一些代码示例,展示了HyperLogLog.merge()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。HyperLogLog.merge()方法的具体详情如下:
包路径:com.clearspring.analytics.stream.cardinality.HyperLogLog
类名称:HyperLogLog
方法名:merge

HyperLogLog.merge介绍

暂无

代码示例

代码示例来源:origin: addthis/stream-lib

@Test
public void testMerge() throws CardinalityMergeException {
  int numToMerge = 5;
  int bits = 16;
  int cardinality = 1000000;
  HyperLogLog[] hyperLogLogs = new HyperLogLog[numToMerge];
  HyperLogLog baseline = new HyperLogLog(bits);
  for (int i = 0; i < numToMerge; i++) {
    hyperLogLogs[i] = new HyperLogLog(bits);
    for (int j = 0; j < cardinality; j++) {
      double val = Math.random();
      hyperLogLogs[i].offer(val);
      baseline.offer(val);
    }
  }
  long expectedCardinality = numToMerge * cardinality;
  HyperLogLog hll = hyperLogLogs[0];
  hyperLogLogs = Arrays.asList(hyperLogLogs).subList(1, hyperLogLogs.length).toArray(new HyperLogLog[0]);
  long mergedEstimate = hll.merge(hyperLogLogs).cardinality();
  long baselineEstimate = baseline.cardinality();
  double se = expectedCardinality * (1.04 / Math.sqrt(Math.pow(2, bits)));
  System.out.println("Baseline estimate: " + baselineEstimate);
  System.out.println("Expect estimate: " + mergedEstimate + " is between " + (expectedCardinality - (3 * se)) + " and " + (expectedCardinality + (3 * se)));
  assertTrue(mergedEstimate >= expectedCardinality - (3 * se));
  assertTrue(mergedEstimate <= expectedCardinality + (3 * se));
  assertEquals(mergedEstimate, baselineEstimate);
}

代码示例来源:origin: addthis/stream-lib

/**
 * should not fail with HyperLogLogMergeException: "Cannot merge estimators of different sizes"
 */
@Test
public void testMergeWithRegisterSet() throws CardinalityMergeException {
  HyperLogLog first = new HyperLogLog(16, new RegisterSet(1 << 20));
  HyperLogLog second = new HyperLogLog(16, new RegisterSet(1 << 20));
  first.offer(0);
  second.offer(1);
  first.merge(second);
}

代码示例来源:origin: Big-Data-Manning/big-data-code

public void operate(FlowProcess process, BufferCall call) {
    Iterator<TupleEntry> it = call.getArgumentsIterator();
    HyperLogLog curr = null;
    try {
      while(it.hasNext()) {
        TupleEntry tuple = it.next();
        byte[] serialized = (byte[]) tuple.getObject(0);
        HyperLogLog hll = HyperLogLog.Builder.build(
                   serialized);
        if(curr==null) {
          curr = hll;
        } else {
          curr = (HyperLogLog) curr.merge(hll);
        }
      }
      call.getOutputCollector().add(
        new Tuple(curr.getBytes()));
    } catch (IOException e) {
      throw new RuntimeException(e);
    } catch(CardinalityMergeException e) {
      throw new RuntimeException(e);
    }
  }
}

代码示例来源:origin: LiveRamp/cascading_ext

HyperLogLog merged = (HyperLogLog)new HyperLogLog(hllError).merge(countParts.toArray(new ICardinality[countParts.size()]));
long cardinality = merged.cardinality();

相关文章