com.clearspring.analytics.stream.cardinality.HyperLogLog.offer()方法的使用及代码示例

x33g5p2x  于2022-01-20 转载在 其他  
字(9.5k)|赞(0)|评价(0)|浏览(158)

本文整理了Java中com.clearspring.analytics.stream.cardinality.HyperLogLog.offer()方法的一些代码示例，展示了HyperLogLog.offer()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。HyperLogLog.offer()方法的具体详情如下：
包路径：com.clearspring.analytics.stream.cardinality.HyperLogLog
类名称：HyperLogLog
方法名：offer

HyperLogLog.offer介绍

暂无

代码示例

代码示例来源：origin: apache/incubator-pinot

/**
 * Helper method to set value for a groupKey into the result holder.
 *
 * @param groupByResultHolder Result holder
 * @param groupKey Group-key for which to set the value
 * @param value Value for the group key
 */
private static void setValueForGroupKey(@Nonnull GroupByResultHolder groupByResultHolder, int groupKey,
  Object value) {
 HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKey);
 hyperLogLog.offer(value);
}

代码示例来源：origin: addthis/stream-lib

Double val = Math.random();
String valString = val.toString();
baseline.offer(valString);
guava128.offerHashed(hf128.hashString(valString, Charsets.UTF_8).asLong());
if (j > 0 && j % 1000000 == 0) {

代码示例来源：origin: apache/incubator-pinot

@Override
public HyperLogLog getInitialAggregatedValue(Object rawValue) {
 HyperLogLog initialValue;
 if (rawValue instanceof byte[]) {
  byte[] bytes = (byte[]) rawValue;
  initialValue = deserializeAggregatedValue(bytes);
  _maxByteSize = Math.max(_maxByteSize, bytes.length);
 } else {
  initialValue = new HyperLogLog(DistinctCountHLLAggregationFunction.DEFAULT_LOG2M);
  initialValue.offer(rawValue);
  _maxByteSize = Math.max(_maxByteSize, DEFAULT_LOG2M_BYTE_SIZE);
 }
 return initialValue;
}

代码示例来源：origin: apache/incubator-pinot

@Override
public HyperLogLog applyRawValue(HyperLogLog value, Object rawValue) {
 if (rawValue instanceof byte[]) {
  try {
   value.addAll(deserializeAggregatedValue((byte[]) rawValue));
  } catch (CardinalityMergeException e) {
   throw new RuntimeException(e);
  }
 } else {
  value.offer(rawValue);
 }
 return value;
}

代码示例来源：origin: apache/incubator-pinot

/**
 * Generate a hll from a single value, and convert it to string type.
 * It is used for default derived field value.
 * @param log2m
 * @param value
 * @return
 */
public static String singleValueHllAsString(int log2m, Object value) {
 HyperLogLog hll = new HyperLogLog(log2m);
 hll.offer(value);
 return convertHllToString(hll);
}

代码示例来源：origin: addthis/stream-lib

@Test
public void testComputeCount() {
  HyperLogLog hyperLogLog = new HyperLogLog(16);
  hyperLogLog.offer(0);
  hyperLogLog.offer(1);
  hyperLogLog.offer(2);
  hyperLogLog.offer(3);
  hyperLogLog.offer(16);
  hyperLogLog.offer(17);
  hyperLogLog.offer(18);
  hyperLogLog.offer(19);
  hyperLogLog.offer(19);
  assertEquals(8, hyperLogLog.cardinality());
}

代码示例来源：origin: apache/incubator-pinot

for (int i = 0; i < length; i++) {
 for (int value : intValues[i]) {
  hyperLogLog.offer(value);
for (int i = 0; i < length; i++) {
 for (long value : longValues[i]) {
  hyperLogLog.offer(value);
for (int i = 0; i < length; i++) {
 for (float value : floatValues[i]) {
  hyperLogLog.offer(value);
for (int i = 0; i < length; i++) {
 for (double value : doubleValues[i]) {
  hyperLogLog.offer(value);
for (int i = 0; i < length; i++) {
 for (String value : stringValues[i]) {
  hyperLogLog.offer(value);

代码示例来源：origin: apache/incubator-pinot

@Override
Object getRandomRawValue(Random random) {
 HyperLogLog hyperLogLog = new HyperLogLog(LOG2M);
 hyperLogLog.offer(random.nextInt(100));
 hyperLogLog.offer(random.nextInt(100));
 return ObjectSerDeUtils.HYPER_LOG_LOG_SER_DE.serialize(hyperLogLog);
}

代码示例来源：origin: apache/incubator-pinot

int[] intValues = blockValSets[0].getIntValuesSV();
for (int i = 0; i < length; i++) {
 hyperLogLog.offer(intValues[i]);
long[] longValues = blockValSets[0].getLongValuesSV();
for (int i = 0; i < length; i++) {
 hyperLogLog.offer(longValues[i]);
float[] floatValues = blockValSets[0].getFloatValuesSV();
for (int i = 0; i < length; i++) {
 hyperLogLog.offer(floatValues[i]);
double[] doubleValues = blockValSets[0].getDoubleValuesSV();
for (int i = 0; i < length; i++) {
 hyperLogLog.offer(doubleValues[i]);
String[] stringValues = blockValSets[0].getStringValuesSV();
for (int i = 0; i < length; i++) {
 hyperLogLog.offer(stringValues[i]);

代码示例来源：origin: addthis/stream-lib

@Test
public void testSerializationUsingBuilder() throws IOException {
  HyperLogLog hll = new HyperLogLog(8);
  hll.offer("a");
  hll.offer("b");
  hll.offer("c");
  hll.offer("d");
  hll.offer("e");
  HyperLogLog hll2 = HyperLogLog.Builder.build(hll.getBytes());
  assertEquals(hll.cardinality(), hll2.cardinality());
}

代码示例来源：origin: addthis/stream-lib

@Test
public void testSerialization() throws IOException, ClassNotFoundException {
  HyperLogLog hll = new HyperLogLog(8);
  hll.offer("a");
  hll.offer("b");
  hll.offer("c");
  hll.offer("d");
  hll.offer("e");
  HyperLogLog hll2 = (HyperLogLog) TestUtils.deserialize(TestUtils.serialize(hll));
  assertEquals(hll.cardinality(), hll2.cardinality());
}

代码示例来源：origin: apache/incubator-pinot

HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKeyArray[i]);
for (int value : intValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKeyArray[i]);
for (long value : longValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKeyArray[i]);
for (float value : floatValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKeyArray[i]);
for (double value : doubleValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKeyArray[i]);
for (String value : stringValues[i]) {
 hyperLogLog.offer(value);

代码示例来源：origin: apache/incubator-pinot

HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKey);
for (int value : intValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKey);
for (long value : longValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKey);
for (float value : floatValues[i]) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKey);
for (double value : doubleValues) {
 hyperLogLog.offer(value);
HyperLogLog hyperLogLog = getHyperLogLog(groupByResultHolder, groupKey);
for (String value : stringValues) {
 hyperLogLog.offer(value);

代码示例来源：origin: addthis/stream-lib

@Test
public void testHighCardinality() {
  long start = System.currentTimeMillis();
  HyperLogLog hyperLogLog = new HyperLogLog(10);
  int size = 10000000;
  for (int i = 0; i < size; i++) {
    hyperLogLog.offer(TestICardinality.streamElement(i));
  }
  System.out.println("time: " + (System.currentTimeMillis() - start));
  long estimate = hyperLogLog.cardinality();
  double err = Math.abs(estimate - size) / (double) size;
  System.out.println(err);
  assertTrue(err < .1);
}

代码示例来源：origin: addthis/stream-lib

@Test
public void testHighCardinality_withDefinedRSD() {
  long start = System.currentTimeMillis();
  HyperLogLog hyperLogLog = new HyperLogLog(0.01);
  int size = 100000000;
  for (int i = 0; i < size; i++) {
    hyperLogLog.offer(TestICardinality.streamElement(i));
  }
  System.out.println("time: " + (System.currentTimeMillis() - start));
  long estimate = hyperLogLog.cardinality();
  double err = Math.abs(estimate - size) / (double) size;
  System.out.println(err);
  assertTrue(err < .1);
}

代码示例来源：origin: apache/incubator-pinot

@Test
 public void testHllFieldSerializedSize()
   throws Exception {
  for (int i = 5; i < 10; i++) {
   HyperLogLog hll = new HyperLogLog(i);
   Assert.assertEquals(HllSizeUtils.getHllFieldSizeFromLog2m(i), hll.getBytes().length);
   LOGGER.info("Estimated: " + hll.cardinality());
   for (int j = 0; j < 100; j++) {
    hll.offer(rand.nextLong());
   }
   Assert.assertEquals(HllSizeUtils.getHllFieldSizeFromLog2m(i), hll.getBytes().length);
   LOGGER.info("Estimated: " + hll.cardinality());
   for (int j = 0; j < 9900; j++) {
    hll.offer(rand.nextLong());
   }
   Assert.assertEquals(HllSizeUtils.getHllFieldSizeFromLog2m(i), hll.getBytes().length);
   LOGGER.info("Estimated: " + hll.cardinality());
  }
 }
}

代码示例来源：origin: addthis/stream-lib

@Test
public void testMerge() throws CardinalityMergeException {
  int numToMerge = 5;
  int bits = 16;
  int cardinality = 1000000;
  HyperLogLog[] hyperLogLogs = new HyperLogLog[numToMerge];
  HyperLogLog baseline = new HyperLogLog(bits);
  for (int i = 0; i < numToMerge; i++) {
    hyperLogLogs[i] = new HyperLogLog(bits);
    for (int j = 0; j < cardinality; j++) {
      double val = Math.random();
      hyperLogLogs[i].offer(val);
      baseline.offer(val);
    }
  }
  long expectedCardinality = numToMerge * cardinality;
  HyperLogLog hll = hyperLogLogs[0];
  hyperLogLogs = Arrays.asList(hyperLogLogs).subList(1, hyperLogLogs.length).toArray(new HyperLogLog[0]);
  long mergedEstimate = hll.merge(hyperLogLogs).cardinality();
  long baselineEstimate = baseline.cardinality();
  double se = expectedCardinality * (1.04 / Math.sqrt(Math.pow(2, bits)));
  System.out.println("Baseline estimate: " + baselineEstimate);
  System.out.println("Expect estimate: " + mergedEstimate + " is between " + (expectedCardinality - (3 * se)) + " and " + (expectedCardinality + (3 * se)));
  assertTrue(mergedEstimate >= expectedCardinality - (3 * se));
  assertTrue(mergedEstimate <= expectedCardinality + (3 * se));
  assertEquals(mergedEstimate, baselineEstimate);
}

代码示例来源：origin: addthis/stream-lib

/**
 * should not fail with HyperLogLogMergeException: "Cannot merge estimators of different sizes"
 */
@Test
public void testMergeWithRegisterSet() throws CardinalityMergeException {
  HyperLogLog first = new HyperLogLog(16, new RegisterSet(1 << 20));
  HyperLogLog second = new HyperLogLog(16, new RegisterSet(1 << 20));
  first.offer(0);
  second.offer(1);
  first.merge(second);
}

代码示例来源：origin: com.github.ddth/ddth-simplehll-core

/**
 * {@inheritDoc}
 */
@Override
public IHLL add(Object obj) {
  if (hll == null) {
    throw new IllegalStateException();
  }
  hll.offer(obj);
  return this;
}

代码示例来源：origin: LiveRamp/cascading_ext

@Override
public void operate(FlowProcess flow, FunctionCall call) {
 byte[] bytes = Bytes.getBytes((BytesWritable)call.getArguments().getObject(0));
 approxCounter.offer(bytes);
 function.hash(bytes, hashResult);
 for (int i = 0; i < hashResult.length; i++) {
  Tuple tuple = new Tuple(hashResult[i] / splitSize, hashResult[i] % splitSize, i);
  call.getOutputCollector().add(tuple);
 }
}

内容来源于网络，如有侵权，请联系作者删除！

HyperLogLog

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com