org.apache.lucene.util.fst.Util.toIntsRef()方法的使用及代码示例

x33g5p2x  于2022-02-01 转载在 其他  
字(12.3k)|赞(0)|评价(0)|浏览(168)

本文整理了Java中org.apache.lucene.util.fst.Util.toIntsRef()方法的一些代码示例，展示了Util.toIntsRef()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Util.toIntsRef()方法的具体详情如下：
包路径：org.apache.lucene.util.fst.Util
类名称：Util
方法名：toIntsRef

Util.toIntsRef介绍

[英]Just takes unsigned byte values from the BytesRef and converts into an IntsRef.
[中]只需从字节引用中获取无符号字节值，并将其转换为IntsRef。

代码示例

代码示例来源：origin: org.apache.lucene/lucene-core

  1. private void append(Builder<BytesRef> builder, FST<BytesRef> subIndex, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<BytesRef> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<BytesRef> indexEnt;
  4.   while((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), indexEnt.output);
  9.   }
  10.  }
  11. }

代码示例来源：origin: org.apache.lucene/lucene-core

  1. assert bytes.length > 0;
  2. scratchBytes.writeTo(bytes, 0);
  3. indexBuilder.add(Util.toIntsRef(prefix, scratchIntsRef), new BytesRef(bytes, 0, bytes.length));
  4. scratchBytes.reset();

代码示例来源：origin: org.elasticsearch/elasticsearch

  1. public void finishTerm(long defaultWeight) throws IOException {
  2.   ArrayUtil.timSort(surfaceFormsAndPayload, 0, count);
  3.   int deduplicator = 0;
  4.   analyzed.append((byte) 0);
  5.   analyzed.setLength(analyzed.length() + 1);
  6.   analyzed.grow(analyzed.length());
  7.   for (int i = 0; i < count; i++) {
  8.     analyzed.setByteAt(analyzed.length() - 1, (byte) deduplicator++);
  9.     Util.toIntsRef(analyzed.get(), scratchInts);
  10.     SurfaceFormAndPayload candiate = surfaceFormsAndPayload[i];
  11.     long cost = candiate.weight == -1 ? encodeWeight(Math.min(Integer.MAX_VALUE, defaultWeight)) : candiate.weight;
  12.     builder.add(scratchInts.get(), outputs.newPair(cost, candiate.payload));
  13.   }
  14.   seenSurfaceForms.clear();
  15.   count = 0;
  16. }

代码示例来源：origin: org.elasticsearch/elasticsearch

  1. analyzed.append((byte) dedup);
  2. Util.toIntsRef(analyzed.get(), scratchInts);

代码示例来源：origin: harbby/presto-connectors

  1. /**
  2.  * Adds a context with boost, set <code>exact</code> to false
  3.  * if the context is a prefix of any indexed contexts
  4.  */
  5. public void addContext(CharSequence context, float boost, boolean exact) {
  6.  if (boost < 0f) {
  7.   throw new IllegalArgumentException("'boost' must be >= 0");
  8.  }
  9.  for (int i = 0; i < context.length(); i++) {
  10.   if (ContextSuggestField.CONTEXT_SEPARATOR == context.charAt(i)) {
  11.    throw new IllegalArgumentException("Illegal value [" + context + "] UTF-16 codepoint [0x"
  12.      + Integer.toHexString((int) context.charAt(i))+ "] at position " + i + " is a reserved character");
  13.   }
  14.  }
  15.  contexts.put(IntsRef.deepCopyOf(Util.toIntsRef(new BytesRef(context), scratch)), new ContextMetaData(boost, exact));
  16. }

代码示例来源：origin: org.infinispan/infinispan-embedded-query

  1. private void append(Builder<BytesRef> builder, FST<BytesRef> subIndex, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<BytesRef> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<BytesRef> indexEnt;
  4.   while((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), indexEnt.output);
  9.   }
  10.  }
  11. }

代码示例来源：origin: org.apache.lucene/lucene-sandbox

  1. private void append(Builder<Pair<BytesRef,Long>> builder, FST<Pair<BytesRef,Long>> subIndex, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<Pair<BytesRef,Long>> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<Pair<BytesRef,Long>> indexEnt;
  4.   while((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), indexEnt.output);
  9.   }
  10.  }
  11. }

代码示例来源：origin: org.apache.servicemix.bundles/org.apache.servicemix.bundles.lucene

  1. private void append(Builder<BytesRef> builder, FST<BytesRef> subIndex, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<BytesRef> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<BytesRef> indexEnt;
  4.   while((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), indexEnt.output);
  9.   }
  10.  }
  11. }

代码示例来源：origin: harbby/presto-connectors

  1. private void append(Builder<BytesRef> builder, FST<BytesRef> subIndex, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<BytesRef> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<BytesRef> indexEnt;
  4.   while((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), indexEnt.output);
  9.   }
  10.  }
  11. }

代码示例来源：origin: harbby/presto-connectors

  1. private void append(Builder<Pair<BytesRef,Long>> builder, FST<Pair<BytesRef,Long>> subIndex, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<Pair<BytesRef,Long>> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<Pair<BytesRef,Long>> indexEnt;
  4.   while((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), indexEnt.output);
  9.   }
  10.  }
  11. }

代码示例来源：origin: org.apache.lucene/lucene-classification

  1. private void updateFST(SortedMap<String, Double> weights) throws IOException {
  2.  PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();
  3.  Builder<Long> fstBuilder = new Builder<>(FST.INPUT_TYPE.BYTE1, outputs);
  4.  BytesRefBuilder scratchBytes = new BytesRefBuilder();
  5.  IntsRefBuilder scratchInts = new IntsRefBuilder();
  6.  for (Map.Entry<String, Double> entry : weights.entrySet()) {
  7.   scratchBytes.copyChars(entry.getKey());
  8.   fstBuilder.add(Util.toIntsRef(scratchBytes.get(), scratchInts), entry
  9.       .getValue().longValue());
  10.  }
  11.  fst = fstBuilder.finish();
  12. }

代码示例来源：origin: org.apache.lucene/lucene-codecs

  1. private void append(Builder<Output> builder, FST<Output> subIndex, long termOrdOffset, IntsRefBuilder scratchIntsRef) throws IOException {
  2.   final BytesRefFSTEnum<Output> subIndexEnum = new BytesRefFSTEnum<>(subIndex);
  3.   BytesRefFSTEnum.InputOutput<Output> indexEnt;
  4.   while ((indexEnt = subIndexEnum.next()) != null) {
  5.    //if (DEBUG) {
  6.    //  System.out.println("      add sub=" + indexEnt.input + " " + indexEnt.input + " output=" + indexEnt.output);
  7.    //}
  8.    Output output = indexEnt.output;
  9.    //long blockTermCount = output.endOrd - output.startOrd + 1;
  10.    Output newOutput = FST_OUTPUTS.newOutput(output.bytes, termOrdOffset+output.startOrd, output.endOrd-termOrdOffset);
  11.    //System.out.println("  append sub=" + indexEnt.input + " output=" + indexEnt.output + " termOrdOffset=" + termOrdOffset + " blockTermCount=" + blockTermCount  + " newOutput=" + newOutput  + " endOrd=" + (termOrdOffset+Long.MAX_VALUE-output.endOrd));
  12.    builder.add(Util.toIntsRef(indexEnt.input, scratchIntsRef), newOutput);
  13.   }
  14.  }
  15. }

代码示例来源：origin: org.apache.lucene/lucene-codecs

  1. @Override
  2. public void add(BytesRef text, TermStats stats, long termsFilePointer) throws IOException {
  3.  if (text.length == 0) {
  4.   // We already added empty string in ctor
  5.   assert termsFilePointer == startTermsFilePointer;
  6.   return;
  7.  }
  8.  final int lengthSave = text.length;
  9.  text.length = indexedTermPrefixLength(lastTerm.get(), text);
  10.  try {
  11.   fstBuilder.add(Util.toIntsRef(text, scratchIntsRef), termsFilePointer);
  12.  } finally {
  13.   text.length = lengthSave;
  14.  }
  15.  lastTerm.copyBytes(text);
  16. }

代码示例来源：origin: org.apache.lucene/lucene-codecs

  1. public void finishTerm(BytesRef text, BlockTermState state) throws IOException {
  2.  // write term meta data into fst
  3.  final FSTTermOutputs.TermData meta = new FSTTermOutputs.TermData();
  4.  meta.longs = new long[longsSize];
  5.  meta.bytes = null;
  6.  meta.docFreq = state.docFreq;
  7.  meta.totalTermFreq = state.totalTermFreq;
  8.  postingsWriter.encodeTerm(meta.longs, metaWriter, fieldInfo, state, true);
  9.  final int bytesSize = (int)metaWriter.getFilePointer();
  10.  if (bytesSize > 0) {
  11.   meta.bytes = new byte[bytesSize];
  12.   metaWriter.writeTo(meta.bytes, 0);
  13.   metaWriter.reset();
  14.  }
  15.  builder.add(Util.toIntsRef(text, scratchTerm), meta);
  16.  numTerms++;
  17. }

代码示例来源：origin: com.strapdata.elasticsearch/elasticsearch

  1. public void finishTerm(long defaultWeight) throws IOException {
  2.   ArrayUtil.timSort(surfaceFormsAndPayload, 0, count);
  3.   int deduplicator = 0;
  4.   analyzed.append((byte) 0);
  5.   analyzed.setLength(analyzed.length() + 1);
  6.   analyzed.grow(analyzed.length());
  7.   for (int i = 0; i < count; i++) {
  8.     analyzed.setByteAt(analyzed.length() - 1, (byte) deduplicator++);
  9.     Util.toIntsRef(analyzed.get(), scratchInts);
  10.     SurfaceFormAndPayload candiate = surfaceFormsAndPayload[i];
  11.     long cost = candiate.weight == -1 ? encodeWeight(Math.min(Integer.MAX_VALUE, defaultWeight)) : candiate.weight;
  12.     builder.add(scratchInts.get(), outputs.newPair(cost, candiate.payload));
  13.   }
  14.   seenSurfaceForms.clear();
  15.   count = 0;
  16. }

代码示例来源：origin: org.apache.servicemix.bundles/org.apache.servicemix.bundles.elasticsearch

  1. public void finishTerm(long defaultWeight) throws IOException {
  2.   ArrayUtil.timSort(surfaceFormsAndPayload, 0, count);
  3.   int deduplicator = 0;
  4.   analyzed.append((byte) 0);
  5.   analyzed.setLength(analyzed.length() + 1);
  6.   analyzed.grow(analyzed.length());
  7.   for (int i = 0; i < count; i++) {
  8.     analyzed.setByteAt(analyzed.length() - 1, (byte) deduplicator++);
  9.     Util.toIntsRef(analyzed.get(), scratchInts);
  10.     SurfaceFormAndPayload candiate = surfaceFormsAndPayload[i];
  11.     long cost = candiate.weight == -1 ? encodeWeight(Math.min(Integer.MAX_VALUE, defaultWeight)) : candiate.weight;
  12.     builder.add(scratchInts.get(), outputs.newPair(cost, candiate.payload));
  13.   }
  14.   seenSurfaceForms.clear();
  15.   count = 0;
  16. }

代码示例来源：origin: apache/servicemix-bundles

  1. public void finishTerm(long defaultWeight) throws IOException {
  2.   ArrayUtil.timSort(surfaceFormsAndPayload, 0, count);
  3.   int deduplicator = 0;
  4.   analyzed.append((byte) 0);
  5.   analyzed.setLength(analyzed.length() + 1);
  6.   analyzed.grow(analyzed.length());
  7.   for (int i = 0; i < count; i++) {
  8.     analyzed.setByteAt(analyzed.length() - 1, (byte) deduplicator++);
  9.     Util.toIntsRef(analyzed.get(), scratchInts);
  10.     SurfaceFormAndPayload candiate = surfaceFormsAndPayload[i];
  11.     long cost = candiate.weight == -1 ? encodeWeight(Math.min(Integer.MAX_VALUE, defaultWeight)) : candiate.weight;
  12.     builder.add(scratchInts.get(), outputs.newPair(cost, candiate.payload));
  13.   }
  14.   seenSurfaceForms.clear();
  15.   count = 0;
  16. }

代码示例来源：origin: harbby/presto-connectors

  1. public void finishTerm(long defaultWeight) throws IOException {
  2.   ArrayUtil.timSort(surfaceFormsAndPayload, 0, count);
  3.   int deduplicator = 0;
  4.   analyzed.append((byte) 0);
  5.   analyzed.setLength(analyzed.length() + 1);
  6.   analyzed.grow(analyzed.length());
  7.   for (int i = 0; i < count; i++) {
  8.     analyzed.setByteAt(analyzed.length() - 1, (byte) deduplicator++);
  9.     Util.toIntsRef(analyzed.get(), scratchInts);
  10.     SurfaceFormAndPayload candiate = surfaceFormsAndPayload[i];
  11.     long cost = candiate.weight == -1 ? encodeWeight(Math.min(Integer.MAX_VALUE, defaultWeight)) : candiate.weight;
  12.     builder.add(scratchInts.get(), outputs.newPair(cost, candiate.payload));
  13.   }
  14.   seenSurfaceForms.clear();
  15.   count = 0;
  16. }

代码示例来源：origin: harbby/presto-connectors

  1. /**
  2.  * Writes all the entries for the FST input term
  3.  */
  4. public void finishTerm() throws IOException {
  5.  int numArcs = 0;
  6.  int numDedupBytes = 1;
  7.  analyzed.grow(analyzed.length() + 1);
  8.  analyzed.setLength(analyzed.length() + 1);
  9.  for (Entry entry : entries) {
  10.   if (numArcs == maxNumArcsForDedupByte(numDedupBytes)) {
  11.    analyzed.setByteAt(analyzed.length() - 1, (byte) (numArcs));
  12.    analyzed.grow(analyzed.length() + 1);
  13.    analyzed.setLength(analyzed.length() + 1);
  14.    numArcs = 0;
  15.    numDedupBytes++;
  16.   }
  17.   analyzed.setByteAt(analyzed.length() - 1, (byte) numArcs++);
  18.   Util.toIntsRef(analyzed.get(), scratchInts);
  19.   builder.add(scratchInts.get(), outputs.newPair(entry.weight, entry.payload));
  20.  }
  21.  maxAnalyzedPathsPerOutput = Math.max(maxAnalyzedPathsPerOutput, entries.size());
  22.  entries.clear();
  23. }

代码示例来源：origin: org.apache.lucene/lucene-codecs

  1. private void writeFST(FieldInfo field, Iterable<BytesRef> values) throws IOException {
  2.  meta.writeVInt(field.number);
  3.  meta.writeByte(FST);
  4.  meta.writeLong(data.getFilePointer());
  5.  PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();
  6.  Builder<Long> builder = new Builder<>(INPUT_TYPE.BYTE1, outputs);
  7.  IntsRefBuilder scratch = new IntsRefBuilder();
  8.  long ord = 0;
  9.  for (BytesRef v : values) {
  10.   builder.add(Util.toIntsRef(v, scratch), ord);
  11.   ord++;
  12.  }
  13.  FST<Long> fst = builder.finish();
  14.  if (fst != null) {
  15.   fst.save(data);
  16.  }
  17.  meta.writeVLong(ord);
  18. }

内容来源于网络，如有侵权，请联系作者删除！

Util

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com