本文整理了Java中com.yahoo.memory.WritableMemory.writableRegion()
方法的一些代码示例,展示了WritableMemory.writableRegion()
的具体用法。这些代码示例主要来源于Github
/Stackoverflow
/Maven
等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。WritableMemory.writableRegion()
方法的具体详情如下:
包路径:com.yahoo.memory.WritableMemory
类名称:WritableMemory
方法名:writableRegion
[英]A writable region is a writable view of this object. This returns a new WritableMemory representing the defined writable region with the given offsetBytes and capacityBytes.
代码示例来源:origin: apache/incubator-druid
@SuppressWarnings("ResultOfObjectAllocationIgnored")
@Override
public void init(final ByteBuffer buf, final int position)
{
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN).writableRegion(position, size);
// Not necessary to keep the constructed object since it is cheap to reconstruct by wrapping the memory.
// The objects are not cached as in BuildBufferAggregator since they never exceed the max size and never move.
// So it is easier to reconstruct them by wrapping memory then to keep position-to-object mappings.
new Union(lgK, mem);
}
代码示例来源:origin: apache/incubator-druid
@Override
public void init(final ByteBuffer buf, final int position)
{
final WritableMemory mem = getMemory(buf).writableRegion(position, size);
putSketchIntoCache(buf, position, new HllSketch(lgK, tgtHllType, mem));
}
代码示例来源:origin: apache/incubator-druid
@Override
public Object get(final ByteBuffer buf, final int position)
{
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN).writableRegion(position, size);
final Lock lock = stripedLock.getAt(HllSketchBuildBufferAggregator.lockIndex(position)).readLock();
lock.lock();
try {
final Union union = Union.writableWrap(mem);
return union.getResult(tgtHllType);
}
finally {
lock.unlock();
}
}
代码示例来源:origin: apache/incubator-druid
/**
* In very rare cases sketches can exceed given memory, request on-heap memory and move there.
* We need to identify such sketches and reuse the same objects as opposed to wrapping new memory regions.
*/
@Override
public void relocate(final int oldPosition, final int newPosition, final ByteBuffer oldBuf, final ByteBuffer newBuf)
{
HllSketch sketch = sketchCache.get(oldBuf).get(oldPosition);
final WritableMemory oldMem = getMemory(oldBuf).writableRegion(oldPosition, size);
if (sketch.isSameResource(oldMem)) { // sketch has not moved
final WritableMemory newMem = getMemory(newBuf).writableRegion(newPosition, size);
sketch = HllSketch.writableWrap(newMem);
}
putSketchIntoCache(newBuf, newPosition, sketch);
}
代码示例来源:origin: apache/incubator-druid
@Override
public synchronized void relocate(int oldPosition, int newPosition, ByteBuffer oldBuffer, ByteBuffer newBuffer)
{
UpdateDoublesSketch sketch = sketches.get(oldBuffer).get(oldPosition);
final WritableMemory oldRegion = getMemory(oldBuffer).writableRegion(oldPosition, maxIntermediateSize);
if (sketch.isSameResource(oldRegion)) { // sketch was not relocated on heap
final WritableMemory newRegion = getMemory(newBuffer).writableRegion(newPosition, maxIntermediateSize);
sketch = UpdateDoublesSketch.wrap(newRegion);
}
putSketch(newBuffer, newPosition, sketch);
final Int2ObjectMap<UpdateDoublesSketch> map = sketches.get(oldBuffer);
map.remove(oldPosition);
if (map.isEmpty()) {
sketches.remove(oldBuffer);
memCache.remove(oldBuffer);
}
}
代码示例来源:origin: apache/incubator-druid
@Override
public synchronized void relocate(int oldPosition, int newPosition, ByteBuffer oldBuffer, ByteBuffer newBuffer)
{
DoublesUnion union = unions.get(oldBuffer).get(oldPosition);
final WritableMemory oldMem = getMemory(oldBuffer).writableRegion(oldPosition, maxIntermediateSize);
if (union.isSameResource(oldMem)) { // union was not relocated on heap
final WritableMemory newMem = getMemory(newBuffer).writableRegion(newPosition, maxIntermediateSize);
union = DoublesUnion.wrap(newMem);
}
putUnion(newBuffer, newPosition, union);
Int2ObjectMap<DoublesUnion> map = unions.get(oldBuffer);
map.remove(oldPosition);
if (map.isEmpty()) {
unions.remove(oldBuffer);
memCache.remove(oldBuffer);
}
}
代码示例来源:origin: apache/incubator-druid
/**
* This method uses locks because it can be used during indexing,
* and Druid can call aggregate() and get() concurrently
* https://github.com/apache/incubator-druid/pull/3956
* The returned sketch is a separate instance of ArrayOfDoublesCompactSketch
* representing the current state of the aggregation, and is not affected by consequent
* aggregate() calls
*/
@Override
public Object get(final ByteBuffer buf, final int position)
{
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
final Lock lock = stripedLock.getAt(lockIndex(position)).readLock();
lock.lock();
try {
final ArrayOfDoublesUpdatableSketch sketch = (ArrayOfDoublesUpdatableSketch) ArrayOfDoublesSketches
.wrapSketch(region);
return sketch.compact();
}
finally {
lock.unlock();
}
}
代码示例来源:origin: apache/incubator-druid
@Override
public void aggregate(final ByteBuffer buf, final int position)
{
final HllSketch sketch = selector.getObject();
if (sketch == null) {
return;
}
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN).writableRegion(position, size);
final Lock lock = stripedLock.getAt(HllSketchBuildBufferAggregator.lockIndex(position)).writeLock();
lock.lock();
try {
final Union union = Union.writableWrap(mem);
union.update(sketch);
}
finally {
lock.unlock();
}
}
代码示例来源:origin: apache/incubator-druid
/**
* This method uses locks because it can be used during indexing,
* and Druid can call aggregate() and get() concurrently
* https://github.com/apache/incubator-druid/pull/3956
* The returned sketch is a separate instance of ArrayOfDoublesCompactSketch
* representing the current state of the aggregation, and is not affected by consequent
* aggregate() calls
*/
@Override
public Object get(final ByteBuffer buf, final int position)
{
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
final Lock lock = stripedLock.getAt(ArrayOfDoublesSketchBuildBufferAggregator.lockIndex(position)).readLock();
lock.lock();
try {
final ArrayOfDoublesUnion union = ArrayOfDoublesSketches.wrapUnion(region);
return union.getResult();
}
finally {
lock.unlock();
}
}
代码示例来源:origin: apache/incubator-druid
private Union createNewUnion(ByteBuffer buf, int position, boolean isWrapped)
{
WritableMemory mem = getMemory(buf).writableRegion(position, maxIntermediateSize);
Union union = isWrapped
? (Union) SetOperation.wrap(mem)
: (Union) SetOperation.builder().setNominalEntries(size).build(Family.UNION, mem);
Int2ObjectMap<Union> unionMap = unions.get(buf);
if (unionMap == null) {
unionMap = new Int2ObjectOpenHashMap<>();
unions.put(buf, unionMap);
}
unionMap.put(position, union);
return union;
}
代码示例来源:origin: apache/incubator-druid
/**
* This method uses locks because it can be used during indexing,
* and Druid can call aggregate() and get() concurrently
* https://github.com/apache/incubator-druid/pull/3956
*/
@Override
public void aggregate(final ByteBuffer buf, final int position)
{
final ArrayOfDoublesSketch update = selector.getObject();
if (update == null) {
return;
}
// Wrapping memory and ArrayOfDoublesUnion is inexpensive compared to union operations.
// Maintaining a cache of wrapped objects per buffer position like in Theta sketch aggregator
// might might be considered, but it would increase complexity including relocate() support.
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
final Lock lock = stripedLock.getAt(ArrayOfDoublesSketchBuildBufferAggregator.lockIndex(position)).writeLock();
lock.lock();
try {
final ArrayOfDoublesUnion union = ArrayOfDoublesSketches.wrapUnion(region);
union.update(update);
}
finally {
lock.unlock();
}
}
代码示例来源:origin: apache/incubator-druid
@Override
public void init(final ByteBuffer buf, final int position)
{
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
new ArrayOfDoublesSetOperationBuilder().setNominalEntries(nominalEntries)
.setNumberOfValues(numberOfValues).buildUnion(region);
}
代码示例来源:origin: apache/incubator-druid
@Override
public synchronized void init(final ByteBuffer buffer, final int position)
{
final WritableMemory mem = getMemory(buffer);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
final UpdateDoublesSketch sketch = DoublesSketch.builder().setK(size).build(region);
putSketch(buffer, position, sketch);
}
代码示例来源:origin: apache/incubator-druid
@Override
public synchronized void init(final ByteBuffer buffer, final int position)
{
final WritableMemory mem = getMemory(buffer);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
final DoublesUnion union = DoublesUnion.builder().setMaxK(k).build(region);
putUnion(buffer, position, union);
}
代码示例来源:origin: apache/incubator-druid
@Override
public void init(final ByteBuffer buf, final int position)
{
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
new ArrayOfDoublesUpdatableSketchBuilder().setNominalEntries(nominalEntries)
.setNumberOfValues(valueSelectors.length)
.setNumberOfValues(valueSelectors.length).build(region);
}
代码示例来源:origin: apache/incubator-druid
/**
* This method uses locks because it can be used during indexing,
* and Druid can call aggregate() and get() concurrently
* https://github.com/apache/incubator-druid/pull/3956
*/
@Override
public void aggregate(final ByteBuffer buf, final int position)
{
for (int i = 0; i < valueSelectors.length; i++) {
values[i] = valueSelectors[i].getDouble();
}
final IndexedInts keys = keySelector.getRow();
// Wrapping memory and ArrayOfDoublesSketch is inexpensive compared to sketch operations.
// Maintaining a cache of wrapped objects per buffer position like in Theta sketch aggregator
// might might be considered, but it would increase complexity including relocate() support.
final WritableMemory mem = WritableMemory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
final WritableMemory region = mem.writableRegion(position, maxIntermediateSize);
final Lock lock = stripedLock.getAt(lockIndex(position)).writeLock();
lock.lock();
try {
final ArrayOfDoublesUpdatableSketch sketch = ArrayOfDoublesSketches.wrapUpdatableSketch(region);
for (int i = 0, keysSize = keys.size(); i < keysSize; i++) {
final String key = keySelector.lookupName(keys.get(i));
sketch.update(key, values);
}
}
finally {
lock.unlock();
}
}
代码示例来源:origin: DataSketches/sketches-core
private static double directUnionTrial2(
WritableMemory heapMem, int[] heapLayout, int sketchNomEntries, int unionNomEntries) {
WritableMemory unionMem = heapMem.writableRegion(heapLayout[0], heapLayout[1]-heapLayout[0]);
WritableMemory sketch1mem = heapMem.writableRegion(heapLayout[1], heapLayout[2]-heapLayout[1]);
WritableMemory sketch2mem = heapMem.writableRegion(heapLayout[2], heapLayout[3]-heapLayout[2]);
WritableMemory sketch3mem = heapMem.writableRegion(heapLayout[3], heapLayout[4]-heapLayout[3]);
WritableMemory resultMem = heapMem.writableRegion(heapLayout[4], heapLayout[5]-heapLayout[4]);
//Recover the 3 sketches
UpdateSketch sk1 = (UpdateSketch) Sketch.wrap(sketch1mem);
UpdateSketch sk2 = (UpdateSketch) Sketch.wrap(sketch2mem);
UpdateSketch sk3 = (UpdateSketch) Sketch.wrap(sketch3mem);
//confirm that each of these 3 sketches is exact.
assertEquals(sk1.getEstimate(), sketchNomEntries, 0.0);
assertEquals(sk2.getEstimate(), sketchNomEntries, 0.0);
assertEquals(sk3.getEstimate(), sketchNomEntries, 0.0);
//Create a new union in the same space with a smaller size.
unionMem.clear();
Union union = SetOperation.builder().setNominalEntries(unionNomEntries).buildUnion(unionMem);
union.update(sk1);
union.update(sk2);
union.update(sk3);
Sketch resSk = union.getResult(true, resultMem);
double est = resSk.getEstimate();
return est;
}
代码示例来源:origin: DataSketches/sketches-core
/**
* @return a byte array representation of this object
*/
public byte[] toByteArray() {
final int sizeBytes = PREAMBLE_SIZE_BYTES + sketch_.getSerializedSizeBytes();
final byte[] byteArray = new byte[sizeBytes];
final WritableMemory mem = WritableMemory.wrap(byteArray);
mem.putByte(PREAMBLE_LONGS_BYTE, (byte) 1); // unused, always 1
mem.putByte(SERIAL_VERSION_BYTE, serialVersionUID);
mem.putByte(FAMILY_ID_BYTE, (byte) Family.TUPLE.getID());
mem.putByte(SKETCH_TYPE_BYTE, (byte) SerializerDeserializer.SketchType.ArrayOfDoublesUnion.ordinal());
mem.putLong(THETA_LONG, theta_);
sketch_.serializeInto(mem.writableRegion(PREAMBLE_SIZE_BYTES, mem.getCapacity() - PREAMBLE_SIZE_BYTES));
return byteArray;
}
代码示例来源:origin: DataSketches/sketches-core
WritableMemory unionMem = heapMem.writableRegion(offset, bytes);
WritableMemory sketch1mem = heapMem.writableRegion(heapLayout[1], heapLayout[2]-heapLayout[1]);
WritableMemory sketch2mem = heapMem.writableRegion(heapLayout[2], heapLayout[3]-heapLayout[2]);
WritableMemory sketch3mem = heapMem.writableRegion(heapLayout[3], heapLayout[4]-heapLayout[3]);
WritableMemory resultMem = heapMem.writableRegion(heapLayout[4], heapLayout[5]-heapLayout[4]);
代码示例来源:origin: DataSketches/sketches-core
final WritableMemory sketchMem = mem.writableRegion(ArrayOfDoublesUnion.PREAMBLE_SIZE_BYTES,
mem.getCapacity() - ArrayOfDoublesUnion.PREAMBLE_SIZE_BYTES);
sketch = new DirectArrayOfDoublesQuickSelectSketch(sketchMem, seed);
内容来源于网络,如有侵权,请联系作者删除!