private static IncompleteCb addIncompleteCompressionBuffer(long cbStartOffset, DiskRangeList target, int extraChunkCountToLog, boolean isTracingEnabled, IoTrace trace) { IncompleteCb icb = new IncompleteCb(cbStartOffset, target.getEnd()); if (isTracingEnabled) { LOG.trace("Replacing " + target + " (and " + extraChunkCountToLog + " previous chunks) with " + icb + " in the buffers"); } trace.logInvalidOrcCb(cbStartOffset, target.getEnd()); target.replaceSelfWith(icb); return icb; }
private IncompleteCb addIncompleteCompressionBuffer( long cbStartOffset, DiskRangeList target, int extraChunkCount) { IncompleteCb icb = new IncompleteCb(cbStartOffset, target.getEnd()); if (isTracingEnabled) { LOG.trace("Replacing " + target + " (and " + extraChunkCount + " previous chunks) with " + icb + " in the buffers"); } target.replaceSelfWith(icb); return icb; }
private final static void checkOrder(DiskRangeList prev, DiskRangeList next, DiskRangeList ref) { if (prev.getEnd() <= next.getOffset()) return; assertInvalidOrder(ref.prev == null ? ref : ref.prev, prev, next); }
private static DiskRangeList findIntersectingPosition( DiskRangeList ranges, long offset, long end) { if (offset < 0) return ranges; ranges = findUpperBound(ranges, offset); ranges = findLowerBound(ranges, end); // We are now on some intersecting buffer, find the first intersecting buffer. while (ranges.prev != null && ranges.prev.getEnd() > offset) { if (ranges.prev.getEnd() > ranges.getOffset()) { throwRangesError(ranges, offset, end); } ranges = ranges.prev; } return ranges; }
if (currentNotCached.getOffset() >= currentCached.getOffset()) { if (currentNotCached.getEnd() <= currentCached.getEnd()) { gotAllData.value = false; if (currentNotCached.getEnd() <= currentCached.getEnd()) { DiskRangeList tail = new DiskRangeList(currentCached.getEnd(), currentNotCached.getEnd()); currentNotCached.insertPartAfter(currentCached); currentCached.insertAfter(tail);
private static DiskRangeList findIntersectingPosition( DiskRangeList ranges, long offset, long end) { if (offset < 0) return ranges; ranges = findUpperBound(ranges, offset); ranges = findLowerBound(ranges, end); // We are now on some intersecting buffer, find the first intersecting buffer. while (ranges.prev != null && ranges.prev.getEnd() > offset) { if (ranges.prev.getEnd() > ranges.getOffset()) { throwRangesError(ranges, offset, end); } ranges = ranges.prev; } return ranges; }
public static DiskRangeList findLowerBound(DiskRangeList ranges, long end) { while (ranges.getOffset() > end) { if (ranges.prev.getEnd() > ranges.getOffset()) { throwRangesError(ranges, end, end); } ranges = ranges.prev; } return ranges; }
while (true) { if ((prev == null) || (prev.getEnd() <= streamStartOffset)) break;
public static DiskRangeList findLowerBound(DiskRangeList ranges, long end) { while (ranges.getOffset() > end) { if (ranges.prev.getEnd() > ranges.getOffset()) { throwRangesError(ranges, end, end); } ranges = ranges.prev; } return ranges; }
while (true) { if ((prev == null) || (prev.getEnd() <= streamStartOffset)) break;
/** Finds compressed offset in a stream and makes sure iter points to its position. This may be necessary for obscure combinations of compression and encoding boundaries. */ private static DiskRangeList findExactPosition(DiskRangeList ranges, long offset) { if (offset < 0) return ranges; ranges = findUpperBound(ranges, offset); ranges = findLowerBound(ranges, offset); if (offset < ranges.getOffset() || offset >= ranges.getEnd()) { throwRangesError(ranges, offset, offset); } return ranges; }
/** Finds compressed offset in a stream and makes sure iter points to its position. This may be necessary for obscure combinations of compression and encoding boundaries. */ private static DiskRangeList findExactPosition(DiskRangeList ranges, long offset) { if (offset < 0) return ranges; ranges = findUpperBound(ranges, offset); ranges = findLowerBound(ranges, offset); if (offset < ranges.getOffset() || offset >= ranges.getEnd()) { throwRangesError(ranges, offset, offset); } return ranges; }
absOffset, currentNotCached.getEnd() + baseOffset) .entrySet().iterator(); long cacheEnd = -1;
public DiskRangeList getIncompleteCbs( DiskRangeList ranges, long baseOffset, BooleanRef gotAllData) { DiskRangeList prev = ranges.prev; if (prev == null) { prev = new MutateHelper(ranges); } DiskRangeList current = ranges; gotAllData.value = true; // Assume by default that we would find everything. while (current != null) { // We assume ranges in "ranges" are non-overlapping; thus, we will save next in advance. DiskRangeList check = current; current = current.next; if (check.hasData()) continue; Integer badLength = cache.get(Long.valueOf(check.getOffset() + baseOffset)); if (badLength == null || badLength < check.getLength()) { gotAllData.value = false; continue; } // We could just remove here and handle the missing tail during read, but that can be // dangerous; let's explicitly add an incomplete CB. check.replaceSelfWith(new IncompleteCb(check.getOffset(), check.getEnd())); } return prev.next; }
if (partOffset == current.getOffset() && current instanceof CacheChunk) { assert current.getOffset() == partOffset && current.getEnd() == partEnd; lastUncompressed = (CacheChunk)current; current = current.next; if (noMoreDataForPart) break; // Done with this part. if (current.getEnd() > partEnd) { + current.getOffset() + ", " + current.getEnd() + ")"); trace.logUncompressedData(current.getOffset(), current.getEnd()); BufferChunk curBc = (BufferChunk)current; hasEntirePartTo = (hasEntirePartTo == current.getOffset()) ? current.getEnd() : -1; if (hasEntirePartTo == -1) {
if (partOffset == current.getOffset() && current instanceof CacheChunk) { assert current.getOffset() == partOffset && current.getEnd() == partEnd; lastUncompressed = (CacheChunk)current; current = current.next; if (noMoreDataForPart) break; // Done with this part. if (current.getEnd() > partEnd) { + current.getOffset() + ", " + current.getEnd() + ")"); hasEntirePartTo = (hasEntirePartTo == current.getOffset()) ? current.getEnd() : -1; if (hasEntirePartTo == -1) {
while (drl != null) { assert drl.hasData(); long from = drl.getOffset(), to = drl.getEnd(); int offsetFromReadStart = (int)(from - readStartPos), candidateSize = (int)(to - from); ByteBuffer data = drl.getData().duplicate(); DiskRangeList candidate = current; current = current.next; long from = candidate.getOffset(), to = candidate.getEnd();
public void verifyResult(DiskRangeList result, long... vals) { for (int i = 0; i < vals.length; i += 3) { switch ((int)vals[i]) { case INCOMPLETE: assertTrue(result instanceof IncompleteCb); break; case DRL: assertFalse(result instanceof IncompleteCb); break; default: fail(); } assertEquals(vals[i + 1], result.getOffset()); assertEquals(vals[i + 2], result.getEnd()); result = result.next; } assertNull(result); }