@JsonCreator public static BatchColumnRangeSelection create(@JsonProperty("startCol") byte[] startCol, @JsonProperty("endCol") byte[] endCol, @JsonProperty("batchHint") int batchHint) { return new BatchColumnRangeSelection(new ColumnRangeSelection(startCol, endCol), batchHint); }
public static ColumnRangeSelection valueOf(String serialized) { // Pass in -1 to split so that it doesn't discard empty strings String[] split = deserializeRegex.split(serialized, -1); byte[] startCol = PtBytes.decodeBase64(split[0]); byte[] endCol = PtBytes.decodeBase64(split[1]); return new ColumnRangeSelection(startCol, endCol); }
private void getAllRowsAndAssert(WideRowTable table, Blackhole blackhole, Consumer<Integer> assertion) { int rowsRead = table.getTransactionManager().runTaskThrowOnConflict(txn -> { Iterator<Map.Entry<Cell, byte[]>> iter = txn.getRowsColumnRange( table.getTableRef(), Collections.singleton(Tables.ROW_BYTES.array()), new ColumnRangeSelection(null, null), 10000); int count = 0; while (iter.hasNext()) { blackhole.consume(iter.next()); ++count; } return count; }); assertion.accept(rowsRead); } }
@Test public void testColumnRangeReadSupported() { Transaction t1 = startTransaction(); // The transactions table is registered as IGNORE_ALL, so the request is supported // Reading at timestamp 0 to avoid any repercussions for in-flight transactions t1.getRowsColumnRange(TransactionConstants.TRANSACTION_TABLE, ImmutableList.of(ValueType.VAR_LONG.convertFromJava(0L)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 1); }
@Override public Map<byte[], RowColumnRangeIterator> getRowsColumnRange(TableReference tableRef, Iterable<byte[]> rows, BatchColumnRangeSelection batchColumnRangeSelection, long timestamp) { Map<byte[], RowColumnRangeIterator> result = Maps.newHashMap(); ConcurrentSkipListMap<Key, byte[]> table = getTableMap(tableRef).entries; ColumnRangeSelection columnRangeSelection = new ColumnRangeSelection( batchColumnRangeSelection.getStartCol(), batchColumnRangeSelection.getEndCol()); for (byte[] row : rows) { result.put(row, getColumnRangeForSingleRow(table, row, columnRangeSelection, timestamp)); } return result; }
@Override protected List<byte[]> getRange(Transaction txn, long startInclusive, long endExclusive) { KvDynamicColumnsTable table = BenchmarksTableFactory.of().getKvDynamicColumnsTable(txn); List<byte[]> data = Lists.newArrayList(); table.getRowsColumnRange( ImmutableSet.of(KvDynamicColumnsRow.of(bucket)), new ColumnRangeSelection( KvDynamicColumnsColumn.of(startInclusive).persistToBytes(), KvDynamicColumnsColumn.of(endExclusive).persistToBytes()), batchSize) .forEachRemaining(entry -> data.add(entry.getValue().getValue())); return data; }
@Benchmark @Threads(1) @Warmup(time = 16, timeUnit = TimeUnit.SECONDS) @Measurement(time = 160, timeUnit = TimeUnit.SECONDS) public Object getAllColumnsAligned(WideRowsTable table) { List<byte[]> rows = IntStream.rangeClosed(0, WideRowsTable.NUM_ROWS - 1) .mapToObj(WideRowsTable::getRow) .collect(Collectors.toList()); RowColumnRangeIterator rowsColumnRange = table.getKvs().getRowsColumnRange( table.getTableRef(), rows, new ColumnRangeSelection(null, null), 10000, Long.MAX_VALUE); int expectedNumCells = WideRowsTable.NUM_ROWS * WideRowsTable.NUM_COLS_PER_ROW; List<Map.Entry<Cell, Value>> loadedCells = new ArrayList<>(expectedNumCells); while (rowsColumnRange.hasNext()) { loadedCells.add(rowsColumnRange.next()); } Preconditions.checkState(loadedCells.size() == expectedNumCells, "Should be %s cells, but were: %s", expectedNumCells, loadedCells.size()); return loadedCells; }
@Test public void testColumnRangeReadUnsupported() { Transaction t1 = startTransaction(); try { t1.getRowsColumnRange(TEST_TABLE, ImmutableList.of(PtBytes.toBytes("row1")), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 1); fail(); } catch (UnsupportedOperationException e) { // expected } }
@Benchmark @Threads(1) @Warmup(time = 16, timeUnit = TimeUnit.SECONDS) @Measurement(time = 160, timeUnit = TimeUnit.SECONDS) public Object getAllColumnsUnaligned(WideRowsTable table) { List<byte[]> rows = IntStream.rangeClosed(0, WideRowsTable.NUM_ROWS - 1) .mapToObj(WideRowsTable::getRow) .collect(Collectors.toList()); RowColumnRangeIterator rowsColumnRange = table.getKvs().getRowsColumnRange( table.getTableRef(), rows, new ColumnRangeSelection(null, null), 10017, Long.MAX_VALUE); int expectedNumCells = WideRowsTable.NUM_ROWS * WideRowsTable.NUM_COLS_PER_ROW; List<Map.Entry<Cell, Value>> loadedCells = new ArrayList<>(expectedNumCells); while (rowsColumnRange.hasNext()) { loadedCells.add(rowsColumnRange.next()); } Preconditions.checkState(loadedCells.size() == expectedNumCells, "Should be %s cells, but were: %s", expectedNumCells, loadedCells.size()); return loadedCells; }
private static ColumnRangeSelection allPossibleColumns() { byte[] startCol = SweepableCellsTable.SweepableCellsColumn.of(0L, MINIMUM_WRITE_INDEX) .persistToBytes(); byte[] endCol = SweepableCellsTable.SweepableCellsColumn.of(TS_FINE_GRANULARITY, 0L) .persistToBytes(); return new ColumnRangeSelection(startCol, endCol); } }
private ColumnRangeSelection getColRangeSelection(long minFineInclusive, long maxFineExclusive) { byte[] start = SweepableTimestampsTable.SweepableTimestampsColumn.of(minFineInclusive).persistToBytes(); byte[] end = SweepableTimestampsTable.SweepableTimestampsColumn.of(maxFineExclusive).persistToBytes(); return new ColumnRangeSelection(start, end); }
@Test public void notAllowColumnRangeGets() { checkThrowsAndNoInteraction(() -> readTransaction.getRowsColumnRange( DUMMY_THOROUGH_TABLE, ImmutableList.of(EMPTY_BYTES), new ColumnRangeSelection(EMPTY_BYTES, EMPTY_BYTES), 1), IllegalStateException.class, "Cannot read"); }
private ColumnRangeSelection columnsBetween(long startTsInclusive, long endTsExclusive, long partitionFine) { long startIncl = exactColumnOrElseBeginningOfRow(startTsInclusive, partitionFine); byte[] startCol = SweepableCellsTable.SweepableCellsColumn.of(startIncl, SweepQueueUtils.MINIMUM_WRITE_INDEX) .persistToBytes(); long endExcl = exactColumnOrElseOneBeyondEndOfRow(endTsExclusive, partitionFine); byte[] endCol = SweepableCellsTable.SweepableCellsColumn.of(endExcl, SweepQueueUtils.MINIMUM_WRITE_INDEX) .persistToBytes(); return new ColumnRangeSelection(startCol, endCol); }
@Test public void testGetRowColumnRangeCellBatchHistorical() { putTestDataForMultipleTimestamps(); RowColumnRangeIterator values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(0)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 1, TEST_TIMESTAMP + 2); assertNextElementMatches(values, TEST_CELL, val(0, 7)); assertFalse(values.hasNext()); values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(0)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 1, TEST_TIMESTAMP + 1); assertNextElementMatches(values, TEST_CELL, val(0, 5)); assertFalse(values.hasNext()); }
@Benchmark @Threads(1) @Warmup(time = 16, timeUnit = TimeUnit.SECONDS) @Measurement(time = 160, timeUnit = TimeUnit.SECONDS) public Object getAllColumnsSingleBigRow(VeryWideRowTable table, Blackhole blackhole) { RowColumnRangeIterator iter = table.getKvs().getRowsColumnRange( table.getTableRef(), Collections.singleton(Tables.ROW_BYTES.array()), new ColumnRangeSelection(null, null), 10000, Long.MAX_VALUE); int count = 0; while (iter.hasNext()) { blackhole.consume(iter.next()); ++count; } Preconditions.checkState(count == table.getNumCols(), "Should be %s cells, but were: %s", table.getNumCols(), count); return count; }
RowColumnRangeIterator values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 1, TEST_TIMESTAMP + 1); values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1)), new ColumnRangeSelection(RangeRequests.nextLexicographicName(column(0)), PtBytes.EMPTY_BYTE_ARRAY), 1, TEST_TIMESTAMP + 1); values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1)), new ColumnRangeSelection(RangeRequests.nextLexicographicName(column(0)), column(2)), 1, TEST_TIMESTAMP + 1); values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1)), new ColumnRangeSelection(RangeRequests.nextLexicographicName(column(2)), PtBytes.EMPTY_BYTE_ARRAY), 1, TEST_TIMESTAMP + 1); values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), Integer.MAX_VALUE, TEST_TIMESTAMP + 1);
@Test public void testGetRowColumnRangeCellBatchMultipleHistorical() { keyValueService.put(TEST_TABLE, ImmutableMap.of(Cell.create(row(1), column(0)), val(0, 5)), TEST_TIMESTAMP); keyValueService.put(TEST_TABLE, ImmutableMap.of(Cell.create(row(1), column(0)), val(0, 7)), TEST_TIMESTAMP + 1); keyValueService.put(TEST_TABLE, ImmutableMap.of(Cell.create(row(1), column(1)), val(0, 5)), TEST_TIMESTAMP); keyValueService.put(TEST_TABLE, ImmutableMap.of(Cell.create(row(1), column(1)), val(0, 7)), TEST_TIMESTAMP + 1); RowColumnRangeIterator values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, RangeRequests.nextLexicographicName(column(1))), 2, TEST_TIMESTAMP + 1); assertNextElementMatches(values, Cell.create(row(1), column(0)), val(0, 5)); assertNextElementMatches(values, Cell.create(row(1), column(1)), val(0, 5)); assertFalse(values.hasNext()); }
private RowsColumnRangeBatchRequest createRequest(int numTotalRows) { ColumnRangeSelection fullColumnRange = new ColumnRangeSelection(col(0), col(5)); ImmutableRowsColumnRangeBatchRequest.Builder request = ImmutableRowsColumnRangeBatchRequest.builder().columnRangeSelection(fullColumnRange); if (hasPartialFirstRow) { request.partialFirstRow(Maps.immutableEntry(row(0), BatchColumnRangeSelection.create(col(3), col(5), 10))); } int firstFullRowIndex = hasPartialFirstRow ? 2 : 1; int lastFullRowIndex = hasPartialLastRow ? numTotalRows - 1 : numTotalRows; for (int rowNum = firstFullRowIndex; rowNum <= lastFullRowIndex; rowNum++) { request.addRowsToLoadFully(row(rowNum)); } if (hasPartialLastRow) { request.partialLastRow( Maps.immutableEntry(row(numTotalRows), BatchColumnRangeSelection.create(fullColumnRange, 10))); } return request.build(); }
@Test public void testGetRowColumnRangeCellBatchMultipleRows() { putTestDataForSingleTimestamp(); RowColumnRangeIterator values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1), row(0), row(2)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 3, TEST_TIMESTAMP + 1); assertNextElementMatches(values, Cell.create(row(1), column(0)), val(1, 0)); assertNextElementMatches(values, Cell.create(row(1), column(2)), val(1, 2)); assertNextElementMatches(values, TEST_CELL, val(0, 0)); assertNextElementMatches(values, Cell.create(row(0), column(1)), val(0, 1)); assertNextElementMatches(values, Cell.create(row(2), column(1)), val(2, 1)); assertNextElementMatches(values, Cell.create(row(2), column(2)), val(2, 2)); assertFalse(values.hasNext()); }
@Test public void testGetRowColumnRangeCellBatchMultipleRowsAndSmallerBatchHint() { putTestDataForSingleTimestamp(); RowColumnRangeIterator values = keyValueService.getRowsColumnRange(TEST_TABLE, ImmutableList.of(row(1), row(0), row(2)), new ColumnRangeSelection(PtBytes.EMPTY_BYTE_ARRAY, PtBytes.EMPTY_BYTE_ARRAY), 2, TEST_TIMESTAMP + 1); assertNextElementMatches(values, Cell.create(row(1), column(0)), val(1, 0)); assertNextElementMatches(values, Cell.create(row(1), column(2)), val(1, 2)); assertNextElementMatches(values, TEST_CELL, val(0, 0)); assertNextElementMatches(values, Cell.create(row(0), column(1)), val(0, 1)); assertNextElementMatches(values, Cell.create(row(2), column(1)), val(2, 1)); assertNextElementMatches(values, Cell.create(row(2), column(2)), val(2, 2)); assertFalse(values.hasNext()); }