private byte[][] readDataBlocks() throws IOException { RAFLock lock = parent.lockUnderlying(); try { byte[][] data = new byte[dataBlockCount][]; for(int i=0;i<dataBlockCount;i++) { data[i] = segments[i].readDataBlock(blockNumbers[i]); if(DEBUG_ENCODE) { ClientCHK key = segments[i].encodeBlock(data[i]).getClientKey(); segments[i].setKey(blockNumbers[i], key); } } return data; } finally { lock.unlock(); } }
protected int getMaxBlockNumber() { // Ignore cross-segment: We either send all blocks, if the segment has been encoded, or // only the data blocks, if it hasn't (even if the cross-segment blocks have been encoded). if(segment.hasEncoded()) return segment.totalBlockCount; else return segment.dataBlockCount; }
private boolean allFinishedEncoding() { for (SplitFileInserterSegmentStorage segment : segments) { if (!segment.isFinishedEncoding()) return false; } return true; }
/** Generate keys for each block and record them. * @throws IOException */ private void generateKeys(byte[][] dataBlocks, int offset) throws IOException { for(int i=0;i<dataBlocks.length;i++) { setKey(i + offset, encodeBlock(dataBlocks[i]).getClientKey()); } }
void setKey(int blockNumber, ClientCHK key) throws IOException { if(logMINOR) Logger.minor(this, "Setting key "+key+" for block "+blockNumber+" on "+this, new Exception("debug")); try { ClientCHK oldKey = readKey(blockNumber); if(!oldKey.equals(key)) throw new IOException("Key for block has changed! Data corruption or bugs in SplitFileInserter code"); } catch (MissingKeyException e) { // Ok. writeKey(blockNumber, key); } // Must be called either way as we don't regenerate blocksHaveKeys on startup. setHasKey(blockNumber); }
BlockInsert chosen = segment.chooseBlock(); assertTrue(chosen != null); keys.addRunningInsert(chosen); assertFalse(chosenBlocks[chosen.blockNumber]); chosenBlocks[chosen.blockNumber] = true; segment.onFailure(chosen.blockNumber, new InsertException(InsertExceptionMode.ROUTE_NOT_FOUND)); BlockInsert chosen = segment.chooseBlock(); assertTrue(chosen == null); for(int i=0;i<segment.totalBlockCount;i++) chosen = segment.chooseBlock(); keys.addRunningInsert(chosen); assertTrue(chosen != null); assertFalse(chosenBlocks[chosen.blockNumber]); chosenBlocks[chosen.blockNumber] = true; segment.onInsertedBlock(chosen.blockNumber, segment.encodeBlock(chosen.blockNumber).getClientKey());
public void testSmallSplitfileCompletion() throws IOException, InsertException, MissingKeyException { Random r = new Random(12121); long size = 65536; // Exact multiple, so no last block LockableRandomAccessBuffer data = generateData(r, size); HashResult[] hashes = getHashes(data); MyCallback cb = new MyCallback(); InsertContext context = baseContext.clone(); KeysFetchingLocally keys = new MyKeysFetchingLocally(); SplitFileInserterStorage storage = new SplitFileInserterStorage(data, size, cb, null, new ClientMetadata(), false, null, smallRAFFactory, false, context, cryptoAlgorithm, cryptoKey, null, hashes, smallBucketFactory, checker, r, memoryLimitedJobRunner, jobRunner, ticker, keys, false, 0, 0, 0, 0); storage.start(); cb.waitForFinishedEncode(); assertEquals(storage.segments.length, 1); SplitFileInserterSegmentStorage segment = storage.segments[0]; assertEquals(segment.dataBlockCount, 2); assertEquals(segment.checkBlockCount, 3); assertEquals(segment.crossCheckBlockCount, 0); assertEquals(storage.getStatus(), Status.ENCODED); for(int i=0;i<segment.totalBlockCount;i++) { segment.onInsertedBlock(i, segment.encodeBlock(i).getClientKey()); } cb.waitForSucceededInsert(); assertEquals(storage.getStatus(), Status.SUCCEEDED); }
BlockInsert chosen = segment.chooseBlock(); assertTrue(chosen != null); keys.addInsert(chosen); assertFalse(chosenBlocks[chosen.blockNumber]); chosenBlocks[chosen.blockNumber] = true; segment.setKey(chosen.blockNumber, segment.encodeBlock(chosen.blockNumber).getClientKey()); segment.onFailure(chosen.blockNumber, new InsertException(InsertExceptionMode.ROUTE_NOT_FOUND)); BlockInsert chosen = segment.chooseBlock(); assertTrue(chosen != null); keys.addInsert(chosen); assertFalse(chosenBlocks[chosen.blockNumber]); chosenBlocks[chosen.blockNumber] = true; segment.onFailure(chosen.blockNumber, new InsertException(InsertExceptionMode.ROUTE_NOT_FOUND));
assertEquals(segment.crossCheckBlockCount, 0); assertEquals(storage.getStatus(), Status.ENCODED); segment.setKey(0, segment.encodeBlock(0).getClientKey()); segment.onFailure(0, new InsertException(InsertExceptionMode.ROUTE_NOT_FOUND)); assertEquals(storage.getStatus(), Status.ENCODED); segment.onFailure(0, new InsertException(InsertExceptionMode.ROUTE_NOT_FOUND)); assertEquals(storage.getStatus(), Status.ENCODED); segment.onFailure(0, new InsertException(InsertExceptionMode.REJECTED_OVERLOAD));
@Override public void onFailure(LowLevelPutException e, SendableRequestItem keyNum, ClientContext context) { InsertException e1 = InsertException.constructFrom(e); if(keyNum == null) { storage.fail(e1); } else { BlockInsert block = (BlockInsert) keyNum; block.segment.onFailure(block.blockNumber, e1); } }
/** Add a random block that has not been added already or decoded already. * @throws IOException */ private boolean addRandomBlock(SplitFileInserterStorage storage, SplitFileFetcherStorage fetcherStorage, Random random) throws IOException { int segCount = storage.segments.length; boolean[] exhaustedSegments = new boolean[segCount]; for(int i=0;i<segCount;i++) { while(true) { int segNo = random.nextInt(segCount); if(exhaustedSegments[segNo]) continue; SplitFileFetcherSegmentStorage segment = fetcherStorage.segments[segNo]; if(segment.isDecodingOrFinished()) { exhaustedSegments[segNo] = true; break; } while(true) { int blockNo = random.nextInt(segment.totalBlocks()); if(segment.hasBlock(blockNo)) { continue; } ClientCHKBlock block = storage.segments[segNo].encodeBlock(blockNo); boolean success = segment.onGotKey(block.getClientKey().getNodeCHK(), block.getBlock()); assertTrue(success); return true; } } } return false; }
public ClientCHKBlock encodeBlock(int blockNo) throws IOException { if(parent.isFinishing()) { throw new IOException("Already finishing reading block "+blockNo+" for "+this+" for "+parent); } synchronized(this) { if(this.blockChooser.hasSucceeded(blockNo)) { Logger.error(this, "Already inserted block "+blockNo+" for "+this+" for "+parent); throw new IOException("Already inserted block "+blockNo+" for "+this+" for "+parent); } } byte[] buf = readBlock(blockNo); return encodeBlock(buf); }
/** Encode the Metadata. The caller must ensure that all segments have encoded keys first. * @throws MissingKeyException This indicates disk corruption or a bug (e.g. not all segments * had encoded keys). Since we don't checksum the blocks, there isn't much point in trying to * recover from losing a key; but at least we can detect that there was a problem. * * (Package-visible for unit tests) */ Metadata encodeMetadata() throws IOException, MissingKeyException { ClientCHK[] dataKeys = new ClientCHK[totalDataBlocks + crossCheckBlocks * segments.length]; ClientCHK[] checkKeys = new ClientCHK[totalCheckBlocks]; int dataPtr = 0; int checkPtr = 0; for(int segNo = 0; segNo < segments.length; segNo++) { SplitFileInserterSegmentStorage segment = segments[segNo]; for(int i=0;i<segment.dataBlockCount+segment.crossCheckBlockCount;i++) { dataKeys[dataPtr++] = segment.readKey(i); } for(int i=0;i<segment.checkBlockCount;i++) { checkKeys[checkPtr++] = segment.readKey(i+segment.dataBlockCount+segment.crossCheckBlockCount); } } assert(dataPtr == dataKeys.length); assert(checkPtr == checkKeys.length); return new Metadata(splitfileType, dataKeys, checkKeys, segmentSize, checkSegmentSize, deductBlocksFromSegments, clientMetadata, dataLength, archiveType, compressionCodec, decompressedLength, isMetadata, hashes, hashThisLayerOnly, origDataSize, origCompressedDataSize, topRequiredBlocks, topTotalBlocks, topDontCompress, cmode, splitfileCryptoAlgorithm, splitfileCryptoKey, specifySplitfileKeyInMetadata, crossCheckBlocks); }
/** Called when a block insert succeeds */ public void onInsertedBlock(int blockNo, ClientCHK key) { try { if(parent.hasFinished()) return; this.setKey(blockNo, key); if(blockChooser.onSuccess(blockNo)) parent.callback.onInsertedBlock(); lazyWriteMetadata(); } catch (IOException e) { if(parent.hasFinished()) return; // Race condition possible as this is a callback parent.failOnDiskError(e); } }
if(keyLength < SplitFileInserterSegmentStorage.getKeyLength(this)) throw new StorageFormatException("Invalid key length "+keyLength+" should be at least "+ SplitFileInserterSegmentStorage.getKeyLength(this)); int compatMode = dis.readInt(); if(compatMode < 0 || compatMode > CompatibilityMode.values().length) long blocks = 0; for(int i=0;i<segmentCount;i++) { segments[i] = new SplitFileInserterSegmentStorage(this, dis, i, keyLength, splitfileCryptoAlgorithm, splitfileCryptoKey, random, maxRetries, consecutiveRNFsCountAsSuccess, keysFetching); underlyingOffsetDataSegments[i] = blocks * CHKBlock.DATA_LENGTH; dis.close(); for(SplitFileInserterSegmentStorage segment : segments) { segment.readStatus();
blockChooser.consecutiveRNFsCountAsSuccess > 0) { try { readKey(blockNo); blockChooser.onRNF(blockNo); parent.clearCooldown(); parent.failTooManyRetriesInBlock(); } else { if(blockChooser.maxRetries >= 0) lazyWriteMetadata(); parent.clearCooldown();
public void testCancel() throws IOException, InsertException, MissingKeyException { Random r = new Random(12124); long size = 32768*6; BarrierRandomAccessBuffer data = new BarrierRandomAccessBuffer(generateData(r, size)); HashResult[] hashes = getHashes(data); data.pause(); MyCallback cb = new MyCallback(); InsertContext context = baseContext.clone(); context.earlyEncode = true; KeysFetchingLocally keys = new MyKeysFetchingLocally(); SplitFileInserterStorage storage = new SplitFileInserterStorage(data, size, cb, null, new ClientMetadata(), false, null, smallRAFFactory, false, context, cryptoAlgorithm, cryptoKey, null, hashes, smallBucketFactory, checker, r, memoryLimitedJobRunner, jobRunner, ticker, keys, false, 0, 0, 0, 0); storage.start(); assertEquals(storage.getStatus(), Status.STARTED); assertEquals(storage.segments.length, 1); SplitFileInserterSegmentStorage segment = storage.segments[0]; segment.onFailure(0, new InsertException(InsertExceptionMode.INTERNAL_ERROR)); data.proceed(); // Now it will complete encoding, and then report in, and then fail. try { cb.waitForFinishedEncode(); assertFalse(true); // Should have failed. } catch (InsertException e) { assertFalse(segment.isEncoding()); assertEquals(storage.getStatus(), Status.FAILED); } }
@Override public void onSuccess(SendableRequestItem keyNum, ClientKey key, ClientContext context) { BlockInsert block = (BlockInsert) keyNum; block.segment.onInsertedBlock(block.blockNumber, (ClientCHK) key); }
@Override public void onEncode(SendableRequestItem token, ClientKey key, ClientContext context) { BlockInsert block = (BlockInsert) token; // Should already be set. This is a sanity check. try { if(storage.hasFinished()) return; block.segment.setKey(block.blockNumber, (ClientCHK) key); } catch (IOException e) { if(storage.hasFinished()) return; // Race condition possible as this is a callback storage.failOnDiskError(e); } }
/** Choose a block to insert. * FIXME make SplitFileInserterSender per-segment, eliminate a lot of unnecessary complexity. */ public BlockInsert chooseBlock() { // FIXME this should probably use SimpleBlockChooser and hence use lowest-retry-count from // each segment? // Less important for inserts than for requests though... synchronized(cooldownLock) { synchronized(this) { if (status == Status.FAILED || status == Status.SUCCEEDED || status == Status.GENERATING_METADATA || failing != null) { return null; } } // Generally segments are fairly well balanced, so we can usually pick a random segment // then a random key from it. randomSegmentIterator.reset(random); while (randomSegmentIterator.hasNext()) { SplitFileInserterSegmentStorage segment = randomSegmentIterator.next(); BlockInsert ret = segment.chooseBlock(); if (ret != null) { noBlocksToSend = false; return ret; } } noBlocksToSend = true; return null; } }