"Invalid account record when reading accountMap in ZNRecord because idKey=null"); account = Account.fromJson(accountJson); if (account.getId() != Short.valueOf(idKey)) { accountServiceMetrics.remoteDataCorruptionErrorCount.inc(); throw new IllegalStateException( "Invalid account record when reading accountMap in ZNRecord because idKey and accountId do not match. idKey=" + idKey + " accountId=" + account.getId()); if (idToAccountMap.containsKey(account.getId()) || nameToAccountMap.containsKey(account.getName())) { throw new IllegalStateException( "Duplicate account id or name exists. id=" + account.getId() + " name=" + account.getName()); idToAccountMap.put(account.getId(), account); nameToAccountMap.put(account.getName(), account);
for (Account account : accountsToSet) { Account accountInMap = accountInfoMap.getAccountById(account.getId()); if (accountInMap != null && account.getSnapshotVersion() != accountInMap.getSnapshotVersion()) { logger.error( "Account to update (accountId={} accountName={}) has an unexpected snapshot version in zk (expected={}, encountered={})", account.getId(), account.getName(), account.getSnapshotVersion(), accountInMap.getSnapshotVersion()); return true; Account potentialConflict = accountInfoMap.getAccountByName(account.getName()); if (potentialConflict != null && potentialConflict.getId() != account.getId()) { logger.error( "Account to update (accountId={} accountName={}) conflicts with an existing record (accountId={} accountName={})", account.getId(), account.getName(), potentialConflict.getId(), potentialConflict.getName()); return true;
for (Account account : accountsToUpdate) { try { accountMap.put(String.valueOf(account.getId()), account.toJson(true).toString()); } catch (Exception e) { String message = "Updating accounts failed because unexpected exception occurred when updating accountId=" + account.getId() + " accountName=" + account.getName();
/** * Checks if there are duplicate accountId or accountName in the given collection of {@link Account}s. * * @param accounts A collection of {@link Account}s to check duplication. * @return {@code true} if there are duplicated accounts in id or name in the given collection of accounts, * {@code false} otherwise. */ static boolean hasDuplicateAccountIdOrName(Collection<Account> accounts) { if (accounts == null) { return false; } Set<Short> idSet = new HashSet<>(); Set<String> nameSet = new HashSet<>(); boolean res = false; for (Account account : accounts) { if (!idSet.add(account.getId()) || !nameSet.add(account.getName())) { logger.debug("Accounts to update have conflicting id or name. Conflicting accountId={} accountName={}", account.getId(), account.getName()); res = true; break; } } return res; } }
/** * Asserts that an {@link Account} exists in the {@link AccountService}. * @param account The {@link Account} to assert existence. * @param accountService The {@link AccountService} to assert {@link Account} existence. */ static void assertAccountInAccountService(Account account, AccountService accountService) { Account accountFoundById = accountService.getAccountById(account.getId()); Account accountFoundByName = accountService.getAccountByName(account.getName()); assertEquals("Account got by name from accountService does not match account to assert.", account, accountFoundByName); assertEquals("Account got by id from accountService does not match the account to assert", account, accountFoundById); assertEquals("The number of containers in the account is wrong.", accountFoundById.getAllContainers().size(), account.getAllContainers().size()); for (Container container : account.getAllContainers()) { assertContainerInAccountService(container, accountService); } }
/** * Tests reading {@link ZNRecord} from {@link HelixPropertyStore}, where the {@link ZNRecord} has a map field * ("key": someValidAccountMap), but ({@link HelixAccountService#ACCOUNT_METADATA_MAP_KEY}: someValidMap) * is missing. This is a good {@link ZNRecord} format that should NOT fail fetch or update. * @throws Exception Any unexpected exception. */ @Test public void testReadBadZNRecordCase3() throws Exception { Map<String, String> mapValue = new HashMap<>(); mapValue.put(String.valueOf(refAccount.getId()), refAccount.toJson(true).toString()); ZNRecord zNRecord = makeZNRecordWithMapField(null, "key", mapValue); updateAndWriteZNRecord(zNRecord, true); }
for (Account account : accountService.getAllAccounts()) { AccountBuilder accountBuilder = new AccountBuilder(account); accountBuilder.name(account.getName() + "-extra"); accountBuilder.status( account.getStatus().equals(AccountStatus.ACTIVE) ? AccountStatus.INACTIVE : AccountStatus.ACTIVE); accountsToUpdate.add(accountBuilder.build()); for (Account account : accountService.getAllAccounts()) { accountBuilder = new AccountBuilder(account); for (Container container : account.getAllContainers()) { ContainerBuilder containerBuilder = new ContainerBuilder(container); containerBuilder.setId((short) (-1 * (container.getId())));
for (Account account : accountService.getAllAccounts()) { AccountBuilder accountBuilder = new AccountBuilder(account); accountBuilder.name(account.getName() + "-extra"); accountsToUpdate.add(accountBuilder.build()); Account updatedAccount = new AccountBuilder(account).name(account.getName() + "-extra").build(); accountsToUpdate = new HashSet<>(Collections.singleton(updatedAccount)); updateAccountsAndAssertAccountExistence(accountsToUpdate, 1 + NUM_REF_ACCOUNT, true);
/** * Assert that a {@link Container} exists in the {@link AccountService}. * @param container The {@link Container} to assert. * @param accountService The {@link AccountService} to assert {@link Container} existence. */ static void assertContainerInAccountService(Container container, AccountService accountService) { Container containerFoundById = accountService.getAccountById(container.getParentAccountId()).getContainerById(container.getId()); Container containerFoundByName = accountService.getAccountById(container.getParentAccountId()).getContainerByName(container.getName()); assertEquals("Container got by id from accountService/account does not match container got by name.", containerFoundById, containerFoundByName); assertEquals("Container got by id from accountService/account does not match the container to assert", containerFoundById, container); }
/** * Tests reading {@link ZNRecord} from {@link HelixPropertyStore}, where the {@link ZNRecord} has a map field * ({@link HelixAccountService#ACCOUNT_METADATA_MAP_KEY}: accountMap), and accountMap contains * ("accountId": badAccountJsonString). This is a NOT good {@link ZNRecord} format that should fail fetch or update. * @throws Exception Any unexpected exception. */ @Test public void testReadBadZNRecordCase5() throws Exception { Map<String, String> mapValue = new HashMap<>(); mapValue.put(String.valueOf(refAccount.getId()), BAD_ACCOUNT_METADATA_STRING); ZNRecord zNRecord = makeZNRecordWithMapField(null, ACCOUNT_METADATA_MAP_KEY, mapValue); updateAndWriteZNRecord(zNRecord, false); }
account.getAllContainers().size()); idToRefAccountMap.put(accountId, account); idToRefContainerMap.put(accountId, idToContainers);
/** * Tests reading {@link ZNRecord} from {@link HelixPropertyStore}, where the {@link ZNRecord} has a map field * ({@link HelixAccountService#ACCOUNT_METADATA_MAP_KEY}: accountMap), and accountMap contains * ("accountId": accountJsonStr) that does not match. This is a NOT good {@link ZNRecord} format that should * fail fetch or update. * @throws Exception Any unexpected exception. */ @Test public void testReadBadZNRecordCase4() throws Exception { Map<String, String> mapValue = new HashMap<>(); mapValue.put("-1", refAccount.toJson(true).toString()); ZNRecord zNRecord = makeZNRecordWithMapField(null, ACCOUNT_METADATA_MAP_KEY, mapValue); updateAndWriteZNRecord(zNRecord, false); }
/** * Check that the provided backup file matches the data in the corresponding serialized accounts. * @param expectedAccounts the expected {@link Account}s. * @param backupPath the {@link Path} to the backup file. * @throws JSONException */ private void checkBackupFile(Collection<Account> expectedAccounts, Path backupPath) throws JSONException, IOException { try (BufferedReader reader = Files.newBufferedReader(backupPath)) { JSONArray accountArray = new JSONArray(new JSONTokener(reader)); int arrayLength = accountArray.length(); assertEquals("unexpected array size", expectedAccounts.size(), arrayLength); Set<Account> expectedAccountSet = new HashSet<>(expectedAccounts); for (int i = 0; i < arrayLength; i++) { JSONObject accountJson = accountArray.getJSONObject(i); Account account = Account.fromJson(accountJson); assertTrue("unexpected account in array: " + accountJson.toString(), expectedAccountSet.contains(account)); } } }
/** * Test updating an account with a conflicting expected snapshot version. */ @Test public void testConflictingSnapshotVersionUpdate() throws Exception { accountService = mockHelixAccountServiceFactory.getAccountService(); // write two accounts (1, "a") and (2, "b") writeAccountsForConflictTest(); Account expectedAccount = accountService.getAccountById((short) 1); int currentSnapshotVersion = expectedAccount.getSnapshotVersion(); for (int snapshotVersionOffset : new int[]{-2, -1, 1}) { int snapshotVersionToUse = currentSnapshotVersion + snapshotVersionOffset; Collection<Account> conflictAccounts = Collections.singleton( new AccountBuilder((short) 1, "c", AccountStatus.INACTIVE).snapshotVersion(snapshotVersionToUse).build()); assertFalse("Wrong return value from update operation.", accountService.updateAccounts(conflictAccounts)); assertEquals("Wrong account number in HelixAccountService", 2, accountService.getAllAccounts().size()); Account account = accountService.getAccountById((short) 1); assertEquals("Account should not have been updated", expectedAccount, account); assertEquals("Snapshot version should not have been updated", currentSnapshotVersion, account.getSnapshotVersion()); } Collection<Account> validAccounts = Collections.singleton( new AccountBuilder((short) 1, "c", AccountStatus.INACTIVE).snapshotVersion(currentSnapshotVersion).build()); updateAccountsAndAssertAccountExistence(validAccounts, 2, true); }
/** * Tests reading {@link ZNRecord} from {@link HelixPropertyStore}, where the {@link ZNRecord} has an invalid account * record and a valid account record. This is a NOT good {@link ZNRecord} format and it should fail fetch or update * operations, with none of the record should be read. * @throws Exception Any unexpected exception. */ @Test public void testReadBadZNRecordCase6() throws Exception { ZNRecord zNRecord = new ZNRecord(String.valueOf(System.currentTimeMillis())); Map<String, String> accountMap = new HashMap<>(); accountMap.put(String.valueOf(refAccount.getId()), refAccount.toJson(true).toString()); accountMap.put(String.valueOf(refAccount.getId() + 1), BAD_ACCOUNT_METADATA_STRING); zNRecord.setMapField(ACCOUNT_METADATA_MAP_KEY, accountMap); updateAndWriteZNRecord(zNRecord, false); }
/** * Tests updating a {@link Account}, which has the same id as an existing record, but the name conflicting with * another existing record. The update operation will fail. This test corresponds to case E specified in the JavaDoc * of {@link AccountService}. * @throws Exception Any unexpected exception. */ @Test public void testConflictingUpdateCaseE() throws Exception { accountService = mockHelixAccountServiceFactory.getAccountService(); // write two accounts (1, "a") and (2, "b") writeAccountsForConflictTest(); Collection<Account> conflictAccounts = Collections.singleton((new AccountBuilder((short) 1, "b", AccountStatus.INACTIVE).build())); assertFalse("Wrong return value from update operation.", accountService.updateAccounts(conflictAccounts)); assertEquals("Wrong account number in HelixAccountService", 2, accountService.getAllAccounts().size()); assertEquals("Wrong account name got from HelixAccountService", "a", accountService.getAccountById((short) 1).getName()); }
/** * Pre-populates a collection of {@link Account}s to the underlying {@link org.apache.helix.store.HelixPropertyStore} * using {@link com.github.ambry.clustermap.HelixStoreOperator} (not through the {@link HelixAccountService}). This method * does not check any conflict among the {@link Account}s to write. * @throws Exception Any unexpected exception. */ private void writeAccountsToHelixPropertyStore(Collection<Account> accounts, boolean shouldNotify) throws Exception { HelixStoreOperator storeOperator = new HelixStoreOperator(mockHelixAccountServiceFactory.getHelixStore(ZK_CONNECT_STRING, storeConfig)); ZNRecord zNRecord = new ZNRecord(String.valueOf(System.currentTimeMillis())); Map<String, String> accountMap = new HashMap<>(); for (Account account : accounts) { accountMap.put(String.valueOf(account.getId()), account.toJson(true).toString()); } zNRecord.setMapField(ACCOUNT_METADATA_MAP_KEY, accountMap); // Write account metadata into HelixPropertyStore. storeOperator.write(HelixAccountService.FULL_ACCOUNT_METADATA_PATH, zNRecord); if (shouldNotify) { notifier.publish(ACCOUNT_METADATA_CHANGE_TOPIC, FULL_ACCOUNT_METADATA_CHANGE_MESSAGE); } }