private SystemStreamMetadata constructSystemStreamMetadata( String streamName, Map<SystemStreamPartition, List<IncomingMessageEnvelope>> sspToMessagesForSystem) { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> partitionMetadata = sspToMessagesForSystem .entrySet() .stream() .collect(Collectors.toMap(entry -> entry.getKey().getPartition(), entry -> { String oldestOffset = "0"; String newestOffset = String.valueOf(entry.getValue().size()); String upcomingOffset = String.valueOf(entry.getValue().size() + 1); return new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset); })); return new SystemStreamMetadata(streamName, partitionMetadata); }
private SystemStreamMetadata constructSystemStreamMetadata( String streamName, Map<SystemStreamPartition, List<IncomingMessageEnvelope>> sspToMessagesForSystem) { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> partitionMetadata = sspToMessagesForSystem .entrySet() .stream() .collect(Collectors.toMap(entry -> entry.getKey().getPartition(), entry -> { String oldestOffset = "0"; String newestOffset = String.valueOf(entry.getValue().size()); String upcomingOffset = String.valueOf(entry.getValue().size() + 1); return new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset); })); return new SystemStreamMetadata(streamName, partitionMetadata); }
private SystemStreamMetadata constructSystemStreamMetadata( String streamName, Map<SystemStreamPartition, List<IncomingMessageEnvelope>> sspToMessagesForSystem) { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> partitionMetadata = sspToMessagesForSystem .entrySet() .stream() .collect(Collectors.toMap(entry -> entry.getKey().getPartition(), entry -> { String oldestOffset = "0"; String newestOffset = String.valueOf(entry.getValue().size()); String upcomingOffset = String.valueOf(entry.getValue().size() + 1); return new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset); })); return new SystemStreamMetadata(streamName, partitionMetadata); }
@Override public Map<String, SystemStreamMetadata> getSystemStreamMetadata(Set<String> streamNames) { Map<String, SystemStreamMetadata> metadataMap = new HashMap<>(); Map<String, Set<Partition>> partitionMap = MSG_QUEUES.entrySet() .stream() .filter(entry -> streamNames.contains(entry.getKey().getSystemStream().getStream())) .map(e -> e.getKey()).<Map<String, Set<Partition>>>collect(HashMap::new, (m, ssp) -> { if (m.get(ssp.getStream()) == null) { m.put(ssp.getStream(), new HashSet<>()); } m.get(ssp.getStream()).add(ssp.getPartition()); }, (m1, m2) -> { m2.forEach((k, v) -> { if (m1.get(k) == null) { m1.put(k, v); } else { m1.get(k).addAll(v); } }); }); partitionMap.forEach((k, v) -> { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> partitionMetaMap = v.stream().<Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata>>collect(HashMap::new, (m, p) -> { m.put(p, new SystemStreamMetadata.SystemStreamPartitionMetadata("", "", "")); }, (m1, m2) -> m1.putAll(m2)); metadataMap.put(k, new SystemStreamMetadata(k, partitionMetaMap)); }); return metadataMap; }
/** * Given some SSPs, but missing metadata for one of the streams, getSSPMetadata should delegate to * getSystemStreamMetadata and only fill in results for the SSPs corresponding to streams with metadata. */ @Test public void testGetSSPMetadataMissingStream() { SystemStreamPartition streamPartition0 = new SystemStreamPartition(SYSTEM, STREAM, new Partition(0)); SystemStreamPartition otherStreamPartition0 = new SystemStreamPartition(SYSTEM, OTHER_STREAM, new Partition(0)); SystemAdmin systemAdmin = mock(MySystemAdmin.class); SystemStreamMetadata.SystemStreamPartitionMetadata streamPartition0Metadata = new SystemStreamMetadata.SystemStreamPartitionMetadata("1", "2", "3"); when(systemAdmin.getSystemStreamMetadata(ImmutableSet.of(STREAM, OTHER_STREAM))).thenReturn(ImmutableMap.of( STREAM, new SystemStreamMetadata(STREAM, ImmutableMap.of(new Partition(0), streamPartition0Metadata)))); Set<SystemStreamPartition> ssps = ImmutableSet.of(streamPartition0, otherStreamPartition0); when(systemAdmin.getSSPMetadata(ssps)).thenCallRealMethod(); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> expected = ImmutableMap.of(streamPartition0, streamPartition0Metadata); assertEquals(expected, systemAdmin.getSSPMetadata(ssps)); verify(systemAdmin).getSystemStreamMetadata(ImmutableSet.of(STREAM, OTHER_STREAM)); }
@Override public Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> getSSPMetadata( Set<SystemStreamPartition> ssps) { LOG.info("Fetching SSP metadata for: {}", ssps); List<TopicPartition> topicPartitions = ssps.stream() .map(ssp -> new TopicPartition(ssp.getStream(), ssp.getPartition().getPartitionId())) .collect(Collectors.toList()); OffsetsMaps topicPartitionsMetadata = fetchTopicPartitionsMetadata(topicPartitions); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> sspToSSPMetadata = new HashMap<>(); for (SystemStreamPartition ssp : ssps) { String oldestOffset = topicPartitionsMetadata.getOldestOffsets().get(ssp); String newestOffset = topicPartitionsMetadata.getNewestOffsets().get(ssp); String upcomingOffset = topicPartitionsMetadata.getUpcomingOffsets().get(ssp); sspToSSPMetadata.put(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset)); } return sspToSSPMetadata; }
@Override public Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> getSSPMetadata( Set<SystemStreamPartition> ssps) { LOG.info("Fetching SSP metadata for: {}", ssps); List<TopicPartition> topicPartitions = ssps.stream() .map(ssp -> new TopicPartition(ssp.getStream(), ssp.getPartition().getPartitionId())) .collect(Collectors.toList()); OffsetsMaps topicPartitionsMetadata = fetchTopicPartitionsMetadata(topicPartitions); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> sspToSSPMetadata = new HashMap<>(); for (SystemStreamPartition ssp : ssps) { String oldestOffset = topicPartitionsMetadata.getOldestOffsets().get(ssp); String newestOffset = topicPartitionsMetadata.getNewestOffsets().get(ssp); String upcomingOffset = topicPartitionsMetadata.getUpcomingOffsets().get(ssp); sspToSSPMetadata.put(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset)); } return sspToSSPMetadata; }
/** * Given some SSPs, but missing metadata for one of the SSPs, getSSPMetadata should delegate to * getSystemStreamMetadata and only fill in results for the SSPs that have metadata. */ @Test public void testGetSSPMetadataMissingPartition() { SystemStreamPartition streamPartition0 = new SystemStreamPartition(SYSTEM, STREAM, new Partition(0)); SystemStreamPartition streamPartition1 = new SystemStreamPartition(SYSTEM, STREAM, new Partition(1)); SystemAdmin systemAdmin = mock(MySystemAdmin.class); SystemStreamMetadata.SystemStreamPartitionMetadata streamPartition0Metadata = new SystemStreamMetadata.SystemStreamPartitionMetadata("1", "2", "3"); when(systemAdmin.getSystemStreamMetadata(ImmutableSet.of(STREAM))).thenReturn(ImmutableMap.of( STREAM, new SystemStreamMetadata(STREAM, ImmutableMap.of(new Partition(0), streamPartition0Metadata)))); Set<SystemStreamPartition> ssps = ImmutableSet.of(streamPartition0, streamPartition1); when(systemAdmin.getSSPMetadata(ssps)).thenCallRealMethod(); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> expected = ImmutableMap.of(streamPartition0, streamPartition0Metadata); assertEquals(expected, systemAdmin.getSSPMetadata(ssps)); verify(systemAdmin).getSystemStreamMetadata(ImmutableSet.of(STREAM)); }
private SystemStreamMetadata constructSystemStreamMetadata( String streamName, Map<SystemStreamPartition, List<IncomingMessageEnvelope>> sspToMessagesForSystem) { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> partitionMetadata = sspToMessagesForSystem .entrySet() .stream() .collect(Collectors.toMap(entry -> entry.getKey().getPartition(), entry -> { String oldestOffset = "0"; String newestOffset = String.valueOf(entry.getValue().size()); String upcomingOffset = String.valueOf(entry.getValue().size() + 1); return new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset); })); return new SystemStreamMetadata(streamName, partitionMetadata); }
private SystemStreamMetadata constructSystemStreamMetadata( String streamName, Map<SystemStreamPartition, List<IncomingMessageEnvelope>> sspToMessagesForSystem) { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> partitionMetadata = sspToMessagesForSystem .entrySet() .stream() .collect(Collectors.toMap(entry -> entry.getKey().getPartition(), entry -> { String oldestOffset = "0"; String newestOffset = String.valueOf(entry.getValue().size()); String upcomingOffset = String.valueOf(entry.getValue().size() + 1); return new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset); })); return new SystemStreamMetadata(streamName, partitionMetadata); }
@Test public void testGetSSPMetadataEmptyPartition() { SystemStreamPartition ssp = new SystemStreamPartition(TEST_SYSTEM, VALID_TOPIC, new Partition(0)); SystemStreamPartition otherSSP = new SystemStreamPartition(TEST_SYSTEM, "otherTopic", new Partition(1)); TopicPartition topicPartition = new TopicPartition(VALID_TOPIC, 0); TopicPartition otherTopicPartition = new TopicPartition("otherTopic", 1); when(mockKafkaConsumer.beginningOffsets(ImmutableList.of(topicPartition, otherTopicPartition))).thenReturn( ImmutableMap.of(topicPartition, 1L)); when(mockKafkaConsumer.endOffsets(ImmutableList.of(topicPartition, otherTopicPartition))).thenReturn( ImmutableMap.of(topicPartition, 11L)); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> expected = ImmutableMap.of(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata("1", "10", "11"), otherSSP, new SystemStreamMetadata.SystemStreamPartitionMetadata(null, null, null)); assertEquals(expected, kafkaSystemAdmin.getSSPMetadata(ImmutableSet.of(ssp, otherSSP))); }
@Override public Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> getSSPMetadata( Set<SystemStreamPartition> ssps) { LOG.info("Fetching SSP metadata for: {}", ssps); List<TopicPartition> topicPartitions = ssps.stream() .map(ssp -> new TopicPartition(ssp.getStream(), ssp.getPartition().getPartitionId())) .collect(Collectors.toList()); OffsetsMaps topicPartitionsMetadata = fetchTopicPartitionsMetadata(topicPartitions); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> sspToSSPMetadata = new HashMap<>(); for (SystemStreamPartition ssp : ssps) { String oldestOffset = topicPartitionsMetadata.getOldestOffsets().get(ssp); String newestOffset = topicPartitionsMetadata.getNewestOffsets().get(ssp); String upcomingOffset = topicPartitionsMetadata.getUpcomingOffsets().get(ssp); sspToSSPMetadata.put(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata(oldestOffset, newestOffset, upcomingOffset)); } return sspToSSPMetadata; }
@Test public void testGetSSPMetadata() { SystemStreamPartition ssp = new SystemStreamPartition(TEST_SYSTEM, VALID_TOPIC, new Partition(0)); SystemStreamPartition otherSSP = new SystemStreamPartition(TEST_SYSTEM, "otherTopic", new Partition(1)); TopicPartition topicPartition = new TopicPartition(VALID_TOPIC, 0); TopicPartition otherTopicPartition = new TopicPartition("otherTopic", 1); when(mockKafkaConsumer.beginningOffsets(ImmutableList.of(topicPartition, otherTopicPartition))).thenReturn( ImmutableMap.of(topicPartition, 1L, otherTopicPartition, 2L)); when(mockKafkaConsumer.endOffsets(ImmutableList.of(topicPartition, otherTopicPartition))).thenReturn( ImmutableMap.of(topicPartition, 11L, otherTopicPartition, 12L)); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> expected = ImmutableMap.of(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata("1", "10", "11"), otherSSP, new SystemStreamMetadata.SystemStreamPartitionMetadata("2", "11", "12")); assertEquals(kafkaSystemAdmin.getSSPMetadata(ImmutableSet.of(ssp, otherSSP)), expected); }
@Override public Map<String, SystemStreamMetadata> getSystemStreamMetadata(Set<String> streamNames) { return streamNames.stream() .collect(Collectors.toMap(Function.identity(), streamName -> new SystemStreamMetadata(streamName, Collections.singletonMap(new Partition(0), new SystemStreamMetadata.SystemStreamPartitionMetadata(null, END_OF_STREAM_OFFSET, null))))); }
/** * Create a new {@link SystemAdmin} that returns the provided oldest and newest offsets for its topics */ private SystemAdmin newAdmin(String oldestOffset, String newestOffset) { SystemStreamMetadata checkpointTopicMetadata = new SystemStreamMetadata(CHECKPOINT_TOPIC, ImmutableMap.of(new Partition(0), new SystemStreamPartitionMetadata(oldestOffset, newestOffset, Integer.toString(Integer.parseInt(newestOffset) + 1)))); SystemAdmin mockAdmin = mock(SystemAdmin.class); when(mockAdmin.getSystemStreamMetadata(Collections.singleton(CHECKPOINT_TOPIC))).thenReturn( ImmutableMap.of(CHECKPOINT_TOPIC, checkpointTopicMetadata)); return mockAdmin; }
@Override public Map<String, SystemStreamMetadata> getSystemStreamMetadata(Set<String> streamNames) { Map<String, SystemStreamMetadata> map = new HashMap<>(); for (String stream : streamNames) { Map<Partition, SystemStreamMetadata.SystemStreamPartitionMetadata> m = new HashMap<>(); for (int i = 0; i < streamToPartitions.get(stream); i++) { m.put(new Partition(i), new SystemStreamMetadata.SystemStreamPartitionMetadata("", "", "")); } map.put(stream, new SystemStreamMetadata(stream, m)); } return map; }
@Test public void testGetSSPMetadataEmptyUpcomingOffset() { SystemStreamPartition ssp = new SystemStreamPartition(TEST_SYSTEM, VALID_TOPIC, new Partition(0)); TopicPartition topicPartition = new TopicPartition(VALID_TOPIC, 0); when(mockKafkaConsumer.beginningOffsets(ImmutableList.of(topicPartition))).thenReturn( ImmutableMap.of(topicPartition, 0L)); when(mockKafkaConsumer.endOffsets(ImmutableList.of(topicPartition))).thenReturn(ImmutableMap.of()); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> expected = ImmutableMap.of(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata("0", null, null)); assertEquals(kafkaSystemAdmin.getSSPMetadata(ImmutableSet.of(ssp)), expected); }
private static SystemStreamMetadata.SystemStreamPartitionMetadata sspMetadata(long baseOffset) { return new SystemStreamMetadata.SystemStreamPartitionMetadata(Long.toString(baseOffset), Long.toString(baseOffset * 100), Long.toString(baseOffset * 100 + 1)); } }
@Test public void testGetSSPMetadataZeroUpcomingOffset() { SystemStreamPartition ssp = new SystemStreamPartition(TEST_SYSTEM, VALID_TOPIC, new Partition(0)); TopicPartition topicPartition = new TopicPartition(VALID_TOPIC, 0); when(mockKafkaConsumer.beginningOffsets(ImmutableList.of(topicPartition))).thenReturn( ImmutableMap.of(topicPartition, -1L)); when(mockKafkaConsumer.endOffsets(ImmutableList.of(topicPartition))).thenReturn( ImmutableMap.of(topicPartition, 0L)); Map<SystemStreamPartition, SystemStreamMetadata.SystemStreamPartitionMetadata> expected = ImmutableMap.of(ssp, new SystemStreamMetadata.SystemStreamPartitionMetadata("0", null, "0")); assertEquals(kafkaSystemAdmin.getSSPMetadata(ImmutableSet.of(ssp)), expected); }
@Override public Map<String, SystemStreamMetadata> getSystemStreamMetadata(Set<String> streamNames) { return streamNames.stream() .collect(Collectors.toMap(Function.identity(), streamName -> new SystemStreamMetadata(streamName, Collections.singletonMap(new Partition(0), new SystemStreamMetadata.SystemStreamPartitionMetadata(null, null, null))))); }