@Override public List<JobDag> sort(@NonNull final Queue<JobDag> inputJobDags) { final List<JobDag> result = new ArrayList<>(inputJobDags.size()); final long lastExecutionTimeThresholdMillis = TimeUnit.HOURS.toMillis(this.lastExecutionTimeThresholdHours); inputJobDags.stream().map(dag -> { try { final Optional<Map<String, String>> contents = this.tracker.get(dag.getDataFeedName()); if (contents.isPresent() && contents.get().containsKey(JobDag.LAST_RUNTIME_METADATA_KEY)) { long lastExecutionTime = contents.get().containsKey(JobDag.LAST_EXECUTION_METADATA_KEY) ? Long.parseLong(contents.get().get(JobDag.LAST_EXECUTION_METADATA_KEY)) : Long.MIN_VALUE; if (this.currentTime - lastExecutionTime > lastExecutionTimeThresholdMillis) { return new Tuple2<>(dag, Long.MAX_VALUE); } return new Tuple2<>(dag, Long.valueOf(contents.get().get(JobDag.LAST_RUNTIME_METADATA_KEY))); } else { return new Tuple2<>(dag, Long.MAX_VALUE); } } catch (IOException e) { throw new JobRuntimeException(String.format( "Unable to get metadata for dag %s : ", dag.getDataFeedName()), e); } }).sorted((o1, o2) -> o2._2().compareTo(o1._2())) .forEach(tuple -> result.add(tuple._1())); return result; } }
@Test public void testReadWriteJobManagerMetadata() throws IOException { final Map<String, String> testData1 = new HashMap<String, String>(); testData1.put("metaKey1","metaValue1"); testData1.put("metaKey2","metaValue2"); this.tracker.get().set(TEST_TOPIC_1, testData1 ); final Map<String, String> testData2 = new HashMap<String, String>(); testData2.put("metaKey1","metaValue1"); testData2.put("metaKey2","metaValue2"); this.tracker.get().set(TEST_TOPIC_2, testData2); this.tracker.get().writeJobManagerMetadata(); final FileStatus[] statuses = fileSystem.listStatus(basePath); Assert.assertEquals(1,fileSystem.listStatus(basePath).length); this.tracker = Optional.absent(); this.tracker = Optional.of(new JobManagerMetadataTracker(config)); Assert.assertEquals(this.tracker.get().contains(TEST_TOPIC_1), true); Assert.assertEquals(this.tracker.get().contains(TEST_TOPIC_2), true); Assert.assertEquals(this.tracker.get().get(TEST_TOPIC_2).get().keySet().size(),2); Assert.assertEquals(this.tracker.get().get(TEST_TOPIC_1).get().keySet().size(),2); Assert.assertEquals(this.tracker.get().get(TEST_TOPIC_1).get().get("metaKey1").toString(),"metaValue1"); Assert.assertEquals(this.tracker.get().get(TEST_TOPIC_1).get().get("metaKey1").toString(),"metaValue1"); }
@Test public void testSortOrder() throws IOException { final JobManagerMetadataTracker tracker = mock(JobManagerMetadataTracker.class); when(tracker.get(any())).then(new ConfigurationAnswer()); final IJobExecutionStrategy strategy = new ExecutionTimeJobExecutionStrategy(tracker); final Queue<JobDag> jobDagQueue = new ConcurrentLinkedDeque<>(); final JobDag jobDag1 = mockJobDag("dag1"); jobDagQueue.add(jobDag1); final JobDag jobDag3 = mockJobDag("dag3"); jobDagQueue.add(jobDag3); final JobDag jobDag2 = mockJobDag("dag2"); jobDagQueue.add(jobDag2); final JobDag jobDag4 = mockJobDag("dag4"); jobDagQueue.add(jobDag4); final JobDag jobDag5 = mockJobDag("dag5"); jobDagQueue.add(jobDag5); final List<JobDag> resultQueue = strategy.sort(jobDagQueue); final List<JobDag> expectedQueue = new ArrayList<>(jobDagQueue.size()); // first jobs with no history and/or no success in 6 hours expectedQueue.add(jobDag3); expectedQueue.add(jobDag4); expectedQueue.add(jobDag5); // then sorted in descending order expectedQueue.add(jobDag2); expectedQueue.add(jobDag1); assertListEquals(expectedQueue, resultQueue); }