taskInfo.getMaxNumberOfParallelSubtasks(), taskInfo.getNumberOfParallelSubtasks(), taskInfo.getIndexOfThisSubtask()); operatorIdentifierText, keySerializer, taskInfo.getMaxNumberOfParallelSubtasks(), keyGroupRange, environment.getTaskKvStateRegistry(),
@Override public String getTaskNameWithSubtasks() { return taskInfo.getTaskNameWithSubtasks(); }
@Override public String getTaskName() { return taskInfo.getTaskName(); }
operatorID, operatorClassName, taskInfo.getIndexOfThisSubtask(), taskInfo.getNumberOfParallelSubtasks());
private <OUT> List<OUT> executeDataSource(GenericDataSourceBase<?, ?> source, int superStep) throws Exception { @SuppressWarnings("unchecked") GenericDataSourceBase<OUT, ?> typedSource = (GenericDataSourceBase<OUT, ?>) source; // build the runtime context and compute broadcast variables, if necessary TaskInfo taskInfo = new TaskInfo(typedSource.getName(), 1, 0, 1, 0); RuntimeUDFContext ctx; MetricGroup metrics = new UnregisteredMetricsGroup(); if (RichInputFormat.class.isAssignableFrom(typedSource.getUserCodeWrapper().getUserCodeClass())) { ctx = superStep == 0 ? new RuntimeUDFContext(taskInfo, userCodeClassLoader, executionConfig, cachedFiles, accumulators, metrics) : new IterationRuntimeUDFContext(taskInfo, userCodeClassLoader, executionConfig, cachedFiles, accumulators, metrics); } else { ctx = null; } return typedSource.executeOnCollections(ctx, executionConfig); }
@Override public int getIndexOfThisSubtask() { return taskInfo.getIndexOfThisSubtask(); }
when(mockTaskInfo.getTaskNameWithSubtasks()).thenReturn("foobar"); when(mockTaskInfo.getIndexOfThisSubtask()).thenReturn(0); Environment mockEnvironment = new MockEnvironmentBuilder().build();
@Override public int getNumberOfParallelSubtasks() { return taskInfo.getNumberOfParallelSubtasks(); }
new TaskExecutionStatus( currentTask.getExecutionState(), currentTask.getTaskInfo().getAttemptNumber(), currentTask.getCreateTimestamp(), currentTask.getJobVertexId(), currentTask.getExecutionId(), currentTask.getTaskInfo().getIndexOfThisSubtask(), resultPartitionIDs, resultPartitionsConsumable,
@Override public int getMaxNumberOfParallelSubtasks() { return taskInfo.getMaxNumberOfParallelSubtasks(); }
@Override public int getAttemptNumber() { return taskInfo.getAttemptNumber(); }
/** * Utility function that composes a string for logging purposes. The string includes the given message, * the given name of the task and the index in its subtask group as well as the number of instances * that exist in its subtask group. * * @param message The main message for the log. * @param taskName The name of the task. * @param parent The task that contains the code producing the message. * * @return The string for logging. */ public static String constructLogString(String message, String taskName, AbstractInvokable parent) { return message + ": " + taskName + " (" + (parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) + '/' + parent.getEnvironment().getTaskInfo().getNumberOfParallelSubtasks() + ')'; }
private <IN, OUT> List<OUT> executeUnaryOperator(SingleInputOperator<?, ?, ?> operator, int superStep) throws Exception { Operator<?> inputOp = operator.getInput(); if (inputOp == null) { throw new InvalidProgramException("The unary operation " + operator.getName() + " has no input."); } @SuppressWarnings("unchecked") List<IN> inputData = (List<IN>) execute(inputOp, superStep); @SuppressWarnings("unchecked") SingleInputOperator<IN, OUT, ?> typedOp = (SingleInputOperator<IN, OUT, ?>) operator; // build the runtime context and compute broadcast variables, if necessary TaskInfo taskInfo = new TaskInfo(typedOp.getName(), 1, 0, 1, 0); RuntimeUDFContext ctx; MetricGroup metrics = new UnregisteredMetricsGroup(); if (RichFunction.class.isAssignableFrom(typedOp.getUserCodeWrapper().getUserCodeClass())) { ctx = superStep == 0 ? new RuntimeUDFContext(taskInfo, userCodeClassLoader, executionConfig, cachedFiles, accumulators, metrics) : new IterationRuntimeUDFContext(taskInfo, userCodeClassLoader, executionConfig, cachedFiles, accumulators, metrics); for (Map.Entry<String, Operator<?>> bcInputs : operator.getBroadcastInputs().entrySet()) { List<?> bcData = execute(bcInputs.getValue()); ctx.setBroadcastVariable(bcInputs.getKey(), bcData); } } else { ctx = null; } return typedOp.executeOnCollections(inputData, ctx, executionConfig); }
@Override public void init() throws Exception { final String iterationId = getConfiguration().getIterationId(); if (iterationId == null || iterationId.length() == 0) { throw new Exception("Missing iteration ID in the task configuration"); } final String brokerID = StreamIterationHead.createBrokerIdString(getEnvironment().getJobID(), iterationId, getEnvironment().getTaskInfo().getIndexOfThisSubtask()); final long iterationWaitTime = getConfiguration().getIterationWaitTime(); LOG.info("Iteration tail {} trying to acquire feedback queue under {}", getName(), brokerID); @SuppressWarnings("unchecked") BlockingQueue<StreamRecord<IN>> dataChannel = (BlockingQueue<StreamRecord<IN>>) BlockingQueueBroker.INSTANCE.get(brokerID); LOG.info("Iteration tail {} acquired feedback queue {}", getName(), brokerID); this.headOperator = new RecordPusher<>(); this.headOperator.setup(this, getConfiguration(), new IterationTailOutput<>(dataChannel, iterationWaitTime)); // call super.init() last because that needs this.headOperator to be set up super.init(); }
@Override public int getNumberOfParallelSubtasks() { return taskInfo.getNumberOfParallelSubtasks(); }
@Override public int getMaxNumberOfParallelSubtasks() { return taskInfo.getMaxNumberOfParallelSubtasks(); }
@Override public int getAttemptNumber() { return taskInfo.getAttemptNumber(); }
int numKeyGroups = getEnvironment().getTaskInfo().getMaxNumberOfParallelSubtasks(); int numSubtasks = getEnvironment().getTaskInfo().getNumberOfParallelSubtasks(); int subtaskIndex = getEnvironment().getTaskInfo().getIndexOfThisSubtask();
/** * Utility function that composes a string for logging purposes. The string includes the given message, * the given name of the task and the index in its subtask group as well as the number of instances * that exist in its subtask group. * * @param message The main message for the log. * @param taskName The name of the task. * @param parent The task that contains the code producing the message. * * @return The string for logging. */ public static String constructLogString(String message, String taskName, AbstractInvokable parent) { return message + ": " + taskName + " (" + (parent.getEnvironment().getTaskInfo().getIndexOfThisSubtask() + 1) + '/' + parent.getEnvironment().getTaskInfo().getNumberOfParallelSubtasks() + ')'; }
/** * Gets the name of the task, in the form "taskname (2/5)". * @return The name of the task. */ public String getName() { return getEnvironment().getTaskInfo().getTaskNameWithSubtasks(); }