@Override public Tuple<Collection<ExecutionLineageNode>, Collection<ChannelInstance>> aggregate( Tuple<Collection<ExecutionLineageNode>, Collection<ChannelInstance>> accumulator, ChannelLineageNode node) { accumulator.getField1().add(node.getChannelInstance()); return accumulator; }
public static <K, V> Map<K, V> createMap(Tuple<K, V>... keyValuePairs) { Map<K, V> result = new HashMap<>(keyValuePairs.length); for (Tuple<K, V> keyValuePair : keyValuePairs) { result.put(keyValuePair.getField0(), keyValuePair.getField1()); } return result; }
/** * Searches for a compatible {@link Class} in the given {@link List} (subclass or equal) for the given parameter * {@link Class}. If a match is found, the corresponding {@link Supplier} is used to create a default parameter. * * @param parameterClass {@link Class} of a parameter * @param defaultParameterSuppliers supply default values for various parameter {@link Class}es * @return the first match's {@link Supplier} value or {@code null} if no match was found */ private static Object getDefaultParameter(Class<?> parameterClass, List<Tuple<Class<?>, Supplier<?>>> defaultParameterSuppliers) { for (Tuple<Class<?>, Supplier<?>> defaultParameterSupplier : defaultParameterSuppliers) { if (parameterClass.isAssignableFrom(defaultParameterSupplier.getField0())) { return defaultParameterSupplier.getField1().get(); } } return null; }
/** * @return whether the {@code enumeration} cannot be expanded anymore (i.e., all {@link PlanEnumeration#getServingOutputSlots()} and * {@link PlanEnumeration#getRequestedInputSlots()} are not connected to an adjacent {@link Slot}) */ public boolean deemsComprehensive(PlanEnumeration enumeration) { return enumeration.getServingOutputSlots().stream().allMatch( outputService -> !deemsRelevant(outputService.getField1()) ) && enumeration.getRequestedInputSlots().stream().allMatch( input -> !deemsRelevant(input) ); }
private ConcatenationActivator getOrCreateConcatenationActivator(OutputSlot<?> output, OptimizationContext optimizationCtx) { Tuple<OutputSlot<?>, OptimizationContext> concatKey = createConcatenationKey(output, optimizationCtx); return this.concatenationActivators.computeIfAbsent( concatKey, key -> new ConcatenationActivator(key.getField0(), key.getField1())); }
@Override protected void doExecute() { TaskActivator readyActivator; while ((readyActivator = this.readyActivators.poll()) != null) { // Execute the ExecutionTask. final ExecutionTask task = readyActivator.getTask(); final Tuple<List<ChannelInstance>, PartialExecution> executionResult = this.execute(readyActivator, task); readyActivator.dispose(); // Register the outputChannelInstances (to obtain cardinality measurements and for further stages). final List<ChannelInstance> outputChannelInstances = executionResult.getField0(); outputChannelInstances.stream().filter(Objects::nonNull).forEach(this::store); // Log executions. final PartialExecution partialExecution = executionResult.getField1(); if (partialExecution != null) { this.executionState.add(partialExecution); } // Activate successor ExecutionTasks. this.activateSuccessorTasks(task, outputChannelInstances); outputChannelInstances.stream().filter(Objects::nonNull).forEach(ChannelInstance::disposeIfUnreferenced); } }
channelsToIndicesChange.getField0(), key -> new Bitmask(this.destChannelDescriptorSets.size()) ).orInPlace(channelsToIndicesChange.getField1());
/** * Models eager execution by marking all {@link LazyExecutionLineageNode}s as executed and collecting all marked ones. * * @param inputs the input {@link ChannelInstance}s * @param outputs the output {@link ChannelInstance}s * @param operatorContext the executed {@link OptimizationContext.OperatorContext} * @return the executed {@link OptimizationContext.OperatorContext} and produced {@link ChannelInstance}s */ static Tuple<Collection<ExecutionLineageNode>, Collection<ChannelInstance>> modelEagerExecution( ChannelInstance[] inputs, ChannelInstance[] outputs, OptimizationContext.OperatorContext operatorContext) { final ExecutionLineageNode executionLineageNode = new ExecutionLineageNode(operatorContext); executionLineageNode.addAtomicExecutionFromOperatorContext(); LazyExecutionLineageNode.connectAll(inputs, executionLineageNode, outputs); final Tuple<Collection<ExecutionLineageNode>, Collection<ChannelInstance>> collectors; if (outputs.length == 0) { collectors = executionLineageNode.collectAndMark(); } else { collectors = new Tuple<>(new LinkedList<>(), new LinkedList<>()); for (ChannelInstance output : outputs) { output.getLineage().collectAndMark(collectors.getField0(), collectors.getField1()); } } return collectors; }
Tuple<Operator, OptimizationContext> enumerationKey = EnumerationActivator.createKey(servedOperator, optimizationCtx); EnumerationActivator enumerationActivator = this.enumerationActivators.computeIfAbsent( enumerationKey, key -> new EnumerationActivator(key.getField0(), key.getField1()) );
/** * Groups all {@link #planImplementations} by their {@link ExecutionOperator}s' {@link OutputSlot}s for the * {@code output}. Additionally preserves the very (nested) {@link PlanImplementation} in that {@code output} resides. * * @param output a (possibly top-level) {@link OutputSlot} that should be connected * @return a mapping that represents each element {@link #planImplementations} by a key value pair * {@code (implementing OutputSlots -> (PlanImplementation, nested PlanImplementation)} */ private MultiMap<OutputSlot<?>, Tuple<PlanImplementation, PlanImplementation>> groupImplementationsByOutput(OutputSlot<?> output) { // Sort the PlanEnumerations by their respective open InputSlot or OutputSlot. final MultiMap<OutputSlot<?>, Tuple<PlanImplementation, PlanImplementation>> basePlanGroups = new MultiMap<>(); // Find and validate implementing OutputSlots. for (PlanImplementation basePlanImplementation : this.getPlanImplementations()) { final Collection<Tuple<OutputSlot<?>, PlanImplementation>> execOpOutputsWithContext = basePlanImplementation.findExecutionOperatorOutputWithContext(output); final Tuple<OutputSlot<?>, PlanImplementation> execOpOutputWithCtx = RheemCollections.getSingleOrNull(execOpOutputsWithContext); assert execOpOutputsWithContext != null && !execOpOutputsWithContext.isEmpty() : String.format("No outputs found for %s.", output); basePlanGroups.putSingle( execOpOutputWithCtx.getField0(), new Tuple<>(basePlanImplementation, execOpOutputWithCtx.getField1()) ); } return basePlanGroups; }
/** * Perform downstream activations for the {@code processedEnumeration}. This means activating downstream * {@link EnumerationActivator}s and updating the {@link ConcatenationActivator}s for its {@link OutputSlot}s. * * @return the number of activated {@link EnumerationActivator}s. */ private int activateDownstream(PlanEnumeration processedEnumeration, OptimizationContext optimizationCtx) { // Activate all successive operators for enumeration. int numDownstreamActivations = 0; for (Tuple<OutputSlot<?>, InputSlot<?>> inputService : processedEnumeration.getServingOutputSlots()) { final OutputSlot<?> output = inputService.getField0(); final InputSlot<?> servedInput = inputService.getField1(); if (!deemsRelevant(servedInput)) continue; // Activate downstream EnumerationActivators. if (this.activateDownstreamEnumeration(servedInput, processedEnumeration, optimizationCtx)) { numDownstreamActivations++; } // Update the ConcatenationActivator for this OutputSlot. if (servedInput != null) { final ConcatenationActivator concatenationActivator = this.getOrCreateConcatenationActivator(output, optimizationCtx); concatenationActivator.updateBaseEnumeration(processedEnumeration); } } return numDownstreamActivations; }
producedChannelInstances = results.getField1(); } catch (Exception e) { throw new RheemException(String.format("Executing %s failed.", task), e);
/** * Brings the given {@code task} into execution. */ private void execute(ExecutionTask task, OptimizationContext optimizationContext, ExecutionState executionState) { final GraphChiExecutionOperator graphChiExecutionOperator = (GraphChiExecutionOperator) task.getOperator(); ChannelInstance[] inputChannelInstances = new ChannelInstance[task.getNumInputChannels()]; for (int i = 0; i < inputChannelInstances.length; i++) { inputChannelInstances[i] = executionState.getChannelInstance(task.getInputChannel(i)); } final OptimizationContext.OperatorContext operatorContext = optimizationContext.getOperatorContext(graphChiExecutionOperator); ChannelInstance[] outputChannelInstances = new ChannelInstance[task.getNumOuputChannels()]; for (int i = 0; i < outputChannelInstances.length; i++) { outputChannelInstances[i] = task.getOutputChannel(i).createInstance(this, operatorContext, i); } long startTime = System.currentTimeMillis(); final Tuple<Collection<ExecutionLineageNode>, Collection<ChannelInstance>> results = graphChiExecutionOperator.execute(inputChannelInstances, outputChannelInstances, operatorContext); long endTime = System.currentTimeMillis(); final Collection<ExecutionLineageNode> executionLineageNodes = results.getField0(); final Collection<ChannelInstance> producedChannelInstances = results.getField1(); for (ChannelInstance outputChannelInstance : outputChannelInstances) { if (outputChannelInstance != null) { executionState.register(outputChannelInstance); } } final PartialExecution partialExecution = this.createPartialExecution(executionLineageNodes, endTime - startTime); executionState.add(partialExecution); this.registerMeasuredCardinalities(producedChannelInstances); }
producedChannelInstances = results.getField1(); } catch (Exception e) { throw new RheemException(String.format("Executing %s failed.", task), e);