SideInputId.newBuilder().setTransformId(transformId).setLocalName(sideInputId).build()); checkArgument(collectionNode != null, "No side input for %s/%s", transformId, sideInputId);
String collectionId = components .getTransformsOrThrow(sideInput.getKey().getTransformId()) .getInputsOrThrow(sideInput.getKey().getLocalName()); DataStream<Object> sideInputStream = context.getDataStreamOrThrow(collectionId); TypeInformation<Object> tpe = sideInputStream.getType(); final int intTag = tagToIntMapping.get(tag); RunnerApi.PTransform pTransform = components.getTransformsOrThrow(sideInput.getKey().getTransformId()); String collectionId = pTransform.getInputsOrThrow(sideInput.getKey().getLocalName()); DataStream<WindowedValue<?>> sideInputStream = context.getDataStreamOrThrow(collectionId); sideInput.getKey().getTransformId() + "-" + sideInput.getKey().getLocalName(); WindowedValueCoder<KV<Void, Object>> kvCoder = kvCoders.get(intTag); DataStream<WindowedValue<KV<Void, Object>>> keyedSideInputStream = context); viewStream.uid(pTransform.getUniqueName() + "-" + sideInput.getKey().getLocalName());
ExecutableStageMatcher.withInput("read.out") .withSideInputs( RunnerApi.ExecutableStagePayload.SideInputId.newBuilder() .setTransformId("sideParDo") .setLocalName("side")
String collectionId = components .getTransformsOrThrow(sideInputId.getTransformId()) .getInputsOrThrow(sideInputId.getLocalName());
pt.putInputs(outerLocalName, sideInput.collection().getId()); payload.addSideInputs( SideInputId.newBuilder() .setTransformId(sideInput.transform().getId()) .setLocalName(sideInput.localName()));
String collectionId = components .getTransformsOrThrow(sideInputId.getTransformId()) .getInputsOrThrow(sideInputId.getLocalName());
String collectionId = components .getTransformsOrThrow(sideInput.getKey().getTransformId()) .getInputsOrThrow(sideInput.getKey().getLocalName()); DataStream<Object> sideInputStream = context.getDataStreamOrThrow(collectionId); TypeInformation<Object> tpe = sideInputStream.getType(); String collectionId = components .getTransformsOrThrow(sideInput.getKey().getTransformId()) .getInputsOrThrow(sideInput.getKey().getLocalName()); DataStream<WindowedValue<?>> sideInputStream = context.getDataStreamOrThrow(collectionId); sideInput.getKey().getTransformId() + "-" + sideInput.getKey().getLocalName(); WindowedValueCoder<KV<Void, Object>> kvCoder = kvCoders.get(intTag); DataStream<WindowedValue<KV<Void, Object>>> keyedSideInputStream =
String sideInputTag = sideInputId.getLocalName(); String collectionId = components .getTransformsOrThrow(sideInputId.getTransformId()) .getInputsOrThrow(sideInputId.getLocalName()); RunnerApi.WindowingStrategy windowingStrategyProto = components.getWindowingStrategiesOrThrow(
String sideInputTag = sideInputId.getLocalName(); String collectionId = components .getTransformsOrThrow(sideInputId.getTransformId()) .getInputsOrThrow(sideInputId.getLocalName()); RunnerApi.WindowingStrategy windowingStrategyProto = components.getWindowingStrategiesOrThrow(
@Override public <T, V, W extends BoundedWindow> SideInputHandler<V, W> forSideInput( String transformId, String sideInputId, RunnerApi.FunctionSpec accessPattern, Coder<T> elementCoder, Coder<W> windowCoder) { PCollectionView collectionNode = sideInputToCollection.get( SideInputId.newBuilder().setTransformId(transformId).setLocalName(sideInputId).build()); checkArgument(collectionNode != null, "No side input for %s/%s", transformId, sideInputId); if (PTransformTranslation.ITERABLE_SIDE_INPUT.equals(accessPattern.getUrn())) { @SuppressWarnings("unchecked") // T == V Coder<V> outputCoder = (Coder<V>) elementCoder; return forIterableSideInput(collectionNode, outputCoder); } else if (PTransformTranslation.MULTIMAP_SIDE_INPUT.equals(accessPattern.getUrn()) || Materializations.MULTIMAP_MATERIALIZATION_URN.equals(accessPattern.getUrn())) { // TODO: Remove non standard URN. // Using non standard version of multimap urn as dataflow uses the non standard urn. @SuppressWarnings("unchecked") // T == KV<?, V> KvCoder<?, V> kvCoder = (KvCoder<?, V>) elementCoder; return forMultimapSideInput(collectionNode, kvCoder.getKeyCoder(), kvCoder.getValueCoder()); } else { throw new IllegalArgumentException( String.format("Unknown side input access pattern: %s", accessPattern)); } }
@Override public <T, V, W extends BoundedWindow> SideInputHandler<V, W> forSideInput( String transformId, String sideInputId, RunnerApi.FunctionSpec accessPattern, Coder<T> elementCoder, Coder<W> windowCoder) { PCollectionView collectionNode = sideInputToCollection.get( SideInputId.newBuilder().setTransformId(transformId).setLocalName(sideInputId).build()); checkArgument(collectionNode != null, "No side input for %s/%s", transformId, sideInputId); if (PTransformTranslation.ITERABLE_SIDE_INPUT.equals(accessPattern.getUrn())) { @SuppressWarnings("unchecked") // T == V Coder<V> outputCoder = (Coder<V>) elementCoder; return forIterableSideInput(collectionNode, outputCoder); } else if (PTransformTranslation.MULTIMAP_SIDE_INPUT.equals(accessPattern.getUrn()) || Materializations.MULTIMAP_MATERIALIZATION_URN.equals(accessPattern.getUrn())) { // TODO: Remove non standard URN. // Using non standard version of multimap urn as dataflow uses the non standard urn. @SuppressWarnings("unchecked") // T == KV<?, V> KvCoder<?, V> kvCoder = (KvCoder<?, V>) elementCoder; return forMultimapSideInput(collectionNode, kvCoder.getKeyCoder(), kvCoder.getValueCoder()); } else { throw new IllegalArgumentException( String.format("Unknown side input access pattern: %s", accessPattern)); } }
@Override protected boolean matchesSafely(ExecutableStage item) { return item.getInputPCollection().getId().equals(inputPCollectionId) && containsInAnyOrder(sideInputIds.toArray()) .matches( item.getSideInputs() .stream() .map( ref -> SideInputId.newBuilder() .setTransformId(ref.transform().getId()) .setLocalName(ref.localName()) .build()) .collect(Collectors.toSet())) && materializedPCollection.matches( item.getOutputPCollections() .stream() .map(PCollectionNode::getId) .collect(Collectors.toSet())) && containsInAnyOrder(fusedTransforms.toArray(new String[0])) .matches( item.getTransforms() .stream() .map(PTransformNode::getId) .collect(Collectors.toSet())); }
/** * Creates a new state handler for the given stage. Note that this requires a traversal of the * stage itself, so this should only be called once per stage rather than once per bundle. */ public static FlinkStreamingSideInputHandlerFactory forStage( ExecutableStage stage, Map<SideInputId, PCollectionView<?>> viewMapping, org.apache.beam.runners.core.SideInputHandler runnerHandler) { ImmutableMap.Builder<SideInputId, PCollectionView<?>> sideInputBuilder = ImmutableMap.builder(); for (SideInputReference sideInput : stage.getSideInputs()) { SideInputId sideInputId = SideInputId.newBuilder() .setTransformId(sideInput.transform().getId()) .setLocalName(sideInput.localName()) .build(); sideInputBuilder.put( sideInputId, checkNotNull( viewMapping.get(sideInputId), "No side input for %s/%s", sideInputId.getTransformId(), sideInputId.getLocalName())); } FlinkStreamingSideInputHandlerFactory factory = new FlinkStreamingSideInputHandlerFactory(sideInputBuilder.build(), runnerHandler); return factory; }
SideInputId.newBuilder().setTransformId(transformId).setLocalName(sideInputId).build()); checkArgument(collectionNode != null, "No side input for %s/%s", transformId, sideInputId);
/** * Creates a new state handler for the given stage. Note that this requires a traversal of the * stage itself, so this should only be called once per stage rather than once per bundle. */ public static FlinkStreamingSideInputHandlerFactory forStage( ExecutableStage stage, Map<SideInputId, PCollectionView<?>> viewMapping, org.apache.beam.runners.core.SideInputHandler runnerHandler) { ImmutableMap.Builder<SideInputId, PCollectionView<?>> sideInputBuilder = ImmutableMap.builder(); for (SideInputReference sideInput : stage.getSideInputs()) { SideInputId sideInputId = SideInputId.newBuilder() .setTransformId(sideInput.transform().getId()) .setLocalName(sideInput.localName()) .build(); sideInputBuilder.put( sideInputId, checkNotNull( viewMapping.get(sideInputId), "No side input for %s/%s", sideInputId.getTransformId(), sideInputId.getLocalName())); } FlinkStreamingSideInputHandlerFactory factory = new FlinkStreamingSideInputHandlerFactory(sideInputBuilder.build(), runnerHandler); return factory; }
/** * Creates a new state handler for the given stage. Note that this requires a traversal of the * stage itself, so this should only be called once per stage rather than once per bundle. */ static FlinkBatchSideInputHandlerFactory forStage( ExecutableStage stage, RuntimeContext runtimeContext) { ImmutableMap.Builder<SideInputId, PCollectionNode> sideInputBuilder = ImmutableMap.builder(); for (SideInputReference sideInput : stage.getSideInputs()) { sideInputBuilder.put( SideInputId.newBuilder() .setTransformId(sideInput.transform().getId()) .setLocalName(sideInput.localName()) .build(), sideInput.collection()); } return new FlinkBatchSideInputHandlerFactory(sideInputBuilder.build(), runtimeContext); }
/** * Creates a new state handler for the given stage. Note that this requires a traversal of the * stage itself, so this should only be called once per stage rather than once per bundle. */ static FlinkBatchSideInputHandlerFactory forStage( ExecutableStage stage, RuntimeContext runtimeContext) { ImmutableMap.Builder<SideInputId, PCollectionNode> sideInputBuilder = ImmutableMap.builder(); for (SideInputReference sideInput : stage.getSideInputs()) { sideInputBuilder.put( SideInputId.newBuilder() .setTransformId(sideInput.transform().getId()) .setLocalName(sideInput.localName()) .build(), sideInput.collection()); } return new FlinkBatchSideInputHandlerFactory(sideInputBuilder.build(), runtimeContext); }
/** Create a side input reference from a SideInputId proto and components. */ public static SideInputReference fromSideInputId( SideInputId sideInputId, RunnerApi.Components components) { String transformId = sideInputId.getTransformId(); String localName = sideInputId.getLocalName(); String collectionId = components.getTransformsOrThrow(transformId).getInputsOrThrow(localName); PTransform transform = components.getTransformsOrThrow(transformId); PCollection collection = components.getPcollectionsOrThrow(collectionId); return SideInputReference.of( PipelineNode.pTransform(transformId, transform), localName, PipelineNode.pCollection(collectionId, collection)); }