@Override public void outputWindowedValue( final KV<K, Iterable<V>> output, final Instant timestamp, final Collection<? extends BoundedWindow> windows, final PaneInfo pane) { windowedValues.add(WindowedValue.of(output, timestamp, windows, pane)); }
@Override public <AdditionalOutputT> void outputWindowedValue( TupleTag<AdditionalOutputT> tag, AdditionalOutputT output, Instant timestamp, Collection<? extends BoundedWindow> windows, PaneInfo pane) { throw new UnsupportedOperationException("GroupAlsoByWindow should not use tagged outputs."); }
/** @deprecated for use only in compatibility with old broken code */ @Deprecated static <T> WindowedValue<T> createWithoutValidation( T value, Instant timestamp, Collection<? extends BoundedWindow> windows, PaneInfo pane) { if (windows.size() == 1) { return of(value, timestamp, windows.iterator().next(), pane); } else { return new TimestampedValueInMultipleWindows<>(value, timestamp, windows, pane); } }
@Override public void outputWindowedValue( OutputT output, Instant timestamp, Collection<? extends BoundedWindow> windows, PaneInfo pane) { outputManager.output( mainOutputTag, WindowedValue.of(output, timestamp, windows, pane)); }
@Override public <AdditionalOutputT> void outputWindowedValue( TupleTag<AdditionalOutputT> tag, AdditionalOutputT output, Instant timestamp, Collection<? extends BoundedWindow> windows, PaneInfo pane) { throw new UnsupportedOperationException( String.format( "%s should not use tagged outputs", DirectGroupAlsoByWindow.class.getSimpleName())); } }
@SuppressWarnings({"unchecked", "rawtypes"}) private WindowedValue<Iterable<?>> valuesInWindow( List<Object> values, Instant timestamp, BoundedWindow window) { return (WindowedValue) WindowedValue.of(values, timestamp, window, PaneInfo.NO_FIRING); } }
/** Returns a {@code WindowedValue} with the given value, timestamp, and windows. */ public static <T> WindowedValue<T> of( T value, Instant timestamp, Collection<? extends BoundedWindow> windows, PaneInfo pane) { checkArgument(pane != null, "WindowedValue requires PaneInfo, but it was null"); checkArgument(windows.size() > 0, "WindowedValue requires windows, but there were none"); if (windows.size() == 1) { return of(value, timestamp, windows.iterator().next(), pane); } else { return new TimestampedValueInMultipleWindows<>(value, timestamp, windows, pane); } }
@Override public void outputWindowedValue( KV<K, Iterable<V>> output, Instant timestamp, Collection<? extends BoundedWindow> windows, PaneInfo pane) { if (traceTuples) { LOG.debug("\nemitting {} timestamp {}\n", output, timestamp); } ApexGroupByKeyOperator.this.output.emit( ApexStreamTuple.DataTuple.of(WindowedValue.of(output, timestamp, windows, pane))); }
@Override public WindowedValue<T> nextRecord(WindowedValue<T> t) throws IOException { if (inputAvailable) { final T current = reader.getCurrent(); final Instant timestamp = reader.getCurrentTimestamp(); // advance reader to have a record ready next time inputAvailable = readerInvoker.invokeAdvance(reader); return WindowedValue.of(current, timestamp, GlobalWindow.INSTANCE, PaneInfo.NO_FIRING); } return null; }
@Override public <T> void output(TupleTag<T> tag, WindowedValue<T> output) { collector.collect( WindowedValue.of( new RawUnionValue(outputMap.get(tag), output.getValue()), output.getTimestamp(), output.getWindows(), output.getPane())); } }
/** Performs a seed {@link DoFn.ProcessElement} call feeding the element and restriction. */ void startElement(InputT element, RestrictionT restriction) throws Exception { startElement( WindowedValue.of( KV.of(element, restriction), currentProcessingTime, GlobalWindow.INSTANCE, PaneInfo.ON_TIME_AND_ONLY_FIRING)); }
@Override public void processElement(WindowedValue<InputT> compressedElement) throws Exception { for (WindowedValue<InputT> element : compressedElement.explodeWindows()) { Collection<? extends BoundedWindow> windows = assignWindows(windowFn, element); outputBundle.add( WindowedValue.of( element.getValue(), element.getTimestamp(), windows, element.getPane())); } }
@Override public void processElement(WindowedValue<InputT> compressedElement) throws Exception { for (WindowedValue<InputT> element : compressedElement.explodeWindows()) { Collection<? extends BoundedWindow> windows = assignWindows(windowFn, element); outputBundle.add( WindowedValue.of( element.getValue(), element.getTimestamp(), windows, element.getPane())); } }
@ProcessElement public void processElement(ProcessContext c) throws Exception { c.output( IsmRecord.of( ImmutableList.of(GlobalWindow.INSTANCE, indexInBundle), WindowedValue.of(c.element(), c.timestamp(), GlobalWindow.INSTANCE, c.pane()))); indexInBundle += 1; } }
@Override public void flatMap(WindowedValue<T> input, Collector<WindowedValue<T>> collector) throws Exception { Collection<W> windows = windowFn.assignWindows(new FlinkAssignContext<>(windowFn, input)); for (W window : windows) { collector.collect( WindowedValue.of(input.getValue(), input.getTimestamp(), window, input.getPane())); } } }
@Override @SuppressWarnings("unchecked") public <T> void output(TupleTag<T> tag, WindowedValue<T> output) { collector.collect( WindowedValue.of( new RawUnionValue(0 /* single output */, output.getValue()), output.getTimestamp(), output.getWindows(), output.getPane())); } }
@Override public void flatMap(WindowedValue<T> input, Collector<WindowedValue<T>> collector) throws Exception { Collection<W> windows = windowFn.assignWindows(new FlinkAssignContext<>(windowFn, input)); for (W window : windows) { collector.collect( WindowedValue.of(input.getValue(), input.getTimestamp(), window, input.getPane())); } } }
/** * Returns a collection of {@link WindowedValue WindowedValues} identical to this one, except each * is in exactly one of the windows that this {@link WindowedValue} is in. */ public Iterable<WindowedValue<T>> explodeWindows() { ImmutableList.Builder<WindowedValue<T>> windowedValues = ImmutableList.builder(); for (BoundedWindow w : getWindows()) { windowedValues.add(of(getValue(), getTimestamp(), w, getPane())); } return windowedValues.build(); }
@ProcessElement public void processElement(final ProcessContext c, final BoundedWindow window) { c.output( KV.of( c.element().getKey(), KV.of( c.timestamp(), WindowedValue.of(c.element(), c.timestamp(), window, c.pane())))); } }
@Test public void testExplodeWindowsInOneWindowEquals() { Instant now = Instant.now(); BoundedWindow window = new IntervalWindow(now.minus(1000L), now.plus(1000L)); WindowedValue<String> value = WindowedValue.of("foo", now, window, PaneInfo.ON_TIME_AND_ONLY_FIRING); assertThat(Iterables.getOnlyElement(value.explodeWindows()), equalTo(value)); }