@ProcessElement public void processElement( ProcessContext context, @StateId("counter") ValueState<Integer> counterState) { int current = (counterState.read() != null ? counterState.read() : 0); if (current < limitCount) { context.output(context.element().getValue()); counterState.write(current + 1); } } }
@Override public void persist() { checkInvariants(); if (activeWindowToStateAddressWindows.isEmpty()) { // Force all persistent state to disappear. valueState.clear(); return; } if (activeWindowToStateAddressWindows.equals(originalActiveWindowToStateAddressWindows)) { // No change. return; } valueState.write(activeWindowToStateAddressWindows); // No need to update originalActiveWindowToStateAddressWindows since this object is about to // become garbage. }
public void prefetchIsClosed(StateAccessor<?> state) { if (isFinishedSetNeeded()) { state.access(FINISHED_BITS_TAG).readLater(); } }
minBufferedIdState.readLater(); long nextId = MoreObjects.firstNonNull(nextIdState.read(), 0L); long minBufferedId = MoreObjects.firstNonNull(minBufferedIdState.read(), Long.MAX_VALUE); minBufferedIdState.write(minBufferedId); elementsBuffered.inc(); continue; minBufferedIdState.clear(); minBufferedId = Long.MAX_VALUE; nextIdState.write(nextId);
private void performStateUpdates( String stateId, ProcessContext ctx, ValueState<Integer> valueState) { Map<String, Integer> stateValues = stateValuesMap.get(stateId); Integer value = ctx.element().getValue(); if (value == null) { throw new IllegalStateException(); } switch (value) { case CLEAR_STATE: valueState.clear(); break; case WRITE_STATE_TO_MAP: stateValues.put(ctx.element().getKey(), valueState.read()); break; default: Integer currentState = valueState.read(); if (currentState == null) { currentState = value; } else { currentState += value; } valueState.write(currentState); } } }));
/** * Retrieve the value as written by {@link #addSideInputValue(PCollectionView, WindowedValue)}, * without applying the SDK specific {@link ViewFn}. * * @param view * @param window * @param <T> * @return */ public <T> Iterable<?> getIterable(PCollectionView<T> view, BoundedWindow window) { @SuppressWarnings("unchecked") Coder<BoundedWindow> windowCoder = (Coder<BoundedWindow>) view.getWindowingStrategyInternal().getWindowFn().windowCoder(); StateTag<ValueState<Iterable<?>>> stateTag = sideInputContentsTags.get(view); ValueState<Iterable<?>> state = stateInternals.state(StateNamespaces.window(windowCoder, window), stateTag); Iterable<?> elements = state.read(); // return empty collection when no side input was received for ready window return (elements != null) ? elements : Collections.emptyList(); }
@ProcessElement public void processElement( ProcessContext context, @StateId("valueState") ValueState<Integer> intValueState, @StateId("bagState") BagState<Integer> intBagState) { Long value = context.element().getValue(); assertNotNull(value); if (value == 0L) { intValueState.write(42); intBagState.add(40); intBagState.add(1); intBagState.add(1); oneShotLatch.countDown(); } } }));
private void clearFinishedBits(ValueState<BitSet> state) { if (!isFinishedSetNeeded()) { // Nothing to clear. return; } state.clear(); }
minBufferedIdState.readLater(); long nextId = MoreObjects.firstNonNull(nextIdState.read(), 0L); long minBufferedId = MoreObjects.firstNonNull(minBufferedIdState.read(), Long.MAX_VALUE); minBufferedIdState.write(minBufferedId); elementsBuffered.inc(); continue; minBufferedIdState.clear(); minBufferedId = Long.MAX_VALUE; nextIdState.write(nextId);
@Test public void testValue() throws Exception { ValueState<String> value = underTest.state(NAMESPACE_1, STRING_VALUE_ADDR); // State instances are cached, but depend on the namespace. assertThat(underTest.state(NAMESPACE_1, STRING_VALUE_ADDR), equalTo(value)); assertThat(underTest.state(NAMESPACE_2, STRING_VALUE_ADDR), not(equalTo(value))); assertThat(value.read(), Matchers.nullValue()); value.write("hello"); assertThat(value.read(), equalTo("hello")); value.write("world"); assertThat(value.read(), equalTo("world")); value.clear(); assertThat(value.read(), Matchers.nullValue()); assertThat(underTest.state(NAMESPACE_1, STRING_VALUE_ADDR), equalTo(value)); }
private FinishedTriggersBitSet readFinishedBits(ValueState<BitSet> state) { if (!isFinishedSetNeeded()) { // If no trigger in the tree will ever have finished bits, then we don't need to read them. // So that the code can be agnostic to that fact, we create a BitSet that is all 0 (not // finished) for each trigger in the tree. return FinishedTriggersBitSet.emptyWithCapacity(rootTrigger.getFirstIndexAfterSubtree()); } @Nullable BitSet bitSet = state.read(); return bitSet == null ? FinishedTriggersBitSet.emptyWithCapacity(rootTrigger.getFirstIndexAfterSubtree()) : FinishedTriggersBitSet.fromBitSet(bitSet); }
public void storeCurrentPaneInfo(ReduceFn<?, ?, ?, ?>.Context context, PaneInfo currentPane) { context.state().access(PANE_INFO_TAG).write(currentPane); }
public void clear(StateAccessor<?> state) { state.access(PANE_INFO_TAG).clear(); }
Iterables.getOnlyElement(c.element().elementsIterable()); WindowedValue<InputT> element = windowedValue.withValue(windowedValue.getValue().getKey()); elementState.write(element); elementAndRestriction = KV.of(element, windowedValue.getValue().getValue()); } else { elementState.readLater(); restrictionState.readLater(); elementAndRestriction = KV.of(elementState.read(), restrictionState.read()); elementState.clear(); restrictionState.clear(); holdState.clear(); return; restrictionState.write(result.getResidualRestriction()); @Nullable Instant futureOutputWatermark = result.getFutureOutputWatermark(); if (futureOutputWatermark == null) {
private void performStateUpdates( ProcessContext ctx, ValueState<Integer> valueState) { Integer value = ctx.element().getValue(); if (value == null) { throw new IllegalStateException(); } switch (value) { case CLEAR_STATE: valueState.clear(); break; case WRITE_STATE: Integer read = valueState.read(); ctx.output( KV.of( ctx.element().getKey(), read == null ? "null" : read.toString())); break; default: Integer currentState = valueState.read(); if (currentState == null) { currentState = value; } else { currentState += value; } valueState.write(currentState); } } }));
@OnTimer(timerId) public void onTimer(OnTimerContext context, @StateId(stateId) ValueState<String> state) { context.output(KV.of(state.read(), timerOutput)); } };
@ProcessElement public void processElement( @TimerId(END_OF_WINDOW_ID) Timer timer, @StateId(BATCH_ID) BagState<InputT> batch, @StateId(NUM_ELEMENTS_IN_BATCH_ID) CombiningState<Long, long[], Long> numElementsInBatch, @StateId(KEY_ID) ValueState<K> key, @Element KV<K, InputT> element, BoundedWindow window, OutputReceiver<KV<K, Iterable<InputT>>> receiver) { Instant windowExpires = window.maxTimestamp().plus(allowedLateness); LOG.debug( "*** SET TIMER *** to point in time {} for window {}", windowExpires.toString(), window.toString()); timer.set(windowExpires); key.write(element.getKey()); batch.add(element.getValue()); LOG.debug("*** BATCH *** Add element for window {} ", window.toString()); // blind add is supported with combiningState numElementsInBatch.add(1L); Long num = numElementsInBatch.read(); if (num % prefetchFrequency == 0) { //prefetch data and modify batch state (readLater() modifies this) batch.readLater(); } if (num >= batchSize) { LOG.debug("*** END OF BATCH *** for window {}", window.toString()); flushBatch(receiver, key, batch, numElementsInBatch); } }
private void persistFinishedSet( StateAccessor<?> state, FinishedTriggersBitSet modifiedFinishedSet) { if (!isFinishedSetNeeded()) { return; } ValueState<BitSet> finishedSetState = state.access(FINISHED_BITS_TAG); if (!readFinishedBits(finishedSetState).equals(modifiedFinishedSet)) { if (modifiedFinishedSet.getBitSet().isEmpty()) { finishedSetState.clear(); } else { finishedSetState.write(modifiedFinishedSet.getBitSet()); } } }
public void prefetchPaneInfo(ReduceFn<?, ?, ?, ?>.Context context) { context.state().access(PaneInfoTracker.PANE_INFO_TAG).readLater(); }