Tabnine Logo
SourceFunction$SourceContext
Code IndexAdd Tabnine to your IDE (free)

How to use
SourceFunction$SourceContext
in
org.apache.flink.streaming.api.functions.source

Best Java code snippets using org.apache.flink.streaming.api.functions.source.SourceFunction$SourceContext (Showing top 20 results out of 540)

origin: apache/flink

synchronized (ctx.getCheckpointLock()) {
  long value = randomKeyRangeStates.incrementAndGet(randomKey);
    StringUtils.getRandomString(random, payloadLength, payloadLength, 'A', 'z'));
  ctx.collect(event);
origin: apache/flink

  @Override
  public void onProcessingTime(long timestamp) throws Exception {
    long minAcrossAll = Long.MAX_VALUE;
    boolean isEffectiveMinAggregation = false;
    for (KafkaTopicPartitionState<?> state : allPartitions) {
      // we access the current watermark for the periodic assigners under the state
      // lock, to prevent concurrent modification to any internal variables
      final long curr;
      //noinspection SynchronizationOnLocalVariableOrMethodParameter
      synchronized (state) {
        curr = ((KafkaTopicPartitionStateWithPeriodicWatermarks<?, ?>) state).getCurrentWatermarkTimestamp();
      }
      minAcrossAll = Math.min(minAcrossAll, curr);
      isEffectiveMinAggregation = true;
    }
    // emit next watermark, if there is one
    if (isEffectiveMinAggregation && minAcrossAll > lastWatermarkTimestamp) {
      lastWatermarkTimestamp = minAcrossAll;
      emitter.emitWatermark(new Watermark(minAcrossAll));
    }
    // schedule the next watermark
    timerService.registerTimer(timerService.getCurrentProcessingTime() + interval, this);
  }
}
origin: apache/flink

@Override
public void run(SourceContext<Integer> ctx) throws Exception {
  for (int i = 0; i < numWatermarks; i++) {
    ctx.collectWithTimestamp(i, initialTime + i);
    ctx.emitWatermark(new Watermark(initialTime + i));
  }
  while (running) {
    Thread.sleep(20);
  }
}
origin: org.apache.flink/flink-streaming-java

  public void close() {
    this.shouldClose = true;
  }
}
origin: org.apache.flink/flink-streaming-java_2.10

  public void close() {
    this.shouldClose = true;
  }
}
origin: king/bravo

  @Override
  public void withCheckpointLock(SourceContext<String> ctx) throws Exception {
    ctx.emitWatermark(new Watermark(watermark));
    Thread.sleep(500);
  }
}
origin: apache/bahir-flink

@Override
public void run(SourceContext<Event> ctx) throws Exception {
  while (isRunning) {
    long timestamp = initialTimestamp + 1000 * number.get();
    ctx.collectWithTimestamp(Event.of(number.get(), "test_event", random.nextDouble(), timestamp), timestamp);
    if (number.incrementAndGet() >= this.count) {
      cancel();
    }
  }
}
origin: apache/bahir-flink

@Override
public void run(SourceContext<Tuple4<Integer, String, Double, Long>> ctx) throws Exception {
  while (isRunning) {
    long timestamp = initialTimestamp + 1000 * number.get();
    ctx.collectWithTimestamp(Tuple4.of(number.get(), "test_tuple", random.nextDouble(), timestamp), timestamp);
    if (number.incrementAndGet() >= this.count) {
      cancel();
    }
  }
}
origin: king/bravo

  @Override
  public void withCheckpointLock(SourceContext<String> ctx) throws Exception {
    ctx.collectWithTimestamp(output, timestamp);
  }
}
origin: dataArtisans/oscon

@Override
public void run(SourceContext<DataPoint<Long>> ctx) throws Exception {
 while (running) {
  synchronized (ctx.getCheckpointLock()) {
   ctx.collectWithTimestamp(new DataPoint<>(currentTimeMs, 0L), currentTimeMs);
   ctx.emitWatermark(new Watermark(currentTimeMs));
   currentTimeMs += periodMs;
  }
  timeSync();
 }
}
origin: apache/bahir-flink

@Override
public void run(SourceContext<String> ctx) throws Exception {
  while (isRunning) {
    long timestamp = initialTimestamp + 1000 * number.get();
    ctx.collectWithTimestamp(WORDS[random.nextInt(WORDS.length)], timestamp);
    if (number.incrementAndGet() >= this.count) {
      cancel();
    }
  }
}
origin: apache/bahir-flink

/**
 * To handle data with timestamp
 *
 * @param data data received from feeder actor
 * @param timestamp timestamp received from feeder actor
 */
private void collect(Object data, long timestamp) {
 ctx.collectWithTimestamp(data, timestamp);
}
origin: org.apache.bahir/flink-connector-akka

/**
 * To handle data with timestamp
 *
 * @param data data received from feeder actor
 * @param timestamp timestamp received from feeder actor
 */
private void collect(Object data, long timestamp) {
 ctx.collectWithTimestamp(data, timestamp);
}
origin: com.alibaba.blink/flink-examples-streaming

@Override
public void run(SourceContext<Tuple3<String, Long, Integer>> ctx) throws Exception {
  for (Tuple3<String, Long, Integer> value : input) {
    ctx.collectWithTimestamp(value, value.f1);
    ctx.emitWatermark(new Watermark(value.f1 - 1));
  }
  ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
}
origin: NationalSecurityAgency/timely

@Override
public void onClose(Session session, CloseReason reason) {
  super.onClose(session, reason);
  try {
    close();
  } catch (Exception e1) {
    LOG.error("Error closing client", e1);
  }
  // Signal done sending data
  ctx.emitWatermark(new Watermark(Long.MAX_VALUE));
}
origin: apache/flink

switch (testMethod) {
  case COLLECT:
    context.collect("msg");
    expectedOutput.add(new StreamRecord<>("msg", processingTimeService.getCurrentProcessingTime()));
    expectedOutput.add(new Watermark(processingTimeService.getCurrentProcessingTime() - (processingTimeService.getCurrentProcessingTime() % watermarkInterval)));
    break;
  case COLLECT_WITH_TIMESTAMP:
    context.collectWithTimestamp("msg", processingTimeService.getCurrentProcessingTime());
    expectedOutput.add(new StreamRecord<>("msg", processingTimeService.getCurrentProcessingTime()));
    expectedOutput.add(new Watermark(processingTimeService.getCurrentProcessingTime() - (processingTimeService.getCurrentProcessingTime() % watermarkInterval)));
    context.emitWatermark(new Watermark(processingTimeService.getCurrentProcessingTime()));
    assertTrue(mockStreamStatusMaintainer.getStreamStatus().isIdle());
    assertEquals(expectedOutput, output);
switch (testMethod) {
  case COLLECT:
    context.collect("msg");
    expectedOutput.add(new StreamRecord<>("msg", processingTimeService.getCurrentProcessingTime()));
    assertTrue(mockStreamStatusMaintainer.getStreamStatus().isActive());
    break;
  case COLLECT_WITH_TIMESTAMP:
    context.collectWithTimestamp("msg", processingTimeService.getCurrentProcessingTime());
    expectedOutput.add(new StreamRecord<>("msg", processingTimeService.getCurrentProcessingTime()));
    assertTrue(mockStreamStatusMaintainer.getStreamStatus().isActive());
    break;
origin: apache/flink

switch (testMethod) {
  case COLLECT:
    context.collect("msg");
    break;
  case COLLECT_WITH_TIMESTAMP:
    context.collectWithTimestamp("msg", processingTimeService.getCurrentProcessingTime());
    break;
  case EMIT_WATERMARK:
    context.emitWatermark(new Watermark(processingTimeService.getCurrentProcessingTime()));
    break;
switch (testMethod) {
  case COLLECT:
    context.collect("msg");
    break;
  case COLLECT_WITH_TIMESTAMP:
    context.collectWithTimestamp("msg", processingTimeService.getCurrentProcessingTime());
    break;
  case EMIT_WATERMARK:
    context.emitWatermark(new Watermark(processingTimeService.getCurrentProcessingTime()));
    break;
origin: apache/flink

  ctx.emitWatermark(Watermark.MAX_WATERMARK);
ctx.close();
if (latencyEmitter != null) {
  latencyEmitter.close();
origin: apache/flink

sourceContext.collectWithTimestamp(record, timestamp);
partitionState.setOffset(offset);
origin: apache/flink

@Override
public void run(SourceContext<Tuple2<Long, IntType>> ctx) throws Exception {
  final RuntimeContext runtimeContext = getRuntimeContext();
  // detect if this task is "the chosen one" and should fail (via subtaskidx), if it did not fail before (via attempt)
  final boolean failThisTask =
    runtimeContext.getAttemptNumber() == 0 && runtimeContext.getIndexOfThisSubtask() == 0;
  // we loop longer than we have elements, to permit delayed checkpoints
  // to still cause a failure
  while (running && emitCallCount < expectedEmitCalls) {
    // the function failed before, or we are in the elements before the failure
    synchronized (ctx.getCheckpointLock()) {
      eventEmittingGenerator.emitEvent(ctx, emitCallCount++);
    }
    if (emitCallCount < failureAfterNumElements) {
      Thread.sleep(1);
    } else if (failThisTask && emitCallCount == failureAfterNumElements) {
      // wait for a pending checkpoint that fulfills our requirements if needed
      while (checkpointStatus.get() != STATEFUL_CHECKPOINT_COMPLETED) {
        Thread.sleep(1);
      }
      throw new Exception("Artificial Failure");
    }
  }
  if (usingProcessingTime) {
    while (running) {
      Thread.sleep(10);
    }
  }
}
org.apache.flink.streaming.api.functions.sourceSourceFunction$SourceContext

Javadoc

Interface that source functions use to emit elements, and possibly watermarks.

Most used methods

  • collect
    Emits one element from the source, without attaching a timestamp. In most cases, this is the default
  • getCheckpointLock
    Returns the checkpoint lock. Please refer to the class-level comment in SourceFunction for details a
  • emitWatermark
    Emits the given Watermark. A Watermark of value t declares that no elements with a timestamp t' late
  • collectWithTimestamp
    Emits one element from the source, and attaches the given timestamp. This method is relevant for pro
  • close
    This method is called by the system to shut down the context.
  • markAsTemporarilyIdle
    Marks the source to be temporarily idle. This tells the system that this source will temporarily sto

Popular in Java

  • Reading from database using SQL prepared statement
  • getExternalFilesDir (Context)
  • requestLocationUpdates (LocationManager)
  • getResourceAsStream (ClassLoader)
  • DateFormat (java.text)
    Formats or parses dates and times.This class provides factories for obtaining instances configured f
  • Arrays (java.util)
    This class contains various methods for manipulating arrays (such as sorting and searching). This cl
  • List (java.util)
    An ordered collection (also known as a sequence). The user of this interface has precise control ove
  • Scanner (java.util)
    A parser that parses a text string of primitive types and strings with the help of regular expressio
  • Executors (java.util.concurrent)
    Factory and utility methods for Executor, ExecutorService, ScheduledExecutorService, ThreadFactory,
  • Loader (org.hibernate.loader)
    Abstract superclass of object loading (and querying) strategies. This class implements useful common
  • Best plugins for Eclipse
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now