Tabnine Logo
StreamExecutionEnvironment.setBufferTimeout
Code IndexAdd Tabnine to your IDE (free)

How to use
setBufferTimeout
method
in
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment

Best Java code snippets using org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.setBufferTimeout (Showing top 13 results out of 315)

origin: apache/flink

env.setBufferTimeout(1);
origin: apache/flink

env.setBufferTimeout(1);
origin: apache/flink

.setBufferTimeout(1);
origin: apache/flink

private static void runPartitioningProgram(int parallelism) throws Exception {
  StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  env.setParallelism(parallelism);
  env.getConfig().enableObjectReuse();
  env.setBufferTimeout(5L);
  env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE);
  env
    .addSource(new TimeStampingSource())
    .map(new IdMapper<Tuple2<Long, Long>>())
    .keyBy(0)
    .addSink(new TimestampingSink());
  env.execute("Partitioning Program");
}
origin: apache/flink

final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setBufferTimeout(77); // set timeout to some recognizable number
origin: apache/flink

env.setBufferTimeout(0);
origin: intel-hadoop/HiBench

 @Override
 public void processStream(final FlinkBenchConfig config) throws Exception {

  final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  env.setBufferTimeout(config.bufferTimeout);

  createDataStream(config);
  DataStream<Tuple2<String, String>> dataStream = env.addSource(getDataStream());

  dataStream.map(new MapFunction<Tuple2<String, String>, Tuple2<String, String>>() {

   @Override
   public Tuple2<String, String> map(Tuple2<String, String> value) throws Exception {
    KafkaReporter kafkaReporter = new KafkaReporter(config.reportTopic, config.brokerList);

    kafkaReporter.report(Long.parseLong(value.f0), System.currentTimeMillis());
    return value;
   }
  });

  env.execute("Identity Job");
 }
}
origin: intel-hadoop/HiBench

 @Override
 public void processStream(final FlinkBenchConfig config) throws Exception {

  final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  env.setBufferTimeout(config.bufferTimeout);

  createDataStream(config);
  DataStream<Tuple2<String, String>> dataStream = env.addSource(getDataStream());

  dataStream.rebalance().map(
    new MapFunction<Tuple2<String, String>, Tuple2<String, String>>() {

     @Override
     public Tuple2<String, String> map(Tuple2<String, String> value) throws Exception {
      KafkaReporter kafkaReporter = new KafkaReporter(config.reportTopic, config.brokerList);

      kafkaReporter.report(Long.parseLong(value.f0), System.currentTimeMillis());
      return value;
     }
    });


  env.execute("Repartition Job");
 }
}
origin: intel-hadoop/HiBench

env.setBufferTimeout(config.bufferTimeout);
env.enableCheckpointing(config.checkpointDuration);
origin: intel-hadoop/HiBench

@Override
public void processStream(final FlinkBenchConfig config) throws Exception {
 final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
 env.setBufferTimeout(config.bufferTimeout);
 env.enableCheckpointing(config.checkpointDuration);
 createDataStream(config);
origin: DTStack/flinkStreamSQL

env.setBufferTimeout(FlinkUtil.getBufferTimeoutMillis(confProperties));
origin: com.alibaba.blink/flink-examples-streaming

.setBufferTimeout(1);
origin: king/bravo

private StreamExecutionEnvironment createJobGraph(int parallelism,
    Function<DataStream<String>, DataStream<String>> pipelinerBuilder) throws Exception {
  final Path checkpointDir = getCheckpointDir();
  final Path savepointRootDir = getSavepointDir();
  checkpointDir.getFileSystem().mkdirs(checkpointDir);
  savepointRootDir.getFileSystem().mkdirs(savepointRootDir);
  StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  env.getConfig().disableSysoutLogging();
  env.getCheckpointConfig().enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
  env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
  env.setBufferTimeout(0);
  env.setParallelism(parallelism);
  env.enableCheckpointing(500, CheckpointingMode.EXACTLY_ONCE);
  env.setStateBackend((StateBackend) new RocksDBStateBackend(checkpointDir.toString(), true));
  DataStream<String> sourceData = env
      .addSource(new TestPipelineSource())
      .uid("TestSource")
      .name("TestSource")
      .setParallelism(1);
  pipelinerBuilder.apply(sourceData)
      .addSink(new CollectingSink()).name("Output").uid("Output")
      .setParallelism(1);
  return env;
}
org.apache.flink.streaming.api.environmentStreamExecutionEnvironmentsetBufferTimeout

Javadoc

Sets the maximum time frequency (milliseconds) for the flushing of the output buffers. By default the output buffers flush frequently to provide low latency and to aid smooth developer experience. Setting the parameter can result in three logical modes:
  • A positive integer triggers flushing periodically by that integer
  • 0 triggers flushing after every record thus minimizing latency
  • -1 triggers flushing only when the output buffer is full thus maximizing throughput

Popular methods of StreamExecutionEnvironment

  • execute
  • getExecutionEnvironment
    Creates an execution environment that represents the context in which the program is currently execu
  • addSource
    Ads a data source with a custom type information thus opening a DataStream. Only in very special cas
  • getConfig
    Gets the config object.
  • enableCheckpointing
    Enables checkpointing for the streaming job. The distributed state of the streaming dataflow will be
  • setStreamTimeCharacteristic
    Sets the time characteristic for all streams create from this environment, e.g., processing time, ev
  • setParallelism
    Sets the parallelism for operations executed through this environment. Setting a parallelism of x he
  • fromElements
    Creates a new data stream that contains the given elements. The elements must all be of the same typ
  • setStateBackend
    Sets the state backend that describes how to store and checkpoint operator state. It defines both wh
  • createLocalEnvironment
    Creates a LocalStreamEnvironment. The local execution environment will run the program in a multi-th
  • fromCollection
    Creates a data stream from the given iterator.Because the iterator will remain unmodified until the
  • getCheckpointConfig
    Gets the checkpoint config, which defines values like checkpoint interval, delay between checkpoints
  • fromCollection,
  • getCheckpointConfig,
  • getParallelism,
  • getStreamGraph,
  • setRestartStrategy,
  • socketTextStream,
  • readTextFile,
  • generateSequence,
  • clean,
  • getStreamTimeCharacteristic

Popular in Java

  • Making http post requests using okhttp
  • getResourceAsStream (ClassLoader)
  • requestLocationUpdates (LocationManager)
  • scheduleAtFixedRate (ScheduledExecutorService)
  • System (java.lang)
    Provides access to system-related information and resources including standard input and output. Ena
  • MessageDigest (java.security)
    Uses a one-way hash function to turn an arbitrary number of bytes into a fixed-length byte sequence.
  • Set (java.util)
    A Set is a data structure which does not allow duplicate elements.
  • Timer (java.util)
    Timers schedule one-shot or recurring TimerTask for execution. Prefer java.util.concurrent.Scheduled
  • TimeUnit (java.util.concurrent)
    A TimeUnit represents time durations at a given unit of granularity and provides utility methods to
  • BoxLayout (javax.swing)
  • Top plugins for Android Studio
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now