org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromParallelCollection java code examples

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type
 * information. This method is useful for cases where the type is generic. In that case, the
 * type class (as given in
 * {@link #fromParallelCollection(org.apache.flink.util.SplittableIterator, Class)} does not
 * supply all type information.
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param typeInfo
 *         The TypeInformation for the produced data stream.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, TypeInformation<OUT>
    typeInfo) {
  return fromParallelCollection(iterator, typeInfo, "Parallel Collection Source");
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type class
 * (this is due to the fact that the Java compiler erases the generic type information).
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param type
 *         The class of the data produced by the iterator. Must not be a generic class.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, Class<OUT> type) {
  return fromParallelCollection(iterator, TypeExtractor.getForClass(type));
}

@Test
@SuppressWarnings("unchecked")
public void testFromCollectionParallelism() {
  try {
    TypeInformation<Integer> typeInfo = BasicTypeInfo.INT_TYPE_INFO;
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    DataStreamSource<Integer> dataStream1 = env.fromCollection(new DummySplittableIterator<Integer>(), typeInfo);
    try {
      dataStream1.setParallelism(4);
      fail("should throw an exception");
    }
    catch (IllegalArgumentException e) {
      // expected
    }
    dataStream1.addSink(new DiscardingSink<Integer>());
    DataStreamSource<Integer> dataStream2 = env.fromParallelCollection(new DummySplittableIterator<Integer>(),
        typeInfo).setParallelism(4);
    dataStream2.addSink(new DiscardingSink<Integer>());
    env.getExecutionPlan();
    assertEquals("Parallelism of collection source must be 1.", 1, env.getStreamGraph().getStreamNode(dataStream1.getId()).getParallelism());
    assertEquals("Parallelism of parallel collection source must be 4.",
        4,
        env.getStreamGraph().getStreamNode(dataStream2.getId()).getParallelism());
  }
  catch (Exception e) {
    e.printStackTrace();
    fail(e.getMessage());
  }
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type
 * information. This method is useful for cases where the type is generic. In that case, the
 * type class (as given in
 * {@link #fromParallelCollection(org.apache.flink.util.SplittableIterator, Class)} does not
 * supply all type information.
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param typeInfo
 *         The TypeInformation for the produced data stream.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, TypeInformation<OUT>
    typeInfo) {
  return fromParallelCollection(iterator, typeInfo, "Parallel Collection Source");
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type
 * information. This method is useful for cases where the type is generic. In that case, the
 * type class (as given in
 * {@link #fromParallelCollection(org.apache.flink.util.SplittableIterator, Class)} does not
 * supply all type information.
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param typeInfo
 *         The TypeInformation for the produced data stream.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, TypeInformation<OUT>
    typeInfo) {
  return fromParallelCollection(iterator, typeInfo, "Parallel Collection Source");
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type
 * information. This method is useful for cases where the type is generic. In that case, the
 * type class (as given in
 * {@link #fromParallelCollection(org.apache.flink.util.SplittableIterator, Class)} does not
 * supply all type information.
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param typeInfo
 *         The TypeInformation for the produced data stream.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, TypeInformation<OUT>
    typeInfo) {
  return fromParallelCollection(iterator, typeInfo, "Parallel Collection Source");
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type
 * information. This method is useful for cases where the type is generic. In that case, the
 * type class (as given in
 * {@link #fromParallelCollection(org.apache.flink.util.SplittableIterator, Class)} does not
 * supply all type information.
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param typeInfo
 *         The TypeInformation for the produced data stream.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, TypeInformation<OUT>
    typeInfo) {
  return fromParallelCollection(iterator, typeInfo, "Parallel Collection Source");
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type class
 * (this is due to the fact that the Java compiler erases the generic type information).
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param type
 *         The class of the data produced by the iterator. Must not be a generic class.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, Class<OUT> type) {
  return fromParallelCollection(iterator, TypeExtractor.getForClass(type));
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type class
 * (this is due to the fact that the Java compiler erases the generic type information).
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param type
 *         The class of the data produced by the iterator. Must not be a generic class.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, Class<OUT> type) {
  return fromParallelCollection(iterator, TypeExtractor.getForClass(type));
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type class
 * (this is due to the fact that the Java compiler erases the generic type information).
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param type
 *         The class of the data produced by the iterator. Must not be a generic class.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, Class<OUT> type) {
  return fromParallelCollection(iterator, TypeExtractor.getForClass(type));
}

/**
 * Creates a new data stream that contains elements in the iterator. The iterator is splittable,
 * allowing the framework to create a parallel data stream source that returns the elements in
 * the iterator.
 *
 * <p>Because the iterator will remain unmodified until the actual execution happens, the type
 * of data returned by the iterator must be given explicitly in the form of the type class
 * (this is due to the fact that the Java compiler erases the generic type information).
 *
 * @param iterator
 *         The iterator that produces the elements of the data stream
 * @param type
 *         The class of the data produced by the iterator. Must not be a generic class.
 * @param <OUT>
 *         The type of the returned data stream
 * @return A data stream representing the elements in the iterator
 */
public <OUT> DataStreamSource<OUT> fromParallelCollection(SplittableIterator<OUT> iterator, Class<OUT> type) {
  return fromParallelCollection(iterator, TypeExtractor.getForClass(type));
}

Javadoc

Creates a new data stream that contains elements in the iterator. The iterator is splittable, allowing the framework to create a parallel data stream source that returns the elements in the iterator.

Because the iterator will remain unmodified until the actual execution happens, the type of data returned by the iterator must be given explicitly in the form of the type class (this is due to the fact that the Java compiler erases the generic type information).

Popular methods of StreamExecutionEnvironment

execute
getExecutionEnvironment
Creates an execution environment that represents the context in which the program is currently execu
addSource
Ads a data source with a custom type information thus opening a DataStream. Only in very special cas
getConfig
Gets the config object.
enableCheckpointing
Enables checkpointing for the streaming job. The distributed state of the streaming dataflow will be
setStreamTimeCharacteristic
Sets the time characteristic for all streams create from this environment, e.g., processing time, ev
setParallelism
Sets the parallelism for operations executed through this environment. Setting a parallelism of x he
fromElements
Creates a new data stream that contains the given elements. The elements must all be of the same typ
setStateBackend
Sets the state backend that describes how to store and checkpoint operator state. It defines both wh
createLocalEnvironment
Creates a LocalStreamEnvironment. The local execution environment will run the program in a multi-th
fromCollection
Creates a data stream from the given iterator.Because the iterator will remain unmodified until the
getCheckpointConfig
Gets the checkpoint config, which defines values like checkpoint interval, delay between checkpoints

Popular in Java

Finding current android device location
getOriginalFilename (MultipartFile)
Return the original filename in the client's filesystem.This may contain path information depending
getSharedPreferences (Context)
onRequestPermissionsResult (Fragment)
MalformedURLException (java.net)
This exception is thrown when a program attempts to create an URL from an incorrect specification.
Map (java.util)
A Map is a data structure consisting of a set of keys and values in which each key is mapped to a si
Executor (java.util.concurrent)
An object that executes submitted Runnable tasks. This interface provides a way of decoupling task s
ThreadPoolExecutor (java.util.concurrent)
An ExecutorService that executes each submitted task using one of possibly several pooled threads, n
Stream (java.util.stream)
A sequence of elements supporting sequential and parallel aggregate operations. The following exampl
Modifier (javassist)
The Modifier class provides static methods and constants to decode class and member access modifiers
Top plugins for WebStorm

How to use fromParallelCollectionmethodin org.apache.flink.streaming.api.environment.StreamExecutionEnvironment

Best Java code snippets using org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromParallelCollection (Showing top 11 results out of 315)

How to use
fromParallelCollection
method
in
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment