congrats Icon
New! Announcing our next generation AI code completions
Read here
Tabnine Logo
StreamExecutionEnvironment.registerCachedFile
Code IndexAdd Tabnine to your IDE (free)

How to use
registerCachedFile
method
in
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment

Best Java code snippets using org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.registerCachedFile (Showing top 9 results out of 315)

origin: apache/flink

/**
 * Registers a file at the distributed cache under the given name. The file will be accessible
 * from any user-defined function in the (distributed) runtime under a local path. Files
 * may be local files (which will be distributed via BlobServer), or files in a distributed file system.
 * The runtime will copy the files temporarily to a local cache, if needed.
 *
 * <p>The {@link org.apache.flink.api.common.functions.RuntimeContext} can be obtained inside UDFs via
 * {@link org.apache.flink.api.common.functions.RichFunction#getRuntimeContext()} and provides access
 * {@link org.apache.flink.api.common.cache.DistributedCache} via
 * {@link org.apache.flink.api.common.functions.RuntimeContext#getDistributedCache()}.
 *
 * @param filePath The path of the file, as a URI (e.g. "file:///some/path" or "hdfs://host:port/and/path")
 * @param name The name under which the file is registered.
 */
public void registerCachedFile(String filePath, String name) {
  registerCachedFile(filePath, name, false);
}
origin: apache/flink

private void distributeFiles() throws IOException {
  this.env.registerCachedFile(pythonTmpCachePath.getPath(), PythonConstants.FLINK_PYTHON_DC_ID);
}
origin: apache/flink

public static void main(String[] args) throws Exception {
  final ParameterTool params = ParameterTool.fromArgs(args);
  final Path inputFile = Paths.get(params.getRequired("inputFile"));
  final Path inputDir = Paths.get(params.getRequired("inputDir"));
  final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  env.setParallelism(1);
  env.registerCachedFile(inputFile.toString(), "test_data", false);
  env.registerCachedFile(inputDir.toString(), "test_dir", false);
  final Path containedFile;
  try (Stream<Path> files = Files.list(inputDir)) {
    containedFile = files.findAny().orElseThrow(() -> new RuntimeException("Input directory must not be empty."));
  }
  env.fromElements(1)
    .map(new TestMapFunction(
      inputFile.toAbsolutePath().toString(),
      Files.size(inputFile),
      inputDir.toAbsolutePath().toString(),
      containedFile.getFileName().toString()))
    .writeAsText(params.getRequired("output"), FileSystem.WriteMode.OVERWRITE);
  env.execute("Distributed Cache Via Blob Test Program");
}
origin: apache/flink

@Test
public void testStreamingDistributedCache() throws Exception {
  String textPath = createTempFile("count.txt", DATA);
  StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  env.registerCachedFile(textPath, "cache_test");
  env.readTextFile(textPath).flatMap(new WordChecker());
  env.execute();
}
origin: DTStack/flinkx

/**
 * Registers a file at the distributed cache under the given name. The file will be accessible
 * from any user-defined function in the (distributed) runtime under a local path. Files
 * may be local files (as long as all relevant workers have access to it), or files in a distributed file system.
 * The runtime will copy the files temporarily to a local cache, if needed.
 *
 * <p>The {@link org.apache.flink.api.common.functions.RuntimeContext} can be obtained inside UDFs via
 * {@link org.apache.flink.api.common.functions.RichFunction#getRuntimeContext()} and provides access
 * {@link org.apache.flink.api.common.cache.DistributedCache} via
 * {@link org.apache.flink.api.common.functions.RuntimeContext#getDistributedCache()}.
 *
 * @param filePath The path of the file, as a URI (e.g. "file:///some/path" or "hdfs://host:port/and/path")
 * @param name The name under which the file is registered.
 */
public void registerCachedFile(String filePath, String name) {
  registerCachedFile(filePath, name, false);
}
origin: org.apache.flink/flink-streaming-java_2.11

/**
 * Registers a file at the distributed cache under the given name. The file will be accessible
 * from any user-defined function in the (distributed) runtime under a local path. Files
 * may be local files (which will be distributed via BlobServer), or files in a distributed file system.
 * The runtime will copy the files temporarily to a local cache, if needed.
 *
 * <p>The {@link org.apache.flink.api.common.functions.RuntimeContext} can be obtained inside UDFs via
 * {@link org.apache.flink.api.common.functions.RichFunction#getRuntimeContext()} and provides access
 * {@link org.apache.flink.api.common.cache.DistributedCache} via
 * {@link org.apache.flink.api.common.functions.RuntimeContext#getDistributedCache()}.
 *
 * @param filePath The path of the file, as a URI (e.g. "file:///some/path" or "hdfs://host:port/and/path")
 * @param name The name under which the file is registered.
 */
public void registerCachedFile(String filePath, String name) {
  registerCachedFile(filePath, name, false);
}
origin: org.apache.flink/flink-streaming-java_2.10

/**
 * Registers a file at the distributed cache under the given name. The file will be accessible
 * from any user-defined function in the (distributed) runtime under a local path. Files
 * may be local files (as long as all relevant workers have access to it), or files in a distributed file system.
 * The runtime will copy the files temporarily to a local cache, if needed.
 *
 * <p>The {@link org.apache.flink.api.common.functions.RuntimeContext} can be obtained inside UDFs via
 * {@link org.apache.flink.api.common.functions.RichFunction#getRuntimeContext()} and provides access
 * {@link org.apache.flink.api.common.cache.DistributedCache} via
 * {@link org.apache.flink.api.common.functions.RuntimeContext#getDistributedCache()}.
 *
 * @param filePath The path of the file, as a URI (e.g. "file:///some/path" or "hdfs://host:port/and/path")
 * @param name The name under which the file is registered.
 */
public void registerCachedFile(String filePath, String name) {
  registerCachedFile(filePath, name, false);
}
origin: org.apache.flink/flink-streaming-java

/**
 * Registers a file at the distributed cache under the given name. The file will be accessible
 * from any user-defined function in the (distributed) runtime under a local path. Files
 * may be local files (which will be distributed via BlobServer), or files in a distributed file system.
 * The runtime will copy the files temporarily to a local cache, if needed.
 *
 * <p>The {@link org.apache.flink.api.common.functions.RuntimeContext} can be obtained inside UDFs via
 * {@link org.apache.flink.api.common.functions.RichFunction#getRuntimeContext()} and provides access
 * {@link org.apache.flink.api.common.cache.DistributedCache} via
 * {@link org.apache.flink.api.common.functions.RuntimeContext#getDistributedCache()}.
 *
 * @param filePath The path of the file, as a URI (e.g. "file:///some/path" or "hdfs://host:port/and/path")
 * @param name The name under which the file is registered.
 */
public void registerCachedFile(String filePath, String name) {
  registerCachedFile(filePath, name, false);
}
origin: DTStack/flinkStreamSQL

for(URL url : classPathSet){
  String classFileName = String.format(CLASS_FILE_NAME_FMT, i);
  env.registerCachedFile(url.getPath(),  classFileName, true);
  i++;
org.apache.flink.streaming.api.environmentStreamExecutionEnvironmentregisterCachedFile

Javadoc

Registers a file at the distributed cache under the given name. The file will be accessible from any user-defined function in the (distributed) runtime under a local path. Files may be local files (which will be distributed via BlobServer), or files in a distributed file system. The runtime will copy the files temporarily to a local cache, if needed.

The org.apache.flink.api.common.functions.RuntimeContext can be obtained inside UDFs via org.apache.flink.api.common.functions.RichFunction#getRuntimeContext() and provides access org.apache.flink.api.common.cache.DistributedCache via org.apache.flink.api.common.functions.RuntimeContext#getDistributedCache().

Popular methods of StreamExecutionEnvironment

  • execute
  • getExecutionEnvironment
    Creates an execution environment that represents the context in which the program is currently execu
  • addSource
    Ads a data source with a custom type information thus opening a DataStream. Only in very special cas
  • getConfig
    Gets the config object.
  • enableCheckpointing
    Enables checkpointing for the streaming job. The distributed state of the streaming dataflow will be
  • setStreamTimeCharacteristic
    Sets the time characteristic for all streams create from this environment, e.g., processing time, ev
  • setParallelism
    Sets the parallelism for operations executed through this environment. Setting a parallelism of x he
  • fromElements
    Creates a new data stream that contains the given elements. The elements must all be of the same typ
  • setStateBackend
    Sets the state backend that describes how to store and checkpoint operator state. It defines both wh
  • createLocalEnvironment
    Creates a LocalStreamEnvironment. The local execution environment will run the program in a multi-th
  • fromCollection
    Creates a data stream from the given iterator.Because the iterator will remain unmodified until the
  • getCheckpointConfig
    Gets the checkpoint config, which defines values like checkpoint interval, delay between checkpoints
  • fromCollection,
  • getCheckpointConfig,
  • getParallelism,
  • getStreamGraph,
  • setRestartStrategy,
  • socketTextStream,
  • readTextFile,
  • generateSequence,
  • clean,
  • getStreamTimeCharacteristic

Popular in Java

  • Parsing JSON documents to java classes using gson
  • onCreateOptionsMenu (Activity)
  • addToBackStack (FragmentTransaction)
  • startActivity (Activity)
  • ServerSocket (java.net)
    This class represents a server-side socket that waits for incoming client connections. A ServerSocke
  • URL (java.net)
    A Uniform Resource Locator that identifies the location of an Internet resource as specified by RFC
  • Dictionary (java.util)
    Note: Do not use this class since it is obsolete. Please use the Map interface for new implementatio
  • Executor (java.util.concurrent)
    An object that executes submitted Runnable tasks. This interface provides a way of decoupling task s
  • ExecutorService (java.util.concurrent)
    An Executor that provides methods to manage termination and methods that can produce a Future for tr
  • IsNull (org.hamcrest.core)
    Is the value null?
  • PhpStorm for WordPress
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimAtomGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyStudentsTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now