Tabnine Logo
StdUDF.processRequiredFiles
Code IndexAdd Tabnine to your IDE (free)

How to use
processRequiredFiles
method
in
com.linkedin.transport.api.udf.StdUDF

Best Java code snippets using com.linkedin.transport.api.udf.StdUDF.processRequiredFiles (Showing top 5 results out of 315)

origin: linkedin/transport

 private synchronized void processRequiredFiles(String[] requiredFiles) {
  if (!_requiredFilesProcessed) {
   _stdUdf.processRequiredFiles(Arrays.stream(requiredFiles)
     .map(path -> Path.getPathWithoutSchemeAndAuthority(new Path(path)).toString())
     .toArray(String[]::new));
   _requiredFilesProcessed = true;
  }
 }
}
origin: com.linkedin.transport/transportable-udfs-hive

private synchronized void processRequiredFiles() {
 if (!_requiredFilesProcessed) {
  String[] localFiles = Arrays.stream(_distributedCacheFiles).map(distributedCacheFile -> {
     try {
      return getLocalFilePath(distributedCacheFile).toString();
     } catch (IOException e) {
      throw new RuntimeException("Failed to resolve path: [" + distributedCacheFile + "].", e);
     }
    }
  ).toArray(String[]::new);
  _stdUdf.processRequiredFiles(localFiles);
  _requiredFilesProcessed = true;
 }
}
origin: linkedin/transport

private synchronized void processRequiredFiles() {
 if (!_requiredFilesProcessed) {
  String[] localFiles = Arrays.stream(_distributedCacheFiles).map(distributedCacheFile -> {
     try {
      return getLocalFilePath(distributedCacheFile).toString();
     } catch (IOException e) {
      throw new RuntimeException("Failed to resolve path: [" + distributedCacheFile + "].", e);
     }
    }
  ).toArray(String[]::new);
  _stdUdf.processRequiredFiles(localFiles);
  _requiredFilesProcessed = true;
 }
}
origin: com.linkedin.transport/transportable-udfs-presto

private synchronized void processRequiredFiles(StdUDF stdUDF, String[] requiredFiles) {
 if (_requiredFilesNextRefreshTime < System.currentTimeMillis()) {
  try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(getClass().getClassLoader())) {
   String[] copiedFiles = new String[requiredFiles.length];
   FileSystemClient client = new FileSystemClient();
   for (int i = 0; i < requiredFiles.length; i++) {
    String localFilename = client.copyToLocalFile(requiredFiles[i]);
    copiedFiles[i] = localFilename;
   }
   stdUDF.processRequiredFiles(copiedFiles);
   // Determine how many times _refreshIntervalMillis needs to be added to go above currentTimeMillis
   int refreshIntervalFactor = (int) Math.ceil(
     (System.currentTimeMillis() - _requiredFilesNextRefreshTime) / (double) getRefreshIntervalMillis());
   _requiredFilesNextRefreshTime += getRefreshIntervalMillis() * Math.max(1, refreshIntervalFactor);
  }
 }
}
origin: linkedin/transport

private synchronized void processRequiredFiles(StdUDF stdUDF, String[] requiredFiles) {
 if (_requiredFilesNextRefreshTime < System.currentTimeMillis()) {
  try (ThreadContextClassLoader ignored = new ThreadContextClassLoader(getClass().getClassLoader())) {
   String[] copiedFiles = new String[requiredFiles.length];
   FileSystemClient client = new FileSystemClient();
   for (int i = 0; i < requiredFiles.length; i++) {
    String localFilename = client.copyToLocalFile(requiredFiles[i]);
    copiedFiles[i] = localFilename;
   }
   stdUDF.processRequiredFiles(copiedFiles);
   // Determine how many times _refreshIntervalMillis needs to be added to go above currentTimeMillis
   int refreshIntervalFactor = (int) Math.ceil(
     (System.currentTimeMillis() - _requiredFilesNextRefreshTime) / (double) getRefreshIntervalMillis());
   _requiredFilesNextRefreshTime += getRefreshIntervalMillis() * Math.max(1, refreshIntervalFactor);
  }
 }
}
com.linkedin.transport.api.udfStdUDFprocessRequiredFiles

Javadoc

Processes the localized files for the StdUDF. This method is called before any records are processed. The Standard UDF framework localizes the files passed through getRequiredFiles() and provides the localized file paths for further processing such as building of lookup tables.

Popular methods of StdUDF

  • getAndCheckNullableArguments
    Returns an array of booleans indicating if any input argument is nullable and also verifies its leng
  • init
    Performs necessary initializations for a StdUDF. This method is called before any records are proces
  • getInputParameterSignatures
    Returns a List of type signature strings representing the input parameters to the UDF
  • getOutputParameterSignature
    Returns a type signature string representing the output parameter to the UDF
  • getNullableArguments
    Returns an array of booleans indicating if any input argument is nullable. Nullable arguments are ar
  • getStdFactory
    Returns a StdFactory object which can be used to create StdData and StdType objects
  • numberOfArguments
    Returns the number of input arguments for the StdUDF

Popular in Java

  • Parsing JSON documents to java classes using gson
  • setContentView (Activity)
  • onRequestPermissionsResult (Fragment)
  • onCreateOptionsMenu (Activity)
  • Kernel (java.awt.image)
  • NumberFormat (java.text)
    The abstract base class for all number formats. This class provides the interface for formatting and
  • TreeSet (java.util)
    TreeSet is an implementation of SortedSet. All optional operations (adding and removing) are support
  • ReentrantLock (java.util.concurrent.locks)
    A reentrant mutual exclusion Lock with the same basic behavior and semantics as the implicit monitor
  • ImageIO (javax.imageio)
  • DateTimeFormat (org.joda.time.format)
    Factory that creates instances of DateTimeFormatter from patterns and styles. Datetime formatting i
  • Github Copilot alternatives
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now