Tabnine Logo
PCollection.getSize
Code IndexAdd Tabnine to your IDE (free)

How to use
getSize
method
in
org.apache.crunch.PCollection

Best Java code snippets using org.apache.crunch.PCollection.getSize (Showing top 1 results out of 315)

origin: org.apache.crunch/crunch-core

 public static <T> int getRecommendedPartitions(PCollection<T> pcollection, Configuration conf) {
  long bytesPerTask = conf.getLong(BYTES_PER_REDUCE_TASK, DEFAULT_BYTES_PER_REDUCE_TASK);
  Preconditions.checkArgument(bytesPerTask > 0);
  int recommended = 1 + (int) (pcollection.getSize() / bytesPerTask);
  int maxRecommended = conf.getInt(MAX_REDUCERS, DEFAULT_MAX_REDUCERS);
  if (maxRecommended > 0 && recommended > maxRecommended) {
   return maxRecommended;
  } else {
   return recommended;
  }
 }
}
org.apache.crunchPCollectiongetSize

Javadoc

Returns the size of the data represented by this PCollection in bytes.

Popular methods of PCollection

  • parallelDo
    Applies the given doFn to the elements of this PCollection and returns a new PCollection that is the
  • getPType
    Returns the PType of this PCollection.
  • by
    Apply the given map function to each element of this instance in order to create a PTable.
  • write
    Write the contents of this PCollection to the given Target, using the given Target.WriteMode to hand
  • materialize
    Returns a reference to the data set represented by this PCollection that may be used by the client t
  • getPipeline
    Returns the Pipeline associated with this PCollection.
  • getTypeFamily
    Returns the PTypeFamily of this PCollection.
  • count
    Returns a PTable instance that contains the counts of each unique element of this PCollection.
  • aggregate
    Returns a PCollection that contains the result of aggregating all values in this instance.
  • asReadable
  • cache
    Marks this data as cached using the given CachingOptions. Cached PCollections will only be processed
  • filter
    Apply the given filter function to this instance and return the resulting PCollection.
  • cache,
  • filter,
  • first,
  • getName,
  • union

Popular in Java

  • Making http post requests using okhttp
  • getContentResolver (Context)
  • onCreateOptionsMenu (Activity)
  • scheduleAtFixedRate (ScheduledExecutorService)
  • OutputStream (java.io)
    A writable sink for bytes.Most clients will use output streams that write data to the file system (
  • BigDecimal (java.math)
    An immutable arbitrary-precision signed decimal.A value is represented by an arbitrary-precision "un
  • SocketTimeoutException (java.net)
    This exception is thrown when a timeout expired on a socket read or accept operation.
  • HttpServlet (javax.servlet.http)
    Provides an abstract class to be subclassed to create an HTTP servlet suitable for a Web site. A sub
  • JButton (javax.swing)
  • JTextField (javax.swing)
  • Best plugins for Eclipse
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now