Tabnine Logo
AvroIO.readGenericRecords
Code IndexAdd Tabnine to your IDE (free)

How to use
readGenericRecords
method
in
org.apache.beam.sdk.io.AvroIO

Best Java code snippets using org.apache.beam.sdk.io.AvroIO.readGenericRecords (Showing top 6 results out of 315)

origin: GoogleCloudPlatform/java-docs-samples

public static void runAvroToCsv(SampleOptions options)
  throws IOException, IllegalArgumentException {
 FileSystems.setDefaultPipelineOptions(options);
 // Get Avro Schema
 String schemaJson = getSchema(options.getAvroSchema());
 Schema schema = new Schema.Parser().parse(schemaJson);
 // Check schema field types before starting the Dataflow job
 checkFieldTypes(schema);
 // Create the Pipeline object with the options we defined above.
 Pipeline pipeline = Pipeline.create(options);
 // Convert Avro To CSV
 pipeline.apply("Read Avro files",
   AvroIO.readGenericRecords(schemaJson).from(options.getInputFile()))
   .apply("Convert Avro to CSV formatted data",
     ParDo.of(new ConvertAvroToCsv(schemaJson, options.getCsvDelimiter())))
   .apply("Write CSV formatted data", TextIO.write().to(options.getOutput())
     .withSuffix(".csv"));
 // Run the pipeline.
 pipeline.run().waitUntilFinish();
}
origin: org.apache.beam/beam-sdks-java-core

/**
 * Reads Avro file(s) containing records of the specified schema. The schema is specified as a
 * JSON-encoded string.
 */
public static Read<GenericRecord> readGenericRecords(String schema) {
 return readGenericRecords(new Schema.Parser().parse(schema));
}
origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testWriteThenReadGeneratedClassWithSchemaString() throws Throwable {
 testWriteThenReadGeneratedClass(
   AvroIO.writeGenericRecords(SCHEMA.toString()),
   AvroIO.readGenericRecords(SCHEMA.toString()));
}
origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(NeedsRunner.class)
public void testWriteThenReadGeneratedClassWithSchema() throws Throwable {
 testWriteThenReadGeneratedClass(
   AvroIO.writeGenericRecords(SCHEMA), AvroIO.readGenericRecords(SCHEMA));
}
origin: org.apache.beam/beam-sdks-java-core

@Test
@Category(ValidatesRunner.class)
public void testPrimitiveReadDisplayData() {
 DisplayDataEvaluator evaluator = DisplayDataEvaluator.create();
 AvroIO.Read<GenericRecord> read =
   AvroIO.readGenericRecords(Schema.create(Schema.Type.STRING)).from("/foo.*");
 Set<DisplayData> displayData = evaluator.displayDataForPrimitiveSourceTransforms(read);
 assertThat(
   "AvroIO.Read should include the file pattern in its primitive transform",
   displayData,
   hasItem(hasDisplayItem("filePattern")));
}
origin: org.apache.beam/beam-sdks-java-core

  readPipeline.apply(
    "read_" + prefix,
    AvroIO.readGenericRecords(schemaFromPrefix(prefix)).from(expectedFilepattern));
PAssert.that(records).containsInAnyOrder(expectedElements.get(prefix));
org.apache.beam.sdk.ioAvroIOreadGenericRecords

Javadoc

Reads Avro file(s) containing records of the specified schema. The schema is specified as a JSON-encoded string.

Popular methods of AvroIO

  • writeGenericRecords
    Writes Avro records of the specified schema.
  • constantDestinations
    Returns a DynamicAvroDestinations that always returns the same FilenamePolicy, schema, metadata, and
  • parseAllGenericRecords
    Like #parseGenericRecords(SerializableFunction), but reads each filepattern in the input PCollection
  • read
    Reads records of the given type from an Avro file (or multiple Avro files matching a pattern).The sc
  • readAll
    Like #read, but reads each filepattern in the input PCollection.
  • write
    Writes a PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
  • writeCustomTypeToGenericRecords
    Similar to #writeCustomType(), but specialized for the case where the output type is GenericRecord.
  • defaultWriteBuilder
  • parseGenericRecords
    Reads Avro file(s) containing records of an unspecified schema and converting each record to a custo
  • readAllGenericRecords
    Like #readGenericRecords(Schema), but reads each filepattern in the input PCollection.
  • sink
    A Sink for use with FileIO#write and FileIO#writeDynamic, writing elements of the given generated cl
  • sinkViaGenericRecords
    A Sink for use with FileIO#write and FileIO#writeDynamic, writing elements by converting each one to
  • sink,
  • sinkViaGenericRecords,
  • writeCustomType

Popular in Java

  • Updating database using SQL prepared statement
  • onCreateOptionsMenu (Activity)
  • addToBackStack (FragmentTransaction)
  • getSystemService (Context)
  • OutputStream (java.io)
    A writable sink for bytes.Most clients will use output streams that write data to the file system (
  • InetAddress (java.net)
    An Internet Protocol (IP) address. This can be either an IPv4 address or an IPv6 address, and in pra
  • ExecutorService (java.util.concurrent)
    An Executor that provides methods to manage termination and methods that can produce a Future for tr
  • ServletException (javax.servlet)
    Defines a general exception a servlet can throw when it encounters difficulty.
  • LogFactory (org.apache.commons.logging)
    Factory for creating Log instances, with discovery and configuration features similar to that employ
  • Get (org.apache.hadoop.hbase.client)
    Used to perform Get operations on a single row. To get everything for a row, instantiate a Get objec
  • Top plugins for Android Studio
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now