Avro file input
Description
The Avro file input transform simply reads Avro records from one or more files.
|
Avro file input does not run on the native Spark engine. It produces an Avro Record field, and the native Spark engine has no Spark data type to carry that field from one transform to the next, so the pipeline is refused before it starts. Run the pipeline that reads the Avro files on the Hop engine or on a Beam engine instead. |
Each record is encapsulated in an Avro field, each value has its own Schema and record.
Options
| Option | Description |
|---|---|
Transform name |
Name of the transform. Note: This name has to be unique in a single pipeline. |
Filename field |
Select the field which contains the filename(s) of the Avro files to read |
Avro output field name |
The name of the field which will contain the Avro records |
Maximum number of rows to read |
Specify a positive number to limit the amount of rows read from all files. |