Avro decode

Description

The Avro decode transform allows you to decode an Avro field and convert it to Hop fields.

Avro decode does not run on the native Spark engine. It decodes an Avro Record field, and the native Spark engine has no Spark data type to carry such a field from one transform to the next, so the pipeline is refused before it starts.

Options

Option Description

Option	Description
Transform name	Name of the transform. Note: This name has to be unique in a single pipeline.
Source field	Select the name of the field (of type: Avro) to convert
Source fields	You can specify the names of the Avro fields to select from the Avro records. You can also specify to which Hop data type you want to convert to. Please note that complex data types like Map and Record are converted into JSON which you can then parse further in subsequent transforms. Note: The "Avro type" column is informational only. It’s not used at runtime.
Get fields from file	Retrieves fields from the schema on the selected Avro Record source field, when that field already has one (for example after Avro Encode). If there is no schema, pick an Avro file. The fields are taken from the first record. Rows are added to the table.
Get fields from JSON	Paste JSON instead of reading a file. Three forms are accepted: a Hop Avro value with `schema` and `data`, an Avro record schema, or one sample record (the JSON shown when previewing an Avro value, including from Kafka). A sample record’s types are a best effort: JSON null becomes String, and only the top-level fields are added. Rows are added to the table.

Transform name

Name of the transform. Note: This name has to be unique in a single pipeline.

Source field

Select the name of the field (of type: Avro) to convert

Source fields

You can specify the names of the Avro fields to select from the Avro records. You can also specify to which Hop data type you want to convert to. Please note that complex data types like Map and Record are converted into JSON which you can then parse further in subsequent transforms. Note: The "Avro type" column is informational only. It’s not used at runtime.

Get fields from file

Retrieves fields from the schema on the selected Avro Record source field, when that field already has one (for example after Avro Encode). If there is no schema, pick an Avro file. The fields are taken from the first record. Rows are added to the table.

Get fields from JSON

Paste JSON instead of reading a file. Three forms are accepted: a Hop Avro value with schema and data, an Avro record schema, or one sample record (the JSON shown when previewing an Avro value, including from Kafka). A sample record’s types are a best effort: JSON null becomes String, and only the top-level fields are added. Rows are added to the table.