Row Denormaliser transform Icon Row Denormaliser

Description

The De-normalizer transform allows you de-normalize data by looking up key-value pairs, with the option to convert data types in the process.

Note: make sure to check the notes on this transform in the Getting started with Beam documentation.

Supported Engines

Hop Engine

Supported

Spark

Maybe Supported

Flink

Maybe Supported

Dataflow

Maybe Supported

Options

Transform name Name of the transform. This name has to be unique in a single pipeline.

Key field

The field that defined the key of the output row.

Group fields

Specify the fields that make up the grouping here.

Target fields

Select the fields to de-normalize by specifying the String value for the key field (see above). Options are provided to convert data types. Strings are most common as key-value pairs so you must often convert to Integer, Number or Date. If you get key-value pair collisions (key is not unique for the group specified) specify the aggregation method to use.

Metadata Injection Support

You can use the Metadata Injection supported fields with ETL Metadata Injection transform to pass metadata to your pipeline at runtime. All fields can be injected, the values used for the aggregation field are the following

key

value

TYPE_AGGR_NONE

No Aggregation is done

TYPE_AGGR_SUM

Sum all values

TYPE_AGGR_AVERAGE

Calculate the average

TYPE_AGGR_MIN

Take the minimal value of the group

TYPE_AGGR_MAX

Take the maximum value of the group

TYPE_AGGR_COUNT_ALL

Count rows

TYPE_AGGR_CONCAT_COMMA

Aggragate values separated by comma

Example

CustomerId Key Value

101

COUSINS_FIRST_NAME

Mary

101

COUSINS_SECOND_NAME

J.

101

COUSINS_LAST_NAME

Blige

101

COUSINS_BIRTH_DATE

1969/02/14

101

COUSINS_INCOME

1723.86

101

…​45 more keys…​

…​