Beam Input Icon Beam Input

Description

The Beam Input transform reads files using a file definition with the Beam execution engine.

Supported Engines

Hop Engine

Not Supported

Spark

Supported

Flink

Supported

Dataflow

Supported

Options

Option Description

Transform name

Name of the transform, this name has to be unique in a single pipeline.

Input location

The location from which to read a file.

File definition to use

The file definition to use stored in a metastore.

Edit

Edits an existing file definition.

New

Creates a new file definition.

Manage

Opens the MetaStore Explorer.

File Definition

A file definition can be used to define a file structure definition.

Option Description

Name

The file definition name.

Description

The file definition description.

Field separator

The character separating the fields.

Field enclosure

The character enclosing fields.

Field definitions

A list of fields.

Field name

The field name.

Type

The field data type.

Format

The field format.

Length

The field length.

Precision

The field precisiom.