Beam Kinesis Produce Icon Beam Kinesis Produce

Description

The Beam Kinesis Produce transform sends data (messages) to an Amazon Kinesis stream using a Beam pipeline execution engine.

Supported Engines

Hop Engine

Not Supported

Spark

Supported

Flink

Supported

Dataflow

Supported

Limitations

The main limitation of the Kinesis produce transform is that it currently only supports sending String data. Please create a request in JIRA if you want to see support for Avro messages and other formats.

Options

Option Description

Transform name

Name of the transform, this name has to be unique in a single pipeline.

AWS access key

The Amazon Web Services access key to authenticate with.

AWS secret key

The Amazon Web Services secret key to authenticate with.

Stream name

The name of the Kinesis stream to write to.

The name of the data field to send

This is the name of the field that contains the actual data (message)

The type of data to send

Only String data (messages) are supported for now.

Partition key

The partition key to use

Configuration options

A list of configuration options. See here for a list of examples.