Beam Kinesis Produce

Description

The Beam Kinesis Produce transform sends data (messages) to an Amazon Kinesis stream using a Beam pipeline execution engine.

Supported Engines

Hop Engine

Not Supported

Spark

Supported

Flink

Supported

Dataflow

Supported

Limitations

The main limitation of the Kinesis produce transform is that it currently only supports sending String data. Please create a request in JIRA if you want to see support for Avro messages and other formats.

Options

Option	Description
Transform name	Name of the transform, this name has to be unique in a single pipeline.
AWS access key	The Amazon Web Services access key to authenticate with.
AWS secret key	The Amazon Web Services secret key to authenticate with.
Stream name	The name of the Kinesis stream to write to.
The name of the data field to send	This is the name of the field that contains the actual data (message)
The type of data to send	Only String data (messages) are supported for now.
Partition key	The partition key to use
Configuration options	A list of configuration options. See here for a list of examples.

Option

Description

Transform name

Name of the transform, this name has to be unique in a single pipeline.

AWS access key

The Amazon Web Services access key to authenticate with.

AWS secret key

The Amazon Web Services secret key to authenticate with.

Stream name

The name of the Kinesis stream to write to.

The name of the data field to send

This is the name of the field that contains the actual data (message)

The type of data to send

Only String data (messages) are supported for now.

Partition key

The partition key to use

Configuration options

A list of configuration options. See here for a list of examples.