Description

The Data Stream Output transform writes rows from a Hop pipeline to a configured Data Stream.

It supports all available Data Stream implementations (Arrow File Stream, Arrow Random Access File, and Arrow Flight).

Supported Engines

Hop Engine

Supported

Single Threaded

Supported

Native Spark

Supported

Beam Spark

Supported

Beam Flink

Supported

Beam Dataflow

Supported

Options

Option Description

Data Stream

The name of the target Data Stream to write to. This is a dropdown that shows all defined Data Streams in the project.

Notes

  • The transform will automatically use the implementation defined in the selected Data Stream (file-based or Flight).

  • When using Arrow Flight, data is sent directly to the running Flight server using the stream name.