Beam MQTT output

Description

Publish one MQTT message for each incoming Hop row using Apache Beam MqttIO. Connect exactly one incoming transform. This is a terminal sink; it does not produce downstream rows. The configured topic must not contain subscription wildcards.

Options

Option Description

Option	Description
Transform name	Unique name in the pipeline.
Server URI	Required broker URI, `tcp://host:1883` or `ssl://host:8883`. Do not embed credentials in the URI.
Topic	Required fixed publish topic.
Client ID prefix	Optional prefix. Beam appends a UUID and truncates the resulting client ID to 23 characters. Leave empty for a random ID; use a short prefix to preserve random suffix characters.
Username	Optional broker username.
Password	Optional broker password, requires a username. Masked in the dialog and encrypted with Hop password conventions in saved pipelines. Prefer a secret variable such as `${MQTT_PASSWORD}`.
Payload field	Required incoming field to publish (default `message`).
Payload type	`String` or `Binary`. String uses Hop value metadata to convert to text, then encodes UTF-8. Binary requires a Hop Binary field and preserves the bytes. Null payloads fail the pipeline; empty payloads are allowed.
Retain messages	Sets the MQTT retained flag. Default is off. A retained publish replaces the broker’s retained message; a retained empty payload clears it.

Transform name

Unique name in the pipeline.

Server URI

Required broker URI, tcp://host:1883 or ssl://host:8883. Do not embed credentials in the URI.

Topic

Required fixed publish topic.

Client ID prefix

Optional prefix. Beam appends a UUID and truncates the resulting client ID to 23 characters. Leave empty for a random ID; use a short prefix to preserve random suffix characters.

Username

Optional broker username.

Password

Optional broker password, requires a username. Masked in the dialog and encrypted with Hop password conventions in saved pipelines. Prefer a secret variable such as ${MQTT_PASSWORD}.

Payload field

Required incoming field to publish (default message).

Payload type

String or Binary. String uses Hop value metadata to convert to text, then encodes UTF-8. Binary requires a Hop Binary field and preserves the bytes. Null payloads fail the pipeline; empty payloads are allowed.

Retain messages

Sets the MQTT retained flag. Default is off. A retained publish replaces the broker’s retained message; a retained empty payload clears it.

All text options support Hop variables and are resolved before the Beam graph is submitted. Required options and invalid numeric limits fail before connecting.

Execution and security

This transform requires a Beam pipeline run configuration; the native Hop engine reports an error. The broker must be reachable from the runner’s worker machines, not just Hop GUI.

Beam MqttIO 2.74.0 publishes with MQTT QoS 1 (at least once) and waits for the broker PUBACK inside its writer. That wait is not a Hop callback, so this transform does not expose an acknowledged-write counter. Hop’s lines-read and lines-output counts are payload conversion: a row was read, or its field was turned into bytes and handed to MqttIO. Lines written stays empty. A converted payload is not a confirmed publish. QoS and reconnect settings are not dialog options. The MQTT client reconnects without a fixed attempt limit (the default is unlimited, starting at 10 ms and backing off to 30 s). A publish that was not acknowledged is sent again after reconnect, so the broker can receive the same payload more than once. The pipeline does not finish successfully until that PUBACK arrives. Connecting waits up to one minute and retries on timeout. A refused connection fails the pipeline. A broker response that acknowledges a different packet than the one in flight also fails the pipeline. A socket that closes before PUBACK does not, by itself, fail the pipeline: the client keeps reconnecting and the publish wait continues. The sink publishes to one configured topic, not a per-row dynamic topic.

Use ssl:// for credentials or sensitive data. TLS uses the worker JVM’s default trust configuration; install the broker’s certificate chain in its trust store. Custom trust stores, client certificates, reconnect controls and QoS selection are not exposed by Beam’s connection API and are not dialog options. tcp:// is not encrypted.

Hop password encryption protects saved metadata from plain-text disclosure; it is not a substitute for a secret manager. Beam’s connection specification must carry usable credentials to workers: protect graph serialization, worker storage and transport. Do not enable Beam MqttIO debug logging for sensitive payloads (the upstream writer logs message text at DEBUG).