Beam partition
Description
The Beam partition transform decides, before the following transforms run, which partition each row belongs to.
This is the counterpart of the partitioning option in the local Hop pipeline engine, and it wraps the Beam Partition transform.
Partitioning does not change the rows themselves: the same rows with the same columns come out, in a possibly different order.
What partitioning is for
On a distributed runner, a transform that groups rows, such as a Beam window on a key, shuffles so that rows sharing a key end up in the same place.
This transform does not remove that shuffle. It assigns each row to a partition and then flattens the partitions back into one stream. A following GroupByKey still shuffles, whether or not it groups on the same field.
Choosing a partitioning type
Single partition sends every row to one partition. It does not serialise the processing that follows. After this transform the rows are one stream again, and the runner schedules the rest of the pipeline as it would without this transform. It is the default, which means a pipeline saved without a partitioning type does not partition.
Keyed partitions on the value of a field: rows with the same value are sent to the same partition, then flattened back into that one stream with every other partition.
| The partition is chosen by hashing the key, so rows are spread over the partitions in no particular order and not evenly. Two keys can share a partition, and some partitions can be heavier than others. |
Options
| Option | Description |
|---|---|
Transform name |
Name of the transform, this name has to be unique in a single pipeline. |
Partitioning type |
|
Key field |
Required for the keyed type: the field to partition on. Ignored for the single-partition type. |
Number of partitions |
How many partitions to create. Defaults to 1. Only used for the keyed type. |
| A key field that is not in the input fails the pipeline with a message naming it, and keyed partitioning without a key field is refused too. Neither is silently ignored. |
Notes on the Beam engines
The number of partitions is decided when the pipeline is built. Changing it therefore changes the size of the resulting Beam graph, which matters on runners with a large number of workers.
After partitioning, Beam hands back one collection per partition. Hop recombines them into a single stream before the next transform, because the partition of a given row is only decided while the pipeline runs and the next transform cannot branch on it. That means partitioning on its own does not filter or split the stream, and the partition does not stick for the transform that follows.