Snowflake ID Generator

Description

The Snowflake ID Generator transform generates globally unique identifiers (IDs) based on the Snowflake algorithm. Each generated ID is a 64-bit long integer that encodes a timestamp, data center ID, machine ID, and sequence number, ensuring uniqueness across distributed systems.

Use this transform to generate unique IDs for your data rows in distributed ETL pipelines. The generated IDs are time-ordered and can be used as primary keys, message identifiers, or other globally unique keys in data processing systems.

Typical use cases:

Generating unique keys for records before inserting into a database.
Ensuring globally unique event IDs in streaming data pipelines.
Replacing database auto-increment IDs in distributed environments.

Supported Engines

Hop Engine

Supported

Single Threaded

Supported

Native Spark

Supported

Beam Spark

Maybe Supported

Beam Flink

Maybe Supported

Beam Dataflow

Maybe Supported

Options

Option	Description
Transform name	The name of this transform as it appears in the pipeline workspace. This name must be unique within a single pipeline.
Name of value	Name of the new snowflake value that is added to the stream.
Data Center No	Identifier for the data center or logical group. Valid range: 0–31 (default: 1).
Machine No	Identifier for the machine generating the ID. Valid range: 0–31 (default: 1).

Option

Description

Transform name

The name of this transform as it appears in the pipeline workspace. This name must be unique within a single pipeline.

Name of value

Name of the new snowflake value that is added to the stream.

Data Center No

Identifier for the data center or logical group. Valid range: 0–31 (default: 1).

Machine No

Identifier for the machine generating the ID. Valid range: 0–31 (default: 1).