OpenLineage

OpenLineage is an open standard for lineage metadata collection. The OpenLineage technology plugin is a Hop lineage sink: it subscribes to the in-engine lineage observation hub and maps Hop’s neutral lineage events to spec-correct OpenLineage RunEvent JSON, which it posts to any OpenLineage-compatible collector. The primary target is Marquez.

Events are generated by the spec’s official openlineage-java client, delivered asynchronously off the pipeline thread, retried on transient HTTP failures, and (by default) buffered losslessly under back-pressure.

Enabling the sink

The lineage hub is disabled by default. Enable it and select the OpenLineage sink with these engine-scoped variables (see the lineage hub for the hub-wide variables):

Variable Default Description

HOP_LINEAGE_ENABLED

N

Set to Y to enable the lineage hub.

HOP_LINEAGE_SINK_IDS

(empty)

Set to openlineage to load this sink.

The sink itself reads the following variables when it initializes:

Variable Default Description

HOP_LINEAGE_OPENLINEAGE_URL

(required)

Collector URL, e.g. http://marquez-api:5000/api/v1/lineage. Must be an absolute http(s) URL.

HOP_LINEAGE_OPENLINEAGE_NAMESPACE

hop

OpenLineage job namespace (the producing Hop environment).

HOP_LINEAGE_OPENLINEAGE_PRODUCER

(plugin URL)

OpenLineage producer field.

HOP_LINEAGE_OPENLINEAGE_SQL_PARSE

Y

Parse SELECT/INSERT statements (Table Input, Execute SQL) to recover their source/target tables and column lineage. Set to N to disable SQL parsing.

HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING

N

Y names file datasets by the OpenLineage naming spec (scheme → namespace, path → name, e.g. s3://bucket + key) so a file reconciles across engines/tools; N keeps the legacy job-namespace + filename identity. Switching re-keys existing file datasets.

HOP_LINEAGE_OPENLINEAGE_API_KEY

(empty)

Optional bearer token sent as Authorization: Bearer ….

HOP_LINEAGE_OPENLINEAGE_CONNECT_TIMEOUT_MS

5000

HTTP connect timeout.

HOP_LINEAGE_OPENLINEAGE_READ_TIMEOUT_MS

30000

HTTP read/response timeout.

HOP_LINEAGE_OPENLINEAGE_BUFFER_SIZE

10000

Async outbound queue capacity.

HOP_LINEAGE_OPENLINEAGE_MAX_RETRIES

3

Retries per event on transient HTTP failure (exponential backoff).

HOP_LINEAGE_OPENLINEAGE_RETRY_BACKOFF_MS

500

Initial retry backoff; doubles each attempt.

HOP_LINEAGE_OPENLINEAGE_OVERFLOW_POLICY

DROP

What to do when the outbound buffer is full. DROP = drop the event and count it, so lineage never slows the pipeline down. BLOCK = wait for capacity, bounded by HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS, for deployments that would rather lose throughput than lose lineage.

HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS

5000

Under BLOCK, how long (ms) to wait for buffer capacity before dropping the event.

HOP_LINEAGE_OPENLINEAGE_SHUTDOWN_DRAIN_MS

30000

Maximum time to drain the queue on shutdown.

HOP_LINEAGE_OPENLINEAGE_METRICS_INTERVAL_MS

0

If > 0, periodically log sent/failed/dropped/queued counts (0 = off).

HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PATH

(empty)

Trust/key store for HTTPS collectors with a custom CA/cert.

HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PASSWORD

(empty)

Store password (used with the trust store path).

HOP_LINEAGE_OPENLINEAGE_KEYSTORE_TYPE

PKCS12

Trust store type (e.g. PKCS12, JKS).

Events are buffered in memory and delivered by a single ordered worker thread; on a hard crash undelivered events are lost. There is no proxy setting — route via the JVM -Dhttps.proxyHost properties if needed.

Event mapping

Hop lineage event OpenLineage output

RUN_LIFECYCLE

RunEvent — START / COMPLETE / FAIL.

FILE_IO

RunEvent (OTHER) with physical file datasets (dataSource facet, plus a schema facet when Hop provides the file’s column schema), correlated to the parent pipeline run.

RELATIONAL_IO

RunEvent (OTHER) with relational table datasets — see Database lineage. Carries a schema facet, a columnLineage facet when column mappings are derivable, and a lifecycleStateChange facet for truncate/overwrite writes.

TRANSFORM_SCHEMA

Skipped (internal row shapes between transforms, not physical datasets).

HTTP_IO

Skipped.

The run id is the executing pipeline/workflow log-channel id; file I/O is correlated to the parent pipeline run so inputs/outputs appear on the pipeline job in Marquez. Transform and action run events carry a ParentRunFacet so their jobs attach to the parent pipeline/workflow run in the lineage graph rather than appearing as disconnected jobs.

Database lineage

RELATIONAL_IO events record the relational tables a transform reads from or writes to. The physical dataset is identified by a database namespace (e.g. postgres://host:5432) and a fully-qualified database.schema.table name (see Dataset identity), so a Hop write and, for example, a dbt read of the same table land on one node in the lineage graph.

The following transforms emit relational lineage:

  • Reads — Table Input (its SELECT is parsed to recover the source tables and column lineage).

  • Writes — Table Output, Insert / Update, Update, Dimension Lookup/Update, Combination Lookup/Update, and the database bulk loaders (CrateDB, MonetDB, MySQL, Oracle, PostgreSQL, Vertica).

  • Deletes — Delete reports the affected table only. A delete produces no columns, so the event carries neither a column schema nor column lineage.

  • Reads and writes — Execute SQL (its statements are parsed for both input and output tables).

SQL parsing

For transforms that only provide a SQL statement (Table Input, Execute SQL), the sink recovers the tables — and, for an INSERT … SELECT, the column-level lineage — by parsing the statement with the OpenLineage SQL parser. Parsing is on by default and controlled by HOP_LINEAGE_OPENLINEAGE_SQL_PARSE. Tables named without a schema or catalog are qualified using the connection’s default database and preferred schema, so a bare SELECT … FROM orders shares one identity with a schema-qualified write of the same table.

Column-level lineage

The sink emits an OpenLineage columnLineage facet on the output dataset in two cases:

  • SQL-derived — an INSERT … SELECT (Execute SQL) maps each target column to the source columns the parser resolves it from.

  • Stream path — a relational write transform whose columns are traced back through the pipeline stream to the read (or lookup) that produced each field, giving target.column ← source.column edges. Covered writers: Table Output, Insert / Update, Update, Dimension Lookup/Update (natural keys and argument-bound attributes), Combination Lookup/Update (keys), and the database bulk loaders. Generated columns — a dimension’s technical key, version and validity dates — have no stream source and carry no column lineage.

Writes that replace their target (a truncate, or the target of an overwrite) additionally carry a lifecycleStateChange facet.

Dataset identity

The (namespace, name) rules the sink uses to identify datasets — and the naming another producer must match to stitch into Hop’s lineage graph — are a stable, versioned contract documented in OpenLineage dataset identity. Read it before integrating a second lineage producer against the same collector.

Verifying

  1. Set HOP_LINEAGE_ENABLED=Y, HOP_LINEAGE_SINK_IDS=openlineage, and HOP_LINEAGE_OPENLINEAGE_URL.

  2. Run a pipeline that reads a file (e.g. a CSV input).

  3. Open the Marquez UI and confirm a job run (START / COMPLETE) and at least one file dataset.

  4. For database lineage, run a pipeline that reads and writes a table (e.g. Table Input → Table Output) and confirm the input/output table datasets — and, for an INSERT … SELECT, the columnLineage facet — in Marquez.

On startup the sink logs OpenLineage lineage sink initialized: url=…, namespace=….