OpenLineage

OpenLineage is an open standard for lineage metadata collection. The OpenLineage technology plugin is a Hop lineage sink: it subscribes to the in-engine lineage observation hub and maps Hop’s neutral lineage events to spec-correct OpenLineage RunEvent JSON, which it posts to any OpenLineage-compatible collector. The primary target is Marquez.

Events are generated by the spec’s official openlineage-java client, delivered asynchronously off the pipeline thread, retried on transient HTTP failures, and (by default) buffered losslessly under back-pressure.

Enabling the sink

The lineage hub is disabled by default. Enable it and select the OpenLineage sink with these engine-scoped variables (see the lineage hub for the hub-wide variables):

Variable Default Description

Variable	Default	Description
`HOP_LINEAGE_ENABLED`	`N`	Set to `Y` to enable the lineage hub.
`HOP_LINEAGE_SINK_IDS`	(empty)	Set to `openlineage` to load this sink.

HOP_LINEAGE_ENABLED

N

Set to Y to enable the lineage hub.

HOP_LINEAGE_SINK_IDS

(empty)

Set to openlineage to load this sink.

The sink itself reads the following variables when it initializes:

Variable Default Description

Variable	Default	Description
`HOP_LINEAGE_OPENLINEAGE_URL`	(required)	Collector URL, e.g. `http://marquez-api:5000/api/v1/lineage`. Must be an absolute `http(s)` URL.
`HOP_LINEAGE_OPENLINEAGE_NAMESPACE`	`hop`	OpenLineage job namespace (the producing Hop environment).
`HOP_LINEAGE_OPENLINEAGE_PRODUCER`	(plugin URL)	OpenLineage `producer` field.
`HOP_LINEAGE_OPENLINEAGE_SQL_PARSE`	`Y`	Parse `SELECT`/`INSERT` statements (Table Input, Execute SQL) to recover their source/target tables and column lineage. Set to `N` to disable SQL parsing.
`HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING`	`N`	`Y` names file datasets by the OpenLineage naming spec (scheme → namespace, path → name, e.g. `s3://bucket` + `key`) so a file reconciles across engines/tools; `N` keeps the legacy job-namespace + filename identity. Switching re-keys existing file datasets.
`HOP_LINEAGE_OPENLINEAGE_API_KEY`	(empty)	Optional bearer token sent as `Authorization: Bearer …`.
`HOP_LINEAGE_OPENLINEAGE_CONNECT_TIMEOUT_MS`	`5000`	HTTP connect timeout.
`HOP_LINEAGE_OPENLINEAGE_READ_TIMEOUT_MS`	`30000`	HTTP read/response timeout.
`HOP_LINEAGE_OPENLINEAGE_BUFFER_SIZE`	`10000`	Async outbound queue capacity.
`HOP_LINEAGE_OPENLINEAGE_MAX_RETRIES`	`3`	Retries per event on transient HTTP failure (exponential backoff).
`HOP_LINEAGE_OPENLINEAGE_RETRY_BACKOFF_MS`	`500`	Initial retry backoff; doubles each attempt.
`HOP_LINEAGE_OPENLINEAGE_OVERFLOW_POLICY`	`DROP`	What to do when the outbound buffer is full. `DROP` = drop the event and count it, so lineage never slows the pipeline down. `BLOCK` = wait for capacity, bounded by `HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS`, for deployments that would rather lose throughput than lose lineage.
`HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS`	`5000`	Under `BLOCK`, how long (ms) to wait for buffer capacity before dropping the event.
`HOP_LINEAGE_OPENLINEAGE_SHUTDOWN_DRAIN_MS`	`30000`	Maximum time to drain the queue on shutdown.
`HOP_LINEAGE_OPENLINEAGE_METRICS_INTERVAL_MS`	`0`	If > 0, periodically log `sent/failed/dropped/queued` counts (`0` = off).
`HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PATH`	(empty)	Trust/key store for HTTPS collectors with a custom CA/cert.
`HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PASSWORD`	(empty)	Store password (used with the trust store path).
`HOP_LINEAGE_OPENLINEAGE_KEYSTORE_TYPE`	`PKCS12`	Trust store type (e.g. `PKCS12`, `JKS`).

HOP_LINEAGE_OPENLINEAGE_URL

(required)

Collector URL, e.g. http://marquez-api:5000/api/v1/lineage. Must be an absolute http(s) URL.

HOP_LINEAGE_OPENLINEAGE_NAMESPACE

hop

OpenLineage job namespace (the producing Hop environment).

HOP_LINEAGE_OPENLINEAGE_PRODUCER

(plugin URL)

OpenLineage producer field.

HOP_LINEAGE_OPENLINEAGE_SQL_PARSE

Y

Parse SELECT/INSERT statements (Table Input, Execute SQL) to recover their source/target tables and column lineage. Set to N to disable SQL parsing.

HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING

N

Y names file datasets by the OpenLineage naming spec (scheme → namespace, path → name, e.g. s3://bucket + key) so a file reconciles across engines/tools; N keeps the legacy job-namespace + filename identity. Switching re-keys existing file datasets.

HOP_LINEAGE_OPENLINEAGE_API_KEY

(empty)

Optional bearer token sent as Authorization: Bearer ….

HOP_LINEAGE_OPENLINEAGE_CONNECT_TIMEOUT_MS

5000

HTTP connect timeout.

HOP_LINEAGE_OPENLINEAGE_READ_TIMEOUT_MS

30000

HTTP read/response timeout.

HOP_LINEAGE_OPENLINEAGE_BUFFER_SIZE

10000

Async outbound queue capacity.

HOP_LINEAGE_OPENLINEAGE_MAX_RETRIES

3

Retries per event on transient HTTP failure (exponential backoff).

HOP_LINEAGE_OPENLINEAGE_RETRY_BACKOFF_MS

500

Initial retry backoff; doubles each attempt.

HOP_LINEAGE_OPENLINEAGE_OVERFLOW_POLICY

DROP

What to do when the outbound buffer is full. DROP = drop the event and count it, so lineage never slows the pipeline down. BLOCK = wait for capacity, bounded by HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS, for deployments that would rather lose throughput than lose lineage.

HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS

5000

Under BLOCK, how long (ms) to wait for buffer capacity before dropping the event.

HOP_LINEAGE_OPENLINEAGE_SHUTDOWN_DRAIN_MS

30000

Maximum time to drain the queue on shutdown.

HOP_LINEAGE_OPENLINEAGE_METRICS_INTERVAL_MS

0

If > 0, periodically log sent/failed/dropped/queued counts (0 = off).

HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PATH

(empty)

Trust/key store for HTTPS collectors with a custom CA/cert.

HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PASSWORD

(empty)

Store password (used with the trust store path).

HOP_LINEAGE_OPENLINEAGE_KEYSTORE_TYPE

PKCS12

Trust store type (e.g. PKCS12, JKS).

Events are buffered in memory and delivered by a single ordered worker thread; on a hard crash undelivered events are lost. There is no proxy setting — route via the JVM -Dhttps.proxyHost properties if needed.

Event mapping

Hop lineage event OpenLineage output

Hop lineage event	OpenLineage output
`RUN_LIFECYCLE`	`RunEvent` — `START` / `COMPLETE` / `FAIL`.
`FILE_IO`	`RunEvent` (`OTHER`) with physical file datasets (`dataSource` facet, plus a `schema` facet when Hop provides the file’s column schema), correlated to the parent pipeline run.
`RELATIONAL_IO`	`RunEvent` (`OTHER`) with relational table datasets — see Database lineage. Carries a `schema` facet, a `columnLineage` facet when column mappings are derivable, and a `lifecycleStateChange` facet for truncate/overwrite writes.
`TRANSFORM_SCHEMA`	Skipped (internal row shapes between transforms, not physical datasets).
`HTTP_IO`	Skipped.

RUN_LIFECYCLE

RunEvent — START / COMPLETE / FAIL.

FILE_IO

RunEvent (OTHER) with physical file datasets (dataSource facet, plus a schema facet when Hop provides the file’s column schema), correlated to the parent pipeline run.

RELATIONAL_IO

RunEvent (OTHER) with relational table datasets — see Database lineage. Carries a schema facet, a columnLineage facet when column mappings are derivable, and a lifecycleStateChange facet for truncate/overwrite writes.

TRANSFORM_SCHEMA

Skipped (internal row shapes between transforms, not physical datasets).

HTTP_IO

Skipped.

The run id is the executing pipeline/workflow log-channel id; file I/O is correlated to the parent pipeline run so inputs/outputs appear on the pipeline job in Marquez. Transform and action run events carry a ParentRunFacet so their jobs attach to the parent pipeline/workflow run in the lineage graph rather than appearing as disconnected jobs.

Database lineage

RELATIONAL_IO events record the relational tables a transform reads from or writes to. The physical dataset is identified by a database namespace (e.g. postgres://host:5432) and a fully-qualified database.schema.table name (see Dataset identity), so a Hop write and, for example, a dbt read of the same table land on one node in the lineage graph.

The following transforms emit relational lineage:

Reads — Table Input (its SELECT is parsed to recover the source tables and column lineage).
Writes — Table Output, Insert / Update, Update, Dimension Lookup/Update, Combination Lookup/Update, and the database bulk loaders (CrateDB, MonetDB, MySQL, Oracle, PostgreSQL, Vertica).
Deletes — Delete reports the affected table only. A delete produces no columns, so the event carries neither a column schema nor column lineage.
Reads and writes — Execute SQL (its statements are parsed for both input and output tables).

SQL parsing

For transforms that only provide a SQL statement (Table Input, Execute SQL), the sink recovers the tables — and, for an INSERT … SELECT, the column-level lineage — by parsing the statement with the OpenLineage SQL parser. Parsing is on by default and controlled by HOP_LINEAGE_OPENLINEAGE_SQL_PARSE. Tables named without a schema or catalog are qualified using the connection’s default database and preferred schema, so a bare SELECT … FROM orders shares one identity with a schema-qualified write of the same table.

Column-level lineage

The sink emits an OpenLineage columnLineage facet on the output dataset in two cases:

SQL-derived — an INSERT … SELECT (Execute SQL) maps each target column to the source columns the parser resolves it from.
Stream path — a relational write transform whose columns are traced back through the pipeline stream to the read (or lookup) that produced each field, giving target.column ← source.column edges. Covered writers: Table Output, Insert / Update, Update, Dimension Lookup/Update (natural keys and argument-bound attributes), Combination Lookup/Update (keys), and the database bulk loaders. Generated columns — a dimension’s technical key, version and validity dates — have no stream source and carry no column lineage.

Writes that replace their target (a truncate, or the target of an overwrite) additionally carry a lifecycleStateChange facet.

Dataset identity

The (namespace, name) rules the sink uses to identify datasets — and the naming another producer must match to stitch into Hop’s lineage graph — are a stable, versioned contract documented in OpenLineage dataset identity. Read it before integrating a second lineage producer against the same collector.

Verifying

Set HOP_LINEAGE_ENABLED=Y, HOP_LINEAGE_SINK_IDS=openlineage, and HOP_LINEAGE_OPENLINEAGE_URL.
Run a pipeline that reads a file (e.g. a CSV input).
Open the Marquez UI and confirm a job run (START / COMPLETE) and at least one file dataset.
For database lineage, run a pipeline that reads and writes a table (e.g. Table Input → Table Output) and confirm the input/output table datasets — and, for an INSERT … SELECT, the columnLineage facet — in Marquez.

On startup the sink logs OpenLineage lineage sink initialized: url=…, namespace=….