OpenLineage

OpenLineage is an open standard for lineage metadata collection. The OpenLineage technology plugin is a Hop lineage sink: it subscribes to the in-engine lineage observation hub and maps Hop’s neutral lineage events to spec-correct OpenLineage RunEvent JSON, which it posts to any OpenLineage-compatible collector. The primary target is Marquez.

Events are generated by the spec’s official openlineage-java client and delivered asynchronously off the pipeline thread, with bounded retries on transient HTTP failures. Delivery is bounded in memory and, with a spool configured, survives a collector outage or a restart — see Durability.

Enabling the sink

The simplest way is a Lineage Sink of type OpenLineage, selected in a pipeline or workflow run configuration: the runs of that configuration report their lineage to it, with the options of the sink, and no variables are needed.

The sink can also be configured with variables, for every run of a Hop process. The lineage hub is disabled by default. Enable it and select the OpenLineage sink with these engine-scoped variables (see the lineage hub for the hub-wide variables):

Variable Default Description

HOP_LINEAGE_ENABLED

N

Set to Y to enable the lineage hub.

HOP_LINEAGE_SINK_IDS

(empty)

Set to openlineage to load this sink.

The sink itself reads the following variables when it initializes:

Variable Default Description

HOP_LINEAGE_OPENLINEAGE_URL

(required)

Collector URL, e.g. http://marquez-api:5000/api/v1/lineage. Must be an absolute http(s) URL.

HOP_LINEAGE_OPENLINEAGE_NAMESPACE

hop

OpenLineage job namespace: the orchestrator instance that produces the events, not the environment. See Job namespace and environments.

HOP_LINEAGE_OPENLINEAGE_PRODUCER

(plugin URL)

OpenLineage producer field.

HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING

N

Y names file datasets by the OpenLineage naming spec (scheme → namespace, path → name, e.g. s3://bucket + key) so a file reconciles across engines/tools; N keeps the legacy job-namespace + filename identity. Switching re-keys existing file datasets. The legacy naming is deprecated; see File datasets.

HOP_LINEAGE_OPENLINEAGE_SQL_PARSE

Y

Parse SELECT/INSERT statements (Table Input, Execute SQL) to recover their source/target tables and column lineage. Set to N to disable SQL parsing.

HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS

N

Y gives each transform its own job, holding the datasets it read plus everything its upstream transforms produced (forward propagation), so a multi-input pipeline reports its real dataset pairs instead of the input × output product. N files every dataset under the pipeline’s job. See Job granularity.

HOP_LINEAGE_OPENLINEAGE_PARENT_RUNS

Y

Emit a parent facet naming the run that started this one (transform → pipeline, action → workflow, pipeline → action or parent pipeline). N leaves it out for a collector that builds the hierarchy itself; see Job hierarchy.

HOP_LINEAGE_OPENLINEAGE_API_KEY

(empty)

Optional bearer token sent as Authorization: Bearer ….

HOP_LINEAGE_OPENLINEAGE_CONNECT_TIMEOUT_MS

5000

HTTP connect timeout.

HOP_LINEAGE_OPENLINEAGE_READ_TIMEOUT_MS

30000

HTTP read/response timeout.

HOP_LINEAGE_OPENLINEAGE_BUFFER_SIZE

10000

Async outbound queue capacity.

HOP_LINEAGE_OPENLINEAGE_MAX_RETRIES

3

Retries per event on transient HTTP failure (exponential backoff).

HOP_LINEAGE_OPENLINEAGE_RETRY_BACKOFF_MS

500

Initial retry backoff; doubles each attempt.

HOP_LINEAGE_OPENLINEAGE_OVERFLOW_POLICY

DROP

What to do when the outbound buffer is full. DROP = drop the event and count it, so lineage never slows the pipeline down. BLOCK = wait for capacity, bounded by HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS, for deployments that would rather lose throughput than lose lineage.

HOP_LINEAGE_OPENLINEAGE_ENQUEUE_TIMEOUT_MS

5000

Under BLOCK, how long (ms) to wait for buffer capacity before dropping the event.

HOP_LINEAGE_OPENLINEAGE_SHUTDOWN_DRAIN_MS

30000

Maximum time to drain the queue on shutdown.

HOP_LINEAGE_OPENLINEAGE_METRICS_INTERVAL_MS

0

If > 0, periodically log sent/failed/rejected/dropped/queued/pending/spooled counts; 0 = off.

HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PATH

(empty)

Trust/key store for HTTPS collectors with a custom CA/cert.

HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PASSWORD

(empty)

Store password (used with the trust store path).

HOP_LINEAGE_OPENLINEAGE_KEYSTORE_TYPE

PKCS12

Trust store type (e.g. PKCS12, JKS).

HOP_LINEAGE_OPENLINEAGE_SPOOL_DIR

(empty)

Folder that holds undelivered events. Empty = no spool (see Durability).

HOP_LINEAGE_OPENLINEAGE_SPOOL_MAX_BYTES

67108864

Ceiling on the spool’s size on disk (64 MiB), per process.

There is no proxy setting — route via the JVM -Dhttps.proxyHost properties if needed.

Durability

Events are sent in the order they were produced. When a delivery fails, the sink stops at that event, pauses (one second, doubling up to a minute) and tries again from there, so a collector that is down costs one attempt per pause rather than one per event, and a run’s COMPLETE never overtakes its START. Meanwhile new events wait: in memory up to HOP_LINEAGE_OPENLINEAGE_BUFFER_SIZE, after which the overflow policy applies.

By default that is all: an event still waiting when the process ends is lost. Set HOP_LINEAGE_OPENLINEAGE_SPOOL_DIR to a writable folder to change that into at least once:

  • every event is written to its own file in the spool before it is posted. A thread of the sink’s own writes it, so the pipeline thread never waits on the disk, and the spool keeps filling while the sink waits on a collector that does not answer;

  • the file is deleted when the collector answers 2xx, or refuses the event (a 4xx other than 401, 403, 404 and 407, which point at configuration and are kept); a refused event is logged and not sent again;

  • when the collector is down at shutdown, the sink does not wait for it: the events stay on disk and the next process that starts sends them, oldest first, before its own.

A collector outage, a GMS restart, or a process that dies between "pipeline finished" and "HTTP 200" therefore delays lineage instead of losing it. A process that dies after a successful POST but before the delete sends one event again; OpenLineage events set aspects on entities rather than append records, so the replay writes the same state again.

The spool is bounded by HOP_LINEAGE_OPENLINEAGE_SPOOL_MAX_BYTES. At the cap, new events are still sent but no longer spooled (counted, and reported once as an error), and the oldest files are kept: those are the ones no later run will re-emit. dropped then counts only events that could not be written to disk and did not fit in memory either.

Several processes can share a spool folder, for example a Hop Server and the hop-run commands on the same machine. Each process writes to a folder of its own inside it and renews a heartbeat file there every 30 seconds. A process only sends the events of a folder whose owner is gone: one that shut down (it removes its heartbeat), or one whose heartbeat is three minutes old (it was killed). It moves those files into its own folder first, so no event is sent by two processes.

Event mapping

Hop lineage event OpenLineage output

RUN_LIFECYCLE

RunEvent — START / COMPLETE / FAIL. A COMPLETE or FAIL also carries the datasets observed for that run since its START.

PIPELINE_GRAPH

The run’s hop graph. With HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS=Y it is remembered for forward propagation (see Job granularity); otherwise it is skipped and the datasets stay on the pipeline job. No event of its own.

WORKFLOW_GRAPH

The workflow’s hop graph. On their COMPLETE or FAIL event, an action’s job names the actions that lead to it (upstreamActions in the hop job facet), and the workflow’s job its first actions (rootActions) and the actions on its enabled hops (actions). No event of its own.

FILE_IO

Physical file datasets (dataSource facet, plus a schema facet when Hop provides the file’s column schema), correlated to the parent pipeline run.

RELATIONAL_IO

Relational table datasets — see Database lineage. A written table carries a schema facet, a columnLineage facet when column mappings are derivable, and a lifecycleStateChange facet for truncate/overwrite writes. A table that is read declares only the columns the statement reads from it: the rows a query returns (aliases, expressions) are not the table’s schema.

TRANSFORM_SCHEMA

Skipped (internal row shapes between transforms, not physical datasets).

HTTP_IO

Skipped.

OpenLineage has no event of its own for a dataset appearing on its own — a collector attaches a dataset to the run that reports it — so FILE_IO and RELATIONAL_IO observations are held against the run they were made in and travel on that run’s COMPLETE or FAIL event. A pipeline therefore reports the files and tables it read and wrote on its own completion, and the collection of a run never has to guess which event a dataset belongs to. An observation whose run has already reported its end, or one still pending when the sink shuts down, is sent as a standalone dataset event instead of being dropped.

Every run event carries a processing_engine run facet (name hop and the Hop version) and says what its job is: a jobType facet (PIPELINE, WORKFLOW, TRANSFORM or ACTION) and a hop job facet with the subject type and the pipeline, workflow, transform or action name and file name. A job name such as nightly/Load orders cannot be split back reliably, so a collector that models pipelines and transforms separately reads those names from the facet.

A transform that ran exactly one SQL statement (Table Input, Execute SQL) has it in the standard sql job facet. A Formula or Calculator lists the expressions it calculates its fields with in a hopTransformLogic job facet: fields, each with its field, its language (formula or calculator) and its expression, such as [price] * [qty] or MULTIPLY(total, rate).

Row counts

A transform’s COMPLETE or FAIL event carries the row counters of that transform copy in a hopRowCounts run facet: linesRead and linesWritten (rows from and to other transforms), linesInput and linesOutput (rows read from and written to files and tables), linesUpdated and linesRejected. These are the transform’s own counters, the numbers it logs as I=, O=, R=, W= and U=; Text File Output counts its header and footer lines as output too, so a file with a header shows one line more than it has rows. A pipeline’s event carries the sum over all its transforms and copies of the counters that add up: linesInput, linesOutput, linesUpdated and linesRejected. That sum is not the pipeline result Hop logs, which takes the highest value of each counter, and a Metadata Injection transform counts the rows of the pipeline it runs too. Only the local pipeline engine sends counters; Beam, Spark and remote runs don’t have final ones when their run ends.

A dataset gets the standard inputStatistics or outputStatistics facet with its rowCount when the count is known to be its own: the transforms that read (or wrote) it read (or wrote) nothing else. The rows written are the rows output plus the rows updated. A transform that writes several files or tables, such as Text File Output with a file per copy or Table Output with the table in a field, gives them no row count rather than a wrong one. On a pipeline’s event a dataset’s count is the sum over its transforms and their copies; with transform jobs on, each copy’s event counts the rows of that copy. Rows a transform deleted from a table (Delete, an Execute SQL with a single DELETE statement) are not rows written; a table emptied and reloaded in one run counts the rows loaded. An Execute SQL script of several statements gives no count, as its counters add all of them up, and neither does a transform that counts 0 while it does handle rows (not every transform counts what it reads or writes).

A pipeline’s COMPLETE or FAIL event also lists every dataset the run wrote, once, in a hopWrites run facet: its namespace and name, its operationType (INSERT, UPDATE, DELETE, CREATE, DROP or ALTER) and, when known, its rows. The DataHub sink turns these into operations. A pipeline that was stopped (Stop, Abort, or a failing parent pipeline stopping it) says so with stopped in its hopRowCounts facet and lists no writes.

Apache Beam and Spark run the transforms of a pipeline in small local pipelines on their workers. With lineage on in the JVM of those workers, as with the Beam DirectRunner or Spark in local mode, each of those reports a run of its own, with the counts and writes of its bundle or partition.

The run id is the executing subject’s log-channel id. By default file I/O is correlated to the parent pipeline run, so inputs and outputs appear on the pipeline job in Marquez. Every run also names the run that started it: a transform’s ParentRunFacet points at its pipeline, an action’s at its workflow, and a pipeline’s at the action that ran it (or at the parent pipeline, for a sub-pipeline). Actions are jobs of their own, named <workflow>/<action>, so a workflow with several actions no longer files them all under the workflow’s job.

Job hierarchy

Jobs are named <owner>/<job>: a transform is <pipeline>/<transform>, an action is <workflow>/<action>. With the parent facets on, a collector that understands parent runs draws the orchestrated run as a tree:

workflow ──▶ action ──▶ pipeline ──▶ transform

A pipeline or workflow that nothing started (a hop-run invocation, the GUI) has no parent.

Set HOP_LINEAGE_OPENLINEAGE_PARENT_RUNS=N to leave the facets out when the collector builds the hierarchy from something else.

Job granularity

OpenLineage derives dataset-to-dataset lineage through jobs: a job’s inputs are paired all-to-all with its outputs. Filing every dataset of a pipeline under the pipeline’s job is what makes Marquez show them on the job a user opens, but it also means an n-input/m-output pipeline produces the full n×m product: reading a and b and writing x and y yields a→x, a→y, b→x and b→y.

Set HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS=Y to make the job mean the transformation. Each transform’s job then carries:

  • the datasets it read itself, plus

  • everything its upstream transforms produced — what they wrote if they wrote anything, otherwise what they read; a transform that does no I/O of its own (a filter, a select) passes what it received through to its downstream transforms.

That gives a writer’s job both ends of a real pair (a→x and b→y above), and lineage still crosses transforms that do not touch a file or table. The pipeline job remains the run-level entry point, without datasets. The trade-off is the one the default was chosen for: with the flag on, the datasets sit on the transform jobs in Marquez, so a user who opens the pipeline has to click through to a transform to see them.

Forward propagation needs the pipeline’s hop graph. The engine emits it as a PIPELINE_GRAPH event next to the run’s START, and this sink uses it only when the flag is on. A run without a graph — an engine that predates the event, or an event stream that lost it — keeps the pipeline-granular mapping, so the flag can stay on across an upgrade.

What a transform consumed is only known once its pipeline is over: an upstream writer reports its table when it finishes, which can be after a downstream transform finished. With the flag on, the transforms' COMPLETE and FAIL events are therefore sent when the pipeline ends, just ahead of the pipeline’s own.

Database lineage

RELATIONAL_IO events record the relational tables a transform reads from or writes to. The physical dataset is identified by a database namespace (e.g. postgres://host:5432) and a fully-qualified database.schema.table name (see Dataset identity), so a Hop write and, for example, a dbt read of the same table land on one node in the lineage graph.

The following transforms emit relational lineage:

  • Reads — Table Input (its SELECT is parsed to recover the source tables and column lineage).

  • Writes — Table Output, Insert / Update, Update, Dimension Lookup/Update, Combination Lookup/Update, and the database bulk loaders (CrateDB, MonetDB, MySQL, Oracle, PostgreSQL, Vertica).

  • Deletes — Delete reports the affected table only. A delete produces no columns, so the event carries neither a column schema nor column lineage.

  • Reads and writes — Execute SQL (its statements are parsed for both input and output tables).

SQL parsing

For transforms that only provide a SQL statement (Table Input, Execute SQL), the sink recovers the tables — and, for an INSERT … SELECT, the column-level lineage — by parsing the statement with the OpenLineage SQL parser. Parsing is on by default and controlled by HOP_LINEAGE_OPENLINEAGE_SQL_PARSE. Tables named without a schema or catalog are qualified using the connection’s default database and preferred schema, so a bare SELECT … FROM orders shares one identity with a schema-qualified write of the same table.

An INSERT INTO summary (order_id, customer) SELECT id, cust FROM … names the output columns after the target list, not after the SELECT expressions: the sink reads the two lists from the statement and renames each parsed edge positionally. The rename is only applied when both lists can be read unambiguously (no *, a plain VALUES insert, or two statements), because attaching column lineage to the wrong column is worse than leaving the parser’s name in place.

Column-level lineage

The sink emits an OpenLineage columnLineage facet on the output dataset in two cases:

  • SQL-derived — an INSERT … SELECT (Execute SQL) maps each target column to the source columns the parser resolves it from.

  • Stream path — a relational write transform whose columns are traced back through the pipeline stream to the read (or lookup) that produced each field, giving target.column ← source.column edges. Covered writers: Table Output, Insert / Update, Update, Dimension Lookup/Update (natural keys and argument-bound attributes), Combination Lookup/Update (keys), and the database bulk loaders. Generated columns — a dimension’s technical key, version and validity dates — have no stream source and carry no column lineage.

Writes that replace their target (a truncate, or the target of an overwrite) additionally carry a lifecycleStateChange facet.

Column types in the schema facet use the lowercase OpenLineage names — string, int, double, decimal, date, timestamp, boolean, binary, and struct for JSON — rather than Hop’s own type descriptions. A type with no honest OpenLineage equivalent (Serializable, or a type a plugin invented) is left out instead of guessed.

Job namespace and environments

The job namespace identifies the orchestrator instance that produced the events, not the environment: hop on a single deployment, hop-prod and hop-dev when one collector holds several. It is part of a job’s identity, so two environments that share it merge into one job in the collector’s UI even though they wrote different datasets. A collector may name the orchestrator from the processing_engine facet instead (DataHub’s OpenLineage endpoint does), in which case the namespace does not appear in its job identity at all.

The environment itself is a property of the collector’s deployment: the same collector marks every dataset it receives with one environment, whichever engine sent it, which is what lets a Hop file dataset and a Spark file dataset join up. Two environments that must not merge therefore need two collector deployments, not two namespaces.

Unit test results

A pipeline unit test that runs with lineage reports each golden data set as an OTHER event with a hopUnitTest run facet: the test, the data set, whether it passed, error when the rows could not be compared at all, what the comparison found, and the datasets the transform declares it writes. Those datasets are not the event’s outputs: the test run didn’t write them. The event belongs to the tested transform’s run with HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS=Y, to the pipeline’s run otherwise.

A unit test runs a changed copy of the pipeline. A lineage sink selected in a run configuration only gets the test’s results, not the lineage of that copy.

Dataset identity

The (namespace, name) rules the sink uses to identify datasets — and the naming another producer must match to stitch into Hop’s lineage graph — are a stable, versioned contract documented in OpenLineage dataset identity. Read it before integrating a second lineage producer against the same collector.

Verifying

  1. Set HOP_LINEAGE_ENABLED=Y, HOP_LINEAGE_SINK_IDS=openlineage, and HOP_LINEAGE_OPENLINEAGE_URL.

  2. Run a pipeline that reads a file (e.g. a CSV input).

  3. Open the Marquez UI and confirm a job run (START / COMPLETE) and at least one file dataset.

  4. For database lineage, run a pipeline that reads and writes a table (e.g. Table Input → Table Output) and confirm the input/output table datasets — and, for an INSERT … SELECT, the columnLineage facet — in Marquez.

On startup the sink logs OpenLineage lineage sink initialized: url=…, namespace=…. Tools > Lineage sinks in Hop GUI, and the status page of a Hop Server, show what the sink sent, what waits and the last error; see Sink health.