Lineage Sink
Description
A Lineage Sink is a place that pipelines and workflows report their lineage to: an OpenLineage collector such as Marquez, or DataHub.
A pipeline run configuration or a workflow run configuration selects one in its Lineage sink option. The runs it starts report their lineage there, and so do the runs those start: the pipelines a workflow runs, the sub-pipelines of a pipeline. Such a run reports to the sink of its own run configuration as well, if that has one. Lineage then doesn’t need to be switched on with variables: a run that selects a sink turns it on for that run.
A project can point all its runs to one sink through its default run configurations, and send the runs of a development and a production run configuration to different places.
Sinks configured with the HOP_LINEAGE_* variables keep working as they did, next to these: they get the lineage of every run.
Do not configure the same place both ways, or it gets every event twice.
Options
| Option | Description |
|---|---|
Name |
The name of the Lineage Sink. |
Description |
A description of the Lineage Sink. |
Sink type |
The kind of sink: OpenLineage or DataHub. The options below it depend on it, and are grouped in tabs. A sink type comes with a plugin, so the list only shows the types of the plugins that are installed. |
Each option has the meaning and the default of the variable that configures the same sink with variables, given in the last column.
An option left empty keeps that default.
Options can use variables, for example '${MARQUEZ_URL}'; they are resolved with the variables of the run that uses the sink.
OpenLineage
| Option | Description | Variable |
|---|---|---|
Collector URL |
The URL events are posted to, for example |
|
API key |
Sent as a bearer token, when the collector needs one. |
|
Namespace |
The OpenLineage namespace of Hop’s pipelines and workflows. |
|
Producer |
The producer URI the events carry. |
|
A job per transform |
Report each transform as a job of its own; see Job granularity. |
|
Parent run facets |
Name the run that started a run in a parent facet. |
|
Name files by the OpenLineage spec |
Name file datasets by the OpenLineage naming spec. On for a new sink; the variable’s default, the naming of Hop 2.20 and earlier, is deprecated. |
|
Parse SQL |
Parse the SQL of Table input and Execute SQL script to find the tables they read and write. |
|
The Delivery tab holds the spool folder and size, the queue size and what happens when it is full, the retries, the timeouts, the metrics log interval and the trust store: the HOP_LINEAGE_OPENLINEAGE_SPOOL_DIR to HOP_LINEAGE_OPENLINEAGE_KEYSTORE_TYPE settings of the OpenLineage page.
DataHub
| Option | Description | Variable |
|---|---|---|
DataHub data catalog |
The Data Catalog of type DataHub to write to: its URL, access token, environment and trust store are the sink’s. Hop then reads from and writes to the same DataHub, and finds the datasets it wrote. The token needs the right to write metadata as well as to read it. |
|
Platform instance |
Keeps the pipelines and workflows of several Hop installations apart in one DataHub. |
|
Write dataset schemas |
When to write the schema of a dataset; see Dataset schemas. |
|
Deprecate deleted datasets |
|
|
Actor |
The DataHub user named as the actor of the runs and deprecations Hop writes. |
|
Parse SQL |
Parse the SQL of Table input and Execute SQL script. |
|
The Delivery tab has the same options as the OpenLineage one, under HOP_LINEAGE_DATAHUB_.
A trust store set there is used instead of the data catalog’s.
Sink health
Tools > Lineage sinks in Hop GUI shows the lineage sinks of the Hop GUI process, and the status page of a Hop Server shows those of the server: the sinks configured with variables, and the sinks runs selected in their run configurations. Per sink:
| Column | Description |
|---|---|
Configured with |
Variables, or a run configuration. |
State |
Running; not started, because no run sent it an event yet; or failed to start, with the reason in Last error. |
Sent |
Events the backend accepted. |
Failed attempts |
Deliveries that failed and are, or were, tried again: the backend could not be reached, or refused the token. |
Rejected |
Events the backend refused. They are not sent again. |
Dropped |
Events dropped because the sink’s queue was full. |
Waiting |
Events handed to the sink and not delivered yet. |
On disk |
Events kept in the spool folder until they are delivered. Empty without a spool. |
Last sent, Last error |
When the backend last accepted an event, and the last error with its time: the HTTP status and the backend’s answer, or why there was none. |
Above the table: the events waiting in the lineage hub for the sinks, and the events the hub dropped because its queue was full. The dialog refreshes itself every two seconds while it is open in Hop GUI.
A wrong token or URL shows up as a growing Failed attempts and Waiting count with the error, while the events wait to be sent once it is fixed.
Notes
-
A run works out its sinks when it starts. A sink edited while a run is going applies to the runs that start after it; the old version is shut down once the runs that used it ended.
-
Runs that resolve a sink to the same settings share it. Runs of another project, or runs whose variables give the sink other settings, get a sink of their own.
-
Only the runs that report to a sink do the work to observe lineage. The other runs of the same Hop process don’t, unless lineage is enabled with variables.
-
A run started on another Hop Server reports to the sinks of the run configuration it runs with there.
-
Give each sink its own spool folder.
-
A sink that doesn’t exist or whose data catalog is missing is logged on the run, and the run goes on without it.
-
A sink that can’t start, for example because its trust store is missing, is logged and tried again a minute later. Until then the runs that select it log that their lineage doesn’t reach it.
-
A run configuration selects one sink. To send the same runs to two places, select one in the run configuration and configure the other with variables.