Lineage observation hub
Apache Hop provides an in-engine lineage observation hub for recording metadata-level observations (pipeline context, file I/O, HTTP traffic, structural hints) and delivering them asynchronously to pluggable lineage sinks. The design keeps third-party integrations (OpenLineage, Datadog, custom exporters) out of the engine: sinks are separate plugins that map Hop’s neutral events to their own formats.
The public API lives under the Java package org.apache.hop.lineage in the hop-engine module.
Goals
-
Central entry point — Producers call
LineageHub.getInstance().emit(…)instead of talking to each backend directly. -
Safe defaults — The hub is off until
HOP_LINEAGE_ENABLEDis set toY, or a run selects a lineage sink in its run configuration. -
Non-blocking pipelines — Events are queued and dispatched on a dedicated worker thread; sinks run in batch with per-sink error isolation.
-
Extensibility — New backends ship as
ILineageSinkimplementations registered via@LineageSinkPlugin.
Package layout
| Package | Role |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
Extension points that call |
Relational lineage is declared, not emitted
A transform does not emit RELATIONAL_IO events itself. It declares the table access its
metadata describes, and the engine derives the lineage from that declaration when the transform
finishes.
Two annotations carry the declaration:
-
@RelationalLineage(org.apache.hop.lineage.api) on theITransformMetaclass — opts the transform in and states theRelationalIoOperation(WRITE,DELETE, …). Opting in is explicit rather than inferred from the presence of a table property, because plenty of metadata names a table without reading or writing its rows (checking that it exists, comparing two of them). -
HopMetadataPropertyTypevalues on the properties themselves —RDBMS_CONNECTION,RDBMS_SCHEMA,RDBMS_TABLE,RDBMS_TRUNCATE, andRDBMS_COLUMNpaired with the database-neutralSTREAM_FIELDon the same mapping object to express "stream field X is written to column Y". These annotations already existed for search and metadata injection; lineage reuses them.
LineageMetadataWalker reads that declaration, recursing into nested metadata objects and lists.
LineageRelationalIoEmitter.emitDeclaredRelationalIo then combines it with the facts only available
at runtime — resolved variables, the connection the name refers to, the row shape, and the transform
that produced each stream field — and emits the event. LineageHubTransformFinishXp drives this for
every transform.
Consequences worth knowing when adding a database transform:
-
Annotating the metadata is all that is required; there is no emission code to write, and no way to silently forget it for a transform that already annotates its properties.
-
A column declaring
RDBMS_COLUMNwith noSTREAM_FIELDis treated as generated by the transform (a dimension’s technical key, for example) and is left out of the column lineage rather than reported with no origin. -
Facts that only exist per row cannot be declared. A target table read from a row field, or a SQL statement built at runtime, is still reported by the transform itself — Table Output’s dynamic table-name mode and the SQL-driven transforms (Table Input, Execute SQL) work that way.
-
A transform reports its table access when it finishes, which is after the pipeline run’s own completion event. Marquez creates dataset versions only from terminal run events, so such a dataset shows
fields: []and no current version in its API and UI, while itscolumnLineagefacet is stored and the lineage graph is complete. This is Marquez’s ingestion rule, not a gap in what Hop emits — worth knowing before investigating a dataset that looks empty.
Event model
Each observation is a LineageEvent:
-
Event id — Unique id (UUID by default).
-
Timestamp — Milliseconds since epoch.
-
LineageEventKind— One ofRUN_LIFECYCLE,TRANSFORM_SCHEMA,FILE_IO,RELATIONAL_IO,HTTP_IO,PIPELINE_GRAPH,WORKFLOW_GRAPH. New kinds are only ever appended: a sink must skip a kind it does not handle rather than fail the batch. -
LineageContext— Who/what produced the observation (see above). -
LineagePayload— Optional typed payload; may be null if the kind and context are enough.
Payload types are plain Java types (strings, longs, enums) so sinks can serialize them without dragging engine objects across boundaries:
-
FileIoLineagePayload— Operation (READ,WRITE,COPY,MOVE,DELETE), source/target URIs, byte counts, success, message, and optionalFileIoContentSchema: tabular columns actually read/written (including JSONPath / YAML path / XPath locators) plus optionalstructureRoots(FileIoSchemaNodetree) merged from those paths for nested file layouts. Text File Output and JSON Output attach the file column layout (not the full transform output row when a subset is written). -
RelationalIoLineagePayload— Relational/JDBC table access: operation (READ,WRITE,DELETE,EXEC), the resolved dataset namespace, the input/outputRelationalTablelist (catalog/schema/table), the raw SQL when the engine cannot resolve the tables itself, an optional column schema, the connection’s default catalog and schema, optional per-column provenance (RelationalWriteColumn: target column, stream field, origin transform), and an optionalRelationalLifecycle(OVERWRITE,TRUNCATE,DROP,CREATE). -
HttpLineagePayload—CLIENTvsSERVER, method, URL, status, request/response bytes, duration, success, message. -
RunLifecycleLineagePayload—STARTED/FINISHEDplus optional detail text. -
PipelineGraphLineagePayload— The pipeline’s hop graph as a list offrom→totransform-name transitions: enabled hops with both ends named, in pipeline order. Informational hops are included. A sink uses it to attribute datasets to the transform that touched them and to propagate them along the graph, instead of pairing all of a pipeline’s inputs with all of its outputs. A pipeline without hops gets an empty list. The same payload carries a workflow’s hop graph (WORKFLOW_GRAPH), its transitions naming actions: every enabled hop, whatever its condition. -
TransformSchemaLineagePayload—INPUTorOUTPUTplus a list of field names, Hop type names, length, and precision (boundary schema; emitted when a transform finishes). Fields usually come from runtime row metadata on the transform or row sets. If input was never observed (e.g. Abort whengetRow()immediately returns null), input is derived viaPipelineMeta#getPrevTransformFields. If output was never written (e.g. Detect empty stream with a non-empty stream), output is derived viaPipelineMeta#getTransformFields. Graph-derived events set context attributetransformSchemaSourcetoDESIGN_GRAPH.
Convenience factory: LineageEvent.of(kind, context, payload) assigns id and current time.
Hub behavior
LineageHub is a singleton (getInstance()). When lineage is disabled, emit returns immediately.
When enabled:
-
The first
emitstarts a daemon worker thread (Hop-LineageHub-Dispatcher). -
Events are placed on a bounded queue. If the queue is full, the event is dropped and an error is logged (drops are counted).
-
The worker builds batches up to
HOP_LINEAGE_BATCH_MAX, waiting up toHOP_LINEAGE_BATCH_LINGER_MSfor additional events before sending a partial batch. -
Each batch is passed to every loaded sink via
ILineageSink.accept(List<LineageEvent>). If one sink throws, others still receive the same batch; failures are logged.
Sinks selected in run configurations
A lineage-sink metadata object (org.apache.hop.lineage.metadata.LineageSink) holds the configuration of one sink: an ILineageSinkConfiguration the sink plugin hands out with ILineageSink.newConfiguration(). A pipeline or workflow run configuration selects one in lineageSinkName.
When a run’s first lineage context is built, LineageSinkRoutes works out the run’s sinks: the one of its run configuration, plus the ones of the run that started it. It resolves each with the run’s variables (ILineageSinkConfiguration.toSettings()) into a LineageSinkTarget, registers it with LineageHub.useSink(), and keeps the keys with the run. Every LineageContext of the run carries them in sinks.
The hub delivers an event to the sinks loaded from variables, as above, and to the sinks of the keys it carries. Such a sink is created and initialized, with the target’s settings as its only variables, on the first event for it. A sink that fails to start is logged and tried again a minute later, and the other sinks carry on; LineageHub.failure(key) lets the next runs log it too.
Code that does extra work only to build an event asks LineageSinkRoutes.isEnabled(run or transform): lineage enabled with variables, or this run has sinks of its own. LineageHub.isEnabled() only answers the first half.
A run holds its sinks in the hub (useSink and hold) and lets go of them when it ends (LineageSinkRoutes.release, which queues the release behind the run’s last event). A sink no run holds whose name a newer target took, after its settings were edited, is shut down.
flush() blocks until queued events (and coalesced flush markers) have been processed. flushQuietly() catches and logs errors. flushIfBacklogged() flushes only once the queue is at least half full: a nested run calls it, so that a run per row does not drain the hub after every row, but still waits for the sinks before the queue overflows and events are dropped.
shutdown() stops the worker and calls shutdown() on sinks. It is invoked from HopEnvironment.shutdown() and HopEnvironment.reset(), and from a JVM exit hook that the hub registers when it starts its worker. That hook waits for a hop-run that is still stopping on a termination signal (holdShutdownOnExit), so the end of the run is delivered too.
After PluginRegistry.init(), HopEnvironment calls LineageHub.getInstance().environmentReady() so sink lists can be rebuilt when the environment is re-initialized.
The queue is allocated lazily on the first emit from the resolved configuration, so HOP_LINEAGE_QUEUE_CAPACITY applies to the singleton as well as to isolated test hubs. The capacity is fixed for the lifetime of the queue; to change it, restart the JVM (or call shutdown() and emit again in tests).
Run lifecycle (extension points)
The org.apache.hop.lineage.xp package registers extension points that emit RUN_LIFECYCLE events (via LineageRunLifecycleEmitter) and flush the hub where needed:
-
PipelineStart—LineageHubPipelineStartXp— phaseSTARTED, followed by onePIPELINE_GRAPHevent with the same context, with no transitions when the pipeline has no enabled hops.PipelineStartfires after the transform threads have started, so the run’s transformSTARTEDevents, and early I/O such as a Table Input read, can arrive before the graph, possibly in an earlier batch. A sink that uses the graph collects a run’s events by the pipeline’s log channel id (transform events carry it in thepipelineLogChannelIdattribute) and resolves them atFINISHEDorFAILED. -
PipelineFinish—LineageHubPipelineFinishXp— phaseFINISHING(transforms stopped, cleanup) -
PipelineCompleted—LineageHubPipelineCompletedXp— phaseFINISHEDorFAILED(from pipeline error count), thenflushQuietly()for a top-level run andflushIfBacklogged()for a nested one -
WorkflowStart—LineageHubWorkflowStartXp— phaseSTARTED, followed by oneWORKFLOW_GRAPHevent with the same context (empty when the workflow has no enabled hops) -
WorkflowFinish—LineageHubWorkflowFinishXp— first actionFAILEDfor every action run that started but never ended (itsexecute()threw), then the workflow’s phaseFINISHEDorFAILED(from workflow/result errors), thenflushQuietly()for a top-level run andflushIfBacklogged()for a nested one -
HopServerShutdown—LineageHubServerShutdownFlushXp— flush only -
TransformBeforeStart—LineageHubTransformStartXp— transform phaseSTARTED(local pipeline engine; transform log channel + pipeline correlation attributes) -
TransformFinished—LineageHubTransformFinishXp— transformFINISHEDorFAILEDfrom transform error count, thenTRANSFORM_SCHEMAevents for observed input/output row shapes when non-empty -
Action start —
LineageActionStartListener, anIActionListenerthatLineageHubWorkflowStartXpadds to each workflow — actionSTARTEDfrombeforeExecution, once the workflow has cloned the action, so the event carries the executing action’s log channel id.WorkflowBeforeActionExecutionfires before that clone exists. An error while emitting is logged and does not fail the action. -
WorkflowAfterActionExecution—LineageHubWorkflowActionAfterXp— actionFINISHEDorFAILEDfrom the actionResult, under the executed action’s log channel id (WorkflowExecutionExtension.actionLogChannelId). An action the workflow skipped has no run of its own and emits nothing. The workflow does not reach this extension point when the action’sexecute()throws;WorkflowFinishends that run instead.
Remote (RemotePipelineEngine) and Beam (BeamPipelineEngine) engines also call PipelineStart and PipelineFinish so they align with the local pipeline engine.
Beam and other engines that do not use the classic Hop thread-per-transform model typically do not fire TransformBeforeStart / TransformFinished the same way as the local Pipeline engine.
Configuration
Variables are defined on org.apache.hop.lineage.LineageVariables and appear in the variable registry (ENGINE scope) like other Hop settings.
| Variable | Default | Meaning |
|---|---|---|
|
|
Set to |
|
|
Bounded queue capacity. Read on first |
|
|
Maximum events per batch per sink. |
|
|
How long the worker waits for more events before sending a smaller batch. |
|
(empty) |
Comma-separated sink plugin ids (case-insensitive). Empty means all discovered sinks are loaded. |
LineageConfiguration.resolve() reads the active IVariables (typically after Variables.initializeFrom so system properties and Hop config apply).
LineageSink plugins
Sinks implement org.apache.hop.lineage.spi.ILineageSink:
-
init(IVariables variables, ILogChannel log)— Optional; called once before the first batch. -
accept(List<LineageEvent> events)— Receives a non-empty batch; should not hold references to the list after returning. -
shutdown()— Optional cleanup. -
health()— Optional; aLineageSinkHealthwith the counts (sent, failed attempts, rejected, dropped, waiting, on disk) and the last error, for Tools > Lineage sinks in Hop GUI and the Hop Server status page (LineageHub.sinkStatus()). Called from any thread: keep it cheap. -
newConfiguration()— Optional; a newILineageSinkConfiguration, an object with@HopMetadataPropertyand@GuiWidgetElementfields (parentLineageSink.GUI_PLUGIN_ELEMENT_PARENT_ID), so the sink can be alineage-sinkmetadata object. ItstoSettings()gives the variablesinitthen reads. Null, the default, means variables only.
The implementation class is annotated with @LineageSinkPlugin (id, optional name, description), mirroring other Hop plugin types (see Plugin Development).
The plugin type LineageSinkPluginType is registered in HopEnvironment.getStandardPluginTypes() alongside transforms, actions, execution info locations, etc. Plugins are discovered from the classpath like other native/engine plugins.
No sink implementations ship with the engine. Integrators implement ILineageSink, annotate the class with @LineageSinkPlugin(id = "…"), and package it as a Hop plugin. With no sinks loaded, the hub remains a no-op even when HOP_LINEAGE_ENABLED=Y.
Declaring files and SQL
A transform whose metadata tells which files it reads or writes, or which SQL it runs, implements org.apache.hop.lineage.api.IDeclaresLineage: declaredLineage(variables, inputRowMeta) returns DeclaredFile (operation, file name, layout) and DeclaredSql (operation, connection, SQL) entries, with variables resolved. Return what a run reports: the same file name and the same layout, so one dataset describes the file whether a run or a publication described it. Declare nothing a run doesn’t write under that name, such as a file name with the date of the run. inputRowMeta holds the fields that reach the transform only when they can be worked out without a database, that is when no transform upstream has a relational connection; it is empty otherwise, and for unit test results and the AI Assistant, which only need the names. Text file input, CSV input, Text file output and Table input implement it; tables are declared with @RelationalLineage.
LineageDefinitions.ofPipeline() and ofWorkflow() turn a pipeline or workflow into the events a run would emit, from those declarations, with the lineageDefinition attribute set: their run is made up, and a sink that records runs must not record it. Pass the variables a run of the file would have: its internal variables (setInternalHopVariables) and its parameters' defaults, as DataHubPublisher.variablesOf() sets them. The DataHub plugin’s DataHubPublisher sends them through a DataHub sink that writes definitions only.
Unit test results
When a pipeline unit test ends, the testing plugin calls LineageUnitTestEmitter.emit() for each golden data set: a UNIT_TEST event on the tested transform’s run with an UnitTestLineagePayload (test, data set, result, comment, and the IO the transform declares, read from the pipeline as it was before the test changed it). The result is PASSED, FAILED, or ERROR when the rows were not compared, for example because the golden data set could not be read. The OpenLineage mapper turns it into the hopUnitTest run facet; the DataHub sink into an assertion and an assertion run per declared dataset.
A unit test runs a changed copy of the pipeline, without a file name. Its other events don’t reach the sinks of run configurations: LineageSinkRoutes.of() is empty for a run with UnitTest_Run=Y, and LineageSinkRoutes.ofUnitTest() gives the sinks for the results. The sinks configured with variables get the run’s events as before.
Documentation
When DataHubPublisher publishes a project with documentation, it sends a DOCUMENTATION event for each pipeline and workflow once every definition is delivered, with a DocumentationLineagePayload: a title, the Markdown DocMarkdown (documentation plugin) builds the way hop doc does, the project name, and the names of the transforms or actions. Runs never emit it. The OpenLineage mapper turns it into the hopDocumentation run facet of an OTHER event; the DataHub sink into a document entity related to the flow and its tasks, under a document of the project.
Emitting events from code
Producers (pipeline/workflow hooks, VFS usage, HTTP client/server filters, transforms) should build a LineageContext (often via LineageContext.builder()), optionally attach a payload, then emit:
// IVariables vars = pipeline; // or workflow — must expose PROJECT_HOME when normalizing
LineageEvent event =
LineageEvent.of(
LineageEventKind.FILE_IO,
LineageContext.builder()
.subjectType(LineageSubjectType.ACTION)
.hopFilename(filename)
.hopFilenamePortableKey(
LineagePortableFilename.portableKey(filename, vars))
.build(),
new FileIoLineagePayload(
FileIoOperation.COPY, sourceUri, targetUri, bytes, true, null));
LineageHub.getInstance().emit(event);
Keep call sites thin: one object construction plus emit. Heavy work belongs in sinks or async code inside the sink implementation.
Related topics
-
Plugin Development — plugin registry,
BasePluginType, annotations. -
Integration testing — patterns that call
HopEnvironment.init()when tests need a full plugin scan.