Pipeline Log


pipeline log

Allows to log the activity of a pipeline with another pipeline.

A pipeline log streams logging information from a running pipeline to another pipeline.

The only requirement for the receiving pipeline is that it starts with a Pipeline Logging transform. Other than that, the logging pipeline is "just another pipeline". In this logging pipeline, you can process the logging information and e.g. write to a relational or NoSQL database, a Kafka topic etc


The samples project comes with an example pipeline log.

Check the metadata perspective for the pipeline log pipeline-log-example. This pipeline log is configured to send logging information for the pipeline ${PROJECT_HOME}/reflection/generate-fake-books.hpl to the logging pipeline ${PROJECT_HOME}/reflection/pipeline-log-example.hpl.

Pipeline Log


Option Default Description


The name to be used for this pipeline log



Logging parent pipelines only


Pipeline executed to capture logging

the pipeline to process the logging information for this pipeline log

Execute at the start of the pipeline?


Should this pipeline log be executed at the start of a pipeline run

Execute at the end of the pipeline?


Should this pipeline log be executed at the end of a pipeline run

Execute periodically during execution?


Should this pipeline log be executed periodically during a pipeline run

Interval in seconds


if executed periodically, indicates the interval at which the pipeline log is executed