Pipeline Log

Description

pipeline log

Allows to log the activity of a pipeline with another pipeline.

A pipeline log streams logging information from a running pipeline to another pipeline.

The only requirement for the receiving pipeline is that it starts with a Pipeline logging transform. Other than that, the logging pipeline is "just another pipeline". In this logging pipeline, you can process the logging information and e.g. write to a relational or NoSQL database, a Kafka topic etc

When changes take effect

A run reads the pipeline logs once, when it first needs them, and every pipeline and workflow it starts uses what it read: the child pipelines of a Pipeline executor, the pipelines of a workflow, a mapping and so on. A run is a top-level pipeline or workflow, the one that Hop Run, Hop GUI, Hop Server or the API starts.

A pipeline log that you add or change while a run is going on takes effect with the next run. This matters for runs that go on for a long time, such as a workflow that loops for hours or a streaming pipeline.

Samples

The samples project comes with an example pipeline log.

Check the metadata perspective for the pipeline log pipeline-log-example. This pipeline log is configured to send logging information for the pipeline ${PROJECT_HOME}/reflection/generate-fake-books.hpl to the logging pipeline ${PROJECT_HOME}/reflection/pipeline-log-example.hpl.

Pipeline Log

Options

Option Default Description

Name

The name to be used for this pipeline log

Enabled?

true

Logging parent pipelines only

false

If you enable this, only the parent pipelines, executed by Hop Run, GUI, Server or API will be logged. When disabled, every pipeline execution will be logged.

Pipeline executed to capture logging

the pipeline to process the logging information for this pipeline log

Execute at the start of the pipeline?

true

Should this pipeline log be executed at the start of a pipeline run

Execute at the end of the pipeline?

false

Should this pipeline log be executed at the end of a pipeline run

Execute periodically during execution?

true

Should this pipeline log be executed periodically during a pipeline run

Interval in seconds

30

if executed periodically, indicates the interval at which the pipeline log is executed