Pipeline Log
Description
Allows to log the activity of a pipeline with another pipeline.
A pipeline log streams logging information from a running pipeline to another pipeline.
The only requirement for the receiving pipeline is that it starts with a Pipeline Logging transform. Other than that, the logging pipeline is "just another pipeline". In this logging pipeline, you can process the logging information and e.g. write to a relational or NoSQL database, a Kafka topic etc
Samples
The samples project comes with an example pipeline log.
Check the metadata perspective for the pipeline log pipeline-log-example. This pipeline log is configured to send logging information for the pipeline ${PROJECT_HOME}/reflection/generate-fake-books.hpl to the logging pipeline ${PROJECT_HOME}/reflection/pipeline-log-example.hpl.
Options
| Option | Default | Description |
|---|---|---|
Name | The name to be used for this pipeline log | |
Enabled? | true | |
Logging parent pipelines only | true | |
Pipeline executed to capture logging | the pipeline to process the logging information for this pipeline log | |
Execute at the start of the pipeline? | true | Should this pipeline log be executed at the start of a pipeline run |
Execute at the end of the pipeline? | false | Should this pipeline log be executed at the end of a pipeline run |
Execute periodically during execution? | true | Should this pipeline log be executed periodically during a pipeline run |
Interval in seconds | 30 | if executed periodically, indicates the interval at which the pipeline log is executed |