Pipeline logging transform Icon Pipeline Logging

Description

the Pipeline Logging transform receives its input from a Pipeline Log metadata item.

The transform requires very little configuration, but provides a lot of information.

Supported Engines

Hop Engine

Supported

Spark

Maybe Supported

Flink

Maybe Supported

Dataflow

Maybe Supported

Related Plugins

Options

Option Description

Transform name

Name of this transform

Also log transform details

  • checked: the transform outputs both pipeline and transform logging and metrics. In this case, the logging will contain one line per transform. For each transform, the pipeline logging and metrics information is repeated.

  • unchecked: the transform only outputs pipeline logging and metrics.

Output Fields

Fieldname Type Description

Pipeline Logging Fields

loggingDate

Date

the date and time this pipeline was executed

loggingPhase

String

the phase this pipeline is in at the time of logging (e.g. 'end')

pipelineName

String

the logged pipeline name

pipelineFilename

String

the logged pipeline file name

pipelineStart

Date

execution start date for the logged pipeline

pipelineEnd

Date

execution end date for the logged pipeline

pipelineLogChannelId

String

logging channel id for the logged pipeline

parentLogChannelId

String

parent logging channel id for the logged pipeline (e.g. the channel id of the parent workflow)

pipelineLogging

String

full logging text for the logged pipeline

pipelineErrorCount

Integer

number of errors that occurred in the logged pipeline execution

pipelineStatusDescription

String

status description for the logged pipeline (e.g. 'Finished')

Transform Logging Fields

transformName

String

the name of the transform in the logged pipeline

transformCopyNr

Integer

the copy number of the current transform

transformStatusDescription

String

status of the current transform (e.g. 'Finished')

transformLogChannelId

String

log channel id for the current transform

transformLoggingText

String

logging text for the current transform

transformLinesRead

Integer

lines read for the current transform

transformLinesWritten

Integer

lines read for the current transform

transformLinesInput

Integer

input lines for the current transform

transformLinesOutput

Integer

output lines for the current transform

transformLinesUpdated

Integer

updated lines for the current transform

transformLinesRejected

Integer

rejected lines for the current transform

transformErrors

Integer

number of errors for the current transform

transformStart

Date

start date and time for the current transform

transformEnd

Date

end date and time for the current transform

transformDuration

Integer

total duration for the current transform