Pipeline logging

Description

The pipeline logging transform provides detailed logging information about a pipeline and its transforms. This transform receives input from a Pipeline Log metadata item.

The transform requires very little configuration, but provides a lot of information.

Supported Engines

Hop Engine

Supported

Single Threaded

Supported

Native Spark

Supported

Beam Spark

Maybe Supported

Beam Flink

Maybe Supported

Beam Dataflow

Maybe Supported

Related Plugins

Pipeline Log

Options

Option	Description
Transform name	Name of this transform
Also log transform details	checked: the transform outputs both pipeline and transform logging and metrics. In this case, the logging will contain one line per transform. For each transform, the pipeline logging and metrics information is repeated. unchecked: the transform only outputs pipeline logging and metrics.

Option

Description

Transform name

Name of this transform

Also log transform details

checked: the transform outputs both pipeline and transform logging and metrics. In this case, the logging will contain one line per transform. For each transform, the pipeline logging and metrics information is repeated.
unchecked: the transform only outputs pipeline logging and metrics.

Output Fields

Fieldname	Type	Description
Pipeline logging Fields
loggingDate	Date	the date and time this pipeline was executed
loggingPhase	String	the phase this pipeline is in at the time of logging (e.g. 'end')
pipelineName	String	the logged pipeline name
pipelineFilename	String	the logged pipeline file name
pipelineStart	Date	execution start date for the logged pipeline
pipelineEnd	Date	execution end date for the logged pipeline
pipelineLogChannelId	String	logging channel id for the logged pipeline
parentLogChannelId	String	parent logging channel id for the logged pipeline (e.g. the channel id of the parent workflow)
pipelineLogging	String	full logging text for the logged pipeline
pipelineErrorCount	Integer	number of errors that occurred in the logged pipeline execution
pipelineStatusDescription	String	status description for the logged pipeline (e.g. 'Finished')
Transform Logging Fields
transformName	String	the name of the transform in the logged pipeline
transformCopyNr	Integer	the copy number of the current transform
transformStatusDescription	String	status of the current transform (e.g. 'Finished')
transformLogChannelId	String	log channel id for the current transform
transformLoggingText	String	logging text for the current transform
transformLinesRead	Integer	lines read for the current transform
transformLinesWritten	Integer	lines read for the current transform
transformLinesInput	Integer	input lines for the current transform
transformLinesOutput	Integer	output lines for the current transform
transformLinesUpdated	Integer	updated lines for the current transform
transformLinesRejected	Integer	rejected lines for the current transform
transformErrors	Integer	number of errors for the current transform
transformStart	Date	start date and time for the current transform
transformEnd	Date	end date and time for the current transform
transformDuration	Integer	total duration for the current transform

Fieldname

Type

Description

Pipeline logging Fields

loggingDate

Date

the date and time this pipeline was executed

loggingPhase

String

the phase this pipeline is in at the time of logging (e.g. 'end')

pipelineName

String

the logged pipeline name

pipelineFilename

String

the logged pipeline file name

pipelineStart

Date

execution start date for the logged pipeline

pipelineEnd

Date

execution end date for the logged pipeline

pipelineLogChannelId

String

logging channel id for the logged pipeline

parentLogChannelId

String

parent logging channel id for the logged pipeline (e.g. the channel id of the parent workflow)

pipelineLogging

String

full logging text for the logged pipeline

pipelineErrorCount

Integer

number of errors that occurred in the logged pipeline execution

pipelineStatusDescription

String

status description for the logged pipeline (e.g. 'Finished')

Transform Logging Fields

transformName

String

the name of the transform in the logged pipeline

transformCopyNr

Integer

the copy number of the current transform

transformStatusDescription

String

status of the current transform (e.g. 'Finished')

transformLogChannelId

String

log channel id for the current transform

transformLoggingText

String

logging text for the current transform

transformLinesRead

Integer

lines read for the current transform

transformLinesWritten

Integer

lines read for the current transform

transformLinesInput

Integer

input lines for the current transform

transformLinesOutput

Integer

output lines for the current transform

transformLinesUpdated

Integer

updated lines for the current transform

transformLinesRejected

Integer

rejected lines for the current transform

transformErrors

Integer

number of errors for the current transform

transformStart

Date

start date and time for the current transform

transformEnd

Date

end date and time for the current transform

transformDuration

Integer

total duration for the current transform