Workflow logging transform Icon Workflow logging

Description

the Workflow Logging transform receives its input from a Workflow Log metadata item.

The transform requires very little configuration, but provides a lot of information.

Supported Engines

Hop Engine

Supported

Spark

Maybe Supported

Flink

Maybe Supported

Dataflow

Maybe Supported

Options

Option Description

Transform name

Name of this transform

Also log action details

  • checked: the transform outputs both workflow and action logging and metrics. In this case, the logging will contain one line per action. For each action, the workflow logging and metrics information is repeated.

  • unchecked: the transform only outputs workflow logging and metrics.

Output Fields

Fieldname Type Description

Workflow Logging Fields

loggingDate

Date

the date and time this workflow was executed

loggingPhase

String

the phase this workflow is in at the time of logging (e.g. 'end')

workflowName

String

the logged workflow name

workflowFilename

String

the logged workflow file name

workflowStart

Date

execution start date for the logged workflow

workflowEnd

Date

execution end date for the logged workflow

workflowLogChannelId

String

logging channel id for the logged workflow

parentLogChannelId

String

parent logging channel id for the logged workflow (e.g. the channel id of the parent workflow)

workflowLogging

String

full logging text for the logged workflow

workflowErrorCount

Integer

number of errors that occurred in the logged workflow execution

workflowStatusDescription

String

status description for the logged workflow (e.g. 'Finished')

Action Logging Fields

actionName

String

the name of the action in the logged pipeline

actionNr

Integer

the number of the current action

actionResult

the result (exit code, Y/N) of the current action

actionLogChannelId

String

log channel id for the current action

actionLoggingText

String

logging text for the current action

actionErrors

Integer

number of errors for the current action

actionLogDate

String

status of the current action (e.g. 'Finished')

actionDuration

Integer

total duration for the current action

actionExitStatus

exit status of the current action

actionNrFilesRetrieved

the number of files retrieved by the current transform (if applicable)

actionFilename

the filename to use for the current action, e.g. the workflow or pipeline the current action runs.

actionComment

an optional comment added to the logs, e.g. Workflow execution finished

actionReason

an optional reason provided by the current action