Publish to DataHub

Description

The Publish to DataHub action publishes the pipelines and workflows of a folder to DataHub without running them. Each becomes the pipeline (data flow) and tasks DataHub shows for its runs, with:

  • the tables its transforms write, from the transforms that declare them (Table output, Insert/Update, the bulk loaders, …​);

  • the tables Table input reads, from its SQL;

  • the files Text file input and CSV input read and Text file output writes;

  • the layouts of those files, as their schemas, which DataHub has no other way to know.

No runs are recorded: DataHub learns what the pipelines are and what they read and write, before, or without, any run reporting it. Put the action at the end of a deployment, so DataHub describes what was deployed.

The same is available on the command line as hop datahub-publish; see Publishing a project without running it.

Options

Option Description

Action name

The name of the action.

DataHub lineage sink

A Lineage Sink of type DataHub: the DataHub, environment and platform instance the runs of the project report to, so publishing and running describe the same entities.

Folder

The folder with the pipelines and workflows to publish, with its subfolders. The default is '${PROJECT_HOME}'. The metadata folder and hidden folders are skipped.

Include documentation

Also publish what hop doc writes for each pipeline and workflow (description, file, when it changed) as a DataHub document, related to the pipeline and its tasks. Needs DataHub 1.7 or later; see Documentation.

Include parameters

With Include documentation: list the parameters of each pipeline and workflow, with their default values, in its document.

Include notes

With Include documentation: add the text of the notes of each pipeline and workflow to its document.

Result

The action succeeds when every pipeline and workflow was read and DataHub accepted all of it. It fails when a file can’t be read (the others are still published) or DataHub refused or didn’t receive part of it; the log says which, with DataHub’s answer.

Notes

  • Names are resolved as a run of each pipeline or workflow resolves them: with the variables of the workflow, the file’s own folder as '${Internal.Entry.Current.Folder}', and the default values of its parameters. A transform whose table or file can’t be resolved without running (a file name from a field, a connection that doesn’t exist) is published without that dataset.

  • Text file input publishes the files that match its file list when the action runs. Text file output with the date or time in its file name publishes no file: no run writes a file of that name.

  • Publishing asks no database for anything. Behind a transform with a relational connection, such as a Table input, the fields that reach a transform are not worked out: a file’s layout then comes from the fields its transform lists, and a table’s columns from the field mapping of its transform, without their types. Tables are best described by DataHub’s own database ingestion.