Publish to DataHub
Description
The Publish to DataHub action publishes the pipelines and workflows of a folder to DataHub without running them.
Each becomes the pipeline (data flow) and tasks DataHub shows for its runs, with:
-
the tables its transforms write, from the transforms that declare them (Table output, Insert/Update, the bulk loaders, …);
-
the tables Table input reads, from its SQL;
-
the files Text file input and CSV input read and Text file output writes;
-
the layouts of those files, as their schemas, which DataHub has no other way to know.
No runs are recorded: DataHub learns what the pipelines are and what they read and write, before, or without, any run reporting it. Put the action at the end of a deployment, so DataHub describes what was deployed.
The same is available on the command line as hop datahub-publish; see Publishing a project without running it.
Options
| Option | Description |
|---|---|
Action name |
The name of the action. |
DataHub lineage sink |
A Lineage Sink of type DataHub: the DataHub, environment and platform instance the runs of the project report to, so publishing and running describe the same entities. |
Folder |
The folder with the pipelines and workflows to publish, with its subfolders. The default is |
Include documentation |
Also publish what hop doc writes for each pipeline and workflow (description, file, when it changed) as a DataHub document, related to the pipeline and its tasks. Needs DataHub 1.7 or later; see Documentation. |
Include parameters |
With Include documentation: list the parameters of each pipeline and workflow, with their default values, in its document. |
Include notes |
With Include documentation: add the text of the notes of each pipeline and workflow to its document. |
Result
The action succeeds when every pipeline and workflow was read and DataHub accepted all of it. It fails when a file can’t be read (the others are still published) or DataHub refused or didn’t receive part of it; the log says which, with DataHub’s answer.
Notes
-
Names are resolved as a run of each pipeline or workflow resolves them: with the variables of the workflow, the file’s own folder as
'${Internal.Entry.Current.Folder}', and the default values of its parameters. A transform whose table or file can’t be resolved without running (a file name from a field, a connection that doesn’t exist) is published without that dataset. -
Text file input publishes the files that match its file list when the action runs. Text file output with the date or time in its file name publishes no file: no run writes a file of that name.
-
Publishing asks no database for anything. Behind a transform with a relational connection, such as a Table input, the fields that reach a transform are not worked out: a file’s layout then comes from the fields its transform lists, and a table’s columns from the field mapping of its transform, without their types. Tables are best described by DataHub’s own database ingestion.