Data Catalog
Description
A Data Catalog is a catalog, such as DataHub, that Hop reads dataset metadata from: whether a dataset exists, whether it is deprecated, when its data last changed and how the checks the catalog runs on it went. The catalog stays the owner of that metadata. Hop only reads it.
The Check data catalog action uses a Data Catalog to check datasets before a workflow goes on.
This metadata type is experimental: its options and what the catalog types can answer may still change.
Options
| Option | Description |
|---|---|
Name |
The name of the Data Catalog. |
Description |
A description of the Data Catalog. |
Catalog type |
The kind of catalog. The options below it depend on it. A catalog type comes with a plugin, so the list only shows the types of the plugins that are installed. |
Test (button at the bottom) |
Connects to the catalog with the options on the screen and the current variables, and shows the answer: the version of the catalog and the user Hop reads as, or why it could not be used. |
How datasets are found
Hop names a dataset the same way the DataHub lineage plugin does, so a catalog finds the dataset that lineage wrote to it.
The OpenLineage plugin names tables the same way, and files too when HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING is Y.
-
A table: the host and port of its relational connection, and
database.schema.table. A table without a schema gets the preferred schema of the connection, and the database name of the connection is always added. A connection without a host, such as an embedded database or a connection with a manual URL, gives a table no name to look for: use its catalog id instead. -
A file: its path or URL, after the variables in it are resolved.
-
A catalog id: the id the catalog itself uses, for example a DataHub URN. Use it for a dataset that Hop doesn’t read or write itself.
DataHub
The DataHub catalog type comes with the DataHub plugin. It reads from DataHub’s GraphQL API, with fields that DataHub has had since version 1.0.
| Option | Description | Default |
|---|---|---|
DataHub GMS URL |
The URL of DataHub’s metadata service (GMS), for example |
|
Access token |
A personal access token, with the right to view the datasets and their assertions. Leave it empty when DataHub doesn’t use token authentication. |
|
Environment |
The environment (fabric) in the dataset URNs, for example |
|
Connect timeout (ms) |
How long to wait for a connection to DataHub. |
|
Read timeout (ms) |
How long to wait for DataHub to answer a question. |
|
Trust store, Trust store password, Trust store type |
A trust store with the certificate of a DataHub whose certificate Java doesn’t trust, its password and its type ( |
|
Every option takes variables, for example '${DATAHUB_URL}' and '${DATAHUB_TOKEN}'.
What DataHub answers:
-
Exists: whether DataHub has the dataset.
-
Deprecated: the deprecation set on the dataset in DataHub, with its note.
-
Last change: the newest operation DataHub has for the dataset. DataHub learns about operations from the sources that report them, such as the usage ingestion of some warehouses or dbt. The DataHub lineage sink of Hop doesn’t report them yet, so a table that only Hop writes has none.
-
Assertions: the latest run of each assertion on the dataset, up to 100 assertions. The assertions themselves are defined and run in DataHub.