Apache Hop
Download
User manual 2.20.0 (pre-release)

Development Documentation

Edit this page

Lakehouse catalog

Description

Lakehouse catalog (called Spark Catalog in earlier versions) is a reusable Hop metadata type (category Connections) that describes a catalog for open table formats. Lake table transforms in TABLE mode reference a catalog by name.

  • On the native Spark pipeline engine Hop expands the entry into spark.sql.catalog.<catalogName>.* configuration when building or preparing the session (LakeSessionPlan / SparkCatalogApplier).

  • On the Hop engine, Lake table input and Lake table output open the catalog with the Iceberg Java library. The hadoop and rest catalog types are supported there. The REST catalog gets the URI, Warehouse and Credential (as token), plus the Extra conf lines as catalog properties; Spark-style keys (spark.sql.catalog.<name>.<key>) are accepted, so one catalog definition serves both engines. Files go through Hop VFS unless Extra conf sets io-impl.

Existing catalogs keep working: the metadata key (spark-catalog) hasn’t changed.

PATH-mode Iceberg tables do not require this metadata type — the engine registers a Hadoop catalog for the folder that holds each table (hop_iceberg_<hash>). Use a lakehouse catalog for production Iceberg warehouses, REST catalogs, and other named catalogs.

Connector packaging and session matrix: Lakehouse tables on the native Spark engine.

Load catalog template

Use Load catalog template at the top of the form to fill fields from a named scenario. Placeholders (bucket, host, URI) must still be adjusted for your environment. Secrets are never stored in templates — put tokens in Credential.

Every template writes a commented documentation link as the first line of Extra conf, for example:

# docs: https://iceberg.apache.org/docs/latest/spark-configuration/

# lines are ignored when Hop builds the Spark session, so the link is guidance only.

Template Type What it sets

Iceberg Hadoop (local)

hadoop

Catalog lake, warehouse file:///tmp/hop-warehouse

Iceberg Hadoop (object store)

hadoop

Catalog lake, warehouse s3a://bucket/warehouse, S3 io-impl in conf extra

Iceberg REST

rest

Catalog lake, URI https://catalog.example.com/v1

Iceberg REST (authenticated)

rest

Same as REST; conf extra notes that Credential maps to catalog .token

Hive Metastore (advanced)

hive

Catalog hive, thrift URI placeholder (extra deps required)

AWS Glue (advanced)

glue

Catalog glue, S3 warehouse + IO placeholders (extra deps required)

Nessie (advanced)

custom

Catalog nessie, Nessie catalog-impl / URI / ref placeholders

Databricks Unity Catalog (advanced)

custom

Catalog unity, commented workspace URI/warehouse examples

Delta named catalog (advanced)

custom

DeltaCatalog implementation; prefer spark_catalog when coexisting with Iceberg

The Hop metadata Name (how transforms select this entry) is not changed by a template — only catalog fields are.

Options

Option Description

Name

Hop metadata name (how transforms reference this entry).

Load catalog template

Button that applies a named scenario (see above). Confirms before replacing non-empty fields.

Catalog name

Spark SQL catalog name — first segment of catalog.namespace.table (e.g. lake).

Catalog type

hadoop — Iceberg Hadoop warehouse (default).
rest — Iceberg REST catalog.
custom — supply implementation + conf extra.
hive / glue — advanced; conf-extra oriented, not packaged by default.

Implementation

Spark catalog class. Default for Iceberg is org.apache.iceberg.spark.SparkCatalog.

Warehouse

Warehouse path for Hadoop (and similar) catalogs, e.g. file:///data/warehouse or an object-store URI.

URI

Catalog service URI (REST and similar).

Credential

Optional secret (password field). For REST catalogs, applied as spark.sql.catalog.<name>.token when set.

Conf extra

Additional key=value lines (one per line; # comments and blank lines ignored).
Short keys (e.g. io-impl) become spark.sql.catalog.<catalogName>.io-impl.
Keys that already start with spark. are applied as full Spark conf keys.

How it is applied

For a catalog named lake of type hadoop with warehouse file:///tmp/wh, Hop effectively sets:

spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type=hadoop
spark.sql.catalog.lake.warehouse=file:///tmp/wh

Lake Table transforms then use identifiers such as lake.db.orders.

When both Delta and Iceberg are used in one session, keep Delta on spark_catalog (DeltaCatalog) and put Iceberg under a different catalog name (e.g. lake or hop_iceberg).

spark-submit

Export project metadata JSON that includes every lakehouse catalog entry referenced by the pipeline (and any used only via run configuration). Alternatively, pass equivalent --conf spark.sql.catalog.… flags on the submit command. Paths and warehouses must be reachable from the driver and executors.

Related pages