Apache Hop
Download
User manual 2.20.0 (pre-release)

Development Documentation

Edit this page

Lake table output transform Icon Lake table output

Description

Lake table output writes the incoming rows to an Apache Iceberg or Delta Lake table, using the table format’s commits so readers never see a half-written run.

The default save mode is ErrorIfExists, so an existing table isn’t overwritten by accident.

The transform runs on two engines:

The transform is part of the Lakehouse plugin (hop-tech-lakehouse). On the native Spark engine the Native Spark engine plugin (hop-engines-spark) is needed as well. Both can be installed with the marketplace when they aren’t in your Hop distribution.

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH or TABLE (see PATH vs TABLE).

Table path

PATH mode: table root directory / URI.

Table identifier

TABLE mode: e.g. lake.db.orders.

Catalog metadata name

The lakehouse catalog for TABLE mode. Optional on the Spark engine.

Save mode

ErrorIfExists (default), Append, Overwrite, or Ignore. Choose Overwrite or Append explicitly when you intend to replace or extend data.

Partition by

Optional comma-separated partition column list.

Coalesce

Optional partition count before write (e.g. 1 for a single file layout — costs a shuffle). Spark engine only.

Extra options

Optional Spark write options as key=value lines (advanced). Spark engine only.

On the Hop engine

On the local and remote Hop engines the transform writes Apache Iceberg tables. Delta Lake tables give an error that asks you to use the Spark engine for now.

One commit per pipeline run

Every copy of the transform writes its own Parquet files. Nothing is visible in the table until the whole pipeline has finished:

  • When the pipeline ends without errors, all files of the run are committed as one Iceberg snapshot. The snapshot summary records the pipeline name (hop.pipeline) and the run’s log channel id (hop.run-id).

  • When the pipeline fails or is stopped, the files are deleted and the table is left as it was. This includes a failure in a transform after Lake table output: the run is rolled back, like a database transaction.

Save modes

Save mode Table exists Table doesn’t exist

ErrorIfExists

The pipeline fails, the table is left alone

The table is created

Append

The rows are added

The table is created

Overwrite

All rows of the table are replaced by the rows of this run. The run fails instead if another writer changed the table since the run started.

The table is created

Ignore

Nothing is written; the rows are still passed on

The table is created

A new table gets its columns from the incoming rows (Integer as long, Number as double, BigNumber as decimal, String as string, Boolean as boolean, Date and Timestamp as timestamp without time zone, Binary as binary) and identity partitions on the Partition by columns. It is created in the same commit as its first rows, so a failed run doesn’t leave an empty table behind. When the table exists, the rows are matched to its columns by name.

  • A BigNumber column gets the field’s length as precision (at most 38) and its precision as scale when the length is greater than 0; otherwise it is decimal(38, 10).

  • Dates and timestamps are written as the wall-clock time in the time zone of the Hop JVM, so they read back unchanged on the Hop engine, as with Parquet Output. Into an existing timestamptz column they are written as instants.

Unlike the Spark engine, Overwrite on the Hop engine keeps the table’s schema and partitioning and only replaces its rows.

Where the table is

  • PATH mode writes the table in the given Hop VFS folder, in the layout of Hadoop catalogs and of the Spark engine’s PATH mode (metadata/v<N>.metadata.json plus version-hint.text). The Spark and Hop engines can read and write the same table. A folder that holds a table managed by a catalog is refused: write that table through its catalog in TABLE mode.

  • TABLE mode writes through the selected lakehouse catalog. The Hop engine supports the hadoop and rest catalog types.

On a local file system (file://), a PATH table commit publishes the new metadata file with an operation that fails if another writer already committed that version: a hard link, or an exclusive create where hard links aren’t supported. Several pipelines can write the same table there; of two conflicting commits one fails and is retried. On HDFS, object stores like S3 and other non-local Hop VFS locations, the commit checks that the version is free and then writes it, which isn’t atomic between writers, so a PATH table there should have one writer at a time. When several pipelines write such a table at the same time, use a REST catalog.

On the Spark engine

  • The write runs as a Spark action when the engine materialises the graph.

  • An empty leaf Dataset is registered so a later engine-wide count() does not re-write the table.

  • Delta PATH: format("delta").mode(…).save(path) (requires Delta extension + DeltaCatalog on the session — hop-run applies these when Delta transforms are present).

  • Iceberg PATH: writeTo(hop_iceberg_<hash>.\`table\).using("iceberg")`, with a Hadoop catalog whose warehouse is the table’s parent folder, so the table is written at its path.

  • Iceberg TABLE: writeTo(catalog.ns.table).using("iceberg").

  • Delta TABLE (advanced): format("delta").saveAsTable(id).

Metadata injection

Supports metadata injection.

Keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, SAVE_MODE, PARTITION_BY, COALESCE, EXTRA_OPTIONS.

Use this with a template write path when landing many tables (different paths/identifiers/save modes) from one injecting pipeline.

Notes

  • Requires exactly one main-stream upstream hop (the data to write).

  • For upserts on the Spark engine, use Spark lake table merge instead of Overwrite.

  • Packaging, session conf, and troubleshooting: lakehouse guide.

Related pages