Spark Lake Table Output transform Icon Spark Lake Table Output

Description

Spark Lake Table Output writes the incoming Spark Dataset to a Delta Lake or Apache Iceberg table using table-format commit semantics on the native Spark pipeline engine.

Unlike Spark File Output (classic file formats), the default save mode is ErrorIfExists so ACID tables are not overwritten by accident.

This transform is native Spark only. Connectors ship with the native Spark plugin; see Lakehouse tables on the native Spark engine.

Supported Engines

Hop Engine

Not Supported

Single Threaded

Not Supported

Native Spark

Supported

Beam Spark

Supported

Beam Flink

Not Supported

Beam Dataflow

Not Supported
Spark here means Hop’s native Spark pipeline engine only (not Beam Spark).

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH or TABLE (see PATH vs TABLE).

Table path

PATH mode: table root directory / URI.

Table identifier

TABLE mode: e.g. lake.db.orders.

Spark Catalog metadata name

Optional Spark Catalog Hop metadata for TABLE mode.

Save mode

ErrorIfExists (default), Append, Overwrite, or Ignore. Choose Overwrite or Append explicitly when you intend to replace or extend data.

Partition by

Optional comma-separated partition column list.

Coalesce

Optional partition count before write (e.g. 1 for a single file layout — costs a shuffle).

Extra options

Optional Spark write options as key=value lines (advanced).

Behaviour

  • The write runs as a Spark action when the engine materialises the graph.

  • An empty leaf Dataset is registered so a later engine-wide count() does not re-write the table.

  • Delta PATH: format("delta").mode(…).save(path) (requires Delta extension + DeltaCatalog on the session — hop-run applies these when Delta transforms are present).

  • Iceberg PATH: writeTo(hop_iceberg.\`uri\).using("iceberg")` with the built-in hop_iceberg Hadoop catalog.

  • Iceberg TABLE: writeTo(catalog.ns.table).using("iceberg").

  • Delta TABLE (advanced): format("delta").saveAsTable(id).

Metadata injection

Supports metadata injection.

Keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, SAVE_MODE, PARTITION_BY, COALESCE, EXTRA_OPTIONS.

Use this with a template write path when landing many tables (different paths/identifiers/save modes) from one injecting pipeline.

Notes

  • Requires exactly one main-stream upstream hop (the data to write).

  • For upserts, use Spark Lake Table Merge instead of Overwrite.

  • Packaging, session conf, and troubleshooting: lakehouse guide.

Related pages