Spark Lake Table Maintenance transform Icon Spark Lake Table Maintenance

Description

Spark Lake Table Maintenance runs operational SQL / procedures against a Delta Lake or Apache Iceberg table on the native Spark pipeline engine: compaction (OPTIMIZE / rewrite data files), vacuum, expire snapshots, rewrite manifests, or conditional deletes.

It is a zero-input action sink (no hop required). Destructive operations require an explicit acknowledgement.

Connectors ship with the native Spark plugin. Session rules: Lakehouse tables on the native Spark engine.

Supported Engines

Hop Engine

Not Supported

Single Threaded

Not Supported

Native Spark

Supported

Beam Spark

Supported

Beam Flink

Not Supported

Beam Dataflow

Not Supported
Spark here means Hop’s native Spark pipeline engine only (not Beam Spark).

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH or TABLE for the target table.

Table path / Table identifier / Spark Catalog

Same semantics as other Lake Table transforms.

Operation

See Operations matrix below.

Retention hours

Required for VACUUM and EXPIRE_SNAPSHOTS. No silent default — you must set how long to retain history.

Retain last

Iceberg expire_snapshots retain_last (defaults to 1 when retention is set and this field is blank).

Where clause

Optional filter for OPTIMIZE / required for DELETE_WHERE (without the WHERE keyword), e.g. id < 10.

Z-Order columns

Optional comma-separated columns for Delta OPTIMIZE … ZORDER BY (…).

Acknowledge destructive

Must be enabled for VACUUM, EXPIRE_SNAPSHOTS, and DELETE_WHERE. Confirms that data files or snapshots may be removed.

Operations matrix

Operation Delta Iceberg

OPTIMIZE

OPTIMIZE delta.\`path\ [ZORDER BY …]` (optional WHERE)

CALL cat.system.rewrite_data_files(table ⇒ '…')

VACUUM

VACUUM … RETAIN n HOURS (n required); destructive ack

Not used — use EXPIRE_SNAPSHOTS

EXPIRE_SNAPSHOTS

n/a

CALL … expire_snapshots(…) with retention / retain_last; destructive ack

REWRITE_MANIFESTS

n/a

CALL … rewrite_manifests(…)

DELETE_WHERE

DELETE FROM … WHERE … (WHERE required); destructive ack

Same SQL pattern; destructive ack

Aliases accepted for operation: COMPACT / REWRITE_DATA_FILES → OPTIMIZE; EXPIRE → EXPIRE_SNAPSHOTS; DELETE → DELETE_WHERE.

Behaviour

  • SQL is executed as a Spark action during graph materialisation; an empty leaf Dataset is registered afterward.

  • Iceberg CALL procedures use the catalog name from TABLE mode (lake in lake.db.t) or hop_iceberg for PATH mode.

  • Metrics are log-only; expect little or no GUI row traffic on this sink.

Metadata injection

Supports metadata injection.

Keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, OPERATION, RETENTION_HOURS, RETAIN_LAST, WHERE_CLAUSE, ZORDER_COLUMNS, ACKNOWLEDGE_DESTRUCTIVE.

Injecting pipelines can drive fleet-wide OPTIMIZE/VACUUM across many tables; still set retention and destructive acknowledgement deliberately.

Safety

  • There is no default retention for vacuum/expire — set hours deliberately for your recovery needs.

  • Destructive ops without acknowledgement fail validation.

  • Prefer OPTIMIZE / rewrite for routine compaction; use VACUUM / expire only when you understand snapshot and time-travel impact.

Related pages