Spark lake table maintenance

Description

Spark lake table maintenance runs operational SQL / procedures against a Delta Lake or Apache Iceberg table on the native Spark pipeline engine: compaction (OPTIMIZE / rewrite data files), vacuum, expire snapshots, rewrite manifests, or conditional deletes.

It is a zero-input action sink (no hop required). Destructive operations require an explicit acknowledgement.

Connectors ship with the native Spark plugin. Session rules: Lakehouse tables on the native Spark engine.

Options

Option Description

Option	Description
Transform name	Unique name of the transform in the pipeline.
Format	`delta` or `iceberg`.
Identifier mode	`PATH` or `TABLE` for the target table.
Table path / Table identifier / Spark Catalog	Same semantics as other Lake Table transforms.
Operation	See Operations matrix below.
Retention hours	Required for `VACUUM` and `EXPIRE_SNAPSHOTS`. No silent default — you must set how long to retain history.
Retain last	Iceberg `expire_snapshots` `retain_last` (defaults to `1` when retention is set and this field is blank).
Where clause	Optional filter for `OPTIMIZE` / required for `DELETE_WHERE` (without the `WHERE` keyword), e.g. `id < 10`.
Z-Order columns	Optional comma-separated columns for Delta `OPTIMIZE … ZORDER BY (…)`.
Acknowledge destructive	Must be enabled for `VACUUM`, `EXPIRE_SNAPSHOTS`, and `DELETE_WHERE`. Confirms that data files or snapshots may be removed.

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH or TABLE for the target table.

Table path / Table identifier / Spark Catalog

Same semantics as other Lake Table transforms.

Operation

See Operations matrix below.

Retention hours

Required for VACUUM and EXPIRE_SNAPSHOTS. No silent default — you must set how long to retain history.

Retain last

Iceberg expire_snapshots retain_last (defaults to 1 when retention is set and this field is blank).

Where clause

Optional filter for OPTIMIZE / required for DELETE_WHERE (without the WHERE keyword), e.g. id < 10.

Z-Order columns

Optional comma-separated columns for Delta OPTIMIZE … ZORDER BY (…).

Acknowledge destructive

Must be enabled for VACUUM, EXPIRE_SNAPSHOTS, and DELETE_WHERE. Confirms that data files or snapshots may be removed.

Operations matrix

Operation Delta Iceberg

Operation	Delta	Iceberg
OPTIMIZE	OPTIMIZE delta.\`path\ [ZORDER BY …]` (optional WHERE)	`CALL cat.system.rewrite_data_files(table ⇒ '…')`
VACUUM	`VACUUM … RETAIN n HOURS` (n required); destructive ack	Not used — use `EXPIRE_SNAPSHOTS`
EXPIRE_SNAPSHOTS	n/a	`CALL … expire_snapshots(…)` with retention / retain_last; destructive ack
REWRITE_MANIFESTS	n/a	`CALL … rewrite_manifests(…)`
DELETE_WHERE	`DELETE FROM … WHERE …` (WHERE required); destructive ack	Same SQL pattern; destructive ack

OPTIMIZE

OPTIMIZE delta.\`path\ [ZORDER BY …]` (optional WHERE)

CALL cat.system.rewrite_data_files(table ⇒ '…')

VACUUM

VACUUM … RETAIN n HOURS (n required); destructive ack

Not used — use EXPIRE_SNAPSHOTS

EXPIRE_SNAPSHOTS

n/a

CALL … expire_snapshots(…) with retention / retain_last; destructive ack

REWRITE_MANIFESTS

n/a

CALL … rewrite_manifests(…)

DELETE_WHERE

DELETE FROM … WHERE … (WHERE required); destructive ack

Same SQL pattern; destructive ack

Aliases accepted for operation: COMPACT / REWRITE_DATA_FILES → OPTIMIZE; EXPIRE → EXPIRE_SNAPSHOTS; DELETE → DELETE_WHERE.

Behaviour

SQL is executed as a Spark action during graph materialisation; an empty leaf Dataset is registered afterward.
Iceberg CALL procedures use the catalog name from TABLE mode (lake in lake.db.t) or hop_iceberg for PATH mode.
Metrics are log-only; expect little or no GUI row traffic on this sink.

Metadata injection

Supports metadata injection.

Keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, OPERATION, RETENTION_HOURS, RETAIN_LAST, WHERE_CLAUSE, ZORDER_COLUMNS, ACKNOWLEDGE_DESTRUCTIVE.

Injecting pipelines can drive fleet-wide OPTIMIZE/VACUUM across many tables; still set retention and destructive acknowledgement deliberately.

Safety

There is no default retention for vacuum/expire — set hours deliberately for your recovery needs.
Destructive ops without acknowledgement fail validation.
Prefer OPTIMIZE / rewrite for routine compaction; use VACUUM / expire only when you understand snapshot and time-travel impact.