Apache Hop
Download
User manual 2.20.0 (pre-release)

Development Documentation

Edit this page

Lake table input transform Icon Lake table input

Description

Lake table input reads an open table format table: Apache Iceberg or Delta Lake.

It supports PATH and TABLE identifiers, optional field projection, and time travel by version/snapshot or timestamp.

The transform runs on two engines:

  • On the Hop engine (local and remote) it reads Apache Iceberg tables with the Iceberg Java library. All files are read through Hop VFS, so a table can live anywhere Hop can reach: local disk, S3, MinIO, Azure, Google Cloud Storage. See On the Hop engine.

  • On the native Spark pipeline engine it reads Delta Lake and Iceberg tables as a Spark SQL Dataset. Connector JARs ship with the native Spark plugin; see Lakehouse tables on the native Spark engine.

It isn’t available on Beam engines.

The transform is part of the Lakehouse plugin (hop-tech-lakehouse). On the native Spark engine the Native Spark engine plugin (hop-engines-spark) is needed as well. Both can be installed with the marketplace when they aren’t in your Hop distribution.

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH — table root path/URI.
TABLE — catalog-qualified name (e.g. lake.db.orders).

Table path

Used when mode is PATH. Must be visible to Spark workers (local path on local[*]; shared URI on cluster).

Table identifier

Used when mode is TABLE. Multi-part name: catalog.namespace.table.

Catalog metadata name

The lakehouse catalog to use in TABLE mode (Iceberg Hadoop/REST, …). On the Spark engine you can leave it empty if the catalog is already configured on the session.

Time travel type

NONE (current), VERSION, or TIMESTAMP.

Time travel version

Delta table version number or Iceberg snapshot id (when type is VERSION). Supports variables.

Time travel timestamp

Timestamp when type is TIMESTAMP, e.g. 2024-01-15 10:30:00. The Hop engine also accepts an ISO-8601 timestamp with an offset and epoch milliseconds.

Extra options

Optional Spark read options as key=value lines (advanced). Not used on the Hop engine.

Fields

Optional projection. When empty, all columns are returned. When set, columns are selected/cast by name (same idea as Spark file input with a schema).

Time travel

Type Delta Iceberg

NONE

Current table version

Current snapshot

VERSION

Read option versionAsOf

SQL VERSION AS OF <snapshot_id>

TIMESTAMP

Read option timestampAsOf

SQL TIMESTAMP AS OF TIMESTAMP '…'

Tip: After two Overwrite writes, Delta version 0 is typically the first commit. For Iceberg, inspect {table}.snapshots (or catalog metadata) for snapshot ids.

On the Hop engine

On the local and remote Hop engines the transform reads Apache Iceberg tables. Delta Lake tables give an error that asks you to use the Spark engine for now.

Finding the table

  • PATH mode reads the table in the given folder. The folder can be any Hop VFS location, for example file:///data/orders or s3://lake/orders. When the table has a metadata/version-hint.text file (tables written by a Hadoop catalog, by the Spark engine in PATH mode, or by Lake table output on the Hop engine) the version it names is read. Otherwise the newest metadata file is read.

  • TABLE mode looks the table up through the selected lakehouse catalog. The Hop engine supports the hadoop and rest catalog types. A catalog name in front of the identifier (lake.db.orders) is dropped, so the same identifier works on both engines. With a REST catalog the files are read through Hop VFS, unless the catalog’s extra configuration names another io-impl, for example org.apache.iceberg.aws.s3.S3FileIO for credentials vended by the catalog.

Reading

  • Iceberg plans the read: it skips the files and row groups that can’t match, using the statistics in the table metadata.

  • Delete files written by other engines (position and equality deletes, for example from Flink or Spark) are applied.

  • Without a field list all columns are returned, with these Hop types: booleans as Boolean, integers as Integer, floating point numbers as Number, decimals as BigNumber, dates as Date, timestamps as Timestamp, binary values as Binary and everything else as String. Dates and timestamps without a time zone (date, timestamp, timestamp_ns) are read as local dates and date-times in the time zone of the Hop JVM, as Parquet Input does, so the wall-clock value in the table is kept. Timestamps with a time zone (timestamptz) are instants and keep their point in time. With a field list the columns are returned in the order of the list and converted to the type of each field.

  • When the transform runs as several copies, the copies share the table between them. Each part of a data file is read by exactly one copy.

  • A time travel timestamp without a time zone is in the time zone of the Hop server.

Metadata injection

All options support metadata injection (ETL Metadata Injection).

Common keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, TIME_TRAVEL_TYPE, TIME_TRAVEL_VERSION, TIME_TRAVEL_TIMESTAMP, EXTRA_OPTIONS.

Field projection group FIELDS: NAME, TYPE, LENGTH, PRECISION, FORMAT_MASK.

Typical pattern: template pipeline with Lake Table Input; injecting pipeline supplies table paths / identifiers (and optional field list) per supplier or table.

Notes

  • On the Spark engine, prefer Lake table input over Spark file input for delta / iceberg so session extensions, Iceberg PATH catalogs, and probes stay consistent.

  • See the lakehouse guide for packaging, PATH vs TABLE, and troubleshooting.

Related pages