PostgreSQL Bulk Loader

Description

The PostgreSQL Bulk Loader transform streams data from Hop to Postgresql using "COPY DATA FROM STDIN" into the database.

replace boolean fields in your pipeline stream by string fields with "Y" or "N" values to avoid errors.

Supported Engines

Hop Engine

Supported

Spark

Maybe Supported

Flink

Maybe Supported

Dataflow

Maybe Supported

The PostgreSQL Bulk Loader is linked to the database type. It will fetch the JDBC driver from the hop/lib/jdbc folder.

Valid locations for the JDBC driver for this transform are the database plugin lib and the main hop/lib folder. It will not work in combination with the SHARED_JDBC_FOLDER variable.

Options

Option Description

Option	Description
Transform name	Name of the transform.
Connection	Name of the database connection on which the target table resides.
Target schema	The name of the Schema for the table to write data to. This is important for data sources that allow for table names with dots '.' in it.
Target table	Name of the target table.
Load action	Insert, Truncate. Insert inserts, truncate first truncates the table.
DB Name Override	(optional) database name to override the database name used in this transform’s connection.
Enclosure	the enclosure character to use in the `QUOTE AS` part of the copy command
Delimiter	the delimiter character to use in the `DELIMITER AS` part of the copy command
Stop on error	Stop the execution of this transform when an error occurs
Fields to load	This table contains a list of fields to load data from, properties include: Table field: Table field to be loaded in the PostgreSQL table; Stream field: Field to be taken from the incoming rows; Date mask: Either "Pass through, "Date" or "DateTime", determines how date/timestamps will be loaded in PostgreSQL.

Transform name

Name of the transform.

Connection

Name of the database connection on which the target table resides.

Target schema

The name of the Schema for the table to write data to. This is important for data sources that allow for table names with dots '.' in it.

Target table

Name of the target table.

Load action

Insert, Truncate. Insert inserts, truncate first truncates the table.

DB Name Override

(optional) database name to override the database name used in this transform’s connection.

Enclosure

the enclosure character to use in the QUOTE AS part of the copy command

Delimiter

the delimiter character to use in the DELIMITER AS part of the copy command

Stop on error

Stop the execution of this transform when an error occurs

Fields to load

This table contains a list of fields to load data from, properties include:

Table field: Table field to be loaded in the PostgreSQL table;
Stream field: Field to be taken from the incoming rows;
Date mask: Either "Pass through, "Date" or "DateTime", determines how date/timestamps will be loaded in PostgreSQL.

Metadata Injection Support

All fields of this transform support metadata injection. You can use this transform with Metadata Injection to pass metadata to your pipeline at runtime.