Parquet file output

Transform name

Name of the transform this name has to be unique in a single pipeline.

Base file name

Specify the base filename. This is composed of where you want to write the Parquet file to as well as the start of the filename. Examples:

Write to Amazon AWS S3 : s3://my-bucket-name/transactions

Write to a local folder : /my/folder/customer-data

Extension

This is the extension of the file. Usually this is simply parquet

Include date?

Check this box if you want to include the date in the filename with mask yyyMMdd

Include time?

Check this box if you want to include the time in the filename with mask HHmmss

Include date-time-format?

Check this box if you want to include a specific custom date-time format in the filename

Include transform copy number?

Enable this option if you run this transform in multiple copies to not have multiple threads write to the same file. The copy number is formatted with mask 00

Split into parts and include number?

Enable this option if you want to split the output into multiple parts. Specify a split size larger than 0 and this is then the number of rows per file. The file part (split) number will be included in the filename to make sure that the same file is not being overwritten. The split number is formatted with mask 0000

Create parent folders?

Create missing parent folders of the output path.

Include compression codec before extension?

When enabled (the default for new transforms), the compression codec is placed before the file extension (for example file.snappy.parquet), which matches the naming used by Spark and other Parquet tools. When disabled, the compression codec is appended after the extension (for example file.parquet.snappy). Existing pipelines that do not have this option set keep the previous behavior.

Compression codec

Here you can indicate which compression codec you want to use. The default is UNCOMPRESSED.

Version

Choose the protocol version of Parquet (1.0 or 2.0)

Row group size (bytes)

The size of a row group in bytes, not rows. A row group is the unit of work for a reader: the transform buffers rows in memory until the row group is full, then encodes, compresses and writes it. The default is 268435456 (256 MB); Parquet itself defaults to 128 MB, and the Parquet project recommends 512 MB to 1 GB for large files. See Row group size and the file footer below before lowering this value.

Data page size (bytes)

The size of a data page in bytes, the unit of encoding and compression inside a column chunk. The default is 8192 (8 kB), which the Parquet project recommends for good read performance; Parquet itself defaults to 1048576 (1 MB).

Dictionary page size (bytes)

The maximum size of a dictionary page in bytes. When the dictionary of a column grows past this size the writer falls back to plain encoding for that column chunk. The default is 1048576 (1 MB).

Existing data

What to do when a partition folder already exists: append, overwrite that partition, fail if it exists, or overwrite the whole base folder. This option is enabled when at least one partition field is set.

Maximum open partitions

How many partition files may be written at the same time. Each open file buffers up to one row group in memory, so a lower number uses less memory but writes more files. This option is enabled when at least one partition field is set.

Partition by

Incoming fields used as directory levels. Leave empty to write a single file set.

Fields

You can specify which fields to write and in which order. You can use the "Get Fields" button to populate the dialog. Leave the list empty to output all input fields with the types in Hop to Parquet types.

Source field

Incoming field to write.

Target field

Column name in the Parquet file. Leave empty to use the source field name.

Parquet type

Logical or physical type of the column. Leave empty to keep the type chosen from the Hop type. Get Fields proposes a type for each field: a Date is proposed as TimestampMillis, the same TIMESTAMP(MILLIS) written when the Parquet type is left empty, and a Timestamp is proposed as TimestampMicros. Select Date for a calendar date, for example so BigQuery loads the column as a DATE. Change the proposed type when you need a different one.

Precision

Total number of digits. Used only when the Parquet type is Decimal. Leave empty to use the length of the source field.

Scale

Digits after the decimal point. Used only when the Parquet type is Decimal. Leave empty to use the precision of the source field.

UTF8

STRING

Text of the value

Boolean

BOOLEAN

Boolean value

Int32

INT32

Whole number. A value outside the 32-bit range stops the transform.

Int64

INT64

Whole number

Float

FLOAT

32-bit floating point number

Double

DOUBLE

64-bit floating point number

Binary

BINARY

Bytes of the value

Date

DATE

Calendar date in the JVM time zone. The time of day is dropped. BigQuery and other warehouses load this logical type as a DATE. Parquet Input reads it back as midnight of that date in the same time zone.

TimeMillis

TIME(MILLIS), not adjusted to UTC

Time of day in the JVM time zone, with millisecond precision

TimeMicros

TIME(MICROS), not adjusted to UTC

Time of day in the JVM time zone, with microsecond precision

TimestampMillis

TIMESTAMP(MILLIS), adjusted to UTC

Instant, with millisecond precision

TimestampMicros

TIMESTAMP(MICROS), adjusted to UTC

Instant, with microsecond precision. The nanoseconds beyond the microsecond are dropped.

Decimal

DECIMAL(precision, scale)

Exact decimal, rounded with the rounding type of the field. Precision is 1 to 38 and scale is 0 to the precision, taken from the Precision and Scale columns or, when those are empty, from the length and precision of the source field. A value with more digits before the decimal point than the column holds stops the transform.

JSON

JSON

JSON text

UUID

UUID

The 16 bytes of the UUID. The value is the text form, for example 00112233-4455-6677-8899-aabbccddeeff.