Parquet file output

Transform name

Name of the transform this name has to be unique in a single pipeline.

Base file name

Specify the base filename. This is composed of where you want to write the Parquet file to as well as the start of the filename. Examples:

Write to Amazon AWS S3 : s3://my-bucket-name/transactions

Write to a local folder : /my/folder/customer-data

Extension

This is the extension of the file. Usually this is simply parquet

Include date?

Check this box if you want to include the date in the filename with mask yyyMMdd

Include time?

Check this box if you want to include the time in the filename with mask HHmmss

Include date-time-format?

Check this box if you want to include a specific custom date-time format in the filename

Include transform copy number?

Enable this option if you run this transform in multiple copies to not have multiple threads write to the same file. The copy number is formatted with mask 00

Split into parts and include number?

Enable this option if you want to split the output into multiple parts. Specify a split size larger than 0 and this is then the number of rows per file. The file part (split) number will be included in the filename to make sure that the same file is not being overwritten. The split number is formatted with mask 0000

Create parent folders?

Create missing parent folders of the output path.

Include compression codec before extension?

When enabled (the default for new transforms), the compression codec is placed before the file extension (for example file.snappy.parquet), which matches the naming used by Spark and other Parquet tools. When disabled, the compression codec is appended after the extension (for example file.parquet.snappy). Existing pipelines that do not have this option set keep the previous behavior.

Compression codec

Here you can indicate which compression codec you want to use. The default is UNCOMPRESSED.

Version

Choose the protocol version of Parquet (1.0 or 2.0)

Row group size (bytes)

The size of a row group in bytes, not rows. A row group is the unit of work for a reader: the transform buffers rows in memory until the row group is full, then encodes, compresses and writes it. The default is 268435456 (256 MB); Parquet itself defaults to 128 MB, and the Parquet project recommends 512 MB to 1 GB for large files. See Row group size and the file footer below before lowering this value.

Data page size (bytes)

The size of a data page in bytes, the unit of encoding and compression inside a column chunk. The default is 8192 (8 kB), which the Parquet project recommends for good read performance; Parquet itself defaults to 1048576 (1 MB).

Dictionary page size (bytes)

The maximum size of a dictionary page in bytes. When the dictionary of a column grows past this size the writer falls back to plain encoding for that column chunk. The default is 1048576 (1 MB).

Existing data

What to do when a partition folder already exists: append, overwrite that partition, fail if it exists, or overwrite the whole base folder. This option is enabled when at least one partition field is set.

Maximum open partitions

How many partition files may be written at the same time. Each open file buffers up to one row group in memory, so a lower number uses less memory but writes more files. This option is enabled when at least one partition field is set.

Partition by

Incoming fields used as directory levels. Leave empty to write a single file set.

Fields

You can specify which fields to write and in which order. You can use the "Get Fields" button to populate the dialog. Leave empty to output all input fields.