Token Replacement transform Icon Token Replacement

Description

The Token Replacement transform replaces tokens in an input string or file.

The transform can then output this data either to a file or a field on the stream.

A token contains a start string, a name, and an end string.

For example ${my_token} could be a token.

The start string, and end string are configurable and can be any series of characters.

When replacing tokens in a file, this transform reads the entire file. It does not operate on a single line at a time. When replacing tokens in a file it is a best practice to output to a file also to prevent Hop from having to read the entire file into memory.

Supported Engines

Hop Engine

Supported

Spark

Maybe Supported

Flink

Maybe Supported

Dataflow

Maybe Supported

Options

Input Tab

  • Input Type - Where to read the field to do the token replacement from. Either text, field, or file.

  • Input Type Text

    • Input Text - The text to token replace

  • Input Type Field

    • Input Field - The input field to token replace

  • Input Type File

    • Input filename - The name of the file to token replace

    • Filename is in field?

      • Is the name of the file to token replace in a field?

    • Input filename field - The field the name of the file is in.

    • Add Input filename to result?

      • Add the input filename(s) to the result files list.

Output Tab

  • Output Type - Where to put the token replaced string. Either field or file.

  • Output Type Field

    • Output field name - The name of the field to put the token replaced string in.

  • Output Type File

    • Output filename - The name of the file to write to.

    • Filename is in field?

      • Is the name of the output file in a field?

    • Output filename field - The name of the field the output filename is in.

    • Append output file?

      • If the output file already exists should it be appended to. If not checked, Pentaho will overwrite the file if it exists.

    • Create parent folder?

      • Should Pentaho create the parent folder?

    • Output format - The format of the new line delimiter for the output file.

    • Output encoding - The character encoding to use when writing the file.

    • Split every - Split the output file into a new file every n rows.

    • Include stepnr in filename?

      • Should the step number be included in the output filename?

    • Include partition nr in filename?

      • Should the partition number be included in the output filename?

    • Include date in filename?

      • Should the curren date be included in the output filename?

    • Include time in filename?

      • Should the current time be included in the output filename?

    • Specify date format?

      • Do you want to specify the date format to include in the output filename?

    • Date time format - The date/time format to include in the output filename.

    • Add output filenames to result?

      • Add the output filename(s) to the result files list.

Tokens Tab

  • Token start string - The string that indicates the start of a token.

  • Token end string - The string that indicates the end of a token. Everything between the token start string and the token end string is the token name.

  • Stream name - The name of the field on the stream containing the value to replace the token with.

  • Token name - The name of the token to replace.

  • Get Fields button - Gets the list of input fields, and tries to map them to an Avro field by an exact name match.