Functionality

We can create a new variable resolver metadata element of type Pipeline Variable Resolver. The options used are fairly simple:

  • The pipeline filename to use: The file name of the pipeline to run to resolve a variable expression.

  • Pipeline Run configuration name: this has to reference a run configuration of type: Native Local.

  • The name of the variable in the pipeline that will contain the expression to resolve

  • The name of the output field to use to pick up the value of the fields.

This setup gives you a lot of flexibility. Maybe you have a configuration file per database, per system, …​ In that case you can set the variable to contain the type or name of the database, system, …​ to read a configuration file for. You can retrieve multiple values from these alternative configuration stores and send them to the output transform in a single row.

Only the first output row in the pipeline is considered.

The variable expression is (as always) in the following format:

#{name:variable:field}

  • name: the name of the variable resolver metadata element to use

  • variable: the variable value to set in the variable specified in the metadata (see above)

  • field: the name of the field for which we want to retrieve a string in the output transform.

Example

Suppose we have a number of databases we have configuration files for in separate properties files:

  • db1.properties

  • db2.properties

  • db3.properties …​

These all contain the same 5 keys, with different values of-course:

username=user1
password=pwd1
hostname=hostname1
port=port1
db=db1

We can then have a pipeline that reads and processes the appropriate file per type:

pipeline variable resolver pipeline

The Variable Resolver metadata would look like this:

pipeline variable resolver conf

Here are some examples of expressions and the results they give back:

  • #{conf:db1:username} : user1

  • #{conf:db2:db} : db2

  • #{conf:db3:port} : port3

This gives you a lot of flexibility.

In the case that you don’t specify any field to retrieve, the whole row will be encoded as a JSON value and given back.