Execute a process
DescriptionThe Execute A Process transform executes a shell script (on the host that runs the pipeline). The transform is similar to the Shell workflow action, but can be used in a pipeline to execute for every row. |
Options
| Option | Description |
|---|---|
Transform name | Name of the transform; this name has to be unique in a single pipeline |
Process field | The field name in the data stream that defines the process to start (shell script / batch file to start). Arguments can also be used, see Command line parsing. |
Arguments in fields | Take the arguments from separate fields in the row instead of parsing them out of the process field. The process field then contains only the executable, see Command line parsing. |
Argument fields | The fields holding the arguments, in the order they should be passed to the process. Only available when Arguments in fields is enabled. |
Fail if not success | Checking this option means if the exit status is different than zero the transform fails. You can use error handling to get these rows. |
Output line delimiter | Without defining a line delimiter, all returned lines are collapsed into a single string with no line delimiters. You can set to any line delimiter and special characters can be set with the format $[value], e.g. $[0D] for CR or $[0D,0A] for CR/LF. |
Result fieldname | Specify here the name of the result fieldname (STRING) added to the output stream of the pipeline. This field is populated by the output stream (stdout) of the process. |
Error fieldname | Specify here the name of the error fieldname (STRING) added to the output stream of the pipeline. This field is filled by the error stream (stderr) of the process. |
Exit value | Specify here the name of the exit fieldname (INTEGER) added to the output stream of the pipeline. This field is filled by the exit output of the process. |
Command line parsing
The process is started directly, not through a shell. This is the single most important thing to know about this transform: no shell interprets what you type, so none of the following work, no matter how the command line is written:
-
pipes and redirection:
|,>,>>,< -
command chaining:
&&,||,; -
wildcard (glob) expansion:
*.csv,?,[a-z] -
environment variable expansion by the shell:
$HOME,%PATH% -
shell built-ins:
cd,export,set,echoon Windows
These are passed to the process as literal arguments. Hop variables such as {openvar}PROJECT_HOME{closevar} are resolved, because Hop resolves them before the process is started.
If you need shell features, run a shell explicitly and pass your command line to it as an argument, for example /bin/sh with arguments -c and ls -la /tmp | wc -l (see Arguments in fields below), or use the Shell workflow action which is built for scripts.
Splitting the command line
When Arguments in fields is disabled, the value of the process field is split into the executable and its arguments:
-
tokens are separated by whitespace
-
a section wrapped in single or double quotes stays a single token, even when it contains whitespace, and the quotes are removed
-
quoting is positional, so quoted and unquoted parts that touch each other end up in the same argument:
--path="/my folder"/xis one argument,--path=/my folder/x -
there is no backslash escaping: on Windows a backslash is a path separator, so
C:\my dirkeeps its backslashes and needs quotes to survive the space -
quotes only take effect when they are balanced. A command line with an unclosed quote is split on whitespace only and keeps its quote characters, so a stray apostrophe such as
--message=don'tstill reaches the process unchanged
| Process field value | Executable | Arguments |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Quote handling was added in Hop 2.19. Up to 2.18 the command line was always split on whitespace only, so ls -la "/work/hop/config" failed with a "does not exist" error on the quoted path. If you have a pipeline that relies on quote characters reaching the process as-is - a JSON argument such as --json={"a":1} for example - switch it to Arguments in fields, where nothing is stripped. |
Arguments in fields
Enable Arguments in fields when you would rather not rely on quoting at all, or when an argument contains characters that are awkward to quote. The process field then holds only the executable and each Argument fields entry is passed as one argument exactly as it is, with no splitting and no quote removal. This is the most reliable option for values that come from your data, such as file paths built at runtime.