Local Pipeline Engine
Local
The local runner runs on the local Hop engine. This is the runtime configuration you’ll use when testing pipelines on your local machine while developing.
Options
| Option | Description | Default |
|---|---|---|
Row set size |
The row set buffer size. |
10.000 |
Safe mode |
Checks every row passed through your pipeline and ensure all layouts are identical. If a row does not have the same layout as the first row, an error is generated and reported. |
false |
Collect metrics |
Collects metrics to monitor the performance of your pipeline. |
false |
Sort transforms |
Performs a topological sort of the transforms in the pipeline before execution. |
false |
Log rows feedback |
Enables log rows feedback, showing a log line every after processing multiple of 50.000 (default) rows. |
false |
Feedback size in rows |
The number of rows to return as feedback. |
50.000 |
Wait time buffer check (ms) |
This indicates the polling frequency when there are no rows in the input buffer of a transform, a lower value will cause higher CPU load when the pipeline has many idle transforms. |
20 |
Sample type while running in the GUI |
Here you can specify which rows you want to see when you click on the little grid icons on the transform icons during a pipeline execution.
You can choose: |
Last |
Number of rows to sample in the GUI |
The number of rows which will be collected |
100 |
Make this pipeline transactional |
If you enable this there will always be just one connection used per database. At the end of the pipeline there will be a commit or rollback (in case of error) on all named connections at the same time. Note that child pipelines or workflows are also automatically run as a transaction. Also be aware that while you can commit and rollback across multiple databases that this still means that you can have a successful commit on one database and a failure on another. This is not a two-phase-commit system. |
false |
Lock-free row sets (experimental) |
An experimental performance option. Rows are passed between transforms through lock-free buffers instead of lock-based ones. The transform that reads a buffer is woken once a batch of rows is ready, instead of for every row. A transform with several inputs waits on all of them at once and is woken by whichever input has rows. A transform that has had no rows for a while waits longer between checks, up to 20 ms, and is then woken by the next row, so idle transforms in long-running pipelines use little CPU. This lowers the cost per row, which matters most in pipelines of fast transforms. A buffer holds at most the configured Row set size, as the lock-based buffers do, and the row counts of the transforms are exact. With this option the Wait time buffer check (ms) setting is not used: transforms no longer poll their inputs. Hops that spill to disk to avoid a buffer deadlock keep their spilling buffer. |
false |