Keep the number of actions in your workflows and transforms in your pipelines within reason.

  • Larger pipelines or workflows become harder to debug and develop against.

  • For every transform you add to a pipeline you start at least one new thread at runtime. You could be slowing down significantly simply by having hundreds of threads for hundreds of transforms.

If you find that you need to split up a pipeline you can write intermediate data to a temporary file using the Serialize to file transform. The next pipeline in a workflow can then pick up the data again with the De-serialize from file transform. While obviously you can also use a database or use another file type to do the same, these transforms will perform the fastest.