The Hop toolbox

You’ll spend most of your time in Hop Gui, but a project that only runs from a desktop GUI is not much use in production.

The split is roughly this: designing pipelines and workflows and creating metadata is what Hop Gui is for. Configuring and running them is where the command line takes over, and that is the half a scheduler, a container or a CI/CD pipeline needs.

One command to start them all

Since Hop 2.x there is a single hop launcher in the installation folder. It replaces the individual hop-run, hop-conf, hop-gui, … scripts, and plugins can add their own commands to it.

Windows
Linux, macOS

shell

hop.bat help

shell

./hop help

help lists the commands available in your installation, which depends on the plugins you have installed:

shell

Commands:
  help     Display help information about the specified command.
  conf     Configure Hop
  encrypt  Encrypt secrets
  gui      The Hop GUI
  import   Import metadata
  run      Run a pipeline or workflow
  search   Search in Hop metadata
  server   Run a Hop server
  setup    Configure Hop environment variables

Two options are worth knowing before anything else, because nearly every command takes them:

Option What it does

Option	What it does
`-j`, `--project`	enable a project before the command runs
`-e`, `--environment`	enable an environment before the command runs

-j, --project

enable a project before the command runs

-e, --environment

enable an environment before the command runs

So running your workflow from the previous chapter looks like this:

shell

./hop run -j getting-started -r local -f '${PROJECT_HOME}/first-workflow.hwf'

the older hop-run.sh, hop-conf.sh and friends still ship and still work. New examples in this manual use hop <command>.

The tools

Tool What you use it for Read more

Tool	What you use it for	Read more
`hop gui`	The visual development environment: build, preview, run and debug pipelines and workflows.	Hop Gui
`hop run`	Run a pipeline or workflow. This is what a scheduler, a CI job or a container calls.	Hop Run
`hop conf`	Manage projects, environments and cloud configuration from a script.	Hop Conf
`hop setup`	Point Hop at a user-level config folder so your projects and settings survive an upgrade.	Hop Setup
`hop server`	Run a Hop Server to execute pipelines and workflows remotely, over REST.	Hop Server
`hop search`	Search across all the metadata in a project: which pipeline uses this connection, where is this field set.	Hop Search
`hop encrypt`	Obfuscate a password so it isn’t stored in clear text in a configuration file.	Hop Encrypt
`hop import`	Convert a Kettle/PDI repository into a Hop project.	Import Kettle projects
`hop doc`	Generate documentation for your own Hop project, in Markdown or HTML.	Hop Doc

hop gui

The visual development environment: build, preview, run and debug pipelines and workflows.

Hop Gui

hop run

Run a pipeline or workflow. This is what a scheduler, a CI job or a container calls.

Hop Run

hop conf

Manage projects, environments and cloud configuration from a script.

Hop Conf

hop setup

Point Hop at a user-level config folder so your projects and settings survive an upgrade.

Hop Setup

hop server

Run a Hop Server to execute pipelines and workflows remotely, over REST.

Hop Server

hop search

Search across all the metadata in a project: which pipeline uses this connection, where is this field set.

Hop Search

hop encrypt

Obfuscate a password so it isn’t stored in clear text in a configuration file.

Hop Encrypt

hop import

Convert a Kettle/PDI repository into a Hop project.

Import Kettle projects

hop doc

Generate documentation for your own Hop project, in Markdown or HTML.

Hop Doc

Some tools are optional plugins rather than part of the standard client:

Tool What you use it for Read more

Tool	What you use it for	Read more
Marketplace	Install optional plugins into a running Hop client from a Maven repository.	Hop Marketplace
`hop lint`	Static analysis over a project: check pipelines, workflows and metadata without running them. Useful as a build step.	Hop Lint
PyHop	Drive Hop from Python.	Hop Python
Hop Arrow	Expose pipeline output over an Apache Arrow Flight server.	Hop Arrow

Marketplace

Install optional plugins into a running Hop client from a Maven repository.

Hop Marketplace

hop lint

Static analysis over a project: check pipelines, workflows and metadata without running them. Useful as a build step.

Hop Lint

PyHop

Drive Hop from Python.

Hop Python

Hop Arrow

Expose pipeline output over an Apache Arrow Flight server.

Hop Arrow

Hop Gui also runs in a browser. See Hop Web and Hop Web in Docker.

Try one

The quickest way to see the command line in action is to run something you already built:

shell

./hop run -j getting-started -r local -f '${PROJECT_HOME}/first-pipeline.hpl'

You get the same log you saw in the Logging tab of Hop Gui, on stdout. That is the whole deployment story for a lot of Hop projects: the same file, the same run configuration, started by cron, a Kubernetes CronJob or Airflow instead of by you.

See Scheduling workflows and pipelines for the details.