Docker container

This is the documentation of the official Apache Hop docker container published on:

https://hub.docker.com/r/apache/hop

It’s a Hop Docker image supporting both short-lived and long-lived setups. A short-lived setup executes a pipeline or workflow and stops right after. A long-lived setup starts a Hop server and waits for work.

This page describes apache/hop (hop-run and Hop Server). For the Hop Web GUI image (apache/hop-web), including which folders to persist across upgrades, see Hop Web in Docker: persistence and upgrades.

The docker container runs a minimal Linux system called Alpine. OpenJDK version 21 is then used to execute Apache Hop. The Linux user used to execute in the container is hop and the group is hop as well.

You can provide values for the following environment variables:

Here are the variables for short-lived containers, to execute a workflow or pipeline:

Below are the variables you can use for a long-lived container, running Hop Server:

Make sure to get the latest updates for the Hop image by pulling them:

If you do not specify a value for :<tag> the value latest will be taken. Latest will contain the last officially released version of Apache Hop. You can also specify Development as a tag. That image will contain the last built Development snapshot of Apache Hop. rxq7777

Apache Hop ships the JDBC drivers it is allowed to redistribute (PostgreSQL, MS SQL Server, Snowflake, …​). Drivers with a restricted license - Oracle, MySQL, MariaDB, IBM DB2 and others - are not bundled and have to be added to the lib/jdbc folder yourself.

The container can do this for you on start-up. Set HOP_DRIVERS_DOWNLOAD to a comma separated list of driver ids and the entrypoint runs hop driver install for each one before Hop is launched, so the drivers are picked up by the normal lib/jdbc scan. Each entry is a driver id, optionally followed by :version. Run hop driver list (inside the container or a local Hop install) to see the available ids.

Because the restricted drivers are downloaded from the vendor / Maven Central under the vendor’s license, you must explicitly accept that license by setting HOP_DRIVERS_ACCEPT_LICENSE=true. Without it, restricted drivers are skipped with an error. Apache Hop neither hosts nor redistributes these drivers; obtaining them this way is your action under the vendor’s terms.

For air-gapped environments, point the download at an internal Maven mirror with HOP_DRIVERS_MAVEN_REPO:

The download is supported by the client/server image (running pipelines, workflows or hop-server) and the Hop web image. The drivers and their transitive dependencies are written to HOP_SHARED_JDBC_FOLDERS (default lib/jdbc). If the container filesystem is read-only, make sure that folder is writable, for example by mounting a volume over it.

The Docker container can install optional or third-party plugins from the Hop marketplace or a corporate Maven repository (e.g. Sonatype Nexus, JFrog Artifactory) on startup before Hop starts. The stock image does this at runtime, so a custom image is not required. A repeated start skips a plugin when its receipt version matches and its files are still present. Every coordinate is installed by one hop marketplace install invocation.

Set HOP_PLUGINS_DOWNLOAD to a comma-separated list of plugin coordinates. Each entry can be:

HOP_PLUGINS_MAVEN_REPO is the repository to try first. The configured Apache and Maven Central repositories are still used when a plugin is not in that repository, so a private plugin and an Apache optional plugin can be requested together. Pass a full groupId:artifactId:version for the private one when that repository cannot be browsed.

A repository that cannot be browsed (a plain Maven repository, or HOP_PLUGINS_REPO_TYPE=maven) does not accept a short name. The install fails and asks for groupId:artifactId:version instead of guessing org.apache.hop. HOP_PLUGINS_REPO_TYPE selects the browse API: auto (default), nexus, forgejo, jfrog, or maven. auto infers Nexus, Artifactory, or Forgejo from the URL.

HOP_PLUGINS_REPO_ID is the id of that repository (default corporate-repo). The entrypoint does not put the password on the hop command line. It exports HOP_MARKETPLACE_<ID>_USERNAME and either HOP_MARKETPLACE_<ID>_PASSWORD or, when the auth type is token, HOP_MARKETPLACE_<ID>_TOKEN. For the default id those names are HOP_MARKETPLACE_CORPORATE_REPO_USERNAME, HOP_MARKETPLACE_CORPORATE_REPO_PASSWORD and HOP_MARKETPLACE_CORPORATE_REPO_TOKEN. Any other id is uppercased, and every character that is not a letter or digit becomes an underscore. You can set those variables yourself instead of HOP_PLUGINS_REPO_USERNAME and HOP_PLUGINS_REPO_PASSWORD.

The global HOP_MARKETPLACE_USERNAME and HOP_MARKETPLACE_PASSWORD variables are also picked up.

HOP_PLUGIN_BASE_FOLDERS replaces the default plugins folder. Include the image plugins path and the mounted path, as in the example. The install is written to the first writable directory in that list that is not the image install itself. Shared lib/core jars are written next to that plugins folder and added to the classpath of hop, hop-run and hop-server.

Plugins written only into the container filesystem are not kept when the container is removed. They are kept when that install directory is a mounted volume, or when the install happened while building an image.

The Hop web image installs marketplace plugins at startup as well. The image rewrites the hop classpath the same way as hop-run.sh, so it includes WEB-INF/lib (where the Hop jars live) as well as lib/core. Shared jars from a plugin are installed into WEB-INF/lib so Tomcat and hop both load them. Plugin folders still follow HOP_PLUGIN_BASE_FOLDERS.

Building a derived image is optional. It avoids the download on every start when a custom image is acceptable:

If you maintain a declarative Hop environment install spec (hop-env.yaml) or repository definition (hop-marketplace-repo.yaml), you can point to it directly using HOP_PLUGINS_ENV_FILE:

The cloud file systems in Hop (s3://, gs://, azure://, googledrive://, dropbox://) get their default credentials from one of two places:

The sample file docker/hop.env.sample lists the options below with comments. Copy it, uncomment what you need and pass it with --env-file:

docker run --env-file takes everything after the first = literally, quotes included, so do not wrap values in quotes. A line with only a variable name, for example AWS_SECRET_ACCESS_KEY, copies that variable from the shell that runs docker run.

The sample is for docker run --env-file only. Do not use it as a Docker Compose env_file: Compose strips quotes and expands ${…​}, which breaks the quoted Azure key and HOP_FILE_PATH.

HOP_CONFIG_OPTIONS is a single variable. To configure more than one file system, put all options on one line, separated by spaces:

The Azure key may be encrypted with hop-encrypt.sh -hop <key>. The result contains a space, so quote it as shown above.

Google Drive needs a token that is created by an interactive browser login. Log in once with Hop GUI on a desktop, then mount the credentials file and the tokens folder into the container. The credentials file can stay on the read-only /keys mount. Hop writes to the tokens folder, so mount it read-write and make it owned by uid 501, the hop user in the image:

with --google-drive-credentials-file=/keys/google-drive-credentials.json --google-drive-tokens-folder=/home/hop/google-drive-tokens in HOP_CONFIG_OPTIONS.

To make the Hop Docker image even more flexible, we added a HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH variable that accepts a path to a custom shell script (that you provide).This shell script will run when you start the container before your Hop project is registered with the container’s Hop config and before your Hop workflow or pipeline gets kicked off. This feature might come in handy when you want to run some custom logic upfront, e.g. source Hop project files from S3 or clone them from GitHub.

The custom shell file can be provided in several ways (this is not a full list):

For the last scenario mentioned, it could be something like this:

We create a simple bash script called clone-git-repo.sh in a sub-folder called resources:

We also make it parameter-driven, so it any other team can use it.We create our custom Dockerfile like so:

Note that apart from defining the new environment variables (that go in line with the parameters we defined in the clone-git-repo.sh earlier on ), we also COPY the clone-git-repo.sh file to user hop’s home folder.

Next let’s build a small script which builds our custom image and then tests it by spinning up a container and running a workflow: