Hop Server

Hop Server is a lightweight server to run workflows and pipelines with the Remote pipeline or Remote workflow run configurations.

Starting and Stopping Hop Server

General Use

Hop Server is available as a script in your Hop installation directory.

Run Hop Server without any parameters to display its usage options. On Windows, this is hop-server.bat, on Mac and Linux, run ./hop-server.sh.

Usage

Usage: <main class> [-k] [-gs] [-e=<environmentOption>] [-id=<id>]
                    [-j=<projectOption>] [-l=<level>] [-n=<serverName>]
                    [-p=<password>] [-ps=<pipelineName>] [-u=<username>]
                    [-ws=<workflowName>] [-s=<systemProperties>[,
                    <systemProperties>...]]... [<parameters>...]
      [<parameters>...]   One XML configuration file or a hostname and port
  -e, --environment=<environmentOption>
                          The name of the lifecycle environment to use
  -gs, --general-status
                          List the general status of the server
      -id=<id>            Specify the ID of the pipeline or workflow to query
  -j, --project=<projectOption>
                          The name of the project to use
  -l, --level=<level>     The debug level, one of NOTHING, ERROR, MINIMAL,
                            BASIC, DETAILED, DEBUG, ROWLEVEL
  -n, --server-name=<serverName>
                          The name of the server to start as defined in the
                            metadata.
  -p, --password=<password>
                          The server password.  Required for administrative
                            operations only, not for starting the server.
  -ps, --pipeline-status=<pipelineName>
                          List the status of the pipeline with this name (also
                            specify the -id option)
  -s, --system-properties=<systemProperties>[,<systemProperties>...]
                          A comma separated list of KEY=VALUE pairs
  -u, --userName=<username>
                          The server user name.  Required for administrative
                            operations only, not for starting the server.
  -ws, --workflow-status=<workflowName>
                          List the status of the workflow with this name (also
                            specify the -id option)

The available Hop Server options are:

Short	Extended	Description
-h	--help	This help text
-p	--password	The server password. Required for administrative operations only, not for starting the server.
-u	--userName	The server password. Required for administrative operations only, not for starting the server.
-s	--system-properties	Manually set system environment variables. Specify a comma separated list of KEY=VALUE pairs.
-e	--environment	The name of the project lifecycle environment to enable before startup. This is provided for by the 'projects' plugin.
-n	--server-name	Name of the server metadata object to start, has to be used in combination with -e to know which environment to load it from
-j	--project	The name of the project to enable before startup. This is provided for by the 'projects' plugin.
-gs	--general-status	List the general status of the server.
-ps	--pipeline-status	List the status of the pipeline with this name (also specify the -id option)
-ws	--workflow-status	List the status of the workflow with this name (also specify the -id option)
-id		Specify the ID of the pipeline or workflow to query

Short

Extended

Description

-h

--help

This help text

-p

--password

The server password. Required for administrative operations only, not for starting the server.

-u

--userName

The server password. Required for administrative operations only, not for starting the server.

-s

--system-properties

Manually set system environment variables. Specify a comma separated list of KEY=VALUE pairs.

-e

--environment

The name of the project lifecycle environment to enable before startup. This is provided for by the 'projects' plugin.

-n

--server-name

Name of the server metadata object to start, has to be used in combination with -e to know which environment to load it from

-j

--project

The name of the project to enable before startup. This is provided for by the 'projects' plugin.

-gs

--general-status

List the general status of the server.

-ps

--pipeline-status

List the status of the pipeline with this name (also specify the -id option)

-ws

--workflow-status

List the status of the workflow with this name (also specify the -id option)

-id

Specify the ID of the pipeline or workflow to query

Start Hop Server with command line parameters

Hop Server can be started with hostname or ip address and port number as unnamed arguments:

Example: ./hop-server.sh 0.0.0.0 8080

Example: hop-server.bat 192.168.1.221 8081

Example: ./hop-server.sh -e aura-gcp gs://apachehop/hop-server-config.xml

Example: hop-server.bat 127.0.0.1 8080 --userName cluster --password cluster

hop-server <Interface address> <Port> [ShutdownPort] [-h] [-p <arg>] [-s] [-u <arg>]

Example startup commands are:

Windows
Linux, macOS

hop-server.bat 127.0.0.1 8080

hop-server.bat 192.168.1.221 8081

 ./hop-server.sh 127.0.0.1 8080

 ./hop-server.sh 192.168.1.221 8081

Listen to all interfaces on the server:

 ./hop-server.sh 0.0.0.0 8080--

Start Hop Server with a configuration file

Specify the xml configuration file as the only argument:

hop-server <Configuration File>

The syntax of this configuration file is fairly simple:

<hop-server-config>

  <hop-server>
    <name>server-8181</name>
    <hostname>localhost</hostname>
    <port>8181</port>
    <shutdownPort>8182</shutdownPort>
  </hop-server>

  <!-- Join the web server thread and wait until it's finished.
       The default is true
  -->
  <joining>true</joining>

  <!-- The maximum number of log lines kept in memory by the server.
       The default is 0 which means: keep all lines
   -->
  <max_log_lines>0</max_log_lines>

  <!-- The time (in minutes) it takes for a log line to be cleaned up in memory.
       The default is 0 which means: never clean up log lines
  -->
  <max_log_timeout_minutes>1440</max_log_timeout_minutes>

  <!-- The time (in minutes) it takes for a pipeline or workflow execution to be removed from the server status.
       The default is 0 which means: never clean executions
  -->
  <object_timeout_minutes>1440</object_timeout_minutes>

  <!-- The folder to read metadata objects from so that web services and database connections for sequences can be found.
       The default is that no metadata is configured: remotely executed pipelines and workflows will have their own metadata.
  -->
  <metadata_folder></metadata_folder>

</hop-server-config>

Example startup commands with a configuration file are:

Windows
Linux, macOS

hop-server.bat C:\<YOUR_PATH>\hop-server-config.xml

Or with a remote configuration file:

hop-server.bat http://www.example.com/hop-server-config.xml

You can also enable a project lifecyfle environment for the Hop server:

hop-server.bat -e graph-aws hop-server.xml

 ./hop-server.sh /foo/bar/hop-server-config.xml

Or with a remote configuration file:

 ./hop-server.sh http://www.example.com/hop-server-config.xml

You can also enable a project lifecyfle environment for the Hop server:

 ./hop-server.sh -e graph-aws hop-server.xml

In the sample above the environment contains configuration files with variables which are loaded. With the environment the server also knows the project home folder. The server configuration file is found in the home folder automatically with the implicit relative path.

SSL configuration

To protect traffic between a Hop Server and its clients (Hop Run, Hop GUI, a browser, Hop Server command line querying, …) it’s possible to use encrypt data using a secure socket layer (SSL) connection on top of the regular web traffic with a hyper treading transfer protocol (HTTP). The combination of both is called HTTPS. To run a Hop server using this https:// protocol you can add an sslConfig section in the hop-server-config/hop-server path.

The 3 main options are:

keyStore : the path to the java keystore file, created with keytool
keyStorePassword : the password to the keystore file
keyPassword : the key password. If this is the same as the keystore password you can omit this option.

The HTTP protocol used is version 1.1 or HTTP/1.1. The type of keystore read is a Java Keystore or type: JKS. Let’s take a look at how we can generate a sample keystore:

# Generate a new key
#
openssl genrsa -des3 -out hop.key

# Make a new certificate
#
openssl req -new -x509 -key hop.key -out hop.crt

# Create a PKCS12 keystore and import it into a JKS keystore
# The resulting file is: keystore
#
keytool -keystore keystore -import -alias hop -file hop.crt -trustcacerts
openssl req -new -key hop.key -out hop.csr
openssl pkcs12 -inkey hop.key -in hop.crt -export -out hop.pkcs12
keytool -importkeystore -srckeystore hop.pkcs12 -srcstoretype PKCS12 -destkeystore keystore

Here is an example of the information to include in your server XML:

<hop-server-config>
<hop-server>
...

    <sslConfig>
      <keyStore>/path/to/keystore</keyStore>
      <keyStorePassword>password</keyStorePassword>
      <keyPassword>keyPassword</keyPassword>
    </sslConfig>

    <!-- Add the following line to support querying over https -->
    <sslMode>Y</sslMode>
  </hop-server>
  ...
</hop-server-config>

Enable detailed server logging

Hop Server provides the -l or --level option to set a logging level for workflows and pipelines that run on the server.

There are scenarios where you may want to see more detailed logging about the server itself. Since Hop Server runs on a Jetty server, you can increase the Jetty server logging by extending the HOP_OPTIONS variable near the end of hop-server.sh or hop-server.bat.

Original:

"$_HOP_JAVA" ${HOP_OPTIONS} -Djava.library.path=$LIBPATH -classpath "${CLASSPATH}" org.apache.hop.www.HopServer "$@"
EXITCODE=$?

With DEBUG logging:

"$_HOP_JAVA" ${HOP_OPTIONS} -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog -Dorg.eclipse.jetty.LEVEL=DEBUG -Djava.library.path=$LIBPATH -classpath "${CLASSPATH}" org.apache.hop.www.HopServer "$@"
EXITCODE=$?

If applied correctly, your Hop Server starts producing a lot of logging information similar to the lines below:

2022/07/15 14:18:00 - HopServer - Installing timer to purge stale objects after 1440 minutes.
2022-07-15 14:18:00.267:INFO::main: Logging initialized @3732ms to org.eclipse.jetty.util.log.StdErrLog
2022-07-15 14:18:00.276:DBUG:oejuc.ContainerLifeCycle:main: Server@3749c6ac{STOPPED}[9.4.35.v20201120] added {QueuedThreadPool[qtp1195781551]@47462daf{STOPPED,8<=0<=200,i=0,r=-1,q=0}[NO_TRY],AUTO}
2022-07-15 14:18:00.283:DBUG:oejuc.ContainerLifeCycle:main: ConstraintSecurityHandler@3f473daf{STOPPED} added {org.eclipse.jetty.util.component.DumpableCollection@390e814c,POJO}
2022-07-15 14:18:00.286:DBUG:oejuc.ContainerLifeCycle:main: HashLoginService@7bfedfb7[null] added {org.eclipse.jetty.security.DefaultIdentityService@6d3194ff,POJO}
2022-07-15 14:18:00.290:DBUG:oejuc.ContainerLifeCycle:main: HashLoginService@7bfedfb7[Hop] added {PropertyUserStore@213c812a[users.count=0,identityService=org.eclipse.jetty.security.DefaultIdentityService@25814d3c],AUTO}
2022-07-15 14:18:00.290:DBUG:oejuc.ContainerLifeCycle:main: ConstraintSecurityHandler@3f473daf{STOPPED} added {HashLoginService@7bfedfb7[Hop],AUTO}
2022-07-15 14:18:00.302:DBUG:oejsh.ContextHandlerCollection:main: ->[{o.e.j.s.ServletContextHandler@1a6df932{/,null,STOPPED},[o.e.j.s.ServletContextHandler@1a6df932{/,null,STOPPED}]}]
2022-07-15 14:18:00.303:DBUG:oejuc.ContainerLifeCycle:main: ContextHandlerCollection@74120029{STOPPED} added {o.e.j.s.ServletContextHandler@1a6df932{/,null,STOPPED},AUTO}
2022-07-15 14:18:00.304:DBUG:oeju.DecoratedObjectFactory:main: Adding Decorator: org.eclipse.jetty.util.DeprecationWarning@48cbb4c5

Start Hop Server with docker

It’s often very convenient to run a Hop docker container since it has all the required software automatically delivered. For a complete description of the standard Hop docker container see the full reference in the technical documentation. Here is an example of how you would start a "long-lived" docker container:

docker run \
  -p 8080:8080 \
  -p 8079:8079 \
  -e HOP_SERVER_PORT=8080 \
  -e HOP_SERVER_SHUTDOWNPORT=8079 \
  -e HOP_SERVER_USER=username \
  -e HOP_SERVER_PASS=password \
  apache/hop

Stopping Hop Server

In a testing setup where Hop Server was started from a terminal, the process can be terminated through CTRL-C.

In headless environments, the same hop-server command used to start the server can be used to stop it: A default listener on shutdown port is configured on port number 8079 when not specifying this in the command it will be used.

Windows
Linux, macOS

hop-server.bat 127.0.0.1 8080 -k -u cluster -p cluster

 ./hop-server.sh 127.0.0.1 8080 -k -u cluster -p cluster

You can also trigger the command on a specific shutdown port

Windows
Linux, macOS

hop-server.bat 127.0.0.1 8080 8079 -k -u cluster -p cluster

 ./hop-server.sh 127.0.0.1 8080 8079 -k -u cluster -p cluster

Verify startup

Starting a Hop Server on the local machine e.g. on port 8081 will only take 1 or 2 seconds.

The console output will look similar to what is listed below:

2020/06/20 18:35:12 - HopServer - Installing timer to purge stale objects after 1440 minutes.
2020/06/20 18:35:12 - HopServer - Created listener for webserver @ address : localhost:8081

Query a server from the command line

You can query the new server with another hop-server command:

Windows
Linux, macOS

hop-server.bat -gs -u cluster -p cluster 127.0.0.1 8080

Expected output:

C:\<YOUR_PATH>\hop>echo off                                                          ===[Environment Settings - hop-server.bat]====================================
Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows -DHOP_PLATFORM_RUNTIME=GUI
-DHOP_AUTO_CREATE_CONFIG=Y
Command to start Hop will be:
"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath lib\core\*;lib\beam\*;lib\swt\win64\*
-Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=GUI -DHOP_AUTO_CREATE_CONFIG=Y org.apache.hop.www.HopServer  -gs -u cluster -p cluster localhost 8181
===[Starting HopServer]=========================================================
2022/12/16 14:02:13 - HopServer - Enabling project 'default'
Pipelines: 0 found.

Workflows: 0 found.

./hop-server.sh -gs -u cluster -p cluster 127.0.0.1 8080

Expected output:

Pipelines: 0 found.

Workflows: 0 found.

Query a pipeline from the command line

sh hop-server.sh -id 375c9113-b538-4559-8e98-ee02a435fbb9 -u cluster -p cluster -ps service-example -j my-project hop-server.xml
2021/10/01 13:27:04 - HopServer - Enabling project 'my-project'
  ID: 375c9113-b538-4559-8e98-ee02a435fbb9
      Name:     service-example
      Status:   Finished
      Start:    2021/10/01 13:26:45.128
      End:      2021/10/01 13:26:45.220
      Log date: 2021/10/01 13:27:04.363
      Errors:   0
      Transforms: 4 found.
        1
          Name:      a,b
          Copy:      0
          Status:    Finished
          Input:     0
          Output:    0
          Read:      1
          Written:   1
          Rejected:  0
          Updated:   0
          Errors:    0
        2
...
        3
...
        4
...
      Logging:
          2021/10/01 13:26:45 - service-example - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
          2021/10/01 13:26:45 - service-example - Execution started for pipeline [service-example]
          2021/10/01 13:26:45 - a,b.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - c,d.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - build JSON.0 - Finished processing (I=0, O=1, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - OUTPUT.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - service-example - Pipeline duration : 0.092 seconds [  0.092" ]
          2021/10/01 13:26:45 - service-example - Execution finished on a local pipeline engine with run configuration 'local'

Query a workflow from the command line

sh hop-server.sh -ws test-workflow -id e24b4549-edf0-4d77-987e-f103b630b4cc -u cluster -p cluster localhost 8181
  ID: e24b4549-edf0-4d77-987e-f103b630b4cc
      Name:     test-workflow
      Status:   Finished
      Log date: 2021/10/01 14:27:45.891
      Result:   true
      Errors:   0
      Logging:
          2021/10/01 14:27:45 - test-workflow - Start of workflow execution
          2021/10/01 14:27:46 - test-workflow - Starting action [sample]
          2021/10/01 14:27:46 - sample - Using run configuration [remote-8181]
          2021/10/01 14:27:46 - sample - Executing this pipeline using the Remote Pipeline Engine with run configuration 'remote-8181'
          2021/10/01 14:27:46 - sample - 2021/10/01 14:27:46 - sample - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
          2021/10/01 14:27:46 - sample - 2021/10/01 14:27:46 - sample - Execution started for pipeline [sample]
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - 1M.0 - Finished processing (I=0, O=0, R=0, W=1000000, U=0, E=0)
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - someString,someInt.0 - Finished processing (I=0, O=0, R=1000000, W=1000000, U=0, E=0)
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - id.0 - Finished processing (I=0, O=0, R=1000000, W=1000000, U=0, E=0)
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - sample - Pipeline duration : 0.977 seconds [  0.977" ]
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - sample - Execution finished on a local pipeline engine with run configuration 'local'
          2021/10/01 14:27:47 - sample - Execution finished on a remote pipeline engine with run configuration 'remote-8181'
          2021/10/01 14:27:48 - test-workflow - Starting action [true]
          2021/10/01 14:27:48 - test-workflow - Starting action [false]
          2021/10/01 14:27:48 - test-workflow - Starting action [log-something]
          2021/10/01 14:27:48 - Subject - Message
          2021/10/01 14:27:48 - test-workflow - Starting action [Success]
          2021/10/01 14:27:48 - test-workflow - Finished action [Success] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [log-something] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [false] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [true] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [sample] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Workflow execution finished
          2021/10/01 14:27:48 - test-workflow - Workflow duration : 2.715 seconds [  2.714" ]

Connect to the Hop Server UI

To connect to the previously started server, point your browser to http://localhost:8081.

You’ll be prompted for your username and password. The default is cluster for both the username and password. The defaults obviously should be changed in any environment that goes beyond a simple local developer setup.

on startup, the pipeline and workflow lists shown below will be empty. Run a workflow or pipeline through the Hop Remote pipeline engine run configuration or through the REST api. When pipelines or workflows are executed on the server, you’ll be able to follow the logging output either from the terminal or a log file (e.g. piped from the startup command).

For each of the options in the pipeline and workflow dialogs described below, select a pipeline and workflow from the list and select the desired option.

The header bar for workflows and pipelines is almost identical (from left to right).

Run
Stop the running pipeline/workflow
Cleanup pipeline	Cleanup a pipeline: close remote sockets etc
View pipeline/workflow details
Remove pipeline/workflow from list

Run

Stop the running pipeline/workflow

Cleanup pipeline

Cleanup a pipeline: close remote sockets etc

View pipeline/workflow details

Remove pipeline/workflow from list

Hop Server web Services

A Hop Server can also be accessed directly through a number of web services and in combination with the Web Service and Asynchronous Web Service metadata types.