Hop Server Web Services Overview

Hop Server has a rich set of web services that can be used to query and manage the server’s status, or to manage pipeline and workflow execution details.

when specified, the id for a workflow or pipeline represents one execution of a workflow or pipeline on the server.

addExport

name

addExport

description

Upload a resources export file. Add a zipped pipeline or workflow to the body payload as a binary file.

endPoint

GET hop/addExport

parameters

  • type: pipeline or workflow

  • load: -

example request

http://localhost:8081/hop/addExport/?type=workflow with zipped workflow as payload

result

A zip file with the export is created on the server’s file system.

<?xml version="1.0" encoding="UTF-8"?>
<webresult>
    <result>OK</result>
    <message>file:///tmp/export_70eb8ef1-9721-4cf5-afa3-940cd0f771d9.zip</message>
    <id/>
</webresult>

addPipeline

name

addPipeline

description

Add a pipeline for execution

endPoint

GET hop/addPipeline

parameters

  • xml Request body should contain xml containing pipeline_configuration (pipeline and pipeline_execution_configuration wrapped in pipeline_configuration tag).

example request

http://localhost:8081/hop/addPipeline/xml=Y with XML payload

result

-

addWorkflow

name

addWorkflow

description

Add a workflow for execution

endPoint

GET hop/addWorkflow

parameters

  • xml Request body should contain xml containing workflow_configuration (pipeline and workflow_execution_configuration wrapped in pipeline_configuration tag).

example request

http://localhost:8081/hop/addWorkflow/xml=Y with XML payload

result

-

getPipelineImage

name

getPipelineImage

description

Generate a SVG image of a pipeline

endPoint

GET hop/pipelineImage

parameters

  • name: name of the pipeline to generate the image for

  • id: id of the pipeline to generate the image for

example request

GET http://localhost:8081/hop/pipelineImage/?name=remote-pipeline&id=c1451bfb-b867-4c76-b123-c29d2b05da17

result

an SVG image of the pipeline graph

getPipelineStatus

name

getPipelineStatus

description

Get the status of a pipeline

endPoint

GET hop/pipelineStatus

parameters

  • xml: use XML (default: Y)

  • json: use JSON (default: Y)

  • name: name of the pipeline to get the status for

  • id: id of the pipeline to get the status for

example request

GET http://localhost:8081/hop/pipelineStatus/?name=<NAME>>&id=<ID>

result

an HTML response with the execution status, transform details and canvas preview for this pipeline

Status

name

status

description

Get the status of the server

parameters

none

example request

GET http://localhost:8081/hop/status

result

an HTML page with an overview of the pipelines and workflows on the server, their execution details and the server’s configuration details.

getWorkflowImage

name

getWorkflowImage

description

Generate an SVG image of a workflow

endPoint

GET hop/workflowImage

parameters

  • name: name of the workflow to generate the image for

  • id: id of the workflow to generate the image for

example request

GET http://localhost:8081/hop/workflowImage/?name=<NAME>>&id=<ID>

result

an SVG image of the workflow graph

getWorkflowStatus

name

getWorkflowStatus

description

Get the status of a workflow

endPoint

GET hop/workflowStatus

parameters

  • xml: use XML (default: Y)

  • json: use JSON (default: Y)

  • name: name of the workflow to get the status for

  • id: id of the workflow to get the status for

example request

GET http://localhost:8081/hop/workflowStatus/?name=<NAME>&id=<ID>

result

an HTML response with the execution status, action details and canvas preview for this workflow

pausePipeline

name

pausePipeline

description

Pause or continue a pipeline

endPoint

GET /hop/pausePipeline

parameters

  • name: name of the pipeline to pause or restart

  • id: id of the pipeline to pause or restart

example request

GET http://localhost:8081/hop/pausePipeline/?name=<NAME>&id=<ID>

result

HTML page with the request status, e.g.

<HTML>

<HEAD>
	<TITLE>Pause pipeline</TITLE>
	<META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>&id=<ID>">
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H1>Pipeline [tmp] : pause requested.</H1>
	<a href="/hop/pipelineStatus?name=<NAME>&id=<ID>">Back to the pipeline status page</a>
	<p>
		<p>
</BODY>

</HTML>

Prepare Execution

name

prepareExec

description

Prepare the execution of a pipeline

endPoint

GET /hop/prepareExec

parameters

  • xml: use xml, default Y

  • name: the name of the pipeline to prepare execution for

  • id: the id of the pipeline to prepare execution for

example request

GET http://localhost:8081/hop/prepareExec/?xml=Y&name=<NAME>&id=<ID>

result

Example result:

<?xml version="1.0" encoding="UTF-8"?>
<webresult>
    <result>OK</result>
    <message/>
    <id/>
</webresult>

Register Pipeline

name

registerPipeline

description

Register a pipeline for execution

endPoint

GET hop/registerPipeline

parameters

  • xml Request body should contain xml containing pipeline_configuration (pipeline and pipeline_execution_configuration wrapped in pipeline_configuration tag).

example request

http://localhost:8081/hop/registerPipeline/xml=Y

with XML payload (example):

<pipeline_configuration>
<pipeline>
  <info>
    <name>generate_rows</name>
    <name_sync_with_filename>Y</name_sync_with_filename>
    <description/>
    <extended_description/>
    <pipeline_version/>
    <pipeline_type>Normal</pipeline_type>
    <parameters>
    </parameters>
    <capture_transform_performance>N</capture_transform_performance>
    <transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
    <transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
    <created_user>-</created_user>
    <created_date>2022/02/03 13:47:49.645</created_date>
    <modified_user>-</modified_user>
    <modified_date>2022/02/03 13:47:49.645</modified_date>
    <key_for_session_key>H4sIAAAAAAAA/wMAAAAAAAAAAAA=</key_for_session_key>
    <is_key_private>N</is_key_private>
  </info>
  <notepads>
  </notepads>
  <order>
    <hop>
      <from>Generate rows</from>
      <to>Delay row</to>
      <enabled>Y</enabled>
    </hop>
    <hop>
      <from>Delay row</from>
      <to>result</to>
      <enabled>Y</enabled>
    </hop>
  </order>
  <transform>
    <name>Delay row</name>
    <type>Delay</type>
    <description/>
    <distribute>Y</distribute>
    <custom_distribution/>
    <copies>1</copies>
    <partitioning>
      <method>none</method>
      <schema_name/>
    </partitioning>
    <scaletime>seconds</scaletime>
    <timeout>1</timeout>
    <attributes/>
    <GUI>
      <xloc>416</xloc>
      <yloc>96</yloc>
    </GUI>
  </transform>
  <transform>
    <name>Generate rows</name>
    <type>RowGenerator</type>
    <description/>
    <distribute>Y</distribute>
    <custom_distribution/>
    <copies>1</copies>
    <partitioning>
      <method>none</method>
      <schema_name/>
    </partitioning>
    <fields>
      <field>
        <length>-1</length>
        <name>value</name>
        <precision>-1</precision>
        <set_empty_string>N</set_empty_string>
        <type>String</type>
        <nullif>test</nullif>
      </field>
    </fields>
    <interval_in_ms>5000</interval_in_ms>
    <last_time_field>FiveSecondsAgo</last_time_field>
    <never_ending>N</never_ending>
    <limit>10000</limit>
    <row_time_field>now</row_time_field>
    <attributes/>
    <GUI>
      <xloc>160</xloc>
      <yloc>96</yloc>
    </GUI>
  </transform>
  <transform>
    <name>result</name>
    <type>Dummy</type>
    <description/>
    <distribute>Y</distribute>
    <custom_distribution/>
    <copies>1</copies>
    <partitioning>
      <method>none</method>
      <schema_name/>
    </partitioning>
    <attributes/>
    <GUI>
      <xloc>720</xloc>
      <yloc>96</yloc>
    </GUI>
  </transform>
  <transform_error_handling>
  </transform_error_handling>
  <attributes/>
</pipeline>
  <pipeline_execution_configuration>
    <pass_export>N</pass_export>
    <parameters>
    </parameters>
    <variables>
    <variable><name>HOP_AUDIT_FOLDER</name><value>/Users/hans/config/audit</value></variable>
    <variable><name>HOP_AUTO_CREATE_CONFIG</name><value>Y</value></variable>
    <variable><name>HOP_CONFIG_FOLDER</name><value>/Users/hans/config</value></variable>
    <variable><name>HOP_DATASETS_FOLDER</name><value>/Users/hans/test/datasets</value></variable>
    <variable><name>HOP_ENVIRONMENT_NAME</name><value>test1</value></variable>
    <variable><name>HOP_METADATA_FOLDER</name><value>/Users/hans/test/metadata</value></variable>
    <variable><name>HOP_PIPELINE_PAN_JVM_EXIT_CODE</name><value/></variable>
    <variable><name>HOP_PROJECTS</name><value>/Users/hans/tmp/</value></variable>
    <variable><name>HOP_PROJECT_NAME</name><value>test</value></variable>
    <variable><name>HOP_UNIT_TESTS_FOLDER</name><value>/Users/hans/test</value></variable>
    <variable><name>NEO4J_CONNECTION</name><value>neo4j</value></variable>
    <variable><name>PROJECT_HOME</name><value>/Users/hans/test</value></variable>
    <variable><name>TEST</name><value>TEST</value></variable>
    <variable><name>ftp.nonProxyHosts</name><value>local|*.local|169.254/16|*.169.254/16</value></variable>
    <variable><name>http.nonProxyHosts</name><value>local|*.local|169.254/16|*.169.254/16</value></variable>
    <variable><name>jdk.debug</name><value>release</value></variable>
    <variable><name>native.encoding</name><value>UTF-8</value></variable>
    <variable><name>p1</name><value>a</value></variable>
    <variable><name>socksNonProxyHosts</name><value>local|*.local|169.254/16|*.169.254/16</value></variable>
    </variables>
    <log_level>Basic</log_level>
    <log_file>N</log_file>
    <log_filename/>
    <log_file_append>N</log_file_append>
    <create_parent_folder>N</create_parent_folder>
    <clear_log>Y</clear_log>
    <show_subcomponents>Y</show_subcomponents>
    <run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json>H4sIAAAAAAAA/+1XbVPbOBD+Kxl/upshzQsUAt/SxBzchSRNTDudpuNRZNlRkSVXkgk5hv9+K0t+
CSl3cDP37coH8Gq1evTs7rPqo6eIvCfSu/j66G3JephlU5QS78Lzjjyl2EjwmCbeBc8ZKww3IoLV
GDFFjrxMiofdXEht/TdCaW539/pn77rw0wNzhpTaChmB2edY7jJNolZ/Tc4HKMaDU4TO4j45wese
Qv3T9zgyJjyAjVzwuTnhCuIqe4QArJJGxH+gSlOewHpGpKZENUE50w1Smkh7H3uBzGId9AYGmMPK
ydZzd7mqbwCmHLhxX5jlJpb39A08aUYY5aQNW9aw+BVsOae6rYnS9lNG61QVnLq/Hr3ldPb5cjL8
wzcfEdJojRSpuc5YnlB+bUiqPY88yiPyEKA1IypDuLrHFkmyEYDPbjbhDp0QxkSpYJeBpbuXnZeS
4lUMVYgcwiUX25ihO2LKoqgY3qAVaS3pOtfE3vR2Pp8tgmUYXN/4y2B4Mw/Hw2AYBl/mcHnvC4T4
eDsL/HA4mYSX1/5kvATzFMzVzg+z2cQfTg/2Xc4WIz+8HvvT4Bp2LuCMWTiZffYXo+HSd1HmC3/p
Lz75ofs9Dj/PFuPQeZgwy4+TcDSbTv1RYO/607gA5nncS3+xgHjL0ZV/Mwynwxuz5j0deSniOWK3
kh1Ujvf0VFVaQjiRFPdhR6My5rNl8Btg/Tj5SWlkkLVEArF7JdLY8mKNvLUomMCIme83tOzZOSa9
bnR6fox6+Lwun/cnx/3nJTS3N7GY/6+ht9TQBnHVrKMsKYQIMnQXM7FtM5FY3cGQN8QjidpYcE6w
poLbFU7Eyfd2IlG2aaeg4cyaQfLbJhfUVAjIlZZwVCxk6pI2ztN01/olEi0u9AYE91dAFlNGHJJV
5xa2q1XHQFx1jAKuOqbyljbou03Gih2ERS7kPWK5kRFmFXypkc5r9XZxFUozRkKAV6KDywmuCde2
dj2UZYxiZG646nxXcM8jD1BBvUPVapmTPamWOTeUwCzLJXKsPII/lCdZ5Hy0v/TofSAoHVMJDM5d
CL/wNWuapNlEYOdbsHGx6pifeyRXnViwqKDkvgu0fE9P77Ik6nbPTjmO7rJ4IJPjaNuFfwkAD1Yd
0w1awnlAxpXIgjID6kMex0Qu6Z9OsnmerokMRRyaxMMR1hwj/TuSdc0MQWVMD0KsF0NfwjjbXAOd
ErJht1aZnxdNO2JQSkQ1R4EKBCQrcWgesgNHU6F7JH9CkhYK5GrQ5rYYk6BPRGFJM8diIYkv52NB
UqEL9smD0ZgQJFHkEpvQJtkwcktDqJFMiA5tHizYetH+3lu0WhRmgrGQ7nECVRNi9wKy+li8crKw
fDS5t0MzArQW2v0jG49VK/cOqThyPQJFXvS5c5WWgzcyNylgwx8xIdEa4btQ2Yp6b2rQgLet5qy9
pk1DqwElYZJTWJqgIm9SbBXQW/u7HTEJ0+bjUG3ENiwPraxaZAKSYKaEsyRIb4C8lID+41oIinpE
hYIZ/IX1lQVWZmqfJyBHtXaQ/lapCqpVeLJda0v1pgUwWkqDfCIZ2ZUWNE3l3bIc24wUarqnsY/V
S3pvkq4LuqR7JJ+dnJ2491pjxhfRvMYAcNdBuRYp3BOX1yxnZa6god1gLk53nEmRG0mtn8KvHuPr
qH8cRYNujAupfXZAfb5XwHwI6uQsIG+7gJbvypqTCb2H545Sow3Bd8YBwD33ecE8xD9yquiBB3S6
0kPGRuZ9Hxv1J+oA26iBICa6QubI2fevKH7O4VzAdNlZp7Vgukzg6eDsBSBVBCgCBaeflMrEn2e5
AXEuBKsVvjHyS+8yWBO2qcBU8ERE64M5nyFAVH8iteO4vT/lm3OxejlUb4m3TspKXw4V4KUWbrZp
/2/6tMT0uj4tvas+fZ0s/jvY/x3qb/Z91Y5ITDltJAAzoXJZvgWa/3U9GB/Fc2tcBXByoggUB9JC
ljWkYK7zO7toJAlk3Xw8/QVlC+SCFRAAAA==
</metastore_json>
</pipeline_configuration>

The metastore_json is base64 encoded GZip content

result

-

Register Workflow

name

registerWorkflow

description

Register a workflow on the server

endPoint

GET /hop/registerWorkflow

parameters

  • xml:

example request

-

result

-

Remove Pipeline

name

removePipeline

description

Remove a pipeline from the server

endPoint

GET /hop/removePipeline

parameters

  • name: the name of the pipeline to remove

  • id: the id of the pipeline to remove

example request

GET http://localhost:8081/hop/removePipeline/?name=<NAME>&id=<ID>

result

Example result:

<HTML>

<HEAD>
	<TITLE>The pipeline was removed</TITLE>
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H3>The pipeline with name [<NAME>] and hopServer object id <ID> was removed from Hop
		Server.</H3>
	<a href="/hop/status">Back to the status page</a><br>
	<p>
</BODY>

Remove Workflow

name

removeWorkflow

description

Remove a workflow from the server

endPoint

GET /hop/removeWorkflow

parameters

  • name: the name of the workflow to remove

  • id: the id of the workflow to remove

example request

GET http://localhost:8081/hop/removeWorkflow/?name=<NAME>&id=<ID>

result

Example result:

<HTML>

<HEAD>
	<TITLE>The workflow was removed</TITLE>
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H3>The workflow with name [<NAME>] and hopServer object id <ID> was removed from Hop
		Server.</H3>
	<a href="/hop/status">Back to the status page</a><br>
	<p>
</BODY>

Sniff Transform

name

sniffTransform

description

Sniff test a pipeline transform

endPoint

GET /hop/sniffTransform

parameters

  • xml: use XML (default: Y)

  • pipeline: name of the pipeline to sniff

  • id: id of the pipeline to sniff

  • transform: name of the transform to sniff

  • type: (input/output) sniff input or output

  • lines: number of lines to wait for

  • copynr: when using multiple copies you can specify the copynr

following parameters are required

  • xml

  • id or pipeline

  • transform

example request

GET http://localhost:8081/hop/sniffTransform/?xml=Y&pipeline=<NAME>&id=<ID>&transform=<TRANSFORMNAME>&type=output&lines=100

result

Example result (empty):

<row-buffer>
<row-meta/>
</row-buffer>

Start Pipeline Execution

name

startExec

description

Start the execution of a pipeline

endPoint

GET /hop/startExec

parameters

  • name: the name of the pipeline to start

example request

GET http://localhost:8081/hop/startExec?name=<NAME>

result

Example result:

<HTML>

<HEAD>
	<TITLE>Start of pipeline</TITLE>
	<META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>">
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H1>Pipeline [<NAME>] was started.</H1>
	<a href="/hop/pipelineStatus?name=<NAME>&id=8bea27db-de97-4bd0-a210-d9bba3aacac2">Back to the status page</a>
	<p>
		<p>
</BODY>

</HTML>

Start Pipeline

name

startPipeline

description

Prepare and start the execution of a pipeline

endPoint

GET /hop/startPipeline

parameters

  • name: the name of the pipeline to start

example request

GET http://localhost:8081/hop/startPipeline?name=<NAME>

result

Example result:

<HTML>

<HEAD>
	<TITLE>Start of pipeline</TITLE>
	<META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>">
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H1>Pipeline [<NAME>] was started.</H1>
	<a href="/hop/pipelineStatus?name=<NAME>&id=8bea27db-de97-4bd0-a210-d9bba3aacac2">Back to the status page</a>
	<p>
		<p>
</BODY>

</HTML>

Start Workflow

name

startWorkflow

description

Prepare and start the execution of a workflow

endPoint

GET /hop/startPipeline

parameters

  • name: the name of the workflow to start

example request

GET http://localhost:8081/hop/startWorkflow?name=<NAME>

result

Example result:

<HTML>

<HEAD>
	<TITLE>Start of workflow</TITLE>
	<META http-equiv="Refresh" content="2;url=/hop/startWorkflow?name=<NAME>">
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H1>Workflow [<NAME>] was started.</H1>
	<a href="/hop/workflowStatus?name=<NAME>&id=8bea27db-de97-4bd0-a210-d9bba3aacac2">Back to the status page</a>
	<p>
		<p>
</BODY>

</HTML>

Stop Pipeline

name

stopPipeline

description

Stop a pipeline

endPoint

GET /hop/stopPipeline

parameters

  • name: the name of the pipeline to stop

  • id: the id of the pipeline to stop

example request

GET http://localhost:8081/hop/stopPipeline?name=<NAME>&id=<ID>

result

Example Result:

<HTML>

<HEAD>
	<TITLE>Stop pipeline</TITLE>
	<META http-equiv="Refresh" content="2;url=/hop/pipelineStatus?name=<NAME>>">
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H1>Pipeline [<NAME>] stop requested.</H1>
	<a href="/hop/pipelineStatus?name=<NAME>&id=<ID>">Back to the pipeline status page</a>
	<p>
		<p>
</BODY>

</HTML>

Stop Workflow

name

stopWorkflow

description

Stop a workflow

endPoint

GET /hop/stopWorkflow

parameters

  • name: the name of the workflow to stop

  • id: the id of the workflow to stop

example request

GET http://localhost:8081/hop/stopWorkflow?name=<NAME>&id=<ID>

result

Example Result:

<HTML>

<HEAD>
	<TITLE>Stop workflow</TITLE>
	<META http-equiv="Refresh" content="2;url=/hop/workflowStatus?name=<NAME>>">
	<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
</HEAD>

<BODY>
	<H1>Workflow [<NAME>] stop requested.</H1>
	<a href="/hop/workflowStatus?name=<NAME>&id=<ID>">Back to the pipeline status page</a>
	<p>
		<p>
</BODY>

</HTML>