Hop Plugin Development

This page explains how to develop new plugins with references to make development easy.

For contributing toolbar buttons and other GUI-only extensions (@GuiPlugin, @GuiToolbarElement, @GuiToolbarElementFilter), see GUI plugins and toolbars. That page covers the TextComposite and TableView extension points (SQL formatters, JSON tools, table export, and similar).

Plugin Registry

Hop keeps a central registry of all the available Hop plugins. They are organized per plugin type (see below). The registry also keeps track of the class loaders that were created and used for the various plugins to avoid duplication. You can get a hold of this plugin registry at any place using

PluginRegistry.getInstance()

Plugin types

Plugins extend Hop in just over thirty different areas. Each of these areas is a plugin type: a class implementing IPluginType<T extends Annotation>, usually by extending BasePluginType<T>, whose job is to find plugins of its kind and register them.

For the catalogue of what those thirty types are, what each one does, and what you need to write one, see Plugin types.

A plugin type declares the annotation that marks a plugin and the interface that plugin has to implement:

@PluginMainClassType(ITransformMeta.class)
@PluginAnnotationType(Transform.class)
public class TransformPluginType extends BasePluginType<Transform> { ... }

Discovery is annotation-driven and works in two passes, both reading a Jandex annotation index (META-INF/jandex.idx) from the jars:

  1. Jars on the classpath, for the plugins built into hop-core, hop-engine and hop-ui.

  2. Jars under the plugin folders — plugins/ by default, configurable with HOP_PLUGIN_BASE_FOLDERS. Each of those gets its own classloader, covering the plugin jar plus the jars in its lib/ folder.

A jar without a Jandex index contains no plugins as far as Hop is concerned. The Hop root pom binds the jandex-maven-plugin for every module; a plugin built outside the Hop repository has to add it, see Porting Kettle plugins.

Plugins

You can use the plugin registry to search for plugins using an ID, a class name. You can list all plugins and types. Plugins contain a lot of information and implement the IPlugin interface. Here are a few of the main fields:

Field Type Optional Description

ids

String[]

Mandatory

The unique ID of the plugin within its type. A plugin can have more than 1 ID to make it possible to merge multiple plugins into one and still remain backward compatible.

name

String

Optional

The name of the plugin as shown in the user interface

description

String

Optional

The description of the plugin as shown in the user interface

category

String

Optional

used to organize certain types of plugins in the UI, for example transforms and actions

imageFile

String

Optional

The path to the SVG file representing the icon of this plugin.

documentationUrl

String

Optional

The URL of the documentation for the plugin

keywords

String[]

Optional

The keywords which represent this plugin to make it easier to find it in the UI

nativePlugin

boolean

false

Indicates if the plugin was found on the classpath (native) or in one of the plugin folders

Aside from this list you can find the main class, the libraries in the plugin classpath and so on. The information in the IPlugin objects is copied from the plugin annotations on the plugin classes by the PluginType class.

Where are the Hop plugins?

In Hop all optional plugins are in the hop/plugins folder, grouped by what they are:

  • transforms: pipeline transforms

  • actions: workflow actions

  • databases: relational database dialects

  • engines: pipeline and workflow engines

  • tech: everything belonging to one technology (Azure, Google, AWS, FTP, SFTP, Arrow, …​), often several plugin types in one module

  • vfs: Virtual File System plugins that are not part of a tech module

  • valuetypes: value types

  • resolvers: variable resolvers

  • misc: miscellaneous plugins. Pick this category only when the other ones don’t fit.

Steps to add a new plugin

Once you have identified the plugin type you need — see Plugin types — and the folder it belongs in, adding it is three edits and two new files.

Create the module

Create the plugin as a new Maven sub-module with the category pom as its parent, and add it to that pom’s <modules>.

The category parent already gives you everything a plugin needs: hop-core, hop-engine, hop-ui and SWT as provided dependencies, the Jandex index, the shared archive resources, and the SWTBot UI test stack on the test classpath. A plugin pom is usually no more than a parent, an artifactId, a name, and whatever extra dependencies the plugin itself needs.

Duplicating a comparable existing plugin is the fastest way to start. When you do, take the time to strip out what does not apply and to rewrite the documentation.

Package it as a plugin zip

Add two files to the module:

src/assembly/assembly.xml

Its presence activates the assembly profile that turns the module into a plugin zip — there is no extra pom configuration. Copy one from a comparable plugin: it names the zip, ships version.xml, and includes the shared component assemblies/shared/hop-plugin-libs.xml.

src/main/resources/version.xml

A one-line file containing ${project.version}.

The shared component routes artifacts by Maven scope: the plugin jar and its runtime dependencies go into the plugin’s own folder and its lib/, non-Hop provided dependencies go to the shared lib/core, and JDBC drivers go to lib/jdbc.

Review what ends up in the zip. Anything already shipped in lib/core must be excluded — the plugin classloader sees the parent’s copy, so a second copy is at best dead weight and at worst a version conflict.

Add it to the distribution

Add a dependency on the module, with <type>zip</type>, to assemblies/plugins/pom.xml. That is what actually puts the plugin in the client, server and web distributions.

Documentation and samples

Two things that are easy to forget and are expected of a plugin in the Hop repository:

  • A page under docs/hop-user-manual, referenced from the plugin’s documentationUrl attribute.

  • A sample pipeline or workflow, see Plugin samples.

Plugins outside the Hop repository

A plugin whose dependencies are not Apache category A cannot ship with Hop. It follows the same shape but builds and releases on its own — see Creating your own plugin and Publishing a plugin to Nexus.