The Hop codebase

Welcome to the Hop project! The size of the codebase can perhaps appear to be a bit daunting at first so remember: Don’t Panic.

When it comes down to it the structure is fairly simple. Three modules carry almost all of the framework — core, engine and ui — and everything else is either a plugin, a packaging concern, or a different front end onto the same three.

The Maven reactor

Module What it is

core

The essentials needed to bootstrap Hop.

engine

Pipelines, workflows, and the tools that run them.

ui

The Hop GUI.

rcp

The desktop fragment of the GUI: SWT-only pieces that cannot run in a browser.

rap

The Hop Web fragment of the GUI, on Eclipse RAP instead of SWT.

rest

A REST API in front of the engine.

plugins

Everything optional: transforms, actions, databases, engines, technology integrations.

lib, lib-jdbc, lib-p2

Dependency aggregators (pom packaging only, no code): shared third-party versions, the JDBC drivers, and the Eclipse p2 artifacts that are not on Maven Central.

assemblies

Turns all of the above into the client, server and web distributions.

Also in the repository, outside the reactor: integration-tests and web-tests (see Integration tests), docker and helm for container images, tools for build and release helper scripts, and docs for the three manuals.

Core

Like the name suggests, this module contains the essentials to bootstrap Hop:

  • The plugin registry which keeps track of all the plugins: PluginRegistry

  • Core data types: IValueMeta, ValueMetaString, …​

  • Rows of data: IRowMeta, RowMeta

  • Logging: HopLogStore, ILogChannel, …​

  • Generic relational database handling: Database, IDatabase, DatabaseMeta, …​

  • The metadata API and its serializers: IHopMetadata, IHopMetadataProvider, HopMetadataProperty, …​

  • Virtual file system access: HopVfs

  • Internationalization a.k.a. i18n to translate stuff: BaseMessages

  • Password obfuscation and encryption: Encr

  • Other utility classes like: Const

  • Bootstrap everything in core with: HopClientEnvironment.init()

HopClientEnvironment.init() registers the eight core plugin types: logging, value types, databases, database type rules, extension points, password encoders, variable resolvers and VFS.

Engine

This module contains the actual framework to handle pipelines and workflows:

  • The hop command line and its sub-commands: hop run, hop server, hop conf, hop search, hop encrypt, hop import

  • Pipelines: PipelineMeta, Pipeline, IPipelineEngine, …​

  • Workflows: WorkflowMeta, Workflow, IWorkflowEngine, …​

  • Web server: HopServer, StartPipelineServlet, …​

  • Execution information, auditing and lineage: IExecutionInfoLocation, AuditManager, LineageHub, …​

  • Other stuff worth mentioning: partitioning, data streams, …​

  • Bootstrap everything in engine with: HopEnvironment.init()

HopEnvironment.init() registers everything in HopClientEnvironment.init() plus the engine-side plugin types: transforms, actions, pipeline and workflow engines, metadata, server servlets, compression, partitioners, configuration options, hop commands, execution information locations and samplers, and lineage sinks.

UI, RCP and RAP

ui contains everything related to the Hop GUI:

  • Tools: hop-gui, hop-translate

  • Hop GUI: HopGui, HopDataOrchestrationPerspective, HopPipelineFileType, …​

  • The rest is mostly a collection of re-usable dialogs and widgets

HopGuiEnvironment.init() adds the four GUI plugin types: GUI plugins, perspectives, file types and searchable analysers.

The same GUI runs on the desktop and in a browser, so it is split: rcp holds the desktop-only pieces and rap the Hop Web ones. Anything you write in ui has to work in both — see Hop Web antipatterns.

Plugins

The plugins module contains all the functionality in Hop which is not absolutely strictly essential. This means you will find a lot of transforms and actions there, along with database dialects, VFS providers, engines and technology integrations.

It is also the best collection of examples: whatever you are trying to build, something in plugins already does something close to it. See How plugins work and the plugin type catalogue.

Where to go next