Metadata and configuration

A metadata plugin adds a new kind of object to the Metadata perspective: relational database connections, run configurations, VFS connections, servers, partition schemas and around forty others are all metadata objects. This is usually the plugin type you want when a plugin needs configuration that is shared between pipelines rather than living inside one.

Any plain Java object will do. Annotate the class with @HopMetadata, the fields with @HopMetadataProperty, and serialisation to JSON is handled for you.

An editor class extending MetadataEditor gives it a GUI.

A configuration plugin adds options to an existing Hop command line tool, and — with one extra annotation — a tab in the GUI options dialog. The projects plugin uses this to add project options to hop run, and the search settings use it to add a Search tab to the options.

The category decides which tool picks the options up. Each tool mixes in only the plugins of its own category:

The options themselves are picocli @CommandLine.Option fields on the class; the plugin is added to the command as a mixin. handleOption(…​) is then called and returns true when it handled something and the tool should not continue with its default behaviour.

Add @GuiPlugin and @GuiWidgetElement annotations to the same fields and the same options become a tab in the Hop GUI options dialog. That path requires a static getInstance() method on the class, because the dialog needs the singleton holding the current values.

Where a configuration plugin extends an existing command, a @HopCommand plugin adds a whole new hop sub-command — this is how hop run, hop conf, hop server, hop encrypt, hop search, hop doc and hop marketplace are all built.

The id is the word the user types. initialize(CommandLine, IVariables, MultiMetadataProvider) hands you the command line and the environment; run() does the work.

An import plugin converts a project from another tool into a Hop project, and shows up in hop import and in the GUI import wizard. Hop ships one, for Kettle/PDI.

The interface is broad because importing is: it covers the input and output folders, the property and shared-object files of the source tool, the target project and environment to create, and an import report.

A variable resolver fetches a value from somewhere else when a variable is dereferenced — a secret manager, a key vault, or another pipeline. It is what makes #{secret/path} work without a password ever being written into a file.

The interface is two methods that matter: init() and resolve(String secretPath, IVariables variables).

The settings are held in a Variable Resolver metadata object, so the resolver’s fields carry both @HopMetadataProperty (to be saved) and @GuiWidgetElement with parentId = VariableResolver.GUI_PLUGIN_ELEMENT_PARENT_ID (to be edited).

This decides how passwords are obfuscated or encrypted in metadata files. Hop ships the default Hop obfuscation, and the passwords plugin adds AES variants.

encode(…​), decode(…​) and getPrefixes(): the prefixes (`Encrypted `, `AES `, …) are how Hop recognises which encoder produced a stored value, so an encoder must claim a prefix nothing else uses.

Only one encoder is active per process. It is selected by the HOP_PASSWORD_ENCODER_PLUGIN variable or system property and defaults to Hop; HopClientEnvironment.init() reads it and calls Encr.init(…​).

Two related plugin types describing credentials rather than connections: a provider is a kind of credential (username/password, Kerberos), a consumer is something that can use one.

Both are small: a display name, and for the consumer the class that performs the authentication. Hop ships two providers and no consumers, so this pair is best treated as a low-traffic extension point — most plugins that need credentials use a metadata object instead.