Execution and observability

The shared mechanics — discovery, annotation attributes, classloading, how a plugin gets into the distribution — are on the plugin types overview.

Extension point

An extension point is a hook: Hop announces that something happened, and every plugin listening for that announcement gets called. There are about a hundred named points, from HopGuiStart through PipelinePrepareExecution and TransformBeforeStart to PipelineCompleted, and the Hop codebase itself uses them heavily.

Annotation

@ExtensionPoint

Implement

IExtensionPoint<T>

Plugin type

ExtensionPointPluginType, id EXTENSION_POINT

Examples

over a hundred in the codebase; engine/src/main/java/org/apache/hop/lineage/xp is a compact set

java
@ExtensionPoint(
    id = "LineageHubPipelineCompletedXp",
    extensionPointId = "PipelineCompleted",
    description = "Emits lineage at pipeline completion")
public class LineageHubPipelineCompletedXp
    implements IExtensionPoint<IPipelineEngine<PipelineMeta>> {
  @Override
  public void callExtensionPoint(
      ILogChannel log, IVariables variables, IPipelineEngine<PipelineMeta> pipeline) { ... }
}

Two ids, and they are not the same thing:

id

the plugin’s own id, unique among extension points.

extensionPointId

which hook to listen to. The names are the constants of the HopExtensionPoint enum in org.apache.hop.core.extension, which is also the list of what is available. Anything can fire an extension point, so a plugin may define and announce its own.

The generic parameter is the object Hop hands over, and it decides where your plugin will and will not work:

Bind to the interface, not the implementation. An extension point declared as IExtensionPoint<Pipeline> or IExtensionPoint<Workflow> works on the local engine and throws a ClassCastException on remote and Beam engines, where the object is a different implementation of IPipelineEngine / IWorkflowEngine. The failure only shows up on the engine you were not testing on.

Extension points run inline, on the thread that fired them. Anything slow belongs on a queue, not in callExtensionPoint.

Failures

Experimental in 2.21: the shape of the failure object can still change.

When a transform, action, pipeline or workflow fails, Hop announces it as an ExecutionFailure (org.apache.hop.core.failure): what failed, in which phase, why, and in which category.

Extension point Fired when

TransformFailed

A transform threw an exception in init() or processRow(), its init() returned false, or it finished with errors without throwing. Once per transform copy.

PipelineFailed

The pipeline failed as a whole: after its finished listeners, before PipelineCompleted. Also when it failed to initialize or to prepare.

ActionFailed

An action threw an exception, its error count went up, a child pipeline or workflow it ran failed, or it returned false after logging an error. An action on a failure hop that only passes on the errors it was given, like Write to log, did not fail.

WorkflowFailed

The workflow ended with a false result, the rule hop-run uses for its exit code. Not when someone stopped it and nothing went wrong.

A transform or action failure names the component (name, plugin id, copy) and carries the cause: the exception class and message, the root cause, the first SQL state, and a stack trace cut off at 16,000 characters. When the transform or action only logged an error, the cause is that last error line, and cause.fromLog is true. A pipeline or workflow failure points at the failure that brought it down through causedBy, and so does an action whose child pipeline or workflow failed: getRootCause() follows the chain from a workflow down to the transform where it started. An action that fails on a failure path, like an Abort after the failure hop of a SQL action, points at the failure that led there too. That failure is passed on until an action recovers from it: one that ends with a true result and no errors.

category comes from fixed rules on exception class names and SQL states (FailureClassifier), tried from the root cause outwards. The wording of a message only counts for "No space left on device", which is RESOURCE. CHILD_FAILURE is reserved for children that run remotely; nothing produces it yet. Every failure in a chain carries the category of its root cause, so a rule like "retry on `CONNECTIVITY`" can look at the top of the chain.

Only the local pipeline and workflow engines report failures: with a remote, Beam or Spark run configuration there is none. The local engines also offer the failures without a plugin: Pipeline.addExecutionFailureListener() and Workflow.addExecutionFailureListener(), and getFailures() / getRunFailure() once the run is done.

Where the failure ends up:

  • Execution information: ExecutionState.failure, on the state of the pipeline or workflow and on the state of a failed transform, at every execution information location. A pipeline that fails to prepare or to initialize is registered at its location as well, with that failure. Elastic and OpenSearch store executionState.failure without indexing it: new indexes get that mapping, Create index adds it to an existing index, and so does the location before its first write to an index. Adding a mapping needs more rights than writing documents: when it is refused, the failure is mapped field by field, which can add a few hundred fields against the field limit of the index. Neo4j stores it as a JSON string in the failure property of the Execution node.

  • Hop Server: the pipeline and workflow status fill errorDescription with one line, for example Transform 'Read orders' (TableInput) failed [CONNECTIVITY]: Connection refused, and the JSON form adds the failure object.

  • Hop Run: the file of --result-json, see the user manual.

The message, the root cause message and the stack trace of a cause are redacted when the failure is built, before any extension point or listener sees it: CredentialRedactor.redact(text, secrets) masks credentials in URLs, name=value and JSON pairs, encrypted passwords, and the values of the variables whose name says they are a credential. cause.redacted is true when that happened.

Logging

A logging plugin participates in the lifecycle of the logging subsystem. It is initialised very early — HopClientEnvironment.init() loads logging plugins immediately after the registry is filled, before anything else — which is what makes it the right place to attach an external logging framework or a log store.

Annotation

@LoggingPlugin

Implement

ILoggingPlugin

Plugin type

LoggingPluginType, id LOGGING

The interface is just init() and dispose(); everything else is done by registering your own listeners with Hop’s logging registry from init().

There are no implementations in the distribution today, so there is no in-tree example to copy. For routing log lines somewhere else, an extension point on the logging-related hooks, or an execution information location, is usually the easier route.

Execution information location

An execution information location is where Hop stores what happened during a run: the execution itself, its state, its logging text, its metrics, and any sampled data. This is what the Hop GUI execution perspective and Hop Server read back.

Annotation

@ExecutionInfoLocationPlugin

Implement

IExecutionInfoLocation

Plugin type

ExecutionInfoLocationPluginType, id EXECUTION_INFO_LOCATIONS

Examples

engine/src/main/java/org/apache/hop/execution/local (files), …​/remote (another Hop Server), …​/caching, plugins/misc/execution-database (a relational database)

The interface has two halves. The writing half — registerExecution, updateExecutionState, registerData — is called by the engines while a pipeline or workflow runs. The reading half — getExecution, getExecutionState, findExecutions, findChildIds, getExecutionData — is called by the UI and the server. Implement both, including the parent/child lookups, or drill-down in the execution perspective will not work.

The settings live in an Execution Information Location metadata object, so the plugin’s fields carry @HopMetadataProperty and @GuiWidgetElement annotations, exactly like a variable resolver.

Locations are written to on every state update of every component. Buffering is normal — unBuffer(executionId) and clearCaches() exist for that reason — and the caching implementations in the engine are the examples to follow.

Execution data sampler

A sampler decides which rows are captured while a pipeline runs, so they can be shown afterwards. The built-ins are first rows, last rows, random rows and basic data profiling.

Annotation

@ExecutionDataSamplerPlugin

Implement

IExecutionDataSampler<Store>

Plugin type

ExecutionDataSamplerPluginType, id EXECUTION_DATA_SAMPLER_LOCATIONS

Examples

engine/src/main/java/org/apache/hop/execution/sampler/plugins

Two methods do the work:

createSamplerStore(ExecutionDataSamplerMeta)

creates the store this sampler accumulates into, one per transform copy and stream

sampleRow(store, streamType, rowMeta, row)

called for every row on that stream

sampleRow is on the hot path: it runs for every row of every sampled transform. Keep it allocation-free and cheap, and put the analysis in the store rather than in the sampling.

Samplers are selected in the pipeline run configuration, and their stores are written to the execution information location.

Lineage sink

A lineage sink is where Hop’s lineage events are delivered — a catalogue, a broker, a file. Hop ships an OpenLineage sink.

Annotation

@LineageSinkPlugin

Implement

ILineageSink

Plugin type

LineageSinkPluginType, id LINEAGE_SINKS

Example

plugins/tech/openlineage

Details

Lineage observation hub

accept(List<LineageEvent>) receives batches from the lineage hub, with optional init(…​) and shutdown() around it. The hub dispatches asynchronously, so a slow sink slows down lineage delivery, not the pipeline.

Hop Server servlet

A server plugin is an HTTP endpoint on Hop Server. Everything the server exposes — status, execution, sniffing, the export of pipelines — is one of these.

Annotation

@HopServerServlet

Implement

IHopServerPlugin, in practice by extending BaseHttpServlet or BaseHopServerPlugin

Plugin type

HopServerPluginType, id HOP_SERVER_SERVLET

Examples

engine/src/main/java/org/apache/hop/www

java
@HopServerServlet(id = "pipelineStatus", name = "Get the status of a pipeline")
public class GetPipelineStatusServlet extends BaseHttpServlet implements IHopServerPlugin { ... }

getContextPath() is the path the servlet is mounted on, and setup(PipelineMap, WorkflowMap) hands over the server’s registries of running pipelines and workflows.

A servlet plugin is an addition to the attack surface of Hop Server. Authentication is handled by the server, not by the servlet, but everything else — what the endpoint exposes, what it accepts, and what it does with it — is on the plugin. See SECURITY.md and the threat model it links before adding one.