Execution and observability
The shared mechanics — discovery, annotation attributes, classloading, how a plugin gets into the distribution — are on the plugin types overview.
Extension point
An extension point is a hook: Hop announces that something happened, and every plugin listening for that announcement gets called.
There are about a hundred named points, from HopGuiStart through PipelinePrepareExecution and TransformBeforeStart to PipelineCompleted, and the Hop codebase itself uses them heavily.
- Annotation
-
@ExtensionPoint - Implement
-
IExtensionPoint<T> - Plugin type
-
ExtensionPointPluginType, idEXTENSION_POINT - Examples
-
over a hundred in the codebase;
engine/src/main/java/org/apache/hop/lineage/xpis a compact set
@ExtensionPoint(
id = "LineageHubPipelineCompletedXp",
extensionPointId = "PipelineCompleted",
description = "Emits lineage at pipeline completion")
public class LineageHubPipelineCompletedXp
implements IExtensionPoint<IPipelineEngine<PipelineMeta>> {
@Override
public void callExtensionPoint(
ILogChannel log, IVariables variables, IPipelineEngine<PipelineMeta> pipeline) { ... }
}
Two ids, and they are not the same thing:
id-
the plugin’s own id, unique among extension points.
extensionPointId-
which hook to listen to. The names are the constants of the
HopExtensionPointenum inorg.apache.hop.core.extension, which is also the list of what is available. Anything can fire an extension point, so a plugin may define and announce its own.
The generic parameter is the object Hop hands over, and it decides where your plugin will and will not work:
|
Bind to the interface, not the implementation.
An extension point declared as |
Extension points run inline, on the thread that fired them.
Anything slow belongs on a queue, not in callExtensionPoint.
Failures
| Experimental in 2.21: the shape of the failure object can still change. |
When a transform, action, pipeline or workflow fails, Hop announces it as an ExecutionFailure (org.apache.hop.core.failure): what failed, in which phase, why, and in which category.
| Extension point | Fired when |
|---|---|
|
A transform threw an exception in |
|
The pipeline failed as a whole: after its finished listeners, before |
|
An action threw an exception, its error count went up, a child pipeline or workflow it ran failed, or it returned false after logging an error. An action on a failure hop that only passes on the errors it was given, like Write to log, did not fail. |
|
The workflow ended with a false result, the rule |
A transform or action failure names the component (name, plugin id, copy) and carries the cause: the exception class and message, the root cause, the first SQL state, and a stack trace cut off at 16,000 characters.
When the transform or action only logged an error, the cause is that last error line, and cause.fromLog is true.
A pipeline or workflow failure points at the failure that brought it down through causedBy, and so does an action whose child pipeline or workflow failed: getRootCause() follows the chain from a workflow down to the transform where it started.
An action that fails on a failure path, like an Abort after the failure hop of a SQL action, points at the failure that led there too.
That failure is passed on until an action recovers from it: one that ends with a true result and no errors.
category comes from fixed rules on exception class names and SQL states (FailureClassifier), tried from the root cause outwards.
The wording of a message only counts for "No space left on device", which is RESOURCE.
CHILD_FAILURE is reserved for children that run remotely; nothing produces it yet.
Every failure in a chain carries the category of its root cause, so a rule like "retry on `CONNECTIVITY`" can look at the top of the chain.
Only the local pipeline and workflow engines report failures: with a remote, Beam or Spark run configuration there is none.
The local engines also offer the failures without a plugin: Pipeline.addExecutionFailureListener() and Workflow.addExecutionFailureListener(), and getFailures() / getRunFailure() once the run is done.
Where the failure ends up:
-
Execution information:
ExecutionState.failure, on the state of the pipeline or workflow and on the state of a failed transform, at every execution information location. A pipeline that fails to prepare or to initialize is registered at its location as well, with that failure. Elastic and OpenSearch storeexecutionState.failurewithout indexing it: new indexes get that mapping, Create index adds it to an existing index, and so does the location before its first write to an index. Adding a mapping needs more rights than writing documents: when it is refused, the failure is mapped field by field, which can add a few hundred fields against the field limit of the index. Neo4j stores it as a JSON string in thefailureproperty of theExecutionnode. -
Hop Server: the pipeline and workflow status fill
errorDescriptionwith one line, for exampleTransform 'Read orders' (TableInput) failed [CONNECTIVITY]: Connection refused, and the JSON form adds thefailureobject. -
Hop Run: the file of
--result-json, see the user manual.
The message, the root cause message and the stack trace of a cause are redacted when the failure is built, before any extension point or listener sees it: CredentialRedactor.redact(text, secrets) masks credentials in URLs, name=value and JSON pairs, encrypted passwords, and the values of the variables whose name says they are a credential.
cause.redacted is true when that happened.
Logging
A logging plugin participates in the lifecycle of the logging subsystem.
It is initialised very early — HopClientEnvironment.init() loads logging plugins immediately after the registry is filled, before anything else — which is what makes it the right place to attach an external logging framework or a log store.
- Annotation
-
@LoggingPlugin - Implement
-
ILoggingPlugin - Plugin type
-
LoggingPluginType, idLOGGING
The interface is just init() and dispose(); everything else is done by registering your own listeners with Hop’s logging registry from init().
There are no implementations in the distribution today, so there is no in-tree example to copy. For routing log lines somewhere else, an extension point on the logging-related hooks, or an execution information location, is usually the easier route.
Execution information location
An execution information location is where Hop stores what happened during a run: the execution itself, its state, its logging text, its metrics, and any sampled data. This is what the Hop GUI execution perspective and Hop Server read back.
- Annotation
-
@ExecutionInfoLocationPlugin - Implement
-
IExecutionInfoLocation - Plugin type
-
ExecutionInfoLocationPluginType, idEXECUTION_INFO_LOCATIONS - Examples
-
engine/src/main/java/org/apache/hop/execution/local(files),…/remote(another Hop Server),…/caching,plugins/misc/execution-database(a relational database)
The interface has two halves.
The writing half — registerExecution, updateExecutionState, registerData — is called by the engines while a pipeline or workflow runs.
The reading half — getExecution, getExecutionState, findExecutions, findChildIds, getExecutionData — is called by the UI and the server.
Implement both, including the parent/child lookups, or drill-down in the execution perspective will not work.
The settings live in an Execution Information Location metadata object, so the plugin’s fields carry @HopMetadataProperty and @GuiWidgetElement annotations, exactly like a variable resolver.
Locations are written to on every state update of every component.
Buffering is normal — unBuffer(executionId) and clearCaches() exist for that reason — and the caching implementations in the engine are the examples to follow.
Execution data sampler
A sampler decides which rows are captured while a pipeline runs, so they can be shown afterwards. The built-ins are first rows, last rows, random rows and basic data profiling.
- Annotation
-
@ExecutionDataSamplerPlugin - Implement
-
IExecutionDataSampler<Store> - Plugin type
-
ExecutionDataSamplerPluginType, idEXECUTION_DATA_SAMPLER_LOCATIONS - Examples
-
engine/src/main/java/org/apache/hop/execution/sampler/plugins
Two methods do the work:
createSamplerStore(ExecutionDataSamplerMeta)-
creates the store this sampler accumulates into, one per transform copy and stream
sampleRow(store, streamType, rowMeta, row)-
called for every row on that stream
|
|
Samplers are selected in the pipeline run configuration, and their stores are written to the execution information location.
Lineage sink
A lineage sink is where Hop’s lineage events are delivered — a catalogue, a broker, a file. Hop ships an OpenLineage sink.
- Annotation
-
@LineageSinkPlugin - Implement
-
ILineageSink - Plugin type
-
LineageSinkPluginType, idLINEAGE_SINKS - Example
-
plugins/tech/openlineage - Details
accept(List<LineageEvent>) receives batches from the lineage hub, with optional init(…) and shutdown() around it.
The hub dispatches asynchronously, so a slow sink slows down lineage delivery, not the pipeline.
Hop Server servlet
A server plugin is an HTTP endpoint on Hop Server. Everything the server exposes — status, execution, sniffing, the export of pipelines — is one of these.
- Annotation
-
@HopServerServlet - Implement
-
IHopServerPlugin, in practice by extendingBaseHttpServletorBaseHopServerPlugin - Plugin type
-
HopServerPluginType, idHOP_SERVER_SERVLET - Examples
-
engine/src/main/java/org/apache/hop/www
@HopServerServlet(id = "pipelineStatus", name = "Get the status of a pipeline")
public class GetPipelineStatusServlet extends BaseHttpServlet implements IHopServerPlugin { ... }
getContextPath() is the path the servlet is mounted on, and setup(PipelineMap, WorkflowMap) hands over the server’s registries of running pipelines and workflows.
|
A servlet plugin is an addition to the attack surface of Hop Server. Authentication is handled by the server, not by the servlet, but everything else — what the endpoint exposes, what it accepts, and what it does with it — is on the plugin. See SECURITY.md and the threat model it links before adding one. |