What is Apache Hop?
Apache Hop is a metadata-driven data integration and data orchestration platform.
Metadata driven
Everything in Hop is metadata.
The pipelines that read, combine and transform your data are metadata. So are the workflows that orchestrate them, the connections to the databases, files and services you work with, and the run configurations that decide where and how any of it executes.
Nothing in a Hop pipeline is bound to a specific database, file format or processing engine. A pipeline describes what should happen to your data; the metadata around it decides where that happens and against what. That separation is what lets the same pipeline run against a developer’s local database and a production warehouse, or on your laptop and on a Spark cluster, without being rewritten.
Hop takes this a step further with metadata injection: a template pipeline that receives its metadata at runtime, so one pipeline can process a hundred different files or tables instead of you building a hundred near-identical pipelines.
Built to fit your architecture
Hop is designed to blend into the data architecture you already run: relational databases, files, object stores on AWS, Azure and Google Cloud, NoSQL and graph databases, message queues, REST and SOAP services, SaaS platforms. Hop connects them and moves data between them.
That is also why Hop is deliberately lightweight. At the core is a small, fast engine; everything else is a plugin, and you add or remove plugins to make your installation exactly what you need and nothing more. The plugin architecture is open, so you can write your own and load them the same way.
The result runs anywhere from an edge device in an IoT scenario to a cluster processing very large volumes, on bare metal, in a container, or on Kubernetes.
Designed visually, managed like software
Hop developers build pipelines and workflows visually, in a full development environment called Hop Gui, available on Windows, macOS and Linux and in the browser as Hop Web.
Visual design means you describe the problem instead of the plumbing, and it means the next person to open your pipeline can see what it does at a glance instead of reading through hundreds of lines of code.
What Hop does not do is trap that work in a proprietary tool. A pipeline is a plain file on disk, and a project is an ordinary folder of them. From that point on, a Hop project is managed like any other software project:
- Version control
-
Projects and environments live in Git. Branch, review, diff and roll back like any other repository. Hop Gui has git integration built in, so you can do most of it without leaving the tool.
- Unit tests
-
Pipeline unit tests validate a pipeline’s output against a golden data set. Logging tells you a pipeline ran; a unit test tells you it produced the right data. Regression tests keep a fixed bug fixed.
- Static analysis
-
Linting checks pipelines, workflows and metadata without running them or connecting to anything, so it is safe to run on every commit.
- CI/CD
-
Everything Hop needs to run is on the command line, so building, testing and deploying a Hop project is an ordinary build pipeline. Promote the same project from development through test to production by pointing it at a different environment.
Flexible runtimes
You design a pipeline once and choose at runtime where it executes. A run configuration is metadata like everything else, so switching engines does not mean redesigning anything.
Workflows run on Hop’s own engine, locally or on a remote Hop Server.
Pipelines have more options:
| Engine | What it’s for |
|---|---|
The default. Runs on the machine you’re on, and handles a very large share of real workloads on its own. |
|
The same engine on a Hop Server, with load balancing across several servers when you need it. |
|
Executes the pipeline directly as a Spark job, without translating it through another framework. Runs locally, on a cluster through |
|
Mainly used to run pipelines on Google Cloud Dataflow. Beam run configurations for Spark and Flink are also available. |
Depending on the engine and the transforms you use, data is processed in batch, streaming, or a mix of the two.
Use cases
Hop is used for, among many other things:
-
Populating and maintaining data warehouses, with built-in support for Slowly Changing Dimensions (SCD), Change Data Capture (CDC) and surrogate keys.
-
Loading large data sets into databases, data warehouses and cloud object storage.
-
Building and maintaining lakehouse tables on Delta Lake and Apache Iceberg.
-
Integrating across diverse architectures: relational databases, files, object stores, NoSQL and graph databases such as MongoDB, Cassandra and Neo4j, REST and SOAP services, and SaaS platforms.
-
Processing streaming data, for example from Apache Kafka, on its own or alongside batch.
-
Migrating data between databases and applications.
-
Data profiling, validation and cleansing.