As always at the start of another new month, let’s have a look at what happened at Hop over the last month. Here’s the recap for February 2021!
Metadata Injection
Metadata injection was added to Hop months ago. As with most of the functionality that was ported from PDI/Kettle, some work was required to lift the code to the new flexibility and quality standards we’ve set for Hop. After we’ve given it some love, Hop Metadata Injection now has:
-
UI improvements:
-
Fixed filter button
-
Added source-to-target (input-to-template) mapping button
-
Removed unneeded top level in the tree
-
Added a "None" button when user clicks on a target key (was missing)
-
-
reimplemented support for additional transforms, among others CSV File Input, Group By.
Dark Mode
"If there’s no dark mode, it doesn’t exist."
It took some tweaks to get the Hop dark mode to work correctly on all major platforms (looking at you, Mac OS!!). Since it appeared in mid-February, all Hop developers seem to have switched to dark mode overnight.
Beam Spark runner improvements
The ability to run Hop pipelines on Apache Spark over Apache Beam has been available since the very early Hop days.
It is now a lot easier to run pipelines on Apache Spark and Apache Flink.
Check out the updated docs to take this for a spin.
New Metadata Types
A number of new metadata types were introduced. These are all brand new, with lots of potential use cases!
-
Webservice: expose data from a pipeline as a webservice
-
Pipeline Log and Workflow Log: send pipeline and workflow logging data to a probe to be picked up in a transform. The pipeline holding that transform can do whatever is possible in Hop: write to files, send to a (graph) database, Kafka producer, …
-
Pipeline Probe: similar to the logging probe, but for actual data. The processing pipeline in this case can be used for data quality, lineage and any imaginable use case.
New Plugins
The list of new plugins in Hop continues to grow. Here are the new additions for February:
-
Transforms:
Various
Docker improvements
A number of small issues have been fixed and improvements have been made. As the number of Docker use cases continues to grow, Hop in Docker has become stable and mature.
Export project to zip
Hop Web
The merge of Hop Web is getting closer. We’ve closed a number of remaining issues, a war file is now included as part of the default Hop build. Expect a fully integrated Hop Web in the default Hop build soon!
Migration updates
A number of bug fixes, tweaks and improvements have been made to support migration for additional PDI/Kettle steps and job entries to Hop. It is now easy and safe to migrate your PDI/Kettle projects to Hop!
Ordered List
In some cases (e.g. the new Coalesce plugin), the order of fields is important. For these use cases, we now have an Ordered List:
Community
The Hop community continues to grow:
Check out the complete list of committers and contributors.
Without community contribution, Hop is just a coding club! Please feel free to join, participate in the discussion, test, file bug tickets on the software or documentation, … Contributing is a lot more than writing code.
Check out our contribution guides to find out more.
JIRA Tickets
The full list of issues that had activity over the last month is:
Resolved| (72)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
Variable from environment config only work after restart | GUI | 2020-10-04 | 2021-02-28 | |
port sample rows step to transform | Transforms | 2020-10-04 | 2021-02-28 | |
"Hop needs a default ""Project"" and ""Local Run Configuration""" | 2020-10-04 | 2021-02-05 | ||
Improve first-time usage experience | GUI | 2020-11-12 | 2021-02-05 | |
change selected text color | Website | 2021-01-07 | 2021-02-03 | |
attach POM to the correct phase | 2021-01-25 | 2021-02-28 | ||
Add UDJE (User Defined Java Expression) to ETL Metadata Injection step | 2021-01-28 | 2021-02-05 | ||
Options Dialog : ConfigPlugins are no longer showing up | GUI | 2021-01-30 | 2021-02-03 | |
Remove IPluginFolder | 2021-01-30 | 2021-02-01 | ||
fix typos in website | Website | 2021-01-31 | 2021-02-02 | |
README for docker image has wrong run instructions | 2021-02-01 | 2021-02-02 | ||
Docker container references wrong deployment folder | 2021-02-01 | 2021-02-03 | ||
Port MonetDB Bulk Loader to Hop | 2021-02-02 | 2021-02-27 | ||
Bump Apache commons-collections version 3.2.2 to commons-collections4 version 4.4 | 2021-02-02 | 2021-02-03 | ||
NPE when loading old project configurations | GUI | 2021-02-03 | 2021-02-04 | |
Code and logging references steps i.s.o. transforms | API | 2021-02-03 | 2021-02-04 | |
Saving a pipeline or workflow before running doesn’t work | GUI | 2021-02-03 | 2021-02-04 | |
CSV File Input: add metadata injection support | Transforms | 2021-02-05 | 2021-02-06 | |
Show last rows widget is showing incorrect data | GUI, Transforms | 2021-02-05 | 2021-02-28 | |
ETL Metadata Injection: variable and parameter passing issues | Transforms | 2021-02-05 | 2021-02-06 | |
Add toolbar icons to edit pipeline and workflow properties | GUI | 2021-02-05 | 2021-02-16 | |
Preview does not work on transforms using variables | 2021-02-05 | 2021-02-28 | ||
Metadata Injection: Logging levels are not propagating | Transforms | 2021-02-06 | 2021-02-08 | |
Group By : add support for metadata injection | Transforms | 2021-02-06 | 2021-02-08 | |
Port the transform Coalesce plugin | Transforms | 2021-02-07 | 2021-02-09 | |
Add EnterOrderedListDialog to select in an order the elements of a list | GUI | 2021-02-07 | 2021-02-08 | |
merge join input steps are not correctly converted to transforms | Import | 2021-02-10 | 2021-02-23 | |
BlockingTransform type not correctly converted | Import | 2021-02-10 | 2021-02-23 | |
remove references to Project Hop | Documentation | 2021-02-10 | 2021-02-23 | |
"Environment dialog: Add a ""Create…"" button to create a new config file" | GUI | 2021-02-11 | 2021-02-13 | |
Add keyboard shortcut Ctrl+Shift+S to save as menu | 2021-02-11 | 2021-02-11 | ||
full connection element not removed from workflow XML | Import | 2021-02-12 | 2021-02-23 | |
EnterMappingDialog : improve guessing and sort mapped values | GUI | 2021-02-13 | 2021-02-15 | |
Meta data injection dialog: various bugs | GUI | 2021-02-13 | 2021-02-15 | |
Integration tests: optionally generate surefire reports | Infrastructure | 2021-02-15 | 2021-02-15 | |
Hop Run: variables in environment or project are not applied to top-level parameters | Hop Run | 2021-02-15 | 2021-02-16 | |
Workflow Executor: NPE in dialog | GUI, Transforms, Workflows | 2021-02-15 | 2021-02-16 | |
Port the SAS Input step to Hop | Transforms | 2021-02-17 | 2021-02-19 | |
Long lived docker container throws NPE | docker, Hop Server | 2021-02-17 | 2021-02-18 | |
Remove inkscape metadata from SVG files | GUI | 2021-02-18 | 2021-02-19 | |
"Add ""View output"" as a context action option" | Transforms | 2021-02-19 | 2021-02-22 | |
JSON Input dialog: get fields shows white text on white background | Transforms | 2021-02-19 | 2021-02-20 | |
"Kettle Import: change pipeline run configuration to ""local""" | Import | 2021-02-22 | 2021-02-23 | |
Kettle Import: change JobExecutor to WorkflowExecutor | Import | 2021-02-22 | 2021-02-23 | |
${Internal.Entry.Current.Folder} not working | API, Import | 2021-02-22 | 2021-02-23 | |
Workflow Executor: assume 1 row grouping if all grouping fields are blank | Transforms | 2021-02-22 | 2021-02-23 | |
Metadata Perspective: sort by metadata type name | GUI | 2021-02-23 | 2021-02-24 | |
Group By transform loses aggregation type | Transforms | 2021-02-23 | 2021-02-25 | |
Docker: simplify hop-server use case | docker, Hop Server | 2021-02-23 | 2021-02-24 | |
ExcelInput accept_stepname not replaced to accept_transform_name | Transforms | 2021-02-24 | 2021-02-25 | |
Allow logging to take place with a Hop pipeline | API | 2021-02-24 | 2021-02-24 | |
Spark runner: missing dependency | API, Beam | 2021-02-24 | 2021-02-24 | |
Spark runner : not finishing application | API, Beam | 2021-02-24 | 2021-02-25 | |
Remote pipeline engine: improve logging | Hop Server | 2021-02-24 | 2021-02-25 | |
Hop Server: broken icons in web interface | Hop Server | 2021-02-24 | 2021-02-25 | |
Docker container: Missing fonts for running hop-server | docker, Hop Server | 2021-02-24 | 2021-02-25 | |
Docker container: Make port configurable | docker, Hop Server | 2021-02-24 | 2021-02-25 | |
Remove Mouse wheel support | Hop Web | 2021-02-24 | 2021-02-24 | |
TableView widget color on Windows are bad | GUI | 2021-02-24 | 2021-02-24 | |
Beam: add a main class for executing using Spark Submit or Flink Run | Beam | 2021-02-25 | 2021-02-25 | |
Docker: libraries like snappy need libc-compat in the container | docker | 2021-02-25 | 2021-02-25 | |
Allow output rows from any transform to be streamed to another pipeline | Transforms | 2021-02-25 | 2021-02-28 | |
Bring back space while mousing over transform | GUI, Transforms | 2021-02-26 | 2021-02-26 | |
Allow a metadata editor to be set to changed or not from outside the editor | API, GUI | 2021-02-26 | 2021-02-26 | |
Expose getter/setter for metadata in IMetadataEditor | API, GUI | 2021-02-26 | 2021-02-26 | |
TableView: when a row is inserted the view is not flagged as modified | API, GUI | 2021-02-26 | 2021-02-26 | |
Keyboard shortcuts are not handled correctly | API, GUI | 2021-02-26 | 2021-02-27 | |
JSON Input step not recognizing .json file extension | Transforms | 2021-02-26 | 2021-02-27 | |
Create a Web Service metadata type | Hop Server, Metadata | 2021-02-27 | 2021-02-27 | |
Deliver a working Hop Server XML configuration file | Hop Server | 2021-03-01 | 2021-03-01 | |
Workflow Logging transform outputs same field name twice | Transforms | 2021-03-01 | 2021-03-01 |
In Progress (7)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
Ensure that Hop complies with ASF encryption policy | 2021-01-27 | 2021-02-23 | ||
disable google analytics tracking when running website locally (draft mode) | Website | 2021-01-31 | 2021-01-31 | |
Add Hop Web Docker image to build process | Containers, Hop Web | 2021-02-14 | 2021-03-01 | |
Add support for a JSON value type | API | 2021-02-24 | 2021-02-24 | |
Create a file explorer perspective | GUI | 2021-02-26 | 2021-03-01 |
Open (29)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
When reloading project variables they are not refreshed | Metadata | 2021-01-17 | 2021-02-09 | |
NPE in file open dialog | GUI | 2021-01-27 | 2021-02-03 | |
There will be a NullPointerException if you only regist a pipeline | Hop Server | 2021-01-31 | 2021-01-31 | |
Translator does not work under Windows | 2021-01-31 | 2021-02-15 | ||
"Picking ""Create hop"" eventually brings up action picker again" | GUI | 2021-02-02 | 2021-02-02 | |
Unit tests UI issues | 2021-02-02 | 2021-02-03 | ||
Issue editing Data Set | GUI | 2021-02-03 | 2021-02-09 | |
Error creating Git metadata | GUI | 2021-02-03 | 2021-02-09 | |
Translator sort i18n messages on the key | 2021-02-03 | 2021-02-12 | ||
Trim option in database lookup | Transforms | 2021-02-05 | 2021-02-05 | |
Transform Fake data call private method domainName() | Transforms | 2021-02-07 | 2021-02-07 | |
Add integration tests for MQTT transforms | 2021-02-08 | 2021-02-08 | ||
Merge join description is incorrect | Documentation | 2021-02-08 | 2021-02-08 | |
add string operations to Metadata Injection | Transforms | 2021-02-12 | 2021-02-26 | |
document salesforce transforms | Documentation | 2021-02-12 | 2021-02-12 | |
Javascript Step does not output booleans | Transforms | 2021-02-12 | 2021-02-12 | |
document file access over VFS | Documentation | 2021-02-12 | 2021-02-12 | |
port Dropbox steps to Hop transforms | Transforms | 2021-02-12 | 2021-02-12 | |
Add Hop Web tests to nightly test process | Hop Web, Integration Testing | 2021-02-14 | 2021-02-14 | |
Add Injection Support to AddSequence Transform | 2021-02-15 | 2021-02-15 | ||
double click or modifier-click on an action or transform name without opening the dialog | GUI | 2021-02-19 | 2021-02-19 | |
add or extend tooltip to action and transform icons and text to | GUI | 2021-02-19 | 2021-02-19 | |
Update the Beam plugin to version 2.28.0 | API, Beam | 2021-02-23 | 2021-02-23 | |
Error while executing Pipeline using Beam Spark pipeline engine | Beam, Hop Run, Pipelines | 2021-02-24 | 2021-02-25 | |
Error while executing Pipeline using Beam Spark pipeline engine - USING HOP GUI | Beam, Pipelines | 2021-02-24 | 2021-02-25 | |