Welcome in 2021, happy new year!
While the world has seen its share of problems in 2020, it’s been a roller coaster for Hop!
Our community continued to work tirelessly in December, even through the holiday period.
Let’s have a look at what happened in this last burst of productivity for 2020. With this pace of development, 2021 will be a great year for Hop!
A special shout out to Nicolas Adment, who was on an absolute roll in December!
Apache Hop Incubation - 0.50 release preparation
We had two voting rounds for 0.50 release candidates: 0.50-rc1 and 0.50-rc2.
We were told in advance that getting used to "the Apache Way" would take a while. Because of some lingering legacy problems (code, dependency licensing etc), we didn’t reach the required total of 3 positive votes to release 0.50-rc2 as Apache Hop (Incubating) 0.50.
Some of the issues mentioned were already resolved in the code after the 0.50 release branch, and development moved on in preparation for the 0.60 release. We therefore won’t be building a 0.50-rc3 and will move on to 0.60-rc1.
Check the mailing list archives for more information:
UI Updates
Hop Gui continues to evolve. A couple of the changes we made in December:
Single Click UI Actions
Since eliminating right click and double actions improves developer productivity and helps us to move towards a unified user interface on all devices and platforms, we increased focus on single-click actions in the user interface.
New Icons
Nicolas worked hard to update the icons in Hop Gui. A couple of areas where Hop Gui was given some icon love:
-
new workflow and pipeline icons
-
new toolbar and menu icons
-
new and/or updated icons for a number of actions and transforms
The result is a Hop Gui that looks and feels lighter and cleaner:
New Metadata Perspective
A lot of metadata operations (relational, database connections, runtime configurations etc) had their own pop-up configuration dialogs.
These various pop-ups were grouped in one central metadata configuration perpective.
Export to SVG
A neat little feature was introduced to Hop Gui: export to SVG.
Although not spectacular in itself, this little menu item gives you a full svg (scalable) export of your workflow or pipeline graph. For Hop, this is a first step towards self documenting workflows and pipelines.
New Plugins
Two new transform plugins are now available in Hop:
-
Transforms:
Integration tests
Since nothing works as well as eating your own dog food, Hop now has an integration testing framework that is included in our daily build.
The purpose of the integration testing framework is as simple as useful: code tests can only test a number of cases. Hop was designed and built to be easy. Easy to use, easy to configure, and now easy to test.
Tests are built as workflows and pipelines. The success or failure of a test workflow is determined through the abort or success action exit code.
There is no doubt about the usefulness of this framework: since its introduction (mid-December), the Hop developers have found and fixed a number of bugs that may have been in the code for over a decade!
Partitioned Stream Lookup
The Stream Lookup transform now supports partitioning.
This significantly helps to improve lookups on large volumes of data on large machines.
Docker, Kubernetes updates
The separate Dockerfile and configuration were moved to the main Hop repository.
A Hop Docker image is now one docker pull away through Docker Hub.
Work on Kubernetes continues in the separate repository.
Jandex
All plugins in Hop now use Jandex.
Jandex uses annotations to find and load all plugins at startup. This reduces the startup times for Hop Gui, Hop Run and Hop Server.
Documentation
Lots of existing pages have been extended and/or updated. A selection of the new documentation we have available:
As always, ping us on mattermost to discuss documentation, and create tickets if you find any missing or incorrect information in the docs. Documentation is code, we take bugs in the documentation just as serious as bugs in the code.
Community
The Hop community continues to grow:
The following people have been added as committers:
-
Rodrigo Haces (late arrival, not included in November update)
-
Jason Chu (late arrival, not included in November update)
-
Yannick Mols
Check out the complete list of committers and contributors.
Without community contribution, Hop is just a coding club! Please feel free to join, participate in the discussion, test, file bug tickets on the software or documentation, … Contributing is a lot more than writing code.
Check out our contribution guides to find out more.
Various
Hop Web
Hop Web has seen some impressive activity from Hiromu over the last months.
The code is in the final stages of being merged into Hop. Once that’s done, you’ll be able to choose to run Hop Gui from a fat (desktop) or web client. Both will be started from the same code base.
Needless to say this will open an entire new world of possible use cases for Hop.
JIRA Tickets
The full list of issues that had activity over the last month is:
Resolved (66)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
Not found from Search option | Documentation | 2020-11-06 | 2020-12-16 | |
The HopGui metadata mangers are not refreshed | GUI | 2020-12-02 | 2020-12-16 | |
The Pipeline action dialog doesn’t remember its size | GUI | 2020-12-03 | 2020-12-04 | |
Workflows: when not pasting XML see if other content can be recognized | GUI | 2020-12-03 | 2020-12-04 | |
creating a new environment with a new name is not possible | GUI | 2020-12-03 | 2020-12-04 | |
Trying to create a fat jar in a project home folder fails | Beam | 2020-12-04 | 2020-12-04 | |
Create a GUI events handler | GUI | 2020-12-04 | 2020-12-04 | |
Separate ActionSpecial into 2 actions START and DUMMY | Actions | 2020-12-04 | 2020-12-13 | |
Metadata perspective: code cleanup | GUI | 2020-12-04 | 2020-12-15 | |
Partition schema editor: doesn’t pick up number of partitions | GUI | 2020-12-04 | 2020-12-05 | |
Beam generating unbounded synthetic data too fast | Beam | 2020-12-05 | 2020-12-05 | |
Beam complaining about slf4j | Beam | 2020-12-05 | 2020-12-16 | |
Avoid state of metadata by not implementing IVariables | API | 2020-12-06 | 2020-12-16 | |
Add Github README with badges | 2020-12-06 | 2020-12-06 | ||
Remove duplicate Hop jar files | Build | 2020-12-06 | 2020-12-16 | |
Declare all plugins with only annotations without penalizing startup time | 2020-12-06 | 2020-12-11 | ||
Error opening files | GUI | 2020-12-07 | 2020-12-15 | |
Options dialog has i18n issues | GUI | 2020-12-07 | 2020-12-08 | |
Add log channel to the JavaScript action | Actions | 2020-12-08 | 2020-12-11 | |
Pipeline and Workflow action dialog: change name after selecting file | Actions, GUI | 2020-12-08 | 2020-12-11 | |
De-select objects on single canvas click | GUI | 2020-12-09 | 2020-12-18 | |
The @HopMetadata annotation must be consistent with the other plugins | API | 2020-12-09 | 2020-12-14 | |
Remove unused annotation EnginePlugin | 2020-12-10 | 2020-12-11 | ||
Images not found for Dummy and Start actions | 2020-12-11 | 2020-12-11 | ||
Define an order of perspective | GUI | 2020-12-11 | 2020-12-12 | |
Output of Sort Rows is suspect | Transforms | 2020-12-11 | 2020-12-11 | |
Set Variables transform throws an error when missing workflow | Transforms | 2020-12-11 | 2020-12-11 | |
Internal Hop variables not set on pipelines and workflows | Pipelines, Workflows | 2020-12-11 | 2020-12-16 | |
Various issues while executing Beam pipelines | Pipelines | 2020-12-11 | 2020-12-12 | |
Move the plugin image to its folder | 2020-12-12 | 2020-12-16 | ||
Improve the sort order of the variables shown with CTRL-SPACE | GUI | 2020-12-12 | 2020-12-12 | |
HopVfsFileDialog is unusable under windows | 2020-12-12 | 2020-12-14 | ||
Port Token Replacement Plugin from Chris Deptula | Transforms | 2020-12-13 | 2020-12-22 | |
Search: when finding metadata objects you can’t open them | GUI | 2020-12-14 | 2020-12-16 | |
Add icon to each type of Hop file | GUI | 2020-12-14 | 2020-12-14 | |
Stream Lookup: allow it to be run partitioned | Transforms | 2020-12-14 | 2020-12-16 | |
remove duplicate 'download' from download link | Website | 2020-12-15 | 2020-12-15 | |
Widget disposed in parameter tabs on Executor transforms | GUI | 2020-12-15 | 2020-12-16 | |
Add support for TLS 1.2 | 2020-12-15 | 2020-12-15 | ||
Exasol dependency error | 2020-12-16 | 2020-12-19 | ||
Add Sequence delivers duplicate ids when running in copies | Transforms | 2020-12-16 | 2020-12-17 | |
Running integration tests locally is dangerous | Integration Testing | 2020-12-17 | 2020-12-17 | |
Create an export to SVG menu options in Hop GUI | GUI | 2020-12-18 | 2020-12-18 | |
Execute row SQL script broken | 2020-12-18 | 2020-12-18 | ||
Add database to test container | Integration Testing | 2020-12-18 | 2020-12-27 | |
top bar menu disappears when using any item in documentation menu | Website | 2020-12-18 | 2020-12-19 | |
With double-click support disabled, allow editing with a single click | GUI | 2020-12-19 | 2020-12-21 | |
Options dialog: mangled option | GUI | 2020-12-19 | 2020-12-21 | |
Add the Apache License header to all .properties files | 2020-12-20 | 2020-12-21 | ||
Add the Apache License header to all shell scripts | CLI | 2020-12-20 | 2020-12-21 | |
Add the Apache License header to all XML files | Build | 2020-12-20 | 2020-12-21 | |
Add right padding in Project dialog | 2020-12-21 | 2020-12-23 | ||
Add right padding in Environment dialog | 2020-12-21 | 2020-12-23 | ||
Environment variables not properly managed | 2020-12-21 | 2020-12-22 | ||
Review Token Replacemen plugin icon | 2020-12-21 | 2020-12-29 | ||
Selection mode entered inadvertently | GUI | 2020-12-21 | 2020-12-21 | |
After selecting a transform in the action dialog with CTRL you get a NPE | GUI | 2020-12-21 | 2020-12-21 | |
No longer able to create a hop in a workflow | GUI | 2020-12-21 | 2020-12-23 | |
Remove mulesoft redshift jdbc dependency | Build | 2020-12-23 | 2020-12-23 | |
Repair image for ValueMeta | 2020-12-23 | 2020-12-25 | ||
Rat should print filenames to console | Build | 2020-12-26 | 2020-12-26 | |
Add a function to the GuiResource class to get image from IValueMeta | GUI | 2020-12-26 | 2020-12-28 | |
Memory Group By - Count - outputs counts with padded zeros. | 2020-12-27 | 2021-01-02 | ||
Excel Input transform with XLSX Streaming don’t work | Transforms | 2020-12-28 | 2020-12-29 | |
Hop About Box opens too small. | GUI | 2020-12-29 | 2020-12-30 | |
Ctrl-+ doesn’t change the zoom level | 2020-12-29 | 2021-01-02 |
In Progress (15)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
Reverse sort the projects drop-down list by use date | GUI | 2020-11-30 | 2020-12-04 | |
Clean GuiResource images | 2020-12-08 | 2020-12-29 | ||
Update kettle to hop plugin porting documentation | API | 2020-12-10 | 2020-12-15 | |
Move hop-docker into ASF HOP project and make docker image available in dockerhub via pipeline | Build, Containers, Infrastructure, Pipelines | 2020-12-11 | 2020-12-29 | |
Add support to Clickhouse database | Database | 2020-12-16 | 2021-01-03 | |
Enable pritty print on JSON output | Transforms | 2020-12-16 | 2021-01-03 | |
Enable pritty print on XML output | Transforms | 2020-12-16 | 2021-01-03 | |
NPE in User Defined Java Class code crashes Hop GUI | GUI | 2020-12-18 | 2020-12-18 | |
add release process information to contribution page | Website | 2020-12-20 | 2020-12-21 | |
Review of italian translation files | Translations | 2020-12-21 | 2020-12-22 | |
Improve integration testing documentation | Documentation, Integration Testing | 2020-12-21 | 2020-12-21 | |
Single click mode: moving a transform or action can cause edit | GUI | 2020-12-21 | 2020-12-21 | |
cleanup integration tests | Integration Testing | 2020-12-27 | 2020-12-27 | |
Remove jxl dependency from hop-transform-excelinput | Transforms | 2020-12-28 | 2020-12-31 | |
Transform ExcelWriter upgrade POI 4.1.2 and remove unnecessary dependencies | Transforms | 2020-12-28 | 2020-12-29 |
Open (40)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
macOS dark mode results in illegible UI text | GUI | 2020-10-14 | 2020-12-15 | |
Need a way to clear a setting in Meta Data Injection Transform | Transforms | 2020-12-02 | 2020-12-02 | |
Pipelines: when pasting allow other content then XML to be recognized | GUI | 2020-12-03 | 2020-12-03 | |
Create page with branding/sample files | Documentation, Website | 2020-12-03 | 2020-12-03 | |
Add a bit of left padding on the Action Dialog | GUI | 2020-12-03 | 2020-12-03 | |
create usage documentation + examples for the various runtime configurations | Documentation, GUI | 2020-12-04 | 2020-12-04 | |
Update Jetty Dependencies | API, Hop Server | 2020-12-04 | 2020-12-04 | |
Add a warning to indicate the consequences of renaming metadata objects | GUI | 2020-12-04 | 2020-12-16 | |
Native core plugins a registered twice | 2020-12-04 | 2020-12-05 | ||
Add support for a HOP_PLUGIN_FOLDERS variable | API | 2020-12-04 | 2020-12-04 | |
Join rows (cartesian products) - Fields should be editable | GUI | 2020-12-04 | 2020-12-04 | |
Allow results to be evaluated in an action | Actions | 2020-12-05 | 2020-12-05 | |
Allow metadata to be read from other projects | Metadata | 2020-12-06 | 2020-12-16 | |
Integration Tests: Add ability to pass parameters to hop-run.sh | Integration Testing | 2020-12-06 | 2020-12-06 | |
Add option to stop pipeline or workflow after some time | Pipelines, Workflows | 2020-12-07 | 2020-12-07 | |
NPE editing Workflow Executor transform | GUI | 2020-12-08 | 2020-12-08 | |
Complete doc help for Start and Dummy actions | Documentation | 2020-12-11 | 2020-12-12 | |
update incubator-hop-docs README, minor cleanups | Documentation | 2020-12-12 | 2020-12-16 | |
GraphQL Input, Output, Update, Delete | 2020-12-14 | 2020-12-14 | ||
Allow unit tests to run on the Beam Direct pipeline execution engine | Pipelines | 2020-12-15 | 2020-12-15 | |
Register project with Hop Server | 2020-12-16 | 2020-12-16 | ||
document Hop server rest api | Documentation, Hop Server | 2020-12-17 | 2020-12-17 | |
Exporting to SVG causes errors | GUI | 2020-12-19 | 2020-12-28 | |
Create documentation for Token Replacement plugin | Documentation | 2020-12-20 | 2020-12-21 | |
Create unit tests for Token Replacement plugin | Integration Testing | 2020-12-20 | 2020-12-21 | |
Remove the old samples in archive-samples | Build | 2020-12-21 | 2020-12-21 | |
Remove the archive-pipeline-transforms folder | Build | 2020-12-21 | 2020-12-21 | |
Use keyboard shortcuts to create transforms and actions in hop gui | 2020-12-21 | 2020-12-21 | ||
generic database connections require properties to be filled | Database, GUI | 2020-12-22 | 2020-12-23 | |
MultiMarkdown support for Notes | 2020-12-22 | 2020-12-22 | ||
The active Hop password encoder plugin is not passed along to the metadata providers | API | 2020-12-22 | 2020-12-22 | |
Dragging a hpl or hwf in the application should open it | GUI | 2020-12-23 | 2020-12-23 | |
Replace the use of StyledText with Text | Web Hop | 2020-12-26 | 2020-12-27 | |
add progress dialog while importing files | GUI, Import | 2020-12-26 | 2020-12-26 | |
Export to SVG menu should de disabled when no file is active | Transforms | 2020-12-26 | 2020-12-28 | |
Wrap Canvas in ScrolledComposite at ContextDialog | Web Hop | 2020-12-28 | 2020-12-28 | |
Action selection dialog not properly positioned on screen | 2020-12-29 | 2020-12-29 | ||
Change variable name HOP_PIPELINE_PAN_JVM_EXIT_CODE | 2020-12-29 | 2020-12-29 | ||
Create a load-balancing row distribution plugin | Pipelines | 2020-12-29 | 2020-12-29 | |
Use SWT’s ToolTip instead of JFace’s (Default)ToolTip | Web Hop | 2020-12-30 | 2020-12-30 |
Closed (8)
| Issue | Summary | Components | Created | Updated |
|---|---|---|---|---|
checkboxes don’twork | 2020-10-04 | 2020-12-04 | ||
Git Repository option is missing | Documentation | 2020-11-06 | 2020-12-16 | |
Integration Tests: Add ability to have inverted (true negative) tests | 2020-12-06 | 2020-12-10 | ||
GetPipelineImageServlet and GetWorkflowImageServlet don’t work | Hop Server | 2020-12-17 | 2020-12-28 | |
GuiContextUtil shares session-unique instances among sessions | Web Hop | 2020-12-27 | 2021-01-02 | |
Do no use CTabFolder.setSimple() | Web Hop | 2020-12-30 | 2021-01-02 | |
RAP only allows painting on the Canvas widget | Web Hop | 2020-12-31 | 2021-01-02 | |
Browser in RAP does not implement back() and forward() | Web Hop | 2020-12-31 | 2021-01-02 |