Advanced project and environment management

Configuration files

Global Project Configuration

The 2 main things that define a project are its name and its home folder. Projects and environments as such are defined in the central Hop configuration file hop-config.json. By default this file lives in the config/ folder of your Hop client distribution. You can change that folder by setting the HOP_CONFIG_FOLDER environment variable on your system.

projectsConfig

In hop-config.json, you’ll find a "projectsConfig" section. By default it contains the following:

json

{
  "projectsConfig" : {
    "enabled" : true,
    "projectMandatory" : true,
    "environmentMandatory" : false,
    "defaultProject" : "default",
    "defaultEnvironment" : null,
    "standardParentProject" : "default",
    "standardProjectsFolder" : null,
    "projectConfigurations" : [ {
      "projectName" : "default",
      "projectHome" : "config/projects/default",
      "configFilename" : "project-config.json"
    }, {
      "projectName" : "samples",
      "projectHome" : "config/projects/samples",
      "configFilename" : "project-config.json"
    } ],
    "lifecycleEnvironments" : [ ],
    "projectLifecycles" : [ ]
  }
}

As you can see the standard Hop client distribution defines 2 projects: default and samples.

Project Configuration

Every project has extra metadata and settings stored in a project configuration file called project-config.json. For the samples project this would be config/projects/samples/project-config.json.

project-config.json

Let’s take a look at it:

json

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "parentProjectName" : "default",
  "parentProjectFolders" : [ {
    "folder" : "templates",
    "copyOnce" : true,
    "copyOnEnable" : false,
    "overwrite" : false,
    "exclusionWildcard" : ".*\\.tmp"
  } ],
  "config" : {
    "variables" : [ ]
  }
}

Variables

You can define variables on a project level as well. This makes it handy to reference things like input and output folders which are not sensitive to being checked into version control.

Hop also sets a few built-in project variables when a project is activated:

Table 1. Built-in project variables
Variable	Description
`PROJECT_HOME`	Home folder of the active project
`HOP_PROJECT_NAME`	Name of the active project
`PARENT_PROJECT_HOME`	Home folder of the immediate parent project (empty if none)
`PARENT_PROJECT_NAME`	Name of the immediate parent project (empty if none)

Use the parent variables to reference shared assets without fragile relative paths, for example:

${PARENT_PROJECT_HOME}/SHARED_PIPELINES/example.hpl

Parent projects

As you can see from the project configuration file (parentProjectName), a project can have a parent from which it will inherit all the metadata objects as well as all the variables that are defined in it. When a parent is configured, Hop also sets ${PARENT_PROJECT_HOME} and ${PARENT_PROJECT_NAME} so pipelines and environment config files can point into the parent project home (for example shared pipelines or secrets kept outside the child repository).

Optional parentProjectFolders entries copy files from those parent folders into the child project home. Each entry can copy once (when the destination is empty), copy every time the project is enabled, overwrite existing files, and skip files with a Java regular expression wildcard. The child’s project-config.json and .git directories are never overwritten. See Projects and Environments for the Project Properties dialog.

Environment configuration

Hop enviroments and their home folders are stored in the hop configuration file 'hop-config.json'. That file lives by default in the config folder of the Hop installation. System property HOP_CONFIG_FOLDER can also be used to point to a different folder

environmentConfig

json

{
 "environmentConfig" : {
  "enabled" : true,
  "openingLastEnvironmentAtStartup" : true,
  "environmentConfigFilename" : "environment.json",
  "environmentFolders" : {
   "Project 1 - DEV" : "/projects/one/dev/",
   "Project 1 - UAT" : "/projects/one/uat/",
   "Project 1 - PRD" : "/projects/one/prd/",
   "Project 2 - DEV" : "/projects/two/dev/",
   "Project 2 - UAT" : "/projects/two/uat/",
   "Project 2 - PRD" : "/projects/two/prd/",
  }
}

Use a project without hop-config.json

You can run or open a project for a single process without adding it to hop-config.json. Hop registers the project in memory, points metadata at ${PROJECT_HOME}/metadata, and does not write the registration back to disk.

This is useful when the project is provided by another system (a checkout, a zip, a container mount) and you do not want to mutate the Hop configuration.

bash

hop --project-locations my-project=/path/to/project run -f pipeline.hpl -r local
hop --project-locations my-project=/path/to/project gui -j my-project -f pipeline.hpl

--project-locations (short -pl) accepts a comma-separated list of:

name=/path/to/folder
name=/path/to/archive.zip or name=/path/to/archive.jar (opened read-only through VFS)
name=/path/to/folder:custom-config.json when the project file is not project-config.json
a bare folder or archive: if the folder itself is a project (project-config.json or hop-project.config), it is registered under the folder name. If it is a parent of several project folders, each child with a project config file is registered.

When more than one project is listed or discovered, Hop enables the leaf project (a child that references a parent). Parent metadata is still inherited.

A zip export may also contain metadata.json and variables.json. Those files are loaded only for archive or JSON-export layouts (no metadata/ folder), not for a normal project that happens to contain them.

Optional environment files for that process:

bash

hop --project-locations edw=/data/edw --environments edw-prod=/data/edw-prod.json \
    run -e edw-prod -f load.hpl -r local

--environments uses name=[project:]file1;file2. If the project name is omitted, Hop infers it from the registered projects. --environment-conf-files (also --environment-config-files) adds extra variable files to the enabled project or environment.

-ime / --in-memory keeps Hop configuration in memory even when you are not using --project-locations. The same behaviour can be enabled with the ${HOP_CONFIG_IN_MEMORY} system variable set to Y.

These options are available on the hop root command (so they apply before run, gui, search, and so on) and on the subcommands themselves (hop run -pl …, hop gui -pl …).

See hop, hop run and hop gui for the full option lists.

Command Line Project Configuration

In addition to the Hop Gui and configuration files, all aspects of and operations on projects and environments can be managed through the Hop Conf command line tool.

Configuration on the command line

The hop-conf script offers many options to edit environment definitions.

Creating an environment

Windows
Linux, macOS

bash

hop-conf.bat --environment-create \
             --environment hop2 \
             --environment-project hop2
             --environment-purpose=Development \
             --environment-config-files="C:\<YOUR_ENV_FILE_PATH>\env-variables.json"

Expected output:

shell

C:\<YOUR_PATH>\hop>echo off
===[Environment Settings - hop-conf.bat]===================================
Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows -DHOP_PLATFORM_RUNTIME=Conf
-DHOP_AUTO_CREATE_CONFIG=Y
Command to start Hop will be:
"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath lib\core\*;lib\beam\*;lib\swt\win64\*
-Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y org.apache.hop.config.HopConfig  --environment-create
--environment hop2 --environment-project hop2 --environment-purpose Development
--environment-config-files "C:\<YOUR_ENV_FILE_PATH\env-variables.json"
===[Starting HopConfig]=========================================================
Creating environment 'hop2'
Environment 'hop2' was created in Hop configuration file C:\<YOUR_PATH>\hop\config\hop-config.json
Warning: referenced project 'hop2' doesn\'t exist
Found existing environment configuration file: C:\<YOUR_ENV_FILE_PATH>\variables.json
Purpose: Development
Project name: hop2
Config file: C:\<YOUR_ENV_FILE_PATH>\env-variables.json

bash

$ sh hop-conf.sh \
     --environment-create \
     --environment hop2 \
     --environment-project hop2 \
     --environment-purpose=Development \
     --environment-config-files=<YOUR_ENV_FILE_PATH>/env-variables.json

Expected output:

shell

Creating environment 'hop2'
Environment 'hop2' was created in Hop configuration file <YOUR_PATH>/hop/config/hop-config.json
Warning: referenced project 'hop2' doesn't exist
Found existing environment configuration file: <YOUR_ENV_FILE_PATH>/env-variables.json
  hop2
    Purpose: Development
    Project name: hop2
      Config file: <YOUR_ENV_FILE_PATH>/env-variables.json

As you can see from the log, an empty file was created to set variables in:

json

{ }

Setting variables in an environment

This command adds a variable to the environment configuration file:

Windows
Linux, macOS

shell

hop-conf.bat --config-file "C:\<YOUR_ENV_FILE_PATH>\env-variables.json" --config-file-set-variables "DB_HOSTNAME=localhost,DB_PASSWORD=abcd"

Expected output:

shell

C:\<YOUR_PATH\hop>echo off
===[Environment Settings - hop-conf.bat]===================================
Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows -DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
Command to start Hop will be:
"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath lib\core\*;lib\beam\*;lib\swt\win64\*
-Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y org.apache.hop.config.HopConfig
--config-file "C:\<YOUR_ENV_FILE_PATH>\env-variables.json"
--config-file-set-variables "DB_HOSTNAME=localhost,DB_PASSWORD=abcd"
===[Starting HopConfig]=========================================================
Configuration file 'C:\<YOUR_ENV_FILE_PATH>/env-variables.json' was modified.

bash

./hop-conf.sh --config-file <YOUR_ENV_FILE_PATH>/env-variables.json --config-file-set-variables DB_HOSTNAME=localhost,DB_PASSWORD=abcd

Expected output:

shell

Configuration file '<YOUR_ENV_FILE_PATH>/env-variables.json' was modified.

If you look at the file env-variables.json, you’ll see that the variables were added:

json

{
  "variables" : [ {
    "name" : "DB_HOSTNAME",
    "value" : "localhost",
    "description" : ""
  }, {
    "name" : "DB_PASSWORD",
    "value" : "abcd",
    "description" : ""
  } ]
}

Please note that you can add descriptions for the variables as well with the --describe-variable option. Please run hop-conf without options to see all the possibilities.

Deleting an environment

The following example deletes an environment from the Hop configuration file:

Windows
Linux, macOS

shell

hop-conf.bat -ed --environment hop2

Expected output:

shell

C:\<YOUR_PATH>\hop>echo off
===[Environment Settings - hop-conf.bat]===================================
Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows -DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
Command to start Hop will be:
"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath lib\core\*;lib\beam\*;lib\swt\win64\* -Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows -DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y org.apache.hop.config.HopConfig  -ed --environment hop2
===[Starting HopConfig]=========================================================
Lifecycle environment 'hop2' was deleted from Hop configuration file C:\<YOUR_PATH>\hop\config\hop-config.json

shell

./hop-conf.sh -ed --environment hop2

Expected output:

shell

Lifecycle environment 'hop2' was deleted from Hop configuration file <YOUR_PATH>/hop/config/hop-config.json

Projects Plugin configuration

There are various options to configure the behavior of the Projects plugin itself. In Hop configuration file hop-config.json we can find the following options:

json

{
    "projectMandatory" : true,
    "environmentMandatory" : false,
    "defaultProject" : "default",
    "defaultEnvironment" : null,
    "standardParentProject" : "default",
    "standardProjectsFolder" : "/home/matt/test-stuff/"
}

Option Description hop-conf option

Option	Description	hop-conf option
projectMandatory	This will prevent anyone from using hop-run without specifying a project	`--project-mandatory`
environmentMandatory	This will prevent anyone from using hop-run without specifying an environment	`--environment-mandatory`
defaultProject	The default project to use when none is specified	`--default-project`
defaultEnvironment	The default environment to use when none is specified	`--default-environment`
standardParentProject	The standard parent project to propose when creating new project	`--standard-parent-project`
standardProjectsFolder	The folder to which you’ll browse by default in the GUI when creating new projects	`--standard-projects-folder`

projectMandatory

This will prevent anyone from using hop-run without specifying a project

--project-mandatory

environmentMandatory

This will prevent anyone from using hop-run without specifying an environment

--environment-mandatory

defaultProject

The default project to use when none is specified

--default-project

defaultEnvironment

The default environment to use when none is specified

--default-environment

standardParentProject

The standard parent project to propose when creating new project

--standard-parent-project

standardProjectsFolder

The folder to which you’ll browse by default in the GUI when creating new projects

--standard-projects-folder