Advanced project and environment management

Configuration files

Global Project Configuration

The 2 main things that define a project are its name and its home folder. Projects and environments as such are defined in the central Hop configuration file hop-config.json. By default this file lives in the config/ folder of your Hop client distribution. You can change that folder by setting the HOP_CONFIG_FOLDER environment variable on your system.

projectsConfig

In hop-config.json, you’ll find a "projectsConfig" section. By default it contains the following:

json

{
  "projectsConfig" : {
    "enabled" : true,
    "projectMandatory" : true,
    "environmentMandatory" : false,
    "defaultProject" : "default",
    "defaultEnvironment" : null,
    "standardParentProject" : "default",
    "standardProjectsFolder" : null,
    "projectConfigurations" : [ {
      "projectName" : "default",
      "projectHome" : "config/projects/default",
      "configFilename" : "project-config.json"
    }, {
      "projectName" : "samples",
      "projectHome" : "config/projects/samples",
      "configFilename" : "project-config.json"
    } ],
    "lifecycleEnvironments" : [ ],
    "projectLifecycles" : [ ]
  }
}

As you can see the standard Hop client distribution defines 2 projects: default and samples.

Project Configuration

Every project has extra metadata and settings stored in a project configuration file called project-config.json. For the samples project this would be config/projects/samples/project-config.json.

project-config.json

Let’s take a look at it:

json

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "parentProjectName" : "default",
  "parentProjectFolders" : [ {
    "folder" : "templates",
    "copyOnce" : true,
    "copyOnEnable" : false,
    "overwrite" : false,
    "exclusionWildcard" : ".*\\.tmp"
  } ],
  "config" : {
    "variables" : [ ]
  }
}

projectId is optional. When it is absent, Hop does not filter execution information by project. A new project suggests the project name; an existing 2.19 configuration file is left unchanged.

Variables

You can define variables on a project level as well. This makes it handy to reference things like input and output folders which are not sensitive to being checked into version control.

Hop also sets a few built-in project variables when a project is activated:

Table 1. Built-in project variables
Variable	Description
`PROJECT_HOME`	Home folder of the active project
`HOP_PROJECT_NAME`	Name of the active project
`HOP_PROJECT_ID`	Project id from `project-config.json`. Empty when the project has no id, which leaves execution information unfiltered
`PARENT_PROJECT_HOME`	Home folder of the immediate parent project (empty if none)
`PARENT_PROJECT_NAME`	Name of the immediate parent project (empty if none)

Use the parent variables to reference shared assets without fragile relative paths, for example:

${PARENT_PROJECT_HOME}/SHARED_PIPELINES/example.hpl

Parent projects

As you can see from the project configuration file (parentProjectName), a project can have a parent from which it will inherit all the metadata objects as well as all the variables that are defined in it. When a parent is configured, Hop also sets ${PARENT_PROJECT_HOME} and ${PARENT_PROJECT_NAME} so pipelines and environment config files can point into the parent project home (for example shared pipelines or secrets kept outside the child repository).

Optional parentProjectFolders entries copy files from those parent folders into the child project home. Each entry can copy once (when the destination is empty), copy every time the project is enabled, overwrite existing files, and skip files with a Java regular expression wildcard. The child’s project-config.json and .git directories are never overwritten. See Projects and Environments for the Project Properties dialog.

Environment configuration

Hop environments and their home folders are stored in the hop configuration file 'hop-config.json'. That file lives by default in the config folder of the Hop installation. System property HOP_CONFIG_FOLDER can also be used to point to a different folder

environmentConfig

json

{
 "environmentConfig" : {
  "enabled" : true,
  "openingLastEnvironmentAtStartup" : true,
  "environmentConfigFilename" : "environment.json",
  "environmentFolders" : {
   "Project 1 - DEV" : "/projects/one/dev/",
   "Project 1 - UAT" : "/projects/one/uat/",
   "Project 1 - PRD" : "/projects/one/prd/",
   "Project 2 - DEV" : "/projects/two/dev/",
   "Project 2 - UAT" : "/projects/two/uat/",
   "Project 2 - PRD" : "/projects/two/prd/",
  }
}

Use a project without hop-config.json

You can run or open a project for a single process without adding it to hop-config.json. Hop registers the project in memory, points metadata at ${PROJECT_HOME}/metadata, and does not write the registration back to disk.

This is useful when the project is provided by another system (a checkout, a zip, a container mount) and you do not want to mutate the Hop configuration.

bash

hop --project-locations my-project=/path/to/project run -f pipeline.hpl -r local
hop --project-locations my-project=/path/to/project gui -j my-project -f pipeline.hpl

--project-locations (short -pl) accepts a comma-separated list of:

name=/path/to/folder
name=/path/to/archive.zip or name=/path/to/archive.jar (opened read-only through VFS)
name=/path/to/folder:custom-config.json when the project file is not project-config.json
a bare folder or archive: if the folder itself is a project (project-config.json or hop-project.config), it is registered under the folder name. If it is a parent of several project folders, each child with a project config file is registered.

When more than one project is listed or discovered, Hop enables the leaf project (a child that references a parent). Parent metadata is still inherited.

A zip export may also contain metadata.json and variables.json. Those files are loaded only for archive or JSON-export layouts (no metadata/ folder), not for a normal project that happens to contain them.

Optional environment files for that process:

bash

hop --project-locations edw=/data/edw --environments edw-prod=/data/edw-prod.json \
    run -e edw-prod -f load.hpl -r local

--environments uses name=[project:]file1;file2. If the project name is omitted, Hop infers it from the registered projects. --environment-conf-files (also --environment-config-files) adds extra variable files to the enabled project or environment.

-ime / --in-memory keeps Hop configuration in memory even when you are not using --project-locations. The same behaviour can be enabled with the ${HOP_CONFIG_IN_MEMORY} system variable set to Y.

These options are available on the hop root command (so they apply before run, gui, search, and so on) and on the subcommands themselves (hop run -pl …, hop gui -pl …).

See hop, hop run and hop gui for the full option lists.

Command Line Project Configuration

In addition to the Hop Gui and configuration files, all aspects of and operations on projects and environments can be managed through the Hop Conf command line tool.

The examples use hop-conf.sh on Linux and macOS. On Windows, run hop-conf.bat with the same options, and put a command on a single line or end its lines with ^ instead of \.

Projects

A project is registered in hop-config.json with its name and its home folder. Its settings and variables are stored in project-config.json, in the home folder.

Create a project

The home folder has to exist: when it doesn’t, hop-conf stops with an error that the home folder does not exist.

bash

mkdir -p /data/projects/sales
./hop-conf.sh --project-create --project=sales --project-home=/data/projects/sales

Output:

shell

Creating project 'sales'
Project 'sales' was created for home folder : /data/projects/sales
Configuration file for project 'sales' was saved to : file:/data/projects/sales/project-config.json

You can set more at the same time, for example a parent project to inherit metadata and variables from (--project-parent), a description (--project-description) or variables (--project-variables, see Project variables).

When the home folder already has a project-config.json, --project-create updates it with the options you pass and keeps its variables. Add --project-keep-config-file to leave the file as it is.

Modify, list and delete projects

--project-modify changes the options you pass, for example ./hop-conf.sh --project-modify --project=sales --project-description="Sales data warehouse".
--projects-list lists the projects with their home folder and configuration file.
--project-delete removes a project from hop-config.json, for example ./hop-conf.sh --project-delete --project=sales. The home folder and its files are not deleted.

Project variables

--project-variables adds variables to the project, or changes the value of a variable that already exists. Variables with other names are kept.

bash

./hop-conf.sh --project-modify --project=sales --project-variables=INPUT_FOLDER=/data/in,OUTPUT_FOLDER=/data/out

Separate variables with commas. To use spaces or commas in a value, wrap the value in double quotes and keep them from the shell:

Option, as typed in a shell Variables

Option, as typed in a shell	Variables
`--project-variables=key1=value1,key2=value2`	`key1` = `value1`, `key2` = `value2`
`--project-variables=key1="This value contains spaces",key2=value2`	`key1` = `This value contains spaces`, `key2` = `value2`
`--project-variables='key1="value1,value2"'`	`key1` = `value1,value2`
`--project-variables=key1=\"value1,value2\"`	`key1` = `value1,value2`
`--project-variables=key1="\"\"String with spaces\"\""`	`key1` = `"String with spaces"`

--project-variables=key1=value1,key2=value2

key1 = value1, key2 = value2

--project-variables=key1="This value contains spaces",key2=value2

key1 = This value contains spaces, key2 = value2

--project-variables='key1="value1,value2"'

key1 = value1,value2

--project-variables=key1=\"value1,value2\"

key1 = value1,value2

--project-variables=key1="\"\"String with spaces\"\""

key1 = "String with spaces"

Environments

An environment belongs to one project. It adds configuration files with variables to that project, usually one environment for every stage of its lifecycle: development, testing, production. The project holds the code, and the environment the configuration, such as server names and folders. See Projects and Environments for how they work together.

Create an environment

bash

./hop-conf.sh --environment-create \
     --environment=sales-dev \
     --environment-project=sales \
     --environment-purpose=Development \
     --environment-config-files=/data/environments/sales-dev/variables.json

Output, shortened:

shell

Creating environment 'sales-dev'
Environment 'sales-dev' was created in Hop configuration file /opt/hop/config/hop-config.json
Created empty environment configuration file : /data/environments/sales-dev/variables.json
  sales-dev
    Purpose: Development
    Project name: sales
      Config file: /data/environments/sales-dev/variables.json

A configuration file that doesn’t exist yet is created, empty. Before creating it, hop-conf logs an error that the file doesn’t exist to read variables from.

json

{ }

Add variables to it as described in Configuration files.

Add configuration files from a directory

--environment-config-files lists files one by one. To add every matching file in a folder, pass the folder and a regular expression. The expression is matched against the file name, the same way as in the JSON Input and Get File Names transforms. .json is not a valid expression; use .\.json.

An exclusion expression and --environment-config-include-subfolders are optional. Files found in the folder are stored as individual paths. They can be combined with --environment-config-files. When either option is set, that selection replaces the environment’s configuration file list.

bash

./hop-conf.sh --environment-create \
     --environment=sales-test \
     --environment-project=sales \
     --environment-purpose=Testing \
     --environment-config-file-directory="/data/environments/sales-test" \
     --environment-config-file-wildcard=".*\.json" \
     --environment-config-file-exclude-wildcard=".*secret.*" \
     --environment-config-include-subfolders

Modify, list and delete environments

--environment-modify changes the options you pass, for example ./hop-conf.sh --environment-modify --environment=sales-dev --environment-purpose=Testing.
--environments-list lists the environments with their purpose, project and configuration files.
--environment-delete removes an environment from hop-config.json, for example ./hop-conf.sh --environment-delete --environment=sales-dev. Its configuration files are not deleted.

Configuration files

The configuration files of an environment hold its variables. --config-file selects the file, --config-file-set-variables sets the value of variables, and --config-file-describe-variables their description:

bash

./hop-conf.sh --config-file=/data/environments/sales-dev/variables.json \
     --config-file-set-variables=DB_HOSTNAME=localhost,DB_PASSWORD=abcd
./hop-conf.sh --config-file=/data/environments/sales-dev/variables.json \
     --config-file-describe-variables="DB_HOSTNAME=The database server"

The file then contains:

json

{
  "variables" : [
    {
      "name" : "DB_HOSTNAME",
      "value" : "localhost",
      "description" : "The database server"
    },
    {
      "name" : "DB_PASSWORD",
      "value" : "abcd",
      "description" : ""
    }
  ]
}

The variables of hop-config.json itself, listed as System variables in the configuration perspective of Hop Gui, are set with --set-variable and described with --describe-variable:

bash

./hop-conf.sh --set-variable=COMPANY_NAME=Acme
./hop-conf.sh --describe-variable="COMPANY_NAME=The company name used in reports"

--set-variable=MY_VARIABLE= sets a variable to an empty value. The variable stays in the list.

Projects plugin configuration

There are various options to configure the behavior of the Projects plugin itself. In Hop configuration file hop-config.json we can find the following options:

json

{
    "projectMandatory" : true,
    "environmentMandatory" : false,
    "defaultProject" : "default",
    "defaultEnvironment" : null,
    "standardParentProject" : "default",
    "standardProjectsFolder" : "/home/matt/test-stuff/"
}

Option Description hop-conf option

Option	Description	hop-conf option
projectMandatory	This will prevent anyone from using hop-run without specifying a project	`--project-mandatory`
environmentMandatory	This will prevent anyone from using hop-run without specifying an environment	`--environment-mandatory`
defaultProject	The default project to use when none is specified	`--default-project`
defaultEnvironment	The default environment to use when none is specified	`--default-environment`
standardParentProject	The standard parent project to propose when creating new project	`--standard-parent-project`
standardProjectsFolder	The folder to which you’ll browse by default in the GUI when creating new projects	`--standard-projects-folder`

projectMandatory

This will prevent anyone from using hop-run without specifying a project

--project-mandatory

environmentMandatory

This will prevent anyone from using hop-run without specifying an environment

--environment-mandatory

defaultProject

The default project to use when none is specified

--default-project

defaultEnvironment

The default environment to use when none is specified

--default-environment

standardParentProject

The standard parent project to propose when creating new project

--standard-parent-project

standardProjectsFolder

The folder to which you’ll browse by default in the GUI when creating new projects

--standard-projects-folder