Environments

Managing projects and environments

Most developers who design and manage data processing on a daily basis work on a multitude of projects and modules. Different sets of workflows and pipelines require management for at least development, acceptance and production environments. Every project or environment comes with its own set of variables and configurations for databases, file paths, ..

How does Hop manage projects and environments

Hop allows developers to manage different projects and environments with their corresponding configurations and variables. Hierarchically each project can have multiple environments with different purposes. Each project will remember opened files, zoom levels and other UI settings.

Creating a project

To create a new project click the Add a new project button.

Add a new project

This button opens the following dialogue:

Create a project

Table 1. Project Properties

Property

Description

Variables Supported

Mandatory

Default

Name

The project name

Yes

No

Home folder

The folder where the project is located

Yes

No

Configuration file (relative path)

The folder where the project’s configuration json is located.

Yes

Yes

Description

A description for this environment

No

No

Company

The company this environment belongs to

No

No

Department

The department this environment is created for

No

No

Metadata base folder

The folder where this environment’s metadata will be stored

Yes

Yes

${ENVIRONMENT_HOME}/metadata

Unit test base path

The folder where this environment’s unit tests will be stored

Yes

Yes

${ENVIRONMENT_HOME}

Data Sets CSV Folder

The folder where this environment’s data files will be stored

Yes

Yes

${ENVIRONMENT_HOME}/datasets

Enforce execution in environment home

Give an error when trying to execute a pipeline or workflow which is not located in the environment home directory or in a sub-directory

Yes

Yes

checked

Project variables to set

A list of variable names, values and variable descriptions to use with this project

No

Yes

After creating a project the user interface will switch to it and ask if you want to create an environment.

Creating an environment

To create a new environment click the Add a new environment button.

Environment

This button opens the following dialogue:

Create a project

Table 2. Environment Properties

Property

Description

Variables Supported

Mandatory

Default

Name

The environment name

No

No

The last created project

Purpose

The purpose of this environment

  • Development

  • Testing

  • Acceptance

  • Production

  • Continuous integration

  • Common build

  • …​

No

No

Project

The project to which this environment belongs

No

No

The last created project

Configuration files

Files to use as configuration for this environment

No

No

After creating an environment the user interface will switch to it.

Switching between projects and environments

To switch between projects the projects list can be used from the main Hop Gui toolbar.

Switch project

After switching to a project the environments list will be updated to the environments which belong to this project. All open files will be restored, including their zoom level and other UI settings.

Switch environment

Editing and deleting projects

To edit an existing project click the Edit the selected project button.

Edit the selected project

Clicking this button will re-open the project dialogue where changes can be made as described in Project Properties.

To delete a project click the Delete the selected project button.

Delete the selected project

Editing and deleting environments

To edit an existing environment click the Edit the selected environment button.

Edit the selected environment

Clicking this button will re-open the project dialogue where changes can be made as described in Environment Properties.

To delete a project click the Delete the selected environment button.

Delete the selected environment

Advanced project and environment management

Configuration files

Configuration files will be stored in the chosen project home directory as a json file. An environment can be set to point to (multiple) different configuration files as is needed. These config files contain the following:

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "config" : {
    "variables" : [ {
      "name" : "foo"
    } ]
  }
}%

Advanced Hop environment configuration

Hop enviroments and their home folders are stored in the hop configuration file 'hop-config.json'. That file lives by default in the config folder of the Hop installation. System property 'HOP_CONFIG_FOLDER' can also be used to point to a different folder

{
 "environmentConfig" : {
  "enabled" : true,
  "openingLastEnvironmentAtStartup" : true,
  "environmentConfigFilename" : "environment.json",
  "environmentFolders" : {
   "Project 1 - DEV" : "/projects/one/dev/",
   "Project 1 - UAT" : "/projects/one/uat/",
   "Project 1 - PRD" : "/projects/one/prd/",
   "Project 2 - DEV" : "/projects/two/dev/",
   "Project 2 - UAT" : "/projects/two/uat/",
   "Project 2 - PRD" : "/projects/two/prd/",
  }
}

The Hop command line configuration tool

Hop Conf is a command line tool to manage environments. The hop-conf.sh script can be run with the -h flag (./hop-conf.sh -h) to display available options.

Hop Conf

The available options are listed below:

Table 3. Hop-conf Options
Short Option Extended Option Description

-h

--help

Displays this help message and quits.

-ec

--environment-create

Create an environment. Also specify the name and its home

-ed

--environment-delete

Delete an environment

-el

--environment-list

List the defined environments

-em

--environment-modify

Modify an environment

-pc

--project-create

Create a new project. Also specify the name and its home

-pd

--prject-delete

Delete a project

-pl

--project-list

List the defined projects

-pm

--project-modify

Modify a project

-dv

--describe-variable=<describeVariable>

Describe a variable

-e

-environment=<environmentName>

The name of the environment to manage

-ep

--environment-project=<environmentProject>

The project for the environment

-eu

--environment-purpose=<environmentPurpose>

The purpose of the environment: Development, Testing, Production, CI, …​

-fj

--generate-fat-jar=<fatJarFilename>

Specify the filename of the fat jar to generate from your current software installation

-p

--project=<projectName>

The project name

-pa

--project-metadata-base=<projectMetadataBaseFolder>

The metadata base folder (relative to home)

-pb

--project-datasets-base-base=<projectDataSetsCsvFolder>

The data sets CSV folder (relative to home)

-pf

--project-config-file=<projectConfigFile>

The configuration file relative to the home folder. The default value is project-config.json

-ph

--project-home=<projectHome>

The home directory of the project

-pp

--project-company=<projectCompany>

The company

-ps

--project-description=<projectDescription>

The description of the project

-pt

--project-department=<projectDepartment>

The department

-pu

--project-unit-tests-base=<projectUnitTestsBasePath>

The unit tests base folder (relative to home)

-px

--project-enforce-execution=<projectEnforceExecutionInHome>

Validate before execution that a workflow or pipeline islocated in the project home folder or a sub-folder (true/false)

-sv

--set-variable=<setVariable>

Set a variable, use format VAR=Value

-cfg

--config-file=<configFile>

Specify the configuration JSON file to manage

-cfd

--config-file-describe-variables=<configDescribeVariables>[,<configDescribeVariables>…​]

A list of variable=description combinations separated by a comma

-cfv

--config-file-set-variables=<configSetVariables> ,<configSetVariables>…​]

A list of variable=value combinations separated by a comma

-eg

--environment-config-files=<environmentConfigFiles>[, <environmentConfigFiles>…​]

A list of configuration files for this lifecycle environment, comma separated

-pv

--project-variables=<projectVariables>[,<projectVariables>…​]

A list of variable=value combinations separated by a comma