Databricks Job Run

Description

The Databricks Job Run action triggers a job on a Databricks workspace via the Jobs REST API and can wait until the run finishes. It stores Job ID, Run ID, and status in workflow variables for downstream actions.

Use a Databricks Connection (workspace host + personal access token). This is not the JDBC SQL warehouse connection.

Modes

Mode Behavior

Run existing job

Calls jobs/run-now with the given Job ID.

One-time submit

Calls jobs/runs/submit with a raw Submit JSON body (full Jobs API payload you provide).

Deploy & run

Uploads fat jar, pipeline .hpl, and exported metadata JSON to a Volume or DBFS base path (optionally a Spark project package zip and environment config file), then creates or updates a JAR job (org.apache.hop.spark.run.MainSpark) with classic existing / new_cluster or serverless compute and runs it. See Native Spark on Databricks.

Wait modes

Wait mode Behavior

Wait for completion

Polls jobs/runs/get until the run is terminal (SUCCESS, FAILED, …) or Timeout is reached (0 = no limit).

Fire and forget

Returns as soon as the run is accepted; does not poll. Pair with Databricks Job Wait later in the workflow (using ${DatabricksRunId}).

If the parent workflow is stopped while waiting, the action attempts to cancel the Databricks run.

Result variables

Default variable names (configurable on the Run tab):

Variable Content

DatabricksJobId

Job ID (when known)

DatabricksRunId

Run ID

DatabricksStatus

Life-cycle or result state (SUCCESS, FAILED, PENDING, …)

DatabricksRunPageUrl

Run page URL when returned by the API

DatabricksError

Error message on failure

Options

Job tab

Option Description

Databricks connection

Databricks Connection metadata name

Mode

Run existing job / One-time submit / Deploy & run

Job ID

Numeric job id for run-now or update-existing deploy

Job name / Update existing job

Name for create; if Update existing job is checked, Job ID is reset with the new settings (Deploy & run)

Submit JSON

Full JSON body for runs/submit (one-time mode)

Deploy tab

Option Description

Fat jar / Pipeline / Run configuration

Deploy mode: local fat jar (prefer hop-conf --generate-fat-jar … --spark-client-version=native-provided), entry pipeline path, and Native Spark run configuration name present in the metadata export (or project package)

DBFS / Volume base directory

Upload root for deploy artifacts. Fat jar: content-addressed hop-native-<sha12>.jar (+ .sha256 sidecar) so Databricks job libraries refresh when jar bytes change (fixed-name overwrite is not enough on long-lived Dedicated clusters — see Databricks lessons). Per entry pipeline stem: pipeline-{stem}.hpl, metadata-{stem}.json, hop-spark-package-{stem}.zip, env-config-{stem}.json. Prefer a Unity Catalog volume when public DBFS root is disabled, e.g. /Volumes/catalog/schema/volume/hop.

Package tab

Option Description

Upload Spark project package

When checked, export (or upload an existing zip) a Native Spark project package so nested Simple Mapping / Pipeline Executor files under PROJECT_HOME resolve on the cluster. MainSpark receives --HopProjectPackage=…. Same layout as Tools → Export project package for Native Spark. Requires the native Spark engine plugin when auto-exporting.

Project home / Project package file

Home used for export (default PROJECT_HOME). Optional existing package zip skips export.

Environment config file

Optional described-variables JSON uploaded as env-config.json and passed to MainSpark as --HopConfigFile (same as MainSpark’s optional env argument).

Compute tab

Option Description

Cluster / compute

Classic existing: paste the all-purpose cluster id — cluster access mode must be Dedicated (not Standard/shared; Standard is Spark Connect and has no SparkContext). Classic job cluster: enter new_cluster (ephemeral cluster for the run). Serverless: enter serverless (not supported for full Native Spark RDD workloads that need SparkContext).

new_cluster spark_version / node_type_id / num_workers / JSON

Used when compute is new_cluster. Defaults: 18.2.x-scala2.13, i3.xlarge (AWS), 1. Prefer i3.* on AWS (local NVMe); m5.xlarge often needs EBS. Optional full JSON can set spark_conf (e.g. match spark.executor.cores / spark.driver.cores to the node’s vCPUs — see Native Spark on Databricks).

Serverless environment_key / client

Used when compute is serverless. Defaults: key default, client 4.

Run tab

Option Description

Wait mode

Wait for completion or fire and forget

Timeout (seconds)

Maximum wait; 0 disables the timeout. For Deploy & run with classic new_cluster, allow for first jar upload and job cluster cold start (often many minutes of Pending/Initializing — see Native Spark on Databricks)

Poll interval (seconds)

Delay between status polls (default 15)

Result variable names

Override default variable names above

Related

Deploy tips

  • See the full guide: Native Spark on Databricks.

  • Generate a fat jar with hop-conf.sh --generate-fat-jar=… --spark-client-version=native-provided when the cluster already provides Spark.

  • The run configuration name must exist in the project metadata that is exported at deploy time.

  • UC Volumes (/Volumes/…): uploads use the Files API (PUT /api/2.0/fs/files/…, up to ~5 GiB).

  • Classic DBFS (dbfs:/FileStore/…): uploads use DBFS create / add-block / close. Public DBFS root is often disabled on newer workspaces — use a volume instead.

  • Large jar uploads can take several minutes.