Databricks job run

Description

The Databricks job run action triggers a job on a Databricks workspace via the Jobs REST API and can wait until the run finishes. It stores Job ID, Run ID, and status in workflow variables for downstream actions.

Use a Databricks Connection (workspace host + personal access token). This is not the JDBC SQL warehouse connection.

Modes

Mode Behavior

Mode	Behavior
Run existing job	Calls `jobs/run-now` with the given Job ID.
One-time submit	Calls `jobs/runs/submit` with a raw Submit JSON body (full Jobs API payload you provide).
Deploy & run	Uploads fat jar, pipeline `.hpl`, and exported metadata JSON to a Volume or DBFS base path (optionally a Spark project package zip and environment config file), then creates or updates a JAR job (`org.apache.hop.spark.run.MainSpark`) with classic existing / `new_cluster` or serverless compute and runs it. See Native Spark on Databricks.

Run existing job

Calls jobs/run-now with the given Job ID.

One-time submit

Calls jobs/runs/submit with a raw Submit JSON body (full Jobs API payload you provide).

Deploy & run

Uploads fat jar, pipeline .hpl, and exported metadata JSON to a Volume or DBFS base path (optionally a Spark project package zip and environment config file), then creates or updates a JAR job (org.apache.hop.spark.run.MainSpark) with classic existing / new_cluster or serverless compute and runs it. See Native Spark on Databricks.

Wait modes

Wait mode Behavior

Wait mode	Behavior
Wait for completion	Polls `jobs/runs/get` until the run is terminal (`SUCCESS`, `FAILED`, …) or Timeout is reached (0 = no limit).
Fire and forget	Returns as soon as the run is accepted; does not poll. Pair with Databricks job wait later in the workflow (using `${DatabricksRunId}`).

Wait for completion

Polls jobs/runs/get until the run is terminal (SUCCESS, FAILED, …) or Timeout is reached (0 = no limit).

Fire and forget

Returns as soon as the run is accepted; does not poll. Pair with Databricks job wait later in the workflow (using ${DatabricksRunId}).

If the parent workflow is stopped while waiting, the action attempts to cancel the Databricks run.

Result variables

Default variable names (configurable on the Run tab):

Variable Content

Variable	Content
`DatabricksJobId`	Job ID (when known)
`DatabricksRunId`	Run ID
`DatabricksStatus`	Life-cycle or result state (`SUCCESS`, `FAILED`, `PENDING`, …)
`DatabricksRunPageUrl`	Run page URL when returned by the API
`DatabricksError`	Error message on failure

DatabricksJobId

Job ID (when known)

DatabricksRunId

Run ID

DatabricksStatus

Life-cycle or result state (SUCCESS, FAILED, PENDING, …)

DatabricksRunPageUrl

Run page URL when returned by the API

DatabricksError

Error message on failure

Options

Job tab

Option	Description
Databricks connection	Databricks Connection metadata name
Mode	Run existing job / One-time submit / Deploy & run
Job ID	Numeric job id for run-now or update-existing deploy
Job name / Update existing job	Name for create; if Update existing job is checked, Job ID is reset with the new settings (Deploy & run)
Submit JSON	Full JSON body for runs/submit (one-time mode)

Option

Description

Databricks connection

Databricks Connection metadata name

Mode

Run existing job / One-time submit / Deploy & run

Job ID

Numeric job id for run-now or update-existing deploy

Job name / Update existing job

Name for create; if Update existing job is checked, Job ID is reset with the new settings (Deploy & run)

Submit JSON

Full JSON body for runs/submit (one-time mode)

Deploy tab

Option Description

Option	Description
Fat jar / Pipeline / Run configuration	Deploy mode: local fat jar (prefer `hop-conf --generate-fat-jar … --spark-client-version=native-provided`), entry pipeline path, and Native Spark run configuration name present in the metadata export (or project package)
DBFS / Volume base directory	Upload root for deploy artifacts. Fat jar: content-addressed `hop-native-<sha12>.jar` (+ `.sha256` sidecar) so Databricks job libraries refresh when jar bytes change (fixed-name overwrite is not enough on long-lived Dedicated clusters — see Databricks lessons). Per entry pipeline stem: `pipeline-{stem}.hpl`, `metadata-{stem}.json`, `hop-spark-package-{stem}.zip`, `env-config-{stem}.json`. Prefer a Unity Catalog volume when public DBFS root is disabled, e.g. `/Volumes/catalog/schema/volume/hop`.

Fat jar / Pipeline / Run configuration

Deploy mode: local fat jar (prefer hop-conf --generate-fat-jar … --spark-client-version=native-provided), entry pipeline path, and Native Spark run configuration name present in the metadata export (or project package)

DBFS / Volume base directory

Upload root for deploy artifacts. Fat jar: content-addressed hop-native-<sha12>.jar (+ .sha256 sidecar) so Databricks job libraries refresh when jar bytes change (fixed-name overwrite is not enough on long-lived Dedicated clusters — see Databricks lessons). Per entry pipeline stem: pipeline-{stem}.hpl, metadata-{stem}.json, hop-spark-package-{stem}.zip, env-config-{stem}.json. Prefer a Unity Catalog volume when public DBFS root is disabled, e.g. /Volumes/catalog/schema/volume/hop.

Package tab

Option Description

Option	Description
Upload Spark project package	When checked, export (or upload an existing zip) a Native Spark project package so nested Simple Mapping / Pipeline Executor files under `PROJECT_HOME` resolve on the cluster. MainSpark receives `--HopProjectPackage=…`. Same layout as Tools → Export project package for Native Spark. Requires the native Spark engine plugin when auto-exporting.
Project home / Project package file	Home used for export (default `PROJECT_HOME`). Optional existing package zip skips export.
Environment config file	Optional described-variables JSON uploaded as `env-config.json` and passed to MainSpark as `--HopConfigFile` (same as MainSpark’s optional env argument).

Upload Spark project package

When checked, export (or upload an existing zip) a Native Spark project package so nested Simple Mapping / Pipeline Executor files under PROJECT_HOME resolve on the cluster. MainSpark receives --HopProjectPackage=…. Same layout as Tools → Export project package for Native Spark. Requires the native Spark engine plugin when auto-exporting.

Project home / Project package file

Home used for export (default PROJECT_HOME). Optional existing package zip skips export.

Environment config file

Optional described-variables JSON uploaded as env-config.json and passed to MainSpark as --HopConfigFile (same as MainSpark’s optional env argument).

Compute tab

Option Description

Option	Description
Cluster / compute	Classic existing: paste the all-purpose cluster id — cluster access mode must be Dedicated (not Standard/shared; Standard is Spark Connect and has no `SparkContext`). Classic job cluster: enter `new_cluster` (ephemeral cluster for the run). Serverless: enter `serverless` (not supported for full Native Spark RDD workloads that need `SparkContext`).
new_cluster spark_version / node_type_id / num_workers / JSON	Used when compute is `new_cluster`. Defaults: `18.2.x-scala2.13`, `i3.xlarge` (AWS), `1`. Prefer `i3.*` on AWS (local NVMe); `m5.xlarge` often needs EBS. Optional full JSON can set `spark_conf` (e.g. match `spark.executor.cores` / `spark.driver.cores` to the node’s vCPUs — see Native Spark on Databricks).
Serverless environment_key / client	Used when compute is `serverless`. Defaults: key `default`, client `4`.

Cluster / compute

Classic existing: paste the all-purpose cluster id — cluster access mode must be Dedicated (not Standard/shared; Standard is Spark Connect and has no SparkContext). Classic job cluster: enter new_cluster (ephemeral cluster for the run). Serverless: enter serverless (not supported for full Native Spark RDD workloads that need SparkContext).

new_cluster spark_version / node_type_id / num_workers / JSON

Used when compute is new_cluster. Defaults: 18.2.x-scala2.13, i3.xlarge (AWS), 1. Prefer i3.* on AWS (local NVMe); m5.xlarge often needs EBS. Optional full JSON can set spark_conf (e.g. match spark.executor.cores / spark.driver.cores to the node’s vCPUs — see Native Spark on Databricks).

Serverless environment_key / client

Used when compute is serverless. Defaults: key default, client 4.

Run tab

Option Description

Option	Description
Wait mode	Wait for completion or fire and forget
Timeout (seconds)	Maximum wait; `0` disables the timeout. For Deploy & run with classic `new_cluster`, allow for first jar upload and job cluster cold start (often many minutes of Pending/Initializing — see Native Spark on Databricks)
Poll interval (seconds)	Delay between status polls (default 15)
Result variable names	Override default variable names above

Wait mode

Wait for completion or fire and forget

Timeout (seconds)

Maximum wait; 0 disables the timeout. For Deploy & run with classic new_cluster, allow for first jar upload and job cluster cold start (often many minutes of Pending/Initializing — see Native Spark on Databricks)

Poll interval (seconds)

Delay between status polls (default 15)

Result variable names

Override default variable names above

Native Spark on Databricks — workspace tiers, classic vs serverless, UC Volumes, Deploy & run checklist
Databricks Connection
Databricks job wait — poll a run after fire-and-forget
Native Spark — fat jar / MainSpark for JAR tasks on the cluster
Databricks JDBC — SQL warehouse (different use case)

Deploy tips

See the full guide: Native Spark on Databricks.
Generate a fat jar with hop-conf.sh --generate-fat-jar=… --spark-client-version=native-provided when the cluster already provides Spark.
The run configuration name must exist in the project metadata that is exported at deploy time.
UC Volumes (/Volumes/…): uploads use the Files API (PUT /api/2.0/fs/files/…, up to ~5 GiB).
Classic DBFS (dbfs:/FileStore/…): uploads use DBFS create / add-block / close. Public DBFS root is often disabled on newer workspaces — use a volume instead.
Large jar uploads can take several minutes.