Databricks job wait

Description

Databricks job wait polls the Jobs API (jobs/runs/get) until a run reaches a terminal state (SUCCESS, FAILED, …). Use it after Databricks job run in Fire and forget mode, or when another system already started a run and you only have a Run ID.

There is no dedicated “wait” endpoint on Databricks: this action repeatedly GETs run status, sleeps for the poll interval, and exits when the run finishes or the timeout is hit.

Typical pattern

  1. Job Run with Fire and forget → sets ${DatabricksRunId}

  2. Other workflow actions (notifications, parallel prep, …)

  3. Job Wait with Run ID ${DatabricksRunId} → success/failure hops

For “submit and block on this step,” use Job Run’s built-in Wait for completion instead.

Options

Option Description

Databricks connection

Databricks Connection (PAT)

Run ID

Numeric run id; default ${DatabricksRunId}

Timeout (seconds, 0=none)

Maximum wait; 0 = no timeout. A value that is not a number is replaced by 3600.

Poll interval (seconds)

Delay between status checks (default 15, at least 1)

Cancel Databricks run if workflow is stopped or the timeout is reached

If checked, calls jobs/runs/cancel when the parent workflow stops, and also when the timeout is reached

Result variables

Same defaults as Job Run: DatabricksJobId, DatabricksRunId, DatabricksStatus, DatabricksRunPageUrl, DatabricksError

Result

What this action passes on to the next action. See Action results for how results travel through a workflow.

Result Value

True

The run ended with life cycle state TERMINATED and result state SUCCESS, with 0 errors.

False

The number of errors is 1. Databricks connection is empty or doesn’t exist, Run ID is empty or not a number, a call to the Jobs API fails, the run ends with another result state (for example FAILED or CANCELED) or with life cycle state SKIPPED or INTERNAL_ERROR, the timeout is reached, or the workflow is stopped while the action waits.

Result files

Passed on unchanged.

Result rows

Passed on unchanged.

After every status check the action sets the result variables for the Job ID, the Run ID, the status (the result state, like SUCCESS or FAILED, once the run is TERMINATED, the life cycle state before that) and the run page URL (the Job ID and the URL only when Databricks returns them), in its workflow and in every parent workflow. The error variable is emptied when the action starts and holds the error message when the action is false.