Add preliminary job_runs resource#5603
Open
radakam wants to merge 17 commits into
Open
Conversation
Collaborator
Integration test reportCommit: f0427e0
23 interesting tests: 13 SKIP, 7 RECOVERED, 3 flaky
Top 26 slowest tests (at least 2 minutes):
|
radakam
force-pushed
the
job-runs-resource
branch
from
63df005 to
ff9faea
Compare
radakam
force-pushed
the
job-runs-resource
branch
from
a899c09 to
3fb23a7
Compare
Contributor
Approval status: pending
|
| if err != nil { | ||
| return "", nil, err | ||
| } | ||
| return strconv.FormatInt(wait.RunId, 10), nil, nil |
Contributor
There was a problem hiding this comment.
We need to store job_params and other fields from https://github.com/databricks/databricks-sdk-go/blob/bcf8559242f04e35469cf6fc6eaf77227f06080d/service/jobs/model.go#L3532 in order for framework to be able compute the diff, you can test it with bundle plan command
Contributor
Author
There was a problem hiding this comment.
Done -- DoRead now maps the run's job_id, job_parameters, and overriding params from GetRun back into the RunNow shape, so the framework can diff them. Please let me know if I missed any fields!
Contributor
There was a problem hiding this comment.
DoRead is called when we read the state of the resource in the workspace and then we construct the state out of it.
The state we return here is what we actually created as a resource. We need to record the correct remote state here in order to calculate "out-of-band" diff correctly (aka comparing what DoRead computes and what we saved as part of DoCreate call)
Contributor
Author
There was a problem hiding this comment.
DoCreate now returns the remote state instead of nil: since RunNow only hands back the run id, I reconstruct it from the request we just sent plus the new id. That's now what the out-of-band diff compares against on the next DoRead. The create-vs-read consistency check in all_test.go passes (the input-only bits like job_parameters/job_settings are covered by the ignore filter), and the job_runs acceptance tests stay green.
| // remote identity lives in RunId. Drift on the embedded request fields is | ||
| // suppressed via ignore_remote_changes in resources.yml. | ||
| type JobRunRemote struct { | ||
| jobs.RunNow |
Contributor
There was a problem hiding this comment.
Maybe it's better to embed jobs.Run here because this is what GetRun returns
Contributor
Author
There was a problem hiding this comment.
DoRead now maps the GetRun result back into RunNow shape. It seems that embedding jobs.Run would make the diff not line up as the remote needs to share field paths with the state (which is RunNow-shaped). Do you think it makes sense?
radakam
force-pushed
the
job-runs-resource
branch
from
d6b00f5 to
f7ec335
Compare
Introduce the job_runs bundle resource backed by the Jobs RunNow API. This is a preliminary skeleton: it wires up the config type, the direct engine resource, registration, and drift handling. Generated schema, validation output, and tests are intentionally left for follow-ups.
Run the schema generator for the new job_runs resource and add real descriptions for the job_runs map, JobRun.lifecycle, and JobRun.python_named_params so the required-annotations guard test passes.
Regenerate the validation rules so job_runs picks up its required field (job_id) and the performance_target enum. Missing required fields surface as warnings via the existing validate:required mutator.
Deploy a bundle with a job and a job_run referencing it, and assert that exactly one RunNow request is made and the returned run id is stored in state. Restricted to the direct engine since job_runs has no Terraform equivalent.
Verify that changing a job_run's configuration re-triggers the run: after the initial deploy, a config change (job_parameters) causes a second, distinct RunNow request on redeploy via the recreate-on-change path.
Several reflection/enumeration tests iterate or hardcode the full resource set and were not updated when the job_runs resource was added, leaving CI red. Add job_runs to each: the run_as allResourceTypes list, the permissions unsupportedResources list, the target-mode mock bundle, the Terraform lifecycle skip list (direct-only), a bind fixture + GetRun mock, and a direct CRUD fixture that triggers a run against a newly created job. The CRUD test also treats job_runs as a no-op delete, since a triggered run cannot be undeployed and stays readable afterward.
Fully populate the GetRunRequest and JobRunRemote literals in job_run.go to satisfy exhaustruct, and regenerate the schema/refschema/apitypes artifacts so the committed job_runs generated files match their generators.
Add job_runs coverage to the two reflection-driven completeness tests: add "job_runs" to the noURL set in TestBundleResourcePluralNamesResolveInWorkspaceURLs (no stable workspace URL yet) and exercise Resources.JobRuns in the StateToBundle test suite.
Add a notebook_change acceptance test that verifies editing a job's notebook_path updates the job (jobs/reset) without re-triggering the run, confirming the run only recreates when its own RunNow config changes. Also extend the basic and redeploy tests with plan/summary/validate traces to make the deploy lifecycle visible in the recorded output.
Map GetRun fields (job_id, job_parameters, overriding parameters) back into
the RunNow shape in DoRead instead of returning an empty RunNow, so the
framework can compute a real diff for the run config.
Add a job_settings snapshot to the job_run config and state: setting
job_settings to a whole-job reference (e.g. ${resources.jobs.my_job}) lets a
change to the targeted job re-trigger the run, since the snapshot is marked
recreate_on_changes. The notebook_change acceptance test now asserts a
notebook_path edit re-triggers the run via this snapshot.
Regenerate schema/validation artifacts for the new job_settings field.
Avoid partial struct literals in job_run DoRead so exhaustruct passes, and raise the config.Root field-count upper bound now that job_runs snapshots a full job_settings.
DoRead now records the run's remote state as what we actually created: the job_id and the overriding parameters GetRun echoes back. It no longer maps run.JobParameters, which GetRun resolves to the job's full parameter set (including defaults the run never overrode) and which therefore does not represent what we created; that field is handled via ignore_remote_changes. The local diff (config vs saved state) still re-triggers a run on any job_parameters change via recreate_on_changes, independent of DoRead. The testserver now resolves the full parameter set like the real GetRun, and a new acceptance test asserts the resolved default does not look like drift.
RunNow's response only carries the new run id, so DoCreate previously returned nil remote state. Reconstruct it from the request we just sent plus the run id, giving the framework a faithful record of what we created to compute the out-of-band diff against DoRead.
Regenerate out.fields.txt after rebasing onto main, which bumped databricks-sdk-go to v0.147.0. The new SDK adds azure_attributes.capacity_reservation_group, which the validate-generated job picks up when generating against the PR-merged-with-main commit.
radakam
force-pushed
the
job-runs-resource
branch
from
9df3417 to
8a71c9e
Compare
These pypi-proxy registry rewrites in python/uv.lock and python/codegen/uv.lock were environment artifacts unrelated to the job_runs resource. Restore both lockfiles to match main.
A run's URL needs both its run id and its parent job id. The run id is
restored from state like any resource, but the job id in config comes from
a ${resources.jobs.*.id} reference that is only resolved at deploy, so at
read time it is 0 and the URL would render as jobs/0/runs/<runid>.
Restore the run's resolved job_id from state at load time (same mechanism
as id and the dashboard etag), then build the URL in JobRun.InitializeURL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add a
job_runsbundle resource (direct engine). Deploying it triggers a job run via the JobsRunNowAPI and stores the run id in state.Includes the resource + direct CRUD, lifecycle config, generated schema/validation,
run_ashandling, the run URL inbundle summary, acceptance tests, and updates to the resource-enumeration tests.Because a run isn't a normal CRUD resource, a few deliberate choices:
RunNow); the run id is the tracked identity.recreate_on_changes— changes re-trigger a fresh run.RunNowre-triggers the run.GetRunechoes the run's overriding parameters faithfully, soDoReadmaps just those back and the framework diffs them directly. EverythingGetRundoes not return faithfully isignore_remote_changes: the request-only inputs (idempotency_token,only,performance_target,queue), the local-onlyjob_settingssnapshot, andjob_parameters—GetRunresolves the job's full parameter set (including defaults the run never overrode), so diffing it directly would look like perpetual drift.RunNowhas norun_as, so a differing bundlerun_asis rejected early.Re-triggering on job changes (
job_settings)RunNowonly carries the stablejob_id, so a run has no way to notice that the targeted job's definition changed. To support that, the resource has an optionaljob_settingsfield that snapshots the job. Point it at a whole-job reference and any change to the job re-triggers the run (the snapshot isrecreate_on_changes):The snapshot is a local-only input:
GetRunnever returns it, so it'signore_remote_changesto avoid spurious drift.Run URL in
bundle summaryA run's URL is
jobs/<job id>/runs/<run id>. The run id comes from state, but the run'sjob_idis a${resources.jobs.*.id}reference that's only resolved at deploy, so at summary it reads0(renderingjobs/0/...) even after a deploy. We restore the run'sjob_idfrom its state blob (like the dashboard etag) and write it back into config likeid;JobRun.InitializeURLthen builds the URL viaworkspaceurls.JobRunURL, guarding on both ids so a not-yet-deployed run stays blank.job_runsstays in thenoURLtest list since its two-id URL can't be a single-id pattern.Closes DECO-27355.
(Waiting for completion is deferred to a later milestone.)
Why
Today, triggering a job as part of
bundle deployrequires custom scripts outside the bundle. This project adds aresources.job_runssection to direct engine so users can declaratively run jobs as part of deployment — e.g. schema migrations, model training, data prep.Tests
job_runs/basic— one run-now, run id stored, redeploy doesn't re-trigger, run URL shown in post-deploy summary, clean destroy.job_runs/redeploy— changingjob_parametersre-triggers a second, different run.job_runs/job_parameters— overriding only one of the job's two parameters stays a stable no-op plan, proving the resolved job default isn't read as drift.job_runs/notebook_change— changing the targeted job'snotebook_pathre-triggers the run via thejob_settingssnapshot.TestAll/job_runsdirect CRUD.workspaceurls.JobRunURLunit test (plain +?w=workspace-id forms).RunNow/GetRunnow echo the job's resolved parameters and the run's overriding parameters so reads round-trip.run_as, permissions, target-mode, Terraform lifecycle, bind-support enumeration, workspace-URL completeness (job_runsstays innoURL— two-id URL built inInitializeURL), andStateToBundleenumeration tests.