Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion extend/docker-runner/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@ all of them are optional:
- `storage` --- configuration of [input and output mapping](/extend/common-interface/folders/); specific options correspond to the options of the
[unload data](https://keboola.docs.apiary.io/#reference/tables/unload-data-asynchronously) and
[load data](https://keboola.docs.apiary.io/#reference/tables/load-data-asynchronously) API calls.
- `runtime` --- configuration for modifying some image parameters at run time
- `runtime` --- [runtime settings](/integrate/jobs/#job-runtime-configuration) (`tag`, `backend`, `parallelism`); most notably `runtime.tag`
pins the Docker image tag that jobs of this configuration run, which is the usual way of testing a development build of a component
- `processors` --- configuration of [Processors](/extend/component/processors/)
- `authorization` --- OAuth authorization [injected to the configuration](/extend/common-interface/oauth/); not stored in the component configuration
- `image_parameters` --- an arbitrary object passed from the [component](/extend/component/); not stored in the component configuration
Expand Down
30 changes: 23 additions & 7 deletions integrate/jobs/index.md
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Original file line number Diff line number Diff line change
Expand Up @@ -242,19 +242,35 @@ When creating a job, you need to provide `mode`. This can be one of `run`, `forc
Use the `forceRun` mode to run a configuration that is disabled. The `debug` can be used during [Component Development & Debugging](https://developers.keboola.com/extend/component/tutorial/debugging/).

### Job Runtime configuration
You may provide runtime settings for a job. Runtime settings do not affect what the job does, it affects how the job does it. Current
available runtime settings are `backend` and `parallelism`. The `backend` parameter defines what Snowflake warehouse is used for the job and currently
affects mostly Snowflake transformations. Available values for backend size come from [Workspace Create API call](https://keboola.docs.apiary.io/#reference/workspaces/workspaces-collection/create-workspace) and are `small`, `medium`, `large`.
You may provide runtime settings for a job. Runtime settings do not affect what the job does, they affect how the job does it. The available runtime settings are:

The `parallelism` allows to run [Configuration Rows](https://help.keboola.com/components/#configuration-rows) (if present in the configuration) in
parallel. Allowed values are integer values and `infinity` which runs all rows in parallel. When parallelism is not specified, the rows are run sequentially.
You may also specify `tag` if you want to run the component with a specific version of code. This is mostly used during component development, testing
and debugging.
- `backend.type` --- for Snowflake transformations this is the size of the [Snowflake warehouse](https://github.com/keboola/developers-docs/pull/380/changes) used for the job; otherwise it affects the [container size](https://help.keboola.com/transformations/python-plain/#dynamic-backends). Available values for backend type are `small`, `medium`, `large`.
- `parallelism` --- runs [Configuration Rows](https://help.keboola.com/components/#configuration-rows) (if present in the configuration) in parallel. Allowed values are integer values and `infinity`, which runs all rows in parallel. When not specified, the rows are run sequentially.
- `tag` --- runs the component with a specific version of code. This is mostly used during component development, testing and debugging.

Runtime parameters can be specified on various levels. The values can be specified in the component configuration. They can also be specified
when creating a job, in which case it overrides the configuration. It may also be specified for an orchestration, in which case it overrides what is specified
in individual jobs of that orchestration.

When stored in the component configuration, the runtime settings live in a top-level `runtime` node of the configuration JSON --- a sibling of `parameters`,
not inside it. For example, to pin all jobs of a configuration to a specific image tag:

{% highlight json %}
{
"parameters": {
"...": "..."
},
"runtime": {
"tag": "my-branch-3"
}
}
{% endhighlight %}

Jobs of this configuration then run the given image tag (the job detail shows the resolved value in its `tag` field) until the `runtime.tag` key
is removed from the configuration. This is the usual way of testing a development build of a component in a single project without affecting
other projects --- the tag in the [Developer Portal](/extend/publish/) stays untouched. Do not forget to remove the key when done; a pinned
configuration keeps running the old image even after new versions of the component are released.

### Job Type
Job can be of one of the four types `standard`, `container`, `phaseContainer` and `orchestrationContainer`. The `standard` is something which does actual work.
Only standard jobs consume billable time and are counted towards consumption of any resources. Other job types are virtual containers encapsulating standard jobs.
Expand Down
Loading