You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
First, provision an OAuth Client for Airflow to use by following the guide on how to [provision client credentials](../single_sign_on.md#provisioning-client-credentials).
55
+
First, provision an OAuth Client for Airflow to use by following the guide on how to [provision client credentials](../security/single_sign_on.md#provisioning-client-credentials).
56
56
57
57
After provisioning the credentials, you can obtain the `Client ID` and `Client Secret` values for Airflow to use to connect to Tobiko Cloud.
On this page, we demonstrate the secrets method Dagster recommends for **local development**.
59
59
60
-
First, provision an OAuth Client for Dagster to use by following the guide on how to [provision client credentials](../single_sign_on.md#provisioning-client-credentials).
60
+
First, provision an OAuth Client for Dagster to use by following the guide on how to [provision client credentials](../security/single_sign_on.md#provisioning-client-credentials).
61
61
62
62
After provisioning the credentials, you can obtain the `Client ID` and `Client Secret` values for Dagster to use to connect to Tobiko Cloud.
|`url`| The Base URL to your Tobiko Cloud instance | str | Y |
377
-
|`oauth_client_id`| OAuth Client ID of the credentials you [provisioned](../single_sign_on.md#provisioning-client-credentials) for Dagster | str | N |
378
-
|`oauth_client_secret`| OAuth Client Secret of the credentials you [provisioned](../single_sign_on.md#provisioning-client-credentials) for Dagster | str | N |
377
+
|`oauth_client_id`| OAuth Client ID of the credentials you [provisioned](../security/single_sign_on.md#provisioning-client-credentials) for Dagster | str | N |
378
+
|`oauth_client_secret`| OAuth Client Secret of the credentials you [provisioned](../security/single_sign_on.md#provisioning-client-credentials) for Dagster | str | N |
379
379
|`dagster_graphql_host`| Hostname of the Dagster Webserver GraphQL endpoint | str | N |
380
380
|`dagster_graphql_port`| Port of the Dagster Webserver GraphQL endpoint | int | N |
381
381
|`dagster_graphql_kwargs`| Extra args to pass to the [DagsterGraphQLClient](https://docs.dagster.io/api/python-api/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) class when it is instantiated | dict | N |
Copy file name to clipboardExpand all lines: docs/cloud/features/scheduler/hybrid_executors_helm.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ Both executors must be properly configured with environment variables to connect
17
17
18
18
- Access to a [data warehouse supported by Tobiko Cloud](../../../integrations/overview.md#execution-engines) (e.g., Postgres, Snowflake, BigQuery)
19
19
- Helm 3.8+
20
-
- A Tobiko Cloud account with [client ID and client secret](../single_sign_on.md#provisioning-client-credentials)
20
+
- A Tobiko Cloud account with [client ID and client secret](../security/single_sign_on.md#provisioning-client-credentials)
21
21
22
22
## Quick start guide
23
23
@@ -267,7 +267,7 @@ run:
267
267
268
268
## Defining Custom Environment Variables
269
269
270
-
If there are additional environment variables that are required to run your project, you will want to define them for both the apply and run executors.
270
+
If there are additional environment variables that are required to run your project, you will want to define them for both the apply and run executors.
Copy file name to clipboardExpand all lines: docs/cloud/features/scheduler/hybrid_executors_overview.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -57,7 +57,7 @@ One important type of environment variable is the `TCLOUD` variables used for co
57
57
58
58
The first required `TCLOUD` variable is a unique Tobiko Cloud URL for your project, which your Solutions Architect will provide after your project is created.
59
59
60
-
You also need the Client ID and Client Secret variables, which are generated when you [create an OAuth Client](../single_sign_on.md#provisioning-client-credentials) in the Tobiko Cloud UI.
60
+
You also need the Client ID and Client Secret variables, which are generated when you [create an OAuth Client](../security/single_sign_on.md#provisioning-client-credentials) in the Tobiko Cloud UI.
61
61
62
62
Specify the URL, Client ID, and Client Secret in these environment variables:
At Tobiko, we treat security as a first-class citizen because we know how valuable your data assets are. Our team follows and executes security best practices across each layer of our product.
4
+
At Tobiko, we treat security as a first-class citizen because we know how valuable your data assets are. Our team follows and executes security best practices across each layer of our product.
5
5
6
6
## Tobiko Cloud Standard Deployment
7
7
8
-
Our standard Tobiko Cloud deployment consists of several components that are each responsible for different parts of the product.
8
+
Our standard Tobiko Cloud deployment consists of several components that are each responsible for different parts of the product.
9
9
10
-
Below is a diagram of the components along with their descriptions.
10
+
Below is a diagram of the components along with their descriptions.
-**Scheduler**: Orchestrates schedule cadence and hosts state metadata (code versions, logs, cost)
15
15
-**Executor**: Applies code changes and runs SQL queries (actual data processing in SQL Engine) and Python models in proper DAG order.
@@ -18,29 +18,29 @@ Below is a diagram of the components along with their descriptions.
18
18
19
19
## Tobiko Cloud Hybrid Deployment
20
20
21
-
For some customers, our hybrid deployment option is a great fit. It provides a seamless experience with Tobiko Cloud but within your own VPC and infrastructure.
21
+
For some customers, our hybrid deployment option is a great fit. It provides a seamless experience with Tobiko Cloud but within your own VPC and infrastructure.
22
22
23
-
In a hybrid deployment, Tobiko Cloud does not execute tasks directly with the engine. Instead, it passes tasks to the executors hosted in your environment, which then execute the tasks with the engine.
23
+
In a hybrid deployment, Tobiko Cloud does not execute tasks directly with the engine. Instead, it passes tasks to the executors hosted in your environment, which then execute the tasks with the engine.
24
24
25
25
Executors are Docker containers that connect to both Tobiko Cloud and your SQL engine. They pull work tasks from the Tobiko Cloud scheduler and execute them with your SQL engine. This is a pull-only mechanism authenticated through an OAuth Client ID/Secret. Whitelist IPs in your network to allow reaching Tobiko Cloud IPs from the executor: 34.28.17.91, 34.136.27.153, 34.136.131.20
26
26
27
-
Below is a diagram of the components along with their description.
27
+
Below is a diagram of the components along with their description.
-**Scheduler**: Orchestrates schedule cadence and hosts state metadata (code versions, logs, cost). **Never pushes** instructions to executor.
32
32
-**Executor**: Appplies code changes and runs SQL queries and Python models in proper DAG order (actual data processing in SQL Engine)
33
33
-**Gateway**: Stores authentication credentials for SQL Engine. Secured through your secrets manager or Kubernetes Secrets.
34
34
-**SQL Engine**: Processes and stores data based on the above instructions
35
-
-**Executor -> Scheduler**: A pull-only mechanism for obtaining work tasks.
35
+
-**Executor -> Scheduler**: A pull-only mechanism for obtaining work tasks.
36
36
-**Helm Chart**: For production environements, we provide a [Helm chart](../scheduler/hybrid_executors_helm.md) that includes robust configurability, secret management, and scaling options.
37
37
-**Docker Compose**: For simpler environments or testing, we offer a [Docker Compose setup](../scheduler/hybrid_executors_docker_compose.md) to quickly deploy executors on any machine with Docker.
38
38
39
39
40
40
41
41
## Internal Code Practices
42
42
43
-
We enforce coding standards throughout Tobiko to write, maintain, and collaborate on code effectively. These practice ensure consistency, maintainability, reliability, and most importantly, trust.
43
+
We enforce coding standards throughout Tobiko to write, maintain, and collaborate on code effectively. These practice ensure consistency, maintainability, reliability, and most importantly, trust.
44
44
45
45
A few key components of our internal code requirements:
46
46
@@ -49,19 +49,19 @@ A few key components of our internal code requirements:
49
49
- We sign commits and register the key with GitHub ([Github Docs](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits)).
50
50
- Binaries are signed using cosign and OIDC for keyless ([Signing docs](https://docs.sigstore.dev/cosign/signing/overview/)).
51
51
- Attestations are created to certify an image, enforced with GCP Binary Authorization ([Attestation docs](https://cloud.google.com/binary-authorization/docs/key-concepts#attestations)).
52
-
- Encryption is a key feature of our security posture and is enforced at each stage of access. For example, the state database automatically encrypts all data. Credentials are also securely encrypted and stored.
52
+
- Encryption is a key feature of our security posture and is enforced at each stage of access. For example, the state database automatically encrypts all data. Credentials are also securely encrypted and stored.
53
53
- We back up each state database nightly and before upgrades. These backups are stored for 14 days.
54
54
55
55
## Penetration Testing
56
56
57
57
At least once a year, Tobiko engages a third-party security firm to perform a penetration test. This test evaluates our systems by identifying and attempting to exploit known vulnerabilities, focusing on critical external and/or internal assets. A detailed report is available upon request.
58
-
59
58
60
-
## Asset and Access Management
59
+
60
+
## Asset and Access Management
61
61
62
62
### How do we protect PGP keys?
63
63
64
-
If an employee loses their laptop, we don't need to get the old PGP key back because we can invalidate the key directly.
64
+
If an employee loses their laptop, we don't need to get the old PGP key back because we can invalidate the key directly.
65
65
66
66
We use GitHub to sign code commits. At the time the code was committed, the PGP key was valid. When an employee loses their laptop, we will invalidate it, and they will regenerate a new key to use in future commits. The old commits are still valid because the PGP key was valid at the time the commit was made.
67
67
@@ -77,4 +77,3 @@ We would revoke access for the GitHub user account associated with the compromis
77
77
- We follow a formal IT asset disposal procedure to prevent key compromise through improper hardware disposal.
78
78
- See above for PGP key protection.
79
79
- Binaries are signed using Cosign and OIDC for keyless signing.
0 commit comments