From 8b6834dae434ccd2ed38c74d43985220508b3aba Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Wed, 21 Jan 2026 12:25:18 -0500 Subject: [PATCH 1/2] 2 step flow is bkac! --- .../build/connectors/data-source/athena.md | 184 +++++++++++++--- .../build/connectors/data-source/bigquery.md | 191 +++++++++++++---- .../build/connectors/data-source/mysql.md | 120 +++++++++-- .../build/connectors/data-source/postgres.md | 185 ++++++++++++++-- .../build/connectors/data-source/redshift.md | 188 ++++++++++++---- .../build/connectors/data-source/snowflake.md | 200 ++++++++++++++++-- 6 files changed, 915 insertions(+), 153 deletions(-) diff --git a/docs/docs/build/connectors/data-source/athena.md b/docs/docs/build/connectors/data-source/athena.md index 02fc25459b5..91c37d1b140 100644 --- a/docs/docs/build/connectors/data-source/athena.md +++ b/docs/docs/build/connectors/data-source/athena.md @@ -11,70 +11,180 @@ sidebar_position: 0 [Amazon Athena](https://docs.aws.amazon.com/athena/) is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. It is serverless, so there is no infrastructure to manage, and you pay only for the queries you run, making it cost-effective for a wide range of data analysis tasks. Athena is designed for quick, ad-hoc querying of large datasets, enabling businesses to easily integrate it into their analytics and business intelligence tools for immediate insights from their data stored in S3. Rill supports natively connecting to and reading from Athena as a source by leveraging the [AWS SDK for Go](https://aws.github.io/aws-sdk-go-v2/docs/). +## Authentication Methods -## Connect to Athena +To connect to Amazon Athena, you need to provide authentication credentials. Rill supports two methods: -To connect to Amazon Athena, you need to provide authentication credentials. You have two options: - -1. **Use Access Key/Secret Key** (recommended for cloud deployment) +1. **Use Access Key/Secret Key** (recommended for production) 2. **Use Local AWS credentials** (local development only - not recommended for production) +:::tip Authentication Methods Choose the method that best fits your setup. For production deployments to Rill Cloud, use Access Key/Secret Key. Local AWS credentials only work for local development and will cause deployment failures. +::: + +## Using the Add Data UI + +When you add an Athena data model through the Rill UI, the process follows two steps: + +1. **Configure Authentication** - Set up your Athena connector with AWS credentials (Access Key/Secret Key) +2. **Configure Data Model** - Define which database, table, or query to execute + +This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable. + +--- + +## Method 1: Access Key and Secret Key (Recommended) -### Access Key and Secret Key +Access Key and Secret Key credentials provide the most reliable authentication for Athena. This method works for both local development and Rill Cloud deployments. -Create a connector with your credentials to connect to Athena. Here's an example connector configuration file you can copy into your `connectors` directory to get started: +### Using the UI + +1. Click **Add Data** in your Rill project +2. Select **Amazon Athena** as the data source type +3. In the authentication step: + - Enter your AWS Access Key ID + - Enter your AWS Secret Access Key + - Specify the output location (S3 bucket for query results) + - Specify the AWS region +4. In the data model configuration step: + - Enter your SQL query + - Configure other model settings as needed +5. Click **Create** to finalize + +The UI will automatically create both the connector file and model file for you. + +### Manual Configuration + +If you prefer to configure manually, create two files: + +**Step 1: Create connector configuration** + +Create `connectors/my_athena.yaml`: ```yaml type: connector - driver: athena + aws_access_key_id: "{{ .env.connector.athena.aws_access_key_id }}" aws_secret_access_key: "{{ .env.connector.athena.aws_secret_access_key }}" -output_location: "s3://bucket/path/folder" +output_location: "s3://my-bucket/athena-results/" region: "us-east-1" ``` -:::tip Using the Add Data Form -You can also use the Add Data form in Rill Developer, which will automatically create the `athena.yaml` file and populate the `.env` file with `connector.athena.aws_access_key_id` and `connector.athena.aws_secret_access_key`. +**Step 2: Create model configuration** + +Create `models/my_athena_data.yaml`: + +```yaml +type: model +connector: my_athena + +sql: SELECT * FROM my_database.my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.athena.aws_access_key_id=AKIAIOSFODNN7EXAMPLE +connector.athena.aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +``` + +:::tip Did you know? +If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. ::: -### Local AWS Credentials (Local Development Only) +--- + +## Method 2: Local AWS Credentials + +For local development, you can use credentials from the AWS CLI. This method is **not suitable for production** or Rill Cloud deployments. :::warning Not recommended for production -Local AWS credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Use Method 1 above for production deployments. +Local AWS credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Always use Access Key/Secret Key for production deployments. ::: -When using Rill Developer on your local machine (i.e., `rill start`), Rill can either use the credentials configured in your local environment using the AWS CLI or use the explicitly set credentials in a [connector](/reference/project-files/connectors#athena) file. +### Setup -To check if you already have the AWS CLI installed and authenticated, open a terminal window and run: -```bash -aws iam get-user --no-cli-pager +1. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) if not already installed +2. Authenticate with your AWS account: + - If your organization has SSO configured, reach out to your admin for instructions on how to authenticate using `aws sso login` + - Otherwise, run `aws configure` and provide your access key, secret, and default region +3. Verify your authentication: + ```bash + aws iam get-user --no-cli-pager + ``` + +### Connector Configuration + +Create `connectors/my_athena.yaml`: + +```yaml +type: connector +driver: athena + +output_location: "s3://my-bucket/athena-results/" +region: "us-east-1" ``` -If it prints information about your user, there is nothing more to do. Rill will be able to connect to any existing Athena instances that your user has privileges to access. -If you do not have the AWS CLI installed and authenticated, follow these steps: +### Model Configuration -1. Open a terminal window and [install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) if it is not already installed on your system. +Create `models/my_athena_data.yaml`: -2. If your organization has SSO configured, reach out to your admin for instructions on how to authenticate using `aws sso login`. +```yaml +type: model +connector: my_athena -3. If your organization does not have SSO configured: +sql: SELECT * FROM my_database.my_table - a. Follow the steps described in [How to create an AWS service account using the AWS Management Console](./s3#how-to-create-an-aws-service-account-using-the-aws-management-console), which you will find below on this page. +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` - b. Run the following command and provide the access key, access secret, and default region when prompted (you can leave the "Default output format" blank): - ``` - aws configure - ``` +When no explicit credentials are provided in the connector, Rill will automatically use your local AWS CLI credentials. -You have now configured AWS access from your local environment. Rill will detect and use your credentials the next time you try to ingest a source. +--- -:::tip Did you know? +## Using Athena Data in Models -If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. +Once your connector is configured, you can reference Athena tables and run queries in your model configurations. -::: +### Basic Example + +```yaml +type: model +connector: my_athena + +sql: SELECT * FROM my_database.my_table + +refresh: + cron: "0 */6 * * *" +``` + +### Custom SQL Query + +```yaml +type: model +connector: my_athena + +sql: | + SELECT + date_trunc('day', event_time) as event_date, + event_type, + COUNT(*) as event_count + FROM my_database.events + WHERE event_time >= date_add('day', -30, current_date) + GROUP BY 1, 2 + +refresh: + cron: "0 */6 * * *" +``` + +--- ## Separating Dev and Prod Environments @@ -82,6 +192,8 @@ When ingesting data locally, consider setting parameters in your connector file For more details, see our [Dev/Prod setup docs](/build/connectors/templating). +--- + ## Deploy to Rill Cloud When deploying a project to Rill Cloud, Rill requires you to explicitly provide an access key and secret for an AWS service account with access to Athena used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#athena) for more information. @@ -92,17 +204,23 @@ rill env push ``` :::tip Did you know? - If you've already configured credentials locally (in your `/.env` file), you can use `rill env push` to [push these credentials](/build/connectors/credentials#rill-env-push) to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running `rill env pull`. - ::: +--- + ## Appendix -### Athena/S3 permissions +### Athena/S3 Permissions + The Athena connector performs the following AWS queries while ingesting data from Athena: + 1. Athena: [`GetWorkGroup`](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetWorkGroup.html) to determine an output location if not specified explicitly. 2. S3: [`ListObjects`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) to identify files unloaded by Athena. 3. S3: [`GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) to ingest files unloaded by Athena. Make sure your account or service account has the corresponding permissions to perform these requests. + +### How to Create an AWS Service Account + +For detailed instructions on creating an AWS service account with the appropriate permissions, see the [S3 connector documentation](./s3#how-to-create-an-aws-service-account-using-the-aws-management-console). diff --git a/docs/docs/build/connectors/data-source/bigquery.md b/docs/docs/build/connectors/data-source/bigquery.md index 4cd459ee94f..4e21bf50c23 100644 --- a/docs/docs/build/connectors/data-source/bigquery.md +++ b/docs/docs/build/connectors/data-source/bigquery.md @@ -1,5 +1,5 @@ --- -title: BigQuery +title: BigQuery description: Connect to data in BigQuery sidebar_label: BigQuery sidebar_position: 10 @@ -11,90 +11,205 @@ sidebar_position: 10 [Google BigQuery](https://cloud.google.com/bigquery/docs) is a fully managed, serverless data warehouse that enables scalable and cost-effective analysis of large datasets using SQL-like queries. It supports a highly scalable and flexible architecture, allowing users to analyze large amounts of data in real time, making it suitable for BI/ML applications. Rill supports natively connecting to and reading from BigQuery as a source by leveraging the [BigQuery SDK](https://cloud.google.com/bigquery/docs/reference/libraries). +## Authentication Methods +To connect to Google BigQuery, you need to provide authentication credentials. Rill supports two methods: -## Connect to BigQuery - -To connect to Google BigQuery, you need to provide authentication credentials. You have two options: - -1. **Use Service Account JSON** (recommended for cloud deployment) +1. **Use Service Account JSON** (recommended for production) 2. **Use Local Google Cloud CLI credentials** (local development only - not recommended for production) +:::tip Authentication Methods Choose the method that best fits your setup. For production deployments to Rill Cloud, use Service Account JSON. Local Google Cloud CLI credentials only work for local development and will cause deployment failures. +::: -### Service Account JSON +## Using the Add Data UI -We recommend using Service Account JSON for authentication as it makes deployment to Rill Cloud easier. The `GOOGLE_APPLICATION_CREDENTIALS` environment variable tells Google Cloud SDK which service account key file to use for authentication. +When you add a BigQuery data model through the Rill UI, the process follows two steps: -Create your Service Account JSON with the following command: +1. **Configure Authentication** - Set up your BigQuery connector with credentials (Service Account JSON) +2. **Configure Data Model** - Define which dataset, table, or query to execute -```bash -gcloud iam service-accounts keys create ~/key.json \ - --iam-account=my-service-account@PROJECT_ID.iam.gserviceaccount.com -``` +This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable. -:::note Permission denied? -You'll need to contact your internal cloud admin to create your Service Account JSON credentials for you. -::: +--- + +## Method 1: Service Account JSON (Recommended) + +Service Account JSON credentials provide the most secure and reliable authentication for BigQuery. This method works for both local development and Rill Cloud deployments. +### Using the UI -Create a connector with your credentials to connect to BigQuery. Here's an example connector configuration file you can copy into your `connectors` directory to get started. The UI will also populate your `.env` with `connector.bigquery.google_application_credentials`. +1. Click **Add Data** in your Rill project +2. Select **Google BigQuery** as the data source type +3. In the authentication step: + - Upload your JSON key file or paste its contents + - Specify your Google Cloud Project ID +4. In the data model configuration step: + - Enter your SQL query + - Configure other model settings as needed +5. Click **Create** to finalize + +The UI will automatically create both the connector file and model file for you. + +### Manual Configuration + +If you prefer to configure manually, create two files: + +**Step 1: Create connector configuration** + +Create `connectors/my_bigquery.yaml`: ```yaml type: connector - driver: bigquery google_application_credentials: "{{ .env.connector.bigquery.google_application_credentials }}" -project_id: "rilldata" +project_id: "my-gcp-project" ``` -:::tip Did you know? +**Step 2: Create model configuration** -If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials/#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. +Create `models/my_bigquery_data.yaml`: + +```yaml +type: model +connector: my_bigquery +sql: SELECT * FROM my_dataset.my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.bigquery.google_application_credentials= +``` + +:::tip Did you know? +If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials/#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. ::: +--- + +## Method 2: Local Google Cloud CLI Credentials -### Local Google Cloud CLI Credentials (Local Development Only) +For local development, you can use credentials from the Google Cloud CLI. This method is **not suitable for production** or Rill Cloud deployments. :::warning Not recommended for production -Local Google Cloud CLI credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Use Method 1 above for production deployments. +Local Google Cloud CLI credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Always use Service Account JSON for production deployments. ::: -Follow these steps to configure your local environment credentials: +### Setup -1. Open a terminal window and run `gcloud auth list` to check if you already have the Google Cloud CLI installed and authenticated. -2. If it does not print information about your user, follow the steps on [Install the Google Cloud CLI](https://cloud.google.com/sdk/docs/install-sdk). Make sure to run `gcloud init` after installation as described in the tutorial. -3. **Important**: Run `gcloud auth application-default login` to set up Application Default Credentials (ADC). If you skip this step, the app will error with missing `GOOGLE_APPLICATION_CREDENTIALS`. +1. Install the [Google Cloud CLI](https://cloud.google.com/sdk/docs/install-sdk) if not already installed +2. Initialize and authenticate: + ```bash + gcloud init + ``` +3. **Important**: Set up Application Default Credentials (ADC): + ```bash + gcloud auth application-default login + ``` :::tip Service Accounts -If you are using a service account, you will need to run the following command instead: -``` +If you are using a service account, run the following command instead: +```bash gcloud auth activate-service-account --key-file=path_to_json_key_file ``` ::: -You have now configured Google Cloud access from your local environment. Rill will detect and use your credentials the next time you try to ingest a source. +### Connector Configuration -## Deploy to Rill Cloud +Create `connectors/my_bigquery.yaml`: -When deploying a project to Rill Cloud, Rill requires you to explicitly provide a JSON key file for a Google Cloud service account with access to BigQuery used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#bigquery) for more information. +```yaml +type: connector +driver: bigquery -If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI: +project_id: "my-gcp-project" ``` -rill env push + +### Model Configuration + +Create `models/my_bigquery_data.yaml`: + +```yaml +type: model +connector: my_bigquery + +sql: SELECT * FROM my_dataset.my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +When no explicit credentials are provided in the connector, Rill will automatically use your local Google Cloud CLI credentials. + +--- + +## Using BigQuery Data in Models + +Once your connector is configured, you can reference BigQuery tables and run queries in your model configurations. + +### Basic Example + +```yaml +type: model +connector: my_bigquery + +sql: SELECT * FROM my_dataset.my_table + +refresh: + cron: "0 */6 * * *" +``` + +### Custom SQL Query + +```yaml +type: model +connector: my_bigquery + +sql: | + SELECT + DATE(created_at) as event_date, + event_type, + COUNT(*) as event_count + FROM my_dataset.events + WHERE created_at >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) + GROUP BY 1, 2 + +refresh: + cron: "0 */6 * * *" ``` +--- + ## Separating Dev and Prod Environments When ingesting data locally, consider setting parameters in your connector file to limit how much data is retrieved, since costs can scale with the data source. This also helps other developers clone the project and iterate quickly by reducing ingestion time. For more details, see our [Dev/Prod setup docs](/build/connectors/templating). +--- + +## Deploy to Rill Cloud + +When deploying a project to Rill Cloud, Rill requires you to explicitly provide a JSON key file for a Google Cloud service account with access to BigQuery used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#bigquery) for more information. + +If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI: +``` +rill env push +``` + +--- + ## Appendix -### How to create a service account using the Google Cloud Console +### How to Create a Service Account Using the Google Cloud Console Here is a step-by-step guide on how to create a Google Cloud service account with access to BigQuery: @@ -104,7 +219,7 @@ Here is a step-by-step guide on how to create a Google Cloud service account wit 3. In the "Create Service Account" window, enter a name for the service account, then click "Create and continue". -4. In the "Role" field, search for and select the following [BigQuery roles](https://cloud.google.com/bigquery/docs/access-control): +4. In the "Role" field, search for and select the following [BigQuery roles](https://cloud.google.com/bigquery/docs/access-control): - [roles/bigquery.dataViewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer) (Lowest-level resources: Table, View) - Provides the ability to read data and metadata from the project's datasets/dataset's tables/table or view. - [roles/bigquery.readSessionUser](https://cloud.google.com/bigquery/docs/access-control#bigquery.readSessionUser) (Lowest-level resources: Project) @@ -123,3 +238,7 @@ Here is a step-by-step guide on how to create a Google Cloud service account wit 7. Choose the "JSON" key type and click "Create". 8. Download and save the JSON key file to a secure location on your computer. + +:::note Permission denied? +You'll need to contact your internal cloud admin to create your Service Account JSON credentials for you. +::: diff --git a/docs/docs/build/connectors/data-source/mysql.md b/docs/docs/build/connectors/data-source/mysql.md index f10ff51be78..ee380d34af4 100644 --- a/docs/docs/build/connectors/data-source/mysql.md +++ b/docs/docs/build/connectors/data-source/mysql.md @@ -11,7 +11,9 @@ sidebar_position: 40 [MySQL](https://dev.mysql.com/doc/refman/8.0/en/introduction.html) is an open-source relational database management system (RDBMS) known for its reliability, performance, and ease of use. It is widely used for a variety of applications, from small to large enterprise projects, supporting structured data storage, retrieval, and management with SQL queries. MySQL offers a comprehensive ecosystem with support for advanced features such as replication, transactions, and full-text indexing, making it a versatile choice for integrating with BI tools. You can connect to and read from MySQL databases directly. -When connecting to MySQL, you need to specify an appropriate Data Source Name (DSN) in the connector's configuration using the following syntax: +## Connection String Format + +When connecting to MySQL, you need to specify an appropriate Data Source Name (DSN) using the following syntax: ```bash ://:@:/ @@ -24,25 +26,115 @@ When connecting to MySQL, you need to specify an appropriate Data Source Name (D For more details, see the [MySQL documentation on DSN formats](https://dev.mysql.com/doc/refman/8.4/en/connecting-using-uri-or-key-value-pairs.html#connecting-using-uri). +## Using the Add Data UI + +When you add a MySQL data model through the Rill UI, the process follows two steps: + +1. **Configure Authentication** - Set up your MySQL connector with connection credentials (host, port, user, password, database) +2. **Configure Data Model** - Define which table or query to execute + +This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable. + +--- + ## Connect to MySQL -Create a connector with your credentials to connect to MySQL. Here's an example connector configuration file you can copy into your `connectors` directory to get started. +### Using the UI + +1. Click **Add Data** in your Rill project +2. Select **MySQL** as the data source type +3. In the authentication step: + - Enter your MySQL host and port + - Enter your database name + - Enter your username and password + - Configure SSL mode if needed +4. In the data model configuration step: + - Enter your SQL query + - Configure other model settings as needed +5. Click **Create** to finalize + +The UI will automatically create both the connector file and model file for you. + +### Manual Configuration + +If you prefer to configure manually, create two files: + +**Step 1: Create connector configuration** + +Create `connectors/my_mysql.yaml`: ```yaml -type: connector -driver: mysql +type: connector +driver: mysql host: "localhost" -port: 3306 -database: "mydatabase" -user: "myusername" +port: 3306 +database: "mydatabase" +user: "myusername" password: "{{ .env.connector.mysql.password }}" -ssl_mode: "DISABLED" +ssl_mode: "DISABLED" +``` + +**Step 2: Create model configuration** + +Create `models/my_mysql_data.yaml`: + +```yaml +type: model +connector: my_mysql + +sql: SELECT * FROM my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.mysql.password=your-secure-password ``` -:::tip Using the Add Data Form -You can also use the Add Data form in Rill Developer, which will automatically create the `mysql.yaml` file and populate the `.env` file with `connector.mysql.*` parameters based on the parameters or connection string you provide. -::: +--- + +## Using MySQL Data in Models + +Once your connector is configured, you can reference MySQL tables and run queries in your model configurations. + +### Basic Example + +```yaml +type: model +connector: my_mysql + +sql: SELECT * FROM customers + +refresh: + cron: "0 */6 * * *" +``` + +### Custom SQL Query + +```yaml +type: model +connector: my_mysql + +sql: | + SELECT + DATE(created_at) as order_date, + status, + COUNT(*) as order_count, + SUM(total_amount) as total_revenue + FROM orders + WHERE created_at >= DATE_SUB(CURDATE(), INTERVAL 30 DAY) + GROUP BY 1, 2 + +refresh: + cron: "0 */6 * * *" +``` + +--- ## Separating Dev and Prod Environments @@ -50,11 +142,13 @@ When ingesting data locally, consider setting parameters in your connector file For more details, see our [Dev/Prod setup docs](/build/connectors/templating). +--- + ## Deploy to Rill Cloud -When deploying a project to Rill Cloud, Rill requires you to explicitly provide the MySQL connection string used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#mysql) for more information. +When deploying a project to Rill Cloud, Rill requires you to explicitly provide the MySQL connection credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#mysql) for more information. If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI: ``` rill env push -``` \ No newline at end of file +``` diff --git a/docs/docs/build/connectors/data-source/postgres.md b/docs/docs/build/connectors/data-source/postgres.md index bf60ff50f20..5d5fa51fb72 100644 --- a/docs/docs/build/connectors/data-source/postgres.md +++ b/docs/docs/build/connectors/data-source/postgres.md @@ -11,35 +11,184 @@ sidebar_position: 50 [PostgreSQL](https://www.postgresql.org/docs/current/intro-whatis.html) is an open-source object-relational database system known for its reliability, feature robustness, and performance. With support for advanced data types, full ACID compliance for transactional integrity, and an extensible architecture, PostgreSQL provides a highly scalable environment for managing diverse datasets, ranging from small applications to large-scale data warehouses. Its extensive SQL compliance, support for various programming languages, and strong community backing make it a versatile choice for a wide range of business intelligence and analytical applications. You can connect to and read from PostgreSQL databases using either a [supported connection string](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) or [connection URI syntax](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS). -As an example of a connection string: +## Authentication Methods + +To connect to PostgreSQL, you need to provide database connection credentials. Rill supports two approaches: + +1. **Use Individual Parameters** (recommended for clarity) +2. **Use Connection String/URI** (alternative format) + +:::tip Authentication Methods +Choose the method that best fits your setup. Both methods work for local development and Rill Cloud deployments. +::: + +## Using the Add Data UI + +When you add a PostgreSQL data model through the Rill UI, the process follows two steps: + +1. **Configure Authentication** - Set up your PostgreSQL connector with database credentials +2. **Configure Data Model** - Define which table or query to execute + +This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable. + +--- + +## Method 1: Individual Parameters (Recommended) + +Using individual parameters provides clear, readable configuration for your PostgreSQL connection. + +### Using the UI + +1. Click **Add Data** in your Rill project +2. Select **PostgreSQL** as the data source type +3. In the authentication step: + - Enter your host (e.g., `localhost` or your database server address) + - Enter the port (default: `5432`) + - Enter your database name + - Enter your username + - Enter your password +4. In the data model configuration step: + - Enter your SQL query + - Configure other model settings as needed +5. Click **Create** to finalize + +The UI will automatically create both the connector file and model file for you. + +### Manual Configuration + +If you prefer to configure manually, create two files: + +**Step 1: Create connector configuration** + +Create `connectors/my_postgres.yaml`: + +```yaml +type: connector +driver: postgres + +host: "localhost" +port: "5432" +user: "postgres" +password: "{{ .env.connector.postgres.password }}" +dbname: "my_database" +``` + +**Step 2: Create model configuration** + +Create `models/my_postgres_data.yaml`: + +```yaml +type: model +connector: my_postgres + +sql: SELECT * FROM my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.postgres.password=your_password_here +``` + +:::tip Did you know? +If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. +::: + +--- + +## Method 2: Connection String/URI + +You can also use a connection string or URI format to configure your PostgreSQL connection. + +### Connection String Format + ```bash host=localhost port=5432 dbname=postgres_db user=postgres_user password=postgres_pass ``` -Using the same example, this would be an equivalent connection URI: +### Connection URI Format + ```bash postgresql://postgres_user:postgres_pass@localhost:5432/postgres_db ``` +### Manual Configuration -## Connect to PostgreSQL +**Step 1: Create connector configuration** -Create a connector with your credentials to connect to PostgreSQL. Here's an example connector configuration file you can copy into your `connectors` directory to get started: +Create `connectors/my_postgres.yaml`: ```yaml type: connector - driver: postgres -host: "localhost" -port: "5432" -user: "postgres" -password: "{{ .env.connector.postgres.password }}" -dbname: "postgres" + +database_url: "{{ .env.connector.postgres.database_url }}" ``` -:::tip Using the Add Data Form -You can also use the Add Data form in Rill Developer, which will automatically create the `postgres.yaml` file and populate the `.env` file with `connector.postgres.*` parameters based on the parameters or connection string you provide. -::: +**Step 2: Create model configuration** + +Create `models/my_postgres_data.yaml`: + +```yaml +type: model +connector: my_postgres + +sql: SELECT * FROM my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.postgres.database_url=postgresql://postgres_user:postgres_pass@localhost:5432/postgres_db +``` + +--- + +## Using PostgreSQL Data in Models + +Once your connector is configured, you can reference PostgreSQL tables and run queries in your model configurations. + +### Basic Example + +```yaml +type: model +connector: my_postgres + +sql: SELECT * FROM my_table + +refresh: + cron: "0 */6 * * *" +``` + +### Custom SQL Query + +```yaml +type: model +connector: my_postgres + +sql: | + SELECT + date_trunc('day', created_at) as event_date, + status, + COUNT(*) as order_count, + SUM(total_amount) as revenue + FROM orders + WHERE created_at >= NOW() - INTERVAL '30 days' + GROUP BY 1, 2 + +refresh: + cron: "0 */6 * * *" +``` + +--- ## Separating Dev and Prod Environments @@ -47,11 +196,17 @@ When ingesting data locally, consider setting parameters in your connector file For more details, see our [Dev/Prod setup docs](/build/connectors/templating). +--- + ## Deploy to Rill Cloud -When deploying a project to Rill Cloud, Rill requires you to explicitly provide the PostgreSQL connection string used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#postgres) for more information. +When deploying a project to Rill Cloud, Rill requires you to explicitly provide the PostgreSQL connection credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#postgres) for more information. If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI: ``` rill env push -``` \ No newline at end of file +``` + +:::tip Did you know? +If you've already configured credentials locally (in your `/.env` file), you can use `rill env push` to [push these credentials](/build/connectors/credentials#rill-env-push) to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running `rill env pull`. +::: diff --git a/docs/docs/build/connectors/data-source/redshift.md b/docs/docs/build/connectors/data-source/redshift.md index 3267001a42a..49fdb8f7f98 100644 --- a/docs/docs/build/connectors/data-source/redshift.md +++ b/docs/docs/build/connectors/data-source/redshift.md @@ -11,66 +11,179 @@ sidebar_position: 55 [Amazon Redshift](https://docs.aws.amazon.com/redshift/) is a fully managed, petabyte-scale data warehouse service in the cloud, offering fast query and I/O performance for data analysis applications. It enables users to run complex analytical queries against structured data using SQL, ETL processes, and BI tools, leveraging massively parallel processing (MPP) to efficiently handle large volumes of data. Redshift's architecture is designed for high performance on large datasets, supporting data warehousing and analytics of all sizes, making it a pivotal component in a modern data-driven decision-making ecosystem. By leveraging the AWS SDK for Go and utilizing intermediary Parquet files in S3 (to ensure performance), you can connect to and read from Redshift data warehouses. +## Authentication Methods -## Connect to Redshift +To connect to Amazon Redshift, you need to provide authentication credentials. Rill supports two methods: -To connect to Amazon Redshift, you need to provide authentication credentials. You have two options: - -1. **Use Access Key/Secret Key** (recommended for cloud deployment) +1. **Use Access Key/Secret Key** (recommended for production) 2. **Use Local AWS credentials** (local development only - not recommended for production) +:::tip Authentication Methods Choose the method that best fits your setup. For production deployments to Rill Cloud, use Access Key/Secret Key. Local AWS credentials only work for local development and will cause deployment failures. +::: + +## Using the Add Data UI + +When you add a Redshift data model through the Rill UI, the process follows two steps: + +1. **Configure Authentication** - Set up your Redshift connector with AWS credentials (Access Key/Secret Key) +2. **Configure Data Model** - Define which database, table, or query to execute + +This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable. + +--- + +## Method 1: Access Key and Secret Key (Recommended) -### Access Key and Secret Key +Access Key and Secret Key credentials provide the most reliable authentication for Redshift. This method works for both local development and Rill Cloud deployments. +### Using the UI -Create a connector with your credentials to connect to Redshift. Here's an example connector configuration file you can copy into your `connectors` directory to get started: +1. Click **Add Data** in your Rill project +2. Select **Amazon Redshift** as the data source type +3. In the authentication step: + - Enter your AWS Access Key ID + - Enter your AWS Secret Access Key + - Specify your database name + - Specify the workgroup (for Serverless) or cluster identifier +4. In the data model configuration step: + - Enter your SQL query + - Configure other model settings as needed +5. Click **Create** to finalize +The UI will automatically create both the connector file and model file for you. +### Manual Configuration + +If you prefer to configure manually, create two files: + +**Step 1: Create connector configuration** + +Create `connectors/my_redshift.yaml`: ```yaml type: connector - driver: redshift + aws_access_key_id: "{{ .env.connector.redshift.aws_access_key_id }}" aws_secret_access_key: "{{ .env.connector.redshift.aws_secret_access_key }}" database: "dev" ``` -:::tip Using the Add Data Form -You can also use the Add Data form in Rill Developer, which will automatically create the `redshift.yaml` file and populate the `.env` file with `connector.redshift.aws_access_key_id` and `connector.redshift.aws_secret_access_key`. +**Step 2: Create model configuration** + +Create `models/my_redshift_data.yaml`: + +```yaml +type: model +connector: my_redshift + +sql: SELECT * FROM my_schema.my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.redshift.aws_access_key_id=AKIAIOSFODNN7EXAMPLE +connector.redshift.aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +``` + +:::tip Did you know? +If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. ::: -### Local AWS Credentials (Local Development Only) +--- + +## Method 2: Local AWS Credentials + +For local development, you can use credentials from the AWS CLI. This method is **not suitable for production** or Rill Cloud deployments. :::warning Not recommended for production -Local AWS credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Use Method 1 above for production deployments. +Local AWS credentials only work for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Always use Access Key/Secret Key for production deployments. ::: -When using Rill Developer on your local machine, you can use credentials configured in your local environment using the AWS CLI instead of explicit credentials in the connector. +### Setup -To check if you already have the AWS CLI installed and authenticated, open a terminal window and run: -```bash -aws iam get-user --no-cli-pager +1. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) if not already installed +2. Authenticate with your AWS account: + - If your organization has SSO configured, reach out to your admin for instructions on how to authenticate using `aws sso login` + - Otherwise, run `aws configure` and provide your access key, secret, and default region +3. Verify your authentication: + ```bash + aws iam get-user --no-cli-pager + ``` + +### Connector Configuration + +Create `connectors/my_redshift.yaml`: + +```yaml +type: connector +driver: redshift + +database: "dev" ``` -If it prints information about your user, there is nothing more to do. You'll be able to connect to any existing Redshift databases that your user has privileges to access. -If you do not have the AWS CLI installed and authenticated, follow these steps: +### Model Configuration -1. Open a terminal window and [install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) if it is not already installed on your system. +Create `models/my_redshift_data.yaml`: + +```yaml +type: model +connector: my_redshift -2. If your organization has SSO configured, reach out to your admin for instructions on how to authenticate using `aws sso login`. +sql: SELECT * FROM my_schema.my_table -3. If your organization does not have SSO configured: +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +When no explicit credentials are provided in the connector, Rill will automatically use your local AWS CLI credentials. + +--- + +## Using Redshift Data in Models + +Once your connector is configured, you can reference Redshift tables and run queries in your model configurations. + +### Basic Example + +```yaml +type: model +connector: my_redshift + +sql: SELECT * FROM my_schema.my_table + +refresh: + cron: "0 */6 * * *" +``` - a. Follow the steps described in [How to create an AWS service account using the AWS Management Console](./s3#how-to-create-an-aws-service-account-using-the-aws-management-console), which you will find below on this page. +### Custom SQL Query - b. Run the following command and provide the access key, access secret, and default region when prompted (you can leave the "Default output format" blank): - ``` - aws configure - ``` +```yaml +type: model +connector: my_redshift + +sql: | + SELECT + date_trunc('day', event_time) as event_date, + event_type, + COUNT(*) as event_count, + SUM(revenue) as total_revenue + FROM analytics.events + WHERE event_time >= DATEADD(day, -30, CURRENT_DATE) + GROUP BY 1, 2 + +refresh: + cron: "0 */6 * * *" +``` -You have now configured AWS access from your local environment. Rill will detect and use your credentials the next time you try to ingest a source. +--- ## Separating Dev and Prod Environments @@ -78,6 +191,8 @@ When ingesting data locally, consider setting parameters in your connector file For more details, see our [Dev/Prod setup docs](/build/connectors/templating). +--- + ## Deploy to Rill Cloud When deploying a project to Rill Cloud, Rill requires you to explicitly provide an access key and secret for an AWS service account with access to Redshift used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#redshift) for more information. @@ -87,15 +202,20 @@ If you subsequently add sources that require new credentials (or if you simply e rill env push ``` +:::tip Did you know? +If you've already configured credentials locally (in your `/.env` file), you can use `rill env push` to [push these credentials](/build/connectors/credentials#rill-env-push) to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running `rill env pull`. +::: + +--- + ## Appendix :::warning Check your service account permissions - Your account or service account will need to have the **appropriate permissions** necessary to perform these requests. - ::: -### Redshift Serverless permissions +### Redshift Serverless Permissions + When using **Redshift Serverless**, make sure to associate an [IAM role (that has S3 access)](https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-iam.html) with the Serverless namespace or the Redshift cluster. :::info What happens when Rill is reading from Redshift Serverless? @@ -109,7 +229,7 @@ Our Redshift connector will place temporary files in Parquet format in S3 to hel ::: -### Redshift Cluster permissions +### Redshift Cluster Permissions Similarly, when using **Redshift Cluster**, make sure to associate an [IAM role (that has S3 access)](https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-iam-authentication-access-control.html) with the appropriate Redshift cluster. @@ -124,10 +244,6 @@ Our Redshift connector will place temporary files in Parquet format in S3 to hel ::: -:::warning Check your service account permissions - -Your account or service account will need to have the appropriate permissions necessary to perform these requests. - -::: - +### How to Create an AWS Service Account +For detailed instructions on creating an AWS service account with the appropriate permissions, see the [S3 connector documentation](./s3#how-to-create-an-aws-service-account-using-the-aws-management-console). diff --git a/docs/docs/build/connectors/data-source/snowflake.md b/docs/docs/build/connectors/data-source/snowflake.md index 9b9ef83a687..b728872feb4 100644 --- a/docs/docs/build/connectors/data-source/snowflake.md +++ b/docs/docs/build/connectors/data-source/snowflake.md @@ -1,5 +1,5 @@ --- -title: Snowflake +title: Snowflake description: Connect to data in Snowflake sidebar_label: Snowflake sidebar_position: 75 @@ -17,30 +17,41 @@ Snowflake has issued a [deprecation notice](https://www.snowflake.com/en/blog/bl [Snowflake](https://docs.snowflake.com/en/user-guide-intro) is a cloud-based data platform designed to facilitate data warehousing, data lakes, data engineering, data science, data application development, and data sharing. It separates compute and storage, enabling users to scale up or down instantly without downtime, providing a cost-effective solution for data management. With its unique architecture and support for multi-cloud environments, including AWS, Azure, and Google Cloud Platform, Snowflake offers seamless data integration, secure data sharing across organizations, and real-time access to data insights, making it a common choice to power many business intelligence applications and use cases. You can connect to and read from Snowflake data warehouses using the [Go Snowflake Driver](https://pkg.go.dev/github.com/snowflakedb/gosnowflake). +## Authentication Methods -## Connect to Snowflake +To connect to Snowflake, you need to provide authentication credentials. Rill supports two methods: -Create a connector with your credentials to connect to Snowflake. Here's an example connector configuration file you can copy into your `connectors` directory to get started: +1. **Use Keypair Authentication** (recommended - required for production) +2. **Use Password Authentication** (deprecated by Snowflake) -```yaml -type: connector -driver: snowflake +:::tip Authentication Methods +Snowflake has deprecated single-factor password authentication. We strongly recommend using keypair authentication to ensure uninterrupted service. +::: -dsn: "{{ .env.connector.snowflake.dsn }}" -``` +## Using the Add Data UI -:::tip Using the Add Data Form -You can also use the Add Data form in Rill Developer, which will automatically create the `snowflake.yaml` file and populate the `.env` file with `connector.snowflake.*` parameters based on the parameters or connection string you provide. -::: +When you add a Snowflake data model through the Rill UI, the process follows two steps: +1. **Configure Authentication** - Set up your Snowflake connector with credentials via connection string +2. **Configure Data Model** - Define which database, schema, table, or query to execute +This two-step flow ensures your credentials are securely stored in the connector configuration, while your data model references remain clean and portable. + +--- + +## Method 1: Keypair Authentication (Recommended) + +Keypair authentication provides enhanced security and is the recommended method for connecting to Snowflake. This method works for both local development and Rill Cloud deployments. + +### Connection String Format Use the following syntax when defining a connection string using a private key: ```sql @//?warehouse=&role=&authenticator=SNOWFLAKE_JWT&privateKey= ``` -See the full documentation to set up [private key authentication](#using-keypair-authentication). + +See the [Appendix](#using-keypair-authentication) for detailed instructions on generating and formatting your private key.
@@ -51,12 +62,155 @@ To determine your [Snowflake account identifier](https://docs.snowflake.com/en/u ::: +### Using the UI + +1. Click **Add Data** in your Rill project +2. Select **Snowflake** as the data source type +3. In the authentication step: + - Enter your connection string with keypair authentication +4. In the data model configuration step: + - Enter your SQL query + - Configure other model settings as needed +5. Click **Create** to finalize + +The UI will automatically create both the connector file and model file for you. + +### Manual Configuration + +If you prefer to configure manually, create two files: + +**Step 1: Create connector configuration** + +Create `connectors/my_snowflake.yaml`: + +```yaml +type: connector +driver: snowflake + +dsn: "{{ .env.connector.snowflake.dsn }}" +``` + +**Step 2: Create model configuration** + +Create `models/my_snowflake_data.yaml`: + +```yaml +type: model +connector: my_snowflake + +sql: SELECT * FROM my_database.my_schema.my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.snowflake.dsn=myuser@myaccount/mydb/myschema?warehouse=mywh&role=myrole&authenticator=SNOWFLAKE_JWT&privateKey= +``` + +:::tip Did you know? +If this project has already been deployed to Rill Cloud and credentials have been set for this connector, you can use `rill env pull` to [pull these cloud credentials](/build/connectors/credentials#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source. +::: + +--- + +## Method 2: Password Authentication (Deprecated) + +:::warning Deprecated by Snowflake +Snowflake has deprecated single-factor password authentication. This method may stop working in the future. We strongly recommend migrating to keypair authentication. +::: + +### Connection String Format + +```sql +:@//?warehouse=&role= +``` + +### Manual Configuration + +**Step 1: Create connector configuration** + +Create `connectors/my_snowflake.yaml`: + +```yaml +type: connector +driver: snowflake + +dsn: "{{ .env.connector.snowflake.dsn }}" +``` + +**Step 2: Create model configuration** + +Create `models/my_snowflake_data.yaml`: + +```yaml +type: model +connector: my_snowflake + +sql: SELECT * FROM my_database.my_schema.my_table + +# Add a refresh schedule +refresh: + cron: "0 */6 * * *" +``` + +**Step 3: Add credentials to `.env`** + +```bash +connector.snowflake.dsn=myuser:mypassword@myaccount/mydb/myschema?warehouse=mywh&role=myrole +``` + +--- + +## Using Snowflake Data in Models + +Once your connector is configured, you can reference Snowflake tables and run queries in your model configurations. + +### Basic Example + +```yaml +type: model +connector: my_snowflake + +sql: SELECT * FROM my_database.my_schema.my_table + +refresh: + cron: "0 */6 * * *" +``` + +### Custom SQL Query + +```yaml +type: model +connector: my_snowflake + +sql: | + SELECT + DATE_TRUNC('day', event_time) as event_date, + event_type, + COUNT(*) as event_count, + SUM(revenue) as total_revenue + FROM analytics.events.user_actions + WHERE event_time >= DATEADD(day, -30, CURRENT_DATE()) + GROUP BY 1, 2 + +refresh: + cron: "0 */6 * * *" +``` + +--- + ## Separating Dev and Prod Environments When ingesting data locally, consider setting parameters in your connector file to limit how much data is retrieved, since costs can scale with the data source. This also helps other developers clone the project and iterate quickly by reducing ingestion time. For more details, see our [Dev/Prod setup docs](/build/connectors/templating). +--- + ## Deploy to Rill Cloud When deploying a project to Rill Cloud, Rill requires you to explicitly provide Snowflake credentials via the connection string as a source configuration `dsn` field used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#snowflake) for more information. @@ -66,31 +220,37 @@ If you subsequently add sources that require new credentials (or if you simply e rill env push ``` +:::tip Did you know? +If you've already configured credentials locally (in your `/.env` file), you can use `rill env push` to [push these credentials](/build/connectors/credentials#rill-env-push) to your Rill Cloud project. This will allow other users to retrieve and reuse the same credentials automatically by running `rill env pull`. +::: + +--- + ## Appendix -### Using keypair authentication +### Using Keypair Authentication You can use keypair authentication for enhanced security when connecting to Snowflake as an alternative to password-based authentication, which Snowflake has deprecated. Per the [Snowflake Go Driver](https://github.com/snowflakedb/gosnowflake) specifications, this requires the following changes to the dsn: -- Remove the password -- Add `authenticator=SNOWFLAKE_JWT` -- Add `privateKey=` +- Remove the password +- Add `authenticator=SNOWFLAKE_JWT` +- Add `privateKey=` ```sql @//?warehouse=&role=&authenticator=SNOWFLAKE_JWT&privateKey= ``` -#### Generate a private key +#### Generate a Private Key Please refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth) on how to configure an unencrypted private key to use in Rill. -The Snowflake Go Driver only supports **unencrypted PKCS#8 keys**. Make sure to include the `-nocrypt` flag, as encrypted keys are not supported. You can generate one using: +The Snowflake Go Driver only supports **unencrypted PKCS#8 keys**. Make sure to include the `-nocrypt` flag, as encrypted keys are not supported. You can generate one using: ```bash # Generate a 2048-bit unencrypted PKCS#8 private key openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt ``` -#### Convert the private key to a URL-safe format for the DSN +#### Convert the Private Key to a URL-Safe Format for the DSN After generating the private key, you need to convert it into a URL-safe Base64 format for use in the Snowflake DSN. Run the following command: @@ -112,4 +272,4 @@ Depending on your OS version, above commands may differ slightly. Please check y If using keypair authentication, consider rotating your public key regularly to ensure compliance with security and governance best practices. -::: \ No newline at end of file +::: From 9765f3a77a6f69d077497d564b97b8a49326d2dd Mon Sep 17 00:00:00 2001 From: royendo <67675319+royendo@users.noreply.github.com> Date: Wed, 21 Jan 2026 12:32:10 -0500 Subject: [PATCH 2/2] link to source-model docs --- .../build/connectors/data-source/athena.md | 6 ++--- .../build/connectors/data-source/azure.md | 27 +++++++------------ .../build/connectors/data-source/bigquery.md | 6 ++--- docs/docs/build/connectors/data-source/gcs.md | 20 +++++--------- .../build/connectors/data-source/mysql.md | 6 ++--- .../build/connectors/data-source/postgres.md | 6 ++--- .../build/connectors/data-source/redshift.md | 6 ++--- docs/docs/build/connectors/data-source/s3.md | 13 +++------ .../build/connectors/data-source/snowflake.md | 6 ++--- 9 files changed, 33 insertions(+), 63 deletions(-) diff --git a/docs/docs/build/connectors/data-source/athena.md b/docs/docs/build/connectors/data-source/athena.md index 91c37d1b140..b1a1040cd9b 100644 --- a/docs/docs/build/connectors/data-source/athena.md +++ b/docs/docs/build/connectors/data-source/athena.md @@ -46,12 +46,10 @@ Access Key and Secret Key credentials provide the most reliable authentication f - Enter your AWS Secret Access Key - Specify the output location (S3 bucket for query results) - Specify the AWS region -4. In the data model configuration step: - - Enter your SQL query - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/azure.md b/docs/docs/build/connectors/data-source/azure.md index de4bd94656b..11df04716be 100644 --- a/docs/docs/build/connectors/data-source/azure.md +++ b/docs/docs/build/connectors/data-source/azure.md @@ -59,13 +59,10 @@ Storage Account Key credentials provide reliable authentication for Azure Blob S - Choose **Storage Account Key** - Enter your Storage Account name - Enter your Storage Account Key - - Name your connector (e.g., `my_azure`) -4. In the data model configuration step: - - Enter your container name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration @@ -118,12 +115,11 @@ Connection String provides an alternative authentication method for Azure Blob S 3. In the authentication step: - Choose **Connection String** - Enter your Connection String - - Name your connector (e.g., `my_azure_conn`) -4. In the data model configuration step: - - Enter your container name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. + ### Manual Configuration **Step 1: Create connector configuration** @@ -173,12 +169,11 @@ SAS tokens provide fine-grained access control with specific permissions and exp - Choose **SAS Token** - Enter your Storage Account name - Enter your SAS Token - - Name your connector (e.g., `my_azure_sas`) -4. In the data model configuration step: - - Enter your container name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. + ### Manual Configuration **Step 1: Create connector configuration** @@ -228,12 +223,10 @@ For publicly accessible Azure Blob Storage containers, you don't need to create 3. In the authentication step: - Choose **Public** - The UI will skip connector creation and proceed directly to data model configuration -4. In the data model configuration step: - - Enter your container name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will only create the model file (no connector file is created). +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/bigquery.md b/docs/docs/build/connectors/data-source/bigquery.md index 4e21bf50c23..bb054373041 100644 --- a/docs/docs/build/connectors/data-source/bigquery.md +++ b/docs/docs/build/connectors/data-source/bigquery.md @@ -44,12 +44,10 @@ Service Account JSON credentials provide the most secure and reliable authentica 3. In the authentication step: - Upload your JSON key file or paste its contents - Specify your Google Cloud Project ID -4. In the data model configuration step: - - Enter your SQL query - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/gcs.md b/docs/docs/build/connectors/data-source/gcs.md index 34c179b2a56..e59988be3a0 100644 --- a/docs/docs/build/connectors/data-source/gcs.md +++ b/docs/docs/build/connectors/data-source/gcs.md @@ -49,13 +49,10 @@ Service Account JSON credentials provide the most secure and reliable authentica 3. In the authentication step: - Choose **Service Account JSON** - Upload your JSON key file or paste its contents - - Name your connector (e.g., `my_gcs`) -4. In the data model configuration step: - - Enter your bucket name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration @@ -107,12 +104,11 @@ HMAC keys provide S3-compatible authentication for GCS. This method is useful wh - Choose **HMAC Keys** - Enter your Access Key ID - Enter your Secret Access Key - - Name your connector (e.g., `my_gcs_hmac`) -4. In the data model configuration step: - - Enter your bucket name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. + ### Manual Configuration **Step 1: Create connector configuration** @@ -166,12 +162,10 @@ For publicly accessible GCS buckets, you don't need to create a connector. Simpl 3. In the authentication step: - Choose **Public** - The UI will skip connector creation and proceed directly to data model configuration -4. In the data model configuration step: - - Enter your bucket name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will only create the model file (no connector file is created). +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/mysql.md b/docs/docs/build/connectors/data-source/mysql.md index ee380d34af4..ab6561a2549 100644 --- a/docs/docs/build/connectors/data-source/mysql.md +++ b/docs/docs/build/connectors/data-source/mysql.md @@ -48,12 +48,10 @@ This two-step flow ensures your credentials are securely stored in the connector - Enter your database name - Enter your username and password - Configure SSL mode if needed -4. In the data model configuration step: - - Enter your SQL query - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/postgres.md b/docs/docs/build/connectors/data-source/postgres.md index 5d5fa51fb72..c1c35ba635a 100644 --- a/docs/docs/build/connectors/data-source/postgres.md +++ b/docs/docs/build/connectors/data-source/postgres.md @@ -47,12 +47,10 @@ Using individual parameters provides clear, readable configuration for your Post - Enter your database name - Enter your username - Enter your password -4. In the data model configuration step: - - Enter your SQL query - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/redshift.md b/docs/docs/build/connectors/data-source/redshift.md index 49fdb8f7f98..97d84d5b2e3 100644 --- a/docs/docs/build/connectors/data-source/redshift.md +++ b/docs/docs/build/connectors/data-source/redshift.md @@ -46,12 +46,10 @@ Access Key and Secret Key credentials provide the most reliable authentication f - Enter your AWS Secret Access Key - Specify your database name - Specify the workgroup (for Serverless) or cluster identifier -4. In the data model configuration step: - - Enter your SQL query - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/s3.md b/docs/docs/build/connectors/data-source/s3.md index 8889a6e4cd3..1a1535cc4b9 100644 --- a/docs/docs/build/connectors/data-source/s3.md +++ b/docs/docs/build/connectors/data-source/s3.md @@ -56,13 +56,10 @@ Access Key/Secret Key credentials provide reliable authentication for S3. This m - Choose **Access Key/Secret Key** - Enter your Access Key ID - Enter your Secret Access Key - - Name your connector (e.g., `my_s3`) -4. In the data model configuration step: - - Enter your bucket name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration @@ -167,12 +164,10 @@ For publicly accessible S3 buckets, you don't need to create a connector. Simply 3. In the authentication step: - Choose **Public** - The UI will skip connector creation and proceed directly to data model configuration -4. In the data model configuration step: - - Enter your bucket name and object path - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will only create the model file (no connector file is created). +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration diff --git a/docs/docs/build/connectors/data-source/snowflake.md b/docs/docs/build/connectors/data-source/snowflake.md index b728872feb4..b3e8ddbd532 100644 --- a/docs/docs/build/connectors/data-source/snowflake.md +++ b/docs/docs/build/connectors/data-source/snowflake.md @@ -68,12 +68,10 @@ To determine your [Snowflake account identifier](https://docs.snowflake.com/en/u 2. Select **Snowflake** as the data source type 3. In the authentication step: - Enter your connection string with keypair authentication -4. In the data model configuration step: - - Enter your SQL query - - Configure other model settings as needed +4. In the data model configuration step, enter your SQL query 5. Click **Create** to finalize -The UI will automatically create both the connector file and model file for you. +After the model YAML is generated, you can add additional [model settings](/build/models/source-models) directly to the file. ### Manual Configuration