Update Salesforce Connector To Create Source Models

docs/docs/developers/build/connectors/data-source/salesforce.md

@@ -9,55 +9,118 @@ sidebar_position: 65
 
 ## Overview
 
-[Salesforce](https://www.salesforce.com/) is a leading cloud-based Customer Relationship Management (CRM) platform designed to help businesses connect with and understand their customers better. It offers a comprehensive suite of applications focused on sales, customer service, marketing automation, analytics, and application development. Salesforce enables organizations of all sizes to build stronger relationships with their customers through personalized experiences, streamlined communication, and predictive insights. Rill can ingest data from Salesforce as a source by utilizing the Bulk API, which requires a Salesforce username and password (and, in some cases, a token, depending on the org configuration) to authenticate against a Salesforce org.
+[Salesforce](https://www.salesforce.com/) is a leading cloud-based Customer Relationship Management (CRM) platform. Rill ingests data from Salesforce by issuing SOQL queries against the Bulk API. Authentication uses the Salesforce OAuth 2.0 endpoints exposed by a Connected App (or, for the client credentials and JWT flows, an External Client App).
 
+The Salesforce connector follows the same shape as other warehouse connectors: credentials live in a connector file under `connectors/`, and each query is its own model file under `models/`.
 
-## Local credentials
+## Authentication
 
-When using Rill Developer on your local machine, you will need to provide your credentials via a connector file. We would recommend not using plain text to create your file and instead use the `.env` file. For more details on your connector, see [connector YAML](/reference/project-files/connectors) for more details.
+The Salesforce connector supports three OAuth flows. All three require a [Connected App](https://help.salesforce.com/s/articleView?id=sf.connected_app_overview.htm) in your Salesforce org; the client credentials and JWT flows also accept an [External Client App](https://help.salesforce.com/s/articleView?id=xcloud.ecapps_intro.htm). The flow you choose determines which other fields you need to provide.
 
-:::tip Updating the project environmental variable
+| Flow | Required fields |
+| --- | --- |
+| Username / Password (OAuth) | `username`, `password`, `client_id`, `client_secret` |
+| Client Credentials | `client_id`, `client_secret` |
+| JWT Bearer | `username`, `client_id`, `key` |
+
+The connector picks a flow based on which credentials are populated: JWT wins when `key` is set; otherwise a `username` plus `password` selects the OAuth password flow; otherwise a `client_secret` selects the client credentials flow.
+
+:::note SOAP login is deprecated
+
+Earlier versions of this connector used the SOAP login endpoint when a username and password were supplied. Salesforce is decommissioning the SOAP login endpoint, so the connector now uses the OAuth password flow instead. This requires the Connected App's Client Secret in addition to the existing Client ID.
+
+Note that the OAuth password flow only works with a Connected App; External Client Apps do not support it. The client credentials and JWT flows work with either.
 
-If you've already deployed to Rill Cloud, you can either [push/pull the credential]( /guide/administration/project-settings/variables-and-credentials#pushing-and-pulling-credentials-to--from-rill-cloud-via-the-cli) from the CLI with:
-```
-rill env push
-rill env pull
-```
 :::
 
-Alternatively, you can include the credentials directly in the underlying source YAML by adding the `username` and `password` parameters. For example, your source YAML may contain the following properties (these can also be configured through the UI during source creation):
+### Connector file
+
+Place your credentials in a connector file at `connectors/<name>.yaml`. Reference secret values from `.env`.
+
+#### Username / Password (OAuth)
+
 ```yaml
-type: "model"
-connector: "salesforce"
-endpoint: "login.salesforce.com"
-username: "user@example.com"
-password: "MyPasswordMyToken"
-soql: "SELECT Id, Name, CreatedDate FROM Opportunity"
-sobject: "Opportunity"
+type: connector
+driver: salesforce
+
+endpoint: login.salesforce.com
+username: user@example.com
+password: "{{ .env.connector.salesforce.password }}"
+client_id: "<Client ID>"
+client_secret: "{{ .env.connector.salesforce.client_secret }}"
 ```
 
-:::tip Did you know?
+#### Client Credentials
 
-If this project has already been deployed to Rill Cloud and credentials have been set for this source, you can use `rill env pull` to [pull these cloud credentials](/developers/build/connectors/credentials/#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source.
+```yaml
+type: connector
+driver: salesforce
 
-:::
+endpoint: login.salesforce.com
+client_id: "<Client ID>"
+client_secret: "{{ .env.connector.salesforce.client_secret }}"
+```
 
-## Deploy to Rill Cloud
+#### JWT Bearer
 
-When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors) for more information.
+```yaml
+type: connector
+driver: salesforce
 
-If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI:
+endpoint: login.salesforce.com
+username: user@example.com
+client_id: "<Client ID>"
+key: "{{ .env.connector.salesforce.key }}"
 ```
-rill env push
+
+PEM keys contain newlines, which break `.env` parsing if stored raw. The UI's file picker base64-encodes the uploaded key automatically; when hand-editing `.env`, base64-encode the PEM file yourself:
+
+```sh
+base64 < key.pem | tr -d '\n' >> .env
 ```
 
+Raw PEM written inline in the connector YAML (without an `.env` reference) is also accepted.
 
-:::note
+## Models
 
-Leave the `key` and `client_id` fields blank unless you are using JWT (described in the next section [below](#jwt)).
+A Salesforce model file references the connector by name and supplies the SOQL query. Rill issues the query through the Salesforce [Bulk API 2.0](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/bulk_api_2_0.htm)
 
+```yaml
+type: model
+materialize: true
+
+connector: salesforce
+
+soql: |
+  SELECT Id, Name, CreatedDate
+  FROM Opportunity
+
+output:
+  connector: duckdb
+```
+
+Use `soql:` for the query. `sql:` is also accepted as an alias for parity with other warehouse drivers (the connector explorer in the UI writes the query into `sql:`). Add `queryAll: true` to include soft-deleted records.
+
+SOQL itself does not accept `SELECT *`, but the connector rewrites the `SELECT * FROM <SObject>` shape (which the explorer's "Table" mode emits) into an explicit field list discovered from the SObject's describe response. Compound types like `Address` and `Location` are skipped — their atomic sub-components (e.g. `BillingStreet`, `BillingCity`) are queried instead.
+
+## Local credentials
+
+When using Rill Developer on your local machine, provide credentials via a connector file as shown above. Keep secrets in `.env` rather than the connector YAML. See [connector YAML](/reference/project-files/connectors) for more details.
+
+:::tip Updating the project environmental variable
+
+If you've already deployed to Rill Cloud, you can either [push/pull the credential]( /guide/administration/project-settings/variables-and-credentials#pushing-and-pulling-credentials-to--from-rill-cloud-via-the-cli) from the CLI with:
+```
+rill env push
+rill env pull
+```
 :::
 
-### JWT
+## Deploy to Rill Cloud
+
+When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. See the [connector YAML reference docs](/reference/project-files/connectors) for more information.
 
-Authentication using JWT instead of a password is also supported. Set `client_id` to the **Client Id** (also known as the _Consumer Key_) of the Connected App to use, and set `key` to contain the PEM-formatted private key to use for signing.
+If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), update them by pushing the `Deploy` button or by running:
+```
+rill env push
+```

go.mod

@@ -13,7 +13,7 @@ require (
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1
 	github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4
 	github.com/ClickHouse/clickhouse-go/v2 v2.41.0
-	github.com/ForceCLI/force v1.1.0
+	github.com/ForceCLI/force v1.12.0
 	github.com/Masterminds/sprig/v3 v3.3.0
 	github.com/MicahParks/keyfunc v1.9.0
 	github.com/NYTimes/gziphandler v1.1.1
@@ -200,7 +200,6 @@ require (
 	github.com/BurntSushi/toml v1.4.0 // indirect
 	github.com/ClickHouse/ch-go v0.69.0 // indirect
 	github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e // indirect
-	github.com/ForceCLI/config v0.0.0-20230217143549-9149d42a3c99 // indirect
 	github.com/ForceCLI/inflect v0.0.0-20130829110746-cc00b5ad7a6a // indirect
 	github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 // indirect
 	github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.51.0 // indirect
@@ -209,7 +208,6 @@ require (
 	github.com/Masterminds/semver/v3 v3.4.0 // indirect
 	github.com/Microsoft/go-winio v0.6.2 // indirect
 	github.com/ProtonMail/go-crypto v1.3.0 // indirect
-	github.com/ViViDboarder/gotifier v0.0.0-20140619195515-0f19f3d7c54c // indirect
 	github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect
 	github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
 	github.com/andybalholm/brotli v1.2.0 // indirect
@@ -347,7 +345,6 @@ require (
 	github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
 	github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
-	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
 	github.com/moby/buildkit v0.29.0 // indirect
 	github.com/moby/docker-image-spec v1.3.1 // indirect

go.sum

@@ -682,10 +682,8 @@ github.com/ClickHouse/clickhouse-go/v2 v2.41.0 h1:JbLKMXLEkW0NMalMgI+GYb6FVZtpaM
 github.com/ClickHouse/clickhouse-go/v2 v2.41.0/go.mod h1:/RoTHh4aDA4FOCIQggwsiOwO7Zq1+HxQ0inef0Au/7k=
 github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e h1:rd4bOvKmDIx0WeTv9Qz+hghsgyjikFiPrseXHlKepO0=
 github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e/go.mod h1:blbwPQh4DTlCZEfk1BLU4oMIhLda2U+A840Uag9DsZw=
-github.com/ForceCLI/config v0.0.0-20230217143549-9149d42a3c99 h1:H2axnitaP3Dw+tocMHPQHjM2wJ/+grF8sOIQGaJeEsg=
-github.com/ForceCLI/config v0.0.0-20230217143549-9149d42a3c99/go.mod h1:WHFXv3VIHldTnYGmWAXAxsu4O754A9Zakq4DedI8PSA=
-github.com/ForceCLI/force v1.1.0 h1:e+KhyNeZF3r98YrvE32Bv7bQOCeg7lj1uiihcs3axVE=
-github.com/ForceCLI/force v1.1.0/go.mod h1:S+csSNhBOHrRsRV7PMKhmPP7N4SBLYJMrQzEc/OHLTs=
+github.com/ForceCLI/force v1.12.0 h1:IUsMtyA+l/pzsD30XMv/47qiheMPObFz8YR67zf+Gf4=
+github.com/ForceCLI/force v1.12.0/go.mod h1:vZhTNH4A1cgnFV21igwqwMRy01TXYsquzdXF4LnrZB8=
 github.com/ForceCLI/inflect v0.0.0-20130829110746-cc00b5ad7a6a h1:mMd54YgLoeupNpbph3KdwvF58O0lZ72RQaJ2cFPOFDE=
 github.com/ForceCLI/inflect v0.0.0-20130829110746-cc00b5ad7a6a/go.mod h1:DGKmCfb9oo5BivGO+szHk2ZvlqPDTlW4AYVpRBIVbms=
 github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 h1:DHa2U07rk8syqvCge0QIGMCE1WxGj9njT44GH7zNJLQ=
@@ -750,8 +748,6 @@ github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbt
 github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
 github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
 github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d/go.mod h1:HI8ITrYtUY+O+ZhtlqUnD8+KwNPOyugEhfP9fdUIaEQ=
-github.com/ViViDboarder/gotifier v0.0.0-20140619195515-0f19f3d7c54c h1:qLWjxZGLdzxp0Gc4Sf6f4w15D+wNKZ28HhkV9y5cAhw=
-github.com/ViViDboarder/gotifier v0.0.0-20140619195515-0f19f3d7c54c/go.mod h1:/nH+y85gO3ta3b6JtRWGA5hPIH35XJr/ZHXlfrBRx3A=
 github.com/XSAM/otelsql v0.27.0 h1:i9xtxtdcqXV768a5C6SoT/RkG+ue3JTOgkYInzlTOqs=
 github.com/XSAM/otelsql v0.27.0/go.mod h1:0mFB3TvLa7NCuhm/2nU7/b2wEtsczkj8Rey8ygO7V+A=
 github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d h1:licZJFw2RwpHMqeKTCYkitsPqHNxTmd4SNR5r94FGM8=
@@ -1947,7 +1943,6 @@ github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceT
 github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
 github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
 github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
-github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
 github.com/mitchellh/gox v0.4.0/go.mod h1:Sd9lOJ0+aimLBi73mGofS1ycjY8lL3uZM3JPS42BGNg=

runtime/drivers/salesforce/authentication.go

@@ -1,10 +1,12 @@
 package salesforce
 
 import (
+	"encoding/base64"
 	"errors"
 	"fmt"
 	"net/url"
 	"os"
+	"strings"
 
 	force "github.com/ForceCLI/force/lib"
 )
@@ -17,33 +19,61 @@ type authenticationOptions struct {
 	Password     string
 	JWT          string
 	ConnectedApp string
+	ClientSecret string
 }
 
+// authMode describes which OAuth flow authenticate() will use for a given set
+// of options. It is exported via selectAuthMode for unit testing.
+type authMode int
+
+const (
+	authModeUnknown authMode = iota
+	authModeJWT
+	authModePassword
+	authModeClientCredentials
+)
+
 func authenticate(options authenticationOptions) (*force.Force, error) {
 	if options.ConnectedApp == "" {
 		return nil, fmt.Errorf("connected app client id is required")
 	}
 	force.ClientId = options.ConnectedApp
 
-	if options.Username == "" {
-		return nil, fmt.Errorf("username missing")
-	}
-
-	isJWTSelected := options.JWT != ""
-	isSOAPSelected := options.Password != ""
-
 	endpoint, err := endpoint(options)
 	if err != nil {
 		return nil, err
 	}
 
-	switch {
-	case isJWTSelected:
+	switch selectAuthMode(options) {
+	case authModeJWT:
+		if options.Username == "" {
+			return nil, fmt.Errorf("username is required for JWT authentication")
+		}
 		return jwtLogin(endpoint, options)
-	case isSOAPSelected:
-		return soapLoginAtEndpoint(endpoint, options.Username, options.Password)
+	case authModePassword:
+		if options.ClientSecret == "" {
+			return nil, fmt.Errorf("client_secret is required for username/password authentication")
+		}
+		return passwordFlowLogin(endpoint, options)
+	case authModeClientCredentials:
+		return clientCredentialsLogin(endpoint, options)
+	}
+	return nil, fmt.Errorf("unable to authenticate: provide a JWT key, a username and password (with client_secret), or a client_secret for the client credentials flow")
+}
+
+// selectAuthMode picks an OAuth flow based on which credentials are populated.
+// JWT wins when a key is present; otherwise username+password selects the
+// password flow; otherwise a client_secret selects client credentials.
+func selectAuthMode(options authenticationOptions) authMode {
+	switch {
+	case options.JWT != "":
+		return authModeJWT
+	case options.Username != "" && options.Password != "":
+		return authModePassword
+	case options.ClientSecret != "":
+		return authModeClientCredentials
 	}
-	return nil, fmt.Errorf("unable to authenticate")
+	return authModeUnknown
 }
 
 func endpoint(options authenticationOptions) (endpoint string, err error) {
@@ -67,13 +97,18 @@ func endpoint(options authenticationOptions) (endpoint string, err error) {
 }
 
 func jwtLogin(endpoint string, options authenticationOptions) (*force.Force, error) {
+	key, err := decodeJWTKey(options.JWT)
+	if err != nil {
+		return nil, err
+	}
+
 	tempfile, err := os.CreateTemp("", "")
 	if err != nil {
 		return nil, fmt.Errorf("creating tempfile to write rsa key failed: %w", err)
 	}
 	defer os.Remove(tempfile.Name())
 
-	if _, err = tempfile.WriteString(options.JWT); err != nil {
+	if _, err = tempfile.Write(key); err != nil {
 		return nil, fmt.Errorf("writing rsa key to tempfile failed: %w", err)
 	}
 
@@ -89,11 +124,42 @@ func jwtLogin(endpoint string, options authenticationOptions) (*force.Force, err
 	return force.NewForce(&session), nil
 }
 
-func soapLoginAtEndpoint(endpoint, username, password string) (*force.Force, error) {
-	session, err := force.ForceSoapLoginAtEndpoint(endpoint, username, password)
+func passwordFlowLogin(endpoint string, options authenticationOptions) (*force.Force, error) {
+	session, err := force.PasswordFlowLoginAtEndpoint(endpoint, options.ConnectedApp, options.ClientSecret, options.Username, options.Password)
 	if err != nil {
-		return nil, fmt.Errorf("SOAP authentication failed: %w", err)
+		return nil, fmt.Errorf("OAuth password authentication failed: %w", err)
 	}
+	return force.NewForce(&session), nil
+}
 
+func clientCredentialsLogin(endpoint string, options authenticationOptions) (*force.Force, error) {
+	session, err := force.ClientCredentialsLoginAtEndpoint(endpoint, options.ConnectedApp, options.ClientSecret)
+	if err != nil {
+		return nil, fmt.Errorf("client credentials authentication failed: %w", err)
+	}
 	return force.NewForce(&session), nil
 }
+
+// decodeJWTKey returns the raw PEM bytes for the JWT private key. The UI
+// base64-encodes uploaded keys before writing them to .env so embedded
+// newlines don't break the dotenv parser; raw PEM is accepted for
+// backwards compatibility with hand-written configs.
+func decodeJWTKey(key string) ([]byte, error) {
+	trimmed := strings.TrimSpace(key)
+	if strings.HasPrefix(trimmed, "-----BEGIN") {
+		return []byte(key), nil
+	}
+	// Tolerate whitespace introduced by line wrapping in .env or YAML.
+	compact := strings.Map(func(r rune) rune {
+		switch r {
+		case ' ', '\t', '\r', '\n':
+			return -1
+		}
+		return r
+	}, trimmed)
+	decoded, err := base64.StdEncoding.DecodeString(compact)
+	if err != nil {
+		return nil, fmt.Errorf("JWT private key is neither PEM nor valid base64: %w", err)
+	}
+	return decoded, nil
+}