Cluster replication

CHANGELOG.md

@@ -1,5 +1,9 @@
 # edge-server-tools
 
+## Unreleased
+
+- deprecated: `autoReplication`. Use `setupDatabase` instead.
+
 ## 0.2.20 (2024-10-11)
 
 - fixed: Add missing TypeScript entrypoint.

docs/couch-setup.md

@@ -9,7 +9,52 @@
   - [Enabling Replication](#enabling-replication)
 - [Background Tasks](#background-tasks)
 
-## Basic Setup
+The edge-server-tools package can create & maintain CouchDB databases, design documents, and replications. This maintenance can take place across multiple clusters or servers.
+
+These tools can also watch for changes to specific synced documents, such as settings.
+
+## Managing Credentials
+
+First, create a `CouchPool` object holding your connections to CouchDB. For simple cases, this just takes a URL:
+
+```js
+import { connectCouch } from 'edge-server-tools'
+
+const pool = connectCouch('http://admin:admin@localhost:5984/')
+```
+
+Or, if you want do auto-replication, you can provide a list of credentials and the name of the default cluster:
+
+```js
+const pool = connectCouch(
+  // The preferred cluster to use for most tasks:
+  'production',
+
+  // All the clusters we can access:
+  {
+    production: {
+      url: 'https://production.example.com:6984/',
+      username: 'admin',
+      password: 'admin',
+    },
+    logs: {
+      url: 'https://admin:admin@logs.example.com:6984/',
+    },
+    backup: 'https://admin:admin@backup.example.com:6984/'
+  }
+)
+```
+
+This example has three clusters, named "production", "logs", and "backup". These credentials would normally be stored on disk, perhaps using [cleaner-config](https://www.npmjs.com/package/cleaner-config). The main app server would generally contain all the credentials for the system and be responsible for maintenance tasks, while secondary servers might just connect directly to their own URL's.
+
+The `pool` object has a `default` property, which corresponds to the default cluster:
+
+```js
+const db = pool.default.use('my-db')
+await db.list()
+```
+
+## Creating Databases
 
 First, create a `DatabaseSetup` object to describe the database:
 
@@ -22,11 +67,10 @@ const productsSetup: DatabaseSetup = {
 Next, when your application starts up, use `setupDatabase` to ensure the database exists and has the right settings:
 
 ```ts
-const connection = nano(couchUri)
-await setupDatabase(connection, productsSetup)
+await setupDatabase(pool, productsSetup)
 ```
 
-This will ensure that the CouchDB instance contains a database named "products".
+If the pool contains credentials to multiple clusters, this will ensure that the database exists on each of them. If you only want to create this database on certain clusters, see the [Replication Setup](#replication-setup) section.
 
 If the database needs special CouchDB settings, pass those as `DatabaseSetup.options`:
 
@@ -54,7 +98,7 @@ This example will create a document named "\_design/mango-upc". The contents wil
 
 ### makeMangoIndex
 
-Use this function to create Mango index design documents. The first parameter is the name of the view (CouchDB doesn't really use this anywhere), and the second parameter is an array of properties to index over, using CouchDB's "sort syntax".
+Use this function to create Mango index design documents. The first parameter is the name of the view (CouchDB doesn't really use this anywhere), and the second parameter is an array of properties to index over, using CouchDB's [sort syntax](https://docs.couchdb.org/en/stable/api/database/find.html#sort-syntax).
 
 ```js
 makeMangoIndex('createdByDate', ['created', 'date'])
@@ -107,7 +151,7 @@ makeJsDesign(
 
 This example uses the `fixJs` option to convert "const" into "var" for the legacy JavaScript engine.
 
-If you need to access to utility functions, you can use CommonJS to inject them into the view:
+If you need access to utility functions, you can use CommonJS to inject them into the view:
 
 ```js
 import { normalizeIp, parseIp } from './ip-utils'
@@ -135,7 +179,7 @@ These library functions need to be completely standalone, and must use CouchDB's
 
 ## Watching Settings Documents
 
-While CouchDB is good at storing application data, it is also good at storing settings. Instead of using environment variables or JSON files on disk, putting settings inside CouchDB provides a nice admin interface for editing them, as well as automatic replication across the various servers.
+CouchDB is a convenient place to store app settings such as API keys or tuning parameters. Doing this provides live updates, automatic replication, and a nice admin interface using Fauxton.
 
 To watch a settings document for changes, use the `syncedDocument` helper function:
 
@@ -156,11 +200,12 @@ const settingsSetup: DatabaseSetup = {
 }
 
 // At app boot time:
-const connection = nano(couchUri)
-await setupDatabase(connection, settingsSetup)
+await setupDatabase(pool, settingsSetup)
 ```
 
-The `setupDatabase` will perform an initial sync, ensuring the document exists, and will then watch the document for any future changes, keeping it up to date. Simply access the `appSettings.doc` property to see the latest value, or subscribe to changes using the `onChange` method:
+The `setupDatabase` will use the `pool.default` connection to perform an initial sync, ensuring the document exists. It will also watch the document for any future changes, keeping it up to date using the cleaner. The [Background Tasks](#background-tasks) section explains how to manage this background process.
+
+To see the latest document contents, simply access the `appSettings.doc` property. You can also subscribe to changes using the `onChange` method:
 
 ```js
 appSettings.onChange((newSettings) => console.log(newSettings))
@@ -187,31 +232,18 @@ The `setupDatabase` function can automatically create documents in the `_replica
   "_rev": "12-6ee74490125b924e323807d9363d64a2",
   "clusters": {
     "production": {
-      "url": "https://production.example.com:6984/",
-      "basicAuth": "ZXhhbXBsZTpleGFtcGxl",
+      "exclude": ["#archived"],
       "pushTo": ["logs", "backup"]
     },
     "logs": {
-      "url": "https://logs.example.com:6984/",
-      "basicAuth": "ZXhhbXBsZTpleGFtcGxl",
-      "include": ["logs-*"]
+      "include": ["logs-*"],
+      "localOnly": ["settings"]
     },
-    "backup": {
-      "url": "https://backup.example.com:6984/",
-      "basicAuth": "ZXhhbXBsZTpleGFtcGxl"
-    }
+    "backup": {}
   }
 }
 ```
 
-This example has three clusters, named "production", "logs", and "backup". Each cluster has a URL, a set of credentials, and options describing what to replicate. You can generate the credential strings by opening a browser console and running:
-
-```js
-btoa('username:password')
-```
-
-You can also turn the base64 back to plain text by using `atob`.
-
 In this example, the "production" cluster has a `pushTo` list, which tells it to push changes out to the "logs" and "backup" clusters. The `setupDatabase` function will not create any replication documents on the other clusters, since they don't have `pushTo` or `pullFrom` properties.
 
 Since the "logs" cluster has an `include` filter, the `setupDatabase` routine will only create and replicate databases with names that start with "logs-" on this cluster. Clusters can also specify an `exclude` filter to avoid creating specific databases, and a `localOnly` filter to create databases but not replicate them. Databases can also have tags, such as "#archived" or "#secret", which also apply to the replicator filters.
@@ -238,41 +270,37 @@ const settingsSetup: DatabaseSetup = {
 }
 
 // At app boot time:
-const connection = nano(couchUri)
-await setupDatabase(connection, settingsSetup)
+await setupDatabase(pool, settingsSetup)
 ```
 
 Finally, pass `replicatorSetup` and `currentCluster` options to the various `setupDatabase` calls in your app:
 
 ```js
 const commonOptions = {
   replicatorSetup,
-
-  // This could be read from a JSON config file instead:
-  currentCluster: process.env.HOSTNAME,
+  // ...
 }
 
 // At app boot time:
-const connection = nano(couchUri)
-await setupDatabase(connection, settingsSetup, commonOptions)
-await setupDatabase(connection, productsSetup, commonOptions)
-await setupDatabase(connection, usersSetup, commonOptions)
+await setupDatabase(pool, settingsSetup, commonOptions)
+await setupDatabase(pool, productsSetup, commonOptions)
+await setupDatabase(pool, usersSetup, commonOptions)
 ```
 
 ## Background Tasks
 
 When setting up replication or synced documents, `setupDatabase` will create long-running background processes. These can interfere with CLI tools, which would like to exit quickly when they are done with their work. To avoid this problem, pass a `disableWatching` flag to `setupDatabase`:
 
 ```js
-setupDatabase(connection, someSetup, {
+setupDatabase(pool, someSetup, {
   disableWatching: true,
 })
 ```
 
 The `setupDatabase` function also returns a cleanup function, which will stop all background work (eventually - it can take a while).
 
 ```js
-const cleanup = setupDatabase(connection, someSetup)
+const cleanup = setupDatabase(pool, someSetup)
 
 // Later, at shutdown time:
 cleanup()
@@ -281,7 +309,7 @@ cleanup()
 The database setup process can generate errors and messages, which go to the console by default, but can be intercepted by passing `log` and `onError` options:
 
 ```ts
-setupDatabase(connection, someSetup, {
+setupDatabase(pool, someSetup, {
   log(message: string) {
     console.log(message)
   },

src/couchdb/couch-pool.ts

@@ -0,0 +1,163 @@
+import { asEither, asObject, asOptional, asString } from 'cleaners'
+import nano, { ServerScope } from 'nano'
+
+/**
+ * A table of CouchDB connections,
+ * with one connection highlighted as the default.
+ */
+export interface CouchPool {
+  /** The default connection, to use for most queries. */
+  default: ServerScope
+
+  /** The cluster name being used for the default connection. */
+  defaultName: string
+
+  /** Lists the clusters that are available for connection. */
+  clusterNames: string[]
+
+  /**
+   * Grab a connection to a specific cluster.
+   * Throws if the name is missing.
+   */
+  connect: (name: string) => ServerScope
+
+  /** Grabs authentication information for a specific cluster. */
+  getCredential: (name: string) => CouchCredential | undefined
+
+  /** Grab a connection, returning undefined if the cluster is missing. */
+  maybeConnect: (name: string) => ServerScope | undefined
+}
+
+export interface CouchCredential {
+  url: string
+  username?: string
+  password?: string
+}
+
+export interface CouchCredentials {
+  [name: string]:
+    | string
+    | CouchCredential
+    // The `setupDatabase` function uses this, but it's deprecated:
+    | ServerScope
+}
+
+export const asCouchCredential = asObject<CouchCredential>({
+  url: asString,
+  username: asOptional(asString),
+  password: asOptional(asString)
+})
+
+export const asCouchCredentials = asObject(
+  asEither(asCouchCredential, asString)
+)
+
+export function connectCouch(url: string): CouchPool
+export function connectCouch(
+  defaultName: string,
+  credentials: CouchCredentials
+): CouchPool
+
+/**
+ * Returns a CouchDB connection pool.
+ */
+export function connectCouch(
+  urlOrDefaultName: string,
+  maybeCreds?: CouchCredentials
+): CouchPool {
+  // Unpack the arguments:
+  const [defaultName, credentials] =
+    maybeCreds == null
+      ? // Just a URL:
+        ['default', { default: urlOrDefaultName }]
+      : // Full version:
+        [urlOrDefaultName, maybeCreds]
+
+  const cache = new Map<string, ServerScope>()
+
+  function maybeConnect(name: string): ServerScope | undefined {
+    // Use the cache, if we have it:
+    const cached = cache.get(name)
+    if (cached != null) return cached
+
+    // Find the credentials:
+    const row = credentials[name]
+    if (row == null) return
+
+    // Plain URL:
+    if (typeof row === 'string') {
+      const connection = nano(row)
+      cache.set(name, connection)
+      return connection
+    }
+
+    // Already-connected nano instance:
+    if ('relax' in row) {
+      cache.set(name, row)
+      return row
+    }
+
+    // A credentials object:
+    const { url, username, password } = row
+    if (username == null || password == null) {
+      const connection = nano(url)
+      cache.set(name, connection)
+      return connection
+    }
+    const connection = nano({
+      url,
+      requestDefaults: {
+        auth: { username, password }
+      }
+    })
+    cache.set(name, connection)
+    return connection
+  }
+
+  function connect(name: string): ServerScope {
+    const connection = maybeConnect(name)
+    if (connection == null) {
+      throw new Error(`Cannot find cluster '${name}'`)
+    }
+    return connection
+  }
+
+  function extractUrlCredentials(url: string): CouchCredential {
+    const parsed = new URL(url)
+    const { username, password } = parsed
+
+    // Remove credentials from the URL:
+    parsed.username = ''
+    parsed.password = ''
+    return {
+      url: parsed.toString(),
+      username,
+      password
+    }
+  }
+
+  function getCredential(name: string): CouchCredential | undefined {
+    const row = credentials[name]
+    if (row == null) return
+    if (typeof row === 'string') return extractUrlCredentials(row)
+    if ('relax' in row) return extractUrlCredentials(row.config.url)
+
+    const cleanUrl = extractUrlCredentials(row.url)
+    return {
+      url: cleanUrl.url,
+      username: row.username ?? cleanUrl.username,
+      password: row.password ?? cleanUrl.password
+    }
+  }
+
+  return {
+    default: connect(defaultName),
+    defaultName,
+
+    clusterNames: Object.keys(credentials),
+
+    connect,
+    getCredential,
+    maybeConnect
+  }
+}

src/couchdb/js-design-document.ts

@@ -72,12 +72,14 @@ export function stringifyCode(code: (...args: any[]) => unknown): string {
 }
 
 interface JsDesignOptions<Lib> {
-  // Methods available using 'require':
+  /** Methods available using 'require'. */
   lib?: Lib
 
-  // Transforms the Javascript source code.
-  // This could be anything from a simple search & replace
-  // to running something complicated like Babel:
+  /**
+   * Transforms the Javascript source code.
+   * This could be anything from a simple search & replace
+   * to running something complicated like Babel.
+   */
   fixJs?: (code: string) => string
 
   // Couch options: