Document Databricks Serverless

[jacobtomlinson]

Looks like Databricks has a serverless notebooks mode which is slightly different to the cluster based approach we use currently. We should make sure we have any differences documented.

https://docs.databricks.com/aws/en/compute/serverless/gpu

Forwarded from @jrbourbeau

Databricks has a new serverless compute product. Long-term that will end up replacing their "classic" Spark compute clusters. I tried a basic pip install cudf in a serverless notebook and it seemed to work out of the box

[jayavenkatesh19]

I looked into this yesterday. I looks like I don't have access to the Serverless GPU cluster as I don't have access to specifying an accelerator in my notebook environment.
The default environment however does not seem to have conda installed and just uses pip for package management.

[jrbourbeau]

(Just posting here for visibility -- @jayavenkatesh19 has access to a DBRX deployment with serverlerss GPU enabled)