diff --git a/source/images/databricks-serverless-environment-panel.png b/source/images/databricks-serverless-environment-panel.png new file mode 100644 index 00000000..1e7ba1f5 Binary files /dev/null and b/source/images/databricks-serverless-environment-panel.png differ diff --git a/source/images/databricks-serverless-select-compute.png b/source/images/databricks-serverless-select-compute.png new file mode 100644 index 00000000..d74d6840 Binary files /dev/null and b/source/images/databricks-serverless-select-compute.png differ diff --git a/source/platforms/databricks.md b/source/platforms/databricks.md index d5ff1a0a..48bcc95c 100644 --- a/source/platforms/databricks.md +++ b/source/platforms/databricks.md @@ -6,11 +6,14 @@ review_priority: "p0" You can install RAPIDS on Databricks in a few different ways: -1. Accelerate machine learning workflows in a single-node GPU notebook environment -2. Spark users can install [RAPIDS Accelerator for Apache Spark 3.x on Databricks](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/databricks.html) -3. Install Dask alongside Spark and then use libraries like `dask-cudf` for multi-node workloads +1. Accelerate machine learning workflows in a single-node GPU notebook environment on [classic compute](classic-gpu-compute) +2. Accelerate machine learning workflows in a single-node GPU notebook environment on [serverless GPU compute](serverless-gpu-compute) +3. Spark users can install [RAPIDS Accelerator for Apache Spark 3.x on Databricks](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/databricks.html) +4. Install Dask alongside Spark and then use libraries like `dask-cudf` for multi-node workloads -## Single-node GPU Notebook environment +(classic-gpu-compute)= + +## Classic GPU compute (create-init-script)= @@ -29,9 +32,7 @@ set -e # Install RAPIDS libraries pip install \ --extra-index-url={{rapids_pip_index}} \ - "cudf-cu{{rapids_cuda_major}}=={{rapids_pip_version}}" "cuml-cu{{rapids_cuda_major}}=={{rapids_pip_version}}" \ - "dask-cuda=={{rapids_pip_version}}" - + "cudf-cu{{rapids_cuda_major}}=={{rapids_pip_version}}" "cuml-cu{{rapids_cuda_major}}=={{rapids_pip_version}}" ``` (launch-databricks-cluster)= @@ -55,9 +56,57 @@ Then expand the **Advanced Options** section, open the **Init Scripts** tab and Select **Create Compute** -### Test RAPIDS +(serverless-gpu-compute)= + +## Serverless GPU compute + +[Serverless GPU compute](https://docs.databricks.com/aws/en/compute/serverless/gpu) gives you a single-node GPU notebook with no cluster to create and no init script to maintain. Databricks provisions the GPU on demand when you attach a notebook to it. + +Because there is no cluster to configure, the [init script](create-init-script) approach above does not apply. Install RAPIDS from inside the notebook instead. + +```{note} +Serverless GPU compute is in public preview and is only available in [certain regions](https://docs.databricks.com/aws/en/compute/serverless/gpu). +``` + +### Connect a notebook -Once your cluster has started, you can create a new notebook or open an existing one from the `/Workspace` directory then attach it to your running cluster. +Open a notebook, click the compute dropdown at the top and select **Serverless GPU**. + +![Screenshot of selecting Serverless GPU from the notebook compute dropdown](../images/databricks-serverless-select-compute.png) + +From the submenu select **Configuration** to open the configuration side panel. Under **Hardware** set **Accelerator** to `1xA10`, and under **Environment** set **Base environment** to **Standard v5**. Then click **Apply** and **Confirm**. + +```{figure} /images/databricks-serverless-environment-panel.png +--- +alt: Screenshot of the Environment side panel +width: 50% +align: center +--- +``` + +```{note} +Choose the **Standard** environment rather than the **AI** environment. The AI environment preinstalls `cupy-cuda12x`, which conflicts with the `cupy-cuda13x` build that the RAPIDS CUDA {{rapids_cuda_major}} wheels depend on. +``` + +### Install RAPIDS + +Install the RAPIDS libraries into your notebook environment. + +```python +%pip install \ + --extra-index-url={{rapids_pip_index}} \ + "cudf-cu{{rapids_cuda_major}}=={{rapids_pip_version}}" "cuml-cu{{rapids_cuda_major}}=={{rapids_pip_version}}" +``` + +Then restart the Python process so that the new packages are picked up. + +```python +%restart_python +``` + +## Test RAPIDS + +You can run the following code snippet to verify that the RAPIDS libraries are installed successfully on your choice of compute. ```python import cudf @@ -70,7 +119,7 @@ gdf 2 3 6 ``` -#### Quickstart with cuDF Pandas +## Quickstart with cuDF Pandas RAPIDS recently introduced cuDF’s [pandas accelerator mode](https://rapids.ai/cudf-pandas/) to accelerate existing pandas workflows with zero changes to code. @@ -100,6 +149,4 @@ df = pd.read_parquet( ) ``` -Upload the [10 Minutes to RAPIDS cuDF Pandas notebook](https://colab.research.google.com/drive/12tCzP94zFG2BRduACucn5Q_OcX1TUKY3) in your single-node Databricks cluster and run through the cells. - -**NOTE**: cuDF pandas is open beta and under active development. You can [learn more through the documentation](https://docs.rapids.ai/api/cudf/~~~rapids_api_docs_version~~~/?_gl=1*1oyfbsi*_ga*MTc5NDYzNzYyNC4xNjgzMDc2ODc2*_ga_RKXFW6CM42*MTcwNTU4NDUyNS4yMC4wLjE3MDU1ODQ1MjUuNjAuMC4w) and the [release blog](https://developer.nvidia.com/blog/rapids-cudf-accelerates-pandas-nearly-150x-with-zero-code-changes/). +Upload the [10 Minutes to RAPIDS cuDF Pandas notebook](https://colab.research.google.com/drive/12tCzP94zFG2BRduACucn5Q_OcX1TUKY3) into your Databricks workspace and run through the cells.