From f88fcffde3a316ca1b8da8a78f555832a7dd6023 Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 12 Aug 2026 16:01:07 -0500 Subject: [PATCH 1/2] Replace DANDI Hub user guide with retirement notice DANDI Hub was retired in August 2026 and all user data deleted, so the usage instructions (server options, conda environments, custom server images) no longer describe anything that exists. This page is the target of the planned hub.dandiarchive.org redirect and of the retirement banner in dandi/dandi-hub, so it keeps its existing path and nav position to preserve inbound links. Inbound references from other pages are not yet updated. Ref: https://github.com/dandi/dandi-hub/issues/296 Co-Authored-By: Claude Opus 5 (1M context) --- docs/user-guide-using/dandi-hub.md | 138 ++--------------------------- 1 file changed, 8 insertions(+), 130 deletions(-) diff --git a/docs/user-guide-using/dandi-hub.md b/docs/user-guide-using/dandi-hub.md index 8b61f10b..7f3aa7cf 100644 --- a/docs/user-guide-using/dandi-hub.md +++ b/docs/user-guide-using/dandi-hub.md @@ -1,134 +1,12 @@ -# Using the DANDI Hub +# DANDI Hub (retired) -[DANDI Hub](http://hub.dandiarchive.org) is a [JupyterHub](https://jupyterhub.readthedocs.io) instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. -For instructions on how to navigate JupyterHub see this [YouTube tutorial](https://www.youtube.com/watch?v=5pf0_bpNbkw&t=09m20s). -Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. +DANDI Hub, the JupyterHub instance at `hub.dandiarchive.org`, was retired in August 2026. +The service is no longer available, and all user data stored on it has been deleted. -## Registration +## What to use instead -To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for an account using the [DANDI website](http://dandiarchive.org). -See the [Create a DANDI Account](../getting-started/creating-account.md) page. +**Example notebooks** are published at [notebooks.dandiarchive.org](https://notebooks.dandiarchive.org) and can be run locally or in Google Colab. +They are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository. -## Choosing a server option - -When you start up the DANDI Hub, you will be asked to select across a number of server options. -For basic exploration, Tiny or Base would most likely be appropriate. -The DANDI Hub also currently offers Medium and Large options, which have more available memory and compute power. -The "T4 GPU inference" server comes with an associated T4 GPU, and is intended to be used for applications that require GPU for inference. -We request that users of this server be considerate of their usage of the DANDI Hub as a free community resource. -Training large deep neural networks is not appropriate. -A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation but you would be required to provide your own license. - -## Using conda environments - -DANDI Hub provides two ways to work with Python environments: shared environments managed through conda-store, and individual environments you create with conda in your home directory. - -**Shared environments** are managed through conda-store and are available to all DANDI Hub users. -These environments contain commonly used packages for neurophysiology analysis and are maintained by administrators. -Use shared environments when: -- You need standard analysis tools and packages -- You want to collaborate with other users using the same environment -- You prefer not to manage package dependencies yourself - -**Individual environments** are created and managed using standard conda commands in your user home directory (`/home/username`). -These are private to your account and **should be used instead of conda-store for personal environments**. -Create individual environments when: -- You need specific package versions not available in shared environments -- You're experimenting with new packages or configurations -- You need a customized environment for your specific analysis workflow - -**Important:** Do not use conda-store for creating individual environments. -Conda-store is a deployment service for shared environments only. -Use regular conda commands for personal environments in your home directory. - -### Using shared environments - -#### Activating in the terminal - -To see available shared environments: -```bash -conda env list -``` - -To activate a shared environment: -```bash -conda activate environment-name -``` - -For example, to activate the dandi environment: -```bash -conda activate nebari-git-dandi -``` - -#### Activating in your Jupyter notebook - -**At startup:** When launching JupyterLab, you can select a shared environment from the kernel dropdown in the launcher. - -**In an existing notebook:** - -1. In the top right of a notebook, click the current environment which will open a "Start a new kernel for mynotebook.ipynb" -2. Select the desired shared environment from the list -3. The notebook will switch to use packages from that environment - -### Creating individual environments - -#### Lightweight `venv` overlay - -If you need to use an environment, ie `dandi` with extra dependencies, please **do not** create a new conda environment. -Instead use `venv` to create a local virtual environment **on top of an existing env.** - -```bash -conda activate nebari-git-dandi -python -m venv --system-site-packages my-dandi-extras -source my-dandi-extras/bin/activate -pip install some-extra-package -``` - -#### Full custom `conda` environment - -If you need to create a conda env make sure it is stored in your home directory using the `--prefix` flag: - -```bash -conda create --prefix /home/username/.conda/envs/my-env-name python=3.9 -``` - -To activate your individual environment: - -```bash -conda activate /home/username/.conda/envs/my-env-name -``` - -To install packages in your individual environment: - -```bash -conda activate /home/username/.conda/envs/my-env-name -conda install package-name -# or -pip install package-name -``` - -**Note:** Replace `username` with your actual username, or use `$HOME` instead of `/home/username`. - -## Custom server image - -If you need additional software installed in the image, you can add a server image that will be made available for all users in the `Server Options` menu. Add a server image by following the instructions below and submitting a pull request to the [dandi-hub repository](https://github.com/dandi/dandi-hub). Once the pull request is merged, the DANDI team will redeploy JupyterHub and the image will be available. - - -1. Fork and clone the [dandi-hub](https://github.com/dandi/dandi-hub) repository. -2. Add a Dockerfile to the [images](https://github.com/dandi/dandi-hub/tree/main/images) directory. -3. Test the Dockerfile with the following commands to build and run the new image, which can be viewed locally in the browser at 127.0.0.1:8888/ -```sh -docker build -f "$(CONTAINERFILE)" -t dandihub-dev:latest . -docker run --rm -p 8888:8888 --name dev_jupyterlab dandihub-dev:latest start-notebook.sh --NotebookApp.token="" -``` -4. Add the Dockerfile to the `include` matrix of both the [docker-push.yaml](https://github.com/dandi/dandi-hub/blob/main/.github/workflows/docker-push.yaml) and [docker-test.yaml](https://github.com/dandi/dandi-hub/blob/main/.github/workflows/docker-test.yaml) files. This will allow the image to be built when new pull requests are opened and pushed to the [DANDI Archive Docker Hub](https://hub.docker.com/u/dandiarchive) when the pull requests are merged. -5. Add the image to the server options by updating the [jupyterhub.yaml](https://github.com/dandi/dandi-hub/blob/main/envs/shared/jupyterhub.yaml) file. - -## Example notebooks - -The best way to share analyses on DANDI data is through the DANDI example notebooks. -These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository which provides more information about their organization. -Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. - -Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository. -Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users. +**To analyze DANDI data**, see [Accessing data](accessing-data/index.md). +Data can be [streamed](accessing-data/streaming.md) directly from the archive without downloading it, which works from a local machine or any cloud environment. From 608fea8f63ca5e4bac9e709d0a1c89b713d3f62c Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 12 Aug 2026 16:25:10 -0500 Subject: [PATCH 2/2] Remove references to the retired DANDI Hub Also drops the developer guide page for deploying your own Hub and its nav entry. The user guide nav entry stays, pointing at the retirement notice. policies.md is handled separately in #242. Ref: https://github.com/dandi/dandi-hub/issues/296 Co-Authored-By: Claude Opus 5 (1M context) --- .../creating-dandi-instance/dandi-hub.md | 14 -------------- docs/getting-started/creating-account.md | 2 +- docs/getting-started/dandi-ecosystem.md | 1 - docs/introduction.md | 1 - docs/user-guide-sharing/contributing-notebook.md | 6 +++--- docs/user-guide-using/accessing-data/index.md | 7 ++----- docs/user-guide-using/accessing-data/streaming.md | 2 +- mkdocs.yml | 1 - 8 files changed, 7 insertions(+), 27 deletions(-) delete mode 100644 docs/developer-guide/creating-dandi-instance/dandi-hub.md diff --git a/docs/developer-guide/creating-dandi-instance/dandi-hub.md b/docs/developer-guide/creating-dandi-instance/dandi-hub.md deleted file mode 100644 index a9ab22ef..00000000 --- a/docs/developer-guide/creating-dandi-instance/dandi-hub.md +++ /dev/null @@ -1,14 +0,0 @@ -The DANDI ecosystem includes a self-hosted Jupyter notebook service. This service is hosted on AWS and orchestrated with a Kubernetes (k8s) cluster -that provides different instance types for users to efficiently interact with data in the DANDI Archive. - -The instructions for configuring and deploying your own JupyterHub instance are available in the [dandi-hub repository](https://github.com/dandi/dandi-hub) (see [README](https://github.com/dandi/dandi-hub/blob/main/README.md#dandihub)). -For example configurations that have been previously generated for the DANDI, LINC, and BICAN projects see the [envs directory](https://github.com/dandi/dandi-hub/tree/main/envs). -**Note: it is important that your k8s cluster is in the same region as your data.** - -Resources - -• [Source code and instructions](https://github.com/dandi/dandi-hub) - -• [DANDI Hub](https://hub.dandiarchive.org/) - -• [LINC Hub](https://hub.lincbrain.org/) \ No newline at end of file diff --git a/docs/getting-started/creating-account.md b/docs/getting-started/creating-account.md index e1657585..01959ac0 100644 --- a/docs/getting-started/creating-account.md +++ b/docs/getting-started/creating-account.md @@ -2,7 +2,7 @@ A DANDI account enhances your capabilities within the DANDI Archive. Without an account, users can freely search, view, and download available datasets. -With an account, users can create and edit Dandisets, and use the DANDI Hub to analyze data. +With an account, users can create and edit Dandisets. DANDI provides two servers: diff --git a/docs/getting-started/dandi-ecosystem.md b/docs/getting-started/dandi-ecosystem.md index 592c5515..3ac54d18 100644 --- a/docs/getting-started/dandi-ecosystem.md +++ b/docs/getting-started/dandi-ecosystem.md @@ -17,7 +17,6 @@ The DANDI project is organized around several **GitHub** repositories: | Repository | Description | |----------|----------| | [DANDI Archive](https://github.com/dandi/dandi-archive) | Contains the code for deploying the client-side Web application frontend based on the [Vue.js](https://vuejs.org/) framework as well as a Django-based backend to run the DANDI REST API. -| [DANDI JupyterHub](https://github.com/dandi/dandi-hub) | Contains the code for deploying a JupyterHub instance to support interaction with the DANDI archive. | [DANDI Python client](https://github.com/dandi/dandi-cli) | Contains the code for the command line tool used to interact with the archive. It allows you to download data from the archive. It also allows you to locally organize and validate your data before uploading to the archive. | [DANDI Docs](https://github.com/dandi/dandi-docs) | Provides the contents of this website. | [helpdesk](https://github.com/dandi/helpdesk) | Contains our community help platform where you can submit [issues](https://github.com/dandi/helpdesk/issues/new/choose). diff --git a/docs/introduction.md b/docs/introduction.md index 407f6e38..124acb48 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -40,6 +40,5 @@ DANDI provides significant benefits: | Non-standardized datasets lead to significant resource needs to understand and adapt code to these datasets. | DANDI standardizes all data using NWB and BIDS standards. | | The multitude of different hardware platforms and custom binary formats requires significant effort to consolidate into reusable datasets. | The DANDI ecosystem provides tools for converting data from different instruments into NWB and BIDS. | | There are many domain general places to house data (e.g. Open Science Framework, G-Node, Dropbox, Google drive), but it is difficult to find relevant scientific metadata. | DANDI is focused on neurophysiology data and related metadata. | -| Datasets are growing larger, requiring compute services to be closer to data. | DANDI provides Dandihub, a JupyterHub instance close to the data. | | Neurotechnology is evolving and requires changes to metadata and data storage. | DANDI works with community members to improve data standards and formats. | | Consolidating and creating robust algorithms (e.g. spike sorting) requires varied data sources. | DANDI provides access to many different datasets. | diff --git a/docs/user-guide-sharing/contributing-notebook.md b/docs/user-guide-sharing/contributing-notebook.md index 72a23c70..fe641c9d 100644 --- a/docs/user-guide-sharing/contributing-notebook.md +++ b/docs/user-guide-sharing/contributing-notebook.md @@ -11,7 +11,7 @@ Example notebooks are Jupyter notebooks that demonstrate how to: - Reproduce figures from associated publications - Showcase the potential uses of your data -These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository and are available to all DANDI Hub users. +These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository and published at [notebooks.dandiarchive.org](https://notebooks.dandiarchive.org). ## Why Contribute an Example Notebook? @@ -39,7 +39,7 @@ To contribute an example notebook: - Add your notebook to the appropriate directory - Submit a pull request with a clear description of your notebook -3. **Wait for review and approval** from the DANDI team. Once approved, your notebook will be merged into the repository and made available to all DANDI Hub users. +3. **Wait for review and approval** from the DANDI team. Once approved, your notebook will be merged into the repository and published at [notebooks.dandiarchive.org](https://notebooks.dandiarchive.org). ## Best Practices for Example Notebooks @@ -50,7 +50,7 @@ To create effective example notebooks: - **Use relative paths** when accessing data to ensure portability - **Include visualizations** to help users understand the data - **Document any assumptions or limitations** of your analyses -- **Test your notebook** in the DANDI Hub environment before submitting +- **Test your notebook** in a clean environment before submitting - **Keep the notebook focused** on demonstrating how to use the data rather than complex analyses ## Example Notebook Organization diff --git a/docs/user-guide-using/accessing-data/index.md b/docs/user-guide-using/accessing-data/index.md index 464abdbd..2aa558d4 100644 --- a/docs/user-guide-using/accessing-data/index.md +++ b/docs/user-guide-using/accessing-data/index.md @@ -10,8 +10,7 @@ DANDI offers several methods for accessing data, each suited to different use ca 2. **DANDI CLI**: Command-line tool for downloading entire Dandisets or specific files. 3. **DataLad**: Access Dandisets as Git repositories with DataLad for version control and reproducibility. 4. **WebDAV**: Access DANDI data using standard WebDAV clients. -5. **DANDI Hub**: Analyze data directly in the cloud using Jupyter notebooks. -6. **Programmatic Access**: Access data programmatically using the DANDI API through Python or other languages. +5. **Programmatic Access**: Access data programmatically using the DANDI API through Python or other languages. ## Choosing the Right Access Method @@ -21,7 +20,6 @@ The best method for accessing data depends on your specific needs: - **For downloading entire Dandisets**: Use the [DANDI CLI](./downloading.md#using-the-python-cli-client). - **For version control and reproducibility**: Use [DataLad](./downloading.md#using-datalad). - **For integration with existing tools**: Use [WebDAV](./downloading.md#using-webdav), [DANDI Python and Command-line Client](./downloading.md#using-the-python-cli-client), or [DANDI API](./external-services.md#custom-integrations) depending on the tool language/interfaces. -- **For cloud-based analysis**: Use [DANDI Hub](../dandi-hub.md). - **For programmatic access**: Use the [DANDI Python Client](https://dandi.readthedocs.io/) for Python or the [DANDI API](./external-services.md#custom-integrations) for other languages. ## Data Access Considerations @@ -29,7 +27,7 @@ The best method for accessing data depends on your specific needs: When accessing data from DANDI, consider the following: - **Data Size**: Large datasets may be better accessed using the DANDI CLI or DataLad rather than the web interface. -- **Bandwidth**: For users with limited bandwidth, consider using DANDI Hub to analyze data in the cloud. +- **Bandwidth**: For users with limited bandwidth, consider [streaming](./streaming.md) data rather than downloading entire files. - **Reproducibility**: DataLad provides version control and reproducibility features that are valuable for scientific workflows. - **Streaming**: For large files, streaming access may be more efficient than downloading entire files. @@ -40,4 +38,3 @@ Explore the following pages for detailed information on each access method: - [Downloading Data](./downloading.md): Learn how to download data using the web interface, DANDI CLI, DataLad, or WebDAV. - [Streaming Data](./streaming.md): Learn how to stream data without downloading entire files. - [External Services](./external-services.md): Learn about external services that can be used to access and analyze DANDI data. -- [DANDI Hub](../dandi-hub.md): Learn how to use DANDI Hub for cloud-based analysis. diff --git a/docs/user-guide-using/accessing-data/streaming.md b/docs/user-guide-using/accessing-data/streaming.md index c0ba5620..68fd1eb5 100644 --- a/docs/user-guide-using/accessing-data/streaming.md +++ b/docs/user-guide-using/accessing-data/streaming.md @@ -8,7 +8,7 @@ DANDI provides several methods for streaming data: ### 1. Python-based streaming methods -Using Python, you can set up data streaming using remfile, fsspec, or ros3. See the [PyNWB streaming tutorial](https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html) for details. Note that these streaming methods tend to work better on [DANDI Hub](../dandi-hub.md), where data access is faster. +Using Python, you can set up data streaming using remfile, fsspec, or ros3. See the [PyNWB streaming tutorial](https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html) for details. ### 2. DataLad FUSE Mount diff --git a/mkdocs.yml b/mkdocs.yml index 370e9583..7b3103e8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -81,7 +81,6 @@ nav: - DANDI Archive: "developer-guide/creating-dandi-instance/dandi-archive.md" - DANDI Authentication: "developer-guide/creating-dandi-instance/dandi-authentication.md" - DANDI Client: "developer-guide/creating-dandi-instance/dandi-cli.md" - - DANDI Hub: "developer-guide/creating-dandi-instance/dandi-hub.md" - Health Status: - Dandisets: https://github.com/dandi/dandisets-healthstatus - Services: https://status.dandiarchive.org