Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions docs/developer-guide/creating-dandi-instance/dandi-hub.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/getting-started/creating-account.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

A DANDI account enhances your capabilities within the DANDI Archive.
Without an account, users can freely search, view, and download available datasets.
With an account, users can create and edit Dandisets, and use the DANDI Hub to analyze data.
With an account, users can create and edit Dandisets.

DANDI provides two servers:

Expand Down
1 change: 0 additions & 1 deletion docs/getting-started/dandi-ecosystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ The DANDI project is organized around several **GitHub** repositories:
| Repository | Description |
|----------|----------|
| [DANDI Archive](https://github.com/dandi/dandi-archive) | Contains the code for deploying the client-side Web application frontend based on the [Vue.js](https://vuejs.org/) framework as well as a Django-based backend to run the DANDI REST API.
| [DANDI JupyterHub](https://github.com/dandi/dandi-hub) | Contains the code for deploying a JupyterHub instance to support interaction with the DANDI archive.
| [DANDI Python client](https://github.com/dandi/dandi-cli) | Contains the code for the command line tool used to interact with the archive. It allows you to download data from the archive. It also allows you to locally organize and validate your data before uploading to the archive.
| [DANDI Docs](https://github.com/dandi/dandi-docs) | Provides the contents of this website.
| [helpdesk](https://github.com/dandi/helpdesk) | Contains our community help platform where you can submit [issues](https://github.com/dandi/helpdesk/issues/new/choose).
Expand Down
1 change: 0 additions & 1 deletion docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,5 @@ DANDI provides significant benefits:
| Non-standardized datasets lead to significant resource needs to understand and adapt code to these datasets. | DANDI standardizes all data using NWB and BIDS standards. |
| The multitude of different hardware platforms and custom binary formats requires significant effort to consolidate into reusable datasets. | The DANDI ecosystem provides tools for converting data from different instruments into NWB and BIDS. |
| There are many domain general places to house data (e.g. Open Science Framework, G-Node, Dropbox, Google drive), but it is difficult to find relevant scientific metadata. | DANDI is focused on neurophysiology data and related metadata. |
| Datasets are growing larger, requiring compute services to be closer to data. | DANDI provides Dandihub, a JupyterHub instance close to the data. |
| Neurotechnology is evolving and requires changes to metadata and data storage. | DANDI works with community members to improve data standards and formats. |
| Consolidating and creating robust algorithms (e.g. spike sorting) requires varied data sources. | DANDI provides access to many different datasets. |
6 changes: 3 additions & 3 deletions docs/user-guide-sharing/contributing-notebook.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Example notebooks are Jupyter notebooks that demonstrate how to:
- Reproduce figures from associated publications
- Showcase the potential uses of your data

These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository and are available to all DANDI Hub users.
These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository and published at [notebooks.dandiarchive.org](https://notebooks.dandiarchive.org).

## Why Contribute an Example Notebook?

Expand Down Expand Up @@ -39,7 +39,7 @@ To contribute an example notebook:
- Add your notebook to the appropriate directory
- Submit a pull request with a clear description of your notebook

3. **Wait for review and approval** from the DANDI team. Once approved, your notebook will be merged into the repository and made available to all DANDI Hub users.
3. **Wait for review and approval** from the DANDI team. Once approved, your notebook will be merged into the repository and published at [notebooks.dandiarchive.org](https://notebooks.dandiarchive.org).

## Best Practices for Example Notebooks

Expand All @@ -50,7 +50,7 @@ To create effective example notebooks:
- **Use relative paths** when accessing data to ensure portability
- **Include visualizations** to help users understand the data
- **Document any assumptions or limitations** of your analyses
- **Test your notebook** in the DANDI Hub environment before submitting
- **Test your notebook** in a clean environment before submitting
- **Keep the notebook focused** on demonstrating how to use the data rather than complex analyses

## Example Notebook Organization
Expand Down
7 changes: 2 additions & 5 deletions docs/user-guide-using/accessing-data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ DANDI offers several methods for accessing data, each suited to different use ca
2. **DANDI CLI**: Command-line tool for downloading entire Dandisets or specific files.
3. **DataLad**: Access Dandisets as Git repositories with DataLad for version control and reproducibility.
4. **WebDAV**: Access DANDI data using standard WebDAV clients.
5. **DANDI Hub**: Analyze data directly in the cloud using Jupyter notebooks.
6. **Programmatic Access**: Access data programmatically using the DANDI API through Python or other languages.
5. **Programmatic Access**: Access data programmatically using the DANDI API through Python or other languages.

## Choosing the Right Access Method

Expand All @@ -21,15 +20,14 @@ The best method for accessing data depends on your specific needs:
- **For downloading entire Dandisets**: Use the [DANDI CLI](./downloading.md#using-the-python-cli-client).
- **For version control and reproducibility**: Use [DataLad](./downloading.md#using-datalad).
- **For integration with existing tools**: Use [WebDAV](./downloading.md#using-webdav), [DANDI Python and Command-line Client](./downloading.md#using-the-python-cli-client), or [DANDI API](./external-services.md#custom-integrations) depending on the tool language/interfaces.
- **For cloud-based analysis**: Use [DANDI Hub](../dandi-hub.md).
- **For programmatic access**: Use the [DANDI Python Client](https://dandi.readthedocs.io/) for Python or the [DANDI API](./external-services.md#custom-integrations) for other languages.

## Data Access Considerations

When accessing data from DANDI, consider the following:

- **Data Size**: Large datasets may be better accessed using the DANDI CLI or DataLad rather than the web interface.
- **Bandwidth**: For users with limited bandwidth, consider using DANDI Hub to analyze data in the cloud.
- **Bandwidth**: For users with limited bandwidth, consider [streaming](./streaming.md) data rather than downloading entire files.
- **Reproducibility**: DataLad provides version control and reproducibility features that are valuable for scientific workflows.
- **Streaming**: For large files, streaming access may be more efficient than downloading entire files.

Expand All @@ -40,4 +38,3 @@ Explore the following pages for detailed information on each access method:
- [Downloading Data](./downloading.md): Learn how to download data using the web interface, DANDI CLI, DataLad, or WebDAV.
- [Streaming Data](./streaming.md): Learn how to stream data without downloading entire files.
- [External Services](./external-services.md): Learn about external services that can be used to access and analyze DANDI data.
- [DANDI Hub](../dandi-hub.md): Learn how to use DANDI Hub for cloud-based analysis.
2 changes: 1 addition & 1 deletion docs/user-guide-using/accessing-data/streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ DANDI provides several methods for streaming data:

### 1. Python-based streaming methods

Using Python, you can set up data streaming using remfile, fsspec, or ros3. See the [PyNWB streaming tutorial](https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html) for details. Note that these streaming methods tend to work better on [DANDI Hub](../dandi-hub.md), where data access is faster.
Using Python, you can set up data streaming using remfile, fsspec, or ros3. See the [PyNWB streaming tutorial](https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html) for details.

### 2. DataLad FUSE Mount

Expand Down
138 changes: 8 additions & 130 deletions docs/user-guide-using/dandi-hub.md
Original file line number Diff line number Diff line change
@@ -1,134 +1,12 @@
# Using the DANDI Hub
# DANDI Hub (retired)

[DANDI Hub](http://hub.dandiarchive.org) is a [JupyterHub](https://jupyterhub.readthedocs.io) instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI.
For instructions on how to navigate JupyterHub see this [YouTube tutorial](https://www.youtube.com/watch?v=5pf0_bpNbkw&t=09m20s).
Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data.
DANDI Hub, the JupyterHub instance at `hub.dandiarchive.org`, was retired in August 2026.
The service is no longer available, and all user data stored on it has been deleted.

## Registration
## What to use instead

To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for an account using the [DANDI website](http://dandiarchive.org).
See the [Create a DANDI Account](../getting-started/creating-account.md) page.
**Example notebooks** are published at [notebooks.dandiarchive.org](https://notebooks.dandiarchive.org) and can be run locally or in Google Colab.
They are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository.

## Choosing a server option

When you start up the DANDI Hub, you will be asked to select across a number of server options.
For basic exploration, Tiny or Base would most likely be appropriate.
The DANDI Hub also currently offers Medium and Large options, which have more available memory and compute power.
The "T4 GPU inference" server comes with an associated T4 GPU, and is intended to be used for applications that require GPU for inference.
We request that users of this server be considerate of their usage of the DANDI Hub as a free community resource.
Training large deep neural networks is not appropriate.
A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation but you would be required to provide your own license.

## Using conda environments

DANDI Hub provides two ways to work with Python environments: shared environments managed through conda-store, and individual environments you create with conda in your home directory.

**Shared environments** are managed through conda-store and are available to all DANDI Hub users.
These environments contain commonly used packages for neurophysiology analysis and are maintained by administrators.
Use shared environments when:
- You need standard analysis tools and packages
- You want to collaborate with other users using the same environment
- You prefer not to manage package dependencies yourself

**Individual environments** are created and managed using standard conda commands in your user home directory (`/home/username`).
These are private to your account and **should be used instead of conda-store for personal environments**.
Create individual environments when:
- You need specific package versions not available in shared environments
- You're experimenting with new packages or configurations
- You need a customized environment for your specific analysis workflow

**Important:** Do not use conda-store for creating individual environments.
Conda-store is a deployment service for shared environments only.
Use regular conda commands for personal environments in your home directory.

### Using shared environments

#### Activating in the terminal

To see available shared environments:
```bash
conda env list
```

To activate a shared environment:
```bash
conda activate environment-name
```

For example, to activate the dandi environment:
```bash
conda activate nebari-git-dandi
```

#### Activating in your Jupyter notebook

**At startup:** When launching JupyterLab, you can select a shared environment from the kernel dropdown in the launcher.

**In an existing notebook:**

1. In the top right of a notebook, click the current environment which will open a "Start a new kernel for mynotebook.ipynb"
2. Select the desired shared environment from the list
3. The notebook will switch to use packages from that environment

### Creating individual environments

#### Lightweight `venv` overlay

If you need to use an environment, ie `dandi` with extra dependencies, please **do not** create a new conda environment.
Instead use `venv` to create a local virtual environment **on top of an existing env.**

```bash
conda activate nebari-git-dandi
python -m venv --system-site-packages my-dandi-extras
source my-dandi-extras/bin/activate
pip install some-extra-package
```

#### Full custom `conda` environment

If you need to create a conda env make sure it is stored in your home directory using the `--prefix` flag:

```bash
conda create --prefix /home/username/.conda/envs/my-env-name python=3.9
```

To activate your individual environment:

```bash
conda activate /home/username/.conda/envs/my-env-name
```

To install packages in your individual environment:

```bash
conda activate /home/username/.conda/envs/my-env-name
conda install package-name
# or
pip install package-name
```

**Note:** Replace `username` with your actual username, or use `$HOME` instead of `/home/username`.

## Custom server image

If you need additional software installed in the image, you can add a server image that will be made available for all users in the `Server Options` menu. Add a server image by following the instructions below and submitting a pull request to the [dandi-hub repository](https://github.com/dandi/dandi-hub). Once the pull request is merged, the DANDI team will redeploy JupyterHub and the image will be available.


1. Fork and clone the [dandi-hub](https://github.com/dandi/dandi-hub) repository.
2. Add a Dockerfile to the [images](https://github.com/dandi/dandi-hub/tree/main/images) directory.
3. Test the Dockerfile with the following commands to build and run the new image, which can be viewed locally in the browser at 127.0.0.1:8888/
```sh
docker build -f "$(CONTAINERFILE)" -t dandihub-dev:latest .
docker run --rm -p 8888:8888 --name dev_jupyterlab dandihub-dev:latest start-notebook.sh --NotebookApp.token=""
```
4. Add the Dockerfile to the `include` matrix of both the [docker-push.yaml](https://github.com/dandi/dandi-hub/blob/main/.github/workflows/docker-push.yaml) and [docker-test.yaml](https://github.com/dandi/dandi-hub/blob/main/.github/workflows/docker-test.yaml) files. This will allow the image to be built when new pull requests are opened and pushed to the [DANDI Archive Docker Hub](https://hub.docker.com/u/dandiarchive) when the pull requests are merged.
5. Add the image to the server options by updating the [jupyterhub.yaml](https://github.com/dandi/dandi-hub/blob/main/envs/shared/jupyterhub.yaml) file.

## Example notebooks

The best way to share analyses on DANDI data is through the DANDI example notebooks.
These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository which provides more information about their organization.
Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications.

Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository.
Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users.
**To analyze DANDI data**, see [Accessing data](accessing-data/index.md).
Data can be [streamed](accessing-data/streaming.md) directly from the archive without downloading it, which works from a local machine or any cloud environment.
1 change: 0 additions & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,6 @@ nav:
- DANDI Archive: "developer-guide/creating-dandi-instance/dandi-archive.md"
- DANDI Authentication: "developer-guide/creating-dandi-instance/dandi-authentication.md"
- DANDI Client: "developer-guide/creating-dandi-instance/dandi-cli.md"
- DANDI Hub: "developer-guide/creating-dandi-instance/dandi-hub.md"
- Health Status:
- Dandisets: https://github.com/dandi/dandisets-healthstatus
- Services: https://status.dandiarchive.org
Expand Down
Loading