Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ A modern Python CLI tool that generates beautiful HTML pages for neuron types us
- **🎨 Beautiful Reports**: Clean, accessible HTML pages with interactive features
- **🔍 Advanced Search**: Real-time filtering by cell count, neurotransmitter, brain regions
- **📊 Rich Analytics**: Hemisphere balance, connectivity stats, ROI summaries
- **⬇️ Neuron Downloads**: One-click zip of SWC skeletons or OBJ meshes for the neurons shown in the 3D view (per-dataset, see `config/README.md`)

## 🚀 Quick Start

Expand Down Expand Up @@ -136,6 +137,12 @@ We welcome contributions! Please see our comprehensive documentation:

This project is licensed under the GPLv3 License - see the LICENSE file for details.

The generated website bundles third-party browser libraries from `static/js/`,
each under the MIT license with its notice kept in the file header: jQuery,
DataTables (with its export buttons and FileSaver), and fflate (zip creation
for the neuron download dialog). The written content of the generated site is
licensed under CC BY 4.0, see `static/LICENSE`.

## 🆘 Support

- **User Documentation**: [User Guide](docs/user-guide.md) for installation and usage
Expand Down
57 changes: 56 additions & 1 deletion config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

Each `config.<dataset>.yaml` file configures one NeuPrint dataset (server,
dataset name, output, discovery, neuroglancer template, HTML metadata, subsets).
This document describes the optional **`hide_features`** block.
This document describes the optional **`hide_features`** block and the optional
**`neuroglancer.download_formats`** list.

## `hide_features`

Expand Down Expand Up @@ -96,3 +97,57 @@ hide_features:
neuron: ["neuroglancer"]
list: ["dimorphism", "hemilineage", "neuromere"]
```

## `neuroglancer.download_formats`

Offer per-neuron file downloads (skeletons, meshes) from the neuron pages. When
the list has at least one entry, the "Neuron Visualization" header shows a
download icon that opens a dialog with one radio button per format. The dialog
fetches one file per displayed neuron and saves them as a single zip archive.
When the list is missing or empty, neither the icon nor the dialog is rendered.

```yaml
neuroglancer:
base_url: "https://neuroglancer-demo.appspot.com/"
template: "neuroglancer-cns.js.jinja"
download_formats:
- key: swc
label: "SWC skeleton (coarse)"
kind: swc
url_template: "https://www.googleapis.com/storage/v1/b/flyem-male-cns/o/v1.0%2Fsegmentation%2Fskeletons-malecns%2Fskeletons-swc%2F{body_id}.swc?alt=media"
- key: obj
label: "OBJ mesh"
kind: ngmesh
url_template: "https://www.googleapis.com/storage/v1/b/flyem-male-cns/o/v1.0%2Fsegmentation%2Fsingle-res-meshes%2F{body_id}:0?alt=media"
```

| Field | Required | Meaning |
|---|---|---|
| `key` | yes | Unique identifier; also used in the zip file name (`<type>_<key>.zip`) |
| `label` | yes | Text of the radio button in the dialog |
| `url_template` | yes | URL with a `{body_id}` placeholder; one request per neuron |
| `kind` | no (default `swc`) | `swc` or `ngmesh`, see below |

Kinds:

| Kind | File in the zip | Behavior |
|---|---|---|
| `swc` | `<body_id>.swc` | The response is stored verbatim. Works for any text or binary per-neuron file, not only SWC |
| `ngmesh` | `<body_id>.obj` | `url_template` must address the Neuroglancer legacy mesh manifest `<body_id>:0`. The fragments it lists are fetched from the same directory and converted to Wavefront OBJ in the browser (vertices in nanometers) |

Requirements and notes:

- The host must send CORS headers, because the browser fetches the files from
the neuView page's origin (or from `file://`). For Google Cloud Storage use
the JSON API form shown above; the plain
`https://storage.googleapis.com/<bucket>/<object>` URL does not send them.
Object paths in the JSON API form are URL-encoded (`/` becomes `%2F`).
- Configuration loading fails with a clear error if `{body_id}` is missing, the
`kind` is unknown, or two entries share a `key`.
- The first entry is preselected. Order the list from cheapest to largest
download; mesh files are roughly a hundred times larger than coarse SWC files.
- Selections above 50 neurons ask the user for confirmation. Neurons without a
file are listed in `missing_body_ids.txt` inside the zip.
- The page only knows the neurons it sent to the viewer. Selections changed
inside Neuroglancer can be downloaded by pasting the viewer URL into the
dialog.
17 changes: 17 additions & 0 deletions config/config.cns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,23 @@ output:
neuroglancer:
base_url: "https://neuroglancer-demo.appspot.com/"
template: "neuroglancer-cns.js.jinja"
# Per-neuron downloads offered next to the Neuroglancer view. URLs go through
# the GCS JSON API so the browser gets CORS headers (the plain
# storage.googleapis.com URLs have none). SWC coordinates are 8 nm voxels,
# mesh vertices are nanometers.
download_formats:
- key: swc
label: "SWC skeleton (coarse, ~16 KB per neuron)"
kind: swc
url_template: "https://www.googleapis.com/storage/v1/b/flyem-male-cns/o/v1.0%2Fsegmentation%2Fskeletons-malecns%2Fskeletons-swc%2F{body_id}.swc?alt=media"
- key: swc-highres
label: "SWC skeleton (high resolution, ~300 KB per neuron)"
kind: swc
url_template: "https://www.googleapis.com/storage/v1/b/flyem-male-cns/o/v1.0%2Fsegmentation%2Fskeletons-malecns%2Fskeletons-highres-swc%2F{body_id}.swc?alt=media"
- key: obj
label: "OBJ mesh (converted from the Neuroglancer mesh, ~3 MB per neuron)"
kind: ngmesh
url_template: "https://www.googleapis.com/storage/v1/b/flyem-male-cns/o/v1.0%2Fsegmentation%2Fsingle-res-meshes%2F{body_id}:0?alt=media"

# Auto-discovery settings for neuron types
discovery:
Expand Down
17 changes: 17 additions & 0 deletions config/config.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,23 @@ output:
neuroglancer:
base_url: "https://clio-ng.janelia.org/"
template: "neuroglancer-cns.js.jinja"
# Optional: per-neuron file formats offered by a "Download" link next to the
# Neuroglancer view. Each entry needs a unique key, a label shown in the
# dialog, a url_template with a {body_id} placeholder, and a kind:
# swc the file is stored as-is (.swc)
# ngmesh Neuroglancer legacy mesh; url_template addresses the manifest
# "<body_id>:0", the browser converts the fragments to OBJ
# The host must allow cross-origin requests (CORS). For Google Cloud Storage
# use the JSON API form shown here, not the storage.googleapis.com path.
# download_formats:
# - key: swc
# label: "SWC skeleton"
# kind: swc
# url_template: "https://www.googleapis.com/storage/v1/b/<bucket>/o/<path>%2F{body_id}.swc?alt=media"
# - key: obj
# label: "OBJ mesh"
# kind: ngmesh
# url_template: "https://www.googleapis.com/storage/v1/b/<bucket>/o/<mesh-path>%2F{body_id}:0?alt=media"

html:
title_prefix: "Neuron Type Report"
72 changes: 72 additions & 0 deletions docs/developer-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,16 @@ The project relies on several key Python packages defined in `pyproject.toml`. T
**Code Quality (dev environment):**
- `ruff>=0.12.11,<0.13` - Fast Python linter and formatter

**Vendored browser libraries (`static/js/`, copied into the generated site):**
- `jquery-3.7.1.min.js` - jQuery (MIT)
- `jquery.dataTables.min.js`, `dataTables.buttons.min.js`, `buttons.html5.min.js` - DataTables and its export buttons (MIT)
- `fflate.min.js` - fflate 0.8.2 (MIT), builds the zip archive for the neuron download dialog client-side

These files are checked in rather than installed through a package manager so
that the generated site works from `file://` without a build step. To update
one, download the new UMD build, keep the one-line license header at the top
of the file, and mention the version bump in the commit message.

**Workflow Tools:**
- `parallel>=20200322,<20250623` - GNU Parallel for batch processing
- `python-lsp-server>=1.13.1,<2` - Python language server
Expand Down Expand Up @@ -401,6 +411,14 @@ Templates are organized hierarchically with inheritance and includes.
- `sections/header.html.jinja` - Page header with search
- `sections/footer.html.jinja` - Page footer
- `sections/connectivity_table.html.jinja` - Connectivity tables
- `sections/neuroglancer.html.jinja` - Embedded Neuroglancer viewer, theme toggle, and the "Download neurons" dialog (rendered only when `config.neuroglancer.download_formats` is non-empty)
- `sections/neuron_page_scripts.html.jinja` - Per-page script includes and the `neuroglancerData` bootstrap

**Browser JavaScript:**
- `templates/static/js/neuroglancer-url-generator.js.jinja` - Rendered once per build into `static/js/neuroglancer-url-generator.js` with the Neuroglancer state template, ROI tables, and the `NEUROGLANCER_DOWNLOAD_FORMATS` list embedded
- `static/js/swc-download.js` - Neuron download dialog logic (body ID collection, fetching, OBJ conversion, zip)
- `static/js/neuron-page.js`, `static/js/responsive-nav.js`, `static/js/iframe-bridge.js` - Page behavior, navigation, and the bridge used when a neuView page itself is embedded in an iframe
- `static/js/fflate.min.js`, `static/js/jquery*.js`, `static/js/*dataTables*.js` - Vendored libraries (see Key Dependencies)

### Template Context

Expand Down Expand Up @@ -774,6 +792,7 @@ Environment variable support for sensitive configuration:
**Configuration Validation** (`src/neuview/config/config.py`):
- Automatic validation with clear error messages using dataclass `__post_init__()` methods
- Example: `NeuPrintConfig.__post_init__()` validates required server and dataset fields
- Example: `DownloadFormat.__post_init__()` (in `src/neuview/config.py`) requires the `{body_id}` placeholder in `url_template` and a known `kind`; `NeuroglancerConfig.__post_init__()` turns the YAML dicts into `DownloadFormat` objects and rejects duplicate keys
- Validation occurs at configuration loading time with descriptive error messages

## API Reference
Expand Down Expand Up @@ -1591,6 +1610,59 @@ Automatic template selection based on dataset:
- FAFB datasets use specialized `neuroglancer-fafb.js.jinja` template
- Other datasets use standard `neuroglancer.js.jinja` template

### Neuron Downloads (Skeletons and Meshes)

The "Download neurons" link next to the embedded Neuroglancer view lets users
save one file per displayed neuron as a single zip archive (issue #175). The
feature is entirely client-side and only appears when the dataset config lists
at least one entry in `neuroglancer.download_formats`.

**Data flow:**

1. `config/config.<dataset>.yaml` lists formats; `NeuroglancerConfig` (`src/neuview/config.py`) parses them into validated `DownloadFormat` objects (`key`, `label`, `url_template`, `kind`, derived `extension`).
2. `NeuroglancerJSService._download_formats()` serialises the list and the JS template embeds it as `const NEUROGLANCER_DOWNLOAD_FORMATS = [...]` in the generated `static/js/neuroglancer-url-generator.js`.
3. `templates/sections/neuroglancer.html.jinja` renders the header icon and a native `<dialog>` with one radio button per format. `templates/sections/neuron_page_scripts.html.jinja` includes `fflate.min.js` and `swc-download.js` under the same condition.
4. `static/js/swc-download.js` collects body IDs, fetches the files with a small concurrency pool, converts meshes, zips everything with fflate, and triggers the download through a Blob URL.

**Body ID sources.** The embedded viewer is a cross-origin iframe, so the page
cannot read its current URL or DOM. `initializeNeuroglancerLinks()` therefore
exposes the page's own state as `window.neuviewNeuroglancerPageData`; its
`visibleNeurons` array is the default source (type neurons plus ticked
partners). As a fallback for selections changed inside the viewer, the dialog
accepts a pasted Neuroglancer URL: the `#!` fragment is decoded, parsed as
JSON, and the segments of the neuron layer are taken. The layer is found with
`findMainSegmentationLayer()` (shared with URL generation, matches `cns-seg` and
`flywire-fafb:v783b`), falling back to the first segmentation layer with
segments. Hidden segments (leading `!`) and ROI layers are skipped.

**Formats.** `kind: swc` stores the response verbatim. `kind: ngmesh` treats
the URL as the Neuroglancer legacy mesh manifest (`<body_id>:0`, a JSON list of
fragment names), derives fragment URLs by replacing `{body_id}:0` in the
template, parses each fragment (uint32 vertex count, float32 x/y/z per vertex
in nanometers, uint32 triangle indices), and writes a Wavefront OBJ with
1-based, offset-corrected face indices. Adding a new kind means extending
`DOWNLOAD_FORMAT_KINDS` in `config.py` (kind to file extension) and adding a
branch in `fetchNeuron()` in `swc-download.js`.

**CORS.** Browsers only allow `fetch()` across origins when the host sends
`Access-Control-Allow-Origin`. The plain `storage.googleapis.com/<bucket>/...`
URLs of the FlyEM buckets do not, but the GCS JSON API endpoint
(`https://www.googleapis.com/storage/v1/b/<bucket>/o/<url-encoded object>?alt=media`)
does, including for the `null` origin of `file://` pages. Configure that form.
Neuroglancer itself is unaffected because it reads through the JSON API too.

**Safeguards.** Selections above 50 neurons ask for confirmation. Missing files
(HTTP 404) are skipped and listed in `missing_body_ids.txt` inside the zip.
Mesh zips use a lower compression level because the OBJ text is large (about
6 MB per neuron before compression). The whole archive is assembled in memory.

**Tests.** `test/test_config_download_formats.py` (config parsing and
validation), `test/services/test_neuroglancer_js_service.py` (embedding in the
generated JS), and `test/test_download_dialog_gating.py` (the link, dialog and
scripts render only with formats configured). The OBJ converter has no
automated test; it was verified against an independent Python parse of a real
mesh fragment.

### Dataset Detection Patterns

Centralized dataset type detection:
Expand Down
66 changes: 66 additions & 0 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,15 @@ neuprint:
dataset: hemibrain:v1.2.1 # or cns:v1.0, optic-lobe:v1.0
```

**Neuron downloads (male CNS only):**
`config/config.cns.yaml` configures three download formats for the "Download
neurons" dialog: coarse SWC skeleton, high-resolution SWC skeleton, and OBJ
mesh, all served from the public `flyem-male-cns` bucket. The hemibrain,
optic-lobe, FAFB and wasp configs define no formats, so their pages show no
download link. See [Neuroglancer Integration](#neuroglancer-integration) for
the user-facing behavior and [Configuration Reference](#configuration-reference)
for the config format.

### Dataset Detection

The system automatically detects dataset type and adjusts:
Expand Down Expand Up @@ -534,6 +543,28 @@ The system automatically detects dataset type and adjusts:
- Multiple selection → Build complex views
- URL updates automatically

**Downloading skeletons and meshes:**

When the dataset config lists `neuroglancer.download_formats`, the Neuron
Visualization header shows a download link next to the "open in new window"
link. It opens a dialog with one radio button per configured format (for the
male CNS: coarse SWC skeleton, high resolution SWC skeleton, OBJ mesh). The
"Download" button fetches one file per neuron the page currently sends to the
viewer (the type's neurons plus ticked partners) and saves them as a single
zip file. Neurons without a file are listed in `missing_body_ids.txt` inside
the zip. Selections above 50 neurons ask for confirmation first. Progress and
errors are shown in the dialog.

Format notes: SWC coordinates are in 8 nm voxels, OBJ vertices are in
nanometers. OBJ files are converted in the browser from Neuroglancer's legacy
mesh format and are large (about 3 MB per neuron before compression).

The page cannot see selection changes made inside the embedded viewer (the
viewer runs on a different origin). To download exactly what is selected in
Neuroglancer, open the viewer in a new window, copy its URL, and paste it into
the URL field of the dialog. The page parses the state in the URL and
downloads the visible segments of the neuron layer.

**Tips:**
- Start with just the main neuron
- Add partners one at a time
Expand Down Expand Up @@ -708,6 +739,21 @@ output:
directory: "output"
clean_before_build: false

neuroglancer:
base_url: "https://neuroglancer-demo.appspot.com/"
template: "neuroglancer-cns.js.jinja"
# Optional: per-neuron downloads offered in the "Download neurons" dialog.
# Leave out (or empty) to hide the download link.
download_formats:
- key: swc
label: "SWC skeleton (coarse)"
kind: swc
url_template: "https://www.googleapis.com/storage/v1/b/<bucket>/o/<path>%2F{body_id}.swc?alt=media"
- key: obj
label: "OBJ mesh"
kind: ngmesh
url_template: "https://www.googleapis.com/storage/v1/b/<bucket>/o/<mesh-path>%2F{body_id}:0?alt=media"

performance:
max_workers: 4
cache_enabled: true
Expand Down Expand Up @@ -738,6 +784,18 @@ subsets:
- The analytics script is automatically included on all generated pages when configured
- No additional setup required - just uncomment and set your site ID

**Neuron Downloads** (`neuroglancer.download_formats`):
- Each entry needs a unique `key`, a `label` shown as a radio button, a
`url_template` containing `{body_id}`, and a `kind`
- `kind: swc` stores the fetched file as `<body_id>.swc`
- `kind: ngmesh` expects the template to address the Neuroglancer legacy mesh
manifest (`<body_id>:0`); the browser fetches the fragments and converts them
to `<body_id>.obj`
- The file host must send CORS headers. For Google Cloud Storage use the JSON
API form shown above; the plain `storage.googleapis.com` URL does not send them
- The first entry is preselected in the dialog; an empty or missing list hides
the download link entirely

### Command Reference

neuView provides a comprehensive command-line interface. All commands are run with the `pixi run neuview` prefix.
Expand Down Expand Up @@ -851,3 +909,11 @@ A: Same data, different formats. JSON for web servers/APIs, JS for local file ac

**Q: Why do some neurons have L/R pages and others don't?**
A: Automatically determined by soma distribution. Bilateral neurons get separate pages.

**Q: How do I download the skeletons or meshes of the neurons I see?**
A: Click the download icon next to the "open in new window" icon above the
Neuroglancer view, pick a format, and press "Download". You get one zip file
with one file per neuron. The page only knows the neurons it sent to the
viewer; if you changed the selection inside Neuroglancer, copy the viewer URL
and paste it into the dialog's URL field first. The link is only present when
the dataset config defines `neuroglancer.download_formats`.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "neuview"
version = "2.7.33"
version = "2.7.34"
requires-python = ">= 3.11"
authors = [
{ name = "Frank Loesche", email = "loeschef@janelia.hhmi.org" },
Expand Down
Loading
Loading