UI overhaul for leaderboards

docs/credits.mdx

@@ -15,22 +15,20 @@ dependencies from the AgML project configuration.
 | opencv-python | >=4.10.0.84 |
 | opencv-python-headless | >=4.10.0.84 |
 | requests | >=2.0.0 |
-| scikit-learn | >=1.3.2 |
 | tqdm | >=4.67.0 |
 | pyyaml | >=6.0.2 |
 | albumentations | >=1.4.18 |
 | dict2xml | >=1.7.6 |
-| ipywidgets | >=8.1.5 |
 | rich | >=14.0.0 |
 | datasets | * |
-| transformers | * |
 
 ## Development dependencies
 
 | Project | Version |
 | --- | --- |
 | boto3 | >=1.35.66 |
 | scikit-image | >=0.21.0 |
+| scikit-learn | >=1.3.2 |
 | shapely | >=2.0.6 |
 | botocore | >=1.35.66 |
 | pandas | >=2.0.3 |

docs/index.mdx

@@ -18,11 +18,7 @@ We're looking to hire a postdoc with both Python library development and ML expe
 
 ## Overview
 AgML is a comprehensive library for agricultural machine learning. Currently, AgML provides
-access to a wealth of public agricultural datasets for common agricultural deep learning tasks. In the future, AgML will provide ag-specific ML functionality related to data, training, and evaluation. Here's a conceptual diagram of the overall framework.
-
-<p align="center">
-    <img src="/img/agml/agml-framework.png" alt="AgML framework" width="350" height="291" />
-</p>
+access to a wealth of public agricultural datasets for common agricultural deep learning tasks. 
 
 AgML supports both the [TensorFlow](https://www.tensorflow.org/) and [PyTorch](https://pytorch.org/) machine learning frameworks.
 
@@ -41,118 +37,58 @@ necessary prerequisites and update WSL. The latest version of WSL includes built
 
 ## Quick Start
 
-AgML is designed for easy usage of agricultural data in a variety of formats. You can start off by using the `AgMLDataLoader` to
-download and load a dataset into a container:
+AgML datasets are hosted on the [Hugging Face Hub](https://huggingface.co/Project-AgML) under the `Project-AgML`
+organization. You can start off by using the `HuggingFaceDataLoader` to download and load a dataset directly into a
+native Hugging Face `DatasetDict`:
 
 ```python
-import agml
-
-loader = agml.data.AgMLDataLoader('apple_flower_segmentation')
-```
-
-You can then use the in-built processing methods to get the loader ready for your training and evaluation pipelines. This includes, but
-is not limited to, batching data, shuffling data, splitting data into training, validation, and test sets, and applying transforms.
-
-```python
-import albumentations as A
-
-# Batch the dataset into collections of 8 pieces of data:
-loader.batch(8)
-
-# Shuffle the data:
-loader.shuffle()
+from agml.data import HuggingFaceDataLoader
 
-# Apply transforms to the input images and output annotation masks:
-loader.mask_to_channel_basis()
-loader.transform(
-        transform = A.RandomContrast(),
-        dual_transform = A.Compose([A.RandomRotate90()])
-)
+# Load a dataset from the Hub
+loader = HuggingFaceDataLoader("Project-AgML/apple_flower_segmentation")
 
-# Split the data into train/val/test sets.
-loader.split(train = 0.8, val = 0.1, test = 0.1)
+# Load a specific config/subset (e.g. an augmented variant)
+loader = HuggingFaceDataLoader("Project-AgML/apple_flower_segmentation", config="augmented")
 ```
 
-The split datasets can be accessed using `loader.train_data`, `loader.val_data`, and `loader.test_data`. Any further processing applied to the
-main loader will be applied to the split datasets, until the split attributes are accessed, at which point you need to apply processing independently
-to each of the loaders. You can also turn toggle processing on and off using the `loader.eval()`, `loader.reset_preprocessing()`, and `loader.disable_preprocessing()`
-methods.
-
-You can visualize data using the `agml.viz` module, which supports multiple different types of visualization for different data types:
-
-```python
-# Disable processing and batching for the test data:
-test_ds = loader.test_data
-test_ds.batch(None)
-test_ds.reset_prepreprocessing()
-
-# Visualize the image and mask side-by-side:
-agml.viz.visualize_image_and_mask(test_ds[0])
-
-# Visualize the mask overlaid onto the image:
-agml.viz.visualize_overlaid_masks(test_ds[0])
-```
+`HuggingFaceDataLoader` automatically casts image-like columns (`image`, `mask`, and image-typed `label` columns) to
+the Hugging Face `Image` type for decoded pixel access.
 
-AgML supports both the TensorFlow and PyTorch libraries as backends, and provides functionality to export your loaders to native TensorFlow and PyTorch
-formats when you want to use them in a training pipeline. This includes both exporting the `AgMLDataLoader` to a `tf.data.Dataset` or `torch.utils.data.DataLoader`,
-but also internally converting data within the `AgMLDataLoader` itself, enabling access to its core functionality.
+You can split the dataset into train/val/test sets, with optional stratification across one or more columns:
 
 ```python
-# Export the loader as a `tf.data.Dataset`:
-train_ds = loader.train_data.export_tensorflow()
+dataset = loader.split(val_size=0.1, test_size=0.1, stratify_cols="label")
+# Returns a DatasetDict with 'train', 'val', and 'test' splits
 
-# Convert to PyTorch tensors without exporting.
-train_ds = loader.train_data
-train_ds.as_torch_dataset()
+# Access the underlying DatasetDict at any time
+dataset = loader.dataset
 ```
 
-You're now ready to use AgML for training your own models! Luckily, AgML comes with a training module that enables quick-start training of standard deep learning models on agricultural datasets. Training a grape detection model is as simple as the following code:
-
-:::warning Deprecation Notice
-`agml.models` is deprecated and will be removed in a future release. The example below still works but will emit a `DeprecationWarning` at import time.
-:::
+**For any preprocessing, inference, or training beyond loading and splitting, use the
+[`datasets`](https://huggingface.co/docs/datasets) and [`transformers`](https://huggingface.co/docs/transformers)
+libraries directly.** Since `loader.dataset` is a native Hugging Face `DatasetDict`, it works out of the box with
+`datasets`' `map`, `filter`, and `with_transform` methods for preprocessing, and with `transformers`' `Trainer`,
+`Pipeline`, and model classes for training and inference — there's no separate AgML-specific processing API to learn.
 
 ```python
-import agml
-import agml.models
+from transformers import AutoImageProcessor, AutoModelForImageClassification, Trainer
 
-import albumentations as A
-
-loader = agml.data.AgMLDataLoader('grape_detection_californiaday')
-loader.split(train = 0.8, val = 0.1, test = 0.1)
-processor = agml.models.preprocessing.EfficientDetPreprocessor(
-        image_size = 512, augmentation = [A.HorizontalFlip(p=0.5)]
+processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
+model = AutoModelForImageClassification.from_pretrained(
+    "google/vit-base-patch16-224",
+    num_labels=dataset["train"].features["label"].num_classes,
 )
-loader.transform(processor)
-
-model = agml.models.DetectionModel(num_classes=loader.num_classes)
-
-model.run_training(loader)
-```
-
-## HuggingFace Integration
-
-AgML now includes a `HuggingFaceDataLoader` for loading datasets directly from the [Hugging Face Hub](https://huggingface.co/datasets) into native HuggingFace `DatasetDict` objects, compatible with `transformers` and `diffusers` pipelines.
-
-```python
-from agml.data import HuggingFaceDataLoader
 
-# Load a dataset from the Hub
-loader = HuggingFaceDataLoader("org/dataset-name")
+def preprocess(batch):
+    batch["pixel_values"] = processor(batch["image"], return_tensors="pt")["pixel_values"]
+    return batch
 
-# Load a specific config/subset
-loader = HuggingFaceDataLoader("org/dataset-name", config="augmented")
-
-# Split into train/val/test with optional stratification
-dataset = loader.split(val_size=0.1, test_size=0.1, stratify_cols="label")
-# Returns a DatasetDict with 'train', 'val', and 'test' splits
+dataset = dataset.with_transform(preprocess)
 
-# Access the underlying DatasetDict at any time
-dataset = loader.dataset
+trainer = Trainer(model=model, train_dataset=dataset["train"], eval_dataset=dataset["val"])
+trainer.train()
 ```
 
-The `HuggingFaceDataLoader` automatically casts image-like columns to the HuggingFace `Image` type for decoded pixel access. Stratified splitting is supported across one or more columns using a composite key.
-
 ## Public Datasets
 
 AgML contains a wide variety of public datasets from various locations across the world:
@@ -184,17 +120,15 @@ the dataset locally from which point it will be automatically loaded from the di
 From this point, the data within the loader can be split into train/val/test sets, batched, have augmentations and transforms
 applied, and be converted into a training-ready dataset (including batching, tensor conversion, and image formatting).
 
-To see the various ways in which you can use AgML datasets in your training pipelines, check out
-the [example notebook](https://github.com/Project-AgML/AgML/blob/main/examples/AgML-Data.ipynb).
-
 ## Annotation Formats
 
 A core aim of AgML is to provide datasets in a standardized format, enabling the synthesizing of multiple datasets
-into a single training pipeline. To this end, we provide annotations in the following formats:
+into a single training pipeline. Datasets on the Hugging Face Hub encode annotations as columns on the underlying
+`Dataset`/`DatasetDict`:
 
-- **Image Classification**: Image-To-Label-Number
-- **Object Detection**: [COCO JSON](https://cocodataset.org/#format-data)
-- **Semantic Segmentation**: Dense Pixel-Wise
+- **Image Classification**: a `label` column of type `ClassLabel`.
+- **Object Detection**: an `objects` column, holding COCO-style bounding boxes (un-normlized [x_min, y_min, width, height]) and corresponding category IDs as a ClassLabel per image.
+- **Semantic Segmentation**: a `mask` column, a single-channel (`L`-mode) image the same size as the corresponding image.
 
 ## Contributions
 

docusaurus.config.ts

@@ -61,66 +61,23 @@ const config: Config = {
     },
     navbar: {
       title: 'AgML',
-      logo: {
-        alt: 'AgML Logo',
-        src: 'img/agml/agml-logo-wide.png',
-      },
       items: [
         {
           type: 'docSidebar',
           sidebarId: 'agmlSidebar',
           position: 'left',
           label: 'Docs',
         },
-        {to: '/datasets', label: 'Dataset Search', position: 'left'},
+        {to: '/datasets', label: 'Datasets', position: 'left'},
+        {to: '/leaderboard', label: 'Leaderboard', position: 'left'},
         {
           href: 'https://github.com/Project-AgML/AgML',
-          label: 'GitHub',
+          label: 'View on GitHub',
           position: 'right',
+          className: 'navbar__github-button',
         },
       ],
     },
-    footer: {
-      style: 'dark',
-      links: [
-        {
-          title: 'Docs',
-          items: [
-            {
-              label: 'Overview',
-              to: '/docs',
-            },
-          ],
-        },
-        {
-          title: 'Explore',
-          items: [
-            {
-              label: 'Dataset Search',
-              to: '/datasets',
-            },
-            {
-              label: 'AgML Library',
-              href: 'https://github.com/Project-AgML/AgML',
-            },
-          ],
-        },
-        {
-          title: 'Project',
-          items: [
-            {
-              label: 'Code of Conduct',
-              to: '/docs/code-of-conduct',
-            },
-            {
-              label: 'License',
-              to: '/docs/license',
-            },
-          ],
-        },
-      ],
-      copyright: `Copyright © ${new Date().getFullYear()} AgML. Built with Docusaurus.`,
-    },
     prism: {
       theme: prismThemes.github,
       darkTheme: prismThemes.dracula,