diff --git a/docs/credits.mdx b/docs/credits.mdx index c634e90..92003fc 100644 --- a/docs/credits.mdx +++ b/docs/credits.mdx @@ -15,15 +15,12 @@ dependencies from the AgML project configuration. | opencv-python | >=4.10.0.84 | | opencv-python-headless | >=4.10.0.84 | | requests | >=2.0.0 | -| scikit-learn | >=1.3.2 | | tqdm | >=4.67.0 | | pyyaml | >=6.0.2 | | albumentations | >=1.4.18 | | dict2xml | >=1.7.6 | -| ipywidgets | >=8.1.5 | | rich | >=14.0.0 | | datasets | * | -| transformers | * | ## Development dependencies @@ -31,6 +28,7 @@ dependencies from the AgML project configuration. | --- | --- | | boto3 | >=1.35.66 | | scikit-image | >=0.21.0 | +| scikit-learn | >=1.3.2 | | shapely | >=2.0.6 | | botocore | >=1.35.66 | | pandas | >=2.0.3 | diff --git a/docs/development.mdx b/docs/development.mdx index 1e345d7..7f09cdf 100644 --- a/docs/development.mdx +++ b/docs/development.mdx @@ -10,211 +10,216 @@ Thank you for choosing to contribute to AgML! ## Contributing Data If you've found (or already have) a new dataset and you want to contribute the dataset to AgML, -then the instructions below will help you format and the data to the AgML standard. +then the instructions below will help you format the data to the AgML standard and publish it +through the Hugging Face Hub, which is how all new datasets are distributed. ### Dataset Formats Currently, we have image classification, object detection, and semantic segmentation datasets available -in AgML. These sources are synthesized to standard annotation formats, namely the following: +in AgML. Every dataset is published as a [Hugging Face `datasets`](https://huggingface.co/docs/datasets) repository +under the `Project-AgML` organization, backed by Parquet, with a `dataset_info` config describing its columns and +splits. The column layout depends on the task: -- **Image Classification**: Image-To-Label-Number -- **Object Detection**: [COCO JSON](https://cocodataset.org/#format-data) -- **Semantic Segmentation**: Dense Pixel-Wise +- **Image Classification**: an `image` column plus a `label` column of type `ClassLabel`. +- **Object Detection**: an `image` column plus an `objects` column holding COCO-style bounding boxes and category IDs. +- **Semantic Segmentation**: an `image` column plus a `mask` column (single-channel `L`-mode image). + +`HuggingFaceDataLoader` (in `agml/data/hf_loader.py`) relies on these names to verify that columns were cast correctly correctly upon loading, so new datasets must follow this schema +for the loader to work. #### Image Classification -Image classification datasets are organized in the following directory tree: +Build a `datasets.Dataset` (or `DatasetDict`, if you have predefined splits) with two columns: -``` - - ├──