Datasets
This page gives an overview of the available datasets in the Mcity Data Engine. The Mcity Data Engine uses fiftyone by Voxel51 internally to represent datasets. This means that any dataset, once imported, can be used with the Data Engine. All necessary information to load a dataset are stored in config/datasets.yaml and used by utils/dataset_loader.py to load the dataset. Using a dataset with the engine is done by choosing a name from datasets.yaml and setting it as the SELECTED_DATASET variable in the config.py.
If a dataset comes with labels, the Data Engine expects them in the field ground_truth. Splits should be defined by sample tags train, val, or test. For training, the Data Engine expects all three splits to be present. We perform some automatic post-processing on datasets to help with slight deviations from these expectations. Voxel51 bounding boxes are defined in the format top_left_x, top_left_y, w, h in rel. coordinates between 0 and 1.
The Mcity Data Engine provides an efficient conversion from a Voxel51 dataset to a Torch dataset for training and dataset iterations. Torch datasets follow the COCO format with bounding boxes in top_left_x, top_left_y, w, h in absolute pixel coordinates.
The Mcity Data Engine provides a conversion from a Torch dataset to a Hugging Face dataset for training and efficient dataset iterations. Hugging Face datasets follow the COCO format with bounding boxes in top_left_x, top_left_y, w, h in absolute pixel coordinates.
To upload datasets to Hugging Face with Voxel51, load private datasets from Hugging Face, and run the tests successfully, you need to set a Hugging Face token. For local usage, create a .secrets file in the root folder with the following content:
HF_TOKEN=<YOUR_HF_TOKEN>
To support private datasets during GitHub Actions, create a secret in Github Actions with the same name and content as above.
- The dataset needs to be available locally or on Huggingface in the Voxel51 format
- The config/datasets.yaml file needs to be extended by an entry. Necessary fields are
nameandload_fct, and eitherlocal_pathto load a dataset from disk orhf_dataset_nameto load it from Huggingface. - In utils/dataset_loader.py, a function with the same name as defined by
load_fctneeds to be implemented to load and return the dataset.
The public Fisheye8K dataset consists of 8,000 annotated images with 157K bounding boxes of five object classes. The Voxel51/fisheye8k version is provided on Hugging Face in the Voxel51 format under Voxel51/fisheye8k. All necessary downloading is performed within the Data Engine itself.
The 'SUN RGB-D' public dataset is supported for local loading. After downloading it manually, the local_path in the datasets.yaml needs to be set accordingly. It contains 10,335 RGB-D images, each of which has a corresponding RGB image, depth image, and camera intrinsics.
⚠️ The loading functions for MARS in utils/dataset_loader.py are not yet fully implemented. However, as MARS follows the NuScenes format, the pip environment includes a custom nuscenes-devkit compatible with the other packages, so the environment is prepared to load the dataset. Details on how to load a Nuscenes dataset with Voxel51 can be found in their Blog.
The Open Mars Dataset is a multimodal autonomous vehicle dataset recorded in Ann Arbor. It consists of the parts mars_multitraversal and mars_multiagent.
To download it, make sure you have entered your SSH key at Hugging Face. Download the MARS dataset with
sudo apt-get install git-lfs
git lfs install
git clone git@hf.co:datasets/ai4ce/MARS
The mcity_fisheye_2000 dataset is currently only available internally at Mcity. Download the data with
cd datasets
scp -r <uniqname>@lighthouse.arc-ts.umich.edu:/nfs/turbo/coe-mcity/tinghanw/midadvrb_2000 .
If images/val contains json files, delete them.
The mcity_fisheye_2100_vru dataset is an extended version of the Mcity Fisheye 2000 dataset that includes 100 additional frames where only vulnerable road users (VRU) were labeled.
This dataset is currently only available internally at Mcity. It is located at
/nfs/turbo/coe-mcity/dbogdoll/datasets/midadvrb_2100
The mcity_fisheye_3_months is a small selection of frames within a time period of 3 months. dataset is currently only available internally at Mcity. It is located at
/nfs/turbo/coe-mcity/tinghanw/midadv_swinl_label_veh_0p3_ped_0p3_2023q4_yolov8_format_v2
The annarbor_rolling Mcity dataset can be generated on a rolling basis based on the provided AWS download code. It creates an AWS S3 bucket with all the available data between two dates, downsampled to a requested framerate. Based on the aws_download workflow, this data can be downloaded locally, extracted, and made available as a Voxel51 dataset.
This dataset is currently only available internally at Mcity. It is located at
/nfs/turbo/coe-mcity/dbogdoll/datasets/dataengine_rolling/<requestes-framerate>
As part of the aws_download workflow, the dataset naming scheme is
now = datetime.datetime.now()
datetime_str = now.strftime("%Y-%m-%d_%H-%M-%S")
dataset_name = f"annarbor_rolling_{datetime_str}"
Thus, to load the dataset as part of a workflow, the name in the datasets.yaml needs to be changed accordingly.
The mcity_fisheye_ano_ped dataset is based on the Mcity Fisheye 2000 (labeled) dataset. It computes the splits newly with a train split that does not contain any pedestrians, and a val split which has pedestrians on every frame. This way, pedestrians can be treated as anomalies. It was first designed to evaluate the Anomalib library.
This dataset is currently only available internally at Mcity. It is located at
/nfs/turbo/coe-mcity/dbogdoll/datasets/ano_ped