Skip to content

Models

Jump to bottom
Daniel Bogdoll edited this page Feb 21, 2025 · 8 revisions

Overview

The Mcity Data Engine supports models from four sources:

This way, models added to Hugging Face, Voxel51, and Ultralytics are automatically supported by the Mcity Data Engine by adding them to the config, as long as their interface does not change. For custom models, we provide an exemplary integration of Co-DETR, encapsulated in a container.

Model Sources

Hugging Face

For object detection, the data engine supports Hugging Face (HF) models of the types AutoModelForObjectDetection and AutoModelForZeroShotObjectDetection. Subscribing to the Transformer releases is an easy way to be notified of new models.

Voxel51

Voxel51 provides a model zoo that is leveraged for embedding models. The Voxel51 Release Notes highlight model additions.

Ultralytics

Ultralytics provides a model zoo for models of the YOLO family for the task of object detection. These are typically small models suitable for real-time edge inference. Subscribing to the Ultralytics releases is an easy way to be notified of new models.

Custom Models

Custom models run in their own containers to avoid any conflicting requirements. To build a dockerfile, mount the repository, test it, and push it do Dockerhub, run the following commands:

cd custom_models/<model>
docker build -t <dockerhub-account>/<image-name>:latest .
docker run --gpus all -v /<root>/mcity_data_engine/custom_models/<model>/<repo>:/launch <dockerhub-account>/<image-name>:latest <optional argument>
docker login
docker push <dockerhub-account>/<image-name>:latest

To run such a container with Singularity (readily available on UofM cluster), run the following command:

singularity run --nv --pwd /launch --bind /<root>/mcity_data_engine/custom_models/<model>/<repo>:/launch docker://<dockerhub-account>/<image-name>:latest <optional argument>

To run it with Docker instead, run the following command:

docker run --gpus all --workdir /launch --volume /<root>/mcity_data_engine/custom_models/<model>/<repo>:/launch <dockerhub-account>/<image-name>:latest <optional argument>

Co-DETR

Co-DETR is an object detection model and is leading the COCO test-dev leaderboard as of January 2025. Co-DETR is included as a submodule at custom_models/CoDETR/Co-DETR. A container image for CoDETR can be found at dbogdollresearch/codetr.

It is necessary to also mount the dataset used for training and increase shared memory:

docker run --gpus all --workdir /launch \
  --volume /<root>/mcity_data_engine/custom_models/CoDETR/Co-DETR:/launch \
  --volume /<root_datasets>/codetr_data/data:/launch/data \
  --shm-size=8g \
  dbogdollresearch/codetr:latest train

Currently supported configurations are co_deformable_detr/co_deformable_detr_r50_1x_coco.py and co_dino_vit/co_dino_5scale_vit_large_coco.py. They load from pre-trained weights. Download and place them in mcity_data_engine/custom_models/CoDETR/Co-DETR/models:

Mcity - Data for AI

Clone this wiki locally