Models

Overview

The Mcity Data Engine supports models from three sources:

• Hugging Face
• Ultralytics
• Custom models

This way, models added to both Hugging Face and Ultralytics are automatically supported by the Mcity Data Engine by adding them to the config. For custom models, we provide an exemplary integration of Co-DETR, encapsulated in a container.

Model Sources

Hugging Face

For object detection, the data engine supports Hugging Face (HF) models of the types AutoModelForObjectDetection and AutoModelForZeroShotObjectDetection. Subscribing to the Transformer releases is an easy way to be notified of new models.

Ultralytics

Custom Models

Custom models run in their own containers to avoid any conflicting requirements. To build a dockerfile, mount the repository, test it, and push it do Dockerhub, run the following commands:

cd custom_models/<model>
docker build -t <dockerhub-account>/<image-name>:latest .
docker run --gpus all -v /<root>/mcity_data_engine/custom_models/<model>/<repo>:/launch <dockerhub-account>/<image-name>:latest <optional argument>
docker login
docker push <dockerhub-account>/<image-name>:latest

To run such a container with Singularity (readily available on UofM cluster), run the following command:

singularity run --nv --pwd /launch --bind /<root>/mcity_data_engine/custom_models/<model>/<repo>:/launch docker://<dockerhub-account>/<image-name>:latest <optional argument>

To run it with Docker instead, run the following command:

docker run --gpus all --workdir /launch --volume /<root>/mcity_data_engine/custom_models/<model>/<repo>:/launch <dockerhub-account>/<image-name>:latest <optional argument>

Co-DETR

Co-DETR is an object detection model and is leading the COCO test-dev leaderboard as of January 2025. Co-DETR is included as a submodule at custom_models/CoDETR/Co-DETR. A container image for CoDETR can be found at dbogdollresearch/codetr.

It is necessary to also mount the dataset used for training and increase shared memory:

docker run --gpus all --workdir /launch \
  --volume /<root>/mcity_data_engine/custom_models/CoDETR/Co-DETR:/launch \
  --volume /<root_datasets>/codetr_data/data:/launch/data \
  --shm-size=8g \
  dbogdollresearch/codetr:latest train