-
Notifications
You must be signed in to change notification settings - Fork 3
Environments
The Mcity Data Engine is designed to run in three types of environments:
We run our tests on a developer machine with the following hardware specifications:
- GPU: 1 x NVIDIA RTX 4090 (24 GB)
- CPU: Intel i9-13900K (32 core)
- RAM: 64 GB
The software specifications can be found in the README.
The clusters Great Lakes and Lighthouse are available for training and inference given UofM affiliation. The compatibility with the provided OS, python versions etc. is tested with every update of the requirements.txt in the Lighthouse Build action.
We run the Data Engine on the following hardware specifications:
- GPU: 8 x NVIDIA H 100 (80 GB)
- CPU: Intel Xeon Gold 6448Y (64 core)
- RAM: 128 GB
To use an interactive VS Code session, follow this Lighthouse Visual Studio Code tutorial and setup python/3.11.5.
Weights and Biases Launch is deprecated. As an expected support of SLURM does not seem to be implemented anytime soon, it was decided to remove the integration from the main branch. If you are interested in how to use WandB Launch, take a look at the wandb-launch branch.
Based on WandB Launch runs can be executed in dedicated Docker containters per run. The Dockerfile.wandb defines that environment.
To fill queues with your local machine, you need to setup an agent:
wandb launch-agent -q <queue-name> --max-jobs <n>
In order to execute jobs on your own machine, the following tools need to be installed:
To launch a job on your machine with Docker, an active agent needs to be running in a terminal. Adapt the config as necessary. Then, run
wandb launch --uri "git@github.com:mcity/mcity_data_engine.git" --job-name <name-run> --project mcity-data-engine --entry-point "python main.py" --dockerfile Dockerfile.wandb --queue data-engine
Locally, you will need to clean up old docker images once in a while. Run docker image prune --all --filter until=48h to delete docker images older than 48 hours.