LLM-FQK-T2M: LLM-Guided Fuzzy Kinematic Modeling for Resolving Kinematic Uncertainties and Linguistic Ambiguities in Text-to-Motion Generation

The official PyTorch implementation of the paper "LLM-Guided Fuzzy Kinematic Modeling for Resolving Kinematic Uncertainties and Linguistic Ambiguities in Text-to-Motion Generation".

Please visit our webpage for more details.

Getting started

This code was tested on Windows11 24H2 and requires:

• Python 3.7.13
• PyTorch 1.7.1
• conda3 or miniconda3

1. Setup FFMPEG

• Download ffmpeg from https://www.ffmpeg.org/download.html#build-windows
• Extract it in C:\ffmpeg.
• Add C:\ffmpeg\bin in PATH environment variable.

2. Setup miniconda environment

conda env create -f environment.yml
conda activate cmdm
python -m spacy download en_core_web_sm
pip install git+https://github.com/openai/CLIP.git

• Download dependencies:

bash protos/smpl_files.sh
bash protos/glove.sh
bash protos/t2m_evaluators.sh

3. Get datasets

HumanML3D

• Follow the instructions in HumanML3D, then copy the resultant dataset to this repository:

cp -r ../HumanML3D/HumanML3D llm-fqk-t2m/dataset/HumanML3D

KIT

• Download from HumanML3D and the place result in ./dataset/KIT-ML

4. Get the pretrained models

• Download the pretrained models and place then unzip and place them in ./checkpoints/.

• HumanML3D (cmdm_humanml3d)

• KIT (cmdm_kitml)

5. Setup LLaMA3.1

• Download the install Ollama: https://ollama.com/download
• Run following command in command prompt:

ollama run llama3.1:8b

Motion Generation

Generate from a single prompt

python -m sample_cmdm.sample --model_path ./checkpoints/cmdm_humanml3d_000294000/cmdm_humanml3d_000294000.pt --text_prompt "a person is jumping on floor" --cuda True --motion_length 9

Generate from test set prompts

python -m sample_cmdm.sample --model_path ./checkpoints/cmdm_humanml3d_000294000/cmdm_humanml3d_000294000.pt --num_samples 10 --num_repetitions 3

Generate from text file

python -m sample_cmdm.sample --model_path ./checkpoints/cmdm_humanml3d_000294000/cmdm_humanml3d_000294000.pt --input_text ./assets/sample_prompts.txt

You may also define:

• --device id.
• --seed to sample different prompts.
• --motion_length in seconds (maximum is 9.8[sec]).

Running those will get you:

• motions.npy file with text prompts and xyz positions of the generated animation
• prompt##_rep##.mp4 - a stick figure animation for each generated motion.

It will look something like this:

Rendering SMPL meshes in Blender

• Download and install blender https://www.blender.org/download/.
• {VER} = your blender version, replace it accordingly.
• Blender>Preferences>Interface> Check Developer Options
• Add the following paths to PATH environment variable.

C:\Program Files\Blender Foundation\Blender {VER}
C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin

• Run CMD as Administrator and follow these commands:

"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m ensurepip --upgrade
"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m pip install matplotlib --target="C:\Program Files\Blender Foundation\Blender {VER}\{VER}\scripts\modules"
"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m pip install hydra-core --target="C:\Program Files\Blender Foundation\Blender {VER}\{VER}\scripts\modules"
"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m pip install hydra_colorlog --target="C:\Program Files\Blender Foundation\Blender {VER}\{VER}\scripts\modules"
"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m pip install shortuuid --target="C:\Program Files\Blender Foundation\Blender {VER}\{VER}\scripts\modules"
"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m pip install omegaconf --target="C:\Program Files\Blender Foundation\Blender {VER}\{VER}\scripts\modules"
"C:\Program Files\Blender Foundation\Blender {VER}\{VER}\python\bin\python.exe" -m pip install moviepy==1.0.3 --upgrade  --target="C:\Program Files\Blender Foundation\Blender {VER}\{VER}\scripts\modules"

• Download Blender Animation Files: https://drive.google.com/file/d/1EbUkwPCt7eB9HAZXtM-nCxsztkfsRImD/view?usp=sharing

• To create SMPL mesh per frame run:

python -m visualize.render_mesh --input_path /path/to/mp4/stick/figure/file

This script outputs:

• prompt##_rep##_smpl_params.npy - SMPL parameters (thetas, root translations, vertices and faces)
• prompt##_rep##_obj - Mesh per frame in .obj format.

Train CMDM

HumanML3D

python -m train.train_cmdm --save_dir checkpoints/cmdm_humanml3d_1 --dataset humanml

KIT

python -m train.train_cmdm --save_dir checkpoints/cmdm_kitml_1 --dataset kit

• Use --diffusion_steps 200 to train the faster model with less diffusion steps.
• Use --device to define GPU id.
• Add --eval_during_training to run a short (90 minutes) evaluation for each saved checkpoint. This will slow down training but will give you better monitoring.

Evaluation

• Single GPU

HumanML3D

python -m evaluations.eval_humanml --model_path ./checkpoints/cmdm_humanml3d_000294000/cmdm_humanml3d_000294000.pt

KIT

python -m evaluations.eval_humanml --model_path ./checkpoints/cmdm_kitml_000294000/cmdm_kitml_000294000.pt

Read Paper

Bibtex

@article{manjotho2025llmfqkt2m,
author = {Ali Asghar Manjotho and Tekie Tsegay Tewolde and Ramadhani Ally Duma and Zhendong Niu},
title = {LLM-guided fuzzy kinematic modeling for resolving kinematic uncertainties and linguistic ambiguities in text-to-motion generation},
journal = {Expert Systems with Applications},
volume = {279},
pages = {127283},
year = {2025},
issn = {0957-4174},
doi = {https://doi.org/10.1016/j.eswa.2025.127283},
url = {https://www.sciencedirect.com/science/article/pii/S0957417425009054},
}

Acknowledgments

This code is based on the MDM implementation. We also thank the following contributors for the progressive development of the domain:

guided-diffusion, MotionCLIP, text-to-motion, actor, joints2smpl, MoDi.

License

This code is distributed under an MIT LICENSE.

Note that our code depends on other libraries, including CLIP, SMPL, SMPL-X, PyTorch3D, and uses datasets that each have their own respective licenses that must also be followed.