Method for live inference?

[jambory]

Hi Lightning Pose Team,

I have been working with a DLC model and would like to try a lightning pose model, but for my use case, I need the model to be able to perform live inference on a video feed. Is there any way you could suggest going about that or does this just not fit the model architecture?

Thanks

[themattinthehatt]

Hi @jambory, yeah Lightning Pose should in theory be just fine with live inference. The context models might be trickier, since they require 5 frames and are (at the moment) acausal, but a standard supervised model will work just like a DLC model. You could lightly rewrite this function:

lightning-pose/lightning_pose/models/heatmap_tracker.py

Line 134 in 73799ce

def get_loss_inputs_labeled(

like:

        # images -> heatmaps
        predicted_heatmaps = model.forward(frame)
        # heatmaps -> keypoints
        predicted_keypoints, confidence = model.head.run_subpixelmaxima(predicted_heatmaps)
        # bounding box coords -> original image coords
        predicted_keypoints = convert_bbox_coords(batch_dict, predicted_keypoints)

in your inference loop.

The one thing you'll have to do is mock a batch_dict to have the info that convert_bbox_coords needs, but take a look and let me know if you have any questions!

[themattinthehatt]

If you have a code snippet for how you currently use the DLC model in your setup that would be helpful too

[jambory]

Thanks for the quick reply! I got the setup ready and am currently training the model, I'll let you know if I can get it to work or if I have any questions!

[themattinthehatt]

Yes please keep me updated, I'm very curious how it goes! I don't know of anyone else using LP for closed loop experiments yet.

Which backbone are you using? And what is the frame rate of your video feed?