only subset of models to train after loading DLC data

[frikyng]

Hi,

I want to use LightningPose with datasets that I label in DLC (b/c you can do k-means clustering). However, when I import the dataset into LightningPose I have 2 issues:

1. The GUI doesn't give me feedback that the import has been successful (it stays gray but the labelled data is present in the label frames tab)
2. I can only selected from supervised and semi-supervised models. I would like to train with the context option. What is the reason they are not available?

FYI I am running the GUI locally. Also, I have added a config.toml for streamlit so I can add larger videos.

Thanks and best,
Friedrich

[themattinthehatt]

Hi @frikyng,

0. I'll point out that we also do k-means clustering in the LP GUI!
1. thanks for pointing this out - there should definitely be some kind of indication that import is successful, I'll look into this
2. this is the advantage of extracting/labeling frames with the LP GUI - when we select fram $n$ for labeling, we automatically extract frames $n-2$, $n-1$, $n+1$, and $n+2$, which are required for the context models (though you do not need to labels these extra context frames). The GUI looks for these context frames in the DLC-labeled frames you've uploaded, and if they're not present then you're not able to train the context models. One thing that you can do to enable the context models is, for each labeled frames, manually extract the context frames yourself as png files, and upload those into the labeled data folder. The naming scheme should follow the same pattern as the frames that are already present (see more info here: https://pose-app.readthedocs.io/en/latest/source/tabs/extract_frames.html#upload-zipped-files-of-frames). I'm happy to discuss this in more detail too if something isn't clear there.

I also want to note that we are a week or two away from releasing version 2.0 of the LP GUI, and it's going to be a much nicer user experience. I'll let you know when it's released!

[frikyng]

Hi @themattinthehatt,

thanks for the quick answer!

A couple more thoughts:
1 - It looks like it's just a bug. What happens is that the GUI never transitions back to give you the green message telling you the credentials for the labelling etc. (like it happens in the demo video). Maybe this is just because I was uploading a larger folder with a single large video (folder size 1.6GB) and it made it hang up on something.

2 - What I actually did to capture interesting frames across all my data to stay within a relatively small file size that is compatible with LightningPose. That is I was applying k-means to extract interesting frames and getting +/- 10 adjacent frames. Then I made a fake video from those that I put into DLC to label. So the contextual data is actually present.

• Is it possible to grab the supplementary frames from the videos contained in the DLC data too?
• The main reason that I decided to use DLC in the first place is that the labelling in Label Studio is quite clunky. DLC has this mode where you click a feature and it automatically advances to the next frame, which makes the labelling process much faster. If this were in LightningPose I would happily do everything there.

Looking forward to version 2 of the GUI 🙌 lmk when it's there!

Another thing I just remembered is that in the inference step I can only ever upload up to 6 videos. LightningPose will hang up on the inference of the 7th, which may not just be a GUI issue.
(Again, I am running my stuff locally and am using an RTX4090, which has plenty of power and memory)

Cheers,
Friedrich

[themattinthehatt]

So to make sure I understand correctly, are the videos you made just 21 frames long each? i.e. the central "interesting" frame +/-10 frames? and then you label all 21 frames? Or are you concatenating contiguous 21-frame chunks into a longer video and labeling from there?

I definitely understand the clunkiness of the Label Studio interface. That's part of the reason for moving to version 2, where we implemented our own custom labeler that will allow us to address pain points like this more easily.

For the inference step, not sure why you're having issues on the 7th video. If they're large videos it might be an issue with the memory consumed in the browser uploading through streamlit. If you're at a point where you're happy with your model and want to run inference at scale I'd recommend doing it from the command line: https://pose-app.readthedocs.io/en/latest/source/guides/inference_on_large_videos.html

[frikyng]

I am concatenating chunks of 21 frames. As I didn't know that LP is doing k-means and I hadn't yet figured I could increase the file size on the local machine, I manually picked 200 or so frames from my own k-means procedure and extracted them along with the temporal context. This was to get the best training set but stay within the file size limits. When I then labelled in DLC and uploaded the project to LP it didn't let me choose the context models.

Great! Looking forward to the new version!

Yeah the command line is probably better. For my 'production runs' I'll move to inferencing from there.