Support for any-point trackers

[sfmig]

Is your feature request related to a problem? Please describe.
The main goal of ethology is to facilitate the application of a wide range of computer vision tasks to animal behaviour research, by providing a unified data analysis interface across these tasks.

Any-point tracking is a good example of a computer vision task that is maturing within the field of computer vision, but it still relatively inaccessible to animal behaviour researchers. The task consists on the following: given a video and a set of query points on a frame of that video, predict the location of those points on every other frame of the video. This is a more general problem than the pose estimation one, which typically focuses on predicting the location of a fixed set of keypoints on an animal's body. As a result, any-point tracking tools could prove to be a very valuable tool for studying animal behaviour.

Describe the solution you'd like
Depending on the quality of the trajectories generated by any-point trackers, these could be useful to study the movement patterns of animals directly, or they may be more helpful as a semi-automatic way to quickly generate labelled data. In recent years, there has been an increase in the development of any-point trackers, such as cotracker3 and TAPIR. The goal would be for users to easily apply these tools to their data.

Also consider Loco track B, from papers with code benchmarks.

Describe alternatives you've considered
\

Additional context

• cotracker3 paper and code
• TAPIR paper and code

[Kannav02]

Hey @sfmig , I was going through all of the papers and libs mentioned

At this point I have an idea, The idea for this would be to have a unified API/ Base class, in leyman terms a prototype class, to unify all the methods for different Libraries like cotracker3 and TAPIR, then we can go about implementing those as concert classes , so at the same time we do have different libraries to test and evaluate , and also providing support for future addition to different libs

Apart from this, support for visualisation and postprocessing can be clubbed into another module specifically for any-point-tracking

I would love to work on this if possible!, Thank you!

[sfmig]

Hi @Kannav02, thanks for your interest!

That sounds like a good way to structure this, but as always the proof is in the pudding! 😁 Would you be up to opening a draft PR that demonstrates the basic idea of your proposed approach?

Even if you aim to go with a prototype class approach, you may want to start focussing on one of the two libraries, to reduce a bit the complexity while you get familiar with their codebases.

I'd recommend reading carefully our Contributing guidelines before opening a draft PR. As you can see ethology is still very early stages, but we closely follow movement's approach, so I also leave you the link to the movement contributing docs too as it may provide more context.

[Kannav02]

Sure, that sounds like a great point to begin with, thank you!, I will update you on this soon

[HARSHDIPSAHA]

Hi @sfmig , can we still work on this?

[sfmig]

Hi @HARSHDIPSAHA, thanks for checking!

I would actually recommend against having a go at this issue as a first time contributor to this repo, since it is part of a long term plan and many things would have to be defined before this. I would recommend having a look at some of our other repos in the neuroinformatics org and check if there are easier issues you can help with.

Thanks for your interest!

[HARSHDIPSAHA]

Thanks for the recommendation @sfmig , I read ethology docs, and understood the repo structure and files, and this issue seemed challenging so I thought to give it a try ! I’m happy to start with smaller or more defined issues first to better familiarize myself with the codebase.. Also i have been already contributing to brainglobe repos:)