Heatmap retrieval can't be automated #2
Description
The heatmap is used to represent the most replayed moments of a given video. I use it to pick which specific moment of a video is the most relevant, is several matches.
The heatmap data is not available through the YouTube API, only through yt-dlp (and even then, not for age-restricted videos). And yt-dlp doesn't work when run from GitHub Actions, as YouTube blocks the request, considering it comes from a bot.
Currently, I have set the heatmap update into its own yarn run update:heatmap script, that is expected to be run locally. The data rarely changes, so I assume it's ok.
But in the future, if we want to automatically add new videos, we might want to look at alternatives. I haven't tried them but maybe:
- We could use the --cookies options of
yt-dlp, and read from Chrome/Firefox cookies. This is brittle as Cookies expires after a while, so would need to be regenerated, which means we'll need another system to use a Chrome headless to login to YouTube and save the Cookies somewhere for the main script to use. Doable, but probably messy. - https://github.com/Benjamin-Loison/YouTube-operational-API seems to be an alternative YouTube API that does include the heatmap. There is no public API endpoint to test it (it has been taken down by YouTube which somehow vouch for its quality), but the repo can be cloned and installed. Maybe we can deploy it on fly.io and request it when we need it? Or maybe it will be plagued with the same bot restrictions. Only one way to find out...
- https://github.com/TarunTomar122/most-replayed-part-of-video seems to include code and video explaining how to get that data.