Bot reports 403 errors sometimes

[feld]

Sometimes the bot reports 403 errors for URLs it cannot scrape

It should probably just be silent on errors for any URL it cannot produce a preview for

[01/23/25 23:32:07] ERROR    403 Client Error: Forbidden for url:
                             https://bitcoinmagazine.com/takes/ethereums-looming-collapse-is-a-lesson-in-blockchain-integrity
                             Traceback (most recent call last):
                               File "/home/feld/deltabots/lib/python3.11/site-packages/instantview/hooks.py", line 72, in on_msg
                                 send_preview(bot, accid, msg, url)
                               File "/home/feld/deltabots/lib/python3.11/site-packages/instantview/web.py", line 41, in send_preview
                                 resp.raise_for_status()
                               File "/home/feld/deltabots/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
                                 raise HTTPError(http_error_msg, response=self)
                             requests.exceptions.HTTPError: 403 Client Error: Forbidden for url:
                             https://bitcoinmagazine.com/takes/ethereums-looming-collapse-is-a-lesson-in-blockchain-integrity

[feld]

I tried changing the User Agent to something like Twitter or Telegram uses but it doesn't help with this URL. This is increasingly becoming a problem because a lot of sites are using anti-bot filters on their edges and CDNs to keep scrapers for AI model training from getting the data. It's breaking link previews in other software as well. Unsure if it's solvable.

[sjehuda]

This might be helpful.

Impersonation

The following provide support for impersonating browser requests. This may be required for some sites that employ TLS fingerprinting.

curl_cffi (recommended) - Python binding for curl-impersonate. Provides impersonation targets for Chrome, Edge and Safari. Licensed under MIT
Can be installed with the curl-cffi group, e.g. pip install "yt-dlp[default,curl-cffi]"
Currently included in yt-dlp.exe, yt-dlp_linux and yt-dlp_macos builds

https://github.com/yt-dlp/yt-dlp#impersonation