Scraping no longer works - requires chromedriver?

[ltning]

Hi,

until some time earlier this year I was using Koillection's scraping feature heavily, with great success. I'm running it on my own FreeBSD installation.

After updating today and attempting to scrape something, I get "chromedriver" binary not found. Install it using the package manager of your operating system or by running "composer require --dev dbrekelmans/bdi && vendor/bin/bdi detect drivers".

Has the scraping implementation changed in the last few months? Is there a way to use the old one?

Thanks!

[ksawyerwsu]

Same issue also observed via proxmox instance setup. It looks like default didn't have some of the chrome tools installed, but I have done apt install chromium, as well as installing from a wget/.deb. When running the command from the error message, it returns mulitple valid chrome instances now, but the error message still persists.

[ltning]

I've never had, nor do I want, chromium/chromedriver or any other browser installed. Until at least 0.7.1 scraping has worked fine without - I presume there is a lighter library in use.

[ksawyerwsu]

I've never had, nor do I want, chromium/chromedriver or any other browser installed. Until at least 0.7.1 scraping has worked fine without - I presume there is a lighter library in use.

Fair. Seems to be the impression I had as well since the prox container didn't install it. Was just trying to do some debug and provide information.

[benjaminjonard]

I've moved the scraping library to use a real browser. Due in part to massive AI scraping of websites, more and more of them are blocking all kinds of scraping, and the previous implementation started failing on a growing number of sites. Using a real browser mitigates this for now. I'll try to keep both systems working together for the next release, but I think fewer and fewer websites will support the old approach.

[ltning]

Thanks @benjaminjonard for confirming I'm not (entirely) crazy.
I'm lucky enough that the sites I'm scraping from are (still) friendly, and I know the operators.

My problem (and I realise I'm in a minority here :) is that on a server platform, installing such very large dependencies - including their deps - may not be desirable. And on my platform (FreeBSD) the choices are limited. In a strictly sandboxed and security-controlled environment, browsers, even chromedriver, won't even run without significant changes.

I'd greatly appreciate if both approaches would continue to work for a while longer :) I put a lot of work into the scraping setup and it is working extremely well for my use case. I am very grateful for the work you've done on Koillection!

[ksawyerwsu]

I've moved the scraping library to use a real browser. Due in part to massive AI scraping of websites, more and more of them are blocking all kinds of scraping, and the previous implementation started failing on a growing number of sites. Using a real browser mitigates this for now. I'll try to keep both systems working together for the next release, but I think fewer and fewer websites will support the old approach.

Understandable, thanks for the work on this. I'll keep poking at what I can do to make the browser version work on my end.

[shineangelic]

So is that "normal" that using the prox container scraping on 1.8.1 is not working?

I get the
"chromedriver" binary not found. Install it using the package manager of your operating system or by running "composer require --dev dbrekelmans/bdi && vendor/bin/bdi detect drivers".

error message

[shineangelic]

I just tested it on 1.8.2 and it looks working to me, thanks

[ltning]

Excellent, thank you @benjaminjonard, also for the MariaDB fixes! I'll test the new release as soon as $dayjob relents a bit..