What is the difference between Selenium and Scrapy splash?

Can you scrape with Selenium?

This is an odd question, because scraping for data with Selenium feels like the same process as scraping for data with PHP.

The scraping process is just different in execution. I'm sure Selenium is more efficient than PHP, but is it the same level of efficiency? If not, what is the difference? Is there anything to make scraping with Selenium slower?

From the Selenium official website: Selenium runs on top of a browser like Firefox or Internet Explorer and uses native browser features for automation, like forms and file uploads. The browser uses the standard DOM for rendering web pages and interacting with forms, so you get a complete browser experience.

And Selenium is used for testing the web browser to see if its functions are working as they should. I think that Selenium is the same level of efficiency as PHP as it is also using the same browser engine.

Can I use Selenium with Scrapy?

Yes, you can use Selenium, it is just not that common (but the documentation is). Scrapy has its own web-driver, this driver is called 'w3lib', and you might want to use this when not in a browser.

There are only a couple of limitations you might run into. I am thinking that the latest Python 2.7. You can either use an older Python or use a browser.

Can you provide more info on this driver? I tried a little bit of Googling, but I didn't find any docs beyond the project docs (which seemed to be old) and the tutorial (I assume it applies to 1.0): Can you elaborate on your proxy issue? It seems it is something that works in a browser, so I'd like to know the exact error or error message (I never dealt with that myself, so I would need guidance). There's a workaround from 2025 that uses curl, but for some reason it doesn't work for me: Do you know of any examples on the internets or in Scrapy codebase where someone is making a call from the spider process to a browser and using its web-driver, ie selenium (if there is a need of doing that)? If I'm not mistaken the documentation is very outdated regarding such scenarios. Thanks! Here is example python module based on your question. It's working for me.

It's just python selenium example where it opens new tab of firefox and navigate to "". From selenium import webdriver. Browser = webdriver.Firefox() browser.get("") print(browser.title)