Why use Selenium with Scrapy?

Which is better beautiful soup Selenium or Scrapy?

I'm interested in which of these two tools is better.

BeautifulSoup has very low memory usage, whereas Scrapy has higher memory requirements. Scrapy requires Python 2.6 or later, whilst BeautifulSoup has 3.x compatibility.

Scrapy can be used for a variety of tasks, whereas BeautifulSoup can only use the parser module for parsing HTML. I am personally much more comfortable with BeautifulSoup, but for most web scraping problems, it is easy enough to switch to Scrapy if needed. I like BeautifulSoup because it is fast and there is more documentation, however the Scrapy project looks really promising. Check out Scrapy docs, its pretty cool.

Is Scrapy Splash faster than Selenium?

Is Scrapy Splash able to execute multiple requests at once, like Selenium, ie load 50 webpages at once and then store their data? I have never used Selenium in Python before but I would like to check if Scrapy Splash is faster than Selenium as some of my tests may execute multiple requests at once and I don't want to get too overwhelmed by waiting for a few minutes on each request as that's what I don't want in Scrapy. Or alternatively: should I just use Selenium for most tests where one doesn't have a need for multiple scraped items from the same webpages, only fetching the first few webpages of a website and getting back their data, or is it overkill to use Selenium for such cases, especially when the number of requests is not critical. What are your experiences with Selenium when executing multiple requests at once? Selenium does the scraping and saving of the data of the rendered web pages. When the requests are sent, that is the moment the pages are rendered into DOM so for each request a new set of pages is rendered. Then selenium picks up the pages, writes the information on them and saves it in memory. If the memory runs low, that's when it stops executing these requests.

Scrapy Splash does exactly the opposite, it sends the requests without rendering the pages into DOM and saves the rendered pages into files, so that the next requests for the same pages can load the information from the files. So the answer is that yes, Scrapy Splash can save many more requests at once because it does not have to render the pages multiple times.

Does Scrapy use Selenium?

Scrapy uses a fork of Selenium which is called Selenium 2 and is available from here: .

I am quite new to python, programming and Selenium, so if anyone knows the reason why Scrapy is using a non-standard Selenium version and what that entails for future development of Scrapy and perhaps even scrapy-selenium package? Please let me know. Thanks.

No. The whole idea of scrapy is that it only loads parts of your website and scrapes just those. So when you run scrapy crawl spider -o response.txt that means scrapy will not scrape any further than the url's in response.