Which is better Selenium or Scrapy?

What are the downsides of Scrapy?

The good.

One feature I really like is that it provides full-on support for multi-processing (scraping tasks run in parallel by your program). This allows you to run long-running queries, such as downloading a very large site (eg Google), without blocking everything else. (Yes, scraping can be CPU-intensive.) However, if you are doing any kind of web scraping with Scrapy then you have no control over how many URLs are created for each process and you need to keep a list of active URLs somewhere. Scrapy takes care of this part for you, but if you use an external project manager such as Splash or Selenium (as discussed later) then there's no need to keep an explicit list of URLs. Also, in Scrapy, all processes share the same list of items, but if you use Splash or another project manager then you can easily have your own copy of each URL's data.

Spiders. In the previous section, I said Scrapy provides full-on support for multi-processing. As we have seen earlier, this is actually the case when running a spider (ie a script to visit a specific URL on the Internet and return some information, perhaps in an Excel spreadsheet format). It just so happens that many people use spiders for general purpose web scraping as well. One of the things I don't really like about Scrapy is that a Scrapy spider (script) doesn't return any information other than what's returned by visiting the specified URL in a browser. Yes, you can include more output, but it only appears on the console and not in a spreadsheet or in an email.

A common solution is to have two different scripts - one for collecting the data and another formatting and displaying it. This way you get the best of both worlds. It's up to you to decide if this is worth the extra complication or not.

I hope my examples illustrate some of the benefits and drawbacks of web scraping. There are many more options and approaches to web scraping that I didn't cover, but hopefully you can identify the ones you think might help you.

Which is better Selenium or BeautifulSoup Scrapy?

These days I am working on writing a scrapy which is going to collect some data from the online and export it to an excel sheet.

I am using PyCharm 2023. I used Python(Anaconda) version 3.6.5.

I am a beginner. Which web scraping tool should I choose between Selenium and BeautifulSoup to get the data in both the languages? As we all know that these frameworks are only used to scraping different tags. But as per my knowledge selenium requires the development process to be long and heavy and beautiful soup is easy to work.

However as per my experience the python-selenium library is not fully supported so you can try beautiful soup and learn it very easily.

Which is better Selenium or Scrapy?

The answer will depend on the type of project you are building.

Selenium tends to focus on unit tests in your main test scripts. Scrapy by contrast, has lots of different features and tools that can be used. The idea being that a single scrapy.Spider can support lots of different things:
Unit test scripts using the spiders framework. Feature files for handling endpoints using Scrapy's HttpErrorMiddleware. Endpoint classes that have various data extraction logic. Custom middlewares. Tests and examples. Scraping tools like scrapyd. Scrapy allows you to easily swap out one feature for another using pip. A tool like this also enables a more iterative approach to building your project with minimal downtime.

I find that scrapy is better when I need to build a crawler for a small set of URLs and I don't need anything other than basic scraping. But I have more experience with Selenium which allows me to do so much more with each page I visit. If you want to have a dynamic crawling system for your site where many URLs can be visited and each of these would yield new information (scraping), then I would go for scrapping and use Scrapy with its extension hooks. Or if you want an advanced tool to manage your scrapping sessions (eg by session variables), then I would use Selenium (because it's easy to use).

Should I use Selenium for web scraping?

In my project I will have to scrape several different websites that use a similar format of html.

The only difference in the sites will be their name and content.

My plan is to start by scraping a single site, save the data in a database (and if possible convert it into an excel sheet). Then make a loop that scrapes the other websites.

Will it better to use Selenium for this or not? What are its downsides and advantages? If you're scraping a single site, I'd think it would better to use Selenium. The benefits of Selenium are quite obvious, and in your situation, it's going to save you a lot of work.

But if you're planning on doing multiple scrappings, you might want to use Selenium. The biggest problem is going to be writing all of the code for scraping the sites, then writing the code to handle the data you've scraped.

With Selenium, you can write all of that code once and then just iterate through the pages in your loop.