What is the difference between BeautifulSoup and web scraping?

What is the difference between BeautifulSoup and web scraping?

BeautifulSoup is used for extracting text from HTML pages. Scrapy is used to scrape data from pages. The main difference between them is that BeautifulSoup makes more assumptions about the input HTML and the output will be the same when doing so. Scrapy on the other hand takes all the possible conditions into account (eg page not available, or page available but with no data, etc.).

I've been using the Scrapy framework to scrape the data for years and now I'm working on a project that requires extracting data out of very different sites such as: And so on. What I find it particularly useful for is having a single framework that can deal with very different pages with the flexibility to deal with data from non-HTML pages such as PDFs and images etc. It's also nice because it works over the wire, which is much nicer if you have lots of machines to run your requests on rather than having to go through my web server for each request.

Having said that though, there's nothing that says a programmer can't do some of the work themselves - you don't necessarily need a full-on framework for that.

Related Answers

What is the eligibility criteria for admission to Web scraping courses?

What resources do I need to learn web scraping? Are there specific skills that...

Why can't beautifulsoup see some HTML elements?

There is a lot of questions here, about both selenium and beautifuls...

What states have the most Web Scraping jobs?

Sure, if you are good enough to make it, but it is also not the future of lar...