What is the difference between BeautifulSoup and web scraping?

BeautifulSoup is used for extracting text from HTML pages. Scrapy is used to scrape data from pages. The main difference between them is that BeautifulSoup makes more assumptions about the input HTML and the output will be the same when doing so. Scrapy on the other hand takes all the possible conditions into account (eg page not available, or page available but with no data, etc.).

I've been using the Scrapy framework to scrape the data for years and now I'm working on a project that requires extracting data out of very different sites such as: And so on. What I find it particularly useful for is having a single framework that can deal with very different pages with the flexibility to deal with data from non-HTML pages such as PDFs and images etc. It's also nice because it works over the wire, which is much nicer if you have lots of machines to run your requests on rather than having to go through my web server for each request.

Having said that though, there's nothing that says a programmer can't do some of the work themselves - you don't necessarily need a full-on framework for that.