Which tool is best for web scraping?

Is Scrapy better than Selenium?

I have worked on scraping projects using both Scrapy and Selenium Webdriver, the former in a Python application where I wanted all the crawling functions to be within the code. Now I am not necessarily doing my own web scraping, I am just getting the raw data via the API, and building my own web applications using it, and then displaying it on my Django site, using Django-tables2.

One of the problems I ran into was I had no real way of easily tracking the status of each item I was scraping, or getting more details. It was also harder for me to use Python's built in data structures to build the items when I wanted them. I did find Selenium would get things done faster, and better.

So I have been looking at the newer "pipelines" system that Scrapy is built around (see link). This seems like it would solve most of my issues, I do need to build my own "pipelines.py" file for each module to process some data from.

So how do you guys track the scrapes as they are completed, so you can easily access which ones have been scraped, and which ones have failed, in order to use these as seed values for the Django data tables module? Is Python's multiprocessing thread pool adequate, with 4 of them going at a time? (My guess is no, but I really don't know if they have been improved since I last coded.) Will it be possible to create a queue and have each of those 4 threads grab an item off the queue and pass it to a different method? I haven't coded much Java/Javascript, though I was using NodeJS quite successfully.

It doesn't appear to work for me, I can not see the status of the task at all. As per the documentation, my first job here is to figure out what command-line option I can use so I can see the status of the scraping tasks: and I figured I would need to pass -t parameter to see if I could do that.

Is this the correct place to ask about this? I'm using Python 2.7 (not python 3.x).

Is Selenium better than Beautiful Soup?

I am fairly new to programming but have been working with Python for a few months. A friend of mine showed me Selenium and I tried it out. To be honest, it was awful! He said that BeautifulSoup was way better. So the question is, which one is better? And why? Please help! Thank you :)

"Better" is always going to be subjective and this article will probably only end up in the comments. However, here are some things that might help: Selenium is a very high-level approach to automating web browser interactions. It makes it super easy to write tests that drive your web browser directly, it's perfect if you want to make sure a page works exactly as it should, not to mention it's super fast.

Beautiful Soup is built upon html parsing and it makes it easy to get a handle on the actual data. The performance and ease of writing code is the primary reason.

Both are wonderful tools, but they do different things very well. If you're looking to start web scraping or are simply looking to create a test suite to ensure your site works, they both have their place.

As for which is better, that's a little more subjective still.

Which tool is best for web scraping?

I have been researching web scraping tools for a project of mine. There are so many tools that it is almost overwhelming. From what I have seen so far, these are my top choices:

Scrapy. Requests. BeautifulSoup. Mechanize. I would really like to know if there is anyone who can offer me advice in selecting ? The answer is based on your stated criteria. "from what I have seen so far, these are my top choices". I'm going to break this down and list the pros and cons, just like the questions/answers site does. Scrapy is probably the best choice if you have a lot of traffic and you want your scraper to be able to deal with it. Scrapy also lets you write tests with the scraped data, a feature that Mechanize doesn't. This will let you write better scrapers with less effort (and since it's Python, it should be much easier to follow than the C# of Mechanize).

BeautifulSoup is a really great general purpose HTML/XML parser. It's easy to use but if you want to do a lot of customization then you might have some issues. There's a python library for it that I like: lxml.

Requests is a good option if you are looking to scrape sites with mixed content (HTML, JS, etc) However, the downside to Requests is it uses a non-standard HTTP library and sometimes has trouble dealing with sites that refuse to be scraped because of security reasons. Mechanize is a bit old and is getting a bit out of date but it's still pretty useful. It's especially useful if you need to scrape a bunch of sites without creating one gigantic program.

Personally I use these different tools for different things. Here's what I'd do: If your looking for content, use BeautifulSoup. Use Requests when the site has mixed content that you need to get through. Use Mechanize when you need to scrape a bunch of sites. I don't think I've ever needed to scrape a site with both mixed and pure content.

I'm sure there are other tools as well, but this is the one I've used the most.