Is selenium better than Beautiful Soup?

Is Scrapy better than Beautiful Soup?

It certainly is very customizable and it can handle some of the most complex sites I've worked with, but I'm always trying to improve my coding skills.

I prefer to do things in shorter ways because: the longer, the more maintenance it requires. I feel that the shorter and more clear my code, the more likely it is for people to be able to understand and use it. Beautiful Soup seemed like the logical way to go (at least at the time) but I'm thinking that Scrapy would be easier for what I'm trying to do as opposed to Beautiful Soup. What are the advantages and disadvantages of using Scrapy versus Beautiful Soup? Scrapy definitely isn't any harder to learn than BeautifulSoup. Scrapy has some cool features (it's designed to crawl websites from top to bottom), such as the request pipeline, middlewares, et cetera, but some of the stuff it does might not be what you're looking for. Eg, I don't know why anyone would want to use Scrapy instead of a regular web server.

Scrapy is certainly flexible enough to make any number of changes to whatever website you want to scrape. But it will take a bit more work than just using BeautifulSoup. And, of course, it takes more time to set up than it would using a standard framework/library.

The choice between Scrapy and BeautifulSoup is largely a trade-off of whether you think learning how to make those changes is a better option than learning how to build scrapers.

Should I use Scrapy or selenium?

I'm developing a scrapping project, it's just about 200 pages of sites, each page scraped with selenium, but there is a big problem that I do not know how to solve.

For some pages, the html is pretty different (ie some page not only have div, but the whole html content changed), and then I don't know how to get the right information, should I use Scrapy to get the good info? Or I need to check all the text, and find out whether it's div or not, and then set the correct attribute? Or I can combine Scrapy and Selenium to do this task? You can use both scrapy and selenium for crawling. I am assuming that you have already decided whether you are going to use Scrapy or Selenium for your requirement. Let's say we decide to go with Selenium. So, if you want to make selenium aware of the content in the HTML tag, You need to first locate that tag. That is you need to first locate the text to which you need to get the HTML.

I would use selenium for this type of tasks. It's pretty straightforward as compared to scraping. You can refer this link for more information.

Is selenium better than Beautiful Soup?

I am not sure why I am asking this question because I know my code sucks but it is also very specific.

I am new to web scraping so I am just wondering what to use? Beautiful Soup or Selenium? What are the advantages and disadvantages of each? And will I be wasting my time if I choose one over the other? Thanks in advance. Lucky. Selenium (if properly used) is much better than BS. But it takes a lot more time to learn.

The advantage of Selenium is that you will get more features like, "How to click on this button". And the main disadvantage is, the initial investment of learning it. So if your budget is tight, then don't bother with Selenium. If your budget is not so tight, try out Selenium.