Can you get banned for web scraping?

Do hackers use web scraping?

So I've been doing a lot of reading around, and every time I read about how hackers use web scraping or phishing or social engineering or anything that allows them to get what they want from someone, it seems like its a common practice, so does anyone know if its a real thing or just a myth?

Does "Web Scraping" Exist? Yes, for all intents and purposes, there is no such thing as web scraping. If you want to call it web harvesting (not necessarily using the term "web scraping" to refer to it), then maybe that's an accurate description of it, but in reality it is not a malicious act by any means. For example, the software I built a few years ago was built to crawl certain websites on a scheduled interval and download all the content to a database. It didn't do anything malicious, it just downloaded the information it needed to know and store. It wasn't malicious, it was only trying to get what it needed. It wasn't web scraping, it was web harvesting.

Is it Common to Hackers? No, but it's certainly not unheard of. What happens when someone hacks my site? The most likely scenario is that the hacker will start uploading user information and/or credit card numbers. This is not malicious in the slightest, and is only because people are not using good security practices (like password protection, etc). A hacker could also find some way to access your database and steal some data or even delete it all together, but again, that's not considered malicious behavior by anyone.

I have been reading a lot about how hackers use web scraping and phishing and social engineering, is that common? Again, it's definitely not unheard of, but it's more like a "thing" in the industry. How can I avoid being hacked? Do not use any type of unauthenticated connection with the website. If the website requires authentication, make sure you're using something other than "login" and "password" to do the authentication. Do not send any of your personal information over unencrypted channels.

If the site doesn't require login, at least make sure it's a "logout" function in order to ensure your account doesn't stay logged in. Never send personal information over the internet.

How can I scrape data from a website for free?

My team of programmers and I are trying to pull out data from different websites in order to build a large dataset for our customers who need it.

We decided to pull the data from www.nii.ox.ac.uk which is the National Institute of Informatics, a public institution in the United Kingdom. They offer two data mining challenges that are free for anyone to participate in and we would like to build an app which can complete these challenges. However, since we are trying to gather this data for free, we can't use any paid methods (such as scraping or using cookies), so, how else can I find these data which contain information about universities (including name, postal code, course title, programme details, etc) and student courses (course/class-level, title, programme, etc. Please let me know if there is a way to find these data otherwise, and thank you for your help in advance!

You could access their website through wget for example, so you can grab all the data that you want. You can look here for the documentation of wget and here for the way to set up your proxy.

If you are not familiar with wget, try cURL (here).

Can you get banned for web scraping?

First of all let me apologise in advance for this post.

The intention is not to troll, I promise! Please do have a read though and if you comment on either of my posts, it will give me the incentive to write more on this.

If you follow me on Facebook, you will be familiar with my occasional use of an app called TidyCam (or just Tidy) which allows people to share live video footage on a daily basis. This is basically a clever way of broadcasting your life at the present moment - it's like a selfie app that goes viral. Some people have also used this as a great time-waster during long train journeys or just a fun gimmick to have around when you need to pass the time.

However, there are times when you want to know what's happening in a certain room while the cam is pointed towards it. I am a big fan of web scraping and am often in situations where I have a large group to feed information to my script from a website - usually some database I don't own the rights to.

One such situation was when we launched the new web shop for the new season of The Real Hustle. If you haven't watched the show yet, I'll try and stay brief as possible but the short story is, we had a really popular season 1 and season 2, so we took a new direction for season 3 where we set a challenge for contestants to get to work. We had an open team shop in which contestants competed over 3 episodes to see who could sell the best. It's not to everyone's taste, but for some people it's a really fun reality show. However, we were always wary of what might happen on the final stage, where contestants were allowed access to many secret product samples. Some of these products would probably be very good and others would be absolute stinker gold. This was really exciting from an IT standpoint and we wanted to scrape the entire list of finalists before the end of the series.

After we'd scraped a few weeks worth of products from finalist web pages, we had a fair bit of data. As you can imagine we had hundreds of products up for sale, each with up to 6 different images representing various stages of the product being sold.