How do I inspect a website for scraping?

I have heard that if I want to protect my website from scrapers and other malicious users, I should perform a "security audit".

However, I do not know what this really entails. What is the "low-hanging fruit" security issues? What is the best way to check if my website is easy to scrape? How do I find out if my website is easy to scrape? Can anyone point me to a link or an example of what a security audit looks like?

I would just like to clarify, as there seems to be some confusion on this. The "low-hanging fruit" security issues are probably as follows: Make sure your server runs a correct httpd config, you should look at the file /etc/httpd/conf.d/security.conf to see how you can improve it. In general, any web server which runs PHP or Python code will have a lot of security.

Make sure your server isn't open to remote exploits (like via SQL injection). You can read up on most common ones here: This should help, as should other exploits which you can look up. Also make sure your servers aren't using a default password - always use strong passwords, and use different usernames for different things.

Make sure your server isn't open to local exploits (file inclusion vulnerabilities). Again, that is covered by the previous link, but also make sure you're only displaying static files (images, javascript, etc) and not allowing users to upload their own files. You could possibly restrict the directory structure of uploaded files to only allow images, javascripts and other commonly used files.

Make sure your server isn't vulnerable to XSS vulnerabilities (you can read more about this here). Make sure you're never echoing user-supplied content, never outputting unencoded/unescaped/raw user data in your HTML and Javascript.

Make sure your server isn't vulnerable to SQL injection vulnerabilities.

Which sites allow web scraping?

My company would like to extract some content from a web service.

This is a big no as it has a reputation cost on us of many tens of pounds. I will be accessing the site via a mobile phone emulator called PhoneGap, does this rule out all sites for scraping? I've heard it is possible to extract the data using XMLHttpRequest with the right parameters but that it's probably not safe and can also affect the end user. I can't see any sites I've tried myself. Does anyone know of any sites which allow scraping, as we need something which doesn't rely on javascript? We don't really have a choice on the API level, its embedded in a game for example. We don't want to hit any sensitive information but we are having to make assumptions that we may need to do. Any advice on the best way to achieve this would be appreciated!

Can websites detect web scraping?

What are some approaches a website can take to identify that a program is web scraping their site and what are some ways it can be done?

I've got a bit of code for a website which is fairly trivial, and want to get a feel for how a website might go about identifying scraping. Thanks. The best way is to have a look at the HTTP headers. Websites that think they've been scraped will send an HTTP header with "Authorization: Basic " for example. A site that's been scraped won't.

Other than that, you might add some sort of unique identifier to your pages (URL fragment), for example by creating a hash of all the parameters in the query string, that would be sent with each request. A couple of tips based on experience: If a site is scraping from multiple sources, use an IP address whitelist. You can use Javascript or other methods to detect that you're on a scraper, but many sites will try to fake it.

Can you get banned for web scraping?

A friend asked me this question yesterday, and I don't know.

So I thought I'd check to see what our community thinks. Here's my best take on the question:

If you write a script to read through the Wikipedia entry for a person and extract their full text, can you get banned for that? That's the big one. I don't think so, but I'm not certain, and I want your opinion. Here are a few things that I think about the question. How much of Wikipedia are we talking about here? Here's a question that I asked several years ago: How does Wikipedia determine if a user is banned? Some months later, someone contacted me with answer. I wrote up the answer on this blog: And while I thought it was interesting, I never fully implemented the recommendation from that post, because I didn't think it would be useful. (I guess I should have realized that there's a pretty good chance that people would want to copy that technique to build a bot to search Wikipedia.)

Anyway, as of today, the only two banned users on Wikipedia who aren't also banned from the Wikipedias at other organizations are those two people in Germany. And I don't think it's very hard to find a reason that they're banned.

And yes, those people are banned from the entire network of Wikipedias. They're not allowed to edit any pages. They're even blocked from viewing pages. There are some people who don't get this kind of block, and I understand that. But this doesn't happen very often, and I don't think it's very hard to come up with a good reason for a ban that's limited to a single network.

There is one other person who has a Wikipedia ban (and isn't at the two German sites), but that's not my question.