What is the best web crawler?

Is web crawler free?

I have no problem using the google web-crawler, but I also want to crawl other websites.

Is the google web-crawler free? Can I install it myself on my web server? If yes, what are the steps? Does it only crawl one URL and not the whole website? There are two separate questions here: (1) whether you can install and use Google's web-crawler on your own web server; and (2) whether you can run the Google's crawler in non-commercial mode -- ie crawl sites only to find links to resources that you will in turn crawl. You need to have a valid license for that type of "commercial use". Google doesn't sell these licenses, as far as I know. You can buy them directly from Microsoft or another distributor.

I haven't crawled any pages using the Google Web Crawl, but I can confirm that you can use the Search API to obtain a list of URLs that can be crawled. Google will limit the requests you make to their servers, depending on the account type you are signed up for, so if you know your approximate limits for crawling (or you can calculate that yourself), you can estimate how many URLs you could reasonably get.

As far as running the Google Web Crawl for your own usage, I think you would need to contact Google's Developer Support staff to get a license. They are the only ones that can grant you the right to do this, and they probably won't unless you have a good idea of how much crawling you will do. But I think it would be possible. As far as non-commercial crawling, it seems that only Google really cares about non-commercial crawling (they even give out the web-crawling feature free). I couldn't find anything on the Google Web Crawler FAQ page that said non-commercial crawling wasn't allowed. The fact that the program says that it has certain limitations when you run it (see here) suggests to me that it is intended to be used non-commercially. But Google itself doesn't want you crawling the web for your own use (as far as I know), so the only option is to talk to them.