How to check for a crawler?

How do I check when Google crawled my site?

I am trying to find the timestamp when google crawled my site.

The reason I am trying to find this is I was doing a link-building campaign on another site, and I was surprised when I saw that I had gotten a large amount of links from a website which I had never heard of before. I did some googling and found out that it was probably due to Google crawling my site.

I have found one way to find out but its not very accurate. This is the way I found it out.

I use Google Webmaster Tools to check my crawl rate. In the statistics tab I see the number of pages crawled. I found out that Google has not crawled my site for 6 days. This is obviously not accurate.

Then I found out that Google Search Console has an option called Crawls. It lists the dates when Google crawled my site. However, it does not give me an exact timestamp.

I also tried to use the Google Crawlable Pages tool. Unfortunately, I could not find out when Google crawled my site.

Do you know of any other way to find out when Google crawled my site? It's hard to say when Google crawled your site because it depends on your server time. If you are using a Linux server, you can check your server time using "date".

KrishnaAug 4 '14 at 19:36. @Krishna I am on a Windows Server. I have checked the date/time and the time seems to be correct. I also tried to turn on the timestamps in Google Webmaster tools but I was not able to find out the date/time when Google crawled my site.

Sagar ChabraAug 4 '14 at 19:40. Ok. If the date/time on your server is correct, and you can't get a timestamp in Google Webmaster Tools, then I think you have two options: 1) If you're running your own servers, you could just wait a few days and see if it comes back. 2) If you don't have the money or resources to wait, you could use Google Analytics (or another 3rd party analytics tool) to track when the most traffic came from your site. Google Analytics is free to use, and will tell you where traffic came from.

How to test Google crawling?

I am testing Google crawling of my website.

My website is built in ASP.NET MVC 3.

I have created a test domain (www.testdomain.com) and I have uploaded a simple static html file in my root. I have updated the Google Webmaster Tools settings to have this URL as a page.

After updating the HTML file I have tried to submit a post request to the URL. And I am not getting any kind of response.

I know that Google should be crawling the website but it isn't. What else can I do? In your webmaster tools, there is a "Submit site test" button. Click that, fill out the form, and hit submit.

If it doesn't work, you need to go into the actual pages of the site and make sure you have the canonical tag.

How do I trigger Google crawl?

I am really having a hard time figuring out how to make Google crawl my website.

Is there a way to trigger it? For instance, say the site has the URL xxxxx.com/about-us and when I do that the links in the page should be crawled by Google (as per Matt Cutts' video) or even a link in the source code that has rel="canonical" will trigger Google to crawl the page. But I don't know what to look for. Can anyone point me to right direction?

The basic issue is, for SEO purposes, Google's spider will follow links that point to other pages on your site. In essence, you are asking Google to crawl a single page with a link on it, and for Google to follow those links and index the related content on the destination page. The good news is that there are ways to automate this process.

First, if you want Google to index your root page, you need to place a small link on the root page that contains the following text and href: . I realize that this isn't exactly the same as a link that actually points to the root of the domain, but it will effectively make the page the "primary" landing page for the rest of the site. If you already have the link, then just place the text within the link tag (without the "rel" attribute). Then, once you have a page indexed, you can use Google's webmaster tools to set up other pages to be linked from the root page. If you create the new page with links pointing to the root page, then the root page will be indexed as well.