Many of you reading this post may remember “Dark Google” was born almost two years ago when Google started its new policy to withhold information that were previously available for free to publishers. This referral data was highly useful because it allowed publishers to find out the exact terms used by people while searching on Google. Prior to October 2011, Google used a standard mechanism to provide this data.
Breaking of this useful system by Google came as a shock to many. According to Google, this was done to ensure better online privacy for the web crawlers. Many people use the search engines to find out information that are rather sensitive in nature. Google started withholding this information to minimize any chance of leakage or eavesdropping. Though it is a good idea in general, this new policy by Google was not free from flaws. That’s because leakage of sensitive search data was still a possibility. This could happen in three different ways.
• It was still possible for the publishers to access search data because certain search terms were provided by Google’s Webmaster Central service.
• It was still possible to access search data that Google Instant autocomplete suggests.
• Certain search terms were still transmitted to the advertisers through the open web by Google.
The last point in particular has been a topic of much debate as Google proactively took measures to ensure access to referral information for the advertisers in the same traditional way. Many well known SEO experts pointed out that Google put a price on search data privacy by implementing this new system. Some though that Google wanted to protect search information privacy without inviting trouble by irritating the advertisers. Google, however, had their own defense against all such accusations.
After the launch of this new system, it was only possible to access data corresponding to the top 2,000 queries for a given day and only for the previous 90 days. The search terms keep changing on a daily basis, and that’s why Google expanded the number of terms in the early part of 2012. Unfortunately, due to this 90 day window, the historical data for many websites has been lost forever. There was simply no way out if someone wanted to compare his/her top terms today to what it was one or two years ago.
However, things dramatically started changing for good towards the end of August 2013 when Google announced their “Paid and Organic” report for the AdWords account holders. This makes it possible now to store any of your search terms that were previous being withheld by Google. All you need to do is to sign up for your AdWords account and link this account to your Webmaster Central account. This will constantly extract search data without using any Python script.
The good news is that this facility is available even without being a paid AdWords account holder. However, some experts have already suggested that Google is making an indirect attempt to encourage people to buy AdWords. Moreover, publishers have been forced to access their non-paid listing information via Google’s ad interface.