Estimating the Real Click Fraud Rate

The controversy surrounding click fraud comes up every year, but it reached a fever pitch during December’s Search Engine Strategies conference in Chicago when participants voiced concerns over experiencing fraudulent click rates ranging from 20 to 40 percent, threatening the entire paid search industry.

At the time, Google’s Business Product Manager for Trust and Safety Shuman Ghosemajumder tried to calm advertisers’ fears explaining that Google was currently “…examining ways to make its fraud-fighting efforts more transparent without revealing crucial information that might help swindlers elude detection.” Ghosemajumder did, however, express concerns over revealing too much information, fearful it would give away algorithm secrets to competitors.

Paid Search Revenues Continue to Rise

While the major search providers have always insisted the clíck fraud rate is a gross overestimation, a 2005 Outsell survey found that clíck fraud was a $1.3 billion problem for publishers. At the time, many advertiser respondents (27 percent) said they planned to cut back and/or eliminate paid search campaigns in 2006.

Outsell respondents may have intended to cut down on paid search, but they certainly didn’t follow through. SEMPO’s year-end search marketing report showed that North American advertisers spent $8 billion on paid placement programs in 2006, amounting to 86 percent of 2006’s total search spend ($9.4 billion). Seventy-one percent of SEMPO respondents said they used paid search campaigns, illustrating that there were not many defectors.

Despite advertisers’ insistent claims that the search engines don’t do enough to eliminate clíck fraud, paid search revenues continue to fill the coffers of Google, Yahoo, Microsoft and many second and third tier search engines. Additionally, there is a huge gap in the professed prevalence of clíck fraud between the search providers and the advertisers and clíck fraud advocates.

Google Click Fraud Estimates

The rate of clíck fraud changes depending on whose numbers you believe. Clíck fraud detection agencies put the clíck fraud rate hovering around 14 percent, while others believe at least 20 percent of all clicks are fraudulent.

Late last month, Google issued a statement on the Inside AdWords blog that insisted invalid clicks consistently remain under 10, typically in the single-digits, and that virtually all malicious activity is found by Google’s filter. Ghosemajumder claimed the percentage of clicks found by advertiser-initiated investigations account for just .02 percent of clicks. All other accounts, he said, are grossly overestimated.

Alchemist Media President Jessie Stricchiola takes issue with Google’s assertion that it refunds advertisers promptly for fraudulent clicks, stating that “Google has been the most stubborn and the least willing to cooperate with advertisers”.

Google Click Fraud Filters

In February, Google outlined the three-layer filtration process it uses to combat and eliminate clíck fraud. They described the system which uses both proactive and reactive filters as follows:

1. Proactive Filters: Automated algorithms analyze and filter out invalid clicks in real-time without billing advertisers for these false clicks. This accounts for the vast majority of invalid click detection.

2. Proactive Offline Analysis: Post billing, Google uses automated and manual analysis to identify fraudulent clicks that somehow made it through the first layer of filtration. Special attention is paid to clicks occurring on the AdSense network. This is done pro-actively and without any involvement from advertisers. When false clicks are found, advertisers’ accounts are immediately credited via Clíck Quality Adjustments.

3. Reactive Investigations: Investigations take place when an advertiser approaches Google concerned about suspicious activity on their account. Each complaint is investigated, though Google says refunds are relatively rare. Google claims that the vast majority of fraudulent clicks, more than 99 percent, are found and thrown out within the first two stages of filtration. The third stage only includes the .02 percent of clicks where advertisers are affected by undetected cases of clíck fraud.Click Fraud Detection Agency Estimates

In April 2006, The Click Fraud Index reported an industry-wide average clíck fraud rate of 13.7 percent. The clíck fraud rate was broken down as follows:

• Tier 1 search providers — 12.1 percent
• Tier 2 search providers — 21.3 percent
• Tier 3 search providers — 29.8 percent

Some of the newer click fraud prevention firms like Click Assurance and ClickLab offër algorithm-based programs to limit bad clicks. These programs estimate the statistical likelihood of a clíck being fraudulent based on behavioral variables and IP address.

Gap in Prevalence of Click Fraud

As noted above, Google admits to a less than 10 percent click fraud rate, while advertisers and clíck fraud detection agencies believe it is more like 14 to 20 percent. Ghosemajumder explained this gap saying that many advertisers and clíck fraud detection agencies are looking at the wrong signals, mistakenly classifying valid clicks as fraudulent. Additionally, he believes many advertisers request refunds for clicks already thrown out during the first two layers of the filtration system.

For example, misclassification might occur when counting reloads of an advertiser’s landing page. Say the customer clicks through to the landing page, views a product page, and then hits the back button, returning to the same landing page. Without proper tagging, that one clíck and five page re-loads could be misclassified as 6 clicks from the same visitor. Google argues that there are hundreds of different signals that must be monitored to detect clíck fraud, signals that are a closely guarded company secrët and known only to the Google clíck quality team.

A Solution for Click Fraud

Like other experts, we believe the only solution to clíck fraud is for independent auditors to evaluate the system using accurate data provided by the search engines and advertisers themselves. It is the only way to get a neutral calculation — the current clíck fraud detection agencies may not be entirely neutral, and certainly the search providers are not neutral. We need an independent agency that has no incentive to íncrease or decrease the clíck fraud rate. One solution could be to use a technology firm like Fair Isaac, which is currently conducting clíck fraud research for SEMPO.

One thing is certain, until advertisers are willing to provide campaign info, and the search engines are willing to share clíck fraud data, we’ll nevër know the actual prevalence of clíck fraud or how much advertisers are losing as a result.

-----
by Nick Guastella, SEM Analyst and PPC expert at Bruce Clay, Inc.