How to Identify and Remove Bot Traffic in Google Analytics

Let's talk about bot traffic in Google Analytics. Bot traffic is usually associated with spam. There is also good bot traffic. Sometimes we intentionally send bots to our websites for testing, diagnostics, or SEO.

(Read full article - goo.gl/1CGJ69)

Sign up for free Google Analytics Course - goo.gl/9GnLyk

But we still don't want bot traffic in our analytics data.

Bots are not real users. Also, heavy bot traffic can skew our data and pollute our analytics.

So the issue is... How do you keep the bots out of your Google Analytics reports?


Google doesn't always block the bot
I love Google Analytics... But sometimes they miss the mark

Now, I've been pretty outspoken about how Google handles spam traffic. And for a long time, my take was that Google wasn't doing much to keep the spam traffic out of our analytics reports.



Google was almost like Fredo in the Godfather II when it came to defending our data against spam - drastically underachieving!



And the community noticed. Like many Google Analytics users, I started checking out other analytics products. Maybe the grass was greener somewhere else?



When users started moving away form Google Analytics, Google responded. And since then things have gotten better. There's been a noticeable reduction of spam traffic in our analytics reports.

But, spam is just one type of bot traffic that can pollute our analytics data. Many kinds of bots hit our websites. Sometimes, we're even the ones sending bots to our site.

How do we keep the Bots out?
Defending our data against bot traffic is a bit like playing whack-a-mole.

You have to identify the bad traffic and respond.

Let's look at how we can identify bot traffic. And let's walk through some strategies for blocking this traffic from our analytics reports.

Analytics Course Student Question
One of our Analytics Course students recently noticed a big issue with bot traffic in his reports. And he wants to know how to respond to the problem.

Keith Asks:

We recently sent an email blast out using an email list that we rented. The company is well known with a good reputation as far as we can tell. We received clicks to the site but no action once on the website and looking through GA I see that 82% of the clicks can be located to the United States but is (not set) for state, city & metro. I have not seen such a high number of (not set), in fact we on another site we have we had 300 not set in the last 250,000 sessions.

The email was targeted at Seattle and the Network/Service Provider dimension shows Microsoft Corporation. Looking at a number of other sites we manage, we don’t have anywhere near the % of location (not set), even for Network/Service Provider: Microsoft Corporation.

So what do you think. Is this a case of a narrow target to Seattle for a number of individuals at Microsoft that happen to block geo location or perhaps something fishy with bot clicks from the email list provider to show results? (no accusations, just haven’t seen numbers like this

Here's Keith problem. His company sent out a targeted email campaign. The campaign resulted in a bunch hits to their site that they tracked in Google Analytics. But the geo location for the majority of this traffic is "(not set)."  And all the not set traffic is coming from one ISP organization - Microsoft Corporation.

Kieth wants to know if this bot traffic?

And I would say most likely it is bot traffic.
Here' why:

Keith's getting a disproportionate amount of location (not set) data in his reports. Also, the traffic doesn't sound like it exhibits human behavior. There are not bunch people at Microsoft sitting around clicking email links, and then immediately bouncing off his site once they click through.

So how do we keep this traffic out of our reports?

The Google black-box solution
The easiest way to keep bot traffic out of your Analytics reports is to use Google's automatic filter. To set up this filter, go to your view settings and check the box that says "Exclude all hits from known bots and spiders."


I've used Google's auto filter on almost every account I've worked on...

And I'd say 60% of the time, it works all the time.

When this filter doesn't work, two things are happening.

False positives and false negatives.

Google's black-box doesn't always exclude the traffic you want to exclude. And it doesn't always include the traffic you want to include.

So, you have to run tests.  You should try to be proactive with testing your filters.

You can't remove bot traffic from your data after the fact. So, if you can, you want to put filters in place before this traffic becomes pervasive in your reports.


(Read full article - goo.gl/1CGJ69)

Sign up for free Google Analytics Course - goo.gl/9GnLyk