Detecting spider and bot traffic through reports
For products:
Webtrends Analytics 9.2x
Webtrends Analytics 8.x
Last modified: 3/1/2011
Situation:
You suspect recent increases in your traffic may be the result of automated processes such as site testing, bots, or site indexing.
Solution:
There are two reports that can help file and determine the impact of automated visits to a web site:
1. Hits Trends
In the Complete View template, navigate to Site Performance > Technical Statistics > Hits Trend. Select the daily view from the calendar for a day which should display typical visitor metrics. For many web sites, normal site traffic should roughly appear as a bell curve, with traffic peaking, possibly several times, and consistently dropping off at times when the majority of visitors would not be expected. If automated traffic is frequenting the site it may cause a plateau which does not decay throughout the day. As machines do not have to care about the time of day this kind of traffic can start high and stay high, producing a flat table effect in the Hits Trend report.
2. Top Visitors
Also in the Complete View template, under Marketing > Visitors, the Top Visitors report displays the highest-ranked visitors by the number of visits made, as well as display the number of hits generated. Automated processes create a large number of hits without ending their visit so one visitor may have few visits but thousands of hits.
Note: The Top Visitor report sorts by Visits in descending order as its default. Click the "Hits" column header to sort visitors by hits in descending order.
The Top Visitors report is subject to table limits so a visitor with few visits may not necessarily show up in the report. To work around this limitation, copy the profile and set the "From the Following Date:" value to the date at which traffic is suspected to be coming from automated processes.
Further information:
While Webtrends has a Browsers filter that can remove known bots and spiders from reporting, it doesn't cover all possible forms of automated traffic. Internal testing and denial-of-service attacks will not be filtered out by the "all spiders and robots" filter. The built-in "All spiders and robots" filter is based on the contents of the browsers.ini file, which contains a list of publicly known bots and spiders.
To remove this unwanted data from reports, find and filter out entries for the IP address of the unwanted visitor prior to analysis.
One indicator of automated traffic is that these clients rarely accept cookies. In the cases where they are capable of accepting cookies the client's IP address will appear as the first part of the cookie value displayed in the Top Visitors report.
Related Articles
How can we include specific domain traffic in our reports?
Products: Webtrends On Demand Webtrends Analytics 8.0x Webtrends Analytics 8.1 Webtrends Analytics 8.5 Webtrends Analytics 8.7 Webtrends Analytics 9.2 Introduction: To include only data from specific domains, hit filters can be created and applied to ...
How do I filter internal traffic based on IP address?
For products: Webtrends Analytics 9.2a Webtrends Analytics 8.x Webtrends Enterprise 7.x Webtrends Professional 7.x Webtrends Small Business 7.x Last modified: 8/5/2010 Situation: How do I filter internal traffic based on IP address? Solution: You can ...
Encoding errors in Asian Search Phrase reports
For products: Webtrends On Demand Webtrends Analytics 9.2x Webtrends Analytics 8.x Last modified: 3/1/2011 Situation: A profile using a data source from Asian web sites shows pound (#) symbols, squares or random characters for search keywords and ...
Why are units appear as all zeroes in Product reports?
For products: Webtrends On Demand Webtrends Analytics 9.2x Webtrends Analytics 8.x Last modified: 1/1/2011 Situation: Although the site is tagged with the minimum commerce tags (WT.pn_sku, WT.tx_u, and WT.tx_s) units are reported as zero in all ...
How to customize page titles in WebTrends reports?
Solution: Title Parameter The Title parameter, WT.ti, supports a single page title per page. WT.ti WT.ti=Title The HTML title of the associated web content. If this parameter is found in parameter list, the value is used in the reports. When present, ...