Playing in AVG Free Traffic...

07-06/08 Update: Upon attempting to do a follow-up post seeing if the AVG Free v8 “SP1” build made a difference in LinkScanner Traffic (as AVG reports they have accomplished), I located some serious problems with the data reported. Turns out I had captured all the data from my network monitoring, but I had not selected the actual summary session totals.  So I have gone-back and re-parsed the data below.

While the totals have changed, the overall conclusions did not, and seem to be even more overwhelming in terms of traffic the initial LinkScanner version in AVG Free 8 (b101) generated.

--Claus

So, in the process of doing some last-minute editing and fact-checking for my guest post over at Houston's chon.com (TechBlog: Guest post: Claus Valca's little AVG 8 Free 'problem') I decided I had to independently confirm if a custom removal of the Search-Shield component from AVG Free did in fact remove the LinkScanner traffic.

First: A Quick LinkScanner GSD Post Review

As reported in this post - AVG disguises fake traffic as IE6 | The Register – AVG Technologies continues to tweak its beloved/despised LinkScanner component.

To refresh, this “feature” pre-checks links as you browse to them in your web-browser for malware and other web-ilk. Great idea in theory.  Seeing as IE, Firefox 3.0, and Opera 9.5 already have a similar feature embedded in them to varying degrees, makes perfect sense for AVG to load-down your pc with even more web-security protection.  I can think of several good images but let’s keep the discussion family-friendly.

As AVG Free has hereto-with been a very popular anti-virus solution and loaded on bazillons of pc’s, and seeing how many of these users have already upgraded to version 8.0 and not done the fancy-pantsy CLI “stripped” version install, the Interwebs are now full of AVG’s additional LinkScanner traffic.

The first versions of LinkScanner registered their “pre-visit” click-through event scans as the unique user agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)."

Clever folks like the guy over at OSBlues figured out quickly how to filter out that cosmic-noise from web-master logs.  Goodness knows it was giving them fits up to that point.

Now it appears that AVG has jiggered LinkScanner to now also report clicks under the following additional user agents:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

According to OSBlues, this actually is the same agent profile used by LinkScanner products before they were bought out by AVG Technologies (Grisoft).

That Register article’s Comments on ‘AVG disguises fake traffic as IE6’ are filled with quite a few good perspectives.

OSBlues has also confirmed that AVG says that LinkScanner does at least NOT click Google AdWords. Not clear that other such pay-per-click providers are also spared this noise.

Adam over at OSBlues offers his perspective which provides great insight into the headache this is causing those who depend on web-stats as well as the detective work he did to uncover this trend, and raises a "bandwidth leaching" concern as well.

AVG Destroys Web Analytics « OSBlues

In fact, LinkScanner analyses results from search engines (not just Google) and is browser independent.  This may sound like a good idea from a security point of view, however, from a webmaster/website owner point of view, this is not good at all.

If your site appears well in the search engines, as everyone strives to do, your website is or is going to be hugely affected by this.  Essentially this means, that everytime your site appears in a users results, regardless of whether they click on it, your website logfiles and thefore your statistics will show that person as a real visitor coming to your site.  Now, because the IP address is the users IP address, we can’t filter on that, at first look it would appear we can filter on this useragent, unfortunately I spotted another one

Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1)

This one however, is even worst.  This time it’s a legitimate user agent which means you can’t filter it out or rewrite it to another page on your site without the risk of blocking or harming real visitors.  The first user agent is different, due to lack of a space (or plus) between the last semi-colon and the 1813, it doesn’t follow the standard pattern used by Microsoft.

So, we get to crux of the problem, AVG has destroyed web analytics for people who use a logfile analysis tool.  Not only have they done this, they are also wasting our bandwidth and our disk space on servers!

Second: LinkScanner Results on a "real" System.

Now back to the show.

As we have just seen, network traffic is increased at the web-site level due to AVG LinkScanner usage.

In addition, many, many AVG users are fussing about the degraded browsing performance on their PC systems where LinkScanner (Search-Shield / Safe-Surf) components are active. Lots of AVG users.

So Dwight and I confirmed that you do not have to use a command-line installation method to disable the Search-Shield/Safe-Surf ...whatever AVG likes to call it) component.  Question that remained was, did this method effectively remove the LinkScanner activity in AVG Free version 8?

I fired up a "real" XP SP3 image in Virtual PC 2007.  Again, by "real" I mean it is a copy of Dad's old XP system I previously had converted to a VPC image. All the junk that a "normal" user would have is on this test-bed.

I downloaded and unpacked Nir Sofer's freeware packet-sniffer SmartSniff inside that virtual system. There were a host of other packet-sniffing tools I could have used for more detail but I was confident this would give me some quick data that I was looking for. It also was light and fast, perfect for my VPC environment.

I then proceeded to run four packet-capture sessions under two different states of an AVG Free v 8.0 build 101 installation; a "Full" install (with Search-Shield) and a "Custom" install (without Search-Shield).

With Search-Shield Installed...

For the first test I ran Internet Explorer 7 and browsed to Google, then did three searches: TechBlog, Grand Stream Dreams, and Starbucks.

I could see the Safe Search icons loading and being added to the Google results page. All were fine and passed the safe-site test (whew!).

According to SmartSniff, I captured a total of 131 TCP/IP conversations resulting in a total of 173 packets and total size of 14,036 Bytes.

According to SmartSniff, I captured a total of 131 TCP/IP conversations resulting in a total of 5,391 packets and total size of  3,615,873 Bytes.

For second test I closed out IE. Reopened it, browsed to Google, ran a search for Grand Stream Dreams, then clicked the link to fully load my main blog page.

According to SmartSniff, I captured a total of 44 TCP/IP conversations resulting in a total of 31 packets and total size of 25,925 Bytes.

According to SmartSniff, I captured a total of 44 TCP/IP conversations resulting in a total of 1,152 packets and total size of 723,115 Bytes.

Without Search-Shield Installed...

For the third test I reinstalled AVG but this time removed the Search-Shield component in the custom setup wizard.

I again ran Internet Explorer 7 and browsed to Google, then did three searches: TechBlog, Grand Stream Dreams, and Starbucks.

This time I could see no Search Shield icons loading and being added to the Google results page.

According to SmartSniff, I captured a total of 37 TCP/IP conversations resulting in a total of 4 packets and total size of 924 Bytes.

According to SmartSniff, I captured a total of 37 TCP/IP conversations resulting in a total of 699 packets and total size of 229,908 Bytes.

For second test I closed out IE. Reopened it, and again browsed to Google, ran a search for Grand Stream Dreams, then clicked the link to fully load my main blog page.

According to SmartSniff, I captured a total of 11 TCP/IP conversations resulting in a total of 11 packets and total size of 3,200 Bytes.

According to SmartSniff, I captured a total of 11 TCP/IP conversations resulting in a total of 182 packets and total size of 68,054 Bytes.

Amazing!  I was stunned to see it with my own eyes in this very simple test.

That Secret AVG LinkScanner User Agent ...

In addition, I could clearly pick out in the AVG Search Shield enabled captures the following user agent, as being reported in various sources earlier noted in this post:

• User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

My regular system browser (and non-SafeSearch loads) requests were the following:

• User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 1.925)

Again, for side-by side comparisons with/without Search Shield on each set of links:

Set #1 (Google, --> Searches only on following words: TechBlog, Grand Stream Dreams, Starbucks)

a. Total of 131 TCP/IP conversations were captured resulting in a total of 173 packets and total size of 14,036 Bytes.
c. Total of 37 TCP/IP conversations were captured resulting in a total of 4 packets and total size of 924 Bytes.

Difference of 97 extra TCP/IP conversations, 169 packets, and total size of 13,112 Bytes transmitted just by using the LinkScanner Safe Search component.

a. Total of 131 TCP/IP conversations were captured resulting in a total of 5,391 packets and total size of 3,615,873 Bytes.
c. Total of 37 TCP/IP conversations were captured resulting in a total of 699 packets and total size of 229,908 Bytes.

Difference of 97 extra TCP/IP conversations, 4,692 packets, and total size of 3,385,965 Bytes transmitted just by using the LinkScanner Safe Search component.

Set #2 (Google, --> search and click-through to Grand Steam Dreams)

b. Total of 44 TCP/IP conversations were captured resulting in a total of 31 packets and total size of 25,925 Bytes.
d. Total of 11 TCP/IP conversations were captured resulting in a total of 11 packets and total size of 3,200 Bytes.

That's a difference of 33 extra TCP/IP conversations, 20 packets, and total size of 22,275 Bytes transmitted just by using the LinkScanner Safe Search component to load a single blog main-page.

b. Total of 44 TCP/IP conversations were captured resulting in a total of 1,152 packets and total size of 784,731 Bytes.
d. Total of 11 TCP/IP conversations were captured resulting in a total of 182 packets and total size of 68,054 Bytes.

That's a difference of 33 extra TCP/IP conversations, 970 packets, and total size of 716,677 Bytes transmitted just by using the LinkScanner Safe Search component to load a single blog main-page.

It is simply amazing.  And this was just a very quick browsing exercise.  Those totals will accrue over a long web-surfing exercise.

You just don't really appreciate the LinkScanner traffic impact on the local system until you see it for yourself.

Granted, AVG home-pc users (and others) who have beefy new systems with lots of RAM and high CPU MHz numbers, along with a broadband network connection to the Inter-webs autobahn lanes might not even notice this as an issue. They are probably still tooling along in their S-class workstations, oblivious to this bad behavior.

However the poor AVG Free users who are clueless, and are stuck driving their air-cooled, four-banger "peoples-wagon" PC with low RAM and CPU MHz's and puttering even slower now on the the dial-up access roads probably are miserable and jealous and confused; maybe even worse.

Yeah, I know it's not going to break any banks or probably overload the Inter-tubes, but you can at least get a simple appreciation on just how much network traffic impact might be going on if just a moderate percentage AVG's claimed 70 million AVG users world-wide install AVG Version 8 and enable the LinkScanner technology in it's current form. It certainly is compounding issues with bandwidth usage already on the rise with streaming media files, torrents, and spam.

What next? Will major ISP's seek to add AVG users to the growing list of throttling targets?

OMG!  What would this do for folks whose ISP's are hard at work lobbying for data-download caps for their subscribers?  Comcast Considering 250GB Cap, Overage Fees - dslreports.com

Yikes!

Wouldn't be a shame if a user's AVG Free product pushed them over the limit not due to downloading torrent files, ISO's, or other stuff, but simply for running their security product with LinkScanner enabled?

Am I falling into a falsely alarmist view? Maybe.

Could it be a problem? Certainly.

But it gets even worse...

Third: AVG's LinkScanner Security Technology; A Tool for 3vil?

In working on my guest post I stumbled on a very interesting website.

AVG Watch.org

Some fellow Texans did some great research and found a neat (don't try this at home kiddies) method to use LinkScanner to use AVG's LinkScanner to bomb a website with a simple DoS (denial of service) attack.

Read the post. It is quite good and has some great technical notes and details.

Oh Bother!

Wonder how AVG is going to close this Pandora's Box up.

Ranger(s) Needed?

Now which is more valuable? Keeping Aunt Lilly and Uncle Bob's pc safe from malicious click-to web links? Or denying the 3vil a new and free security tool to lightly-nuke a website?

So in the words of our poor AVG friend michaelhd, is LinkScanner still a "...valuable security tool to protect users while they surf"?  Really?

I suppose the jury's still out, but the court of public opinion seems to be reaching a clear and loud verdict in advance of the final decision outside in the Texas summer heat.

Better call in a Texas Ranger to help guard the defendant.

Hmmm. 70 million users of AVG and the web-masters to boot?

Might want to break the rules and send two Texas Rangers just to be sure the jury gets its opportunity to render a decision first.

Now, where did I put my Stetson.....

--Claus