Sunday, June 29, 2008

Playing in AVG Free Traffic...

07-06/08 Update: Upon attempting to do a follow-up post seeing if the AVG Free v8 “SP1” build made a difference in LinkScanner Traffic (as AVG reports they have accomplished), I located some serious problems with the data reported. Turns out I had captured all the data from my network monitoring, but I had not selected the actual summary session totals.  So I have gone-back and re-parsed the data below.

While the totals have changed, the overall conclusions did not, and seem to be even more overwhelming in terms of traffic the initial LinkScanner version in AVG Free 8 (b101) generated.

--Claus

So, in the process of doing some last-minute editing and fact-checking for my guest post over at Houston's chon.com (TechBlog: Guest post: Claus Valca's little AVG 8 Free 'problem') I decided I had to independently confirm if a custom removal of the Search-Shield component from AVG Free did in fact remove the LinkScanner traffic.

First: A Quick LinkScanner GSD Post Review

As reported in this post - AVG disguises fake traffic as IE6 | The Register – AVG Technologies continues to tweak its beloved/despised LinkScanner component.

To refresh, this “feature” pre-checks links as you browse to them in your web-browser for malware and other web-ilk. Great idea in theory.  Seeing as IE, Firefox 3.0, and Opera 9.5 already have a similar feature embedded in them to varying degrees, makes perfect sense for AVG to load-down your pc with even more web-security protection.  I can think of several good images but let’s keep the discussion family-friendly.

As AVG Free has hereto-with been a very popular anti-virus solution and loaded on bazillons of pc’s, and seeing how many of these users have already upgraded to version 8.0 and not done the fancy-pantsy CLI “stripped” version install, the Interwebs are now full of AVG’s additional LinkScanner traffic.

The first versions of LinkScanner registered their “pre-visit” click-through event scans as the unique user agent "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)."

Clever folks like the guy over at OSBlues figured out quickly how to filter out that cosmic-noise from web-master logs.  Goodness knows it was giving them fits up to that point.

Now it appears that AVG has jiggered LinkScanner to now also report clicks under the following additional user agents:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

According to OSBlues, this actually is the same agent profile used by LinkScanner products before they were bought out by AVG Technologies (Grisoft).

That Register article’s Comments on ‘AVG disguises fake traffic as IE6’ are filled with quite a few good perspectives.

OSBlues has also confirmed that AVG says that LinkScanner does at least NOT click Google AdWords. Not clear that other such pay-per-click providers are also spared this noise.

Adam over at OSBlues offers his perspective which provides great insight into the headache this is causing those who depend on web-stats as well as the detective work he did to uncover this trend, and raises a "bandwidth leaching" concern as well.

AVG Destroys Web Analytics « OSBlues

In fact, LinkScanner analyses results from search engines (not just Google) and is browser independent.  This may sound like a good idea from a security point of view, however, from a webmaster/website owner point of view, this is not good at all.

If your site appears well in the search engines, as everyone strives to do, your website is or is going to be hugely affected by this.  Essentially this means, that everytime your site appears in a users results, regardless of whether they click on it, your website logfiles and thefore your statistics will show that person as a real visitor coming to your site.  Now, because the IP address is the users IP address, we can’t filter on that, at first look it would appear we can filter on this useragent, unfortunately I spotted another one

Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1)

This one however, is even worst.  This time it’s a legitimate user agent which means you can’t filter it out or rewrite it to another page on your site without the risk of blocking or harming real visitors.  The first user agent is different, due to lack of a space (or plus) between the last semi-colon and the 1813, it doesn’t follow the standard pattern used by Microsoft.

So, we get to crux of the problem, AVG has destroyed web analytics for people who use a logfile analysis tool.  Not only have they done this, they are also wasting our bandwidth and our disk space on servers!

Second: LinkScanner Results on a "real" System.

Now back to the show.

As we have just seen, network traffic is increased at the web-site level due to AVG LinkScanner usage.

In addition, many, many AVG users are fussing about the degraded browsing performance on their PC systems where LinkScanner (Search-Shield / Safe-Surf) components are active. Lots of AVG users.

So Dwight and I confirmed that you do not have to use a command-line installation method to disable the Search-Shield/Safe-Surf ...whatever AVG likes to call it) component.  Question that remained was, did this method effectively remove the LinkScanner activity in AVG Free version 8?

I fired up a "real" XP SP3 image in Virtual PC 2007.  Again, by "real" I mean it is a copy of Dad's old XP system I previously had converted to a VPC image. All the junk that a "normal" user would have is on this test-bed.

I downloaded and unpacked Nir Sofer's freeware packet-sniffer SmartSniff inside that virtual system. There were a host of other packet-sniffing tools I could have used for more detail but I was confident this would give me some quick data that I was looking for. It also was light and fast, perfect for my VPC environment.

I then proceeded to run four packet-capture sessions under two different states of an AVG Free v 8.0 build 101 installation; a "Full" install (with Search-Shield) and a "Custom" install (without Search-Shield).

With Search-Shield Installed...

For the first test I ran Internet Explorer 7 and browsed to Google, then did three searches: TechBlog, Grand Stream Dreams, and Starbucks.

I could see the Safe Search icons loading and being added to the Google results page. All were fine and passed the safe-site test (whew!).

According to SmartSniff, I captured a total of 131 TCP/IP conversations resulting in a total of 173 packets and total size of 14,036 Bytes.

According to SmartSniff, I captured a total of 131 TCP/IP conversations resulting in a total of 5,391 packets and total size of  3,615,873 Bytes.

For second test I closed out IE. Reopened it, browsed to Google, ran a search for Grand Stream Dreams, then clicked the link to fully load my main blog page.

According to SmartSniff, I captured a total of 44 TCP/IP conversations resulting in a total of 31 packets and total size of 25,925 Bytes.

According to SmartSniff, I captured a total of 44 TCP/IP conversations resulting in a total of 1,152 packets and total size of 723,115 Bytes.

Without Search-Shield Installed...

For the third test I reinstalled AVG but this time removed the Search-Shield component in the custom setup wizard.

I again ran Internet Explorer 7 and browsed to Google, then did three searches: TechBlog, Grand Stream Dreams, and Starbucks.

This time I could see no Search Shield icons loading and being added to the Google results page.

According to SmartSniff, I captured a total of 37 TCP/IP conversations resulting in a total of 4 packets and total size of 924 Bytes.

According to SmartSniff, I captured a total of 37 TCP/IP conversations resulting in a total of 699 packets and total size of 229,908 Bytes.

For second test I closed out IE. Reopened it, and again browsed to Google, ran a search for Grand Stream Dreams, then clicked the link to fully load my main blog page.

According to SmartSniff, I captured a total of 11 TCP/IP conversations resulting in a total of 11 packets and total size of 3,200 Bytes.

According to SmartSniff, I captured a total of 11 TCP/IP conversations resulting in a total of 182 packets and total size of 68,054 Bytes.

Amazing!  I was stunned to see it with my own eyes in this very simple test.

That Secret AVG LinkScanner User Agent ...

In addition, I could clearly pick out in the AVG Search Shield enabled captures the following user agent, as being reported in various sources earlier noted in this post:

  • User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

My regular system browser (and non-SafeSearch loads) requests were the following:

  • User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 1.925)

Again, for side-by side comparisons with/without Search Shield on each set of links:

Set #1 (Google, --> Searches only on following words: TechBlog, Grand Stream Dreams, Starbucks)

a. Total of 131 TCP/IP conversations were captured resulting in a total of 173 packets and total size of 14,036 Bytes.
c. Total of 37 TCP/IP conversations were captured resulting in a total of 4 packets and total size of 924 Bytes.

Difference of 97 extra TCP/IP conversations, 169 packets, and total size of 13,112 Bytes transmitted just by using the LinkScanner Safe Search component.

a. Total of 131 TCP/IP conversations were captured resulting in a total of 5,391 packets and total size of 3,615,873 Bytes.
c. Total of 37 TCP/IP conversations were captured resulting in a total of 699 packets and total size of 229,908 Bytes.

Difference of 97 extra TCP/IP conversations, 4,692 packets, and total size of 3,385,965 Bytes transmitted just by using the LinkScanner Safe Search component.

Set #2 (Google, --> search and click-through to Grand Steam Dreams)

b. Total of 44 TCP/IP conversations were captured resulting in a total of 31 packets and total size of 25,925 Bytes.
d. Total of 11 TCP/IP conversations were captured resulting in a total of 11 packets and total size of 3,200 Bytes.

That's a difference of 33 extra TCP/IP conversations, 20 packets, and total size of 22,275 Bytes transmitted just by using the LinkScanner Safe Search component to load a single blog main-page.

b. Total of 44 TCP/IP conversations were captured resulting in a total of 1,152 packets and total size of 784,731 Bytes.
d. Total of 11 TCP/IP conversations were captured resulting in a total of 182 packets and total size of 68,054 Bytes.

That's a difference of 33 extra TCP/IP conversations, 970 packets, and total size of 716,677 Bytes transmitted just by using the LinkScanner Safe Search component to load a single blog main-page.

It is simply amazing.  And this was just a very quick browsing exercise.  Those totals will accrue over a long web-surfing exercise.

You just don't really appreciate the LinkScanner traffic impact on the local system until you see it for yourself.

Granted, AVG home-pc users (and others) who have beefy new systems with lots of RAM and high CPU MHz numbers, along with a broadband network connection to the Inter-webs autobahn lanes might not even notice this as an issue. They are probably still tooling along in their S-class workstations, oblivious to this bad behavior.

However the poor AVG Free users who are clueless, and are stuck driving their air-cooled, four-banger "peoples-wagon" PC with low RAM and CPU MHz's and puttering even slower now on the the dial-up access roads probably are miserable and jealous and confused; maybe even worse.

Yeah, I know it's not going to break any banks or probably overload the Inter-tubes, but you can at least get a simple appreciation on just how much network traffic impact might be going on if just a moderate percentage AVG's claimed 70 million AVG users world-wide install AVG Version 8 and enable the LinkScanner technology in it's current form. It certainly is compounding issues with bandwidth usage already on the rise with streaming media files, torrents, and spam.

What next? Will major ISP's seek to add AVG users to the growing list of throttling targets?

OMG!  What would this do for folks whose ISP's are hard at work lobbying for data-download caps for their subscribers?  Comcast Considering 250GB Cap, Overage Fees - dslreports.com

Yikes!

Wouldn't be a shame if a user's AVG Free product pushed them over the limit not due to downloading torrent files, ISO's, or other stuff, but simply for running their security product with LinkScanner enabled?

Am I falling into a falsely alarmist view? Maybe.

Could it be a problem? Certainly.

But it gets even worse...

Third: AVG's LinkScanner Security Technology; A Tool for 3vil?

In working on my guest post I stumbled on a very interesting website.

AVG Watch.org

Some fellow Texans did some great research and found a neat (don't try this at home kiddies) method to use LinkScanner to use AVG's LinkScanner to bomb a website with a simple DoS (denial of service) attack.

Read the post. It is quite good and has some great technical notes and details.

Oh Bother!

Wonder how AVG is going to close this Pandora's Box up.

Ranger(s) Needed?

Now which is more valuable? Keeping Aunt Lilly and Uncle Bob's pc safe from malicious click-to web links? Or denying the 3vil a new and free security tool to lightly-nuke a website?

So in the words of our poor AVG friend michaelhd, is LinkScanner still a "...valuable security tool to protect users while they surf"?  Really?

I suppose the jury's still out, but the court of public opinion seems to be reaching a clear and loud verdict in advance of the final decision outside in the Texas summer heat.

Better call in a Texas Ranger to help guard the defendant.

Hmmm. 70 million users of AVG and the web-masters to boot?

Might want to break the rules and send two Texas Rangers just to be sure the jury gets its opportunity to render a decision first.

Now, where did I put my Stetson.....

--Claus

10 comments:

Ren said...

Hello again, Claus. Ok I just read this blog post of yours http://grandstreamdreams.blogspot.com/2008/06/remove-linkscanner-from-avg-simply.html and now I finished reading this one, and now I have a question.

I am going to quote you here, from this blog post: "Question that remained was, did this method effectively remove the LinkScanner activity in AVG Free version 8?"
And I am guessing, that you are referring to the simple method of just using the *Custom Installation* to prevent the installation of LinkScanner(SafeSearch Module), and the AVG Toolbar, and thereby removing them from the Free AVG 8.0

So this simple *Custom Installation* method, did it work? I am still a little confused here. I mean, did it prevent the total install of those LinkScanner and AVG Toolbar components? Because, if that's the case..then I'd rather install the AVG 8.0 in this custom installation way..instead of the complicated command line method. I mean, when I decide to install it. I am still going to wait out a week or 2 before performing the real install/update to 8.0, because of all the developments going on. Oh, I so wish AVG didn't do all this, and screwed it up so badly. I checked out your guest blog at TechChron too (another excellent, detail oriented post..with lots of facts; kudos to you) and was checking your suggestions for other Free AntiVirus Solutions..and right now, I am quite in the right mood to ditch the whole AVG thing and go for Comodo or AntiVir or something. I've heard good things about Comodo. And Comodo has free Firewall too, which is supposedly real good. This screwup by AVG is getting a little frustrating right now. Wish they didn't antagonize their loyal, trusted users so badly. Once again, amazing work Claus..I'll be around quite frequently, actively checking your updates.

Claus Valca said...

@ Ren - Ooohh. You caught me. I didn't actually answer that question explicitly. My bad.

I need to update that post to make that point clear. thanks for pointing it out!

Yes. As far as I can effectively tell by monitoring the network traffic, using the "custom install method" and removing the Search-Shield component does remove all traces of the LinkScanner behavior on the system.

Also, let me clarify, while the AVG Security Toolbar feature does work to help manage the Search-Shield (and other) feature of AVG, it doesn't contribute in-of-itself to the LinkScanner activity. I dumped it because I just don't like toolbars in my browsers. Icky.

I use and have been pretty pleased with Comodo's firewall product on both XP and Vista systems. It also has built-in HIP's protections (Defense+). I have that turned off myself as I use the freeThreatFire product to cover that base. But I really do like Comodo Firewall and it's blended protection model. As to their A/V product I really can't say. I've heard mixed reviews--mostly centering around their GUI interface being difficult to navigate. I've installed in on a test system just fine, but haven't played enough with it to come to a conclusion for myself yet.

The "Tools -> Advanced..." options of AVG give a lot more control over the AVG Free 8 program than one would believe. However, the average user just isn't going to take the time to tweak and adjust all the settings from the defaults to get the perfect balance between performance and protection custom to their own system.

That's too bad. It didn't have to be that way....

Thanks again for the support and encouragement. It's usually fun to share and I feel like I am able to give back a little bit to a community of IT users who have taught me so very much.

Jimi said...

I believe avg8 "SP1", or build 135 has been released, yesterday july 2nd. I haven't checked my pc running v8 yet so I can't confirm.
An entry with info on this would be much appreciated by readers, I believe.

Claus Valca said...

@ Jimi - Yes. AVG Free 8 "SP1" was released as build 135.

See this comment.

I did have time to verify that my AVG Free 8 build did auto-update to this version yesterday. The XP SP3 system this occured on went smoothly. First it applied a signatures file update, then it applied the SP1 update (135 build). It took about four-five minutes on my system. A single reboot was required.

I didn't have time to test for performance gains, etc. I could tell some GUI changes with the system-tray icon, and extra elements in the tools->advanced menu window.

I do plan on making another post this weekend specifically on this.

-Cheers!

dan said...

Hey nice to find your site. I have gone through your archives from the start of May to the present see what you have to say about AVG. You come up as the first rank on google for "is avg 7.5 still being updated" for your article on updating avg manually.

I do have a few questions as a home user and wonder if you have the time to answer or direct me to good resources for answers.

I am unclear as to the nature of the malware threat that one is evidently vulnerable to when browsing to certain web sites, enough so that avg felt it made sense to include the much maligned link scanner in version 8.0.? ( I seem to be receiving updates and plan to stick with 7.5 awhile longer.) How does one protect them selfs for these threats without using a linkscanner type tool?

I am also interested in finding a definitive discussion on the best strategies for securing ones home computer. I know that there is ton of info out there but that seems to be the problem there are just to many hits when searching. I would love a recommendation.

Thank you and I plan on coming back in the future now that I have found your site.

Yours, Dan.

Ian said...

Hi
Found your site from a link on the AVG forum, as I've been tracking the AVG8 saga as many others have. Some great stuff on here!

It's really good to see some proper analysis of the network traffic created by the Link Scanner component, rather than just have users say they 'have a feeling' that things are running slower etc.

Now, we know the effect of omitting Search Shield at installation time. How about the effect of installing the component, but disabling it (and now ignoring the fault condition). Does this have the same effect as not installing, ie also stops any extra traffic? It would be nice to install Search Shield, but only switch it on as required, rather than having to go back thru the installation process.

One other factor stopping me upgrading is that since 8.0.135, my update manager is trying to connect approx every 15 mins - as seen in the event history log. I thought the free edition only updated once per day (previous versions did). This can't be good for traffic either, and can cause an error state if the LAN cable is unplugged, as it sometimes is on my laptop. Not seen any info on this one yet.

Keep up the good work!

Regards, Ian.

Claus Valca said...

@ Ian - Thanks for the kind words!

My observations were very basic, but I hope showed that user's were actually had a basis for fact on their complaints about poor system performance caused by LinkScanner on some system configurations.

I haven't explored the scenario that you propose about installing the component but disabling it until needed. The new AVG Free v8 SP1 build should allow you to do that. So that seems a good suggestion for those seeking a middle-ground.

As to the update frequency, you can set a very high-degree of update control.

In version 8, open the AVG User Interface. Then on the top menu-bar go to "Tools" and "Advanced settings..."

Pick and expand the "Schedules".

Click on "Virus database update schedule" and verify the settings are as you want. I have mine to check once a day only.

Click on "Program update schedule" and do the same thing there.

If the update manager is indeed kicking off more frequently despite those settings...then I'm not sure. I checked my logs and it's only updating once-per-day (unless I request manual updates).

--Cheers!

Claus Valca said...

@ Dan - I'm not sure if it is a good thing or not to become first-ranked on Google searches for information on another company's program. ;)

You ask a number of very good questions.

Let me give you advice based only from what I do on my systems; I believe in a layered defense.

I have a hardware based firewall/router to start things off. Properly configured this should prevent intrusion into your home network. Mine sits right-behind my broadband cable modem.

Behind that are my computers.

I work very hard to keep all my systems updated and patched (Windows Updates). I also recommend using The Secunia Software Inspector (online) or their Personal (PSI) tool installed locally to look for vulnerable applications.

Next I currently run the free Comodo Firewall Software set up for both inbound and outbound "leak" protection. (I do have the Defense+ element turned off.)

Then, AVG Anti-Virus Free Edition (without LinkScanner) to provide real-time and scan protection for virus/trojan/malware threats.

I then follow that layer up with ThreatFire AntiVirus - Behavioral Virus and Spyware Protection which is freeware software that provides heuristic and behavior-based malware protection (for threats that don't have a signature yet).

Finally I recommend using a next-generation build web-browser.

The Opera 9.5 browser, Internet Explorer 8 Beta, and Firefox web browser 3.0 all now provide anti-phishing/anti-threat site protection to varying degrees. While not perfect, they do operate in a much more efficient manner to provide web-surfing protection from malicious sites.

I prefer Firefox coupled with the NoScript - JavaScript/Java/Flash blocker Add-on. This keeps most all malicious software activity from running via the Firefox browser, but allows you to enable Java/Flash on sites you trust (banking/commerce/etc.).

For extra checking on the safety of web-links, check out this post I did a while back: Pre-Scanning of URL Links for Safe Web Surfing.

For "embedded" link-checking on search-result websites, these software options might provide a measure more of protection without the overhead of Linkscanner which checks ALL links on every page you visit:

LinkScanner Lite

TrendProtect

SiteAdvisor

Scandoo

I don't use these, preferring to only use the "on-demand" tools and references mentioned in that same post.

Hope this helps....

--Claus

Ian said...

Claus, many thanks for your thoughts. I'll try some tests with Link Scanner installed but disabled when I get time. I'm focussed on the update manager problem at the moment.

As I'm running the Free version, the scheduling options are limited to one per day, as confirmed by the parameters displayed. The Every/Periodically option is greyed out, and the drop-down for the Specified Time Interval option only shows Every Day. This would be fine for me if it was working properly!

I had seen an approx 15 min frequency in the event history log, but there's even more weird behaviour with the times shown in the update manager window. For example, yesterday, after a successful update at 18:18, he showed the next scheduled update as 18:28. At 18:25, the time displayed for the next scheduled update moved to 18:33. At 18:33, another update completed ok, and the next scheduled time was displayed as 18:43. At 18:40, this moved to 18:49, and at 18:49 the next update occurred. And so on.

The event history log continued to show the correct time (15 min frequency) for actual updates, but I can't explain the intermediate time changes on the update manager screen. All very odd.

As I say, I'm running the Free version, but I think I've read that the full version's minimum schedule period is 15 mins. If so, it may be a case of screwed up parameters defaulting to this value.

The 8.0.135 I'm currently running was updated from 8.0.100, so I think I'll do a clean install of the latest downloaded exe and see if that makes any difference.

As things stand, I think there's too much flaky stuff going on to allow the version onto my main desktop machine just yet.

Ian.

Claus Valca said...

@ Ian - I think it is indeed a bug of sorts.

In an earlier build of AVG Free 8 I was looking at when I responded, I could select from incremental/hourly or less update checks or the once-per-day.

In version build 138 I can't do that and am locked in (like you) to the once a day update schedule.

Curious. But makes sense as AVG wants us to buy the full version so we can make those changes.

I'm also seeing those staggered multiple updates in my event history log.

Remember there are two types of updates, signature updates and program updates. both can't run at the same time so you will likely see one first, then the other. The event description doesn't differentiate between them.

I am seeing a pattern of three updates per day, each about 10-15 minutes apart. Probably just the way the updates go on and are staggered.

Besides the standard "run at specific time" update check, there are also checkboxes to run update if task was missed, and run update again as soon as internet connection is available. I wonder if these may also pay something in having it repeat the update process.

Finally, if I recall, AVG first downloads and updates the update catalog file(s), then based on that "update", downloads and applies the signature updates, then the program updates. That also might explain the series of three updates I am seeing.

Just not sure.

Great comments and information from you Ian. Thanks for taking the time to share!

--Claus