Saturday, February 09, 2008

Firefox 3 Security Blocker: Going In Deep

I've been using Mozilla's "Minefield" nightlies as my daily browser at home now for the better part of two weeks. These are the iterations that will eventually become the final Firefox 3 release.

Major issues with the Places bookmark handling, while not fully fixed, have been resolved enough to allow me to be comfortable with regular usage.

Only last night I ran into an unexpected "problem" and it is really frustrating me that I can't get more information!

Scanwith Saga

One of the websites I like to check in at periodically is This simple site provides links to freeware and $-ware software that is geared towards system-scanning. Process scanners, anti-virus scanners, malware-scanners, etc. While I don't download the files from this site (rather I go to the developer's site) it provides a great all-in-one stopping place for the latest and greatest in this class of tools.

So Friday morning I was doing a quick link-check for new utility leads before work and the site loaded great.

Friday night, after work, I fired up Firefox 3 (Nightly) and was hit in the face with this:


Apparently, someone thinks that ScanWith is an "attack site."


But why? And is it true or not? Have I put my own system at risk in my past visits?

Oh bother!

Claus has questions and he's out to take down names!

First Things First: Assessing the Threat

The first thing that I noticed was the frightful language and lack of options on the displayed page (Mozilla feature sample link here).

Notice there is no link here to allow you to click-through, despite your better judgement. That's normally a good thing, especially if your pc is used in a shared-user environment.

There are only two links: the "request a review" link which takes you to

Here website owners can register and plead their case to the courts.

However, despite all my attempts and searches through the "Clearinghouse" I could not locate any information on why this site got black-listed. Nothing. Nada. Zilch.

The only other link available on the site is the panic-soothing "Get me out of here!" button. That returns you to your browser's homepage.

There is no way to tell Mozilla that you wish to pass-through to the site anyway. You just can't do it.

If a site is blocked, it's blocked.

Just for kicks, I cross-checked ScanWith's URL against McAfee's SiteAdvisor for the site. It comes back clean and gets positive marks from this trusted anti-virus company. Long searches on Google turned up no additional information reported by other users.

I also left-clicked the icon to the left of the address-bar. In Firefox 3 this gets you a little drop-down mini-window that gives you some site-security information. If you want more, click the "More Information" button.

I did. While helpful, I suspect that the site's lack of supplying owner information and the fact that the connection isn't encrypted isn't what caused the alarm. This is a nice feature, but could be much more informative.

Second: The Workaround - It's all or nothin' baby!

Because there are no options at all here on the warning page to "whitelist" the site manually, there is only one option to get to the site(s) that are being blocked.

On the menu bar, go to Tools > Options... > and click on the Security icon at the top.

Now uncheck the "Tell me if the site I'm visiting is a suspected attack site."

Apply the changes and re-click.

You're in!

But now you've turned off the global-protection setting for this feature. It's not entirely clear to me, but I am pretty sure that you have also just turned off the anti-phishing feature also built into Firefox 3. So now the doors are open!

(Correction: as Asa Dotzler kindly points out in his comment, turning off the "attack warning" feature indeed does NOT turn off the anti-phishing protection in Firefox 3. In Firefox 3 there are two different control options as seen in the screenshots below. Firefox 2 only has an anti-phishing protection option so it wouldn't alert on an "attack designated" website, regardless. Thanks Asa!)

A Mozilla Dr. Jekyll and Mr. Hyde?

Actually it isn't really a "new" feature. Firefox 2.0.0.x also contains the functionality.

In Firefox 2.0.0.x menu-bar, go to Tools > Options and then click on the Security icon at the top.

Notice the options are very similar. You have the option to "Tell me if the site I'm visiting is a suspected forgery" and either "Check by using a downloaded list of suspected sites" or "Check by asking [Google] about each site I visit."

Notice below that I do have this feature enabled in Firefox on my system.


As I do in Firefox 3.0.0.x (Minefield nightly)


However, when I load in Firefox 2, it loads just fine with no interception or warning messages. When I load it with Firefox 3.0, it gets blocked.

The difference? Different download lists!

(Edit: Actually, this is true, but as Asa pointed out I'm comparing apples and oranges here. FF2's list is geared soley for checking for phishing sites. FF3's list (as I break into futher down) differentiates between "attack" sites and "phishing" sites. So this easily explains the differences in website blocking behavior I'm observing between the two, even though the lists are coming from Google.)

Because I am using the one from (presumably) Mozilla in 2.0 (and not downloading Google's) I'm cool. Because Firefox 3.0 uses Google's black-lists, "No soup for you!"

(Even more curiously, when I did toggle the "Check by asking Google..." option in Firefox 2, the site remained unblocked. Maybe I'm not waiting long-enough for the Google-list to be updated after toggling it on? I really don't know.)

More blockage screenshots: Firefox 3: Suspected Attack Site Screenshots « RJO Blog

Mozilla and Google and StopBadware...Sitting in a tree, k-i-s-s-i-n-g....

Wikipedia has some information on the relationship between Mozilla and Google for their anti-phishing/attack filtering:

The release of the anti-phishing protection in Firefox 2 especially raised controversy. By default, anti-phishing protection is enabled, based on a list that is updated by downloads to the user's computer about twice an hour from Google's server. The user cannot change the data provider within the GUI, and is not informed who the default data provider is. The browser also sends Google's cookie with each request for update. An additional, explicitly opt-in security feature has been added to recent builds by the Mozilla Foundation. This anti-phishing feature provides live protection by checking each visited URL with Google.

And the relationship between Google and StopBadware is pretty clear as well, Google's Webmaster Central blog: Better badware notifications for webmasters.

In the fight against badware, protecting Google users by showing warnings before they visit dangerous sites is only a small piece of the puzzle. It's even more important to help webmasters protect their own users, and we've been working on this with A few months ago we took the first step and integrated malware notifications into webmaster tools. I'm pleased to announce that we are now including more detailed information in these notifications, and are also sending them to webmasters via email.

The post comments were quite interesting to read and seemed to reflect some issues with getting "un-blacklisted" and the impact this event has on site-ranking.

It appears that the general process for getting tagged and blocked is this:

  1. Google uses an undisclosed and proprietary method (magic?) to locate badware on websites.

  2. Google passes this information to StopBadware.

  3. Google adds a warning under the search-results links for sites it tags.

  4. Mozilla's Firefox 3 downloads (periodically) updated lists from Google's servers and these are folded into the user's browser for protection.

Here is a current Google search result (as of this post's date) showing Google's identification on


You can still click through to their, if you are interested and not sufficiently frightened.

In theory, the website owner finds the badware on their page (embedded in the page-code or hosted for download), cleans up their webpage/website, petitions Google and StopBadware, and is released from the black-lists. More on this in a bit.

Pause for Review

So, at this stage of the game I had figured out that depending on which version of Firefox I was using, I was going to see different results. That the blacklist was provided by Google, and that any attempts by me (the public) to assess for myself what the threat was and make an informed decision to continue or not had been removed from my control in (this version) of Firefox 3 (Minefield).

So now I was OCD locked-on to this issue.

I had to know, what is going on under the Mozilla hood and could I manually "un-blacklist" a site?

Get out the Tools!

Out of curiosity, I decided to see if, perhaps, I could locate the file that was being checked against to see if a site is blacklisted.

I fired up Microsoft Sysinternal's Process Monitor and set a filter to exclude all results except the process firefox.exe.

I swapped over to Firefox and clicked the bookmark for

I then toggled back to Process Monitor and culled through the results. I pretty quickly located a file called "urlclassifier3.sqlite".

Only trouble was, it wasn't viewable in my Windows Explorer file list at the location claimed.

So back to Process Monitor. I right-clicked one of the lines with it listed and selected "jump-to."

Bam, a new Windows Explorer window opened up and I found the file buried in a super-hidden "OfflineCache" folder in my Firefox/Minefield profile folder.


Now. What goodies/baddies are in here?

The file-extension told me that it was a SQL "lite" database file.

I'm down with Access databases, but SQL isn't something I'm knowledgeable about. Some quick web-searching located what I needed: About SQLite.

Turns out it is basically an embedded SQL database engine which contains all the tables, indices, triggers and views..all self-contained in a single file. I can really see the benefits of using this from Mozilla's standpoint.

Unfortunately (for me and other prying novices) working with it demands knowledge and interaction with it from a command-line level. I'm pretty impatient. Anybody got GUI?

Yep: SQLite Database Browser. This freeware tool allows one to "to create, design and edit database files compatible with SQLite. It is meant to be used for users and developers that want to create databases, edit and search data using a familiar spreadsheet-like interface, without the need to learn complicated SQL commands." Awesome!

I downloaded the file from SourceForge and unzipped it. It is a single EXE file with no need for installation, click and run. My kind of program!

Once running I was able to dive into the blacklist files! Well, kind-of.

Welcome to urlclassifier2.sqlite versus urlclassifier3.sqlite

See, actually the first time I went hunting for urlclassifier3 on my drive, I didn't pay attention to the full path location.

I ended up first working with urlclassifier2.sqlite which I did find plainly in the root of my Firefox (Minefield) profile. This is the same location and file used by Firefox 2 builds.

When I opened it up with SQLite Database Browser I found the following structure:

goog_black_enchash - details on encrypted hash format can be found here
goog_black_url - contains blacklisted URLs of suspected phishing sites
goog_white_domain - contains whitelisted hostnames as determined by Google
goog_white_url - not currently used

When I eventually realized my mistake, I went back and found the correct one - urlclassifier3.sqlite.

moz_classifier - hashed domain and chunk_id's (my current one lists 18453 records).
moz-subs - hashed domain and chunk_id's (my current one lists 18453 records).
moz-tables - contains four names; test-malware-simple, test-phish-simple, goog-malware-shavar (with add_chunks and sub_chunks values), and goog-phish-shavar (with more add_chunks and sub_chunks values).

Turns out that with a bit of work, you can actually and easily "decode" most of the URL data in the urlclassifier2.sqlite file.

John Oberheide and the oiepoie blogs both have fantastic information on the Firefox 2 blacklist file

Here's how I figured out how to decrypt them using John's information:

  1. Using my new friend SQLite Database Browser, I exported both the goog_black_url and goog_white_domain files as separate text files to my desktop.

  2. If opened in notepad or another application, they appear to have a URL-like format, but are unreadable.

  3. The trick here is knowing (from John and Mozilla's posts) that they are actually in a ROT13 encryption format.

  4. If you open either of the exported files in NOTEPAD++ or EditPad Lite (thanks for the tip TinyApps!) both have a "convert ROT13" function. Run that function and there are the URL's in their normal glory! Or if you just want to pick out one or two, use an on-line ROT13 converter,

Explore to your heart's content. I ran a search for against these lists, and didn't find it. Not that I expected to, as Firefox 2 doesn't block, despite checking that urlclassifier2.sqlite file in my Firefox 2 profile folder was only minutes old when examined and clearly from Google's own servers.

Making Mozilla Hash

The goog_black_enchash I never could work out how to decrypt. Not that it's a big secret.

I was able to follow the steps provided in Mozilla's Wiki to create several MD5 hashes for I figured, if I couldn't decode them, maybe I could encode it and compare the hash against those hashes in the list.

So I ran the following URLS through the ropes: and the hashing process:

  1. Based on the Mozilla Wiki page I take the current database salt "oU3q.72p" and concatenate it with the sub-hostname string, and uppercase the results, thusly:



  2. I then computed the MD5 hash of each item:

    • 55D5ADE94D9C39E4EA9EC58E932EE62A

    • C4590E8D8FABD03ED33F3C4090DE4050

  3. I then ran a search for each of these strings in the exported table, but no hits.

Again, I didn't expect any as Firefox 2 doesn't block the site, so they shouldn't be there.

I wish I could have decoded all the URL's listed in the goog_black_enchash export file. However, I'm not an advanced script-writer or a noobie cryptanalyst. So I was quickly lost following Mozilla's instructions on how to decode the data using base64 encoding, along with the "database salt, random salt, and hostname. You also have to toss some RC4 into this mix (Here is a java-based RC4 tool.) If anyone reading this has those skills and can write something to help, or provide steps for the witless like me to follow, it would be nice.

Why all the Obfuscation?

Simple. No dark secrets; it's just Mozilla's attempt to keep anti-malware scanners from alerting on the bad URL's contained in their database file.

Unfortunately, I was unable to find ANY documentation on the structure and format Mozilla is using for their revised URL checking file urlclassifier3.sql. Too bad for me.

It appears to actually be designed to function pretty coolly. From the very limited information I have, the Firefox browser is able to request small-sized database update "chunks" from Google's server to help with bandwidth utilization on the network. This way it can perform frequent "micro-updates" to avoid some earlier performance issues with this updating process in Firefox 3 (Betas). I'm guessing that the add_chunks are blacklisted URLs being added to the database while sub_chunks are coming in to remove URL's now deemed safe.

I still wonder if it would be possible, were a particular URL to be identified, if someone with enough technical knowledge could go in and, using a SQLite tool" delete the offending record out of the database or modify it. Of course, if I could, then I have no doubts that an attacker could use this as a vector to remove "true" listed phishing/attack site URLS or even falsely seed it with legitimate websites. Probably not really a useful attach vector, but certainly worth considering.

Full Circle - Back to the Beginning

So, by 2 AM this morning I decided I had squeezed all the blood I could get from this stone.

Here's where I leave off with what I know:

Jesse Ruderman 2007-11-17 16:51:06 PST Comment #3

How should the UI for this work?  Should Firefox ship with a predefined list of
phishing/malware site list providers, like we do for search engines? For ones
that aren't on the list, should there be a way for users to add them?

I imagine we could get one or more of McAfee SiteAdvisor, Netcraft, or GeoTrust
to provide their data in the correct format if we offered to add them as

Mike Beltzner 2007-11-20 08:25:50 PST Comment #4

(In reply to comment #3)
> I imagine we could get one or more of McAfee SiteAdvisor, Netcraft, or GeoTrust
> to provide their data in the correct format if we offered to add them as
> options.

You'd imagine incorrectly. We've asked and extended offers to all of those
companies, and been rebuffed.

This doesn't block, but is wanted. The best way to do it, I'd imagine, would be
to allow alternative providers to register themselves similar to how feed
readers and other web apps do, providing the endpoints for the Safe Browsing
protocol. The UI would be a drop-down of available providers in the prefPane.

If we don't get it for Firefox 3, alternative providers can still support
themselves by creating an add-on.

  • At least in the case of, StopBadware and Google provide no way for the public to find what triggered the alert "attack" ranking blocking the website.
  • My unprofessional review of the site's page-code doesn't find anything alarming. I am curious if any experts could verify this.
  • My "gut" feeling is that there may be one or more programs available for download on the site that could be "potentially" used for ill-purposes/hacking against a system or network. Maybe the availability alone got them hit?
  • StopBadware swears that a site can't be listed in error or for no good reason.
  • Firefox 2's current urlclassifier2.sqlite file doesn't blacklist the site, Firefox 3's urlclassifier3.sqlite file does. Does anyone see an issue with consistency here? Hard to know who and what to trust now.
  • Mozilla doesn't (currently) offer any options for Firefox 3 (Minefield) users to click-through on their own to their desired website. We can only entirely disable the protection globally to get to the site, or leave it engaged and give up or switch to a different browser.
  • Mozilla's own developers have in the past openly wrestled with the philosophic issues around the Phishing/Attack protection design problem:
Warning Dialog UI

The hard part of a feature like this is not identifying phishing pages, but rather figuring out how to present this information to users. With this in mind, Google did usability testing of a multitude of potential UIs. In the process we learned:

  • When a phishing warning is encountered, users have three primary reactions: a strong desire to continue to use the page, a near-panicked desire to get away from the page, and a desire to see the villains brought to justice. In cases where users heeded the warning, they typically also needed a sense of closure, so it's important to give it to them (hence the "report to google" option).
  • We originally showed the warning only when the user began interacting with a problematic page, but it quickly became clear that users prefer we just disable the page as it is loading.
  • Warnings of "suspicious" pages (pages we're not positive are phishing, but that might be) were by and large ineffective. Users either ignored the warning or said they'd prefer us to make a judgment for them, and that they'd trust us on it. Such warnings probably don't give the user enough information to go on; perhaps they'd be more effective if accompanied by a way for users to do a "background check" on the page.

<snip>....This is by no means final; we're still doing testing. Additionally there are obvious branding issues here that need to be considered. But this is the basic idea.

For those of you who have stuck this post out and are still with me, I'm NOT complaining or questioning either Google or StopBadware's mission and efforts to try to keep the web a safer-place for its citizens. God Bless Them for working hard and establishing some sort of review and process to help folks stay safe as they search and surf the web.

Same goes for Firefox. For the most-part, I applaud Mozilla's efforts to integrate a phishing/attack filter on their wonderful browser.

I am strongly complaining about the lack of information than both StopBadware and Google provide to help us gauge for ourselves the specific reason(s) for the blacklist rating provided. As well as the inconstancy in filtering results depending on which of their browsers I use.

I'm not trying to erroneously defend ScanWith here. They may or may not be hosting malware on their site. They may or may not just find themselves blacklisted due to the content of programs they host for download. That's my point. I can't say and no-one seems to be able to or is willing to tell me (and the public at large).

I also regret that Mozilla doesn't have some kind of mechanism in place for me to click-through anyway to get to the website in Firefox 3. Even if it would take a discouragingly painful amount of prompts to do so would be fine (like how Firefox requires me to add some Add-on developer's websites to an internal white-list first when downloading an XPI file from a non-Mozilla hosted website).

I have learned how to view many of the files blacklisted/whitelisted by Firefox 2. I really would like to be able to do the same thing for the ones in Firefox 3's urlclassifier3 file. (Nir, are you thinking what I am hoping for?)

Despite all this, I am pleased that I have now gained an deeper understanding of Mozilla's use of SQLite database files and found the SQLite Database Browser tool. Poking around in additional sqlite files in my Mozilla profiles with the utility provided some interesting information as to what and how Mozilla stores user data. Cool stuff.

More Information on the Mozilla Phishing (and Attack) Protection

If anyone is more familiar with these issues and can leave clarifications or corrections, comments would be appreciated. Just in case I'm out in left-field here singing to myself.



Asa Dotzler said...

I may have missed it in my quick read, but in case not, then let me note here that the phishing protection and the malware/attack site protection are two different lists with two somewhat different concerns and two different resulting UIs.

The phishing protection will let you advance to the site but tries to warn you that it could be a social engineering attack (phishing).

The malware/attack protection won't let you advance to the site because the site has been determined to try to install evil software on your machine without your consent or otherwise attack your browser or computer.

Phishing protection existed in Firefox 2. Malware/attack site prevention didn't and is new to Firefox 3.

- Asa

Claus said...

Asa, you are correct.

I talked around the differences but may not have pointed this out nearly as clearly as you did. Good call! I really appreciate you taking the time to leave your comment and refine that point for me.

The two "options" screenshots I have show what you are describing. In FF2 you can only enable/disable "suspected forgery" warning.

In the current iteration of FF3, you can enable/disable "suspected forgery" websites and also now have the option for "suspected attack" sites.

Looking at the differences between the urlclassifierX.sqlite files/contents shows that FF2 uses blacklists and FF3 breaks up into an internal distinction between malware (attack-sites) and phishing lists. Clever stuff.

This is a new feature of FF3 and in my opinion, is a good one to have. I 100% agree with it.

My fussing is about the fact that for the sites labeled "attack" I don't have either any way to find out exactly WHAT factors triggered the "Attack" designation for the site, nor (as an advanced and danger-loving techie) do I have any option to reach the so-designated site in my preferred browser outside of disabling this feature entirely or using an alternative browser.

I think many folks can understand the warning about a phishing site. Seems clear to me why they are trying to protect me...but I am allowed to click through anyway and loose my identity and/or $$$? (Though I would like to see the use of the word "phishing" instead of "forgery" here on the Options page; that seems fairly well understood now in Net-land.)

With the vague and alarming "attack" title, I can't evaluate the designation. Does the site host malware-infected downloads? Is the site compromised and page-code exists that will maliciously download software and execute it on my system? Or maybe the site is hosting "potentially unwanted applications (PUA's)" that are not in-of-themselves malicious (network scanners) but could be used by hackers maliciously.

Point is that I now have to "trust" outside parties (Google) to control the web-browsing alerting with no recourse or way to find additional information about the threat.

Let me be clear. I'm not at all making a "Google/Mozilla" are 3vil and censoring everyone argument. Not in the least. I am actually a strong supporter of both.

However, in the case of ScanWith it is a site that I have-up until now-trusted, how can I evaluate the threat and (if it was a real attack-site) use the threat-vector information to evaluate if my prior visits may have compromised my system? I have to "Trust" them blindly or not trust them at all.

I just want more information and a more refined bypass option (however many hoops I need to jump through to verify I am really doing this with forethought) if I decide I don't agree with the designation instead of just turning off the entire attack filtering capability. That's all.

Aside from this truly minor fuss, it really was a neat learning experience for me and I really do like the way the sqlite files work and are being used in Firefox. Especially with the incremental update handling of the urlclassifier files.

Again, cool stuff!

Claus said...


Updated my post in a few spots to make the point you make clearer and to correct the erroneous comparison I made in the FF2 and FF3 blocking behaviors.

Thanks again for helping me clear that up!


Anonymous said...

Hi Claus,

I can't offer you any more insight into the rating methodology, but I think it's important to note that they appear to use cyclical scans similar to SiteAdvisor and others. Obviously depending on criteria, timing and frequency, one may find something that is gone or unrated by the time another takes a look (I think you said essentially the same thing above). Another option is to use a type of on-demand scanner service such as LinkScanner Online. I think of it not so much as a replacement but rather as a supplement; another tool that may help in those "tie-breaker" moments.

And if you're not familiar with Exploit Prevention Labs already, you probably will be soon since they were just purchased by AVG. It should be interesting to see how they bring this and Ewido together.

Anyway, thanks again for a great post and keep 'em coming!


Claus said...

Hi Steve,

Thanks for the extra info. I hadn't heard about Exploit Prevention Labs getting picked up by AVG. It will be interesting to see what becomes of this combination. Thanks for the link!

More for the curious:

Exploit Prevention Labs Blog
Exploit Prevention Labs : LinkScanner
Grisoft acquires Exploit Prevention Labs | Tech news blog - CNET ...

I've used Dr.Web link scanner myself as an add-on in Firefox.

Free Dr.Web link checkers in your browser.3


BartZilla said...

[Citations from your aricle.]
"Because I am using the one from (presumably) Mozilla in 2.0 (and not downloading Google's) I'm cool. "

Nope. It is downloaded from Google, too. (Browser connects regularly to address, sends also Google's cookie.) I guess GUI is designed in misleading way deliberately.

""Because Firefox 3.0 uses Google's black-lists, "No soup for you!"

(Even more curiously, when I did toggle the "Check by asking Google..." option in Firefox 2, the site remained unblocked. Maybe I'm not waiting long-enough for the Google-list to be updated after toggling it on? I really don't know.)

You don't have to wait. It is purely evil spyware -- your full visited URLs (including https and ftp; including revisiting site with "back" / "forward" buttons) are sent to Google in real time. See my little project for details and demonstration (if you are brave enough ;-)).

[From your comment above.]
"(...) or using an alternative browser."

Yup, that's the best option. Firefox should be renamed to "Googlefox". It is definitely not privacy-friendly browser anymore.

BartZilla said...

Few more things.
If switching to another browser is too radical step for you, there is another possibility - disable so called "safebrowsing" (i.e. phishing/malware "protection") at compile time. Just add these lines:
ac_add_options --disable-safe-browsing
ac_add_options --disable-url-classifier
to your .mozconfig. (Yes, I know that it can be disabled from GUI too, but I don't want spyware as part of the browser at all.)

Comment regarding links to specifications at the end of the article: those links are all related to FF2 only. FF3 implements new protocol, specification is not hosted under domain anymore, but at Google:

Especially "3.7. HTTP Request for Full-Length Hashes" is interesting, since it allows Google to check what pages users are visiting...
I am going to make some more detailed analysis in the future and publish it on my website.

BartZilla said...

I said:
"Especially "3.7. HTTP Request for Full-Length Hashes" is interesting, since it allows Google to check what pages users are visiting..."

Well, just to be clear here, I should reformulate it: it potentially allows Google to check whether user visited chosen pages (domains) and when.
(I'm not native English speaker, so bear with me :>).

Claus said...


Very interesting work you have done.

Firefox Antiphishing Server-side Project

Thank you for sharing your findings on my humble blog. (and your English seems very good to me!)

It appears that Firefox 2 and Firefox 3 were using different blacklists. The one in Firefox 2 only checked against "phishing" sites while Firefox 3 URL database has components for both the "phishing" and "attack" URL's. That's why the website worked in Firefox 2 and not Firefox 3.

Thanks for the link to the FF3/Google protocols. I'm going to spend some time reading the 3.7 HTTP Request section based on your recommendation.

All in all, I really am concerned that Firefox isn't being more up front with this. I'm deeply conflicted. I personally am a bit turned off by it all, but have (a tiny bit) of ability to dig for and understand the technicals in the design. Most "average" home users don't and probably could care less.

I only hope that as Firefox 3 becomes close to release, they might make these "features" a lot more visible, so folks can make educated decisions.

If it does indeed come to fruition that Google (or others) are able to use the features you have illustrated to "track" browsing behavior in Firefox 3, it certainly gives pause for thought, especially since most hard-core/average users seem to consider Firefox to be a "safer" browser choice when it comes to security and privacy.

Thank you again, BartZilla for the outstanding comments you have shared!


BartZilla said...

Thanks for your warm comments.

"It appears that Firefox 2 and Firefox 3 were using different blacklists. The one in Firefox 2 only checked against "phishing" sites while Firefox 3 URL database has components for both the "phishing" and "attack" URL's. That's why the website worked in Firefox 2 and not Firefox 3."

Yes, indeed. Well, my main point was that "ask google" mode in Firefox 2 sends full URLs to Google (or to anyone else, who is able to change configuration on client side and enable this mode).

"Thanks for the link to the FF3/Google protocols. I'm going to spend some time reading the 3.7 HTTP Request section based on your recommendation."

Well, AFAICT new protocol works like this: browser connects in regular intervals (how often? Google decides, see "n:" parameter in spec., "3.5.2 Response Body") to Google and downloads only partial hashes of domains/URLs. When user visits some site, browser calculates hash of visited domain/page and if it matches one of the stored hash then browser requests full hash. What does it mean? Google can say whether and when user visited particular domain/page, since they know which hash matches what URL. (After all, they have calculated it on the server side and sent to the browser in the first place.) It seems that Mozilla tries to close this particular loophole, but actually they failed to do it properly (perhaps by mistake, perhaps deliberately; Google/Mozilla definitely know the meaning of "plausible deniability"). I will make some comment in this bug soon.

[From main article:]
"I have learned how to view many of the files blacklisted/whitelisted by Firefox 2. I really would like to be able to do the same thing for the ones in Firefox 3's urlclassifier3 file."

Well, after reading my summary above I think this is now clear that it is impossible. There are stored only partial hashes of URLs/domains, not URLs/domains itself. So, there isn't other way than bruteforcing. And well, this is probably not very practical approach anyway (especially that 32-bit hash is not very long, so there are possible some collisions).

It has interesting implication: you'are totally unable to say what pages are blocked -- unless you visit one of them.
Moreover, Google is able to differentiate between users thanks to cookie -- so, in theory, they can send different blacklists (i.e. different hashes) to different users. So, they can track/block chosen pages for chosen users. At least these are technical possibilities with current implementaion of "safebrowsing" in FF3 (which is enabled by default and, as you probably noticed, the options still don't say that in practice this means connecting with Google).

"have (a tiny bit) of ability to dig for and understand the technicals in the design"

Well, IMO this specification is not very clear... If you know C++/JS and Mozilla's way of doing things in source code, then the source is, well, the best source of knowledge about Firefox. See browser/components/safebrowsing/ and toolkit/components/url-classifier/ in source tree. Probably one of the most interesting files is nsUrlClassifierDBService.cpp.

"Most "average" home users don't and probably could care less.
(...) most hard-core/average users seem to consider Firefox to be a "safer" browser choice when it comes to security and privacy.

Yes, indeed. That's one of the reasons why it (this whole stuff with phishing/malware "protection") really gets on my nerves.

But well, think about it. What is the best place to plant some kind of spyware? Popular, open-source browser with many devoted zealots seems like a very smart choice (Google is smart, isn't it?). First, it is popular and cross-platform, so you can gather a lot of interesting data not available by using other means. Open-source -- since source is available most people think it is totally spyware-less -- but actually how many users audit source code of their applications, especially as big as WWW browser and as complicated and involved as this "safebrowsing" thing?
Just some food for thought, and sorry for my lame English (BTW -- feel free to tell me what grammatical and other mistakes I've made; I'm still learning, after all :)).

BartZilla said...

I said:
"IMO this specification is not very clear"

Well, OK, it is not that bad after all. :-) And definitely easier to read than source code.
Cheers! (And sorry for spamming your blog with one more of my comments...)

BartZilla said...

Just FYI: after my chatting on IRC (on #firefox, someone has filled a bug in Bugzilla: "make "safebrowsing by google" more discoverable". I've created a patch that adds " according to Google" in descriptions of options in Security window (ie. changes "Tell me if the site I'm visiting is a suspected attack site" to "Tell me if the site I'm visiting is a suspected attack site according to Google" and "Tell me if the site I'm visiting is a suspected forgery" to "Tell me if the site I'm visiting is a suspected forgery according to Google"), however I doubt they're going to do something with this...

Anonymous said...

hi. just want to add a head spinner. with all these anti phising and malware protection stuff, why is my firewall showing this:

[03/Nov/2008 16:24:05] VIRUS charset="en" file=";7776-7780;:" firewall="[computer name]" hostip="[IP]" hostname="computer name" protocol="HTTP" time="Mon Nov 03 16:24:05 2008" username="[user]" virus="External AV verdict: probably unknown STEALTH.POLY.CRYPT.TSR.DRIVER virus"

btw, items inside brackets are substituted. I have a NOD32 external AV plugin integrated in my firewall and i found this in the security and alert logs of the firewall. i'm really just wondering. i searched any info on this but i guess your blog has the closest answer i can find. this has something to do with firefox's protection measures. what really bothers me is why the heck is my AV classifying this as an unknown/possible threat. does google safebrowsing contain threats? hope i can hear from you guys soon.

Claus said...

@ Anonymous - I can't say for certain but based on the general information you have provided, here is what I think is going on.

Looking at the URL, I suspect your Firefox 3.x version build was caught downloading a "chunk" update for the browser's antimalware "attack site" SQLite database portion.

That normally happens on a semi-regular basis, but this time I suspect that this particular chunk had some kind of sequence in it's data-packet that matched the A/V classification pattern. Thus the alert got triggered.

I suppose if you were really concerned you could try uploading the urlclassifier3.sqlite file in your firefox profile up to Jotti or Virus Total for a file-scan but I doubt you will get a hit on it. The chunk data gets incorporated into the main SQLite file as part of the update process.

You might want to look at these (semi-related) posts as well as they contain either specific information on the process or urls with elements similar to the one you encountered:

Read - See Update 4 at the bottom.

Google Safe Browsing

Bug#503127: iceweasel: privacy violation in default settings of Iceweasel (connect to Google, Mozilla and BBC)

Protocolv2Spec Client specification for the Google Safe Browsing v2.1 protocol

If you are still feeling unsure about this possible threat, I would suggest you leave a comment at the bottom of that last Google Code page in the comments. Looks like it is pretty active. I'm sure someone will have a more technical answer.

Great catch and find! Thanks for sharing it here!


BartZilla said...

Hello again!

It took me some time to figure out all details of the new protocol (ie. the one used in FF3 and also in Google Chrome BTW) and then to implement it, but finally I made it. Firefox 3 "Antiphishing/Antimalware" (so-called "safebrowsing") Server-side Project tries to demonstrate how "safebrowsing" works.

The guy in Debian's bug #503127 commenting that there is no "privacy concern" is obviously misguided IMO...