| Informal Trojan Detection Tests # 2 Hi All:
If you read the web page detailing my previous series of Informal Trojan Detection Tests...
»www.staff.uiuc.edu/~ehowes/troja···ests.htm
...then you'll recall that popular anti-virus and anti-trojan applications can encounter difficulties when confronted with trojan servers that have been packed with the many executable compression programs available on the Internet.
In response to a number of questions about how well other applications would perform under the same conditions, I've run a new round of tests. As before, I used the Sub7 2.13 MUIE server, packed several different ways (and even left completely unpacked in one case). Once again, I selected seven programs to test: four dedicated anti-trojan programs and three standard anti-virus programs. I also ran the same types of tests as before, mixing on-demand scans file scans with tests designed to check the performance of real time memory and process monitoring components.
In essence, this is the same series of tests as detailed on the original Informal Trojan Detection Test page, but with a different set of applications.
You can read the results of this new series of tests here:
»www.staff.uiuc.edu/~ehowes/troja···ts-2.htm
Before you jump to any hasty conclusions based on what you see on that web page, please take the time to read the "Disclaimers & Limitations" section at the bottom of the page.
I hope you find these tests interesting and useful. As always, comments, questions, and criticisms are welcome.
Eric L. Howes
eburger68@yahoo.com |
|
| Eric, my compliments on an excellent set of tests, well-described and documented.
I find only one (1) fault with the test; the fact that it omits some applications that are in wide use, which I'm sure we'd all like to see compared alongside those you did test.
I find myself curious about Tauscan, Kaspersky/AVP and TDS-3, to name a few.
Nonetheless, it's a remarkable job, very revealing, and the information provides a real service to the Net. You've targeted a weak zone in a lot of people's defenses that they need to know about.
pchelp |
|
 Jamming777$Time Is Running OutPremium
join:2001-07-25
USA | reply to eburger68
Those are in the Test #1 pchelp.
What I object to is using only elements of programs while using many of the others at maximum search ability, also there should of been more than one trojan used, at least three, I would of preferred more. If one definition/ rule/ profile is wrong it skews the report to seem as any one program is completely ineffective. Also, I would like to see someone else run the same report not connected to the college, to see if they duplicate the testing. This is too limited of a test to base your decision on. This is not useful, as your learn in statistics class you need a large sample and blind tests using the same equipment. Unfortunately from a statistical point of view it is all most useless.
--
Jamming *Team Z Member* |
|
|
|
 gt7697cPremium
join:2001-02-16
The Hive | reply to eburger68
Nice job and nice to see a comparison of some of the products. I wish you could add NAV 2002, SwatIt, and Trojan Check on the next round of testing. (If there will be a next round of testing.)
--
Just my 2 bits. |
|
 R2R NotPremium,MVM
join:2000-09-18
Long Beach, CA
kudos:1 | reply to eburger68
Excellent job -- very interesting. |
|
 cjsmithPremium
join:2000-11-03
Villa Rica, GA | reply to eburger68
Thank you Eric for the indepthness, simplicity, and very informative post. Your involvement here is very much appreciated!!!  |
|
 ChipPremium
join:2001-12-23
Connecticut | reply to eburger68
DITTO!! |
|
| reply to eburger68
Very informative, thanks! 
It also triggered a thread (one of many to come, I'm sure) here
Cheers, Tony |
|
| reply to Jamming777$
Jamming777:
You wrote:
said by Jamming777:
What I object to is using only elements of programs while using many of the others at maximum search ability,
I tried to set all programs to their highest "sensitivity" level to give all a fair shot at detecting the various versions of the trojan. There are at least two instances where I used something less than the "highest" level:
ANTS 2.1: "Scan priority" was set left at the default "Normal." My understanding of this setting is that it determines the process priority and is more useful if you're interested in running background file scans while you work. (Later, I'll be hopping on to the other machine, so I'll double-check that.) As ANTS 2.1 doesn't do real time memory/process scanning, it wasn't an issue for those tests.
Tauscan 1.6: In the original set of tests I didn't enable the "Advanced Trojan Analyzer" for the two memory/process scanning tests as the program's description of this option seemed to indicate that this was not a viable/realistic option for real time use. Even with the option on in the first file scanning test, however, Tauscan missed the same trojans as in the memory/process scanning tests.
said by Jamming777:
also there should of been more than one trojan used, at least three, I would of preferred more. If one definition/ rule/ profile is wrong it skews the report to seem as any one program is completely ineffective.
... This is too limited of a test to base your decision on. This is not useful, as your learn in statistics class you need a large sample and blind tests using the same equipment. Unfortunately from a statistical point of view it is all most useless.
As I noted in the "Limitations and Disclaimers" section of both pages, the tests I ran aren't designed to provide a comprehensive picture of any application's complete capabilities. One would need a much, much larger malware sample to even think about doing that.
And I certainly wouldn't base a decision about any of these programs solely on my tests -- the tests simply can't support that kind of decision. In fact, over in grc.security.software, I've already responded to one person who is trialing TrojanHunter, encouraging him to continue with his trial and base his final decision on his own experiences with the program.
Nonetheless, the tests do raise interesting questions, though, esp. since the trojan used was hardly an obscure one.
All too many of the trojan tests I've seen simply involve dumping a couple hundred (or thousand) known pieces of malware on a hard drive and then setting various scanners loose on it. I wanted to do something a bit different -- to explore the effects of compression programs on the ability of popular anti-malware scanners to detect well-known trojans. And I wanted to see use several different scenarios, to see how the programs would react.
Thus, my tests are less concerned with tallying up the number of instances of malware detected, and more concerned with seeing what these programs do when confronted with different types of challenges, esp. real time memory/process scanning.
In doing that, I think the tests have succeeded. At the very least the tests point up troubling problems with some of these applications. And, as I note in the conclusion, the tests ultimately reinforce a lesson that so many users seem to ignore: that anti-virus and anti-trojan programs are not magic bullets, and that they must take responsibility for their own systems (and their own PC habits and practices) themselves.
said by Jamming777:
Also, I would like to see someone else run the same report not connected to the college, to see if they duplicate the testing.
I would certainly encourage folks to test these programs themselves. I've described exactly what files I used, how I created them, and how I tested them. The whole testing process was a valuable learning experience for me, one that I would wholeheartedly recommend to others with an interest in this area. Still further, I can only hope that others would start investigating anti-malware apps themselves instead of relying solely on the existing file scan tests (which do have some value) and the hard-sell marketing done by some (not all) anti-malware vendors.
One last thing: Nancy McAleavey just announced an update to BOClean, partly in response to those tests:
said by PrivSoft Corp.:
Two new nasties added today for a total of 1179 UNIQUE trojans (8,953 variants) covered in today's update for BOClean 4.09, along with adjustments for an "ASP file wrapper" memory offset problem with a specific obsolete version of ASPack. Come and get it at:
»www.nsclean.com/update.html
You can also click down below to download directly from this email if your security settings permit by using the link below:
»ftp://ftp.nsclean.com/pub/update.exe
Click the above to download. The update is safe to run from the internet if you'd like for automatic install from this email.
PSC product forum:
»www.morelerbe.com/cgi-bin/ubb-cg···rum&f=40
I haven't tested this new update (I will later tonight), and I'm interested in getting a more complete description of just what this update involves, but it is available.
In any case, thanks for having a look at the test page.
Best,
Eric L. Howes |
|
TAG97Premium
join:2002-02-28
Bridgeport, CT | reply to eburger68
Thank you Eric. Could you post your findings on the BoClean update? I beleave their response to the results is very profesional. Me being a user of BoClean is hoping for a much improve performance.
Regards
Tim |
|
| Tim:
You asked:
said by TAG97:
Thank you Eric. Could you post your findings on the BoClean update? I beleave their response to the results is very profesional. Me being a user of BoClean is hoping for a much improve performance.
Yep, I plan to do just that. I also plan to test the latest update from Magnus Mischel for TrojanHunter. There's a set of links near the top of the test page now to both updates.
I'm going to try to run these tests tonight, but it may not be until tomorrow. I'm still sorting through the reactions so far -- discussion threads in five separate forums.
I'll post an update notice when I have more info.
Best,
Eric L. Howes |
|
Anon | reply to pchelp7
> I find only one (1) fault with the test; the fact that it omits some applications that are in wide use, which I'm sure we'd all like to see compared alongside those you did test.
> I find myself curious about Tauscan, Kaspersky/AVP and TDS-3, to name a few.
Keith,
From what I understand this is only "part 2" of his tests -- part 1 which Eric released several weeks back contains test information for the other programs you mention
The URL for the first tests -- »www.staff.uiuc.edu/~ehowes/troja···ests.htm
Best regards,
Wayne |
|
| said by Wayne DiamondCS:
Keith,
From what I understand this is only "part 2" of his tests -- part 1 which Eric released several weeks back contains test information for the other programs you mention
The URL for the first tests -- »www.staff.uiuc.edu/~ehowes/troja···ests.htm
Whoops!
Sorry, Eric.
Looking back, I see it's virtually the identical procedure that was followed in January. For some reason I thought it wasn't.
pchelp |
|
