how-to block ads
|
DrTCP
Yours truly
Premium,ExMod 1999-04
join:1999-11-09
Round Rock, TX
| reply to Network Guy
Re: Linksys vs. Netgear...
First, small packet performance is not really meaningful. You generate large packet during any significant transfer. Whatever is using 64 byte packets are not using much bandwidth anyway so the test is very artificial.
Secondly, a NAT router that does not even implement packet filters has an edge over one that implements packet filters. Netgear RP114 in test probably came preconfigured with a couple filters in default configuration (though these filters are no longer needed in 3.25 firmware Netgear is shipping routers with these filters installed anyway).
Third, 10 sec test is not enough. Some router significantly degrade when test is extended.
Fourth, PC Mag is not the only company that did tests. Here is another picture from Tolly Group.
»www.2wire.com/products/pdfs/tolly_hp0501.pdf
RP114 is basically the same as RT314 plus some rudimentary content filtering.
Finally, the reviews did not consider the stability of firmware. Also, the reviews did not note that Linksys is not properly translating the port numbers as the packet goes through NAT and solely relying on randomness of packets. If two PC uses the same fixed source port it does not work with Linksys router. Linksys routers do not support UDP based traceroute of Unix like systems either. These are just two examples that demonstrate corners cut in the design of the router. Of course, when you implement less you have more CPU cycles to move data.
[text was edited by author 2002-02-05 20:22:30] | |
okaven
join:2001-12-02
New York, NY
| [[First, small packet performance is not really meaningful. You generate large packet during any significant transfer. Whatever is using 64 byte packets are not using much bandwidth anyway so the test is very artificial.]]
Yes, the test is artificial. However, unfortunatly do all hardware manufacturers indicate their switch or router performance based on 64-byte frames. Also, the traffic generator is layed out to actually stress the device harder using 64-byte frames. While it is true that the frames are smaller, the DUT (device under test) has to work harder to move the same amount of data comprised of 64-byte frames than 1518-byte frames.
[[Secondly, a NAT router that does not even implement packet filters has an edge over one that implements packet filters. Netgear RP114 in test probably came preconfigured with a couple filters in default configuration (though these filters are no longer needed in 3.25 firmware Netgear is shipping routers with these filters installed anyway).]]
This is true! However, you can see that we chose the Netgear FR318 as Editor's Choice-- despite its performance. We did this especially with small business implementations in mind and we did value the existance of a SPI firewall higher than raw throughput. A small business router without SPI would not have been an option for the Editor's Choice.
[[Third, 10 sec test is not enough. Some router significantly degrade when test is extended.]]
We did run the tests over an extended period of time to see if we run into any problems. As a matter of fact, the Netgear router was one of the routers I let run for over 8 hours.
[[Fourth, PC Mag is not the only company that did tests. Here is another picture from Tolly Group.
»www.2wire.com/products/pdfs/tolly_hp05..[?]]]
This is very true and I am not arrogant enough to say that our testing was the ultimate router testing ever conducted. There are lots of additional tests that I would like to have done, which was largely impossible because of our relatively short testing cycles that are dictated by editorial deadlines. Firmware stability testing and most importantly customer service and long term reliability are items that we unfortunately do not have the time to investigate. However (maybe contrary to public opinion) we do try to design our tests as meaningful as possible and most importantly unbiased. And no, we cannot keep the devices or accept any gifts from vendors that exceed the usual t-shirt or ballpen contributions 
[[RP114 is basically the same as RT314 plus some rudimentary content filtering.]]
Yes
[[Finally, the reviews did not consider the stability of firmware. Also, the reviews did not note that Linksys is not properly translating the port numbers as the packet goes through NAT and solely relying on randomness of packets. If two PC uses the same fixed source port it does not work with Linksys router. Linksys routers do not support UDP based traceroute of Unix like systems either. These are just two examples that demonstrate corners cut in the design of the router. Of course, when you implement less you have more CPU cycles to move data.]]
Firmware stability, please see above. As far as packet translation and UDP traceroute capabilities go, you might very well be correct. You have to keep in mind though that our tests were primarily targeted at the home consumer and very simple small business applications. This audience will most likely not care about the depth of problem we discuss here at the geek-fest. In any case, if you do have more sophisticated applications that you wish to run, I would not recommend any of the tested devices anyway and point to more powerful routers from Cisco & Co.
I hope that you don't feel like I am picking apart your post, but you have to understand that this is the fruits of my agonizing labs life that we are talking about here  )
Oliver Kaven
----------------------------------------
Oliver Kaven
Project Leader, Network Infrastructure
PC Magazine Labs
oliver_kaven@ziffdavis.com
-------------------- | |
DrTCP
Yours truly
Premium,ExMod 1999-04
join:1999-11-09
Round Rock, TX
| said by okaven:
We did run the tests over an extended period of time to see if we run into any problems. As a matter of fact, the Netgear router was one of the routers I let run for over 8 hours.
My point was based on the article that claimed the tests are based on 10sec of data passing. Anyway, 8 hours indicates Netgear router tested is pretty stable. I wished you did the same with Linksys as well but I understand time constraints in such large scale tests. As far as I remember, you could actually test several devices in parallel using Ixia or Smartbits.
quote:
There are lots of additional tests that I would like to have done, which was largely impossible because of our relatively short testing cycles that are dictated by editorial deadlines.
I understand. Perhaps, the objectives of the tests and their real life applicability needs to be explained. Given a lot of users will base their purchasing decisions solely based on the tests without understanding what the numbers really mean. Even the techy types have trouble interpreting the results.
quote:
In any case, if you do have more sophisticated applications that you wish to run, I would not recommend any of the tested devices anyway and point to more powerful routers from Cisco & Co.
There is no excuse for crippled NAT implementation. SOHO users deserve better designed equipment as well.
quote:
I hope that you don't feel like I am picking apart your post, but you have to understand that this is the fruits of my agonizing labs life that we are talking about here )
No problem. It was a nice chat! I see there exists agreement in most points I've raised.
My main concern is that as the review results results stands in current form, the user is given somewhat wrong messages in terms of performance.
Also, the advanced features of Netgear routers (well basically RT311/314/RP114) are often ignored by the reviewers.
ZyXEL which supply the design for Netgear routers is completely ignored. ZyXEL Prestige 310/314/316, Z1, Z10 all have significant extras that speak to power users (or as the needs of the user become more advanced) | |
Daniel
Premium,MVM
join:2000-06-26
Pleasanton, CA
clubs: 
|  reply to okaven
Overall...
said by okaven:
I hope that you don't feel like I am picking apart your post, but you have to understand that this is the fruits of my agonizing labs life that we are talking about here )
Oliver Kaven
----------------------------------------
Oliver Kaven
Project Leader, Network Infrastructure
PC Magazine Labs
oliver_kaven@ziffdavis.com
--------------------
I know you were addressing the Dr. but allow me to retort. I enjoyed your report and I have no doubts as to its accuracy. My argument is only that its presentation could cause it to be misleading even for people with a decent amount of networking knowledge.
However, even if someone hated your report they couldn't deny the fact that it was accurate and did us the service of brining our attention to the performance of our routing products.
Keep up the good work, man.
--
"While we are postponing, life speeds by." - Seneca (3BC - 65AD) | |
okaven
join:2001-12-02
New York, NY
| reply to DrTCP
Re: Linksys vs. Netgear...
DrTCP:
I enjoyed the chat myself. You raised some really good points. Besides, this is the stuff that I am (and many of the guys here at the labs)living and breathing. And I am sure that everyone here has more than just an "interest" in networking and broadband related issues.
I am also glad that you guys question everything. After all, nobody wants to get false information-- of which there is plenty out there.
Great talking to you and great job on the site, too! I think this is a great resource for all broadband and cable related issues.
Oliver | |
okaven
join:2001-12-02
New York, NY
| reply to Daniel
Re: Overall...
Thank you! I appreciate it and I am glad that you found the info valuable. And, of course, everyone here at my end will always try to do their best when we work on a story.
We have some wireless stuff coming up in issue 10, so keep your eyes open! (just had to do a little advertising )
Oliver | |
|
