dslreports logo
 
    All Forums Hot Topics Gallery
spc
Search similar:


uniqs
1905
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Need 10GB recommendations

Click for full size
Dataline.pdf
142,151 bytes
I am designing a new network for a very data intensive company. Currently the idea is 3 networks - SAN, Backup, Production. The SAN connects directly to the servers via 8GB Fibre channel aggregated to 16GB by bonding. From there the servers (2) will have a dual port 10GB card. One port will go to the production network and end users. The other to a switch for the backup network. On the backup switch, it will connect to the production network by a 10GB link. Goal is to have the SAN backup to the backup equipment via the direct connection. But if an end user has to access it then it will follow the switch to switch connection.

With that said... I need to decide what type of switches the production and backup should be. By my count each will need 3 10GB connections. I am finding that hard to find. Any suggestions?

Brano
I hate Vogons
MVM
join:2002-06-25
Burlington, ON

Brano

MVM

Don't have the answer for you, but you may want to update your diagrams and show Gb instead of GB which is incorrect.
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Yeah it was a quick throw together thing. More for me to see it working.
LittleBill
join:2013-05-24

LittleBill

Member

to be honest your diagram is still vague. i work in the san/lan world for a living.

i don't see any redundancy in your san network, no dual pathed hosts into the san, no dual fiber switches etc....

and generally backup of sans are done via some form of vtl through a host or if natively supported by the san itself.

nothing is gained by direct connect, most enterprise level switches can do wires speed on all ports.

you obviously have not looked at switches well. any of the cisco e series with slots support a numerous amount of card options including 10G ports fiber or copper based depending on your needs.

i have no idea on your budget but most stuff mentioned is in the 6 figure range

Anav
Sarcastic Llama? Naw, Just Acerbic
Premium Member
join:2001-07-16
Dartmouth, NS

Anav

Premium Member

If I was LittleBIll I would be better positioned to day it looks like homework, I can only say it may smell like it.
LittleBill
join:2013-05-24

LittleBill

Member

said by Anav:

If I was LittleBIll I would be better positioned to day it looks like homework, I can only say it may smell like it.

not really sure on the comment, he has homework or i do?
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

This is no homework, well unless you consider quote a large project for a client is homework? Anyways... I didn't build any redundancy into it yet. The SAN is a 48TB unit with dual controller/PS. I may or may not add the 2nd fiber switch. As for the backup network, that is just for long term storage and archiving. They can fill 24TB in a week with what they do. Once they are done with the current data they archive then move the new project into the SAN. As for switches I just got a reply back on some Cisco 4948E's for $1600 a pop. So I'll probably go with those.

As for the separate networks, again they can easily be transferring 24-30TB to the archive at any given time. I don't want that to bog down the connection from the server through the production lan switch.

Anav
Sarcastic Llama? Naw, Just Acerbic
Premium Member
join:2001-07-16
Dartmouth, NS

Anav to themagicone

Premium Member

to themagicone
My apologies.
LittleBill
join:2013-05-24

LittleBill to themagicone

Member

to themagicone
said by themagicone:

This is no homework, well unless you consider quote a large project for a client is homework? Anyways... I didn't build any redundancy into it yet. The SAN is a 48TB unit with dual controller/PS. I may or may not add the 2nd fiber switch. As for the backup network, that is just for long term storage and archiving. They can fill 24TB in a week with what they do. Once they are done with the current data they archive then move the new project into the SAN. As for switches I just got a reply back on some Cisco 4948E's for $1600 a pop. So I'll probably go with those.

As for the separate networks, again they can easily be transferring 24-30TB to the archive at any given time. I don't want that to bog down the connection from the server through the production lan switch.

good luck with your project, hope you got a hell of a backup system if you plan on archiving 24 tb a week. thats a significant amount of data even for a enterprise customer.

hope you have some form of deduplication/compression for your system.

personally i would be more concerned with the backup. archiving that much data is a significant project

if you run the numbers thats a pretty high sustained transfer rate. you will prolly find the IOPS on the server to be the limiting factors, but i digress. Good Luck
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Server wise we are looking at Quad AMD 16 Core's and 256GB of memory. The goal is to limit the amount of times they have to move from production to archive. Right now, they just buy a dang new computer, throw some drives in it, wait till it fills then shove it in the corner. Really anything is better. Any design suggestions are gladly taken. Thanks
LittleBill
join:2013-05-24

LittleBill

Member

sadly

this is all over the place. this require a significant amount of prep to be successful. a budget etc.

24tb in the real world is expensive, 256GB of ram is a joke ,especially shared over esx, if your really dealing with TB size volumes, not to mention the fact you are prolly using sata type drives, maybe sas for the esx servers. based on what your saying, you can't afford fiber or enterprise ssd type storage.

i will simply finish with 10Gb while nice, will probably be overkill.

filling that pipe would be 1.25GB of data moved a minute. i will tell you right now.

we have arrays that can do this. they start in the 7 figures. and that data generally has to reside in cache for those numbers to be hit.

that being said. performance on your servers/san/and backup will be fastly more important
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Budget is right at $100,000. We are looking at Aberdeen for the SAN. They have a 48TB model with SAS drives and dual 8Gb Fibre channel to the servers for $26k. Servers are around $10k each. Dual 4948E Cisco's with 8 ports of 10Gb and cables is $4k. Software/Labor is eating up the rest. This isn't a huge company, I think 10 users? Just lots and lots of data.
LittleBill
join:2013-05-24

LittleBill

Member

well like i said, good luck i think 10 gig is overkill stated for the reasons above.

but you found what you needed with the switches.

good luck
HELLFIRE
MVM
join:2009-11-25

HELLFIRE to themagicone

MVM

to themagicone
Thanks for posting your budget and throughput numbers, themagicone See Profile. Got any other requirements / musthaves?
Specifically in size, power, cooling, etc?

Really dumb question, what kind of 10Gbe PHY interface(s) are you using?

For size-constrained setups, the Catalyst 4500X is a rather nice unit, with a Juniper equivalent being the EX4550.

I'm wondering if a WS-X6704-10GE (4x 10GBE interfaces) linecard wouldn't be too much off the mark, if
you're looking to push packets 24x7 with no oversubscription.

Other than that, I suspect there's products from the likes of HP, Force10, etc. as well... I just can't
name any off the top of my head.

My 00000010bits

Regards

tubbynet
reminds me of the danse russe
MVM
join:2008-01-16
Gilbert, AZ

tubbynet

MVM

said by HELLFIRE:

Other than that, I suspect there's products from the likes of HP, Force10, etc. as well... I just can't
name any off the top of my head.

arista.
cheap. 10gbe.

q.
HELLFIRE
MVM
join:2009-11-25

HELLFIRE

MVM

...ahh, forgot about them tubbynet See Profile. Thanks!

Regards
mkaishar
join:2000-12-20
00000

mkaishar to themagicone

Member

to themagicone
Looking at your requirements, I would go a different route:

Get a pair of TrueNAS ZFS servers with multiple 10Gbe NICs and lots of RAM or ssd drives for arc and l2arc

setup your vmware vmfs volumes on nfs (unless you have MS Exchange - Exchange databases are not supported on nfs vmfs volumes)

get 2 10Gbe switches for redundancy, vlans if you want to separate production and backup traffic

setup snapshot replication from the primary storage server to the backup

we have a pair with ~120TB of storage dedicated for HD videos and video archiving, for the money nothing came close to it, we add about 8TB of data about every quarter

i run snapshot replication from primary to backup, the initial replication took a while, but it was almost 80TBs, I did that over 10Gbe xover connections

Once completed, I moved the backup server to another building connected over gigabit wireless.

We bought the pair for about $60K with ~80TB each, when we came close to 80% utilization, bought another 18 4TB NL-SAS HDDs (9 drives per server), used zfs to add to the existing pool, no downtime necessary

i would also not use cisco, instead force10, much better for the money investment and they are fast.
AsherN
Premium Member
join:2010-08-23
Thornhill, ON

AsherN to themagicone

Premium Member

to themagicone
said by themagicone:

Budget is right at $100,000. We are looking at Aberdeen for the SAN. They have a 48TB model with SAS drives and dual 8Gb Fibre channel to the servers for $26k.

Methinks you're in over your head.

That SAN does not have 48TB. It has 12 4TB drives. Once Raided, that's really 24TB of usable space. And they are not SAS drives. They are NL-SAS. Which are essentially Enterprise SATA, 7.2K RPM drives. I have 100s of TB of those. They doo a job, but they are not the fastest.

What RAID level were you looing at? That many large drives, and any parity type RAID is asking for the array to never be able to rebuild.
mkaishar
join:2000-12-20
00000

mkaishar to themagicone

Member

to themagicone
I revise my recommendation after re-reading your scope, for 10 users, odds are they are all connected to the network over 1Gbe, I would not even bother with multiple networks, just stick everything into a single network, you can bond multiple 10G ports if necessary.

Go with Truenas and zfs with snapshot replication, no need complicate things with different technologies (fiber san to tcp, etc...)

Also if you go with nfs for vmfs, then you can ultimately just buy esxi standard which will save $$$.
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Its 24 3TB drives. Raid 6.
AsherN
Premium Member
join:2010-08-23
Thornhill, ON

AsherN

Premium Member

24 3TB 7.2K NL-SAS in a RAID6 array? you must hate performance and hate your data even more.

Every single write operation will generate 24 IOs.

Rebuild times aside, you will likely never see that array rebuild.

Read this: »www.zdnet.com/blog/stora ··· 2019/805

The crux of the problem RAID arrays are groups of disks with special logic in the controller that stores the data with extra bits so the loss of 1 or 2 disks won't destroy the information (I'm speaking of RAID levels 5 and 6, not 0, 1 or 10). The extra bits - parity - enable the lost data to be reconstructed by reading all the data off the remaining disks and writing to a replacement disk.

The problem with RAID 5 is that disk drives have read errors. SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. Which means that once every 200,000,000 sectors, the disk will not be able to read a sector.

2 hundred million sectors is about 12 terabytes. When a drive fails in a 7 drive, 2 TB SATA disk RAID 5, you'll have 6 remaining 2 TB drives. As the RAID controller is reconstructing the data it is very likely it will see an URE. At that point the RAID reconstruction stops.

Here's the math: (1 - 1 /(2.4 x 10^10)) ^ (2.3 x 10^10) = 0.3835

You have a 62% chance of data loss due to an uncorrectable read error on a 7 drive RAID with one failed disk, assuming a 10^14 read error rate and ~23 billion sectors in 12 TB. Feeling lucky?


To rebuild your array, you will need to read 66TB. With those URE numbers, you should plan to encounter 6 of them during a rebuild. Because you are using RAID6, you will survive the first one. But only the first one.

A NL-SAS drive transfers data at about 150Mbps. it will take 3.7 hours to fully read or write a drive. You will need 88.8 hours, 3.7 days, to rebuild the array, if allowed to consume 100% of the resources. If the array is in production, the controller card may allocate as much as 5% to the rebuild. Meaning you will need 74 days to rebuild the array.

Best of luck.
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

1 recommendation

cramer

Premium Member

A) His math is flawed. If his BS reasoning is correct, badblocks -w should kill almost any modern TB drive. (hint: IT DOESN'T)

B) You're thinking RAID 4. RAID 5 and 6 use distributed parity. It doesn't have to read all disks to write a single block. And if you're writing a large amount of data with any measure of caching, it's very likely the blocks you'll need to read are also being written. (OSes and RAID controllers are very smart about minimizing IO and seeks.)

C) I had a 12TB Dell MD3200i. It initialized as RAID 6 in about 4hrs. (hint: the bus is faster than the drives.)
AsherN
Premium Member
join:2010-08-23
Thornhill, ON

AsherN

Premium Member

Real life rebuild times in production arrays, a failed 4TB drive in a RAID10 array takes around 5 hours to rebuild.

There is a reason RAID5 is no longer an option in Enterprise arrays, RAID6 is next.

Will the rebuild fail everytime? no, but the odds are damn good it will. Do you want to take the chance with you're data?
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

Alright then... since my logic is flawed, would you enlighten me with which method I should use? Should I do 6 x 4 (3TB) setup? Or 4 x 6? I can set it up any way but what is the best?
AsherN
Premium Member
join:2010-08-23
Thornhill, ON

AsherN

Premium Member

For reliability and speed, at those array sizes, RAID10 is the recommendation.
LittleBill
join:2013-05-24

LittleBill

Member

raid 6 is used in every major data center in the world along with raid5 that said, generally we try to keep it 7+2 for 6.

the drives will take a day or 2 to rebuild.
AsherN
Premium Member
join:2010-08-23
Thornhill, ON

AsherN

Premium Member

RAID5 is no longer available on Tier1 arrays anymore. That should tell you something.

Sweet spot for parity is around the 6-7 drive per set. I use RAID6, mostly on SSD arrays because they do not suffer from the same URE limitation. Or on large arrays where I don't care if it ever rebuilds.

24 3TB drives I would never think about it.
cramer
Premium Member
join:2007-04-10
Raleigh, NC
Westell 6100
Cisco PIX 501

cramer to themagicone

Premium Member

to themagicone
The URE number is not "per random block", but "per same bit". If you read the same sector 10e14 times, then you are statically likely to have 1 failure. It's not 1 failure per 10e14 READ's (across many blocks.)

Most arrays today do RAID6 because that's how people ran them anyway... RAID5 + hot spare, periodically rebuilding onto that spare. RAID6 puts the hot-spare into the live array.
AsherN
Premium Member
join:2010-08-23
Thornhill, ON

AsherN

Premium Member

ummm, no. URE ratings on drives are not per bit, but per drive. That is why RAID5 has been deprecated, and RAID6 is following fast on it's heels. A URE of 10^14 translates on a URE for every 12TB read. In the heydays of RAID5, when a large drive was 146GB, it did not matter. As drive capacity has increased, the URE ratings have not. The larger drives get, the better the odds of URE during rebuilds. In theory, if we can ever get to spinning drives greater than 12TB with the same URE, even RAID1 would not rebuild.

Rebuilding a parity array, requires you to read the entire drive at least once. So rebuilding a #TB drive based array, you use 25% of the possible read life. Add to that reads already performed during the life of the drive, and new reads required as new data is being written to disk, and the odds are not good.

RAID5 + HS is NOT RAID6. A rebuilding RAID5 array cannot tolerate a single URE. A rebuilding RAID6 could.

Recent firmware on Enterprise grade arrays have removed RAID5 as an option. Not a good idea anymore.
themagicone
join:2003-08-13
Osseo, MN

themagicone

Member

If I did Raid 10, that would give me 12 usable drives or 36TB. That might work. We are probably going to do a 6 to 10 drive SSD array on top of this storage to provide quicker IO to the server.