Topic 'Re: money money money money..... MONEY!' in forum '' - dslreports.com

Re: money money money money..... MONEY!

Tue, 13 Jan 2009 13:39:45 EDT

kamm posted :

said by fifty nine:

Yeah I know. :(

I used to work at a much larger DC where we had Veritas Cluster filesystem, RAC and the like. They had a couple hundred TB of storage. This was a financial company.

Now I'm at a different place that deals primarily in media. It's quite a different environment. A lot of our storage is for video content, including HD streaming and on-demand for cable companies.

Completely different needs.

Not necessarily: we deal with high-res media as well, scientific stuff but the main SAN users are usually the 3d/fx guys.
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]

Re: money money money money..... MONEY!

Tue, 13 Jan 2009 06:58:12 EDT

kamm posted :
...

Re: money money money money..... MONEY!

Tue, 13 Jan 2009 00:55:03 EDT

houkouonchi posted : Odds are they would use some sort of NAS setup that multiple servers access. 500TB is not that big of a deal. Take this server at work for example:

# df -HlFilesystem             Size   Used  Avail Use% Mounted on/dev/sda1               30G   5.1G    24G  18% /tmpfs                  2.1G      0   2.1G   0% /lib/init/rwudev                    11M    82k    11M   1% /devtmpfs                  2.1G      0   2.1G   0% /dev/shm/dev/sda2              7.9G   135M   7.4G   2% /tmp/dev/sdb1               17T   7.0T   9.3T  44% /vol/raid3052/1/dev/sdb2               17T   7.2T   9.1T  45% /vol/raid3052/2/dev/sde1               17T   141M    17T   1% /vol/raid3057/1/dev/sde2               17T    13T   4.1T  75% /vol/raid3057/2/dev/sdd1               17T   141M    17T   1% /vol/raid3058/1/dev/sdd2               17T   141M    17T   1% /vol/raid3058/2/dev/sdc1               17T   141M    17T   1% /vol/raid3059/1/dev/sdc2               17T   141M    17T   1% /vol/raid3059/2

That is 128TB all hooked up to a single machine. It is a total of 16U (1x4U server and 3x4U boxes with just drives in them) each with 24x1.5 TB drives in them in raid6.

Of course this is just used for backups as for any type of data-base like driven stuff you would want to use much higher performance hardware. Of course this 128TB of storage probably only cost less than 20k. I did not order the hardware so I am not sure on the exact cost.

--
Chugging along on 2x 6016/768k DSL Extreme DSL lines and one 6016/768 ATT DSL DIrect line as well as one 16mb/2mb Charter cable line for a combined total of just over 32 meg download and 3 meg up (after overhead). yay!

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 17:31:22 EDT

bigjoesmith posted : (To the original anonymous poster with comments about 7.5Gbyte drives under windows.)
a) Your numbers on the largest logical drive possible in Windows are way off.
b) You can mount drives any number of ways, including off folders, etc. You are not limited to 26 letters.
c) Nobody would organize a VLDB database around a collection of drives, a through z. The storage hierarchy in these very large databases is much more complex and layered.
d) Here's an example of a 1.1 petabyte database running on a Windows platform:

The Panoramic Survey Telescope and Rapid Response System, (Pan-STARRS) is the example offered by Microsoft when it comes to one of the world's largest databases, with SQL Server 2008 at its infrastructure. The Redmond company indicated that the Pan-STARRS architects are building a massive 1.1 petabytes (quadrillion bytes) of disk storage for the celestial imagery which would be provided by four telescopes via 1.4-gigapixel resolution digital cameras. At this point in time, with just a single functioning telescope, Pan-STARRS is producing 1.4 terabytes of imagery each night.

“There are only a handful of databases that large in the world,” revealed Ted Kummert, corporate vice president of the Data and Storage Platform Division at Microsoft.

(»news.softpedia.com/news/Microsof···88.shtml)

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 17:25:08 EDT

cdru posted :

said by DarkLogix:

in windows 64bit the largest logical drive you can have is 7.5TB so 7.5*26(letteres in the alphabet) 195TB
not lets assume not windows
500TB=minimum 334 1.5TB harddrives (but of course add redundancy so) 336 drives if one large array is even possible

Windows NTFS can support 2^32 clusters. At the default cluster size of 4KB, that translates to 16terabytes. Using teh max size of 64kb/cluster, the size goes up to 256TB.

Using other file systems, Windows can theoretically address up to 18exabytes (that's 1000 petabytes or 1,000,000 terabytes).

Windows, just like *nix servers, also aren't limited to just 26 drive letters. You aren't going to put 500TB as a single volume.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 16:46:40 EDT

jchambers28 posted : their goes a waste of money :o

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 16:35:37 EDT

anon posted : There is no such limit in Windows XP on logical drive size limitations. Window XP supports partitioning drives up to 18 exabytes in size.

I doubt this level will be reached in the next ten years.

This is an OLD document, but proves my point:

»support.microsoft.com/kb/100108

First, NTFS has greatly increased the size of files and volumes, so that they can now be up to 2^64 bytes (16 exabytes or 18,446,744,073,709,551,616 bytes). NTFS has also returned to the FAT concept of clusters in order to avoid HPFS problem of a fixed sector size. This was done because Windows NT is a portable operating system and different disk technology is likely to be encountered at some point. Therefore, 512 bytes per sector was viewed as having a large possibility of not always being a good fit for the allocation. This was accomplished by allowing the cluster to be defined as multiples of the hardware's natural allocation size. Finally, in NTFS all filenames are Unicode based, and 8.3 filenames are kept along with long filenames.

You are forgetting that there are other variables for NTFS that determine the maximum partition size:

- BIOS limits
- Controller limitations
- Cluster Size
- Maximum clusters per partition
- NTFS Revision

NTFS 3.1 and newer (Windows XP, 20003 and up) support mount points just like Unix. There is no practical limit to the number of mount points. No extra drive letters required.

The REAL "logical" NTFS volume size limit in Windows XP is 256TB with a cluster size of 64K. Using the more common 4K cluster size the maximum limit is 16TB. More than is needed for most any home user. You can always create multiple mount points on a single drive letter to vastly increase the total addressable space on single computer. There are other file system solutions for Windows that do not have these limits.

And finally, no one uses Windows XP as a server environment anyway so this statement is just silly.

Cluster size Maximum NTFS Volume Size (bytes RAW)
512 2,199,023,255,040 (2TB)
1,024 4,398,046,510,080 (4TB)
2,048 8,796,093,020,160 (8TB)
4,096 17,592,186,040,320 (16TB)
8,192 35,184,372,080,640 (32TB)
16,384 70,368,744,161,280 (64TB)
32,768 140,737,488,322,560 (128TB)
65,536 281,474,976,645,120 (256TB)

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 15:19:15 EDT

fifty nine posted :

said by kamm:

said by fifty nine:

said by kamm:

What FS are you guys using?

Filesystems?

For the Linux boxes it's LVM with ReiserFS. For Solaris we're using vxvm/vxfs.

Ahh so you're not running shared FS. That's the nice and easy way of living. :)

BTW Reiser got 15 to life last year, you know that, right?

Yeah I know. :(

I used to work at a much larger DC where we had Veritas Cluster filesystem, RAC and the like. They had a couple hundred TB of storage. This was a financial company.

Now I'm at a different place that deals primarily in media. It's quite a different environment. A lot of our storage is for video content, including HD streaming and on-demand for cable companies.

Completely different needs.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 14:13:15 EDT

kamm posted :

said by KodiacZiller:

said by DarkLogix:

Ya but would/could a single server have that much capasity?

in windows 64bit the largest logical drive you can have is 7.5TB so 7.5*26(letteres in the alphabet) 195TB
not lets assume not windows
500TB=minimum 334 1.5TB harddrives (but of course add redundancy so) 336 drives if one large array is even possible

It would be easy to see its possible but it would take some planning as to how to orgenize and various other aspects and possibly a need for ZFS

you could actually fit the drives all in one 48U Rack with space to spare

it would be cool to get more details as to how they set it up(although I know how I would)

They won't be using Windows (but knowing cable companies, I wouldn't doubt they'd try).

Like you said, this is a job for a UNIX variant and ZFS (most likely Solaris). ZFS is as scalable as you want it to be.

If you go down the SAN way I doubt this level of number-crunching (analyzing the habits of ~16M subs) can be done without using some sort of shared FS.
--

said by bicker:

Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 14:10:44 EDT

kamm posted :

said by fifty nine:

said by kamm:

What FS are you guys using?

Filesystems?

For the Linux boxes it's LVM with ReiserFS. For Solaris we're using vxvm/vxfs.

Ahh so you're not running shared FS. That's the nice and easy way of living. :)

BTW Reiser got 15 to life last year, you know that, right?
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 13:58:14 EDT

fifty nine posted :

said by kamm:

What FS are you guys using?

Filesystems?

For the Linux boxes it's LVM with ReiserFS. For Solaris we're using vxvm/vxfs.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 13:54:01 EDT

DarkLogix posted : well I guess I'd add a new step 1
1. find some good proformance data

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 13:53:51 EDT

kamm posted : Mine is from here: »www.datadirectnet.com/

I put two Storage Server 2k3 R2 w/ multi-Gb eth on front of it via 4 gig FC, to do FC->LAN and FS/metadata and the rest of the host ports connected to a 16-port 4 gig switch.

What FS are you guys using?
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 13:48:10 EDT

KodiacZiller posted :

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 13:45:35 EDT

kamm posted :

said by DarkLogix:

If I were building it I'd do somthing allong the following

If absolutely insane speed is needed then
HP SAS MSA's on a 10G local net (maybe faster) and that would just be to get the capasity then add however many servers that are needed to handle the processing and load ZFS so I could make the partitions as big as I want

MSA is a joke, seriously, it's a lot slower even than my system here. Also when you need HPC performance you have to forget any eth layer.
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 13:07:34 EDT

jp10558 posted : Come on - what hardware? They're going to use an EMC SAN or the like...

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 12:55:24 EDT

DarkLogix posted : If I were building it I'd do somthing allong the following

If absolutely insane speed is needed then
HP SAS MSA's on a 10G local net (maybe faster) and that would just be to get the capasity then add however many servers that are needed to handle the processing and load ZFS so I could make the partitions as big as I want

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 12:41:05 EDT

fifty nine posted :

said by kamm:

Exactly and beyond the storage size and bandwidth your shared fs licenses ($3-4k per seat) can run into tens of thousands of dollars, depending on your server/access configuration, all this on top of your FC or Infiniband hw costs.

What kind of array did you buy for 200k btw?

We bought a 3Par S400. Nice price, but one of the system controllers had RAM that failed twice. No problems since though.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 12:30:38 EDT

kamm posted :

said by fifty nine:

said by ThrowDemsOut:

said by TCub:

If they have the cash to burn on crap like this.... They must charge their customers an arm and a leg. I have Cox and we have our TV and Internet for about $100 I think. Not too bad for digital cable and WWW.

A 500 terabyte server farm costs next to nothing for even a medium size business. For Comcast the cost is inconsequential and would be easily dwarfed by the extra revenue they would get from the higher prices targeted ads would garner.

Depends on what type of storage. I've worked on everything from small arrays to large EMC Symmetrix SANs. We paid over $100,000 for a 15TB SAN array the other day. This was the low end. The high end gets into the millions of dollars.

If you're using cheap drives in a cheap array it can be cheap. But if you're using enterprise class storage it can get pricey.

Exactly and beyond the storage size and bandwidth your shared fs licenses ($3-4k per seat) can run into tens of thousands of dollars, depending on your server/access configuration, all this on top of your FC or Infiniband hw costs.
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 12:24:59 EDT

kamm posted :

said by TCub:

Actually ya, now that you mention it I'm curious as to what hardware they're using. That's pretty crazy!

I wonder if this is going to bring up any privacy issues? I would guess it won't because google does a similar thing with your searches. Bringing up ad's specific to what you search for or write in a website or blog.

Still all it takes is one person whoes outraged to start a lawsuit.

See my post - it's nothing special: multiple servers using a shared file system (probably StorNext) over FC.
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 12:24:24 EDT

fifty nine posted :

said by ThrowDemsOut:

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 11:53:00 EDT

kamm posted :

said by ThrowDemsOut:

A 500 terabyte server farm costs next to nothing for even a medium size business.

What an ignorant BS - you obviously don't have a clue about HPC storage pricing otherwise you wouldn't call a $500k starter price tag nothing (and besides the iron you'll need the shared fs, the collaboration environment, the data processing apps etc).

FYI: when I bought a 20TB but 1.5GB/s rated system ~18 months ago I paid over $200k - and I got a bargain.

For Comcast the cost is inconsequential and would be easily dwarfed by the extra revenue they would get from the higher prices targeted ads would garner.

Assuming they can fool people into consent - otherwise they will get SUED and handed over their corporate stupid asses for stealing and selling private information.

Yes, my browsing habits, search info etc are MY information thus it's MY PROPERTY - and no parasite monopolies like Comcast can get away with it, mwhahaha.
--

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 11:20:04 EDT

TCub posted : Actually ya, now that you mention it I'm curious as to what hardware they're using. That's pretty crazy!

I wonder if this is going to bring up any privacy issues? I would guess it won't because google does a similar thing with your searches. Bringing up ad's specific to what you search for or write in a website or blog.

Still all it takes is one person whoes outraged to start a lawsuit.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 10:43:38 EDT

DarkLogix posted : Ya but would/could a single server have that much capasity?

in windows 64bit the largest logical drive you can have is 7.5TB so 7.5*26(letteres in the alphabet) 195TB
not lets assume not windows
500TB=minimum 334 1.5TB harddrives (but of course add redundancy so) 336 drives if one large array is even possible

It would be easy to see its possible but it would take some planning as to how to orgenize and various other aspects and possibly a need for ZFS

you could actually fit the drives all in one 48U Rack with space to spare

it would be cool to get more details as to how they set it up(although I know how I would)

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 09:51:43 EDT

TCub posted : Whoah! You changed your avatar.

Anyway. I was considering the entire cost. I sure a server wouldn't be that much but then they have to come up with the idea, create some funky software to monitor EVERY single person watching TV with comcast. Analyse the data, etc etc.

But like Kearnstd said, they're trying to do the Google thing and make a shit ton of money off ad's. So if it works out for them then all this stuff will just pay for itself then.

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 09:43:44 EDT

ThrowDemsOut posted :

Re: money money money money..... MONEY!

Mon, 12 Jan 2009 09:36:37 EDT

Kearnstd posted : everyone also wants to be the next google when it comes to success from ad profits. google nearly prints its money somedays it seems like.
--
[65 Arcanist]Filan(High Elf) Zone: Broadband Reports

money money money money..... MONEY!

Mon, 12 Jan 2009 09:33:13 EDT

TCub posted : If they have the cash to burn on crap like this.... They must charge their customers an arm and a leg. I have Cox and we have our TV and Internet for about $100 I think. Not too bad for digital cable and WWW.
--
And if you go, furious angels will bring you back to me.