Comcast Constructing 500TB User Data Warehouse > money money money money..... MONEY!

reply to TCub

Re: money money money money..... MONEY!

Actually ya, now that you mention it I'm curious as to what hardware they're using. That's pretty crazy!

I wonder if this is going to bring up any privacy issues? I would guess it won't because google does a similar thing with your searches. Bringing up ad's specific to what you search for or write in a website or blog.

Still all it takes is one person whoes outraged to start a lawsuit.

If I were building it I'd do somthing allong the following

If absolutely insane speed is needed then
HP SAS MSA's on a 10G local net (maybe faster) and that would just be to get the capasity then add however many servers that are needed to handle the processing and load ZFS so I could make the partitions as big as I want

reply to TCub
Come on - what hardware? They're going to use an EMC SAN or the like...

reply to DarkLogix

reply to DarkLogix

reply to kamm
well I guess I'd add a new step 1
1. find some good proformance data

reply to KodiacZiller

reply to DarkLogix
There is no such limit in Windows XP on logical drive size limitations. Window XP supports partitioning drives up to 18 exabytes in size.

I doubt this level will be reached in the next ten years.

This is an OLD document, but proves my point:

»support.microsoft.com/kb/100108

First, NTFS has greatly increased the size of files and volumes, so that they can now be up to 2^64 bytes (16 exabytes or 18,446,744,073,709,551,616 bytes). NTFS has also returned to the FAT concept of clusters in order to avoid HPFS problem of a fixed sector size. This was done because Windows NT is a portable operating system and different disk technology is likely to be encountered at some point. Therefore, 512 bytes per sector was viewed as having a large possibility of not always being a good fit for the allocation. This was accomplished by allowing the cluster to be defined as multiples of the hardware's natural allocation size. Finally, in NTFS all filenames are Unicode based, and 8.3 filenames are kept along with long filenames.

You are forgetting that there are other variables for NTFS that determine the maximum partition size:

- BIOS limits
- Controller limitations
- Cluster Size
- Maximum clusters per partition
- NTFS Revision

NTFS 3.1 and newer (Windows XP, 20003 and up) support mount points just like Unix. There is no practical limit to the number of mount points. No extra drive letters required.

The REAL "logical" NTFS volume size limit in Windows XP is 256TB with a cluster size of 64K. Using the more common 4K cluster size the maximum limit is 16TB. More than is needed for most any home user. You can always create multiple mount points on a single drive letter to vastly increase the total addressable space on single computer. There are other file system solutions for Windows that do not have these limits.

And finally, no one uses Windows XP as a server environment anyway so this statement is just silly.

Cluster size Maximum NTFS Volume Size (bytes RAW)
512 2,199,023,255,040 (2TB)
1,024 4,398,046,510,080 (4TB)
2,048 8,796,093,020,160 (8TB)
4,096 17,592,186,040,320 (16TB)
8,192 35,184,372,080,640 (32TB)
16,384 70,368,744,161,280 (64TB)
32,768 140,737,488,322,560 (128TB)
65,536 281,474,976,645,120 (256TB)

reply to kamm
their goes a waste of money

reply to DarkLogix

reply to DarkLogix
(To the original anonymous poster with comments about 7.5Gbyte drives under windows.)
a) Your numbers on the largest logical drive possible in Windows are way off.
b) You can mount drives any number of ways, including off folders, etc. You are not limited to 26 letters.
c) Nobody would organize a VLDB database around a collection of drives, a through z. The storage hierarchy in these very large databases is much more complex and layered.
d) Here's an example of a 1.1 petabyte database running on a Windows platform:

The Panoramic Survey Telescope and Rapid Response System, (Pan-STARRS) is the example offered by Microsoft when it comes to one of the world's largest databases, with SQL Server 2008 at its infrastructure. The Redmond company indicated that the Pan-STARRS architects are building a massive 1.1 petabytes (quadrillion bytes) of disk storage for the celestial imagery which would be provided by four telescopes via 1.4-gigapixel resolution digital cameras. At this point in time, with just a single functioning telescope, Pan-STARRS is producing 1.4 terabytes of imagery each night.

“There are only a handful of databases that large in the world,” revealed Ted Kummert, corporate vice president of the Data and Storage Platform Division at Microsoft.

reply to DarkLogix
Odds are they would use some sort of NAS setup that multiple servers access. 500TB is not that big of a deal. Take this server at work for example:

# df -Hl
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sda1               30G   5.1G    24G  18% /
tmpfs                  2.1G      0   2.1G   0% /lib/init/rw
udev                    11M    82k    11M   1% /dev
tmpfs                  2.1G      0   2.1G   0% /dev/shm
/dev/sda2              7.9G   135M   7.4G   2% /tmp
/dev/sdb1               17T   7.0T   9.3T  44% /vol/raid3052/1
/dev/sdb2               17T   7.2T   9.1T  45% /vol/raid3052/2
/dev/sde1               17T   141M    17T   1% /vol/raid3057/1
/dev/sde2               17T    13T   4.1T  75% /vol/raid3057/2
/dev/sdd1               17T   141M    17T   1% /vol/raid3058/1
/dev/sdd2               17T   141M    17T   1% /vol/raid3058/2
/dev/sdc1               17T   141M    17T   1% /vol/raid3059/1
/dev/sdc2               17T   141M    17T   1% /vol/raid3059/2