Search:  

 
 
   All ForumsHot TopicsGallery






how-to block ads


 
Forums » Comcast Constructing 500TB User Data Warehouse » money money money money..... MONEY!
Search Topic:
Uniqs:
892
Share Topic:
RSS topic:
toggle:
flat / full
normal / watch
Post a:
Post a:
500TB doesn't sound like very much »
« Where are the privacy whackos?  
page: 1 · 2
AuthorAll Replies


Eat Me

join:2002-09-25
Sussex, NJ
·PenTeleData
·Future Nine Corpor..
·VOIPo
·Vonage

reply to kamm
Re: money money money money..... MONEY!

said by kamm See Profile :

said by Eat Me See Profile :

said by kamm See Profile :

What FS are you guys using?
Filesystems?

For the Linux boxes it's LVM with ReiserFS. For Solaris we're using vxvm/vxfs.
Ahh so you're not running shared FS. That's the nice and easy way of living.

BTW Reiser got 15 to life last year, you know that, right?
Yeah I know.

I used to work at a much larger DC where we had Veritas Cluster filesystem, RAC and the like. They had a couple hundred TB of storage. This was a financial company.

Now I'm at a different place that deals primarily in media. It's quite a different environment. A lot of our storage is for video content, including HD streaming and on-demand for cable companies.

Completely different needs.


moon1234

@tds.net

reply to DarkLogix
There is no such limit in Windows XP on logical drive size limitations. Window XP supports partitioning drives up to 18 exabytes in size.

I doubt this level will be reached in the next ten years.

This is an OLD document, but proves my point:

»support.microsoft.com/kb/100108

First, NTFS has greatly increased the size of files and volumes, so that they can now be up to 2^64 bytes (16 exabytes or 18,446,744,073,709,551,616 bytes). NTFS has also returned to the FAT concept of clusters in order to avoid HPFS problem of a fixed sector size. This was done because Windows NT is a portable operating system and different disk technology is likely to be encountered at some point. Therefore, 512 bytes per sector was viewed as having a large possibility of not always being a good fit for the allocation. This was accomplished by allowing the cluster to be defined as multiples of the hardware's natural allocation size. Finally, in NTFS all filenames are Unicode based, and 8.3 filenames are kept along with long filenames.

You are forgetting that there are other variables for NTFS that determine the maximum partition size:

- BIOS limits
- Controller limitations
- Cluster Size
- Maximum clusters per partition
- NTFS Revision

NTFS 3.1 and newer (Windows XP, 20003 and up) support mount points just like Unix. There is no practical limit to the number of mount points. No extra drive letters required.

The REAL "logical" NTFS volume size limit in Windows XP is 256TB with a cluster size of 64K. Using the more common 4K cluster size the maximum limit is 16TB. More than is needed for most any home user. You can always create multiple mount points on a single drive letter to vastly increase the total addressable space on single computer. There are other file system solutions for Windows that do not have these limits.

And finally, no one uses Windows XP as a server environment anyway so this statement is just silly.

Cluster size Maximum NTFS Volume Size (bytes RAW)
512 2,199,023,255,040 (2TB)
1,024 4,398,046,510,080 (4TB)
2,048 8,796,093,020,160 (8TB)
4,096 17,592,186,040,320 (16TB)
8,192 35,184,372,080,640 (32TB)
16,384 70,368,744,161,280 (64TB)
32,768 140,737,488,322,560 (128TB)
65,536 281,474,976,645,120 (256TB)


jchambers28

join:2007-05-12
Alma, AR
reply to kamm
their goes a waste of money


cdru
Go Colts
Premium,MVM
join:2003-05-14
Fort Wayne, IN

reply to DarkLogix
said by DarkLogix See Profile :

in windows 64bit the largest logical drive you can have is 7.5TB so 7.5*26(letteres in the alphabet) 195TB
not lets assume not windows
500TB=minimum 334 1.5TB harddrives (but of course add redundancy so) 336 drives if one large array is even possible
Windows NTFS can support 2^32 clusters. At the default cluster size of 4KB, that translates to 16terabytes. Using teh max size of 64kb/cluster, the size goes up to 256TB.

Using other file systems, Windows can theoretically address up to 18exabytes (that's 1000 petabytes or 1,000,000 terabytes).

Windows, just like *nix servers, also aren't limited to just 26 drive letters. You aren't going to put 500TB as a single volume.

bigjoesmith

join:2000-11-21
Peoria, IL
·Cable Onda


1 edit
reply to DarkLogix
(To the original anonymous poster with comments about 7.5Gbyte drives under windows.)
a) Your numbers on the largest logical drive possible in Windows are way off.
b) You can mount drives any number of ways, including off folders, etc. You are not limited to 26 letters.
c) Nobody would organize a VLDB database around a collection of drives, a through z. The storage hierarchy in these very large databases is much more complex and layered.
d) Here's an example of a 1.1 petabyte database running on a Windows platform:
The Panoramic Survey Telescope and Rapid Response System, (Pan-STARRS) is the example offered by Microsoft when it comes to one of the world's largest databases, with SQL Server 2008 at its infrastructure. The Redmond company indicated that the Pan-STARRS architects are building a massive 1.1 petabytes (quadrillion bytes) of disk storage for the celestial imagery which would be provided by four telescopes via 1.4-gigapixel resolution digital cameras. At this point in time, with just a single functioning telescope, Pan-STARRS is producing 1.4 terabytes of imagery each night.

“There are only a handful of databases that large in the world,” revealed Ted Kummert, corporate vice president of the Data and Storage Platform Division at Microsoft.
news.softpedia.com/news/Microsof···88.shtml)


houkouonchi

join:2002-07-22
Corona, CA
clubs:
·AT&T U-Verse
·DSL EXTREME
·OCN
·Pacific Bell - SBC
·Charter Pipeline

reply to DarkLogix
Odds are they would use some sort of NAS setup that multiple servers access. 500TB is not that big of a deal. Take this server at work for example:


That is 128TB all hooked up to a single machine. It is a total of 16U (1x4U server and 3x4U boxes with just drives in them) each with 24x1.5 TB drives in them in raid6.

Of course this is just used for backups as for any type of data-base like driven stuff you would want to use much higher performance hardware. Of course this 128TB of storage probably only cost less than 20k. I did not order the hardware so I am not sure on the exact cost.

--
Chugging along on 2x 6016/768k DSL Extreme DSL lines and one 6016/768 ATT DSL DIrect line as well as one 16mb/2mb Charter cable line for a combined total of just over 32 meg download and 3 meg up (after overhead). yay!


kamm

join:2001-02-14
Brooklyn, NY

1 edit
reply to TKJunkMail

...


kamm

join:2001-02-14
Brooklyn, NY
·T-Mobile US

reply to Eat Me
said by Eat Me See Profile :

Yeah I know.

I used to work at a much larger DC where we had Veritas Cluster filesystem, RAC and the like. They had a couple hundred TB of storage. This was a financial company.

Now I'm at a different place that deals primarily in media. It's quite a different environment. A lot of our storage is for video content, including HD streaming and on-demand for cable companies.

Completely different needs.
Not necessarily: we deal with high-res media as well, scientific stuff but the main SAN users are usually the 3d/fx guys.
--
[BQUOTE=[user=bicker]]Waaaa waaaa waaaa. You just want what you want and don't care to factor in what is right or true. Your perspectives are un-American, and deserve far more ridicule than I'm prepared to pile on them.
[/BQUOTE]
-
Forums » Comcast Constructing 500TB User Data Warehouse500TB doesn't sound like very much »
« Where are the privacy whackos?  
page: 1 · 2


Thursday, 26-Nov 17:11:37 Terms of Use | Privacy Policy | Hosting by www.nac.net - DSL,Hosting & Co-lo | feedback | contact
over 10 years online! © 1999-2009 dslreports.com.
page compression OFF
Most commented news this week
· [109] New AT&T Ad Campaign Hits Back At Verizon
· [106] Time Warner Cable Fires Broadside At Broadcasters
· [95] Apple Joins AT&T Verizon Snark Fest
· [87] New Bill Takes Aim At Higher Verizon ETFs
· [69] TiVo Sees Record Customer Losses
· [60] In-Flight Internet Headed For Bumpy Landing?
· [41] Thanksgiving Open Thread
· [37] ICANN Slams DNS Redirection
· [34] Senators Want ACTA Made Public
· [34] Despite Billions In USF Fees, U.S. Libraries Lack Bandwidth
Most people now reading
· I'll Just Unplug That... [No, I Will Not Fix Your #@$!! Computer]
· Newegg Black Friday Sale started [Users Find Hot Deals]
· 3.x Feral Druid - Bear Tanking Guide [World of Warcraft]
· Connecting to Google Voice Via SIP [VOIP Tech Chat]
· Slow speeds in the evenings [TekSavvy]
· Windows 7 boot manager editing questions [Microsoft Help]
· Whats the big deal about being "Old School"....? [World of Warcraft]
· HOW-TO: QoS and Tomato (fixes "choppy voice") [MagicJack]
· What is the spell hit cap for a lvl 80 full arcane spec mage [World of Warcraft]
· Not strictly "Home" related - but WOW anyways... [Home Repair & Improvement]