| reply to knightmb
Re: FBI need to find more important things to do said by knightmb:I can understand the FBI doing an investigation where they bring in equipment and record something for some user using the courts to get the wiretap, that I'm fine with. CALEA already requires that US ISPs have snooping equipment in place to allow any law enforcement agency to view all your Internet traffic, not just your IP addresses and URLs.
This new thing with logs is just to get access to IP addresses and URLs for 2 yr old activity that they didn't have a subpoena for at the time. |
|
 knightmbEverybody Lies
join:2003-12-01
Franklin, TN | said by Supervisor:said by knightmb:I can understand the FBI doing an investigation where they bring in equipment and record something for some user using the courts to get the wiretap, that I'm fine with. CALEA already requires that US ISPs have snooping equipment in place to allow any law enforcement agency to view all your Internet traffic, not just your IP addresses and URLs. This new thing with logs is just to get access to IP addresses and URLs for 2 yr old activity that they didn't have a subpoena for at the time. Already worked with the FCC about that, a lot didn't apply to us. Technically we do have snooping equipment, it's called a 5 port hub that sits in between everything, they are welcome to come plug in any equipment they want to snoop provided they have a court order to do so. 
--
Fight Insight Ready (Was NebuAD) and the like:
Click Here to pollute their data |
|
| You must not pass very much traffic. I sure as heck wouldn't let some dinky 5 port hub be my single point of failure. |
|
knightmbEverybody Lies
join:2003-12-01
Franklin, TN
1 edit | reply to knightmb
Fun with Math Here's another reason why this wouldn't be practical.
I've uploaded a simple text with 10 URLs recorded. It's the most basic info, time/date, IP, and URL
Now that is just 10 URL and the file is already 1,163 bytes
That won't included the URL for every other file that is accessed, because what good is it to record that you visit »somebadplace.gov/ without logging all the files you got from the site? Well that's additional "lines" in the log file.
Have you seen those *long* URL that you get with Google or other sites when you start going deep into them, yeah that's more space being burned up.
Ok, right now (I'm a small ISP), I see about 20 to 30 URL coming through a second from customers.
Ok, so I'm using about 3,000 bytes per second of recording, figured in more since I will also have to record every *other* place that URL goes (graphics, CSS, javascript, etc.)
So now I'm recording 5,000 bytes per second.
Doesn't sound too bad? 5,000 bytes X 60 seconds = 30,000 bytes per minute.
30,000 bytes per minute X 60 minutes = 1,800,000 bytes per hour
1,800,000 X 24 hours/day = 43,200,000 bytes per day
43,200,000 bytes per day X 365 days = 15,768,000,000 bytes per year
15,768,000,000 X 2 years = 31,536,000,000 bytes per 2 years
That's 30,075.0732 megabytes
29.37 gigabytes
That's being generated by 15 users (provided they are only visiting a few sites).
If you have people visiting e-bay, google, etc. They will generate a ton more URL data, but I'm trying to be fair.
So if 15 users can generate 29 gigabytes a year (very low estimate of course), then you can only imagine what hundreds or even thousands of users would generate.
I can see that Hard Drive companies that produce multi-terabyte drives will be getting a lot of business if this every passes. 
--
Fight Insight Ready (Was NebuAD) and the like:
Click Here to pollute their data |
|
knightmbEverybody Lies
join:2003-12-01
Franklin, TN | reply to battleop
Re: FBI need to find more important things to do said by battleop:You must not pass very much traffic. I sure as heck wouldn't let some dinky 5 port hub be my single point of failure. Of course not, that's why we have 3
--
Fight Insight Ready (Was NebuAD) and the like:
Click Here to pollute their data |
|
| reply to knightmb
Re: Fun with Math Whats the name of your ISP company? |
|
 SLDPremium
join:2002-04-17
San Francisco, CA | reply to knightmb
One of my web servers generates 2-4GB of log files per day! |
|
Romney2012Defeat Obama 2012-Chg we can believe inPremium
join:2002-03-03
USA
kudos:4
1 edit | reply to knightmb
And what if you don't need URLs(since that isn't in the request to save for 2 years) and only have to collect date, time, and IP address? That would be much less data. |
|
patcat88
join:2002-04-05
Jamaica, NY
kudos:1 | reply to knightmb
Well, with metered billing, you be paying for your ISP to buy a HD for data retention every month to store your traffic. It works perfectly. |
|
 RARPSL
join:1999-12-08
Suffern, NY | said by patcat88:Well, with metered billing, you be paying for your ISP to buy a HD for data retention every month to store your traffic. It works perfectly. Metered Billing only counts how much you use NOT what you are using it for. It just updates a counter associated with your account.
IOW: It is like the trip meter on you car's odometer - It shows how far you have driven since it was last reset, NOT where you were driving (which would need a "Black Box" with a GPS unit). |
|
patcat88
join:2002-04-05
Jamaica, NY
kudos:1
1 edit | But when you pay $40 a month for 80 GB, $5 went to pay for HD that will the 80GB you moved that month, full packet capture. If you use less than your bucket, cheaper for the ISP to store your data traffic. If if you move 250GB a month, about $20 of your monthly bill went to buy HD space to store your 250GB for 2 years. |
|
chimera
join:2009-06-09
Washington, DC | reply to knightmb
Don't forget that for large ISPs most 1TB drives (which run at 7200 RPM) will be too slow to actually keep up with the data, and since they would need failure protection everything would need a RAID setup adding even more cost per GB. |
|
 ReformCRTCSupport Your Independent ISP
join:2004-03-07
Canada | reply to knightmb
Re: FBI need to find more important things to do Zinnnngage! |
|
| reply to chimera
Re: Fun with Math that a moot point. they would never consider a non-raid like solution to begin with. i'm 90% sure that servers only use RAID. they aren't stupid enough to only store it on a couple 1TB hard drives with no backups.
also, don't underestimate the speed of high density drives. |
|
chimera
join:2009-06-09
Washington, DC Reviews:
 ·Comcast
| I'm not trying to underestimate the speed, but this sort of setup would most likely be stored in a SQL database on an external SAN for large ISPs, and when you are writing THAT many entries the latency difference between 7200RPM and 15000RPM makes a difference. My point is the cost for storage shouldn't be thought of as just being the cost of storing this data on a desktop because all of the extra costs to ensure that it operates safely and smoothly REALLY adds up. |
|
