Sonic CEO Discusses The "Five Levels of ISP Evil" → Disk is cheap and the data is immutable...

Can we define "costlier"? Even if it's exceedingly large volumes of data, they can use tiered storage, compress the hell out of it and move it off-line onto something cheap. Off-line also adds a security barrier. Furthermore, if they separate url logs, DHCP logs and customer data (i.e. mac address), that's a second security barrier. Access logs are anonymous until cross-referenced with the DHCP logs and customer data. Finally, off-line retention can be encrypted which adds another security barrier.

Of course if the customer is at risk because the ISP is tempted... If you cannot trust the palace guards perhaps it's time to flee.

Let's say you're streaming data at 5 Gbps (probably the size of Sonic.net's network) on average. That's 1.62 PB of data per month. To cope with the speed at which data is being collected, they'll need to have a rack of Backblaze Pods (cheapest storage system out there) running, at a cost of a few hundred thousand dollars.

Of course, URL logs may be a bit smaller. Maybe one Pod (135TB) gets used per month (remember, lots of AJAX stuff going on). That's still thousands of dollars per month that could be used to add a few more fiber customers, or to add another gigabit or two of network transit...

Costlier means it would cost more money.

And more than likely make us all a lot less safe.

I didn't realize the government is asking them to store ALL the data. I thought it was just the access logs.

OK...I'll reword it just for you...

How much does this raise my monthly bill if they have to comlpy?

Whatever your ISP decides it costs to hire people to do this, buy the equipment, and establish new protocols for compliance and various legal fee's for the drove of Patriot act requests.

Take that cost and add at least 20%

I'm unsure myself. Maybe someone can come on here and clarify

Regulations are nothing new. I work for a grocery retailer and years ago we were forced to implement country of origin labeling (COOL) in response to the mad cow scare. This required changing package labels and retaining invoice/shipment/PO records in case the worst happened. We're also a wholesaler and our customers looked to us to keep their invoices in electronic format, which we did. Of course now COOL turns out to be a pretty good idea since purposely tainted food supplies are possible with the uptick in terrorism. Did it cost money to implement? Absolutely. Did it run anyone out of business? Not to my knowledge. Given the seasonal volatility of various supply costs and the length of time we had to implement compliance, I doubt consumers noticed whatever tiny price increase that might have been passed on to them.

It's easy to say something costs more but that's like taking a poll and making a statement about how America thinks. Without the supporting methods of data collection (how many samples, what confidence interval, how random is the sample), the results have zero context. In fact I'm sure people have repeated polls until they get the answer they want to serve dubious purposes.

In an era where minorities complain about skewed results on standardized tests because the tests might have context that cannot be universally understood, how can we be so arrogant as to believe those polled even understand the issues or the question?

I'm sure retention will cost more but if we cannot put context to that cost, there's simply not enough information to form an opinion and take sides. Anyone who does is ignorantly coming to a conclusion that they cannot support.

its not about the size and tier of the storage -- its just as much about the performance of said architecture.
i work in the network architecture/design field and routinely have to engage technical architects around performance requirements for the virtualized compute environments and storage arrays. there are many "gotchas" around each of the major storage vendors and even down to the drives and protocols/access methods in use. it comes down to having a storage architect designing the entire array and infrastructure to handle that.

plus -- you have to start looking at the infrastructure to even capture or log this data. inline taps on the network are great -- but are you simply going to look at them at the egress point(s) on the network? can you find a device or cluster of devices to archive and log this data -- then dump it onto the storage network without losing any information? if you can't -- how do you spread this load out so that it is workable? can you fit it within your existing architecture. what about capex for the gear and redesign work? what about opex for the spin-up, training, and management of these new components in the network?

its not like a home network where you just throw another drive in the nas and go. there is a lot of high-performance infrastructure design needs that need to be met to fit this into a network. there isn't a 'one size fits all' solution -- nor is it something that is easy to overcome. dane has it right -- this is *costly*.

q.

In whole I would agree with you, but regardless it shouldnt be done.

To implement the scope of this information gathering in hopes of catching maybe 1000 nonviolent criminals and 2-3 violent ones a year is ridiculous at any cost.

The only logs that should be required are DHCP to MAC logs so they can see who had an IP at any one time. And the only time that information should ever be used is when there are crimes against a person that has involved physical or mental harm.

I thought that was all that was required. As I said in another reply, I didn't know they required ALL the data to be kept. That's ridiculous and it seems like an unreasonable request that isn't even feasible. Are we sure they want ISPs to keep ALL the data and not just the logs for 18 months?

An excerpt from the bill:

(a) IN GENERAL. Section 2703 of title 18, United States Code, is amended by adding at the end the following:
(h) RETENTION OF CERTAIN RECORDS.—A provider of an electronic communication service or remote computing service shall retain for a period of at least 18 months the temporarily assigned network addresses the service assigns to each account, unless that address is transmitted by radio communication (as defined in section 3 of the Communications Act of 1934).

They don't have to capture all the data and it doesn't appear to apply to wireless providers (assuming mobile, not fixed). Mobile wireless would be a nightmare since the IP logs are probably a mess with folks moving from place to place.

Are you honestly going to tell me they cannot retain DHCP logs?!?!?! According to this, they don't even have to keep URL access logs. In my opinion, the ISPs are whining about nothing. And, as usual, information is power and the uninformed crowd has taken sides and is crowing about NOTHING!

What's ridiculous is that I posed a question, "define costlier". I get tripe responses like "it would cost more" or folks going off the deep end talking about the architectural complexities and costs of storing hundreds of terabytes of data per month.

Now I'll be the first to apologize if I missed something in the bill that forces ISPs to keep the DATA and the logs. Keeping the DATA is unreasonable and if true, everyone who said it was crazy is correct. However, I read the ENTIRE bill and the only part that I can find that applies to ISPs seems like a VERY REASONABLE request from our government. ISPs will incur practically NO additional cost to implement regardless of size.

I read the bill. It's just DHCP access logs.