Search similar:
|
|
uniqs 1857 |
|
|
|
maartenaElmo Premium Member join:2002-05-10 Orange, CA |
to Black Box
Re: Backup solution ideasQuestion: How much data do you CHANGE every day? I know a cloud backup seems daunting, and with a 1 Mbit/s upstream it will probably take 2 months to upload 6 TB to a cloud, but if the data only changes at the rate of around 1 GB a day, a cloud backup may be the way to go. I "only" have 350 GB on my Carbonite backup, which is my crucial data such as digital photo's, documents, and things I know I can't get back. Everything else is pretty much replaceable, such as my media collection, all of which can be redownloaded if I still want it at that time. My thing was a house fire. I have a colleague who lost everything house fire. They were insured, and got a new house built, a new car (burned out in garage too), new furniture, new computers, everything. But all of that didn't matter. He and his family were out for the weekend, and now the earliest photo he has of his daughters is starting at around 8 years old, the photo's taken that weekend. He lost all of his pictures from their childhood, and all of their videos. Did he have a backup? Yes, it was melted to a crisp right next to the black goo that once was a computer.... After that, and after having to deal with someone who got their laptop AND external backup drive stolen after a burglary, I decided that cloud backups aren't so bad. And they have saved me. Last December I had not 1 but 2 hard drives fail on me in the same week... and I was dumb enough to think: Ah, no problem I have the OTHER drive still, i'll buy a replacement after work some time this week.... got busy, and 4 days later a 2nd drive crashed. Hard. Murphy's law had struck. This wasn't a raid setup, but 2 drives crashing in a raid setup pretty much also means unrecoverable, unless there is more then 8 drives in the setup, and 2 drives are setup as hot spares. Luckily for me I had those 350 Gb saved in the cloud, and I did not lose my photos, did not lose my outlook PST file, did not lose my family videos (including all my childhood stuff converted from 8 mm film reels), and I did not lose all my documents with a history dating back to the 90s. (and yes, there is a lot, including my wife's unpublished poetry that she wrote back then, which we still want to save). So I have become an advocate of online backups, regardless of how much data it actually is. Some backup providers allow you to deliver an initial backup on a hard drive so you don't have to upload for 2 months just to get it there, but if your data doesn't change more then around 50 GB a month or so.... I'd say it is so worth it. Peace of mind. I have it now. | | |
Not too much data is changing daily. rsync would work just fine (already does). The issue is not bandwidth, is compromised credentials.
For the most critical data (apart of the offline backups) I have an online backup literally on the other side of the Atlantic. Thing is, if my credentials are compromised, what could stop a cracker to send to the backup machine the commands I mentioned in my prior post? If your cloud credentials are compromised what stops a cracker to delete all backup data or delete your backup account altogether? There is a lot of money to be made in data ransom.
I understand very well the concept of online backup (on site/off site/cloud). I already have some form of online backup. I use it to recover from hardware failures and it works. The issue is I don't trust myself to make the online backup impregnable if my credentials are compromised. As simple as that. I don't have neither the time nor the hardware to do it. There is no one else to fall back to.
I get it. Pretty much everyone here relies on online backups, and that's fine. Please stop trying to convince me to depend exclusively on online backups too. That's what I have now and I consider that is not enough. I want to be protected against a scenario described above (all machines and accounts compromised). Until now only a solution proposed in jest (back up 4-8 TB data on DVD/Bluray) kind of works.
Offline backup stored offsite would protect very nicely against fire and burglary, so my solution would for your needs too. Your friends made the mistake of storing the only backup onsite. Scenario number one for disaster recovery: what if I find a crater in the ground instead of the shop in the morning?
The solutions I am considering currently are HDD docks and and LTO6 tapes, but I'm not to fond on blowing 2-3k bucks on it. I did not want to mention them, to see what ideas other people throw around and if anyone proposes them too. Anyone with other ideas? I would like to have more alternatives before pulling the trigger. | | |
to Black Box
said by Black Box:Actually yes. You just have to repeat it for mirrors and RAIDZx, but you can reliably nuke the pool. Updated code to completely nuke a system:
# Run as root NOT!
for d in /dev/rdsk/c*t*d* do
cat /dev /zero > $d &
done
That is completely destroying a pool by simultaneously trying to write to all devices. A very simple 'zpool destroy' would be much more effective, and if you have access to implement something like that you can much more easily do a 'zpool destroy'. I'm not quite sure the point you are making here. At that point you've achieved your catastrophic failure scenario that you are trying to protect against already. There is nothing further constructive to come of this discussion since all you seem to want to do is continue to find new scenarios to throw up. Any solution anyone proposes would be exposed to similar issues. You've been presented with several options by several individuals. It is up to you to now figure out how to proceed. Seriously - you really need to define your true requirements first. If you keep changing these while trying to identify a solution you'll never have success. How do they not cover your scenario? The solutions proposed rely on online replicas/snapshots. If it is online in my scenario it is as good as gone. You need to copy something. The snapshot is a point in time copy of the data and facilitates an efficient native mechanism for transfer to another system, only transferring changed blocks and fully protected by the same checksums used for the storage itself. It is not intended to be used as your actual backup solution (I've noted that already) If you've reached the point where you can't trust the snapshot for the short amount of time you need it to send your stream then you can't trust the underlying data either. If this is a real concern you really need to be using exclusively write-once media. I don't think you really understand ZFS, its value, or its features/functionality. As I noted before if you really don't trust it then don't use it. Again - you really need to identify your true requirements first. Insisting on shooting holes in suggestions made without properly defining your requirements Your statement misses one key word: "online". Online backups can be destroyed, offline (as in sitting in a drawer somewhere, even if writable) cannot. I cannot spend the time and money to properly lock down an online system so it won't go down if my credentials are compromised. Optical media could work, just it is not scalable. It is hard to manage terabytes of data dumped on myriad of discs. You obviously did not read the suggestions made by me or several others here and are now wasting our time. The target for your backups would not be "online" except when you are actually performing your copy. "online" would only need to be connectivity to the host you are backing up and nowhere else. If you want to make a backup without ever having any form of connectivity to your target then you are reaching into science fiction. Again - shooting arbitrary holes into solutions simply for the sake of argument is not productive and (based on a private conversation I have had) has actually turned others away from offering suggestions. I too have no interest in trying to help further for the same reason. | | Shady Bimmer |
to Black Box
said by Black Box:I get it. Pretty much everyone here relies on online backups, and that's fine. Please stop trying to convince me to depend exclusively on online backups too. I don't think you have the same definition of 'online' that everyone else has. You've been presented with several options that provide an 'offline' backup, including options to make this 'offsite' as well. You seem to keep wanting to shoot holes in those, even though you consider 'rsync' OK. If your concern is over compromised credentials then that is a separate issue that can be addressed with the multiple various options offered already. You'll need to do your own research on this since you seem to only want to make swiss cheese out of everyone else's suggestions rather than have constructive discussion. | | |
to Shady Bimmer
said by Shady Bimmer:That is completely destroying a pool by simultaneously trying to write to all devices. A very simple 'zpool destroy' would be much more effective, and if you have access to implement something like that you can much more easily do a 'zpool destroy' The example was not how on how to destroy a zpool (btw zpool import -D can restore destroyed zpools). It was about how easy is to cause data loss if the system is accessible online. I define online as accessible either locally or over a network connection (regardless of clear text/encrypted/VPN or permanent/on demand). Offline would be something that you need to perform a physical action to bring it online (e.g. bridging air gap/insert media in socket), again local or over the network. said by Shady Bimmer:I don't think you really understand ZFS, its value, or its features/functionality. As I noted before if you really don't trust it then don't use it. I chose ZFS for the normal usage and I am pretty happy with it. Things fall apart only when a malicious intruder comes into play. I demonstrated above how easy is for a malicious intruder to destroy any file system, ZFS included. No finesse needed, brute force works too. I stated in my first post: said by Black Box:My current "backup" solution is a combination of data replication amongst machines and USB key/DVD+RW copies for the most important files rsync covers online replication and that's already handled. Less important data is replicated, more important data is also backed up on removable media (however not scalable). I am looking for a proper backup solution, media rotation, offsite storage and all. The disaster scenario has not changed: Everything accessible either locally or remote (permanently or on-demand) is maliciously corrupted/destroyed. The recovery starts by physically turning off and disconnecting the compromised systems (to prevent reinfection for the rebuilt systems), connect back systems only after nuke and pave (old disk zeroed, partitioned and OS installed afresh). Then the backup(s) are brought online for data recovery, connected only to fresh systems. To perform the backups the media would be brought online only for the time needed to write the backup then removed and then transported to a safe location. Multiple rotating sets of media would be better. Creating replacement credentials for those compromised would be part of the nuke and pave step. I am not concerned with that right now. | | |
All I can say once again is to read the responses provided. You clearly have not and clearly only have an interest in arguing with others.
You've been presented with very viable options that meet your requirements and if those don't suffice you've also been provided with options for the absolute worst case. Beyond what has been suggested only Science Fiction offers better options.
Seriously. Please stop the arguing and actually read what multiple people have offered. | | dennismurphyPut me on hold? I'll put YOU on hold Premium Member join:2002-11-19 Parsippany, NJ |
to Black Box
said by Black Box:I get it. Pretty much everyone here relies on online backups, and that's fine. Please stop trying to convince me to depend exclusively on online backups too. That's what I have now and I consider that is not enough. I want to be protected against a scenario described above (all machines and accounts compromised). Until now only a solution proposed in jest (back up 4-8 TB data on DVD/Bluray) kind of works. Look, it's a question of how important your data is. The more threats you try to solve for, the more it's going to cost. For me, the NAS to NAS backup/VPN solution is the best compromise I've found. It protects me against accidental data loss, as well as physical loss (i.e. my house burns down.) If I wanted to get extreme, I'd add the WORM USB key I mentioned and put it on the remote site, then have a SHA key written to it so I can verify the data is as I wrote it. Now, the other thing I've done is to occasionally (define that however you'd like) plug a USB drive into the NAS, perform a full backup, and then transport said USB disk to my safe deposit box. That way, in the event the NAS thing completely melts down and I lose everything, I can recover back to the point-in-time when I made the backup. My data is VERY important to me. But as said above, it doesn't change much. Recovering to, say, even 3 months ago, gives me back 95% of my data. Combined with the "live" NAS-to-NAS backups, the periodic, manual backup to the safe deposit box is about as many situations as I care to protect against. Frankly, if that's still not sufficient for you, drop the coin and buy an SL8500. Not much more I can tell you. Sign a contract with Iron Mountain and store your tapes in their vault. Encrypted. What else do you want me to say? If you want to solve enterprise-grade problems, that's going to cost enterprise-grade money. | | |
Well, the thread was pretty good in covering online backups. Not much so on offline backups. The whole point was to avoid a SL8500 solution, while still covering all the data. USB keys/Optical media scale only that much. I've already hit the practical limits. said by dennismurphy:plug a USB drive into the NAS, perform a full backup Where can I find an 8TB USB? Guess that my solution will be based on HDD docks. HDDs will be rotated and stored safely. I just have to figure out in which configuration. If anyone has a better solution or implementation hints I would still like to hear it. | | dennismurphyPut me on hold? I'll put YOU on hold Premium Member join:2002-11-19 Parsippany, NJ |
Where can I find an 8TB USB?
Guess that my solution will be based on HDD docks. HDDs will be rotated and stored safely. I just have to figure out in which configuration.
If anyone has a better solution or implementation hints I would still like to hear it. » www.amazon.com/Seagate-B ··· ords=8tbYou didn't even try. Your google-fu is weak. | | |
Indeed it is. I have to admit I didn't follow the latest developments. According to google it was released only mid January. I had no idea such thing existed already, so i didn't look for it.
BTW, that is more in line with the idea of a dock, I would likely buy an USB dock and put bare 8TB drives (for rotation) in it. Or maybe find an eSATA dock (does it even exist? everything seems to be USB3.0) and do the same. BTW, the server doesn't have an available USB3.0 or eSATA port, I'll have to buy an card for that anyway.
To nitpick, that's not technically an USB key as implied in the conversation above, but thanks anyway. | | maartenaElmo Premium Member join:2002-05-10 Orange, CA |
to Black Box
said by Black Box:I understand very well the concept of online backup (on site/off site/cloud). I already have some form of online backup. I use it to recover from hardware failures and it works. The issue is I don't trust myself to make the online backup impregnable if my credentials are compromised. As simple as that. I don't have neither the time nor the hardware to do it. There is no one else to fall back to. What you do is you make a passphrase that is unbreakable. (30 characters or more and even the NSA's datacenters would need 200 years to crack it) You setup the backup, you write down the username and passphrase, and you put it in a safety deposit box in your bank. The backup is setup, and will run for many years until you need to re-install the device that does the backup. Then, if you don't remember the passphrase you go to the bank to pick it up for re-install or restore. And you really only need that safety deposit box in case of a fire, for any daily restore/backup duties and/or re-installing it, a note on your monitor has the same security level as the tapes and/or hard drives that are most likely in the same room. Since i had a few drives crash within the same WEEK last year, i'd like to take the hardware failure thing out of the equation. Let someone else worry about daily management of datacenter storage. The data is stored encrypted in a manner no one else can get to it but you, and it just means I can forget the backup, and don't have to do weekly off-site carrying to ensure my latest set of digital pictures is included. You should also calculate your investment over time. My online backup is $60 a year for unlimited, but even "unlimited" providers (I have 350 GB on there) will at some point cry about too much data, so you will probably have to spend around $100 a year. That said, hard drives have an average lifespan of around 5 years, so in 5 years from now you will need to replace your current setup most likely. LTO tapes also have a lifespan, but it depends on how often it is used. If a tape is used once a week, it needs to be replaced after 4 years. If you use the tape for offline storage, you have about 15 years. Tape drives themselves last 5-10 years, but maintenance can be done on them. | | |
Remember: original disaster scenario is all online machines compromised. Don't pick on the detail of the possible compromise (I can adjust them), concentrate on the result. So, I have this backup server with NSA proof passwords, stored in the bank box only. I assume that you mean that none of the other machines have access to it, so the only way to use it for backups is for the server to have access to the unencrypted contents of all the protected machines. Correct until now? Let's put aside for a sec the privacy issues. Now the backup server gets infected. Does not matter how. Infected update (I hope you don't suggest to run an unpatched machine for years) because stolen developer keys, fraudulent SSL certificate, bash bug, wife uses it to browse for kittens-vs-lime videos. Not really important. Now I have an intruder foothold on a machine connected to everything. The intruder methodically destroys everything accessible (on the secure backup and on all the protected machines). How do I recover from this? Or: backup server managed by Fort Knocks Unlimited. Disgruntled employee or software bug proceeds to methodically destroy data backup and machines connected to a bunch of accounts, including mine. Whatever. Not important. Everything accessible online is gone. Hard disk life is not really an issue here. Usage patterns similar to tapes. They would be on for weeks or maybe month during their entire life (most time sitting in a drawer somewhere), before I have to upgrade for capacity issues. The shelf life is definitely greater than 5 years. Failure? Rotating backups should take care of that too. | | rayik join:2005-08-04 united state |
rayik
Member
2015-Mar-23 5:20 pm
said by Black Box:Now the backup server gets infected. Does not matter how. Infected update (I hope you don't suggest to run an unpatched machine for years) because stolen developer keys, fraudulent SSL certificate, bash bug, wife uses it to browse for kittens-vs-lime videos. Not really important. If your backup server is used by your wife to browse the internet, it's no longer a backup server It's a workstation for end users. It is really important how it is used. | | |
I said that somehow it gets infected. Does not matter how. I was just being silly with that one. Use the first scenario, poisoned update. | |
|