Rsync assistance or right tool for the job?

Author	All Replies
Matt All noise, no signal. Premium join:2003-07-20 Jamestown, NC kudos:12	Rsync assistance or right tool for the job? I am implementing a pair of servers running Nginx with heartbeat clustering. I'd like to keep a few of the Nginx config directories in sync, so if Server 1 is edited, Server 2 pulls the changes or if Server 2 is edited, Server 1 pulls the changes. Ideally, this would run from cron every 15-30 seconds. The config file directories don't need to be diff patched or anything exotic, the config directories and their contents can be overwritten with the newer versions. I have been reading up on Rsync, as I think this is the tool I want, but I don't see any way to tell it only to pull the file/directory if it is newer than the existing. Is that functionality simply assumed, will I need to script the logic to determine if the files are newer, or is there a better tool for what I am trying to do? Thanks for pointing me in the right direction.
	to forum · permalink · 2012-01-12 15:48:24 ·
pablo MVM join:2003-06-23 kudos:1	Howdy, `rsync' is the right tool for the job. If you need under one minute refreshing, you'll need to write a script to put `rsync' in a loop as `cron's most granular unit is per minute. Cheers, -pablo -- openSUSE 12.1/KDE 4.x ISP: TekSavvy Bonded DSL; backhauled via a 6KM wireless link Assorted goodies: »pablo.blog.blueoakdb.com
	to forum · permalink · 2012-01-12 16:14:06 ·

Brano I hate Vogons Premium,MVM join:2002-06-25 Burlington, ON kudos:3 1 edit	reply to Matt Typically in clustered environment you'd put such files on shared external storage and mount it from both machines.
	to forum · permalink · 2012-01-12 16:41:44 ·
ethics Premium join:2008-05-09 Brooklyn, NY	reply to Matt 1. You need to set up SSH keys so you can run this across servers. Use -e which specifies the files will be sent through SSH. 2. Looks like this kid did it. Personally, I would just use the -u switch to NOT update newer files on each of the servers. So if you have server named ethics and server named matt The file on ethics is updated. Next rsync run when ethics tries to rsync it will allow it and matt will then be updated. While the other server tries it, it won't because the file is updated. I would look around and rsync has a -dry run option which will allow you to see what it will do and not actually do it. Once the results are what you want, you can then put it in to cron to run whatever intervals you want.
	to forum · permalink · 2012-01-12 16:57:25 ·
sbconslt join:2009-07-28 Los Angeles, CA	reply to Matt While you can accomplish this with rsync technically speaking, given the polling frequency you mentioned, I would recommend a shared filesystem instead, being the better tool for the job. -- Scott Brown Consulting
	to forum · permalink · 2012-01-12 17:22:06 ·
Matt All noise, no signal. Premium join:2003-07-20 Jamestown, NC kudos:12	reply to Matt Thanks for the advice gentlemen. Unfortunately, a shared filesystem is not an option, which is why I am exploring rsync. A 1 minute interval will be sufficient. Cheers.
	to forum · permalink · 2012-01-12 20:26:32 ·
Gomez Exile in waiting Premium,Ex-Mod 06-11 join:2001-02-21 Atlanta, GA	reply to Matt A lot depends on your process. I'm not about to automatically propagate a config file across the cluster simply because it happened to have changed. I can't risk our SLAs and bring down the entire operation because someone fat fingered a configuration, or worse. We tend to follow a more pessimistic approach that favors a formal Q/A process and validation of production configurations. There are configuration managament tools (Puppet, Chef et al) designed to solve this problem. -- It's a fact : Chicks dig Mafia players. 'Wanna help buy a goat?' - »www.kiva.org
	to forum · permalink · 2012-01-13 00:14:36 ·
DeHackEd Bill Ate Tux's Rocket join:2000-12-07	reply to Matt Right tool for the job? You may want drbd. Google will find it quickly. Drbd is shared storage running like RAID-1 over the network, with the special case that generally only one host can have access at a time. To take over the other host, you must first enable access to the device (subject to the host either not being in control itself, or being dead enough not to object) then mount the device locally. As such use a journaled filesystem like ext3. Heartbeat should include scripts to help manage this automatically. -- That's odd...
	to forum · permalink · 2012-01-13 05:28:57 ·
Matt All noise, no signal. Premium join:2003-07-20 Jamestown, NC kudos:12	reply to Gomez said by Gomez: A lot depends on your process. I'm not about to automatically propagate a config file across the cluster simply because it happened to have changed. I can't risk our SLAs and bring down the entire operation because someone fat fingered a configuration, or worse. We tend to follow a more pessimistic approach that favors a formal Q/A process and validation of production configurations. There are configuration managament tools (Puppet, Chef et al) designed to solve this problem. We are a small business. I am the only one who would be making changes. Knowing that, would you elaborate on the tools you mention?
	to forum · permalink · 2012-01-13 22:01:23 ·
Matt All noise, no signal. Premium join:2003-07-20 Jamestown, NC kudos:12	reply to DeHackEd said by DeHackEd: Right tool for the job? You may want drbd. Google will find it quickly. Drbd is shared storage running like RAID-1 over the network, with the special case that generally only one host can have access at a time. To take over the other host, you must first enable access to the device (subject to the host either not being in control itself, or being dead enough not to object) then mount the device locally. As such use a journaled filesystem like ext3. Heartbeat should include scripts to help manage this automatically. Thanks, but drdb was evaluated and just doesn't fit us.
	to forum · permalink · 2012-01-13 22:02:24 ·
Gomez Exile in waiting Premium,Ex-Mod 06-11 join:2001-02-21 Atlanta, GA	reply to Matt said by Matt: said by Gomez: A lot depends on your process. I'm not about to automatically propagate a config file across the cluster simply because it happened to have changed. I can't risk our SLAs and bring down the entire operation because someone fat fingered a configuration, or worse. We tend to follow a more pessimistic approach that favors a formal Q/A process and validation of production configurations. There are configuration managament tools (Puppet, Chef et al) designed to solve this problem. We are a small business. I am the only one who would be making changes. Knowing that, would you elaborate on the tools you mention? Hopefully small shops become big shops and stuff like this starts to matter. If you start with a good infrastructure, you don't have to undo all the hacks later. Not to say syncing is a hack, we use it extensively, but not for configuration. »puppetlabs.com/ We're in the process of ramping up for deployment with this. -- It's a fact : Chicks dig Mafia players. 'Wanna help buy a goat?' - »www.kiva.org
	to forum · permalink · 2012-01-13 22:46:54 ·
Cabal Premium join:2007-01-21 Austin, TX Reviews: ·Suddenlink	reply to Matt Agreed, a hacked-together rsync, ssh, or filesystem solution is no replacement for proper configuration management. »bitfieldconsulting.com/puppet-tutorial »bitfieldconsulting.com/puppet-tutorial-2 -- Are you now or have you ever been a member of the Islamic religion?
	to forum · permalink · 2012-01-13 23:24:17 ·
Matt All noise, no signal. Premium join:2003-07-20 Jamestown, NC kudos:12	said by Cabal: Agreed, a hacked-together rsync, ssh, or filesystem solution is no replacement for proper configuration management. »bitfieldconsulting.com/puppet-tutorial »bitfieldconsulting.com/puppet-tutorial-2 Are you really accusing me of poor configuration management when I came here asking for help?
	to forum · permalink · 2012-01-13 23:55:37 ·
tubbynet reminds me of the danse russe Premium,MVM join:2008-01-16 Chandler, AZ	said by Matt: said by Cabal: Agreed, a hacked-together rsync, ssh, or filesystem solution is no replacement for proper configuration management. »bitfieldconsulting.com/puppet-tutorial »bitfieldconsulting.com/puppet-tutorial-2 Are you really accusing me of poor configuration management when I came here asking for help? while i can't comment on the actual solution -- i can only comment on the actual perception of those posting. i work in a field where 'good enough' has to make due for the time being (and this is from a consultants perspective). i make my bread and butter from ensuring customers are never on the bleeding edge. that being said -- there are times when a 'good enough' solution has to be met in order to fill the immediate business requirements in order to decide on which way to proceed. in this case -- is Matt 's rsync solution the best -- by consensus -- i would assume "no". but in the interest of time, will it work until process (read: layer 8, 9) gets sorted, yes. sometimes, the level of abstraction and 'best practice' is removed from the day to day. its easy to armchair quarterback -- but we've all done crazy and wonky shit to solve an immediate problem. (disclaimer -- i'm a network architect -- not a dba. shakers of salt may be necessary). q. -- "...if I in my north room dance naked, grotesquely before my mirror waving my shirt round my head and singing softly to myself..."
	to forum · permalink · 2012-01-14 00:08:09 ·
Gomez Exile in waiting Premium,Ex-Mod 06-11 join:2001-02-21 Atlanta, GA	reply to Matt If you're plan is to use blind syncing for key configuration files.. The reluctant answer is yes, thus the proposed alternatives. Blind sync is full of problems, from security to application races, to basic process control, and it won't scale. As in most things there are right ways, and wrong ways of doing things. The solution presented, would be considered the wrong way by those that have dealt with the problem in terms of scale, security and SLA requirements. The request for help resulted in some best practices, that in fact might be better solutions. The intent was no to insult, but to solve the problem as it could best be solved, via established proven patterns. -G -- It's a fact : Chicks dig Mafia players. 'Wanna help buy a goat?' - »www.kiva.org
	to forum · permalink · 2012-01-14 00:17:31 ·
Matt All noise, no signal. Premium join:2003-07-20 Jamestown, NC kudos:12 1 edit	said by Gomez: If you're plan is to use blind syncing for key configuration files.. The reluctant answer is yes, thus the proposed alternatives. Blind sync is full of problems, from security to application races, to basic process control, and it won't scale. As in most things there are right ways, and wrong ways of doing things. The solution presented, would be considered the wrong way by those that have dealt with the problem in terms of scale, security and SLA requirements. The request for help resulted in some best practices, that in fact might be better solutions. The intent was no to insult, but to solve the problem as it could best be solved, via established proven patterns. -G I am not trying to scale. Nor are they "key" configuration files. My question was answered here. If you know anything about Nginx, you'd know the virtual hosts file is decidedly NOT key. So stop trying to tell me what is best for me. Man, you *nix guys are assholes. I hate asking you for help because you always seem to know better.
	to forum · permalink · 2012-01-14 03:50:37 ·
pablo MVM join:2003-06-23 kudos:1	Howdy, Although I confirmed you could use `rsync' as you were intending, the other posters provided other ideas I hadn't considered. Perhaps how the message was delivered wasn't to your liking but you shouldn't torch the bridge by calling the people who are trying to help you (be calling them assholes). That's simply unkind. Good luck, -pablo -- openSUSE 12.1/KDE 4.x ISP: TekSavvy Bonded DSL; backhauled via a 6KM wireless link Assorted goodies: »pablo.blog.blueoakdb.com
	to forum · permalink · 2012-01-14 06:44:05 ·
ethics Premium join:2008-05-09 Brooklyn, NY	reply to Matt We use rsync in a corporate environment and it's part of our contingency. I've worked for big investment banks and small ones and rsync is more than fine, as we performed quarterly disaster recovery tests. The rsync method we used is a bit more elaborate than what you have but I don't see an issue in using it in what you want. If you need to grow and scale, it would be easy to do just that once there is growth not in just company's profits but personnel. Someone then could just take your rsync solution and scale it to whatever they want but I am pretty sure it will work just fine. Start off with rsync -u and go from there.
	to forum · permalink · 2012-01-14 09:01:07 ·
No_Strings Premium,Mod join:2001-11-22 The OC kudos:6 Host: Wireless Networking All Things Unix Cox HSI Efficient Southwest Chat	reply to Matt One of the great aspects of Linux environments is that there are many ways to skin a cat. One of the things I like about ATU is knowing someone else will have a different approach to the problem. That way, I can decide which is best for my situation. Few things have gotten me into more trouble than thinking I know more than I do.
	to forum · permalink · 2012-01-14 10:15:59 ·
sbconslt join:2009-07-28 Los Angeles, CA	reply to Matt I use rsync over ssh to keep server pairs (primary and failback) in sync however the sync frequency is every 4 hours. What made me want to suggest a different approach for you is the extremely short frequency you mentioned of 15 to 30 seconds. Rsync is a lot of overhead spawning a process establishing a connection inventorying what is to be transfered etc. Rsync is appropriate every 4 hours but at this frequency a solution that persisted the link would have less unnecessary performance impact. We were trying to get you thinking about aspects of the solution design that might not have crossed your mind such as performance and security. -- Scott Brown Consulting
	to forum · permalink · 2012-01-14 12:35:37 ·

OS and Software > All Things Unix > Rsync assistance or right tool for the job?