Simple yet powerful backup solution for Linux user data

Simple yet powerful backup solution for Linux user data

By apexwm, 24 June, 2011 14:38

One of the things I love about Linux is that it comes with a huge suite of software and utilities that can be very powerful. Some of these utilities go un-noticed or are not widely known. In this example I'll focus on backing up user data; I am a strong believer of making good backups of data at all times, to prepare for the worst. There's nothing worse than losing data and having to live with older copies or missing files.

With Linux, the entire user's profile folders are stored in /home. What's even better about this, is all applications that the user runs also store settings in this profile folder. So, my main focus is to get good backups of the user's profile folders that are stored in /home, and this will ensure that if the hard drive goes up in flames, that all data can be restored allowing the user to be back up and running with little effort.

For doing these backups, I use a lightweight but extremely powerful utility called "rsync". It's a command-line based utility, but can handle gigs of data across slow links, making it very effective. In fact, it's one of the leading utilities used to replicate large public software mirrors among sites. What makes rsync most appealing to me is it can identify changes in files, and will only transfer data that has changed since the last sync, making it extremely efficient. I've used various similar programs in Windows to rsync, and have NEVER found anything as effective as rsync on Linux. Mainly, issues like "files in use" and other problems crept up in Windows, which caused incomplete backups and errors, and never gave us 100% reliable backups. Not to mention, the ability to run a sync in Windows silently seemed to be a challenge also, where rsync on Linux can be run silently in the background while the user continues to work, and there are never any file locking issues which is extremely important for getting a complete backup.

Using rsync involves a little setup and knowledge of editing text files on a Linux system. You must have rsync running on a host/server system where the files are stored, and also on the client where the files are backed up from. And the beauty about Linux is that the host/server and client computers can be any sort of machines because Linux distributions contain the same software and rsync is relatively light on system load.

On the host/server:

Make sure the "xinetd" and "rsync" packages are installed. Edit the file /etc/xinetd.d/rsync, and make sure it has contents similar to below:

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
# allows crc checksumming etc.
service rsync
{
disable = no
# flags = IPv6
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = --daemon
log_on_failure += USERID
}

This makes xinetd start the rsyncd service. You can double check by running "ntsysv" and making sure "rsyncd" is checked so that it starts automatically on boot.

Then edit the file /etc/rsyncd.conf and add a backup point where the data will be stored, similar to the sample below:

[backup]
path = /ext/rsync/backupfolder
uid = localuser
gid = localgroup
read only = false
comment = Workstation backup
auth users = localuser
secrets file = /etc/rsyncd.secrets

Edit the file /etc/rsyncd.secrets (or create it if it doesn't exist), add a line with the username/password specified in the file above, as "username:password". In this example, "localuser:backuppassword" would suffice. You can create mutiple backup points (one per user) and specify separate usernames and passwords by creating multiple entries in both of these files.

On the client system:

Create a file (I usually store this in the user's folder, i.e. /home/localuser/.rsync.passwd). Putting the dot in front of the file makes it hidden. In that file, put in the same password specified in the server's settings file above, in this case "backuppassword"; you do not need the username in this file.

Next, you simply need to add a cron job to run the backup job on the client machine. I do this by creating a new file in /etc/cron.hourly, which cron will run on the hour. I usually name it : /etc/cron.hourly/z-rsync-backup, so that cron will run it last. Be sure to "chmod 775" to this file to allow cron to run it. Then in this file, put a line in it similar to below:

rsync -aq /home/localuser/ localuser@rsyncserverip::backupfolder/profilefolder --prune-empty-dirs --delete --delete-excluded --exclude-from=/home/localuser/.rsync-exclude --password-file=/home/localuser/.rsync.passwd

This creates a live "copy" or sync of the user's profile folder on the host/server.

The final step is to create the file referenced in the run command above, /home/localuser/.rsync-exclude , and put in entries for items that you do NOT want to back up. I usually add the following entries:

.mozilla/**/Cache
*.o
temp/*
.gvfs

This skips over Firefox's cache, any object files, any files if there's a folder called "temp", and the .gvfs folder.

Conclusion

The possibilities are endless for rsync. Read the man page by using "man rsync" if you want to read what all of the options do.

This solution can be added or enhanced for added functionality. I use one scenario like this and on the host/server system, I set rsync on the host/server to back up to the path "/media/backupdrive" (which is a mounted ext4 USB drive named "backupdrive"). This is done with the "path =" line in /etc/rsyncd.conf. This way the USB drive can be removed and stored in a safe place. Another solution I use is to have rsync back up remote systems to another site over the Internet, using ssh tunnelling to provide a secure connection.

I currently run my own backups like this with multiple workstations that save the files to a server, then on the server the files are backed up to tape. I have put it to the test by formatting a hard drive and installing Fedora from scratch again, and restoring the data backed up with rsync. When logging in, all profile data is read like normal and everything works just like it did before. In fact, an rsync command could be written to do the opposite where it connects to the host/server and restores the data back to the client. These are all examples of how GNU/Linux is really excellent stuff, and just works every time without a lot of hassle like we have in Windows.

Talkback

Any average computer user would look at this and say you are crazy. Windows 7, simply attach an external hard disk and click create a system image, in an hour, you have 120 GBs of data backed up.

After 20 years, Linux still succumbs to the command line. ..And you wonder why the year of the Linux desktop has not arrived yet.
Andre Da Costa 23 July, 2011 07:29
Report offensive content Reply

Andre Da Costa :

Please provide to me a solution that works in this fashion in Windows. I have provided a solution where I can back up multiple computers to one single file server, and ensure that I retain ALL DATA. And, data is backed up on the hour. So, if any one of the PCs were to burst into flames, I would have a complete backup of data that is less than an hour old. Oh, and don't forget Windows scatters settings and user data among the filesystem and registry, so make sure your recommendation gets both.

If you are looking for a PC imaging solution as in your example, Clonezilla would be my open source answer. Fedora Linux also has backup tools like Deja Dup Backup Tool, Nepomuk, and BackupPC.

Unfortunately I have news on the command line as well. Any operating system needs knowledge of the command line. That's why Microsoft developed PowerShell.

As I usually state, the issue at hand is that users must have the patience to learn and make a change, in order to change operating systems.
apexwm 3 August, 2011 00:33
Edit Delete Report offensive content Reply

Excellent post apex, thanks for the info!

I can confirm that Clonezilla is also an excellent application, albeit with a different purpose to rsync as you described. Even if Andre's post was relevant in the slightest, the command line is what makes rsync more flexible than the Windows imaging method he described. True, it is designed for administrators, not average users, but as Apex outlined above, the "average user" has plenty of varying options when backing up Linux systems.
JamieKB 3 August, 2011 09:26
Report offensive content Reply

There you are! You found the answer you were looking for!
"users must have the patience to learn and make a change, in order to change operating systems"
A lot of information crammed in such a compact phrase:
1 - Users must have patience; not that linux must raise their appetite for a better working environment, users must have patience.
2 - Users must learn and make a change, if you ever expect users to learn to backup in this fashion with editing obscure files with some command line knowledge, you better get comfy, it might take a while.
3 - In order to change operating systems, why exactly should I change operating systems? just for the fun of it? For the vast majority of users windows is what they like, what they want and if they get anything else they return the product, just check return statistics for hardware manufacturers that sell the same hardware in both windows and linux, maybe it will sober you up and make you return back to earth from that ivory tower from where you shout linux is just better.

Cheers
ndl007 3 August, 2011 09:48
Report offensive content Reply

ndl007 :

"For the vast majority of users windows is what they like, what they want and if they get anything else they return the product, just check return statistics for hardware manufacturers that sell the same hardware in both windows and linux, maybe it will sober you up and make you return back to earth from that ivory tower from where you shout linux is just better."

And I agree with you, many users are happily suited with Windows. If you like to be controlled on how to use your PC, enjoy troubleshooting it often, and enjoy having funds sucked out of your pocket on a continuous basis, and don't have any desire to change that, Windows is definitely for you. I provided this backup solution as a demonstration of the powerful utilities available with GNU/Linux right out of the box. I'm not urging everybody to try setting it up. There are many utilities and software available for backups that are easier to use than my solution posted here, so it's up to the user to choose which one best suits.
apexwm 3 August, 2011 14:28
Edit Delete Report offensive content Reply

I work in the high tech industry and use a laptop every working day for email, web surfing, office and additionally some very advanced design applications. For us it would be impossible to use linux as those applications exist for mac and windows only.
I'm not controlled in any way shape or form in my use of windows.
I have used windows xp since 2004 and I got a blue screen of death once in an IBM T41 back in 2004; I got service pack 2 installed and never I got any error in windows. It has been heavy operation continuously for 7 years without a glitch.
Actually I like the minimal interface of google chrome and that is my browser of choice but every now and again it pulls a face and tells me it must kill some tab. Those are my errors in 7 years, is that trouble shooting it often?
Money wise I really don't see much of a difference, when I buy a computer it comes with OS installed and the price difference between models with linux and windows is minimal if any at all.
I really don't see how linux can ever make it to the desktop, windows is vastly superior if what you want to do is produce real work, rather than hack around loosing time and money in things that are not your core business.

Cheers
ndl007 4 August, 2011 09:46
Report offensive content Reply

ndl007:

"I'm not controlled in any way shape or form in my use of windows."

So, you weren't forced to activate your copy of Windows XP? What about other proprietary software that requires activation or system profiling?

"I got service pack 2 installed and never I got any error in windows."

I've had this happen too, on one PC, no problems. However, out of a group of 5 PCs, at least 2 of them have some sort of issues over a span of 7 years as you stated. Temp files accumulate, the PC needs defragging, all issues that the user should not have to deal with, if the operating system were somewhat efficient.

"Money wise I really don't see much of a difference, when I buy a computer it comes with OS installed and the price difference between models with linux and windows is minimal if any at all."

Yes, but what about additional software? Or when the next version of Windows is out, everybody is buying a new PC with it, do you pay for a full upgrade at that point? Or buy a new PC (even if the current PC will run the new version of Windows just fine)?

"I really don't see how linux can ever make it to the desktop, windows is vastly superior if what you want to do is produce real work, rather than hack around loosing time and money in things that are not your core business."

Based on that statement I would have to ask you, have you used a GNU/Linux distribution, and actually tried it for at least a couple of weeks? If you use niche applications that MUST run in Windows, then I see your point. But you actually describe exactly what I have seen with Windows, where technicians are constantly maintaining the OS rather than getting some productive work done.
apexwm 6 September, 2011 19:30
Edit Delete Report offensive content Reply