Quick and dirty workstation replication

[Kint]

Here's the situation :

I've got 20 or so servers at this small company with absolutely no failover backups. Most of these servers were installed on the fly by the 'in-house' programmers, with no thought given to proper layout, security and backups. Some are Mandrakes, some are RedHats, some are Slackwares. Some IDE, some SCSI. Some of them have more than 1 HD, some don't. You get the point. The only hardware standard is Intel CPUs on all the linux servers. The actual data (user files, code, documents) is backed up safely using the LEGATO client for Linux. What I need is some sort of failover in the case of HD failure or worse, a break-in. I had considered mirroring, but that would imply buying 20 or so HDs. Plus, if a break-in occurs, that doesn't help me much. The mirror can be wiped quite easily.

Also, it seems that LILO has been installed on the MBR on all servers. I tried simply tar'ring (and dump'ing) up the OS filesystem (minus the data, basically /home and /usr), but that proved cumbersome when trying to replicate to another disk. It took too long to recreate a server, and I had problems getting LILO to rewrite itself to the MBR on the new disk. I solved that problem by using another boot loader altogether.

Then I tried 'dd'. That worked well, except that in some cases, /usr and /home are not on a separate partition and I end up backing up huge ammounts of data that's already been picked up by LEGATO. Also, that pretty much prevents me from doing incremental backups. On some servers, the configuration does change quite often.

Then I tried the latest version of Norton Ghost. Works well enough, lets me read ext2 partitions (thus only creating an image as big as the data being used, not the total size of the HD), but I have to shutdown the servers to ghost them. The full enterprise verrsion of Ghost (which lets you do live backups remotely) seems to lack a Linux client.

So there I am. I'd like to hear from SysAdmins who experienced something similar. Ideally, the solution would be something native to linux (tar, dump, etc.), and I don't mind 'combining' solutions (local mirroring for all the servers, plus some off-site backups).

A quick note on LILO : When recreating filesystems with restore or tar, I used a boot disk and chroot'ed my newly created filesystem as /, then ran lilo -b /dev/hda. For some reason, on reboot the system hung at 'LI...'.

Many, many thanks.

[ 08 August 2001: Message edited by: Kint ]

[Gnu/Vince]

how big are the HD's? A thing I like to do, is to use mkisofs to make an ISO image and then burn it to a CD. When you want to restore: dd if=/dev/cdrom of=/

Of course, with big hard drives, this is totally useless. Another thing is to make a script (perl or bash) in which you program everything that must be done to have security and backup problems. Then, you could make another script, which would scan machines on your subnet to see if the script has been ran (by checking for a file for example or I don't know what) and if it hasn't been, email the user to get him to download and install the script.

[milanuk]

I haven't tried this myself per se, but it seems that you might be able to do something like make a kickstart disk for each machine configuration, so that if a machine gets hosed, basically pop in the right kickstart disk, reinstall over the network and you're off and running again, assuming you have most of your data backed up/mirrored across the network. Just an idea, the details you'll have to work out for yourself, since I've never tried it myself.

The downside? Kickstart would probably be easier if you used RedHat.

HTH,

Monte

[ 08 August 2001: Message edited by: milanuk ]

[Kint]

Sounds good. I'll look into that.

Thanks!