[CLUE-Tech] Managing multiple servers (long)

[David Anselmi]

Dan Harris wrote:
> I wanted to get some advice from others here on distro suggestions, 
> hopefully without causing an all-out distro war. But we're all adults 
> here, right?

[...]

> I'd really like some advice here from people who have been faced with 
> the problem of managing multiple servers (i expect to be up to 20 
> servers within the next two years).

My advice is to simplify, and be anal about it.

First, all your servers should run the same distro and versions of 
software.  All the packages should be identical on every machine 
(including the test machine you try out *any* changes on).  There is a 
trade off here--you don't want to run a database on a machine that's a 
web server only, but you have to balance that with the difficulty of 
managing more configurations.

Second, all your config files should be as similar as possible and you 
should be able to identify who changed what, when.  You might consider 
putting templates into CVS, with machine specific adjustments (host 
name, IP, etc) made by a script.  I have done this before (on a web 
application, not a whole server) and it worked well.  The process for 
changing a config file and pushing the change out (including what needs 
restarting/rebooting) should be written down and you should follow it 
exactly, always.  There are many ways to manage config files, but in the 
end you should be able to diff everything in /etc on 2 machines and get 
a small number of well understood differences.

Third, document everything.  You should be able to check (easily) what 
packages/versions are installed on each server.  They should be exactly 
the same except for deliberate differences.  You should be able to take 
a bare metal machine, install, update, and configure and know that it 
matches your other servers.  This doesn't have to be difficult--all the 
info should be on the machine.  You just have to be able to find it 
easily (use a script to generate a database, text file, web page, whatever).

Fourth, test any changes you contemplate on an expendable server.  This 
is where documentation is important--the test machine must be exactly 
like the production machines.  Whether you're tweaking a config or 
applying a security patch, do it on the test machine first.  Make sure 
you can follow the same change process on each server, write it down, 
and follow it to the letter (that way you don't forget to restart 
something the 19th time you do it).  Ideally you would have some 
automated tests to make sure the test machine still functions after the 
change.  More than just "does it serve up index.html", more like "does 
it have the same performance profile it did before".

Of course this is a lot of work.  If you're trying to provide 20 servers 
for your hobby activities, it probably isn't worth it.  But if you want 
to provide consistent, reliable service to customers, it's worth the 
effort to build something reliable.

I will also say that the added effort to install packages from source 
probably isn't worth it.  If you have to use source (including source 
you are writing), set up a build process (on a build machine, not a test 
or production server) that will produce a binary package for you.  Then 
you can install it on the test machine, verify it, and use the same 
process to install it on a production machine.  That saves you from 
needing compilers et. al. on your servers (not to mention the 
performance hit when you ask them to compile something).

As far as distros go, Debian seems pretty good.  The update process (apt 
or dselect) is easy to understand, and stable doesn't change except for 
security fixes (which are easy to follow on their mail list).  Probably 
there is a way to get similar features out of an rpm distro, the key is 
how YOU manage it.

In implementing the above, you can take an evolutionary approach.  Start 
by setting up an easy way to list and compare what's installed on each 
server.  Work on standardizing that and setting up a test machine that 
represents the standard.  Once you have one piece (say updating packages 
for security fixes), work on another piece.

I wonder whether there are any tools to help with this problem.  It 
seems that everything is aimed at easing configuration by making web or 
gui front ends.  But they don't seem to address issues of consistency or 
repeatability.  I would think that running Linux on the desktop has a 
similar set of needs.

HTH,
Dave