How to set up master server to manage multiple servers in ISP type of data center

In an ISP or stock exchange data center, there are hundreds or even thousands of similar servers running same version of UNIX/Linux OS, and other application software.

To manage those servers and applications, the challenge is if you have an OS patch or application patch needs be installed, you will need to install the patch to all the servers that run the same version of OS or application. This will be a lengthy and very boring task: doing the same thing for 500 servers, and each take one hour or 2, means you will need your UNIX/Linux system administrators to do the task again and again for many days. Not many system administrators like that job.

WZIS Software has a solution to help you to significantly improve the efficiency of system administrators and security of the systems.

The solution will use our AutoSSH and CaclMgr.

AutoSSH is a very secure ssh job automation software, it contains 2 ssh automation tools: assh and asshc. assh is for auto ssh remote login, which is designed for flexibility. asshc, in the other hand, is for remote job execution, which is designed with command log for auditing purpose in mind.

CaclMgr is a very secure privilege delegation software. Unlike other similar function software, CaclMgr is designed to be usable for all kinds of command execution jobs in mind, i.e. not just interactive jobs, but also batch jobs. CaclMgr has many security features to help users to maintain good security on the system. For example, in-experienced system administrator may use sudo to grant an account the root privilege to run /bin/view, thought that will only allow that account to read files which require root privilege to read. But that’s very wrong：the receiving account will be able to run arbitrarily commands with the escalated privilege and able to write to the file. With CaclMgr, the system administrator will get a warning when he/she tries to grant the command to an account.

To use asshc to run commands on remote machine and log the commands for auditing purpose, you need first set up the encrypted password or passphrase file for asshc:

1. Run ssh-keygen to generate a pair of private/public keys for use by the asshc. To make sure no single person know the whole passphrase, you should have data/information security officer present to generate a dual controlled passphrase.

2. Rename the private key file to asshc.key, and the public key to asshc.pub.

3. Put the public key into remote root account’s .ssh/authorized_keys on the remote server.

4. Run

$ asshckey root@remote-server

to generate the encrypted passphrase file for asshc’s use.

When you bought enough connection/link capabilities for the AutoSSH, you can also set up a generic encrypted file for use by asshc connection to many servers, by running the following command exactly:

$ asshckey user@server

Now, you can use the asshc to run batch jobs on remote server.

But as now the user who sets up the encrypted passphrase file can run arbitrary command on the remote server without need to know the passphrase, this account’s security is very important for the remote server, so this account should be put under dual control. We name this account as sshdoer.

To understand how our AutoSSH and CaclMgr together can provide a very secure central server management solution, we will discuss the solution using an example.

Say for example, there are 1000 Linux servers in an ISP data center, they all run the same version of Linux and application.

There is a critical patch for the OS which needs be applied to all the 1000 servers.

Based on company’s policy, that patch needs be first installed on a test machine to confirm its effects.

Then you should develop a script, named as patchdoer.sh:

#!/bin/sh

mkdir /tmp/patchlog

for SRV in server1 server2 … servern

do

cat OSPatchXYZ.rpm|asshc root@$SRV dd of=/root/ OSPatchXYZ.rpm

asshc root@$SRV “cd /root; rpm –Uvh OSPatchXYZ.rpm" >/tmp/patchlog/$SRV.log 2>&1

done

put the script under sshdoer’s ownership, and then login as sshdoer, run

$ cacl –a admgroup /path/to/patchdoer.sh

After that, accounts in the admgroup will be able to run the patchdoer.sh to apply patch to the 1000 servers:

$ cacl –e sshdoer patchdoer.sh

For more examples, please check the “Cross Server Job Security and Solution for UNIX” “Case Studies” at www.wziss.com.