Summary: It is imperative that you can determine what state a machine was in before you can ascertain how crackers compromised your security measures.

When your machine is cracked[1] it's a good idea to figure out exactly what happened. If you don't know what was or who was on your machine, there's little chance you can ascertain what occurred to breach the security. Regardless of your post-compromise preference (fixing the problem or reinstalling from scratch) it's imperative that you understand what happened. If you end up leaving the exact same vulnerability open, you haven't fixed anything.

For example, I once consulted at a pharmecutical company that was required by FDA regulations to keep detailed paper logs of every piece of software installed, every configuration change, ever user added or removed. While most of us grumbled about the annoyance of doing so, it made it absolutely clear what the state of a machine was over time, which made it easier to correlate problems with configuration changes, and find other bugs created by users^H^H^H^H misconfiguration.

Unfortunately, most folks don't have the time or energy to list everything they do to a system. However there are many steps you can take to make things more automated.

Backup configuration directories

In addition to your normal system backups, create simple cron jobs that back up your configuration settings - The /etc directory may be sufficient in itself. Backups should ideally go to removable media (tape, CD, etc) for long term storage, but you may find it's also helpful to have it stored on a separate machine as well for faster access in a crisis. Take, for example, the following script:

  #!/bin/sh

  cd /etc

  tarball=`mktemp` || exit 1

  tar cvzf $tarball .
  scp $tarball backup-server:/backups/`hostname`.etc.`date '+%Y-%m-%d'`.tgz
  rm $tarball

This shell script, ripe for use from cron, will create a tarball of /etc and scp it over to a backup-server. (This, of course, assumes that you have some form of passwordless scp access available. I'll probably cover that in a later article.) Certainly not the most elegant of solutions, but it will capture the essence of the machine's state.

Of course, if you liberally comment your configuration files, this makes everything much clearer as well.

CVS

My personal favourite method is to keep all your configuration in CVS. You can put all of /etc in a CVS repository and maintain your configuration via CVS that way. This makes it easy for you to see exactly how a machine was configured at any time, and makes it trivial to recover a compromised system or clone a functional one.

You can use RCS locally, if you prefer, and in combination with some sort of backups such as the previous example you'll have a similar revision history without the overhead of learning and implementing CVS.

Internal Mail Archives

If you have multiple administrators at work, it's a good idea to create a mailing list that goes to all parties, and have each mail archived automatically. Though not as seamless as the previous two options, it does provide a minimal audit trail that may be helpful.

This assumes, of course, that you're all keeping each other informed about what you're doing. Communication is good for systems stability. Then again, keeping secret what you do is good for job security.

This article is written partially in response to a request from a reader. He found that several processes were running as uid 1032, but no such user existed on the system. In fact, more processes kept being run over time - it wasn't just leftover processes that were still running. It took a while to track down how the processes were being started before they were all killed off.

There used to be an actual user with uid 1032, but it had been deleted some time in the past. Since the record from /etc/passwd was gone, it was not immediately obvious who the user himself actually was.

Next week I'll teach you a very simple trick you can use to lock out user without loosing their actual login information. However until that time, I offer my latest challenge...

The challenge

Come up with fun, ingenious, or wacky ways that you think a user who is removed from the system (entry in /etc/passwd is deleted, for example) can have processes still running, and/or the ability to create new processes at will or periodically.

I'll send out a post card from lovely yet rainy Seattle to an entry (or entries) of my choice based on Originality, Performance, and Lip Sync, or any other criteria that comes to mind.

(Send your responses to me at bri@hackinglinuxexposed.com. Don't reply to the list address.)

[1] Yes, "when", not "if". Sooner or later, it happens to all of us.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He is a bit CVS-happy. His websites are all stored as WML in CVS, and any cvs commit automatically triggers a rebuild of the site and rsync to the webservers. Now if only he could get this to work for the non-computing areas of his life. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002

This is the November 06, 2002 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.