Hacking Linux Exposed





previous article
next article
/etc/inittab - The Most Overlooked Cracker Haven
By Bri Hatch.

Summary: Crackers can cause their software to be run by adding entries to /etc/inittab, a file frequently missed by administrators.

Last week I provided a challenge, which can be summarised as follows:

  • A cracker broke into a server,
  • Said cracker installed various back doors,
  • The administrator removed everything he could find, but,
  • Each time the machine was booted, even in single user mode, mysterious processes ran from /tmp/.qw,
  • The /tmp directory was created clean from scratch at every bootup, so the files could not be stored there and usable between reboots,
  • All xinetd and /etc/rc?.d services were cleaned out and nothing therein could have started these mystery processes, or start processes that started those processes, etc,
  • No LKMs or kernel modifications were used to start these processes[1],

Many[2] people wrote in with ideas which fell into the following categories:

  • changes to the master boot record
  • changes to /etc/inittab
  • changes to bootloader (lilo/grub) command-line arguments
  • changes to initrd ram disks
  • changes to /root/.bashrc, /etc/profile, and such.

I told folks to discount any change which required kernel modifications. A change to the MBR probably doesn't qualify in that regard, but it would certainly seem out of scope nonetheless.[3] Similarly, a change to the bootloader or initrd ram disk[4] fall under kernel changes to some degree, and thus weren't the correct answer, based on my phrasing.

Many folks suggested that it was root's .bashrc, or global /etc/profile and similar files that was the culprit. By logging in, root's shell would execute commands that were hidden in those files. That's definitely a possibility, but the fact that moniker had PID 15 meant that it had to have started before root logged in, so that's not the culprit.

In fact, the problem on this compromised box was caused by cracker changes to /etc/inittab. To explain, let's recap some of the evidence. The processes were visible via ps as:

  # ps -ef | grep qw
  root     15    1    0 11:24 ?        00:00:00 /tmp/.qw/moniker
  root     242   15   0 11:25 ?        00:00:00 /tmp/.qw/cr8
  root     244   1    0 11:25 ?        00:00:00 /tmp/.qw/rucc

Several folks said that it is quite obvious that 'moniker' and 'rucc' were started from init since their parent is process #1. (Init always has process ID #1). Unfortunately, that's faulty logic. Any daemonized process will be inherited by init when as seen in this "ps -ef" snippet:

$ ps -ef | grep '  1  '
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  0 Jan23 ?        00:00:19 init [2]
  root         2     1  0 Jan23 ?        00:00:00 [powerd]
  root         3     1  0 Jan23 ?        00:00:00 [keventd]
  root        10     1  0 Jan23 ?        00:00:00 [scsi_eh_0]
  root        11     1  0 Jan23 ?        00:00:00 [khubd]
  root        88     1  0 Jan23 ?        00:00:04 perl /etc/swatch/swatch
  root       483     1  0 Jan23 ?        00:00:00 /sbin/klogd
  root       574     1  0 Jan23 ?        00:02:23 /usr/lib/postfix/master
  daemon     612     1  0 Jan23 ?        00:00:00 /usr/sbin/atd
  root       619     1  0 Jan23 ?        00:00:13 /usr/sbin/cron
  root       632     1  0 Jan23 ttyS0    00:00:00 /sbin/getty 9600 ttyS0
  root       859     1  0 Jan23 ?        00:02:15 /sbin/syslogd
  root       893     1  0 Jan23 ?        00:28:28 /usr/sbin/sshd

These processes also have a PPID of 1, but most were started by standard /etc/rc?.d scripts. The fact that our cracker's processes had init as the parent doesn't prove they were started by init. However the 'moniker' program has a PID of 15 - a very very low number for processes. Assuming it was started as the 15th process (as opposed to being several thousand process IDs later when the PIDs wrapped around) then it must be extremely early in the boot process, before many rc?.d scripts run, and likely is started from init.

The administrator had re-installed all his software to be safe. By design, these re-installs did not wipe out his configuration files, so though he reinstalled init, it did not wipe /etc/inittab clean.

He scanned the machine for any files that had been touched since the breach, but this is not reliable. A cracker can easily change the modified-date timestamp on a file using the 'touch' command. Doing so will change the inode change time, however:

  $ cd /tmp
  $ ls -l somefile; ls -cl somefile
  -rw-------    1 bri      bri         20476 Sep 17 14:49 somefile
  -rw-------    1 bri      bri         20476 Sep 17 14:49 somefile
  $ date
  Wed Dec  4 13:31:37 PST 2002

  $ echo "blah" >> somefile

  $ ls -l somefile; ls -cl somefile
  -rw-------    1 bri      bri         20481 Dec 04 12:00 somefile
  -rw-------    1 bri      bri         20481 Dec 04 12:00 somefile

  $ touch 09201419 somefile
  $ ls -l somefile; ls -lc somefile
  -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
  -rw-------    1 bri      bri         20481 Dec  4 13:32 somefile

Note that after our 'touch' command, the file modification time (mtime) was set to the original Sept 17th date. However the inode change time (ctime) was not. The ctime is always updated whenever the inode has any changes made, such as file modifications, chmod permissions changes, etc. The administrator was smart by looking for files with mtime or ctime changes during the window where the cracker had access.

If the cracker modified /etc/inittab, even though he could reset the mtime with touch, he shouldn't be able to change the ctime back. Thus you'd think that his find process would have caught the /etc/inittab changes. Unfortunately, the cracker can still modify the ctime of a file simply by changing the system clock:

  $ date 09201419 
  $ touch 09201419 somefile
  $ date 12041200 
  $ ls -l somefile; ls -lc somefile
  -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
  -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile

So, faking both the mtime and ctime is trivial, and this is why he missed the changes to /etc/inittab - his find command didn't indicate any change, so he didn't examine the file at all.

So, what were those changes? Here's a snippet of the file

  # The default runlevel.
  # Boot-time system configuration/initialisation script.
  # This is run first except when booting in emergency (-b) mode.
  # What to do in single-user mode.

  # /etc/init.d executes the S and K scripts upon change
  # of runlevel.
  l0:0:wait:/etc/init.d/rc 0
  l1:1:wait:/etc/init.d/rc 1
  l2:2:wait:/etc/init.d/rc 2
  l3:3:wait:/etc/init.d/rc 3
  l4:4:wait:/etc/init.d/rc 4

The line that begins with C1 above was the new entry. It tells init to run the program /dev/st0z at boot time, regardless of runlevel. This entry came before the /etc/init.d scripts (the l# entries) which is how it was able to get such a low process id.

The /dev/st0z file was the cracker's program. Sticking the file in /dev/ made it less likely the administrator would find it, since many admins don't understand what all the /dev devices do. This file was a simple shell script:

  # head /dev/st0z
  cat <<'EOF' > /tmp/st0z.uu
  begin 700 st0z
  cd /tmp
  uudecode st0z.uu
  tar xzf st0z
  rm  st0z.uu st0z
  exec /tmp/.qw/moniker

This script contains a uuencoded file within it, which it puts in /tmp. This uuencoded file is just a compressed tarball, so it uudecodes and untars it. It then deletes the temporary files, and then runs the moniker program (another shell script). The actual script had more comments at the top to make it look more like a system program, and obfuscated the commands a bit, but you get the idea.

Moniker was responsible for running the cr8 program -- obvious from ps output, since moniker is cr8's parent -- and for running rucc, which is not obvious from ps, but obvious if you read moniker's source.[6]

This st0z script also explains why the files in /tmp/.qw were always coming back, even though they weren't there between reboots. The files were copies out of the /dev/st0z file each time, so wiping /tmp had no effect.

So what were the problems this administrator faced when trying to track down and remove the mysterious cracker code?

  • He blindly trusted timestamps on files, thus missing /etc/inittab.
  • Had no file integrity tools on the system, thus missing inittab and /dev/st0z.
  • He didn't understand the Linux boot process enough to know that programs can be launched by init directly.

Of course, you may argue having a weak root password was another problem. There were a few other backdoors found later, and eventually they decided to wipe the machine clean and start from scratch - a very good idea given how long the machine had been out of their control.

Thanks to the many folks who wrote in with ideas - you came up with some things I hadn't thought of either. And congratulations to Mandi Walls, who nailed pretty much everything that the cracker did. A beautiful copy of Hacking Linux Exposed, Second Edition will be coming your way.

Also, thanks to Simon Walter who pointed out the 'From Power Up To Bash Prompt' webpage at http://www.tldp.org/HOWTO/From-PowerUp-To-Bash-Prompt-HOWTO.html. If you want to understand the bootup process in more depth, take a look.


[1] This wasn't obvious based on the facts presented, but was offered by me since I already knew the actual cause, in order to narrow the scope of the challenge.

[2] Proof that a free book is much better incentive than a postcard - I shall have to remember this.

[3] A change to the MBR would need to run the kernel in a way different than the default, and really is no different than modifying the kernel or the bootloader. Thus a theoretical MBR change would seem to fit within the other alternatives anyway.

[4] Inirtd ram disks are typically used to help a Linux kernel load needed modules, and thus this fits into the kernel-changes category again.

[5] Becoming a daemon process involves code such as "fork && exit; fork && exit" which creates a child process and kills off the parent processes, such that they have no running parent process, and are inherited by init.

[6] Rucc contained code to daemonize, which is why it's parent is init.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. He launches programs from /etc/inittab quite regularly, or at worst through DJB's supervise. It's a heck of a lot easier than writing your own daemon checking and restarting code. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002

This is the December 05, 2002 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.

previous article
next article