Summary: This week, Bri shares some more /proc tricks for investigating programs running on your machine.

Last week, I introduced you to a bit of wizardry using the /proc filesystem to recover a file deleted from the hard drive that was still held open by a process on the machine. This week, I'll show you a few other tricks you can use to investigate programs that are running on your machine.

Again, the two tools we'll look at are lsof and the /proc filesystem. Let's say we find a program called 'crackit' running on our system that uses a significant bit of CPU. (Perhaps we noticed our machine was sluggish and singled this process out by looking at 'top' output, for example.)

First, we should always perform a simple ps of the program:

$ ps -f -p 24061
UID        PID  PPID  C STIME TTY      STAT   TIME CMD
axe      24061  1087 99 09:36 pts/9    R     16:06 ./crackit 18277

A quick search for crackit on the hard drive turns up nothing:

$ locate crackit
$ find / -name crackit

Ok, so something is fishy here. Either the file has been deleted or the command line has been altered.[1] Let's perform a full lsof of the process to see what's going on:

$ lsof -p 24061
CMD      PID USER   FD  TYPE DEVICE    SIZE  NODE NAME
john  24061  axe  cwd   DIR    3,5    4096 49430 /tmp/r (deleted)
john  24061  axe  rtd   DIR    3,7    1024     2 /
john  24061  axe  txt   REG    3,5  201840 49439 /tmp/r/john (deleted)
john  24061  axe  mem   REG    3,7  340771 40255 /lib/ld-2.1.3.so
john  24061  axe  mem   REG    3,7 4101836 40258 /lib/libc-2.1.3.so
john  24061  axe    0u  CHR  136,9         29484 /dev/pts/9
john  24061  axe    1u  CHR  136,9         29484 /dev/pts/9
john  24061  axe    2u  CHR  136,9         29484 /dev/pts/9
john  24061  axe    3w  REG    3,5      39 49446 /tmp/r/john.pot (deleted)
john  24061  axe    4r  CHR    5,0         29477 /dev/tty
john  24061  axe    5r  REG    3,5   16157 49435 /tmp/r/password.lst (deleted)
john  24061  axe   6uW  REG    3,5      57 49447 /tmp/r/restore (deleted)

ps duplicates a lot of this info, such as the tty (/dev/pts/9) that's in use, the process id, and username. Hmmn, the command name here, however, is 'john' instead of crackit. lsof retrieves the program name from a different location in /proc, rather than from the command line of the process, which can be faked. Looks like someone is trying to stealthily run the damned-fine password cracker John the Ripper[2].

The lsof output shows that this program was being run in the /tmp/r directory, which has been deleted since it was created (along with all the files therein, such as the john binary itself, john.pot, password.lst, and restore). We won't see the /tmp/r directory if we go looking for it:

$ ls /tmp/r
ls: /tmp/r: No such file or directory

However, we could still recover those files. Let's copy them to /root/mystery by accessing the filedescriptors themselves from /proc, just like we did last week to recover our precious movie file:

# mkdir /root/mystery
# cd /root/mystery
# cp /proc/24061/fd/3 ./john.pot
# cp /proc/24061/fd/5 ./password.lst
# cp /proc/24061/fd/6 ./restore

Ok, we have a copy of the files in use by the password cracker, but don't have a copy of the password cracker itself. Sure, we probably don't need it, but it's nice to be able to see if the attacker has improved this one by disassembling it or running it under a debugger. However the binary that's running isn't open on a file descriptor, so we can't copy from /proc/PID/fd. However it is available via a different file in /proc:

# cp /proc/24061/exe ./john

This 'exe' file is a copy of the executable itself, so we've snagged it too.

Ok, we now have a copy of the program and it's support files, despite the attacker's attempts to deny them to us. Next week, we'll investigate this a bit further, including an important answer to the question 'So if the files are deleted, how will the cracker get access to the decrypted passwords when they're cracked?' Don't forget, as soon as the password cracker process is done, the /proc/PID directory that belonged to it will disappear, taking those files with it....

NOTES

[1] Altering your command line is trivial, just edit your argv array and ps/lsof/etc will all report whatever you want. To test this for yourself, you could simply run this command line perl script, which changes the $0 variable and then looks itself up in ps:

$ perl -e '$0 = "Fake command line"; system "ps -f $$"'
UID        PID  PPID  C STIME TTY      STAT   TIME CMD
arioch   24403 24126  0 09:59 tty4     S      0:00 Fake command line

See how easy that was?

[2] http://www.openwall.com/john/

Bri Hatch is Chief Hacker at Onsight, Inc, and author of Hacking Linux Exposed and Building Linux VPNs. He has 'ps' aliased to 'lsof' on half his systems, and doesn't mind the information overload one little bit. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2002.

This article was first published here in ITworld.com Inc., 118 Turnpike Rd., Southborough, MA 01772  on 14-May-2002.