By Bri Hatch.
Summary: This week, Bri shares some more /proc tricks for investigating programs running on your machine.
Last week, I introduced you to a bit of wizardry using the /proc filesystem to recover a file deleted from the hard drive that was still held open by a process on the machine. This week, I'll show you a few other tricks you can use to investigate programs that are running on your machine.
Again, the two tools we'll look at are lsof and the /proc filesystem. Let's say we find a program called 'crackit' running on our system that uses a significant bit of CPU. (Perhaps we noticed our machine was sluggish and singled this process out by looking at 'top' output, for example.)
First, we should always perform a simple ps of the program:
$ ps -f -p 24061 UID PID PPID C STIME TTY STAT TIME CMD axe 24061 1087 99 09:36 pts/9 R 16:06 ./crackit 18277
A quick search for crackit on the hard drive turns up nothing:
$ locate crackit $ find / -name crackit
Ok, so something is fishy here. Either the file has been deleted or the command line has been altered. Let's perform a full lsof of the process to see what's going on:
$ lsof -p 24061 CMD PID USER FD TYPE DEVICE SIZE NODE NAME john 24061 axe cwd DIR 3,5 4096 49430 /tmp/r (deleted) john 24061 axe rtd DIR 3,7 1024 2 / john 24061 axe txt REG 3,5 201840 49439 /tmp/r/john (deleted) john 24061 axe mem REG 3,7 340771 40255 /lib/ld-2.1.3.so john 24061 axe mem REG 3,7 4101836 40258 /lib/libc-2.1.3.so john 24061 axe 0u CHR 136,9 29484 /dev/pts/9 john 24061 axe 1u CHR 136,9 29484 /dev/pts/9 john 24061 axe 2u CHR 136,9 29484 /dev/pts/9 john 24061 axe 3w REG 3,5 39 49446 /tmp/r/john.pot (deleted) john 24061 axe 4r CHR 5,0 29477 /dev/tty john 24061 axe 5r REG 3,5 16157 49435 /tmp/r/password.lst (deleted) john 24061 axe 6uW REG 3,5 57 49447 /tmp/r/restore (deleted)
ps duplicates a lot of this info, such as the tty (/dev/pts/9) that's in use, the process id, and username. Hmmn, the command name here, however, is 'john' instead of crackit. lsof retrieves the program name from a different location in /proc, rather than from the command line of the process, which can be faked. Looks like someone is trying to stealthily run the damned-fine password cracker John the Ripper.
The lsof output shows that this program was being run in the /tmp/r directory, which has been deleted since it was created (along with all the files therein, such as the john binary itself, john.pot, password.lst, and restore). We won't see the /tmp/r directory if we go looking for it:
$ ls /tmp/r ls: /tmp/r: No such file or directory
However, we could still recover those files. Let's copy them to /root/mystery by accessing the filedescriptors themselves from /proc, just like we did last week to recover our precious movie file:
# mkdir /root/mystery # cd /root/mystery # cp /proc/24061/fd/3 ./john.pot # cp /proc/24061/fd/5 ./password.lst # cp /proc/24061/fd/6 ./restore
Ok, we have a copy of the files in use by the password cracker, but don't have a copy of the password cracker itself. Sure, we probably don't need it, but it's nice to be able to see if the attacker has improved this one by disassembling it or running it under a debugger. However the binary that's running isn't open on a file descriptor, so we can't copy from /proc/PID/fd. However it is available via a different file in /proc:
# cp /proc/24061/exe ./john
This 'exe' file is a copy of the executable itself, so we've snagged it too.
Ok, we now have a copy of the program and it's support files, despite the attacker's attempts to deny them to us. Next week, we'll investigate this a bit further, including an important answer to the question 'So if the files are deleted, how will the cracker get access to the decrypted passwords when they're cracked?' Don't forget, as soon as the password cracker process is done, the /proc/PID directory that belonged to it will disappear, taking those files with it....
 Altering your command line is trivial, just edit your argv array and ps/lsof/etc will all report whatever you want. To test this for yourself, you could simply run this command line perl script, which changes the $0 variable and then looks itself up in ps:
$ perl -e '$0 = "Fake command line"; system "ps -f $$"' UID PID PPID C STIME TTY STAT TIME CMD arioch 24403 24126 0 09:59 tty4 S 0:00 Fake command line
See how easy that was?
Bri Hatch is Chief Hacker at Onsight, Inc, and author of Hacking Linux Exposed and Building Linux VPNs. He has 'ps' aliased to 'lsof' on half his systems, and doesn't mind the information overload one little bit. Bri can be reached at firstname.lastname@example.org.
Copyright Bri Hatch, 2002.
This article was first published here in ITworld.com Inc., 118 Turnpike Rd., Southborough, MA 01772 on 14-May-2002.