Summary: A non-standard kernel patch provides some insight into a cracker's activities.

The /proc filesystem is populated by files that are hooks into the kernel. For example running cat /proc/version will let you see the version of the Linux kernel that's running, who it was compiled by, and when. The /proc/### directories contain info about the processes that are running on your system. The files under the /proc/sys directory can let you view and, in some cases, change characteristics of the kernel at run time.

So what is the purpose of /proc/config? Heck - I'd never even heard of /proc/config until this issue came up. /proc/config is only available in your kernel if you apply a patch and enable it using CONFIG_PROC_CONFIG when you compile.

/proc/config is actually pretty handy -- it shows you the compile-time configuration settings for the kernel:

  $ head /proc/config
  CONFIG_X86=y
  CONFIG_UID16=y
  CONFIG_EXPERIMENTAL=y
  CONFIG_MODULES=y
  CONFIG_MODVERSIONS=y
  CONFIG_KMOD=y
  CONFIG_M386=y
  CONFIG_X86_L1_CACHE_SHIFT=4
  CONFIG_RWSEM_GENERIC_SPINLOCK=y
  CONFIG_X86_PPRO_FENCE=y

Say you want to upgrade to the next available kernel. Your current kernel works fine, so you'd like to use the same parameters, but you accidentally lost your original .config configuration file. Simply cat /proc/config > /usr/src/linux/.config and you're ready to go.

Anyway, a friend has a computer that has /proc/config enabled. He runs a file integrity check every night out of cron.[1] One day he found that /proc/config had been changed.

The first thing I thought to myself is that he must be crazy. Checking all of /proc with a file integrity tool is pretty painful - one might even say pointless. Since the files there aren't real files, they're not as static as real files. The timestamp of files in /proc, with a few exceptions, is always equal to the current time. The file size of most files is reported as 0, even when they do present data when read.

  $ date
  Thu May 22 14:00:00 PDT 2003
  $ ls -la /proc/cmdline
  -r--r--r--    1 root     root            0 May 22 14:00 /proc/cmdline
  $ cat /proc/cmdline
  BOOT_IMAGE=2.4.18 ro root=305 apm=on

This must lead to huge numbers of false positives with file integrity checkers, unless you turn off timestamp tests and such. Some of these files are constantly changing, such as /proc/{stat,uptime,PID}. He assures me that /proc has lots of false positives, which he's cleaned up by liberal use of post processing with egrep...

But, my worries aside, it seemed that indeed /proc/config had been changed. He'd compiled his kernel on his own, and checked his kernel sources in /usr/src/linux, and nothing seemed amiss. Comparing his old .config kernel compilation configuration file with the output of /proc/config clearly showed something fishing was going on:

  $ cd tmp
  $ scp backupserver:/path/to/.config oldconfig
  $ sort oldconfig > oldconfig.sort
  $ sort /proc/config > curconfig.sort
  $ diff oldconfig.sort curconfig.sort
  > CONFIG_RARBOG_PROC_BD=y
  > CONFIG_RARBOG_UDP_BD=y
  > CONFIG_RARBOG_UDP_PRT=18872
  > CONFIG_RARBOG_HIDE=y

Those three lines represent kernel configuration parameters that were not in his original .config file.[2] While I have no idea what RARBOG is, the _BD bits makes me think "backdoor", UDP_PRT makes me think "UDP port", and _HIDE is non too encouraging.

There wasn't any process actively listening on port 18872,[3] however it's quite possible for the kernel to be watching for packets on that port without actively binding it.[4]

We scoured the machine looking for the kernel modifications that allowed the CONFIG_RARBOG, but couldn't find them on the filesystem, or by looking through the unallocated sectors of the disk. Most likely the cracker compiled the kernel elsewhere and uploaded it. (It was polite that he used the same configuration file for maximum compatibility...)

If anyone has a copy of this RARBOG kernel patch, I'd love to see it. Regardless, the machine has been taken off line and is currently having the software re-installed from clean sources.

And hopefully this'll teach him not to administer his machine from a university network over telnet any more...

NOTES:

[1] Running a file integrity tool out of cron isn't the best method - a cracker who has gotten root access can easily munge with your database or program invocation to hide his activities. The most sure method to run a file integrity tool is at the console, booted from secure cd/floppy media, in single user mode. However, it's better than nothing.

[2] For completeness, we ran the same checksum algorithms on the oldconfig file to verify that they matched the original (pre-intrusion) /proc/config checksums.

[3] We tested lsof locally, which isn't the most reliable thing to use when the kernel is suspect, and then nmap from the neighboring machine.

[4] Some port scan detectors work in this fashion - rather than binding an unused port and accepting connections they bind raw sockets and watch for any inbound accesses to ports that are not currently bound to processes.

Bri Hatch is Chief Hacker at Onsight, Inc and author of Hacking Linux Exposed and Building Linux VPNs. Given his wetware's shortage of /proc/kcore and the excessive number of /proc/interrupts, it's amazing he ever gets any real work done. Bri can be reached at bri@hackinglinuxexposed.com.

Copyright Bri Hatch, 2003

This is the May 22, 2003 issue of the Linux Security: Tips, Tricks, and Hackery newsletter. If you wish to subscribe, visit http://lists.onsight.com/ or send email to Linux_Security-request@lists.onsight.com.