Zero Downtime Post-Compromise Forensics

(Ping! Zine Issue 11) – The title of this article sounds like an oxymoron, I know. Proper forensics requires downing the server. But often, in small, budget-conscious shops, downtime is unacceptable or undesirable.  While the steps outlined below are no substitute for true forensics, if you (or your employer) cares less about tracking down the bad guy and ensuring it doesn’t happen again than about keeping servers and applications available, these guidelines could help you clean the server and prevent future attacks, often in under an hour. I wouldn’t recommend this path for any server in which sensitive data could be obtained by the badguys (e-commerce, personnel, or classified data). 

Compromised!

In most situations, the first symptom that leads us to suspect (either superstitiously or scientifically) that a server has been compromised somehow is unusual behavior. If we’re fortunate enough to have network intrusion detection [[reference previous article?]], a compromise may manifest itself as unusual and unexpected outbound traffic from the server. Otherwise, unexplained reboots, sluggish performance or disk drives filling up may expose the problem.

“Find and Clean” on Unix Servers

Many of the Unix rootkits (pre-made intrusion/privilege escalation tools made available by smart hackers to script kiddies) include the ability to hide their processes and files. This used to be an obstacle to detection. Modern testing utilities such as chkrootkit (http://www.chkrootkit.org) have the ability to detect hidden processes and alert you to them. I like to use chkrootkit as a first step.  If chkrootkit reports an infection, read the chkrootkit code itself to see exactly what it found and where it was looking when it found them. This allows you to gather more information (timestamps, filenames, etc) and examine the intruder’s droppings more carefully at a later time.

You can also assist yourself by making sure that you always have known-clean binaries of the major system tools for your OS available to you – keep a copy of ps, ls, ldd, top, df, du, w, awk, cut, echo, egrep, find, head, id, netstat, strings, uname and sed in an accessible location, preferably another server, accessible by FTP or wget. This way, if you suspect intrusion, you can download them to your server and get results which are not likely to be filtered through your badguy’s process and file hiders.

“Find and Clean” on Windows Servers

Some people like to start Windows server investigations by running Housecall, from TrendMicro (http://www.trendmicro.com). I prefer to check the usual suspects first. Open Task Manager.  See anything you’re not used to seeing? Maybe a slightly different spelling of an expected Windows process?  On one infected server, I noticed a process running called CSRS.exe which I hadn’t remembered seeing before. Googling the process name produced some hits, so I did a Windows search for the filename. Noting the location and timestamps of the directories and files involved, I continued my hunt.

Another useful tool is Fport, from Foundstone (http://www.foundstone.com), which shows which ports are open on a Windows server, and by which process. Compare it to Unix’s ‘netstat –ap’ command.

It’s important to note the timestamps of any files or directories involved in a compromise, even if you’re not doing a full-blown forensic investigation, because timestamps help to narrow down the date and time, and thus the vector, of intrusion. There’s no point in doing any of this work if you’re not prepared to spend the time it takes to determine (and close) the intrusion vector. Why clean the house if you’re going to leave the front door wide open? Keep in mind that often files obtained from archives will retain their original timestamps, and the directory created by the bad guy will often be more useful in reflecting the time of infection.

Typical places to check for backdoors include ANYWHERE in the tree structure of the default website, especially under \inetpub\scripts and \inetpub\iissamples. To prevent abuse of the default IIS website, I typically stop the site, and add a nonexistent host header to it (such as dontgothere.nowhere.com) to prevent it from being exploited if it is later made activate by a badguy. I have seen backdoor programs deposited in \inetpub\scripts, where they are executable later, in case their badguy programs are detected and cleaned by the administrator. 

Aftermath, and Determining How the Server was Compromised

When you find files left by an intruder, don’t simply delete them. Often they leave clues, such as a script which was executed.  Note the timestamps of the files. If the files are readable, note the contents. Keep them in a safe place for further investigation. If you know what he did once he got in, you have a pretty good roadmap off how to clean up after him. 

Tools and running processes left behind may help to indicate the intrusion vector. If the first thing he does when he gets into your server is scan for awstats vulnerabilities on other networks, it’s a pretty good bet that that’s how he got into your server. If you get a report that your server is scanning other networks for LDAP, you’re probably running a vulnerable version of Imail with LDAP turned on, and that LDAP was exploited. 

If you manage to obtain what you feel are valid timestamps, do a search on the server, this time for all files modified on or since the date(s) involved. This often exposes other files left behind by the badguys. Also search the registry (Windows) for the filenames.

If you suspect a Web vulnerability was exploited, use grep (Unix) or findstr (Windows) against your web logs. Search for things like ‘%20’, ‘..\’, ‘wget’, ‘echo’, etc. 

Sometimes file locations can give you a clue to the intrusion vector. For example, files deposited in a cgi directory of a webserver sometimes indicate that a vulnerable cgi script in that directory was exploited. Files deposited in the temp directory of a PHP-enabled webserver could indicate abuse of PHP’s file upload capabilities combined with insecure PHP.INI settings.

Frequently, search engine results on a filename found during forensics will allude to frequently-seen vectors of intrusion for that particular worm, virus or Trojan. Look for phrases such as “Usually spread by weak passwords on remote shares” or “exploits SQL stored procedure vulnerability,” for example, to help isolate the source of the intrusion.

Obviously, this article could not possibly cover every compromise situation, but hopefully I’ve provided a reasonable set of guidelines to help clean an infected server when it is impractical or impossible to perform true cleanup, which would generally require a takedown and rebuild from scratch and involve downtime.  I’ve cleaned dozens of servers using these general methodologies, and nine times out of ten, I’m able to isolate the root vulnerability and prevent re-infection.

Bob is Director of Operations for Irides, LLC in Arlington, Virginia. In what little free time remains, he enjoys playing EyeToy games with his daughters, photography, cycling and Basil Hayden’s bourbon.

Advertisement