Investigate compromised servers
This article lists the tools that are available for performing an analysis of a
compromised server. (Cleaning the compromised server is outside of its scope.)
Using these tools helps you determine the following information:
- The point of entry
- The origin of the attack
- Which files are compromised
- The level of access that the attacker obtained
- The audit trail of the attacker's footprints
Many different types of compromises can exploit a Unix® server. Attackers
might launch a brute force attack, guess a weak password, or attempt to use
known software vulnerabilities in the hope that the server isn't on a regular
patch schedule. It's important to understand how the machine was compromised to
determine the extent of the damage to your server and other hosts
that are accessible to the compromised machine.
For most root-level compromises, the most straightforward recovery approach is
to perform a clean installation of the server and restore any critical data from
backups. However, until you know the entry point of the compromise, this step
might not be sufficient because you need to understand the compromise so that
you can properly close the security hole.
Document the attack
When notified that a system under your control might be compromised,
ensure that you obtain as much information as possible from the reporter,
including the following items:
- How the initial problem was found
- The estimated time that the compromise occurred
- Whether the server was modified after the compromise was detected
- Anything else that the reporter says that is important
Important: If you plan to involve law enforcement, it is imperative that you
take no additional actions on the server. The server must remain in its
current state for evidence collection purposes.
If you choose to proceed with the investigation, document anything that you find
on the server. This step can be as simple as copying and pasting a command and
its results.
Investigation tools
In some compromises, the attacker manages to delete all important log files to
hide their tracks. However, this doesn't always occur. As a result, the log
files contain valuable clues about what the attacker did to the server. The
log files might also help you determine if the attack was a basic web hack or a
root-level compromise. Use the commands in this section to find clues to
help you unravel the extent of the compromise.
last command
The last
command lists the sessions of users who recently logged in to the
system. Its output includes the timestamps and hostnames and indicates whether
the user is still logged in. If an odd Internet Protocol (IP) address appears in
the output, you can cross-reference it against a brute force Secure Shell (SSH)
attack in the /var/log/messages or /var/log/secure directory. This step
might indicate how the attacker gained entry, what username they used to gain
access, and if they were able to escalate their privileges to root
.
ls -lart command
The ls -lart
command outputs a time-ordered list of files and directories that
you can correlate against when the compromise occurred. This output can
help you determine what the attack added or removed from the system.
netstat -na command
The netstat -na
command displays the current listening sockets on the machine.
Running this command might reveal any back doors that are listening or errant
services that are running.
ps -wauxef command
This command helps you track down any errant processes that are listening and
shows other odd processes (for example, the user www
running a Bash process).
You can also run the command lsof |grep <pid>
to find more information about
open files that a process is using. Concurrently, running cat /proc/<pid>/cmdline
might also tell you where the file that controls a process
exists.
bash_history command
The history file is often the Rosetta stone of tracking down what took place
during a compromise. Using the bash_history
command to look through the user's
.bash_history
file often shows exactly what commands they executed, what
malicious programs they downloaded, and the directories on which they focused.
top command
Malicious processes often cause central processing unit (CPU) contention issues
within the environment, and therefore appear near the top of the list of
processes. Use the top
command to display this list. When you're tracking down
a compromise, consider any processes that cause CPU contention issues
suspicious.
strace command
When you run the strace -p pid
command on a suspicious process, the strace
command might yield important insight into what the process is doing.
Other tools
The preceding commands might not provide many clues regarding what occurred
during the attack. If this is the case, you can use more specialized tools.
Important: Before you use the tools in this section, you should confirm that
the binaries you are using to investigate are not trojanned versions.
Trojanned versions can perform tasks on behalf of the attacker, such as omitting
information that might reveal what the compromise was trying to accomplish.
Run the following command to verify that you have a good working set of tools:
rpm -Va
Verifying a package compares information about the package's installed files
with the metadata for the package that the RPM Package Manager (RPM)
database stores. Verification compares information about the size, MD5 sum,
permissions, type, owner, and group that is associated with each file. The
output displays any discrepancies.
Important: Flagged packages in the following directories might indicate that
you are using a trojanned version of the binary, and therefore you cannot trust
its output:
/bin
/sbin
/usr/bin
/usr/sbin
The following example shows a trojanned file:
S.5….T /bin/login
rpm -qa command
You can use the command rpm -qa
to show recently installed packages in
chronological order. However, in the case of a root compromise, the rpm database
might also be compromised.
lsattr command
If the attacker gains root access and trojans certain binaries, they
might make those binaries immutable so that you cannot reinstall clean versions
of them. Check the following directories:
/bin
/sbin
/usr/bin
/usr/sbin
The following example shows a file that an attacker has made immutable:
-------i----- /bin/ps
Under normal circumstances in these directories, the rules should all look similar to:
------------- /bin/ps
find command
find
is a Unix tool that can be critical in finding recently modified files.
For example, you can find files modified within the past five days by
running the following command:
find / -mtime 5
Common directories for web exploits
Check the following world-writable directories to which Apache® commonly
writes temporary files:
ls -al /tmp
ls -al /var/tmp
ls -al /dev/shm
Look for any files that you don't recognize or that look suspicious. Be on the
lookout for hidden files and files that have execute permissions.
If you have set permissions for directories on your website to 777, check those
as well.
Find the point of entry
If you find helpful information by using the tools in the previous sections, you
might also have a timestamp for when the hacker installed the malicious file or files
on the server.
You can use that timestamp to review your website's access logs for suspicious
entries that were added during that time period. If you find something
suspicious, you can cross-reference it with the location of the malicious files
to narrow down the point of entry.
While most of compromises come from exploitable code within your
website, you cannot rule out other entry points. Ensure that you review
/var/log/* for anything that appears suspicious during the reported time
frame.
Example investigation
The example investigation in this section demonstrates the process you
should use when investigating a suspected root-level compromise.
Identify the type of attack
Verify whether it was a basic web hack or if the attacker really gained root privileges.
In most cases, the attack is a simple web hack that you can safely clean.
-
Run the following commands to determine if the attacker gained root privileges:
lsattr /usr/sbin | less
lsattr /usr/bin | less
lsattr /bin | less
lsattr /sbin | less
-
Look for modified attributes, such as binaries that have been set to immutable.
Output:
s---ia------- /sbin/shs
When you use the
strings
command on that file, you see that it's a backdoor
shell.
Check if the attacker cleaned their tracks
In many cases, attackers are inexperienced or sloppy and have not erased their
tracks. Use the following steps to check if the attacker has left clues:
-
Verify that all of the user accounts in
/etc/passwd
have a valid shell by
running the following command:cat /home/$USER/.bash_history
-
Retrieve the root user's history by running the following commands:
history
cat /root/.bash_history
In this example, the output from the /root/.bash_history
command reveals that
the attacker performed the following actions on the server:
- Downloaded malicious tools to serve up through Apache® in /var/www/html/*.
- Installed Internet Relay Chat (IRC) tools and other tools in
/var/tmp/.ICE-unix. - Modified the root crontab to re-download the malicious tools if someone removes
them from the server (* * * * * /var/tmp/.ICE-unix/update >/dev/null 2>&1
).
Check for basic web hacks
Up to this point, we have determined that the attack is probably a simple web hack
that you can easily clean without formatting the server.
However, in this example, we know that the attacker gained root privileges. They
also might have exploited phpMyAdmin
. After the backdoor PHP Shell loaded, the
attacker was able to perform a local root exploit to escalate their privileges.
-
Run the following commands to find hidden files and directories in the
world-readable directories to which Apache typically writestmp
files:ls -al /var/tmp |less ls -al /tmp ls -al /dev/shm
-
In this example, the commands return the following output:
drwx------ 3 70 70 4096 Nov 19 02:00 /var/tmp/.ICE-unix
-
If you find items here, you must attempt to track down the entry point so that
you can take down the site, upgrade site code, or otherwise fix the exploitable
code. Step 5 presents a quick way to accomplish this task. However, if the
output of theps -waux
command shows that IRC bots are running, then you can
try to catch where the process is running from by using thelsof
command or
ps -wauxxef |grep <pid>
.
Look for process identifiers that are listening for incoming connections
- Run the following commands to look for process identifiers (PIDs) that are
listening for incoming connections:
-
netstat -natp
: Looks for any suspicious connections that are running on odd
ports -
ps -wauxxef
: Looks for suspicious files, such asbash
running under a
www
context -
lsof <pid>
: Helps to determine where the PID is running fromThe output appears similar to the following example:
tcp 0 0 0.0.0.0:1144 0.0.0.0:* LISTEN 1008/bash tcp 0 1 172.16.23.13:60968 22.22.22.22:7000 SYN_SENT 6860/sshd
In this example, several other SSH-established connections are also running from
high-level ports, as shown in the following example:[root@www tmp]# netstat -natp |grep sshd |awk '{print $4,$5,$6,$7}' 0.0.0.0:22 0.0.0.0:* LISTEN 1046/sshd 172.16.23.13:60986 22.22.22.22:6667 SYN_SENT 6860/sshd 123.123.123.123:22 22.22.22.22:59361 ESTABLISHED 22795/sshd 123.123.123.123:22 22.22.22.22:57434 ESTABLISHED 22796/sshd 123.123.123.123:57139 143.143.143.143:6667 ESTABLISHED 6860/sshd 123.123.123.123:57402 22.22.22.22:6667 ESTABLISHED 6860/sshd 123.123.123.123:22 143.143.143.143:49238 ESTABLISHED 8860/sshd 123.123.123.123:57134 22.22.22.22:6667 ESTABLISHED 6860/sshd 123.123.123.123:56845 22.22.22.22:6667 ESTABLISHED 6860/sshd 123.123.123.123:57127 143.143.143.143:6667 ESTABLISHED 6860/sshd
This output indicates that the attackers are still connected to this machine.
However, you can't see them because they probably modified the binaries
to hide themselves.
Determine the point of entry for the original compromise
Use the following steps to determine the point of entry for the original
compromise:
-
Check
/var/log/[messages|secure]
for brute force SSH attempts. -
Check the Apache access logs and error logs. This step might help determine
which site is exploitable.You should also cross-reference IPs against the logs if you think that there
is a chance that it might have originated from there. This is a quick and
straightforward way to trace down the point of origin.You can quickly check servers that have a large number of web logs by using
the following commands:cd /var/log/httpd for i in `ls * |grep access`; do echo $i && grep wget $i; done for i in `ls * |grep access`; do echo $i && grep curl $i; done
Note: This example searches
wget
becausewget
was in root's history
file under what might have been part of the entry point.
Outcome
In this example, our investigation revealed that an attacker exploited the phpMyAdmin
installation in the /var/www/html
directory, most likely because the version of
phpMyAdmin
installed on the server was severely outdated. Patching phpMyAdmin
on
a regular schedule prevents this situation from occurring.
Updated 10 months ago