The first day of the new year has passed. It's time for us (after a long day, it is 3:25am at the time of writing) to reflect on what we accomplished today. The list starts from the most important and goes to the least important.
- Performed a critical systems upgrade: Our systems were vulnerable to the CVE-2014-8142 vulnerability. This affected PHP by allowing an attacker to execute code remotely. The patch to fix our systems came out on December 31st, and we immediately started patching "background" (systems not in use directly by our clients). Today we patched the remaining systems, while testing that everything works out as expected. At the time of writing, the announcement about the patch has not been made available on Debian's page. Yes patched up systems before the announcement for the patch is made.
- Identified and fixed a bug affecting synchronization of certain hosted websites to backup systems: The bug was identified immediately before switching from the primary systems to the secondary systems, to proceed with the upgrade. The bug affected a small number of hosted clients. The synchronization between the primary and the secondary systems failed to execute, causing the backup systems to have an old website copy. Meaning if we switched from the primary systems to the secondary systems as is, it would be like going back in time.
- Identified and rectified a mistake that caused certain hosted websites to fail enabling their automatic backups: Each hosted website has 7 backup pairs (website files + databases). A daily backup pair, kept for a week. Not only on the primary servers, but also on the secondary servers. The secondary servers do NOT copy the backup files from the primary servers, they take automated backups of the files that they have a copy of. Correlating to 2 from above, if said certain websites failed for any reason, the results would be catastrophic.
- Identified and reported a bug in a WordPress theme that we have been waiting on a fix for, for a long...LONG time: The bug is caused by a typo in one of the theme's files. To protect the guilty, we will not publicly name the theme, nor the authors of the theme. The bug does not (severely) affect the theme's functionality, but it kept nagging our servers. The bug was first spotted a few months back, and we have been waiting on the theme author to do what every responsible author would do: fix it. We like to keep our servers happy, so we eventually caved in, in their requests for peace. No, we haven't lost our minds. Yet.
- Added a few hosts on our permanently banned IP list: When no means no, you don't come back asking for more. Said hosts were flagged subsequently temporarily banned by our security systems. Those systems don't ban an IP for a day, but a number of days instead. Banned hosts are still monitored for bad activity. If the host continues to attempt to break in, the ban is renewed for another X number of days. We like to keep the number of days a secret for obvious reasons. IPs earn a place on our List of Shame, when they have had multiple security incidents logged against them. We keep a record of ALL past security incidents, and are not afraid to use it. At the present time, our permanently banned list numbers about 4,000,000 IPs (four million IPs).
That's all for today. Time to trick our bodies into thinking we are asleep. Didn't you know? deZillium administrators aren't chosen based on their knowledge, they are chosen based on their dedication to their jobs. And how long they can go before passing out.