Maintenance Schedule for October 18th, 2014

Production systems upgrades
Debian Linux 7.7 is scheduled for release on Saturday, October 18th, 2014. As a result, we'll be going into maintenance mode on that day. Stop foaming and put the phone down! We are NOT expecting service interruptions, since we'll be following a strict update and testing schedule, published in the table below. We have chosen an out-of-increased-load time for our maintenance, to ensure that we don't accidentally interrupt any services during normal office hours.
In (a not so short) summary, we'll be updating non-essential internal systems first, testing them, then essential internal systems (and testing). Up to this point, we'll not even touch the customer facing servers. At this point all leaves-of-absences will be recalled, non-essential personnel will be evacuated out of the building...oops wrong movie :-P
If (fingers crossed) everything is operating satisfactory, we'll update our secondary servers first, fail-over internally the services to them (customers and their visitors will still hit the primary systems), test each of the websites we host and perform essential testing to other services (like FTP and email). Fix any problems that come up, then fail-over customer facing services to the secondary systems.
After all the services have completely failed over to the secondary systems, we'll start upgrading the primary systems. As soon as that finishes, we'll fail-over internally the services back to the primary systems (customers still hitting secondary systems), test and apply fixes if problems occur, then finally fail-over customer facing services back to the primary systems and re-test. At this point the maintenance is completed and everybody can go on about their lives.
Why such a fuzz about this particular upgrade? Aren't you always performing upgrades?
We are constantly performing updates, and failing over services back and forth between primary systems and secondary systems. Just this past week we performed a full database cluster upgrade without publishing an announcement, since we have tested the upgrade on our testing lab beforehand. This particular upgrade brings an update to the Linux kernel though, and we can't properly test that, unless it's on our production systems.
Again we remind all our clients that this is NOT a service outage announcement. We are NOT expecting any service interruptions and will go out of our way to ensure we don't get any interruptions.
Schedule (GMT+2 timezone)
Scheduled time Actual time Current status Duration Notes
8:00pm 8:00pm COMPLETED Maintenance starts. Local on-premises Debian mirror expected to be up to date.
8:10pm 8:10PM COMPLETED 20mins Local Debian mirror updates itself
8:30pm 8:30pm COMPLETED 20mins Non-essential systems update
8:50pm 8:50pm COMPLETED 20mins Essential systems update. No customer facing servers yet.
9:10pm 9:10pm COMPLETED 20mins Making sure everything works as expected before moving on to the production servers.
9:30pm 9:30pm COMPLETED 30mins Secondary systems upgraded and tested. Operating on reduced capacity.
10:00pm 9:40 COMPLETED 10mins Internal fail-over to secondary systems. All customer websites and services tested.
10:10pm 9:50pm COMPLETED n/a Public services fail-over to secondary systems.
10:15pm 9:53pm COMPLETED 30mins Primary systems upgraded and tested.
10:45pm 10:00pm COMPLETED 10mins Internal fail-over to public systems. All websites and services will be tested (yet again!) before moving on.
10:55pm 10:10pm COMPLETED n/a Public services fail-over to primary systems. Operating capacity returns to normal.
11:00pm 10:11pm COMPLETED n/a Stepping out of maintenance mode.
Color key:
Normal operation Slight possibility of someone getting fired Nail-chewing, nerve-racking, de-caf time. If it's going to hit the fan, it's happening now.
We'll keep this post updated, if a release re-schedule or anything else out of our control (like an asteroid headed for Earth) comes up. Or if we spot a typo or two.