The websites were all down since Sunday morning. There were no raids, no chaos, no legal trouble, nothing like that. Here’s a summary of the server events on the weekend:
According to Peer1 (who hosts our servers), Harbour Center (a landmark building nearby) was conducting a power/generator test on Saturday June 25 that went very wrong, so that Peer1 (with all our servers) lost power for 20 minutes. All our servers should have rebooted automatically after the power was restored, however, a few critical servers didn’t reboot properly and didn’t come back online, so couldn’t be reached for corrective measures or a reboot.
Our back-up webmaster had access to all the root passwords, so there wasn’t a missing password problem. Since the servers didn’t boot no logins were available.
Our webmaster wasn’t in communication for the weekend and didn’t hear about the problem until 3:30am on June 27, and by 10am managed to get to a computer terminal and look at the situation, and contact Peer1. They didn’t inform him of the earlier power failure, but he had Peer1 power-cycle the servers in question. It didn’t seem to work, so he began his return to Vancouver so he could directly access the servers at Peer1.
The servers came back online around 6pm Tuesday night.