Sorry about the (15 hours of) downtime.
We have protection in place to guard our server against DDoS. Although the one that brought a lot of sites down a few weeks back targeted upstream providers (i.e., the folks who connect your server to the wider web), and we likely still could've been brought down had they targeted ours.
What we had today were two incidents that may or may not have been related:
- We had a table crash in the database that we could not repair
- Then we suffered a massive brute force SSH attack in the midst of trying to get the site back up
Not sure if the two were related or if it was just someone kicking us while we were down (quite literally).
I was offline for the first 9 hours of the broken DB so didn't even know it was going on until then. We repaired the table, but it kept re-breaking. So we had to restore the site from a backup, which took about 3 hours (big DB).
While we were in the middle of that, we suffered the brute force attack, which brought the whole server down and interrupted the backup (so we had to restart the backup, after we restarted the server).
We're still not sure what caused the DB table to crash, but I've never seen that happen in 8 years of running this site (random DB crash for no obvious reason) and it doesn't seem to be a common problem, so that one's hopefully a 1-in-a-million black swan. But I will ask our devs to investigate regardless. The brute force attack we have some protections against now (all our passwords are super long and complicated; it would take hundreds of years of brute forcing to crack them... and credit card data is additionally stored off-site with our payment gateway anyway, so even if someone got in, there's nothing for them to get even if you're a customer with us).
Anyway, we're back up now and all seems to be running normally again. We'll continue to keep a close eye on things this week to make sure we don't see any repeats, though.
Chase
