Today, March 5, 2015, the Freckle app experienced an outage of approximately 2:45 hours.
First of all, this sucks—and we're sorry that it happened. Freckle is supposed to save you time—not make you bite your fingernails in anxiety and wear out your ⌘R keys to see if it's back up again.
Here's the sequence of things as they happened:
There was a scheduled emergency server maintenance from our hosting provider Rackspace, which happened from about 1am EST to 5am EST. They installed a critical, as yet not publicly available security update to the Xen virtualization software used in their hosting environment. (Yay, security! Boo, timing in the middle of the workweek!)
After this maintenance was complete, the Freckle servers, as expected were automatically booted back up. However, one of the servers, while booting normally, did not start the nginx service, which, as our main web server software is critical.
The underlying reason why nginx didn't start was that DNS was not working properly—nginx checks SSL certificates and it couldn't resolve one of the hosts needed to verify our main SSL certificate. We don't know why DNS didn't resolve, but it's likely that to the large number of booted servers in the Rackspace datacenter there was a temporary problem with DNS resolution requests.)
This in turn made the Freckle web app not load, with requests timing out. (Same for related apps, like Mac app and iPhone app.
Our monitoring alerted us immediately—however there was a window of time this morning were no one was reacting to the alerts (as a small business, we mostly have one person on on-call duty, and in this case, while alerts did go through, the CTO simply failed to wake up [Thanks, allergies!]). This is 100% our fault, and we'll make changes so someone is actively monitoring any maintenance that happens by our hosting provider.
As soon as alerts were received, we immediately announced the outage and investigated the cause. A few reboots later, and with working DNS resolution, Freckle, and the Freckle apps are now fully up and running again.
Again, this sucks, and it's our fault. We'll make changes so it won't happen again.
Thomas Fuchs, Partner & CTO Freckle Time Tracking
Note that the landing page and help/support site were not affected by this problem.