We're preparing a larger feature update, part of which is includes a complete rewrite of our budget and budget notification engine.

In order to roll out the (secret!) feature in the future, we first need to see how the new engine for this works in production to make any adjustments neccessary to keep performance up—fast response time is our number one goal in Freckle and any feature we add can't make things slower.

The slow migration was caused by having to create a new database row for every existing budget in Freckle, for hundreds of thousands of your projects. Each of these updates needed to recalculate some project statistics, like the number of billable or unbillable hours. Despite having run the migration on test servers against backups of the database, the production database was unexpectedly slow in running the queries neccessary to fill the new table.

Additionally, during the migration, the database backup kicked in (we're doing hourly backups), majorly slowing down DDL (data definition language) commands that deleted some columns which are no longer required.

In the future, we'll make sure that we get more realistic estimates of computationally intensive migrations like this one and if neccessary roll out out new features without any (unexpected or not) downtime.

We're really sorry for this—it sucks when an app you rely on is down, especially if it's avoidable and due to human error.