You're Hired! You're Fired!

NOTE: I originally authored this article for Big Astronaut's blog, which Lance Ennen posted on my behalf. I am sharing here as well for those following my blog directly.

What is the #1 thing everyone complains about with regards to using Heroku as a Rails Web Hosting platform? 

If you guessed "price" you would not be far off from the truth. In fact, Heroku gives a lot of bang for the buck to get websites started and on the way to mega-stardom, but the inability to scale dynos up and down automatically based on web server load can be such a boon when you know the site is a ghost town at night not worth having 5 dynos fired up at $34/month each.

Enter HireFire: a dyno manager for Heroku, allowing you to autoscale your web and worker dynos, saving on server costs at nighttime and handling greater traffic during the day. It can integrate directly with NewRelic to get response times, request rates, and Apdex as data to use for scaling Heroku dynos automatically.

Testing

Very recently, I experimented with HireFire.io on a Staging web server for a Chicago healthy gourmet food site called Factor 75 (highly recommended approach to testing the waters before taking the plunge). 

The first step was to configure HireFire for Web Dynos with a minimum number of dynos set to 1 and a maximum set to 5. Additionally, it was configured to scale up when response time (obtained From NewRelic) surpassed 950ms and scale down when response time went below 700ms.

Next, I ran the following command to try and overwhelm the single dyno that was originally configured for the Heroku Staging application:

ab -v 1 -c 5 -n 1000 http://factor75-branch.herokuapp.com/menu

It caused HireFire to scale the Heroku application gradually until there were 5 dynos. After the command finished and a little bit of time passed, HireFire scaled the Heroku application back down

The result was a great success yielding both better traffic handling and better cost saving depending on the traffic situation.

Lastly, I experimented with enabling HireFire for Worker Dynos, and it scaled them up to 2 and down to 0 depending on whether the Sidekiq background processing queues got populated with jobs. The configuraiton stipulated adding an additional dyno for every 15 queued jobs waiting to be processed. Again, the result was a great success, yielding $0 cost for worker dynos the majority of the time.

Recommendation

It would be good to start with a minimum of 2 dynos to avoid Heroku's single dyno sleeping policy and support 2 concurrent users to boot. The maximum can be set to 5 dynos to support spikes of 5 concurrent users (the number may be increased further if there is higher traffic and the web application owner can afford more dynos). 

Next, it would be good to set the Upscaling Sensitivity to LV1 in order for the website to scale up and service users immediately when load is increased and the need arises. It would also be good to set the Downscaling Sensitivity to LV5 to give the server time to prove it needs to scale down before doing so, thus avoiding a jittery experience for website users.

I configured the HireFire Upscale Quantity to 2 to ensure handling sudden spikes with more fire power, yet configured the Downscale Quantity to 1 to scale it down gradually when the extra server load is gone.

With regards to timeouts between dyno scaling, 3 minutes (minimum) is good for upscaling and 12 minutes is good for downscaling.

I recommend a maximum response time threshold of 950ms as that is about the point when user experience starts to degrade. As for the minimum response time threshold, 700ms should work well leaving some room to downscale gradually.

Last but not least, time boundaries can be very useful to lower the minimum number of dynos at night (e.g. 11pm-7am) to 1, thus running the website completely FOR FREE during the night.

Pricing

With regards to pricing, HireFire costs about $10 a month. Given that it scales Heroku Web dynos from 5 down to 1 and Worker dynos from 2 to 0 at nighttime (8 hours), it would save at least 33.3% over a month on dyno cost. That can add up to roughly $70 in cost saving for a 5 web dyno / 2 worker dyno setup. Trading that for HireFire's $10 a month fee results in a $60 net saving. And, that is the pessimistic estimate. In actual use, HireFire should result in much greater savings.

Gotchas

Note that when a Heroku dyno is added, it takes about a 20 seconds to boot up according to data captured from Heroku logs.

Also, HireFire warns about hitting the Heroku API rate limit of 1200 calls an hour when enabling the "Restart Crashed Dynos" feature on a web application. It costs about 60 API calls to Heroku per hour, so with one application utilizing HireFIre under a Heroku client account, HireFire will work well with Heroku with regards to using the "Restart Crashed Dynos" feature. It is recommended for the sake of ensuring minute by minute restart of crashed dynos during emergencies, which is faster than Heroku's increasing intervals restart policy. 

HireFire makes Redis connections to check the number of jobs in Sidekiq queues. It is important to ensure Redis can handle these extra connections in addition to the workers connecting to do work. For example, with Redis To Go Medium, which supports a maxclients number of 512, HireFire has plenty of room to connect.

Conclusion

HireFire did work as advertised, scaling Heroku dynos up within a second of load surpassing the maximum response threshold. It also did work as advertised with regards to scaling Heroku dynos down once the load has let up, resulting in great savings, at least 33% if the website gets light traffic at night.

In other words, go ahead and HIRE HireFire.io so you'd finally FIRE your DevOps team!