We have an application on Heroku that sometimes gets huge influxes of requests at unpredictable times. These are processed as background jobs, and we occasionally encounter huge backlogs in our queue, which results in requests taking many hours to be picked up and processed.
In the past we’ve had to handle these cases manually by monitoring and increasing the number of background workers as we saw fit. This wasn’t sustainable or convenient (especially if it happened in the middle of the night!), so we decided to write a script to automate the scaling of heroku workers based on the current number of queued requests.
To use the Heroku API, you need to get an API key. This is available on the account page of Heroku.
I opted to use the heroku-api gem. To open a new connection:
1 2 3 4 5
To get the current number of workers:
This returns an array of the web and worker dynos currently running on your app (your Dyno configuration).
1 2 3 4 5 6 7 8 9 10 11 12 13 14
There are two different kinds of dynos:
- Web dynos: Handle web processes and serve web requests. They are not used to handle high-latency or long-running requests because these can tie up all web dynos and result in a slow, inconsistent user experience
- Worker dynos: workers running independently from web processes. They take items off a work queue and process one by one.
We were interested in scaling the number of worker dynos when their queue became too large. These dynos are identifiable by their process name (“worker.x”):
Changing the number of workers to ‘x’ is easy:
So using the current queue size, we scale automatically using a script something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Run this script periodically and the number of workers will scale automatically based on the current load. Hopefully, this will allow you to stay soundly asleep the next time your application gets hit with tons of requests at 3am =)