Heroku Auto Scale
Here we go… I need to get some Heroku auto scaling magic going on.
Problem: Heroku doesn’t care if a worker process is busy. When scaling down workers on Heroku, jobs that are off the queue and in process may be killed. However, I don’t want to have unused workers just sitting there doing nothing but looking for work and racking up my (company’s) bill.
Of many possible solutions, this is one:
I’m assuming if you’ve found this post, you’ve been googling the hell out of “heroku auto scale” and have already learned a lot, if not, go forth and google, read, learn, then come back.
Heroku’s cedar stack can launch processes defined in a Procfile. When you use the heroku scale worker=3 command, Heroku spins up or down as many of your dynos is as needed to hit that number. The gotcha is that Heroku also doesn’t care nor is there a way to tell them that a dyno is busy, and should only be scaled once it’s finished with it’s job. So if you just monitor queue size and try to scale based off of the number, you’ll kill jobs before they’re finished. So I decided that if I added in the ability to give a resque worker a specific tag when launched (see my fork of hone’s heroku branch) I would be able to tell which workers were working on which of heroku’s dynos.
I created a number of grouped processes, worker1 worker2 worker3 … Then passed the name of the process to my resque:work call as the worker’s tag. I also added in the ability to put queues in the rake task arguements, which leaves me with Procfile lines like these:
worker1 bundle exec rake resque:work[“worker1”,”tasks”]
worker2 bundle exec rake resque:work[“worker2”,”tasks”]
Where the managed dyno name is worker, and I can start and scale 2 dynos that process the “tasks” queue.
First: Get standard vanilla resque working
Then: Use unicorn as your webserver for your ‘web’ process
Next: Set up an initializer for unicorn:
Then: Create or edit your Procfile
Next: Create a rake task that manages the workers, scaling up if needed.
Is it messy: true
Is super efficient: false
Does it do it’s job so the previous 2 stop mattering: true