How to implement a distributed Redis message processing system by using Redis / Resque in Rails

I’ve written a post introducing how to implement a distributed message processing system in standalone Ruby application. Now I will do the same, but in Rails.

Architecture design

Add dependencies

add it in Gemfile:

1
gem 'resque'

Also, modify Rakefile:

1
2
require "resque"
require "resque/tasks"

Add resque web admin UI:

In Gemfile:

1
gem 'resque-web', require: 'resque_web'

Later you should be able to access it via: http://SERVER:PORT/resque

Define a worker (message consumer)

Add file app/workers/my_worker.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
class MyWorker
@queue = "mymessages"
def self.perform(message)
puts "worker is processing job: #{message}"
# implement your logic here ...
puts "Successfully processed: #{message}"
end
end

Don’t forget to update Rakefile:

1
require_relative "app/workers/my_worker.rb"

Connect to your remote Redis server in your Rails application

1
RAILS_RESQUE_REDIS=YOUR_REDIS_SERVER_IP:PORT rails s -b0.0.0.0

If you want to set up a basic HTTP authentication for your Resque web admin UI, run:

1
RESQUE_WEB_HTTP_BASIC_AUTH_USER=YOUR_USERNAME RESQUE_WEB_HTTP_BASIC_AUTH_PASSWORD=YOUR_PASSWORD RAILS_RESQUE_REDIS=YOUR_REDIS_SERVER_IP:PORT rails s -b0.0.0.0

To load the worker

1
RAILS_RESQUE_REDIS=YOUR_REDIS_SERVER_IP:PORT rake environment resque:work QUEUE=*

Note: there is an additional parameter ‘environment’ which is required by Rails 5 (don’t know why - it just works)

The client side (message producer)

First, you need to copy over ‘MyWorker.rb’ to your client application, e.g. the message producer.

In the appropriate class, e.g. controller / actor class, add:

1
require_relative 'my_worker'

Also, make a call to send message to Redis, which will be consumed by your worker later.

1
Resque.enqueue(MyWorker, your_message)
Share