How to implement a distributed message processing system by using Redis / Resque in standalone Ruby applications

Most Ruby developers are using Rails for their application. Integrating Redis/Resque in Rails are quite easy and there are lots of resources. But standalone applications are still needed. Here is how to use Reids/Resuqe in a distributed standalone Ruby application (without Rails).

Architecture design

Define a processor (message consumer)

File: file_processor.rb

1
2
3
4
5
6
7
8
9
10
11
class FileProcessor
@queue = "file_process"
def self.perform(file)
puts "Processing file: #{file}"
sleep 3 # fake file processing here ...
puts "Successfully processed: #{file}"
end
end

Define a client (message producer)

File: file_processor_client.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
require "yaml"
require "resque"
require_relative "file_processor"
# Configuring (remote) redis server
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(File.join(__dir__, 'config/', 'resque.yml'))
Resque.redis = resque_config[rails_env]
params = ARGV
puts "Send files for processing: #{params.join(" ")}"
params.each do |param|
puts "Sending a file to process: #{param}"
Resque.enqueue(FileProcessor, param)
end

Configurations

File: config/resque.yml

1
2
3
development: 127.0.0.1:6379
test: YOUR_TEST_SERVER:6379
production: YOUR_PRODUCTION_SERVER:6379

File: config/initializers/resque.rb

1
2
3
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(File.join(__dir__, '../', 'resque.yml'))
Resque.redis = resque_config[rails_env]

File: Rakefile

1
2
3
4
5
6
7
8
9
10
11
12
require 'yaml'
require "resque"
require "resque/tasks"
require_relative "file_processor"
namespace :resque do
desc "config resque ENV"
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(File.join(__dir__, 'config/', 'resque.yml'))
Resque.redis = resque_config[rails_env]
end

Start worker

Run:

1
rake resque:work QUEUE=*

Start client to send messages to Redis

1
RAILS_ENV=production ruby file_processor_client.rb 1.jpg 2.png 3.tif
Share