Skip to content

One consumer only for specific queue #436

Open
@TarasBardiuk

Description

Hi, guys.

I need to fetch and process messages from one specific queue in a strict stream, messages should be processed one by one, I mean NOT in parallel. We need this to avoid parallel creation of records, only one by one.

I was trying to achieve that by local worker config. As a result, thread: 1 doesn't help because I still have 4 consumers (workers: 4 specified in a config yml file). Then I was trying to use local workers option in a specific worker class, but it doesn't work, I guess it is not supported at all.

Then I found Handling different workloads Wiki-page, but it didn't help as well.

  1. I configured ./config/sneaker_worker_groups.yml file:
notparallelgroup:
  classes: MyWorkerClass
  workers: 1
  1. Then I executed bundle exec rake sneakers:run task
    After this step, I assumed that now all my queues should be handled by 4 consumers (as by default), but one specific queue (managed by MyWorkerClass) by a single consumer only. But no, all queues where listened by 4 consumers. Okay, maybe I need to run spawner manually?
  2. So I executed bundle exec ruby -e "require 'sneakers/spawner';Sneakers::Spawner.spawn" as mentioned in a Wiki page and... nothing changed. Every queue was still listened by 4 consumers.

Then I tried to create my own rake task to run it instead of default sneakers:run:

require 'sneakers'
require 'sneakers/runner'

task :environment

namespace :sneakers do
  desc 'Custom sneakers process start'
  task start: :environment do
    ::Sneakers.server = true
    ::Rake::Task['environment'].invoke
    ::Rails.application.eager_load!

    default, specific = ::Sneakers::Worker::Classes.partition do |classy|
      !classy.queue_opts.key?(:workers) || classy.queue_opts[:workers] ==
        ::Sneakers.const_get(:CONFIG)[:workers]
    end

    ::Sneakers::Runner.new(default, {}).run

    specific = specific.group_by { |classy| classy.queue_opts[:workers] }
    specific.each do |workers, classes|
      ::Sneakers::Runner.new(classes, workers: workers).run
    end
  end
end

But I found it impossible to be a single task only, because after the first #run method call, the rest of the rake task is not executed at all. So in such case I will have to define separate rake tasks for each local workers config option value.

It may be worth to say that on a server I need to start sneakers in a daemonize: false mode, but locally it is always daemonize: true.

Please help me get one specific queue to be processed by a single consumer only.

P.S. Why don't worker groups work? Or am I wrong and they are useless for my needs?

Thank you in advance.

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions