Job profiling is an amazing superpower for those who have mastered its complexities. Sidekiq makes it as easy as possible to profile your jobs.

Profiling allows you to see the code executed by your job and how much time it took to execute. This makes it really easy to find slow spots in your job where you can optimize, e.g. you might find a slow query which is missing a database index.

Teaching you how to use a profiler is beyond the scope of this wiki page but I recommend watching this Youtube video by the author of Vernier:

Add Vernier to your Gemfile:

gem "vernier"

1. Start Sidekiq in production.
2. Open a console in production.
3. Pick a job type you wish to profile and kick off a job with the special profile token:

$ rails r 'SomeJob.set(profile: "token").perform_async(...)'

The purpose of token is to associate profile data to you, making it easier to find your profiles if other team members are also profiling their jobs. Typically I just use "mike" as my token.

• Vernier requires Ruby 3.2.1+.
• Profiling works fine with Active Jobs.
• Your profiling data is uploaded to profiler.firefox.com when you click the View button in the Web UI. Click "Invert call stack" to quickly find hotspots.
• Remember to profile in production mode. Development mode will add a lot of noise and won't reflect production exactly.
• Never add profile to a job's default sidekiq_options. Profiling slows down execution and adds a large chunk of data to Redis; you want to profile manually only.

The Web UI provides a new Profiles tab with a listing of all profile recordings in Redis. You can click the View button and see your profile in the Firefox Viewer. You can click the Data button to see or download the profile's raw JSON.

You can access profile data via Sidekiq::ProfileSet:

ps = Sidekiq::ProfileSet.new
ps.each do |profile|
  p profile # profile.started_at, profile.jid, profile.token, etc
end

The profiled data is stored in Redis as a Hash with key "token-jid", e.g. "mike-1234567890abcdef".

#{token}-#{jid}:
{
  started_at: <timestamp>,
  token: <token>,
  data: <gzip'd JSON>,
  jid: <jid>,
  type: <jobtype>,
  size: 50123, # data size in bytes
  elapsed: 24.345 # seconds
  sid: <storeid> # the profile's storage id at profiler.firefox.com
}

Vernier's profiler output can be large so Sidekiq uses gzip to compress it for storage. Be careful not to profile every job and quickly fill up your Redis! Profiling data expires after 24 hours.

Profiling only saves data if the job is successful. Failures will retry as normal.

Firefox Profiler is a complex JavaScript single page application which can't be maintained or distributed by the Sidekiq project. Instead Sidekiq uses the public instance at profiler.firefox.com which is run and maintained by Mozilla. When you click the View button in the Web UI, your profile data is uploaded to p.f.c for viewing. Profiling data generally does not contain anything sensitive but it does contain the names of your application classes and methods. It should never contain any application data.

I'm not aware of any official data retention policy from Mozilla for your uploaded profiles but your profiling data does not leave your system until you click View. You can use the Data button to access the raw JSON should you wish to run your own Firefox Profiler instance locally.

If you wish to disable remote profile uploads, adjust your config/routes.rb like this:

require "sidekiq/web"

Sidekiq::Web.configure do |config|
  config[:profile_store_url] = nil
end

The View button will disappear in the Web UI. Downloading the JSON data directly via the Data button will be your only option.