Rails 4.2 Autoloading not thread-safe Rails 4.2 Autoloading not thread-safe multithreading multithreading

Rails 4.2 Autoloading not thread-safe


I recently ran into a very similar issue when trying to use an extra custom library placed in the [rails_root]/lib directory.

TL;DR:

You can use eager loading to around the issue, as that makes sure all constants/modules/classes are in memory before any actual code runs. However for this to work:

  1. You must have config.eager_load = true set in the Rails config (this is done by default in the Production environment)
  2. The file that your class-to-be-eager-loaded is in must be in the config.eager_load_paths, as opposed to config.autoload_paths.

OR

You can use require or require_dependency (another ActiveSupport feature) to make sure the code you need it explicitly loaded before it would otherwise get autoloaded by Rails.

More info

As digidigo mentioned in his reply, the circular dependency error comes from the ActiveSupport::Dependencies module, or the Rails autoloader in more general terms. This code is not threadsafe, as it uses that class/module variable to store files that it is loading. If two threads end up autoloading the same thing at the same time, one of them can get mislead by seeing the file to load already in that class variable and throwing a 'circular dependency' error.

I ran into this issue when running Rails in production mode with the (threaded) Puma webserver. We had added a small library to the lib directory in our Rails root, and initially added lib to config.autoload_once_paths. Everything was fine in Development, but in Production (with config.eager_load and config.cache_classes enabled), very occasionally we would get these same circular dependency issues with near-simultaneous requests. A few hours of debugging later, I ended up seeing the non-thread-safety happening in front of my eyes, when stepping through the ActiveSupport code around the circular dependency and seeing the different threads pick up at different points in the code. The first thread would add the file to load into the loading array, then the second thread would find it there and raise the circular dependency error.

It turns out adding something to autoload_paths or autoload_once_paths does NOT also mean that it will get picked up by eager loading. However the opposite is true - paths added to eager_load_paths will be considered for autoloading if eager_load is disabled (see this article for more info). We switched to eager_load_paths and have had no further issues so far.

Interestingly enough, just before the Rails 4 beta, autoloading was disabled in the Production environment by default, which meant that an issue like this would have caused a hard fail 100% of the time, rather than a quirky threading fail 5% of the time. However this was reverted in time for the 4.0 beta release - you can see some (passionate) discussion about it here (including the choice phrase 'honestly, you're telling me to go f*** myself?'). Since then though, that revert has been reverted ahead of the Rails 5.0.0beta1, so hopefully less people will have to deal with this headache of an issue again in the future.

Extra notes:

The Rails autoloader is totally separate from the Ruby autoloader - this seems to be because Rails does more inference on directory structure when trying to autoload constants.

Ruby's autoload appears to have been made threadsafe as of Ruby 2.0, however this has nothing to do with the Rails autoloading code. Rails's autoloader appears to be definitely not threadsafe, as previously mentioned.


This isn't really an answer but I do have more information. The error being thrown is from ActiveSupport

 if file_path    expanded = File.expand_path(file_path)    expanded.sub!(/\.rb\z/, '')    if loading.include?(expanded)      raise "Circular dependency detected while autoloading constant #{qualified_name}"    else      require_or_load(expanded, qualified_name)      raise LoadError, "Unable to autoload constant #{qualified_name}, expected #{file_path} to define it" unless from_mod.const_defined?(const_name, false)      return from_mod.const_get(const_name)    end  elsif mod = autoload_module!(from_mod, const_name, qualified_name, path_suffix)    return mod  elsif 

Upon further research we can see that loading is a class variable.

# Stack of files being loaded.mattr_accessor :loadingself.loading = []

Two threads checking the same file:

First thread hits this code and puts a path into loading

      loading << expanded

Then second thread goes to check the path represented by expanded and hits

 if loading.include?(expanded)      raise "Circular dependency detected while autoloading constant #{qualified_name}"

What am I missing? ActiveSupport::Dependencies is not threadsafe?


After some research it turns out, that autoload is thread-safe now. So it is propbably a regression. Checkout Threading with the AWS SDK for Ruby. The patch was introduced by Charles Nutter in ruby 2.0.0 autoload is not thread-safe

Anyway if it is only this class, you can avoid autoloading it by requiring it manually.Just require it manually.

require 'message_poro'class Userdef self.send_to_all(content)   ... end