A RubyGems plugin that generates binary (pre-compiled) gems.

• home
• bugs

gem-compiler is a RubyGems plugin that helps generates binary gems from already existing ones without altering the original source code. It compiles Ruby C extensions and bundles the result into a new gem.

It uses an outside-in approach and leverages on existing RubyGems code to do it.

Using gem-compiler removes the need to install a compiler toolchain on the platform used to run the extension. This means less dependencies are required in those systems and can reduce associated update/maintenance cycles.

Additionally, by having only binaries, it reduces the time it takes to install several gems that normally take minutes to compile themselves and the needed dependencies.

Without gem-compiler, takes more than a minute to install Nokogiri on Ubuntu 18.04:

$ time gem install --local nokogiri-1.10.7.gem
Building native extensions. This could take a while...
Successfully installed nokogiri-1.10.7
1 gem installed

real    1m22.670s
user    1m5.856s
sys     0m18.637s

Compared to the installation of the pre-compiled version:

$ gem compile nokogiri-1.10.7.gem --prune
Unpacking gem: 'nokogiri-1.10.7' in temporary directory...
Building native extensions. This could take a while...
  Successfully built RubyGem
  Name: nokogiri
  Version: 1.10.7
  File: nokogiri-1.10.7-x86_64-linux.gem

$ time gem install --local nokogiri-1.10.7-x86_64-linux.gem
Successfully installed nokogiri-1.10.7-x86_64-linux
1 gem installed

real    0m1.697s
user    0m1.281s
sys     0m0.509s

To install gem-compiler you need to use RubyGems:

$ gem install gem-compiler

Which will fetch and install the plugin. After that the compile command will be available through gem.

As requirement, gem-compiler can only compile local gems, either one you have generated from your projects or previously downloaded.

If you don't have the gem locally, you can use fetch to retrieve it first:

$ gem fetch yajl-ruby --platform=ruby
Fetching: yajl-ruby-1.1.0.gem (100%)
Downloaded yajl-ruby-1.1.0

Please note that I was explicit about which platform to fetch. This will avoid RubyGems attempt to download any existing binary gem for my current platform.

You need to tell RubyGems the filename of the gem you want to compile:

$ gem compile yajl-ruby-1.1.0.gem

The above command will unpack, compile any existing extensions found and repackage everything as a binary gem:

Unpacking gem: 'yajl-ruby-1.1.0' in temporary directory...
Building native extensions.  This could take a while...
  Successfully built RubyGem
  Name: yajl-ruby
  Version: 1.1.0
  File: yajl-ruby-1.1.0-x86-mingw32.gem

This new gem do not require a compiler, as shown when locally installed:

C:\> gem install --local yajl-ruby-1.1.0-x86-mingw32.gem
Successfully installed yajl-ruby-1.1.0-x86-mingw32
1 gem installed

There are native gems that will invalidate their own specification after compile process completes. This will not permit them be repackaged as binary gems. To workaround this problem you have the option to prune the package process:

$ gem fetch nokogiri --platform=ruby
Fetching: nokogiri-1.6.6.2.gem (100%)
Downloaded nokogiri-1.6.6.2

$ gem compile nokogiri-1.6.6.2.gem --prune
Unpacking gem: 'nokogiri-1.6.6.2' in temporary directory...
Building native extensions.  This could take a while...
  Successfully built RubyGem
  Name: nokogiri
  Version: 1.6.6.2
  File: nokogiri-1.6.6.2-x86_64-darwin-12.gem

$ gem install --local nokogiri-1.6.6.2-x86_64-darwin-12.gem
Successfully installed nokogiri-1.6.6.2-x86_64-darwin-12
1 gem installed

Gems compiled with gem-compiler be lock to the version of Ruby used to compile them, following Ruby's ABI compatibility (MAJOR.MINOR)

This means that a gem compiled with Ruby 2.6.1 could be installed in any version of Ruby 2.6.x (Eg. 2.6.4).

You can tweak this behavior by using --abi-lock option during compilation. There are 3 available modes:

• ruby: Follows Ruby's ABI. Gems compiled with Ruby 2.6.1 can be installed in any Ruby 2.6.x (default behavior).
• strict: Uses Ruby's full version. Gems compiled with Ruby 2.6.1 can only be installed in Ruby 2.6.1.
• none: Disables Ruby compatibility. Gems compiled with this option can be installed on any version of Ruby (alias for --no-abi-lock).

Warning: usage of none is not recommended since different versions of Ruby might expose different APIs. The binary might be expecting specific features not present in the version of Ruby you're installing the gem into.

By default, RubyGems do not strip symbols from compiled extensions, including debugging information and can result in increased size of final package.

With --strip, you can reduce extensions by using same stripping options used by Ruby itself (see RbConfig::CONFIG["STRIP"]):

$ gem compile oj-3.10.0.gem --strip
Unpacking gem: 'oj-3.10.0' in temporary directory...
Building native extensions. This could take a while...
Stripping symbols from extensions (using 'strip -S -x')...
  Successfully built RubyGem
  Name: oj
  Version: 3.10.0
  File: oj-3.10.0-x86_64-linux.gem

Or you can provide your own stripping command instead:

$ gem compile oj-3.10.0.gem --strip "strip --strip-unneeded"
Unpacking gem: 'oj-3.10.0' in temporary directory...
Building native extensions. This could take a while...
Stripping symbols from extensions (using 'strip --strip-unneeded')...
  Successfully built RubyGem
  Name: oj
  Version: 3.10.0
  File: oj-3.10.0-x86_64-linux.gem

Gem servers like RubyGems or Gemstash treat gems as immutable, so once a gem has been pushed, you cannot replace it.

When playing with compilation options or library dependencies, you might require to build and push an updated version of the same version.

You can use --build-number to add the build number to the compiled version and push an updated build, maintaining gem dependency compatibility:

$ gem compile oj-3.11.3.gem --build-number 10
Unpacking gem: 'oj-3.11.3' in temporary directory...
Building native extensions. This could take a while...
  Successfully built RubyGem
  Name: oj
  Version: 3.11.3.10
  File: oj-3.11.3.10-x86_64-linux.gem

This new version remains compatible with RubyGems' dependency requirements like ~> 3.11 or ~> 3.11.3.

Most of the times, as gem developer, you would like to generate both kind of gems at once. For that purpose, you can add a task for Rake similar to the one below:

desc "Generate a pre-compiled native gem"
task "gem:native" => ["gem"] do
  sh "gem compile #{gem_file}"
end

Of course, that assumes you have a task gem that generates the base gem required.

It's assumed you have Ruby and RubyGems installed. gem-compiler requires RubyGems 2.6.x to work.

If you don't have RubyGems 2.6.x, you can upgrade by running:

$ gem update --system

In order to compile a gem, you need a compiler toolchain installed. Depending on your Operating System you will have one already installed or will require additional steps to do it. Check your OS documentation about getting the right one.

For those using RubyInstaller-based builds, you will need to download the DevKit from their downloads page and follow the installation instructions.

To be sure your installation of Ruby is based on RubyInstaller, execute at the command prompt:

C:\> ruby --version

And from the output:

ruby 2.4.9p362 (2019-10-02 revision 67824) [x64-mingw32]

If you see mingw32, that means you're using a RubyInstaller build (MinGW based).

rake-compiler has provided to Ruby library authors a tool for compiling extensions and generating binary gems of their libraries.

You can consider rake-compiler's approach be an inside-out process. To do its magic, it requires library authors to modify their source code, adjust some structure and learn a series of commands.

While the ideal scenario is using a tool like rake-compiler that endorses convention over configuration, is not humanly possible change all the projects by snapping your fingers 😉

The MIT License