Add support for cross compiling x64-mingw32 fat binary gem and add support for native builds on Windows. #989

larskanis · 2013-10-27T20:22:34Z

This PR fixes #864 and obsoletes #976.

The PR moves all MiniPortile recipes into extconf.rb. That enables native builds on Windows and ensures that all depending libraries are build with the same compiler version as the C-ext when cross compiling.

Probably the most easy way to cross build the mingw gems is by using the rake-compiler-dev-box: https://github.com/tjschuck/rake-compiler-dev-box

There are currently some outstanding external issues:

libxslt needs to be patched for mingw-w64: https://bugzilla.gnome.org/show_bug.cgi?id=676511 This patch is already included in this PR.
There is an issue with the strip command in the dev-box, so you probably need this patch: Add wrappers for strip commands so that files are copied to the local drive before stripped. tjschuck/rake-compiler-dev-box#4
There is an issue with bundle exec in combination with rake-compiler's faking mechanism, that can be worked around by this patch: Pre-load resolver library before faking. rake-compiler/rake-compiler#83

I've successfully tested the extension by running all unit tests on Windows-7 64-Bit with Ruby-x64 2.0.0-p0 and Ruby-x86 1.9.3-p392. Both with cross compiled fat binary gem and with natively compiled source gem. They currently don't pass completely ( https://gist.github.com/larskanis/7186885 ), but they also don't pass with plain nokogiri-1.6.0 gem. I'll look at these failures later.

Update: Since #993 is merged to master, all tests pass on all above versions.

johnnyshields · 2013-10-31T05:12:08Z

👍 awesome work!!

giamma13 · 2013-11-04T20:41:43Z

Sorry I do I get the fix ?

knu · 2013-11-05T01:30:44Z

@luislavena Hope you could take the time for this... I would be able to test the build, but don't have a 64bit Windows environment at hand to test a resulted binary.

@larskanis Thanks for the work. In the meantime, could you rebase this on the current master again?

larskanis · 2013-11-06T20:55:02Z

@knu I merged with master instead of rebase to better show the changes. My patch for libxslt now integrates with the patching done on master and is version independent therefore, now.

johnnyshields · 2013-11-18T19:33:44Z

👍 please merge

larskanis · 2013-11-19T14:08:50Z

@johnnyshields did you test it? It would be great, if a windows user could give some feedback 😄

Testing the native build on Windows should be as simple as:

git clone https://github.com/larskanis/nokogiri.git
cd nokogiri
git config core.autocrlf false
git checkout mingw32-support -f
rake gem
gem inst pkg\nokogiri-1.6.1.beta.1.gem

This should work for all current RubyInstaller versions, when the corresponding DevKit is installed. It then downloads and builds zlib, libiconv, libxml and libxslt, so gem install takes some time.

You should also be able to do a full nokogiri test suite run like this:

rake -rdevkit test

Building a fat binary gem per cross build is somewhat more complicated, so I uploaded the binries here for testing.

@luislavena If you find some minutes, it would be great, if you could leave a comment to this PR. One thing is that I updated the cross ruby patch levels here, but I think this is obsolete now. In particular since I added a check of the binary dll files (about correct platform, dll entry point and imported system dlls).

@knu I did another rebase and removed some patches in b57294c. Do you agree with this?

johnnyshields · 2013-11-20T16:34:23Z

@larskanis will take a detailed look soon, am in the middle of a product launch this week. Very much looking forward to this, thanks!

arteezy · 2013-11-21T19:21:01Z

@larskanis I've tested these binaries that you've provided in my project and everything seems to be working. Thanks!

larskanis · 2013-11-22T21:48:42Z

I did another rebase targeting the master branch. Now the fat binary gems tell about the libraries they are build with, while gem install. The loud messages while source package install are not printed on Windows. but it's quite common to bundle all libraries within a product on Windows, anyway. Moreover the fat binary gem will certainly still keep the most used version.

Updated gems are here for download. Sill looking forward to receive any reports 😄

hk-dd · 2013-11-27T17:06:04Z

Thanks!
After download of nokogiri-1.6.1.beta.1.mingw.1-x64-mingw32.gem it works on Win7 with Ruby 2.0

maxfelsher · 2013-11-27T21:21:04Z

I was also able to download and install nokogiri-1.6.1.beta.1.mingw.1-x64-mingw32.gem on Windows 7, Ruby 2.0.

aprinslo1 · 2013-12-01T03:17:53Z

I was also able to download and install nokogiri-1.6.1.beta.1.mingw.1-x64-mingw32.gem on Windows 7 (64), Ruby 2.0. Thank you!

luislavena · 2013-12-09T00:52:02Z

@larskanis I need to look at the strip issue you're mentioning.

It didn't happen with other projects I tested, so I think this is limited to ffi.

Will have some time over the holiday/new year break to look at it.

Thank you.

larskanis · 2013-12-09T07:33:44Z

@luislavena nokogiri suffers from the same strip-issue because zlib makes
use of strip in it's build process. Furthermore I added striping of the
final so files, to shrink the gem size similar to what is done when
building the ffi gem.

The issue only happens in the VirtualBox and only when started on it's
shared folders.

lypanov · 2014-01-14T07:43:13Z

I was able to install your replacement nokogiri git src (v1.6.1.beta.1) in Windows 8.1 on top of ruby 2.0.0-p353 x64.

Thank you!

utensil · 2014-01-14T09:08:01Z

I'm also able to install nokogiri-1.6.1.beta.1-x64-mingw32.gem in Windows 7 (64) on top of ruby 2.0.0p353 (2013-11-22) [x64-mingw32].

Thank you!

RyoYamamotoJP · 2014-01-30T04:36:54Z

I was also able to install nokogiri-1.6.1.beta.1-x64-mingw32.gem in Windows 7 Professional 64bit on top of ruby 2.0.0p353 (2013-11-22) [x64-mingw32]. Thank you!

maxfelsher · 2014-02-03T16:42:17Z

@larskanis @luislavena Is there any way to help with this pull request other than reporting that it seems to work, or is it at the point where the only ones who can help are the repository owners?

I'd love to be able to use something that's been merged into the main repository, if only to have access to other improvements made to Nokogiri since @larskanis forked. But if that's not feasible right now, that's understandable!

knu · 2014-02-04T05:58:42Z

Apart from the third-party issues mentioned, I think we should merge this to move forward.
@larskanis Could you resolve conflicts?

I also want to know what is left to be done after merging this, namely the third-party issues.

…-mingw32) fat binary gem and support for native builds on Windows. This moves all MiniPortile recipes into extconf.rb. This enables native build on Windows and ensures that all depending libraries are build with the same compiler version as the C-ext when cross compiling.

…ller-1.9.3

Modifications to configure.in trigger a re-run of autotools, which adds additional dependencies to nokogiri. This breaks the build of libxslt when there is a version mismatch between the one used to release the libxslt tar file and the installed one. The removed patches itself are IMHO not necessary for nokogiri.

…' error.

is really used for the packaged fat binary gems. Avoid adding dependency.yml twice - once per Manifest.txt and once per add_file_to_gem.

It was called as prerequisites and is called as part of 'rake cross native gem'

larskanis · 2014-02-04T20:24:36Z

@knu: You should be able to do a clean merge now.

The still open third party issues I encountered are:

libxslt needs to be patched for mingw-w64: https://bugzilla.gnome.org/show_bug.cgi?id=676511 This patch is already included as file in this PR.
There is an issue with the strip command in the dev-box, so you probably need this patch: Add wrappers for strip commands so that files are copied to the local drive before stripped. tjschuck/rake-compiler-dev-box#4
There is a character escaping bug in mingw, that affects native windows builds: http://sourceforge.net/p/mingw/bugs/2142 . It is worked around by larskanis@d0ea7c2

Apart from these, all other third party issues were resolved in the meanwhile.

Some more internal things should be mentioned:

There is a LF vs. CRLF issue for the native windows build from git. Line endings of text files in a git working directory are typically converted to CRLF. This applies also for the patch files. But the tar files that are downloaded are LF encoded, so the patching fails. This can be avoided by setting "git config core.autocrlf false"
Issue mini_portile should not be a dependency for java and windows builds #1019 is not yet addressed for the windows binary gems.
There could be some documentation for the cross build.
Several gcc warnings are printed in the windows build.

larskanis · 2014-02-24T08:46:06Z

Issue 2 in the list above is now finally solved. So the rake-compiler-dev-box is now suitable to cross build nokogiri.
Any chance to merge this pull request soon?

knu · 2014-02-24T09:55:54Z

Done. Thanks for all the work @larskanis, and the testers!

larskanis · 2014-02-24T15:02:26Z

Don't hesitate to contact me in case of questions/issues regarding the merged code.
Is there a documentation to the cross build process, that needs to be updated? Also the native build from a git repository on Windows could be documented.

ccoenen · 2014-03-15T14:33:08Z

The gem provided 4 months ago installs perfectly!
Any chance of getting a release candidate containing this on rubygems? (I'm just asking because of the Y U NO GEMSPEC ;-) )

knu · 2014-03-26T09:38:03Z

The developers of Nokogiri are starting to work as a team again, so please be patient just a bit more for us to be able to release an official gem.

flavorjones · 2014-04-03T15:01:49Z

We're getting close to a release. Thanks to everyone for their patience,
and thanks to everyone who contributed.

On Wed, Mar 26, 2014 at 5:38 AM, Akinori MUSHA [email protected]:

The developers of Nokogiri are starting to work as a team again, so please
be patient just a bit more for us to be able to release an official gem.

Reply to this email directly or view it on GitHubhttps://github.com//pull/989#issuecomment-38664345
.

This was referenced Oct 27, 2013

Add support for cross compiling for the x64-mingw32 platform. #976

Closed

Work around broken vsnprintf implementation in mingw. #993

Merged

This was referenced Nov 20, 2013

Fail install Nokogiri on Windows7 x86 with Ruby2.0 #951

Closed

Support Ruby x64 on Windows #864

Closed

larskanis and others added 7 commits February 4, 2014 20:32

Use 'host' as alternative, because host_alias is not set on RubyInsta…

2a602c8

…ller-1.9.3

Add missing files to Manifest.txt

284eac2

Update rake-compiler version to 0.9.2 in order to avoid 'win32/resolv…

0d83a37

…' error.

Move gem post install message into cross compiling callback, so that it

24383cf

is really used for the packaged fat binary gems. Avoid adding dependency.yml twice - once per Manifest.txt and once per add_file_to_gem.

Don't trigger gem task twice in 'rake gem:windows'.

64931bd

It was called as prerequisites and is called as part of 'rake cross native gem'

Use a better fix for the character escaping bug in MSYS

d0ea7c2

knu merged commit d0ea7c2 into sparklemotion:master Feb 24, 2014

larskanis mentioned this pull request Mar 5, 2014

Workaround the MSYS character escaping bug on native mingw builds only. #1061

Merged

larskanis mentioned this pull request Jun 25, 2015

Make use of rake-compiler-dock for building windows binary gems #1302

Closed

Uh oh!

Add support for cross compiling x64-mingw32 fat binary gem and add support for native builds on Windows. #989

Add support for cross compiling x64-mingw32 fat binary gem and add support for native builds on Windows. #989

Uh oh!

Conversation

larskanis commented Oct 27, 2013

Uh oh!

johnnyshields commented Oct 31, 2013

Uh oh!

giamma13 commented Nov 4, 2013

Uh oh!

knu commented Nov 5, 2013

Uh oh!

larskanis commented Nov 6, 2013

Uh oh!

johnnyshields commented Nov 18, 2013

Uh oh!

larskanis commented Nov 19, 2013

Uh oh!

johnnyshields commented Nov 20, 2013

Uh oh!

arteezy commented Nov 21, 2013

Uh oh!

larskanis commented Nov 22, 2013

Uh oh!

hk-dd commented Nov 27, 2013

Uh oh!

maxfelsher commented Nov 27, 2013

Uh oh!

aprinslo1 commented Dec 1, 2013

Uh oh!

luislavena commented Dec 9, 2013

Uh oh!

larskanis commented Dec 9, 2013

Uh oh!

lypanov commented Jan 14, 2014

Uh oh!

utensil commented Jan 14, 2014

Uh oh!

RyoYamamotoJP commented Jan 30, 2014

Uh oh!

maxfelsher commented Feb 3, 2014

Uh oh!

knu commented Feb 4, 2014

Uh oh!

larskanis commented Feb 4, 2014

Uh oh!

larskanis commented Feb 24, 2014

Uh oh!

knu commented Feb 24, 2014

Uh oh!

larskanis commented Feb 24, 2014

Uh oh!

ccoenen commented Mar 15, 2014

Uh oh!

knu commented Mar 26, 2014

Uh oh!

flavorjones commented Apr 3, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants