Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

guix: Linux and macOS builds are not cross-arch reproducible with powerpc64le build arch #31207

Open
1 task done
JeremyRand opened this issue Nov 3, 2024 · 6 comments
Open
1 task done
Assignees

Comments

@JeremyRand
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Current behaviour

The Guix output hashes for Windows and the source tarball match between my powerpc64le build machine and the official binaries, but the hashes for macOS and Linux do not match.

This seems to be a partial regression from 27.1, where Windows, Linux, and the source tarball all matched for me. macOS did not even build without errors for 27.1 on powerpc64le (that is at least fixed in 28.0), so I have nothing to compare there.

It is not clear to me whether the root cause might be the same as the converse problem referenced in #27897 (powerpc64le hashes didn't match between x86_64 and aarch64 build machines).

(Maybe I've done something stupid here, but I definitely can't tell what I would have done wrong, and the different behavior from 27.1 certainly seems like I've hit a real bug.)

Expected behaviour

All Guix output hashes should match between my powerpc64le build machine and the official binaries.

Steps to reproduce

  1. Build Bitcoin Core via ./contrib/guix/guix-build
  2. Compare results against the guix-sigs repo (I used pinheadmz's Guix sigs as a reference).

Relevant log output

Hashes from my powerpc64le build machine:

3908cbe1339fc10a220e9634629b6937dba77de926bf8dd618283fd090c61f5e  bitcoin-28.0-aarch64-linux-gnu-debug.tar.gz
1e58898a1b48296fd7e31ddf96c248a0a45c07709d980e3dadfd66ab9e5d2845  bitcoin-28.0-aarch64-linux-gnu.tar.gz
dbbb90b05c2e390ba6130cac074135a52c83dcc762a28257de1abfe89d69675f  bitcoin-28.0-arm-linux-gnueabihf-debug.tar.gz
54a30faa5a61f628d890f464c48def63858eca9b655873f1e60c4e220af41c2e  bitcoin-28.0-arm-linux-gnueabihf.tar.gz
da85a7758d272ca5ea345a2b549147acfbbdf61c556c98dc4f49e8f7a27af30d  bitcoin-28.0-arm64-apple-darwin-unsigned.tar.gz
c24ff655b0cbbdc116b19474fa621b1c803e3af25f3e540ea10d53a9ba9c4a2d  bitcoin-28.0-arm64-apple-darwin-unsigned.zip
17033ff17cb91c09cf27fefb42801a584844385b2c6c50902ae63cebb7fc632e  bitcoin-28.0-arm64-apple-darwin.tar.gz
700ae2d1e204602eb07f2779a6e6669893bc96c0dca290593f80ff8e102ff37f  bitcoin-28.0.tar.gz
a182b39beb4603c3cad55f9f45eaae730eaa1a8c5ee26718707a35554d28654a  bitcoin-28.0-powerpc64-linux-gnu-debug.tar.gz
252a1e89b369ce1895a58ae4425c765e6c654214550e14245f6bdb84ec161c2b  bitcoin-28.0-powerpc64-linux-gnu.tar.gz
cf4cbfa876142bfdb4c781271cd9a1a5410e6cb6e7f0e3996eb67bf08909d4ef  bitcoin-28.0-riscv64-linux-gnu-debug.tar.gz
fd91dee4518444891549d41f995ef007ae81a426cc5914a4767f1d4410eb1103  bitcoin-28.0-riscv64-linux-gnu.tar.gz
a98b0fdddcd71fdc16d29d3c4cc95baf408cc2210e23aa96fa1b808267a25387  bitcoin-28.0-x86_64-apple-darwin-unsigned.tar.gz
898f790d4941e106158ba2c38f8cb439a6e1e4f58cad8c9cd86dd0ddff08d117  bitcoin-28.0-x86_64-apple-darwin-unsigned.zip
d2c74eebfa6ac1e7ce64b986cbbd28e4838bd8abf2de70493d971b373d559044  bitcoin-28.0-x86_64-apple-darwin.tar.gz
3b5808f3070e88b5f6eee25678d80f4159827ea391abb0ee25ba96af2a598f75  bitcoin-28.0-x86_64-linux-gnu-debug.tar.gz
ffa78c8e87ea20f13780e780589351484fd22defdabb6aa3234d48f9518104a1  bitcoin-28.0-x86_64-linux-gnu.tar.gz
8990def2e611323d4c7a8cf17187a138dca64f98fc0ecebda0a3e999dbdd083d  bitcoin-28.0-win64-debug.zip
d8170c342ac049fab953f87841cbbba6c0e3f277703ddc29c678b6ab93dae966  bitcoin-28.0-win64-setup-unsigned.exe
8ec39e7bf66ea419ea79e5f1b7bee1b03a28b51ddd1daa6e167bff6abac0a5d2  bitcoin-28.0-win64-unsigned.tar.gz
85282f4ec1bcb0cfe8db0f195e8e0f6fb77cfbe89242a81fff2bc2e9292f7acf  bitcoin-28.0-win64.zip

For comparison, see https://github.com/bitcoin-core/guix.sigs/blob/main/28.0/pinheadmz/noncodesigned.SHA256SUMS

I would be happy to upload my binaries and/or run diffoscope on request. I would also be happy to run Guix builds of additional commits on my powerpc64le machine if there are commits that seem likely to be the culprit.

How did you obtain Bitcoin Core

Compiled from source

What version of Bitcoin Core are you using?

v28.0

Operating system and version

Debian 12.7 running inside KVM; host is also Debian 12.7

Machine specifications

powerpc64le; HDD; wired network on residential Internet connection.

@maflcko
Copy link
Member

maflcko commented Nov 4, 2024

I would be happy to upload my binaries and/or run diffoscope on request. I would also be happy to run Guix builds of additional commits on my powerpc64le machine if there are commits that seem likely to be the culprit.

I'd say all of that would be useful. You link to the GCC bump to 12, which seems one of the more significant changes to the Linux build, so testing commit e1ce5b8 (and e1ce5b8ae9124717c00dca71a5c5b43a7f5ad177~1), or otherwise bisecting could be useful as well.

@laanwj
Copy link
Member

laanwj commented Nov 14, 2024

I would be happy to upload my binaries

Yes, please do. i'm happy to try to help find the difference but i don't have access to a ppc64 system.

@fanquake
Copy link
Member

Last I was looking at this, if you guix built (master) for powerpc64le-linux-gnu, once on x86_64 and once on aarch64, and compared the resulting binaries, you'd end up with (a small number) of differences where two instructions are swapped. For example:

--- a/x86_64.txt
+++ b/aarch64.txt
@@ -176,8 +176,8 @@
    24ea8:      78 1b 77 7c     mr      r23,r3
    24eac:      66 01 17 7c     mtfprd  f0,r23
    24eb0:      14 f2 37 7d     add     r9,r23,r30
-   24eb4:      d0 00 b8 3a     addi    r21,r24,208
-   24eb8:      10 00 40 39     li      r10,16
+   24eb4:      10 00 40 39     li      r10,16
+   24eb8:      d0 00 b8 3a     addi    r21,r24,208
    24ebc:      00 00 00 60     nop
    24ec0:      10 8e 22 f9     std     r9,-29168(r2)
    24ec4:      50 00 00 f0     xxspltd vs0,vs0,0

In this case, in:

0000000000024bf0 <_GLOBAL__sub_I_descriptor.cpp>:
   24bf0:	4e 00 4c 3c 	addis   r2,r12,78
   24bf4:	10 32 42 38 	addi    r2,r2,12816
   24bf8:	a6 02 08 7c 	mflr    r0
<snip>
   24eb0:	14 f2 37 7d 	add     r9,r23,r30
   24eb4:	d0 00 b8 3a 	addi    r21,r24,208
   24eb8:	10 00 40 39 	li      r10,16
   24ebc:	00 00 00 60 	nop

@laanwj
Copy link
Member

laanwj commented Nov 14, 2024

Last I was looking at this, if you guix built (master) for powerpc64le-linux-gnu, once on x86_64 and once on aarch64

It might be, but it's unclear to me if that is the same issue, this OP talks about building on PPC64.

(that's a pretty awful issue btw! apparently the optimizer is non-deterministic?)

@JeremyRand
Copy link
Contributor Author

I would be happy to upload my binaries

Yes, please do. i'm happy to try to help find the difference but i don't have access to a ppc64 system.

I've uploaded my non-debug binaries here:

https://ynqrnzvuotdgvqj2g73f5tyj4rcvwutdemu23jgadawihbkc3lxkbwid.onion/bitcoin-core/28.0-guix-binaries-built-on-powerpc64le/

If the debug variants would be helpful, I don't mind uploading them too. I don't have any plans to delete these files or decommission this server, but it's personal hobbyist infrastructure, so I cannot make any guarantees about uptime or permanence.

@laanwj laanwj self-assigned this Nov 25, 2024
@laanwj
Copy link
Member

laanwj commented Nov 25, 2024

i've compared against the binaries from bitcoincore.org and looked at the ELF metadata and disassembly and it's a significant difference, not just a few different instructions. The entire .text section ends up with a different size, which shifts the offsets, making it very hard to determine what the source of the discrepancy is.

Debug symbols could help but ideally i'd need a linker map.

Can you please build the following branch: https://github.com/laanwj/bitcoin/tree/2024-11-linker-map-28.0 — it is v28.0 with a patch to generate .map files.
And then upload the .maps for x86_64-linux-gnu (i think this is enough).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants