Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

4.0.8 sdist tarball signature on pypi is bad #77

Closed
dvzrv opened this issue Oct 23, 2021 · 17 comments
Closed

4.0.8 sdist tarball signature on pypi is bad #77

dvzrv opened this issue Oct 23, 2021 · 17 comments

Comments

@dvzrv
Copy link

dvzrv commented Oct 23, 2021

Hi! When trying to package 4.0.8 for Arch Linux I ran into a bad signature for the sdist tarball on pypi.org:

gpg --verify gitdb-4.0.8.tar.gz.asc
gpg: assuming signed data in 'gitdb-4.0.8.tar.gz'
gpg: Signature made 2021-10-23T02:39:33 CEST
gpg:                using RSA key EAF9CCFCD0876408F297C60A9CB5EE7895E8268B
gpg: BAD signature from "Sebastian Thiel (YubiKey USB-C) <[email protected]>" [unknown]

To reproduce:

curl -L -O https://files.pythonhosted.org/packages/source/g/gitdb/gitdb-4.0.8.tar.gz -O https://files.pythonhosted.org/packages/source/g/gitdb/gitdb-4.0.8.tar.gz.asc && gpg --verify gitdb-4.0.8.tar.gz.asc
@Byron
Copy link
Member

Byron commented Oct 24, 2021

Thanks for reporting, I am able to reproduce the issue as well.

Now I wonder what could have happened there, since it looks 'like always' on my end. My gpg is able to properly sign commits, --show-signature considers these valid, so I hope this was just a cosmic ray.

@Byron
Copy link
Member

Byron commented Oct 24, 2021

curl -L -O https://files.pyAthonhosted.org/packages/source/g/gitdb/gitdb-4.0.9.tar.gz -O https://files.pythonhosted.org/packages/source/g/gitdb/gitdb-4.0.9.tar.gz.asc && gpg --verify gitdb-4.0.9.tar.gz.asc still says it's a bad signature.

I have no idea how to fix this or what the problem even is :/.

@dvzrv
Copy link
Author

dvzrv commented Oct 26, 2021

Can you create a signature locally to test whether it is also bad?

e.g. gpg --detach-sign gitdb-4.0.9.tar.gz and then verify it via gpg --verify gitdb-4.0.9.tar.gz.asc

How are you creating the signature and how is it pushed to pypi.org?

@Byron
Copy link
Member

Byron commented Oct 26, 2021

Great idea, as this easily reproduces the issue while showing what's going wrong:

gpg -u 27C50E7F590947D7273A741E85194C08421980C9 --detach-sign gitdb-4.0.9.tar.gz
gpg: error reading symlink '/proc/curproc/file': No such file or directory

It still creates a signature though, which might be a signature over no bytes at all. Initially I thought it would be related to me using gpg from nix-pkg, but now I am using the homebrew one with the same outcome.

Funnily enough, I am not the only one: https://stackoverflow.com/questions/69699986/gpg-error-reading-symlink-proc-curproc-file-no-such-file-or-directory . The issue seems to be all but harmless, and is probably related to breakage in MacOS 12/12.1. The latter one I am using.

By the looks of it there won't be valid signatures from me for a while until this is fixed. Now that I can reproduce it, I can keep trying as updates come in.

How are you creating the signature and how is it pushed to pypi.org?

	twine upload -s -i 27C50E7F590947D7273A741E85194C08421980C9 dist/*

Twine does it, and I wouldn't be surprised if it just calls gpg under the hood.

@dvzrv
Copy link
Author

dvzrv commented Nov 3, 2021

@Byron can you use a different machine/ a VM for an intermediate release?

@Byron
Copy link
Member

Byron commented Nov 4, 2021

In the linked stackoverflow question one of the answers says it's fixed in 2.3.3.1, so I upgraded to this one. Unfortunately, despite not showing the error message anymore, the signature is still bad right after creation. Maybe they also considered the issue harmless and squelched it. Maybe it's something very else.

Setting up a VM to sign should be possible and I will see when I get to it. Alternatively I think I could try MacGPG as I'd expect them to maintain a working version.

@Byron
Copy link
Member

Byron commented Nov 4, 2021

No, MacGPG and its binary have the same issue despite claiming support for MacOS 12. I'd think my key got corrupted, but that's something it would probably detect, besides, signing commits works and yields valid signatures.
So maybe gpg for detached signatures is thoroughly broken on MacOS right now (as are a few other things, like the linker 😅)

@dvzrv
Copy link
Author

dvzrv commented Dec 10, 2021

You could also have a look at sq.

@Byron
Copy link
Member

Byron commented Dec 12, 2021

I took a look and alongside the --sign-with=<path-to-gpg-wrapper> one should be able to make twine use sq for signing.

Now one would just have to bring in the GPG keys to sq. Probably all doable and desirable to open a new chapter on the GPG story which definitely left a bitter taste thus far.

@dvzrv
Copy link
Author

dvzrv commented Dec 12, 2021

Probably all doable and desirable to open a new chapter on the GPG story which definitely left a bitter taste thus far.

I agree. We also just recently switched the Arch Linux distribution keyring handling over to sq, as its tooling and CLI is less overengineered and nicer to use.

@dvzrv
Copy link
Author

dvzrv commented Jan 17, 2022

Any update on this? I unfortunately can not upgrade the package at the moment.

@Byron
Copy link
Member

Byron commented Jan 18, 2022

Sorry for the wait and the hassle it must be.

For now I don't know when I will spend the time to deal with this. It's certainly not helped by my expectation that it will be painful to touch a combination of python tooling and handle loose GPG keys.

At the latest it will happen once GitPython NG lands, the one that uses Gitoxide under the hood.

In the mean time, after each update of MacOS or GPG I try again to see if something changes, but have no hopes.

@dvzrv
Copy link
Author

dvzrv commented Feb 20, 2022

I am sorry, but I will now drop all PGP signature validation from all gitpython related packages (this ticket has been open since October 2021).

Having PGP signature validation does not justify the breakage that is caused by trying to make this work (see e.g. https://bugs.archlinux.org/task/73193) and I have spent considerable amounts of time trying to make this work by providing numerous tickets (and I do not see myself doing this anymore).
This is not feasible and as this has been broken for months already I would suggest you consider either properly fixing this or just dropping it. As it stands it is just not possible to rely on the PGP signatures for the gitpython projects :-/

@Byron
Copy link
Member

Byron commented Feb 21, 2022

Thanks for speaking up and sharing your experience, and I am sorry to be the cause of this hassle and hope that dropping PGP signatures will fix the issue.

Now that I know it's an option, I will do that right away which hopefully allows GitPython packages to become available again on arch linux. PGP was always a hassle for me as well and even though sequoia could be the solution, I am just unable to invest the time needed to migrate to it. It's sad that suddenly perfectly valid keys just fail to produce valid signatures in this particular setting, without error message or anything like that.

Byron added a commit to gitpython-developers/GitPython that referenced this issue Feb 21, 2022
Byron added a commit that referenced this issue Feb 21, 2022
The key stopped producing correct signatures when upgrading MacOS,
at least when used by `twine`.
Byron added a commit to gitpython-developers/smmap that referenced this issue Feb 21, 2022
@Byron
Copy link
Member

Byron commented Feb 21, 2022

Would re-releasing smmap, gitdb and GitPython help to get affairs back on track? If so, please let me know and it will be done. Thank you

@Byron
Copy link
Member

Byron commented Feb 15, 2024

Closing this issue as signed releases, deprecated for a while, have now officially been removed from the last location that mentioned them, the main readme file.

Please see this comment for an anchor to the conversation that led to this.

It may well be that I am missing something which makes closing this issue the incorrect choice, and if so please let me know and we can reopen.

@EliahKagan
Copy link
Contributor

EliahKagan commented Feb 26, 2024

I've noticed this is still open. Some time has passed since #77 (comment); should this be closed?

@Byron Byron closed this as not planned Won't fix, can't repro, duplicate, stale Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

3 participants