Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git push in Cygwin gets stuck uploading LFS objects at 0% and then times out #5870

Open
hashimaziz1 opened this issue Sep 19, 2024 · 5 comments

Comments

@hashimaziz1
Copy link

hashimaziz1 commented Sep 19, 2024

For the past few hours I've been unable to push changes to my GitHub origin answer because it hangs indefinitely - the only output I get is:

Connection to github.com closed by remote host.
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
batch request: Received disconnect from 20.26.156.215 port 22:11: Bye Bye
Disconnected from 20.26.156.215 port 22: exit status 255

This is definitely not a network issue as similar issues have been closed for, because the network is working fine when LFS is not involved.

Running with GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push gives the following:

02:18:12.318186 git.c:465               trace: built-in: git push
02:18:12.331066 run-command.c:657       trace: run_command: unset GIT_PREFIX; ssh [email protected] 'git-receive-pack '\''hashimaziz1/mysite.git'\'''
02:18:13.605682 run-command.c:657       trace: run_command: .git/hooks/pre-push central [email protected]:hashimaziz1/mysite.git
02:18:13.716654 git.c:750               trace: exec: git-lfs pre-push central [email protected]:hashimaziz1/mysite.git
02:18:13.717291 run-command.c:657       trace: run_command: git-lfs pre-push central [email protected]:hashimaziz1/mysite.git
02:18:13.797585 trace git-lfs: exec: git 'version'
02:18:13.862025 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
02:18:13.937940 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
02:18:14.000599 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
02:18:14.079623 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
02:18:14.152077 trace git-lfs: exec: uname
02:18:14.244277 trace git-lfs: exec: cygpath '-w' '.git'
02:18:14.306389 trace git-lfs: exec: cygpath '-w' '/cygdrive/d/Users/Hashim/Projects/MySite/mysite'
02:18:14.357196 trace git-lfs: exec: git 'config' '--includes' '-l'
02:18:14.418597 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
02:18:14.481155 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
02:18:14.543444 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
02:18:14.605213 trace git-lfs: attempting pure SSH protocol connection
02:18:14.608736 trace git-lfs: run_command: ssh -oControlMaster=auto -oControlPath=C:\cygwin64\tmp\sock-3188800215\sock-%C [email protected] git-lfs-transfer hashimaziz1/mysite.git upload
02:18:14.623388 trace git-lfs: exec: ssh '-oControlMaster=auto' '-oControlPath=C:\cygwin64\tmp\sock-3188800215\sock-%C' '[email protected]' 'git-lfs-transfer hashimaziz1/mysite.git upload'
02:18:15.856192 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF
02:18:15.857629 trace git-lfs: pre-push: refs/heads/master 46d96490bb3932861ab252447325d5106b552c11 refs/heads/master a162c7a1bafd9f70a000ddf602333bfffa11c555
02:18:15.859898 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
02:18:15.951570 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'central'
02:18:17.550074 trace git-lfs: tq: running as batched queue, batch size of 100
02:18:17.552590 trace git-lfs: run_command: git rev-list --objects --ignore-missing --stdin --
02:18:17.553150 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-list' '--objects' '--ignore-missing' '--stdin' '--'
02:18:17.569575 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
02:18:17.596280 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
02:18:17.699104 trace git-lfs: exec: cygpath '-w' '.git'
02:18:17.763395 trace git-lfs: exec: cygpath '-w' '.git'
02:18:17.815095 trace git-lfs: tq: sending batch of size 1
02:18:17.815638 trace git-lfs: run_command: ssh [email protected] git-lfs-authenticate hashimaziz1/mysite.git upload
02:18:17.828492 trace git-lfs: exec: ssh '[email protected]' 'git-lfs-authenticate hashimaziz1/mysite.git upload'
Connection to github.com closed by remote host.
02:24:19.348742 trace git-lfs: ssh: [email protected] failed, error: exit status 255, message: Received disconnect from 20.26.156.215 port 22:11: Bye Bye
Disconnected from 20.26.156.215 port 22 (try: 0/6)
02:24:19.349339 trace git-lfs: run_command: ssh [email protected] git-lfs-authenticate hashimaziz1/mysite.git upload
02:24:19.376833 trace git-lfs: exec: ssh '[email protected]' 'git-lfs-authenticate hashimaziz1/mysite.git upload'
02:29:58.780014 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
02:29:58.782813 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
02:29:58.783793 trace git-lfs: filepathfilter: accepting "tmp"

02:29:58.784864 trace git-lfs: filepathfilter: accepting "2626421661"

2:24 is the point at which I hit Ctrl+C and the subsequent commands are always the same no matter when I do so, indicating no actual progress is being made. This seems to be an authentication issue, and I do use an SSH key file with a passphrase, but running with the wrapper script from #3779 (pointing to /usr/bin/ssh instead of Windows Git) makes no difference.

Letting the command run all the way until the end eventually terminates with:

...
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
batch request: Received disconnect from 20.26.156.215 port 22:11: Bye Bye
Disconnected from 20.26.156.215 port 22: exit status 255
error: failed to push some refs to 'github.com:hashimaziz1/mysite.git'

System environment

Cygwin running on Windows 10, which ssh shows usr/bin/ssh and ssh -V shows:

OpenSSH_9.8p1, OpenSSL 3.0.14 4 Jun 2024

Output of git config -l

user.tool
merge.tool=vscode
mergetool.vscode.cmd=code --wait $MERGED
diff.tool=vscode
difftool.vscode.cmd=code --wait --diff $LOCAL $REMOTE
alias.ac=!git add -A && git commit -m
init.defaultbranch=master
core.autocrlf=true
core.eol=crlf
core.protectntfs=false
core.longpaths=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.autocrlf=false
[email protected]:hashimaziz1/mysite.git
remote.central.fetch=+refs/heads/*:refs/remotes/central/*
branch.master.remote=central
branch.master.merge=refs/heads/master
pull.rebase=false
advice.addignoredfile=false
lfs.repositoryformatversion=0
lfs.activitytimeout=2000
lfs.tlstimeout=2000
lfs.dialtimeout=2000
lfs.keepalive=2000
lfs.https://github.com/hashimaziz1/mysite.git/info/lfs.locksverify=false

Output of git lfs env

git-lfs/3.3.0 (GitHub; windows amd64; go 1.19.3; git 77deabdf)
git version 2.45.1

Endpoint (central)=https://github.com/hashimaziz1/mysite.git/info/lfs (auth=none)
  [email protected]:hashimaziz1/mysite.git
LocalWorkingDir=D:\Users\Hashim\Projects\MySite\mysite
LocalGitDir=D:\Users\Hashim\Projects\MySite\mysite\.git
LocalGitStorageDir=D:\Users\Hashim\Projects\MySite\mysite\.git
LocalMediaDir=D:\Users\Hashim\Projects\MySite\mysite\.git\lfs\object
s
LocalReferenceDirs=
TempDir=D:\Users\Hashim\Projects\MySite\mysite\.git\lfs\tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=D:\Users\Hashim\Projects\MySite\mysite\.git\lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/libexec/git-core
GIT_LFS_PATH=C:\Program Files\Git LFS
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context

I had previously committed the 130MB file and forgotten to track it with git lfs track, then ran git lfs track and later git lfs migrate, which successfully rewrote the refs for those commits and added the file to .gitattributes. git lfs ls-files gives me the two files I expect to be tracked with LFS (the first one was pushed a few weeks ago without issues).

Would really appreciate some help here because I'm out of ideas and I'd really like to be able to push my changes sometime soon.

@hashimaziz1
Copy link
Author

hashimaziz1 commented Sep 19, 2024

Running ssh [email protected] git-lfs-authenticate hashimaziz1/mysite upload also gives me valid-looking JSON with the keys href. header, expires_at and expires_in, and the header contains a key Authorization. Anyone able to look into this issue?

@hashimaziz1
Copy link
Author

Likely related to #2791 since I disabled locks-verify to fix a previous issue preventing me from pushing my first LFS file a few weeks ago.

@hashimaziz1 hashimaziz1 changed the title Git push times gets stuck uploading LFS objects at 0% and then times out Git push in Cygwin gets stuck uploading LFS objects at 0% and then times out Sep 19, 2024
@chrisd8088
Copy link
Member

Hey, I'm sorry you're having trouble. I tried to replicate this but I only have success when pushing to GitHub right now, using an SSH endpoint, and testing with Git LFS objects in the pushes.

As this project is only for the Git LFS client, I would suggest reaching out to GitHub's Support team if you continue to experience problems.

Two things I happen to notice, in case they have any utility in diagnosing the problem with GitHub's people. One is that your version of Git LFS is 3.3.0 and mine is 3.5.1; there are a number of SSH-related changes between those versions, but, that said, I don't think any of them would be related to your disconnection problems.

The other detail is that from the IP address in your logs (20.26.156.215) I suspect you're connecting to one of GitHub's regional proxy sites; you might want to see what happens if you override your DNS and use an IP for github.com like 140.82.114.3 instead, which should hit one of their datacenters more directly.

And, of course, we always recommend removing any monitoring or firewall software you have installed locally, to make sure that's not interfering in some way.

Regardless, good luck and I hope you get the issue resolved! Let us know how it turns out.

@hashimaziz1
Copy link
Author

hashimaziz1 commented Sep 19, 2024

I was able to work around this for now by adding sshCommand = C:/Windows/System32/OpenSSH/ssh.exe under gitconfig.core and copying my private key and config in ~.ssh to the equivalent Windows folder D:\Users\Hashim\.ssh, so this is clearly a Git LFS bug related to its Cygwin/Windows implementation, just like several other bugs in the history of this repo that have been raised and fixed. If all those bugs were immediately written off as problems on GitHub's side as seems to be the strategy for new issues in this repo then none of them would have been fixed at all.

In case anyone does have an interest in finally getting this fixed, I have a feeling this is something to do with trying to use an SSH identity file when no ASKPASS binary is set (Cygwin doesn't come with one). Here are a few virtually identical issues that have also never been resolved: #2349, git-for-windows/git#1613, #1843, #3216.

@chrisd8088
Copy link
Member

I'm glad you found a working configuration which helps mitigate your problem!

We're always interested in trying to resolve issues. The key challenge for us is that we need a reproducible test case in order to be able to diagnose something; in this particular case, from your initial description, an SSH issue on GitHub's side seemed most likely to me, particularly as I wasn't able to reproduce the problem myself.

As you're obviously expert with Windows development, if you're able to pin down the exact cause of the issue, that would be most welcome! Patches and PRs are also always greatly appreciated.

In the meantime, I'll mark this as a bug and put it into the backlog, and if anyone is able to look at it before we're able, so much the better. Thanks again for your detailed report, and I'm glad you've got something working for you now, even if it's a workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Bugs
Development

No branches or pull requests

2 participants