Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Git LFS behind a proxy #5730

Open
thanksshu opened this issue May 3, 2024 · 3 comments
Open

Git LFS behind a proxy #5730

thanksshu opened this issue May 3, 2024 · 3 comments
Labels
bug

Comments

@thanksshu
Copy link

Describe the bug
I'm using git lfs behind an entreprise proxy that rejects requests to https://git.my-entreprise.com (I can access those domains directly).
I've set environment variables "HTTP_PROXY" and "HTTPS_PROXY" and updated gitconfig as below to ignore proxy for that domain.
When I clone a repo from the internal site, It fails to download and prints out:

Output

Cloning into 'project'...
remote: Enumerating objects: xxx, done.
remote: Counting objects: 100% (xxx), done.
remote: Compressing objects: 100% (xxx), done.
remote: Total xxx (delta xxx), reused xxx (delta xxx), pack-reused xxx
Receiving objects: 100%, xxx MiB | xxx MiB/s, done.
Resolving deltas: 100% (xxx), done.
Updating files: 100% (xxx), done.
Downloading project/file (xx KB)
Error downloading object: project/file (xx): Smudge error: Error downloading project/file (xxx): batch response: Post "https://git.my-entreprise.com/me/project.git/info/lfs/objects/batch": Bad Gateway

Errors logged to 'xxx.log'.
Use git lfs logs last to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: project/file: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

After unset "HTTP_PROXY" and "HTTPS_PROXY", I can clone the repo without any problem.
But I have to set those enviroment variables in order to make other softwares to work.

So in my case, did git lfs really respect proxy settings in .gitconfig?

To Reproduce
Steps to reproduce the behavior:

  1. Set HTTP_PROXY and HTTPS_PROXY to a proxy server that rejects requests of certain destination
  2. Set git config as below
  3. Clone the repo at a rejected destination
  4. See error downloding object

Expected behavior
Proxy settings in .gitconfig overrides "HTTP_PROXY" and "HTTPS_PROXY" environment variables

System environment

  • git-lfs 3.5.1
  • git 2.45.0
  • Windows 10 22H2
  • git config -l 
    filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
core.ignorecase=false
core.editor=code --wait
http.proxy=http://proxy.my-entreprise.com:80
http.https://git.my-entreprise.com.proxy=

Output of git lfs env

git lfs env 

LocalWorkingDir=
LocalGitDir=
LocalGitStorageDir=
LocalMediaDir=lfs\objects
LocalReferenceDirs=
TempDir=lfs\tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneVerifyUnreachableAlways=false
PruneRemoteName=origin
LfsStorageDir=lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context

GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git clone 

13:53:24.529379 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/bin
13:53:24.545072 git.c:465               trace: built-in: git clone https://git.my-entreprise.com/Me/project.git
Cloning into 'project'...
13:53:24.567374 run-command.c:657       trace: run_command: git remote-https origin https://git.my-entreprise.com/Me/project.git
13:53:24.614279 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:24.614279 git.c:750               trace: exec: git-remote-https origin https://git.my-entreprise.com/Me/project.git
13:53:24.614279 run-command.c:657       trace: run_command: git-remote-https origin https://git.my-entreprise.com/Me/project.git
13:53:24.667736 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:24.667736 http.c:868              == Info: Couldn't find host git.my-entreprise.com in the .netrc file; using defaults
13:53:24.683361 http.c:868              == Info: Host git.my-entreprise.com:443 was resolved.
13:53:24.683361 http.c:868              == Info: IPv6: (none)
13:53:24.683361 http.c:868              == Info: IPv4: my-ip
13:53:24.683361 http.c:868              == Info:   Trying my-ip:443...
13:53:24.698964 http.c:868              == Info: Connected to git.my-entreprise.com (my-ip) port 443
13:53:24.698964 http.c:868              == Info: ALPN: curl offers h2,http/1.1
13:53:24.698964 http.c:868              == Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
13:53:24.967251 http.c:868              == Info:  CAfile: C:/Users/Me/AppData/Local/Programs/Git/mingw64/etc/ssl/certs/ca-bundle.crt
13:53:24.967251 http.c:868              == Info:  CApath: none
13:53:24.967251 http.c:868              == Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
13:53:24.967251 http.c:868              == Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
13:53:24.967251 http.c:868              == Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
13:53:24.967251 http.c:868              == Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
13:53:24.967251 http.c:868              == Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
13:53:24.967251 http.c:868              == Info: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
13:53:24.967251 http.c:868              == Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
13:53:24.987800 http.c:868              == Info: TLSv1.2 (IN), TLS handshake, Finished (20):
13:53:24.987800 http.c:868              == Info: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / x25519 / RSASSA-PSS
13:53:24.987800 http.c:868              == Info: ALPN: server accepted h2
13:53:24.987800 http.c:868              == Info: Server certificate:
13:53:24.987800 http.c:868              == Info:  subject: C=XX; ST=XXXX; L=XXXX; O=My entreprise; CN=git.my-entreprise.com
13:53:24.987800 http.c:868              == Info:  start date: Dec  8 00:00:00 2023 GMT
13:53:24.987800 http.c:868              == Info:  expire date: Dec 11 23:59:59 2024 GMT
13:53:24.987800 http.c:868              == Info:  subjectAltNaMe: host "git.my-entreprise.com" matched cert's "git.my-entreprise.com"
13:53:24.987800 http.c:868              == Info:  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=GeoTrust TLS RSA CA G1
13:53:24.987800 http.c:868              == Info:  SSL certificate verify ok.
13:53:24.987800 http.c:868              == Info:   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
13:53:24.987800 http.c:868              == Info:   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
13:53:24.987800 http.c:868              == Info:   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
13:53:24.987800 http.c:868              == Info: using HTTP/2
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] OPENED stream for https://git.my-entreprise.com/Me/project.git/info/refs?service=git-upload-pack
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [:Method: GET]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [:scheMe: https]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [:authority: git.my-entreprise.com]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [:path: /Me/project.git/info/refs?service=git-upload-pack]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [user-agent: git/2.45.0.windows.1]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [accept: */*]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [accept-encoding: deflate, gzip, br, zstd]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [pragma: no-cache]
13:53:24.987800 http.c:868              == Info: [HTTP/2] [1] [git-protocol: version=2]
13:53:24.987800 http.c:815              => Send header, 0000000243 bytes (0x000000f3)
13:53:24.987800 http.c:827              => Send header: GET /Me/project.git/info/refs?service=git-upload-pack HTTP/2
13:53:24.987800 http.c:827              => Send header: Host: git.my-entreprise.com
13:53:24.987800 http.c:827              => Send header: User-Agent: git/2.45.0.windows.1
13:53:24.987800 http.c:827              => Send header: Accept: */*
13:53:24.987800 http.c:827              => Send header: Accept-Encoding: deflate, gzip, br, zstd
13:53:24.987800 http.c:827              => Send header: Pragma: no-cache
13:53:24.987800 http.c:827              => Send header: Git-Protocol: version=2
13:53:24.987800 http.c:827              => Send header:
13:53:24.987800 http.c:868              == Info: Request completely sent off
13:53:25.006006 http.c:815              <= Recv header, 0000000013 bytes (0x0000000d)
13:53:25.006006 http.c:827              <= Recv header: HTTP/2 401
13:53:25.006006 http.c:815              <= Recv header, 0000000026 bytes (0x0000001a)
13:53:25.006006 http.c:827              <= Recv header: server: GitHub-Babel/3.0
13:53:25.006006 http.c:815              <= Recv header, 0000000054 bytes (0x00000036)
13:53:25.006006 http.c:827              <= Recv header: content-security-policy: default-src 'none'; sandbox
13:53:25.006006 http.c:815              <= Recv header, 0000000041 bytes (0x00000029)
13:53:25.006006 http.c:827              <= Recv header: content-type: text/plain; charset=UTF-8
13:53:25.006006 http.c:815              <= Recv header, 0000000040 bytes (0x00000028)
13:53:25.006006 http.c:827              <= Recv header: www-authenticate: Basic realm="GitHub"
13:53:25.006006 http.c:815              <= Recv header, 0000000020 bytes (0x00000014)
13:53:25.006006 http.c:827              <= Recv header: content-length: 21
13:53:25.006006 http.c:815              <= Recv header, 0000000037 bytes (0x00000025)
13:53:25.006006 http.c:827              <= Recv header: date: Fri, 03 May 2024 11:53:25 GMT
13:53:25.006006 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.006006 http.c:827              <= Recv header: x-fraMe-options: DENY
13:53:25.006006 http.c:815              <= Recv header, 0000000002 bytes (0x00000002)
13:53:25.006006 http.c:827              <= Recv header:
13:53:25.006006 http.c:868              == Info: Connection #0 to host git.my-entreprise.com left intact
13:53:25.006006 run-command.c:657       trace: run_command: 'git credential-manager get'
13:53:25.122205 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:25.122205 git.c:750               trace: exec: git-credential-manager get
13:53:25.122205 run-command.c:657       trace: run_command: git-credential-manager get
13:53:25.238064 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:25.253762 git.c:465               trace: built-in: git config --null --list
13:53:25.353675 http.c:868              == Info: Found bundle for host: 0xxxxxxxxxxxx [can multiplex]
13:53:25.353675 http.c:868              == Info: Re-using existing connection with host git.my-entreprise.com
13:53:25.353675 http.c:868              == Info: Server auth using Basic with user 'Me'
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] OPENED stream for https://git.my-entreprise.com/Me/project.git/info/refs?service=git-upload-pack
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [:Method: GET]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [:scheMe: https]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [:authority: git.my-entreprise.com]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [:path: /Me/project.git/info/refs?service=git-upload-pack]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [authorization: Basic <redacted>]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [user-agent: git/2.45.0.windows.1]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [accept: */*]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [accept-encoding: deflate, gzip, br, zstd]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [pragma: no-cache]
13:53:25.353675 http.c:868              == Info: [HTTP/2] [3] [git-protocol: version=2]
13:53:25.353675 http.c:815              => Send header, 0000000334 bytes (0x0000014e)
13:53:25.353675 http.c:827              => Send header: GET /Me/project.git/info/refs?service=git-upload-pack HTTP/2
13:53:25.353675 http.c:827              => Send header: Host: git.my-entreprise.com
13:53:25.353675 http.c:827              => Send header: Authorization: Basic <redacted>
13:53:25.353675 http.c:827              => Send header: User-Agent: git/2.45.0.windows.1
13:53:25.353675 http.c:827              => Send header: Accept: */*
13:53:25.353675 http.c:827              => Send header: Accept-Encoding: deflate, gzip, br, zstd
13:53:25.353675 http.c:827              => Send header: Pragma: no-cache
13:53:25.353675 http.c:827              => Send header: Git-Protocol: version=2
13:53:25.353675 http.c:827              => Send header:
13:53:25.353675 http.c:868              == Info: Request completely sent off
13:53:25.400624 http.c:815              <= Recv header, 0000000013 bytes (0x0000000d)
13:53:25.400624 http.c:827              <= Recv header: HTTP/2 200
13:53:25.400624 http.c:815              <= Recv header, 0000000026 bytes (0x0000001a)
13:53:25.400624 http.c:827              <= Recv header: server: GitHub-Babel/3.0
13:53:25.406815 http.c:815              <= Recv header, 0000000059 bytes (0x0000003b)
13:53:25.406815 http.c:827              <= Recv header: content-type: application/x-git-upload-pack-advertiseMent
13:53:25.406815 http.c:815              <= Recv header, 0000000054 bytes (0x00000036)
13:53:25.406815 http.c:827              <= Recv header: content-security-policy: default-src 'none'; sandbox
13:53:25.406815 http.c:815              <= Recv header, 0000000040 bytes (0x00000028)
13:53:25.406815 http.c:827              <= Recv header: expires: Fri, 01 Jan 1980 00:00:00 GMT
13:53:25.406815 http.c:815              <= Recv header, 0000000018 bytes (0x00000012)
13:53:25.406815 http.c:827              <= Recv header: pragma: no-cache
13:53:25.406815 http.c:815              <= Recv header, 0000000053 bytes (0x00000035)
13:53:25.406815 http.c:827              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
13:53:25.406815 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.406815 http.c:827              <= Recv header: vary: Accept-Encoding
13:53:25.406815 http.c:815              <= Recv header, 0000000037 bytes (0x00000025)
13:53:25.406815 http.c:827              <= Recv header: date: Fri, 03 May 2024 11:53:25 GMT
13:53:25.406815 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.406815 http.c:827              <= Recv header: x-fraMe-options: DENY
13:53:25.406815 http.c:815              <= Recv header, 0000000002 bytes (0x00000002)
13:53:25.406815 http.c:827              <= Recv header:
13:53:25.406815 http.c:868              == Info: Connection #0 to host git.my-entreprise.com left intact
13:53:25.406815 run-command.c:657       trace: run_command: 'git credential-manager store'
13:53:25.508940 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:25.508940 git.c:750               trace: exec: git-credential-manager store
13:53:25.508940 run-command.c:657       trace: run_command: git-credential-manager store
13:53:25.607381 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:25.607381 git.c:465               trace: built-in: git config --null --list
13:53:25.707162 http.c:868              == Info: Found bundle for host: 0xxxxxxxxxxxx [can multiplex]
13:53:25.707162 http.c:868              == Info: Re-using existing connection with host git.my-entreprise.com
13:53:25.707162 http.c:868              == Info: Server auth using Basic with user 'Me'
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] OPENED stream for https://git.my-entreprise.com/Me/project.git/git-upload-pack
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [:Method: POST]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [:scheMe: https]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [:authority: git.my-entreprise.com]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [:path: /Me/project.git/git-upload-pack]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [authorization: Basic <redacted>]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [user-agent: git/2.45.0.windows.1]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [accept-encoding: deflate, gzip, br, zstd]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [content-type: application/x-git-upload-pack-request]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [accept: application/x-git-upload-pack-result]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [git-protocol: version=2]
13:53:25.707162 http.c:868              == Info: [HTTP/2] [5] [content-length: 185]
13:53:25.707162 http.c:815              => Send header, 0000000406 bytes (0x00000196)
13:53:25.707162 http.c:827              => Send header: POST /Me/project.git/git-upload-pack HTTP/2
13:53:25.707162 http.c:827              => Send header: Host: git.my-entreprise.com
13:53:25.707162 http.c:827              => Send header: Authorization: Basic <redacted>
13:53:25.707162 http.c:827              => Send header: User-Agent: git/2.45.0.windows.1
13:53:25.707162 http.c:827              => Send header: Accept-Encoding: deflate, gzip, br, zstd
13:53:25.707162 http.c:827              => Send header: Content-Type: application/x-git-upload-pack-request
13:53:25.707162 http.c:827              => Send header: Accept: application/x-git-upload-pack-result
13:53:25.707162 http.c:827              => Send header: Git-Protocol: version=2
13:53:25.707162 http.c:827              => Send header: Content-Length: 185
13:53:25.707162 http.c:827              => Send header:
13:53:25.707162 http.c:868              == Info: upload completely sent off: 185 bytes
13:53:25.760777 http.c:815              <= Recv header, 0000000013 bytes (0x0000000d)
13:53:25.760777 http.c:827              <= Recv header: HTTP/2 200
13:53:25.760777 http.c:815              <= Recv header, 0000000026 bytes (0x0000001a)
13:53:25.760777 http.c:827              <= Recv header: server: GitHub-Babel/3.0
13:53:25.760777 http.c:815              <= Recv header, 0000000052 bytes (0x00000034)
13:53:25.760777 http.c:827              <= Recv header: content-type: application/x-git-upload-pack-result
13:53:25.760777 http.c:815              <= Recv header, 0000000054 bytes (0x00000036)
13:53:25.760777 http.c:827              <= Recv header: content-security-policy: default-src 'none'; sandbox
13:53:25.760777 http.c:815              <= Recv header, 0000000040 bytes (0x00000028)
13:53:25.760777 http.c:827              <= Recv header: expires: Fri, 01 Jan 1980 00:00:00 GMT
13:53:25.760777 http.c:815              <= Recv header, 0000000018 bytes (0x00000012)
13:53:25.760777 http.c:827              <= Recv header: pragma: no-cache
13:53:25.760777 http.c:815              <= Recv header, 0000000053 bytes (0x00000035)
13:53:25.760777 http.c:827              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
13:53:25.760777 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.760777 http.c:827              <= Recv header: vary: Accept-Encoding
13:53:25.760777 http.c:815              <= Recv header, 0000000037 bytes (0x00000025)
13:53:25.760777 http.c:827              <= Recv header: date: Fri, 03 May 2024 11:53:25 GMT
13:53:25.760777 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.760777 http.c:827              <= Recv header: x-fraMe-options: DENY
13:53:25.760777 http.c:815              <= Recv header, 0000000002 bytes (0x00000002)
13:53:25.760777 http.c:827              <= Recv header:
13:53:25.760777 http.c:868              == Info: Connection #0 to host git.my-entreprise.com left intact
13:53:25.760777 http.c:868              == Info: Found bundle for host: 0xxxxxxxxxxxx [can multiplex]
13:53:25.760777 http.c:868              == Info: Re-using existing connection with host git.my-entreprise.com
13:53:25.760777 http.c:868              == Info: Server auth using Basic with user 'Me'
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] OPENED stream for https://git.my-entreprise.com/Me/project.git/git-upload-pack
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [:Method: POST]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [:scheMe: https]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [:authority: git.my-entreprise.com]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [:path: /Me/project.git/git-upload-pack]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [authorization: Basic <redacted>]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [user-agent: git/2.45.0.windows.1]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [accept-encoding: deflate, gzip, br, zstd]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [content-type: application/x-git-upload-pack-request]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [accept: application/x-git-upload-pack-result]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [git-protocol: version=2]
13:53:25.760777 http.c:868              == Info: [HTTP/2] [7] [content-length: 212]
13:53:25.760777 http.c:815              => Send header, 0000000406 bytes (0x00000xxx)
13:53:25.760777 http.c:827              => Send header: POST /Me/project.git/git-upload-pack HTTP/2
13:53:25.760777 http.c:827              => Send header: Host: git.my-entreprise.com
13:53:25.760777 http.c:827              => Send header: Authorization: Basic <redacted>
13:53:25.760777 http.c:827              => Send header: User-Agent: git/2.45.0.windows.1
13:53:25.760777 http.c:827              => Send header: Accept-Encoding: deflate, gzip, br, zstd
13:53:25.760777 http.c:827              => Send header: Content-Type: application/x-git-upload-pack-request
13:53:25.760777 http.c:827              => Send header: Accept: application/x-git-upload-pack-result
13:53:25.760777 http.c:827              => Send header: Git-Protocol: version=2
13:53:25.760777 http.c:827              => Send header: Content-Length: 212
13:53:25.760777 http.c:827              => Send header:
13:53:25.760777 http.c:868              == Info: upload completely sent off: 212 bytes
13:53:25.823451 http.c:815              <= Recv header, 0000000013 bytes (0x0000000d)
13:53:25.823451 http.c:827              <= Recv header: HTTP/2 200
13:53:25.823451 http.c:815              <= Recv header, 0000000026 bytes (0x0000001a)
13:53:25.823451 http.c:827              <= Recv header: server: GitHub-Babel/3.0
13:53:25.823451 http.c:815              <= Recv header, 0000000052 bytes (0x00000034)
13:53:25.823451 http.c:827              <= Recv header: content-type: application/x-git-upload-pack-result
13:53:25.823451 http.c:815              <= Recv header, 0000000054 bytes (0x00000036)
13:53:25.823451 http.c:827              <= Recv header: content-security-policy: default-src 'none'; sandbox
13:53:25.823451 http.c:815              <= Recv header, 0000000040 bytes (0x00000028)
13:53:25.823451 http.c:827              <= Recv header: expires: Fri, 01 Jan 1980 00:00:00 GMT
13:53:25.823451 http.c:815              <= Recv header, 0000000018 bytes (0x00000012)
13:53:25.823451 http.c:827              <= Recv header: pragma: no-cache
13:53:25.823451 http.c:815              <= Recv header, 0000000053 bytes (0x00000035)
13:53:25.823451 http.c:827              <= Recv header: cache-control: no-cache, max-age=0, must-revalidate
13:53:25.823451 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.823451 http.c:827              <= Recv header: vary: Accept-Encoding
13:53:25.823451 http.c:815              <= Recv header, 0000000037 bytes (0x00000025)
13:53:25.823451 http.c:827              <= Recv header: date: Fri, 03 May 2024 11:53:25 GMT
13:53:25.823451 http.c:815              <= Recv header, 0000000023 bytes (0x00000017)
13:53:25.823451 http.c:827              <= Recv header: x-fraMe-options: DENY
13:53:25.823451 http.c:815              <= Recv header, 0000000002 bytes (0x00000002)
13:53:25.823451 http.c:827              <= Recv header:
remote: EnuMerating objects: 2867, done.
remote: Counting objects: 100% (2421/2421), done.
13:53:25.838908 run-command.c:657       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 11020 on MyDevice' --check-self-contained-and-connected
13:53:25.892413 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:25.892413 git.c:465               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 11020 on MyDevice' --check-self-contained-and-connected
remote: Compressing objects: 100% (1193/1193), done.
13:53:26.077315 http.c:868              == Info: Connection #0 to host git.my-entreprise.com left intact
remote: Total 2867 (delta 1436), reused 1909 (delta 968), pack-reused 446
Receiving objects: 100% (2867/2867), 1.18 MiB | 8.59 MiB/s, done.
Resolving deltas: 100% (1537/1537), done.
13:53:26.139809 run-command.c:657       trace: run_command: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
13:53:26.177816 exec-cmd.c:243          trace: resolved executable dir: C:/Users/Me/AppData/Local/Programs/Git/mingw64/libexec/git-core
13:53:26.177816 git.c:465               trace: built-in: git rev-list --objects --stdin --not --all --quiet --alternate-refs '--progress=Checking connectivity'
13:53:26.209065 run-command.c:657       trace: run_command: 'git-lfs filter-process'
13:53:26.614958 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
13:53:26.679873 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
13:53:26.740552 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
13:53:26.803507 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.repositoryformatversion' '0'
13:53:26.880174 trace git-lfs: exec: git 'config' '--includes' '-l'
13:53:26.940483 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
13:53:27.000624 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
13:53:27.061138 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
13:53:27.123797 trace git-lfs: Install hook: pre-push, force=false, path=C:\Users\Me\project\.git\hooks\pre-push
13:53:27.124594 trace git-lfs: Install hook: post-checkout, force=false, path=C:\Users\Me\project\.git\hooks\post-checkout
13:53:27.125211 trace git-lfs: Install hook: post-commit, force=false, path=C:\Users\Me\project\.git\hooks\post-commit
13:53:27.126404 trace git-lfs: Install hook: post-Merge, force=false, path=C:\Users\Me\project\.git\hooks\post-Merge
13:53:27.132273 trace git-lfs: Initialize filter-process
13:53:27.133379 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-naMe' 'HEAD'
13:53:27.199025 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
13:53:27.265460 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
13:53:27.326414 trace git-lfs: tq: running as batched queue, batch size of 100
13:53:27.328428 trace git-lfs: filepathfilter: accepting "statics/large_file_1"
13:53:27.329491 trace git-lfs: filepathfilter: accepting "statics/large_file_2"
13:53:27.330111 trace git-lfs: filepathfilter: accepting "statics/large_file_3"
13:53:27.330625 trace git-lfs: filepathfilter: accepting "statics/large_file_4"
13:53:27.331169 trace git-lfs: filepathfilter: accepting "statics/large_file_5"
13:53:27.331708 trace git-lfs: filepathfilter: accepting "statics/large_file_6"
13:53:27.332777 trace git-lfs: filepathfilter: accepting "statics/large_file_7"
13:53:27.333324 trace git-lfs: filepathfilter: accepting "statics/large_file_8"
13:53:27.333868 trace git-lfs: filepathfilter: accepting "statics/large_file_9"
13:53:27.334384 trace git-lfs: filepathfilter: accepting "statics/large_file_10"
13:53:27.334929 trace git-lfs: filepathfilter: accepting "statics/large_file_11"
13:53:27.592152 trace git-lfs: filepathfilter: accepting "statics/large_file_12"
13:53:27.597781 trace git-lfs: filepathfilter: accepting "statics/large_file_13"
13:53:27.600513 trace git-lfs: filepathfilter: accepting "statics/large_file_14"
13:53:27.601029 trace git-lfs: filepathfilter: accepting "statics/large_file_15"
13:53:27.601062 trace git-lfs: filepathfilter: accepting "statics/large_file_16"
Updating files: 100% (404/404), done.
13:53:27.601575 trace git-lfs: tq: sending batch of size 16
13:53:27.602087 trace git-lfs: api: batch 16 files
13:53:27.604792 trace git-lfs: HTTP: POST https://git.my-entreprise.com/Me/project.git/info/lfs/objects/batch
> POST /Me/project.git/info/lfs/objects/batch HTTP/1.1
> Host: git.my-entreprise.com
> Accept: application/vnd.git-lfs+json
> Content-Length: 1562
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.5.1 (GitHub; windows amd64; go 1.21.7; git e237bb3a)
>
13:53:27.988516 trace git-lfs: api error: Post "https://git.my-entreprise.com/Me/project.git/info/lfs/objects/batch": Bad Gateway
13:53:27.991315 trace git-lfs: tq: running as batched queue, batch size of 100
13:53:27.991851 trace git-lfs: filepathfilter: accepting "statics/large_file_0"
Downloading statics/large_file (45 KB)
13:53:27.992401 trace git-lfs: tq: running as batched queue, batch size of 100
13:53:27.992401 trace git-lfs: tq: sending batch of size 1
13:53:27.992910 trace git-lfs: api: batch 1 files
13:53:27.992986 trace git-lfs: HTTP: POST https://git.my-entreprise.com/Me/project.git/info/lfs/objects/batch
> POST /Me/project.git/info/lfs/objects/batch HTTP/1.1
> Host: git.my-entreprise.com
> Accept: application/vnd.git-lfs+json
> Content-Length: 230
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.5.1 (GitHub; windows amd64; go 1.21.7; git e237bb3a)
>
13:53:28.366778 trace git-lfs: api error: Post "https://git.my-entreprise.com/Me/project.git/info/lfs/objects/batch": Bad Gateway
Error downloading object: statics/large_file (66fdef91e9739348df7a096aa384a5685f4e875584cce89386a7a47251c4d8e9): batch response: Post "https://git.my-entreprise.com/Me/project.git/info/lfs/objects/batch": Bad Gateway
13:53:28.380324 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir'
13:53:28.455990 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'

Errors logged to 'C:\Users\Me\project\.git\lfs\logs\20240503T135328.3674382.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: statics/large_file: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

@chrisd8088
Copy link
Member

Hey, I'm sorry you're having trouble, and thanks for the detailed report.

I think I've been able to reproduce this issue with a fairly simple case. Assuming there's a repository on GitHub like https://github.com/chrisd8088/test:

$ export HTTPS_PROXY=https://google.com
$ git config --global http.https://github.com.proxy ''
$ git config --global http.https://github.com.version 'HTTP/1.1'

$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git clone https://github.com/chrisd8088/test
...
> POST /chrisd8088/test.git/info/lfs/objects/batch HTTP/1.1
...
20:23:44.355924 trace git-lfs: api error: Post "https://github.com/chrisd8088/test.git/info/lfs/objects/batch": Not Found

The issue, I believe, stems from a differing interpretation of an empty http.proxy/http.<url>.proxy value. In Git, an empty value sets the CURLOPT_PROXY option to an empty string, and according to the cURL library, that explicitly disables proxying, so that any HTTP_PROXY or similar environment variables are ignored. In Git LFS, though, we only consider the http.proxy/http.<url>.proxy value found in the Git configuration if it is not empty, and otherwise we ignore it, so the environment variables take effect.

I'm therefore going to mark this as a bug, since we aim to follow Git's implementation in these sorts of matters.

@chrisd8088 chrisd8088 added the bug label May 6, 2024
@thanksshu
Copy link
Author

Thanks for the detailed investigation, and I think this is exactly the problem I encountered. Now by setting NO_PROXY I temporarily bypass this bug.

@chrisd8088
Copy link
Member

chrisd8088 commented May 6, 2024 •
edited
Loading

A few extra notes from some investigation into ways to address this issue:

There's a bit of a confusing history to the relevant code. The original implementation dates from PR #1358, where an empty Git configuration setting was ignored and treated as an unset value, so any environment variables would then take effect.

There was a change in PR #3062 to a later version of this code with the stated intention of fixing the bug reported in #3060, where environment variables supposedly took precedence over Git configuration settings. However, from what I can tell, that change (which persists in the current version of the code) had no real effect, which suggests that the bug report was either incorrect or reflected some particular condition which was not documented at the time.

Confusingly, in commit 5db9370 of PR #3062 the TestHttpsProxyFromGitConfig test was revised to have the opposite meaning from what is desired, and from what the stated intent of the PR was, making the test check that environment variables were preferred over Git configuration settings. This apparently then caused CI failures (since, as far as I can tell, the PR did not alter the behaviour of the code, which both before and after the PR would have preferred non-empty Git config proxy settings over the corresponding environment variables), and the test was returned to its original, correct expectations in PR #3138.

So far as I can tell, though, at no point were empty Git configuration settings (or empty but not unset environment variables) considered in any of these changes.

Meanwhile, ever since the first implementation in PR #1358 we've honoured the HTTP_PROXY environment variable (note the uppercase name), preferring its value to that of any http_proxy variable. However, neither Git nor cURL read the uppercase version of that variable. The curl(1) command is explicitly documented to only accept the lowercase variant:

The environment variables can be specified in lower case or upper case. The lower case version has precedence. http_proxy is an exception as it is only available in lower case.

On that note, we also incorrectly prefer the uppercase version of HTTPS_PROXY to the lowercase version. This should work, but doesn't, because while Git clones successfully, Git LFS tries to contact the wrong host:

$ export HTTPS_PROXY=https://google.com
$ export https_proxy=
$ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git clone https://github.com/chrisd8088/test
...
> POST /chrisd8088/test.git/info/lfs/objects/batch HTTP/1.1
...
00:06:13.128663 trace git-lfs: api error: Post "https://github.com/chrisd8088/test.git/info/lfs/objects/batch": malformed HTTP response

The behaviour above appears to stem from the Go language's support for these environment variables, which reads both HTTP_PROXY and http_proxy, unlike either the Git or cURL implementations.

However, the Go implementation in the x/net/http/httpproxy package has been changed from the version on which our code was based in one particular respect; namely, it now only applies the value of any HTTP_PROXY or http_proxy environment variable to HTTP connections, never HTTPS connections. Previously, it used those as a fallback for HTTPS connection proxies if one was not defined in HTTPS_PROXY or https_proxy, but this was altered in commit golang/net@7b1cca2 to address the issues raised in golang/go#40909.

But we still treat any proxy defined for HTTP as a fallback for HTTPS connections, reflecting how the Go language implementation used to behave.

As well, there are some oddities in our getProxyServers() function left over from the change in PR #3062. For instance, the function returns immediately if osEnv is nil. This used to make sense because the function first checked for any http.proxy/http.<url>.proxy settings, and then could return early if osEnv was not valid as there was no point in using it to look for environment variables. Now, though, we never reach the checks for Git configuration settings if osEnv is nil, although it would still be possible to do so if we skipped over the environment variable lookups.

We also perform some checks, like the check for an empty httpsProxy string, at points where it's not possible they could be anything other than an empty string because they are defined as return variables and haven't been set yet. Again, these checks used to make sense because the Git configuration values were read first and used to set the httpsProxy and httpProxy variables, but with the change in PR #3062, they are now redundant.

Whether we can fix all of these concerns in the v3.x release series I'm not immediately certain. We should be able to address most of the issues, but removing support for HTTP_PROXY in particular might be considered a change which could break some existing user configurations (even though Git ignores that environment variable) and have to wait for v4.0.0.

@github-project-automation github-project-automation bot moved this to Bugs in Backlog Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
Backlog
Status: Bugs
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chrisd8088 @thanksshu