Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

big PR view details is extremely slow #31896

Closed
phoedos opened this issue Aug 21, 2024 · 11 comments
Closed

big PR view details is extremely slow #31896

phoedos opened this issue Aug 21, 2024 · 11 comments
Labels
performance/speed performance issues with slow downs

Comments

@phoedos
Copy link

phoedos commented Aug 21, 2024

Description

Faced with long-time gitea work during view details of big PR

http://giteaurl/MYORG/myrepo/pulls/68/files

dev #68
Merged
myuser merged 13 commits from dev  into dev_adaptive yesterday
Conversation
0
Commits
13
Files Changed
58
+1701 -284

58 changed files
with
1701 additions
and
284 deletions

Powered by Gitea
Version:
1.22.1
Page:
360658ms
Template:
559ms

this request took up to 360(!) seconds to display page

docker logs gitea:

2024/08/21 00:13:16 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /MYORG/myrepo/pulls/68/files for 10.10.2.100:0, elapsed 3209.6ms @ repo/pull.go:900(repo.ViewPullFilesForAllCommitsOfPr)
2024/08/21 00:19:15 ...eb/routing/logger.go:102:func1() [I] router: completed GET /MYORG/myrepo/pulls/68/files for 10.10.2.100:0, 200 OK in 360658.2ms @ repo/pull.go:900(repo.ViewPullFilesForAllCommitsOfPr)

tried made changes to indexer_type bleve->db->elasticsearch: no improvements
tried to enable Postgresql query timing: no queries with latency >100ms. 99% of gitea queries hit in <10ms responce.

Issue reproduce stable on server/local environment (SSD/nVME m2 drives), typical hdparm from drive where gitea folder is mounted:

hdparm -tT  /dev/sdb
/dev/sdb:
 Timing cached reads:   40268 MB in  2.00 seconds = 20169.03 MB/sec
 Timing buffered disk reads: 4640 MB in  3.00 seconds = 1546.34 MB/sec

docker-compose.yml example

services:
  gitea:
    image: gitea/gitea:1.22.1
    container_name: gitea
    restart: unless-stopped
    environment:
      USER_UID: 1000
      USER_GID: 1000
      #database connection
      GITEA__database__DB_TYPE: postgres
      GITEA__database__HOST: db:5432
      GITEA__database__NAME: gitea
      GITEA__database__USER: gitea
      GITEA__database__PASSWD: gitea
      #domain, rooturl, server settings
      GITEA__server__APP_DATA_PATH: /data/gitea
      GITEA__server__HTTP_PORT: 8080
      GITEA__server__DOMAIN: localhost
      GITEA__server__ROOT_URL: http://127.0.0.1:8080
      #indexer
      GITEA__indexer__ISSUE_INDEXER_TYPE: 'elasticsearch'
      GITEA__indexer__ISSUE_INDEXER_CONN_STR: 'http://elastic:password@es:9200'
      GITEA__indexer__ISSUE_INDEXER_NAME: 'gitea_issues'
      GITEA__indexer__REPO_INDEXER_ENABLED: true
      GITEA__indexer__REPO_INDEXER_REPO_TYPES: 'sources'
      GITEA__indexer__REPO_INDEXER_TYPE: 'elasticsearch'
      GITEA__indexer__REPO_INDEXER_CONN_STR: 'http://elastic:password@es:9200'
      GITEA__indexer__REPO_INDEXER_NAME: 'gitea_codes'
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 8080:8080
    depends_on:
       db:
         condition: service_healthy
       es:
         condition: service_healthy

  db:
    image: postgres:16.3
    restart: unless-stopped
    environment:
      POSTGRES_USER: gitea
      POSTGRES_PASSWORD: gitea
      POSTGRES_DB: gitea
    networks:
      - gitea
    volumes:
      - ./postgres16:/var/lib/postgresql/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    healthcheck :
      test: [ "CMD", "pg_isready", "-q", "-d", "gitea", "-U", "gitea" ]
      interval : 5s
      timeout : 5s
      retries : 5

  es:
    image: elasticsearch:8.14.2
    environment:
      - node.name=es
      - cluster.name=gitea
      - discovery.type=single-node
      - ELASTIC_PASSWORD=password
      - xpack.security.enabled=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - gitea
    volumes:
      - ./esdata:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD", "curl", "-s", "http://elastic:passwors@es:9200"]
      interval: 30s
      timeout: 10s
      retries: 5

networks:
 gitea:

Gitea Version

1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

docker-compose

Database

PostgreSQL

@yp05327 yp05327 added the performance/speed performance issues with slow downs label Aug 21, 2024
@jpraet
Copy link
Member

jpraet commented Oct 16, 2024

Same issue here after upgrading from Gitea 1.21.8 to 1.22.3.

2024/10/16 08:42:52 ...eb/routing/logger.go:68:func1() [W] router: slow GET /CBSS/ssh-communication-trace/pulls/52/files for 172.19.0.2:34192, elapsed 3014.0ms @ repo/pull.go:900(repo.ViewPullFilesForAllCommitsOfPr)
2024/10/16 08:44:19 ...ers/web/repo/pull.go:746:viewPullFiles() [E] SyncAndGetUserSpecificDiff: context canceled
2024/10/16 08:44:19 ...eb/routing/logger.go:102:func1() [I] router: completed GET /CBSS/ssh-communication-trace/pulls/52/files for 172.19.0.2:34192, 500 Internal Server Error in 90007.3ms @ repo/pull.go:900(repo.ViewPullFilesForAllCommitsOfPr)

(It stops after 90 seconds because HAProxy timeout)

The PR isn't even very big (7 changed files with 1005 additions and 219 deletions).

What's also strange is that on some occasions the same page does respond quickly (Page: 1504ms Template: 1103ms).

Database: PostgreSQL 14

@jpraet
Copy link
Member

jpraet commented Nov 14, 2024

Not quite sure if I'm experiencing the same issue as @phoedos, but in my case, when enabling debug logging, I see this:

2024/11/14 08:38:04 ...dules/git/command.go:345:Run() [D] slow git.Command.Run: /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= check-attr --stdin -z --cached linguist-vendored linguist-generated linguist-documentation linguist-detectable linguist-language gitlab-language (1m29.714398091s)
2024/11/14 08:38:04 ...dules/git/command.go:345:Run() [D] slow git.Command.Run: /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check (1m29.97483912s)
2024/11/14 08:38:04 ...dules/git/command.go:345:Run() [D] slow git.Command.Run: /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch (1m29.965427616s)
2024/11/14 08:38:04 ...dules/git/command.go:291:Run() [D] git.Command.RunDir(/data/git/repositories/cbss/dismissal-compensation-mutations-batch.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= diff --shortstat 6004812c49a57bba34a52915a6abc75870909c85...81ad60adf88c1889be3cae257425cd6d5f10a234
2024/11/14 08:38:04 ...ers/web/repo/pull.go:746:viewPullFiles() [E] SyncAndGetUserSpecificDiff: context canceled
2024/11/14 08:38:04 .../context_response.go:70:HTML() [D] Template: status/500
2024/11/14 08:38:04 ...eb/routing/logger.go:102:func1() [I] router: completed GET /CBSS/dismissal-compensation-mutations-batch/pulls/14/files for 172.19.0.2:43842, 500 Internal Server Error in 90012.2ms @ repo/pull.go:900(repo.ViewPullFilesForAllCommitsOfPr)

So it's some git commands that are slow.

@jpraet
Copy link
Member

jpraet commented Nov 15, 2024

Not quite sure if I'm experiencing the same issue as @phoedos, but in my case, when enabling debug logging, I see this:

2024/11/14 08:38:04 ...dules/git/command.go:345:Run() [D] slow git.Command.Run: /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= check-attr --stdin -z --cached linguist-vendored linguist-generated linguist-documentation linguist-detectable linguist-language gitlab-language (1m29.714398091s)
2024/11/14 08:38:04 ...dules/git/command.go:345:Run() [D] slow git.Command.Run: /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check (1m29.97483912s)
2024/11/14 08:38:04 ...dules/git/command.go:345:Run() [D] slow git.Command.Run: /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch (1m29.965427616s)
2024/11/14 08:38:04 ...dules/git/command.go:291:Run() [D] git.Command.RunDir(/data/git/repositories/cbss/dismissal-compensation-mutations-batch.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= diff --shortstat 6004812c49a57bba34a52915a6abc75870909c85...81ad60adf88c1889be3cae257425cd6d5f10a234
2024/11/14 08:38:04 ...ers/web/repo/pull.go:746:viewPullFiles() [E] SyncAndGetUserSpecificDiff: context canceled
2024/11/14 08:38:04 .../context_response.go:70:HTML() [D] Template: status/500
2024/11/14 08:38:04 ...eb/routing/logger.go:102:func1() [I] router: completed GET /CBSS/dismissal-compensation-mutations-batch/pulls/14/files for 172.19.0.2:43842, 500 Internal Server Error in 90012.2ms @ repo/pull.go:900(repo.ViewPullFilesForAllCommitsOfPr)

So it's some git commands that are slow.

This is similar to what's described in #31600.

@phoedos
Copy link
Author

phoedos commented Nov 22, 2024

dear @jpraet
Unfortunately the issue is still exist
Gitea version now 1.22.1 -> 1.22.3
seems the issue related to diff visualisation or diff building? updated gitea show messages related to repo.CompareDiff functions

2024/11/22 11:03:39 .../web/repo/compare.go:626:PrepareCompareDiff() [E] GetDiffRangeWithWhitespaceBehavior: context canceled
2024/11/22 11:03:39 ...eb/routing/logger.go:102:func1() [I] router: completed GET /MYORG/myrepo/compare/master...xCICD2 for 192.168.0.200:0, 500 Internal Server Error in 102409.0ms @ repo/compare.go:705(repo.CompareDiff)
2024/11/22 11:03:39 ...eb/routing/logger.go:102:func1() [I] router: completed GET / for 10.171.69.138:0, 200 OK in 56.4ms @ web/home.go:32(web.Home)

2024/11/22 11:03:42 ...eb/routing/logger.go:102:func1() [I] router: completed GET /MYORG/myrepo/src/branch/xCICD2 for 192.168.0.200:0, 200 OK in 104.3ms @ repo/view.go:774(repo.Home)
2024/11/22 11:03:45 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/events for 192.168.0.200:0, 200 OK in 2657.4ms @ events/events.go:18(events.Events)
2024/11/22 11:03:48 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /MYORG/myrepo/compare/master...xCICD2 for 192.168.0.200:0, elapsed 3079.6ms @ repo/compare.go:705(repo.CompareDiff)

@phoedos
Copy link
Author

phoedos commented Nov 22, 2024

In my example repository, the master branch is configured as the default branch.
I created an xCICD2 branch based on the dev branch.

Currently, when I click on the "PR" button, Gitea automatically computes the diff between the default branch (master) and the xCICD2 branch.

I believe a possible workaround would be to defer the diff calculation until the user selects both the source and destination branches for the pull request.

Anyway, performance of the diff computation should also be reviewed and optimized.

@jpraet
Copy link
Member

jpraet commented Nov 22, 2024

OK. The issue you are describing now is a bit different than the issue in your original post, as now you are on the compare page to create a new PR, and not on the diff page of an actual PR.

See also #13949 and #6900.

For your original issue with the PR diff page, it would be interesting to know if it's also caused by slow git.Command.Run.

@lunny
Copy link
Member

lunny commented Nov 22, 2024

Can somebody send a branch compare or pull request to demo.gitea.com?

@phoedos
Copy link
Author

phoedos commented Nov 22, 2024

Dear @jpraet yes this is a several different cases but think the root cause is the same related with diff
Dear @lunny I was able to simulate the issue on demo

  • made migration of big repo into demo
  • made with script with 1000 changes of random lines add, deletion and commited them to a separate big-commit-simulation branch
  • trying click with PR button and see a perfomance degradation on PR page when trying to select source and destination branches

Hope this example will help you with investigation
https://demo.gitea.com/phoedos/pmd

@lunny
Copy link
Member

lunny commented Nov 22, 2024

There are some configurations can be used to adjust the first time loading diff files and lines.

[git]
;MAX_GIT_DIFF_LINES = 1000
;;
;; Max number of allowed characters in a line in diff view
;MAX_GIT_DIFF_LINE_CHARACTERS = 5000
;;
;; Max number of files shown in diff view
;MAX_GIT_DIFF_FILES = 100

After adjusting like below.

[git]
MAX_GIT_DIFF_LINES = 300
MAX_GIT_DIFF_FILES = 30

The page will display total in 1 second on my macBook.

@lunny lunny removed the type/bug label Nov 22, 2024
@phoedos
Copy link
Author

phoedos commented Nov 25, 2024

made several experiments with documented options from configuration cheat sheet, reduced value

      GITEA__GIT__DISABLE_DIFF_HIGHLIGHT: 'true'
      GITEA__GIT__MAX_GIT_DIFF_LINES: 50
      GITEA__GIT__MAX_GIT_DIFF_LINE_CHARACTERS: 100
      GITEA__GIT__MAX_GIT_DIFF_FILES: 10

yes this options really helpful, during diff compare. If developer want to get more details in list of changed files / code lines, there is a 'Show More' button.

The issue is solved now, thanks!. Hope this expirience will be helpful for other users.

@lunny lunny closed this as completed Nov 25, 2024
@jpraet
Copy link
Member

jpraet commented Nov 25, 2024

There is definitely a performance regression causing slower diffs in Gitea 1.22 vs 1.21.

Closing this issue by suggesting to lower the diff size as a workaround feels to me like sweeping the problem under the rug.

But I guess further discussion and problem analysis can continue in #31600.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance/speed performance issues with slow downs
Projects
None yet
Development

No branches or pull requests

4 participants