Skip to content

Conversation

@Pixel998
Copy link
Contributor

Prerequisites checklist

What is the purpose of this pull request? (put an "X" next to an item)

[ ] Documentation update
[x] Bug fix (template)
[ ] New rule (template)
[ ] Changes an existing rule (template)
[ ] Add autofix to a rule
[ ] Add a CLI option
[ ] Add something to the core
[ ] Other, please explain:

Fix #19234

What changes did you make? (Give an overview)

Replace WebdriverIO with Cypress

Is there anything you'd like reviewers to focus on?

@Pixel998 Pixel998 requested a review from a team as a code owner February 26, 2025 20:57
@github-project-automation github-project-automation bot moved this to Needs Triage in Triage Feb 26, 2025
@eslint-github-bot eslint-github-bot bot added the chore This change is not user-facing label Feb 26, 2025
@netlify
Copy link

netlify bot commented Feb 26, 2025

Deploy Preview for docs-eslint canceled.

Name Link
🔨 Latest commit bb8bbca
🔍 Latest deploy log https://app.netlify.com/sites/docs-eslint/deploys/67c60330234ba90008599d70

@fasttime
Copy link
Member

Thanks for the new pull request @Pixel998. I can see that the browser test is still running quite slowly in CI. The current run took over 8 minutes. I was trying to use cypress-io/github-action to speed it up, but it didn't help. So I'm not sure. Either Cypress is just this lame in GitHub or we don't know how to set it up properly.

@nzakas
Copy link
Member

nzakas commented Feb 27, 2025

Maybe we could try https://modern-web.dev/docs/test-runner/overview/ instead.

@nzakas nzakas moved this from Needs Triage to Implementing in Triage Feb 27, 2025
@Pixel998
Copy link
Contributor Author

Pixel998 commented Mar 1, 2025

@fasttime So it appears that making Cypress not render the runner ui fixes the performance issue.

@fasttime
Copy link
Member

fasttime commented Mar 2, 2025

I think this is quite acceptable now. It's not as fast as WebdriverIO but still faster then Mocha unit tests and it never timed out in my test runs. Probably it isn't even necessary to use Firefox as a browser.

@fasttime fasttime added the accepted There is consensus among the team that this change meets the criteria for inclusion label Mar 2, 2025
@fasttime
Copy link
Member

fasttime commented Mar 2, 2025

Note that the output of the Test step is showing two strange warnings:

Couldn't find tsconfig.json. tsconfig-paths will be skipped

tput: No value for $TERM and no -T specified

The first warning is filed as a bug in Cypress (cypress-io/cypress#21805 and cypress-io/cypress#22273), so maybe we could leave it as it is until they fix it. If we want to get rid of the warning, we could create a directory cypress with a TS config file cypress/tsconfig.json in it, move cypress.config.js to cypress/config.js and launch Cypress with the CLI option --config-file=cypress/config.js. That would remove the warning without adding an unnecessary tsconfig.json file at the repo root.

The second warning seems to only occur when --no-runner-ui is specified. It can be fixed by defining a TERM environment variable in .github/workflows/ci.yml:

  test_on_browser:
    name: Browser Test
    runs-on: ubuntu-latest
+   env:
+     TERM: xterm-256color

Clearly there are also other options.

@Pixel998
Copy link
Contributor Author

Pixel998 commented Mar 3, 2025

Probably it isn't even necessary to use Firefox as a browser.

You're right. Chrome is now as fast locally, so I've switched back to it.

The first warning is filed as a bug in Cypress ...

I think we can avoid the workaround for now. Given that the warning isn't causing any major issues, waiting for the Cypress fix seems like the best approach. Unless you have a strong preference for implementing the workaround, I'd suggest we leave it as is.

The second warning seems to only occur when --no-runner-ui is specified. It can be fixed by defining a TERM environment variable in .github/workflows/ci.yml:

Done.

@fasttime
Copy link
Member

fasttime commented Mar 4, 2025

I think the output of Cypress is also easier to read. For comparison, this is what a test failure in GitHub Actions currently looks like with WebdriverIO — notice that there's no stack trace:

image

And this is the same failure with Cypress and the progress reporter, the same one we use for Mocha unit tests:

image

Copy link
Member

@fasttime fasttime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! I would like another review from @nzakas.

@fasttime fasttime moved this from Implementing to Second Review Needed in Triage Mar 4, 2025
@fasttime fasttime requested a review from nzakas March 4, 2025 08:17
Copy link
Member

@mdjermanovic mdjermanovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Copy link
Member

@nzakas nzakas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I tested this locally and it has roughly the same speed as the WebDriver approach. Well done!

Just need to wait for the patch release window to close.

@nzakas nzakas moved this from Second Review Needed to Merge Candidates in Triage Mar 10, 2025
@mdjermanovic mdjermanovic merged commit c99db89 into eslint:main Mar 10, 2025
30 checks passed
@github-project-automation github-project-automation bot moved this from Merge Candidates to Complete in Triage Mar 10, 2025
@Pixel998 Pixel998 deleted the test/cypress branch March 23, 2025 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accepted There is consensus among the team that this change meets the criteria for inclusion chore This change is not user-facing contributor pool github actions

Projects

Status: Complete

Development

Successfully merging this pull request may close these issues.

Change Request: Switch browser test runner

4 participants