Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable whether process should exit with non-zero exit code when tests failed? #1300

Closed
davidparsson opened this issue Feb 5, 2015 · 14 comments · Fixed by JetBrains/ring-ui#461, Narshe1412/nuarca-sql#25 or karronoli/redpen#10

Comments

@davidparsson
Copy link
Contributor

We run tests on Jenkins using Karma and Grunt. Karma produces JUnit reports using the karma-junit-reporter, and we've configured Jenkins to set the build as unstable (yellow) if test failures are found in these report files.

The problem is that whenever a test fails, Karma (and Grunt) exits with a non-zero exit code and this makes the build failed (red).

I think the process still should fail in all those cases where the tests haven't been completely executed, for example when a browser has been disconnected.

Is there any way to configure this today? If not, I'll gladly provide a pull request.

@mightyiam
Copy link

I'm quite sure you can do this if you run Karma through its API, instead of through karma-cli. See API.

@davidparsson
Copy link
Contributor Author

I don't think so. I found no way to distinguish if there were failed assertions in the tests or if the tests failed to execute, for example if a browser disconnected. Do you?

The API only callbacks with the exit code, and the exit code can be 1 both if one test fails and if a browser disconnects.

@mightyiam
Copy link

@davidparsson It seems that you're right. It may be that what you can use a reporter for this purpose, while also running through the API.

@dignifiedquire
Copy link
Member

I think the way to do this to expand the api support to give more insight to the exit reasons of Karma and then add an option for this to grunt-karma. Happy to accept a pr for this

@larryhe
Copy link

larryhe commented Aug 12, 2015

Hi @davidparsson, how did you fix your problem? I had the exactly same issue that running karma tests will have random DISCONNECTED issue (for example, If I run the same tests 10 times on my local, I probably will see that DISCONNECTED issue 2~3 times). Even worse, karma exit code is 1 on DISCONNECTED issue. In this case,I want the exit code 0 because technically there is no any unit tests failure. Or is there any way to distinguish unit test failure and DISCONNECTED? Any suggestion is highly appreciated. Thanks for your time.

@davidparsson
Copy link
Contributor Author

@larryhe, this is still an issue for me. For your disconnect issues, take a look at browserDisconnectTimeout, browserDisconnectTolerance and browserNoActivityTimeout in the docs.

@davidparsson
Copy link
Contributor Author

@dignifiedquire, I think the use-case described in the feature request could be desirable when running karma from the CLI as well. Would you consider allowing this to be a configured using a karma.conf.js/command-line option?

@davidparsson
Copy link
Contributor Author

Sadly my situation has changed, so I'm not likely going to submit a PR for this in the near future.

@ianwremmel
Copy link

I'm looking for something similar. I'd like to treat test failures as successes when generated structured output. Since karma seems to always exit with code 1, whether as a result of test failures or e.g. failure to get a browser, I can't use my CI system to determine the difference.

@costincaraivan
Copy link

I'm kind of surprised that this is the case. Is no one using Karma with tools like Jenkins? :(

@kintax
Copy link

kintax commented Apr 26, 2018

I do. We are using Karma w/ ChromeHeadless, and trying to figure out how to run tests in Jenkins and have failing tests NOT fail the build. Ultimately we want them to fail the build but we're doing heavy dev right now so we want the UT fails to be more like warnings.

@andreaspsson
Copy link
Contributor

andreaspsson commented Aug 21, 2018

I also run tests using Jenkins and have the same problem. Would be really neat with a proper solution for this problem

@dougludlow
Copy link

Just to help any other googlers, the failOnFailingTestSuite option was added. Looks like maybe --no-fail-failing-test-suite could work from the command line?

@jslegers
Copy link

jslegers commented Apr 2, 2019

I'm having sort-of the opposite problem here.

I'm starting my Karma like this (in Java) :

ArrayList<String> karmaTestCommand = new ArrayList<String>();
boolean isWindows = System.getProperty("os.name").startsWith("Windows");
if (isWindows) {
  karmaTestCommand.add("cmd"); //on Windows, NPM is not an executable, but a .cmd file
  karmaTestCommand.add("/c");
}
karmaTestCommand.add("npm");
karmaTestCommand.add("run");
karmaTestCommand.add("test");

ProcessBuilder karmaProcessBuilder = new ProcessBuilder(karmaTestCommand);
karmaProcessBuilder.inheritIO();
Map<String, String> karmaProcessEnv = karmaProcessBuilder.environment();
karmaProcessEnv.put("RIA_TESTSERVER_PORT", Integer.toString(testserverPort));
karmaProcessEnv.put("KARMA_SERVER_PORT", Integer.toString(karmaServerPort));
Process karmaProcess = karmaProcessBuilder.start();

boolean completed = new ProcessWatchDog(karmaProcess).waitFor(20, TimeUnit.MINUTES);
if (!completed) {
  System.err.println("Tests timed out after 20 minutes!");
  RIATestReport.writeTestTimeoutReport("target/junit_reports/test-results.xml", RunKarmaTests.class.getName());
}

This starts the following script :

"scripts": {
  "test": "karma start common/test/karma/karma.conf.js --singleRun=true || (exit 0)"
}

Due to some issue with Firefox on one of our Jenkins slaves, Karma behaves like this :

[java] 02 04 2019 14:06:25.834:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
[java] 02 04 2019 14:06:25.891:DEBUG [launcher]: Process Firefox exited with code 0
[java] 02 04 2019 14:06:25.891:DEBUG [temp-dir]: Cleaning temp dir C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:06:25.897:INFO [launcher]: Trying to start Firefox again (1/2).
[java] 02 04 2019 14:06:25.897:DEBUG [launcher]: Restarting Firefox
[java] 02 04 2019 14:06:25.897:DEBUG [temp-dir]: Creating temp dir at C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:06:25.898:DEBUG [launcher]: C:\Program Files\Mozilla Firefox\firefox.exe http://localhost:51005/?id=95416947 -profile C:\Users\release\AppData\Local\Temp\karma-95416947 -no-remote
[java] 02 04 2019 14:07:25.898:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
[java] 02 04 2019 14:07:25.961:DEBUG [launcher]: Process Firefox exited with code 0
[java] 02 04 2019 14:07:25.961:DEBUG [temp-dir]: Cleaning temp dir C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:07:25.967:INFO [launcher]: Trying to start Firefox again (2/2).
[java] 02 04 2019 14:07:25.967:DEBUG [launcher]: Restarting Firefox
[java] 02 04 2019 14:07:25.967:DEBUG [temp-dir]: Creating temp dir at C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:07:25.968:DEBUG [launcher]: C:\Program Files\Mozilla Firefox\firefox.exe http://localhost:51005/?id=95416947 -profile C:\Users\release\AppData\Local\Temp\karma-95416947 -no-remote
[java] 02 04 2019 14:08:25.968:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
[java] 02 04 2019 14:08:26.038:DEBUG [launcher]: Process Firefox exited with code 0
[java] 02 04 2019 14:08:26.038:DEBUG [temp-dir]: Cleaning temp dir C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:08:26.045:ERROR [launcher]: Firefox failed 2 times (timeout). Giving up.

This causes Karma to become unresponsive and will result in the process reaching a timeout after 20 minutes.

Is there any was I can get this process to end immediately, so I don't block my Jenkins slave for nothing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants