Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coverage summary always 100% #2210

Closed
vvenv opened this issue Jun 27, 2016 · 7 comments · May be fixed by Omrisnyk/npm-lockfiles#122 or Omrisnyk/npm-lockfiles#132
Closed

coverage summary always 100% #2210

vvenv opened this issue Jun 27, 2016 · 7 comments · May be fixed by Omrisnyk/npm-lockfiles#122 or Omrisnyk/npm-lockfiles#132

Comments

@vvenv
Copy link

vvenv commented Jun 27, 2016

Expected behavior

should report coverage summary correctly

Actual behavior

always output

Statements   : 100% ( 0/0 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 100% ( 0/0 )

Enviroment Details

  • Karma version (output of karma --version): 1.1.0
  • Relevant part of your karma.config.js file
coverageReporter: {
    reporters: [
        { type: 'text-summary' }, // always 100%
        { type: 'text-summary', file: 'lcov.txt' } // reported file is correct
    ]
  }

Steps to reproduce the behaviour

1.just run
2.
3.

@dignifiedquire
Copy link
Member

this is probably an issue with karma-coverage. please file an issue there

@nicojs
Copy link
Contributor

nicojs commented Jun 28, 2016

I think i have the same issue. It only occurs with karma 1.1.0, not 1.0.0 or 0.13.x. As the problem seems to be introduced in karma 1.1, i think the issue should be in the karma issue tracker

There seems to be a raise condition. I think karma is collecting the reports too soon. Usually this doesn't result in an error because the html reporter first needs to verify that the directory structure is created (with fs.stats, see reporter.js). By then, the collector is filled. However for Stryker we're using the in-memory reporter. That one tries to collect the report immediately which usually fails with karma 1.1.0.

If i surround reporter.js L290 with a setTimeout the collector is always filled.

I have been debugging this issue for hours, but my inside karma knowledge is just too small. Can someone point me in the right direction? Namely: did something in the life cycle of reporting change what could explain this raise condition?

See this build output (with karma 1.1.0): https://travis-ci.org/stryker-mutator/stryker-karma-runner/jobs/140456228

Note: I see in the bug request that the text-summary has the same issue, which makes sense as it is also reporting immediately, without the creation of the file system.

@nicojs
Copy link
Contributor

nicojs commented Jun 29, 2016

I did some more research. Apparently the OnBrowserComplete (reporter.js line 228) is sometimes called after onRunComplete (reporter.js line 237). The report is only created in the OnBrowserComplete so it is important that that one is called first.

Can anyone speculate why the ordering of the events might have changed in this new release?

Note: This means that the fact that other reporters are still working is just luck. If they didn't do the call to mkdirIfNotExists and create the report in the async callback, it would break them as well.

@RIAstar
Copy link

RIAstar commented Jun 29, 2016

@nicojs' analysis is correct. I had the same issue and tracked it down to the same reason.
( too bad I found this issue after the fact :( )

Perhaps I can offer some more information: I have one case where it happens systematically and another where it systematically doesn't.

case 1: order always correct:
Just launch Karma server with singlerun, no autowatch.

const config = {
    singleRun: true,
    autoWatch: false,
    ...
};
const server = new Server(cfg, onFinish(done));
server.start();

case 2: order always inversed:
Launch Karma server without singlerun or autowatch and use the runner to launch the tests.

const serve = done => {
    const config = {
        singleRun: false,
        autoWatch: false,
        ...
    };
    const server = new Server(config, onFinish(done));
    server.on('browsers_ready', () => done());
    server.start();
};

const runTests = done => runner.run({}, () => done()});

This case will always fire OnBrowserComplete after onRunComplete.

(for information: these 2 functions are in fact 2 Gulp tasks and gulp-watch will trigger runTests when source files change, but I don't think that is really relevant here)

@nicojs
Copy link
Contributor

nicojs commented Jun 29, 2016

@RIAstar I've been able to reproduce your assumption. See karma-bug-in-reporter for a really small test project which proves it.

Now, can anyone point us in the right direction on how to solve it?

@dignifiedquire
Copy link
Member

I've implemented a fix, thanks for the research

@nicojs
Copy link
Contributor

nicojs commented Jun 30, 2016

@dignifiedquire Thanks for your help!

I understand that the setTimeout fixes the issue, but what caused it? Maybe we can solve it without adding the overhead of a setTimeout. Even with 0ms, you'll have to wait till the next microtask loop (or whatever) before the event is triggered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants