v6 coverage runs can fail dues to maxHttpBufferSize change in socket.io v3 #3621
Comments
|
This looks similar to an issue we have been working on (but it happens in
karma v5). A colleague analyzed it and believes that the problem is
related to lost socket io packets during the second (reconnecting) upgrade.
The first upgrade happens early and has no contention from the page load.
But the problem cases are attempting to reconnect during the page load
processing when the browser is running flat out to compile and load
incoming js files. The contention delays the upgrade and during that time
socket events get queued or lost. Try setting longer timeouts until we
sort this out one out.
…On Fri, Jan 15, 2021 at 9:24 AM David DOLCIMASCOLO ***@***.***> wrote:
Hi guys,
Our build starts to fail when we upgrade karma to v6.0.0. All tests run
without problem, but Karma thinks the browser disconnects right after
executing the tests (or the browser actually disconnects, but that doesn't
make Karma yell in v5.2).
I tracked it down and reproduced with a single test running in .only
(we're using Mocha as a framework) so I suppose it's not a problem with the
actual tests, but maybe related to the actual amount of JS code we're
oading in the browser. Again, in v5.2 they run fine.
I'm attaching karma.v5.log
<https://github.com/karma-runner/karma/files/5821996/karma.v5.log> and
karma.v6.log
<https://github.com/karma-runner/karma/files/5821998/karma.v6.log> in
DEBUG level that show the exact same run, one with 5.2.3 and the other with
6.0.0. The only step in-between these two runs is the upgrade of Karma.
A few additional information on our setup (and you have the full config
object at the beginning of the log file)
- The actual application is using AngularJS
- We use Mocha as our test framework
- We run tests in Chrome headless only, using karma-chrome-launcher
<https://www.npmjs.com/package/karma-chrome-launcher>
- We use karma-coverage-istanbul-instrumenter
<https://www.npmjs.com/package/karma-coverage-istanbul-instrumenter>
and karma-coverage-istanbul-reporter
<https://www.npmjs.com/package/karma-coverage-istanbul-reporter> to
generate coverage report information
- We preprocess HTML templates to make them AngularJS modules (yeah...
more JS code loaded :p)
I'm kinda stuck now, not knowing where to go to track the problem even
further. I will provide any additional information you may require but
would appreciate help from Karma's community. In addition, I volounteer to
make a PR if we eventually find a problem in Karma itself !
Thanks,
Regards,
David
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#3621>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABSGAW62I5EBYCBBKCZKI3S2B25NANCNFSM4WELTU3Q>
.
|
|
Thx, I'm not sure which timeout you're referring to. Tests do run at "normal" speed and complete just fine. But a few milliseconds later instead of the typical tests-done-browser-closing scenario I'm seeing an error about browser disconnecting, then reconnecting and then nothing until "no message in 30s" error, which I assume is expected because tests are done so there nothing left for the browser to do... |
|
My interpretation of your scenario is that the test "complete just fine"
but then fail to report results to the server. The results get lost during
the reconnect and thus the server eventually assumes the browser crashed
and says "no message". Try setting a larger pingTimeout
http://karma-runner.github.io/6.0/config/configuration-file.html
…On Sat, Jan 16, 2021 at 7:49 AM David DOLCIMASCOLO ***@***.***> wrote:
Thx, I'm not sure which timeout you're referring to. Tests do run at
"normal" speed and complete just fine. But a few milliseconds later instead
of the typical tests-done-browser-closing scenario I'm seeing an error
about browser disconnecting, then reconnecting and then nothing until "no
message in 30s" error, which I assume is expected because tests are done so
there nothing left for the browser to do...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#3621 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABSGATROJT25HCQ3VUMCTLS2GYSFANCNFSM4WELTU3Q>
.
|
|
Thx, I'll try and report here |
|
We experienced a similar issue after upgrading from v5.2.3 to 6.0.0. I attempted changing the I discovered that whether the issue occurred depended on the amount of code in the test bundle. I tried various combinations of files and eventually discovered that it would reliably succeed with a specific number of files fully included and one file included with certain statements commented out. If I uncommented one more statement, it would fail. After adding some low-level logging to Socket.IO, I found that when the amount of code was just small enough to allow the tests to pass, the raw This is coming from In our project we are using Istanbul for code coverage. The "complete" message sent at the end of the test run includes all of the code coverage data. It seems that if there is enough code, this data can push the size of the "complete" message past this threshold. I see from the changelog that one of the changes in Karma v6 is an upgrade to Socket.io v3, so I guess this is probably related. Edit: I see the original reporter was also using Istanbul as well, so they probably had the same issue. |
|
The maximum payload size is coming from here in engine.io: https://github.com/socketio/engine.io/blob/e5b307c16d8e7594fcec4eb23508f23f78546dc6/lib/server.js#L29. Changing the The fact that Istanbul sends all of its code coverage data over the WebSocket in one request at the end of the test run is probably not ideal, but that's going to be harder to fix. |
|
I just ran into the same issue. After executing tests in my project the debug output shows a message similar to the one mentioned in #3359 (comment): And the last sent message is rather large (due to coverage data):
Increasing the default buffer size for the socket.io server also fixes the problem for me. |
|
Good catch @robertknight and thx for investigating this. As you've noted, we're indeed using Istanbul for our code coverage report and probably fall into the same hole... So the idea is to make socket.io's maxHttpBufferSize configurable through Karma config right? If so, this is something I can handle in a PR. Will look into it ASAP |
|
@johnjbarton Meanwhile, do you mind pinning this issue so that it's easier to find for users suffering the regression? |
|
I think reading |
|
@cyco I think you're right, actually. Identifying and logging the issue is probably the best option. Otherwise people will never know they need to increase that buffer size... Detecting, logging and suggesting to increase the timeout would be a good way to fix the issue I suppose |
Is there anything that karma can do to surface this error message to the user? |
|
I think we can set the From https://socket.io/docs/v3/migrating-from-2-x-to-3-0/index.html |
johnjbarton
changed the title
|
🎉 This issue has been resolved in version 6.0.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
## [6.0.1](karma-runner/karma@v6.0.0...v6.0.1) (2021-01-20) ### Bug Fixes * **server:** set maxHttpBufferSize to the socket.io v2 default ([karma-runner#3626](karma-runner#3626)) ([69baddc](karma-runner@69baddc)), closes [karma-runner#3621](karma-runner#3621) * restore `customFileHandlers` provider ([karma-runner#3624](karma-runner#3624)) ([25d9abb](karma-runner@25d9abb))
Hi guys,
Our build starts to fail when we upgrade karma to v6.0.0. All tests run without problem, but Karma thinks the browser disconnects right after executing the tests (or the browser actually disconnects, but that doesn't make Karma yell in v5.2).
I tracked it down and reproduced with a single test running in
.only(we're using Mocha as a framework) so I suppose it's not a problem with the actual tests, but maybe related to the actual amount of JS code we're oading in the browser. Again, in v5.2 they run fine.I'm attaching karma.v5.log and karma.v6.log in DEBUG level that show the exact same run, one with 5.2.3 and the other with 6.0.0. The only step in-between these two runs is the upgrade of Karma.
A few additional information on our setup (and you have the full
configobject at the beginning of the log file)I'm kinda stuck now, not knowing where to go to track the problem even further. I will provide any additional information you may require but would appreciate help from Karma's community. In addition, I volounteer to make a PR if we eventually find a problem in Karma itself !
Thanks,
Regards,
David
The text was updated successfully, but these errors were encountered: