-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gracefully shutting down a TLS server sometimes leads to the client not receiving a response #3792
Comments
Have you been able to trace what is happening? One thing that sort of sounds like is that the TLS stream perhaps hasn't flushed the response before closing? |
But do you know for sure all the requests have indeed started? Or could the shutdown be triggered just before hyper has been able to see the request bytes? |
Seems to be the problem. Most of this comment is how I arrived at that. Skip to the end for some potential solutions. Did a bit of Hmm, so if I sleep for Inlining the snippet code here for convenience: loop {
tokio::select! {
_ = &mut shutdown_receiver => {
// ADDING THIS SLEEP MAKES THE TEST PASS
tokio::time::sleep(std::time::Duration::from_millis(1)).await;
drop(shut_down_connections_rx);
break;
}
conn = tcp_listener.accept() => {
tokio::spawn(
handle_tcp_conn(
conn,
wait_for_request_to_complete_rx.clone(),
shut_down_connections_tx.clone(),
tls_config
)
);
}
}
} The Inlining the snippet here for convenience: tokio::select! {
result = conn.as_mut() => {
if let Err(err) = result {
dbg!(err);
}
}
_ = should_shut_down_connection => {
// TEST STILL FAILS IF WE SLEEP RIGHT HERE
conn.as_mut().graceful_shutdown();
let result = conn.as_mut().await;
if let Err(err) = result {
dbg!(err);
}
}
}; Key PointsThe test passes if we sleep for a millisecond before sending on the channel that leads to If I instead move the sleep to just before the This suggests that the problem occurs when we call It looks like
Yeah seems like this is the problem. ProblemCurrently, if a user opens a TCP connection to a server and the server calls This means that if the client has just begun transmitting packets, but the server has not received them, the client will get an error. Potential SolutionsPotential Solution - change how
|
I'm observing errors while testing graceful shutdown of a hyper server.
When gracefully shutting down a TLS connection the client will sometimes get an
IncompleteMessage
error.This only happens for TLS connections. The graceful shutdown process is always successful for non-TLS connections.
Given the following testing steps:
(Since the request handler has a random sleep between 5-10ms we can be reasonably confident
that when we receive a response there are still some other requests that are in-progress.)
When the hyper server is not using TLS, the test pass.
When the hyper server is using TLS, the test fails with an
IncompleteMessage
error.I've created a repository that reproduces the issue https://github.com/chinedufn/hyper-tls-graceful-shutdown-issue
Here's a quick snippet of the graceful shutdown code:
Here is the full source code for convenience (also available in the linked repository)
Cargo.toml (click to expand)
Rust code (click to expand)
The text was updated successfully, but these errors were encountered: