Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Akka.IO] TcpListener connection queue problem #5988

Open
Arkatufus opened this issue Jun 10, 2022 · 14 comments
Open

[Akka.IO] TcpListener connection queue problem #5988

Arkatufus opened this issue Jun 10, 2022 · 14 comments

Comments

@Arkatufus
Copy link
Contributor

Arkatufus commented Jun 10, 2022

Version Information
Version of Akka.NET? dev branch
Which Akka.NET Modules? Akka.IO

Describe the bug
The way TcpListener is accepting new connection is buggy. JVM managed TCP sockets through the nio channel selector pattern while we're doing it closer to bare metal by requesting actual sockets for each incoming connection.

The way it is done in JVM is to use an accepting limit that is being decremented each time a new channel is created, in our case we used SocketAsyncEventArgs as an async non-blocking way of accepting connections (https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka/IO/TcpListener.cs#L68-L79).

  1. In JVM all channels are passed to the TcpConnection class and have its lifetime managed by it, in ours, SAEA are stored inside an array and they are supposed to be disposed on TcpListener PostStop. But they're not. This array is being overwritten when a ResumeAccepting command is received, possibly creating a memory leak (https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka/IO/TcpListener.cs#L100-L103)
  2. Instead of being limited by the accept number, we're reusing SAEA objects (which is a really bad idea) and kept accepting new connections when new socket connections were made (https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka/IO/TcpListener.cs#L90-L97)

We can probably improve this this by ditching SAEA and use _socket.AcceptAsync(null, cancellationToken) instead. The SAEA would be managed by the method and all pending accept command can be cancelled using the token.

@Aaronontheweb
Copy link
Member

We can probably improve this this by ditching SAEA and use _socket.AcceptAsync(null, cancellationToken) instead. The SAEA would be managed by the method and all pending accept command can be cancelled using the token.

Is that .NET 6 only?

@Arkatufus
Copy link
Contributor Author

Not sure, I can see the code in .NET 5.0, would have to check the .NET Standard API to see if its there

@Arkatufus
Copy link
Contributor Author

Nope, not available in .NET Standard 2.0. Introduced in .NET 5.0.

@Aaronontheweb
Copy link
Member

We can do an #if #endif for that in the .NET 6 version in Akka.NET v1.5 then

@Aaronontheweb Aaronontheweb added this to the 1.5.0 milestone Jun 10, 2022
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.0, 1.5.1 Mar 2, 2023
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.1, 1.5.2 Mar 15, 2023
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.2, 1.5.3 Apr 6, 2023
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.3, 1.5.4, 1.5.5 Apr 20, 2023
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.5, 1.5.6, 1.5.7 May 4, 2023
@F0b0s
Copy link
Contributor

F0b0s commented May 14, 2023

I'm working with this issue and find that TcpListener and its PullMode params:
_acceptLimit = bind.PullMode ? 0 : _tcp.Settings.BatchAcceptLimit;
probably doesn't work as expected.
If it mean that TcpListener should accept new connection only on Pull, then it doesn't work in this way. It will accept any new connection once it has accepted connection:

                case SocketEvent evt:
                    var saea = evt.Args;
                    if (saea.SocketError == SocketError.Success)
                        Context.ActorOf(Props.Create<TcpIncomingConnection>(_tcp, saea.AcceptSocket, _bind.Handler, _bind.Options, _bind.PullMode).WithDeploy(Deploy.Local));
                    
                    saea.AcceptSocket = null;
                    if (!_socket.AcceptAsync(saea))
                        Self.Tell(new SocketEvent(saea));
                    return true;

_socket.AcceptAsync will be called automatically after accepting current connection.

Do we need to change it to real pull mode?

@F0b0s
Copy link
Contributor

F0b0s commented May 14, 2023

Second question about io.tcp.batch-accept-limit setting. It manages size of array of SocketAsyncEventArgs in TcpListener and documentations says:

The maximum number of connection that are accepted in one go,
higher numbers decrease latency, lower numbers increase fairness on
the worker-dispatcher
batch-accept-limit = 10

If we take a look at MSDN example of SocketAsyncEventArgs usage at this page we can see that it use only one object for socket.AcceptAsync() because its usage is very short and all io operations is performed on connected socket:

SocketAsyncEventArgs readEventArgs = m_readWritePool.Pop();
readEventArgs.UserToken = e.AcceptSocket;

// As soon as the client is connected, post a receive to the connection
bool willRaiseEvent = e.AcceptSocket.ReceiveAsync(readEventArgs);

Moreover, we have this code in TcpListener

case SocketEvent evt:
    var saea = evt.Args;
    if (saea.SocketError == SocketError.Success)
        Context.ActorOf(Props.Create<TcpIncomingConnection>(_tcp, saea.AcceptSocket, _bind.Handler, _bind.Options, _bind.PullMode).WithDeploy(Deploy.Local));
                    
    saea.AcceptSocket = null;
    if (!_socket.AcceptAsync(saea))
        Self.Tell(new SocketEvent(saea));
    return true;

If usage of SAEA is very short in fact we can use only one element of array SAEA.
My proposal is to remove array of SAEA and mark io.tcp.batch-accept-limit setting as obsolete. Then create single SAEA object and hardly use it inside TcpListener.

@Arkatufus
Copy link
Contributor Author

We can't reduce the SAEA to a single SAEA. The point of using asynchronous event model is to make sure that the server socket can accept multiple incoming connection asynchronously without blocking.

This is described in the MS documentation:

In the new System.Net.Sockets.Socket class enhancements, asynchronous socket operations are described by reusable SocketAsyncEventArgs objects allocated and maintained by the application. High-performance socket applications know best the amount of overlapped socket operations that must be sustained. The application can create as many of the SocketAsyncEventArgs objects that it needs. For example, if a server application needs to have 15 socket accept operations outstanding at all times to support incoming client connection rates, it can allocate 15 reusable SocketAsyncEventArgs objects for that purpose.

@Arkatufus
Copy link
Contributor Author

I'm working with this issue and find that TcpListener and its PullMode params: _acceptLimit = bind.PullMode ? 0 : _tcp.Settings.BatchAcceptLimit; probably doesn't work as expected. If it mean that TcpListener should accept new connection only on Pull, then it doesn't work in this way. It will accept any new connection once it has accepted connection:

                case SocketEvent evt:
                    var saea = evt.Args;
                    if (saea.SocketError == SocketError.Success)
                        Context.ActorOf(Props.Create<TcpIncomingConnection>(_tcp, saea.AcceptSocket, _bind.Handler, _bind.Options, _bind.PullMode).WithDeploy(Deploy.Local));
                    
                    saea.AcceptSocket = null;
                    if (!_socket.AcceptAsync(saea))
                        Self.Tell(new SocketEvent(saea));
                    return true;

_socket.AcceptAsync will be called automatically after accepting current connection.

Do we need to change it to real pull mode?

Wouldn't AcceptAsync() be actually needed to start the socket? It just returns a bool to signal if the connection has been accepted, if false then then connection is still being negotiated (handshake) and we can't use the socket yet.

@F0b0s
Copy link
Contributor

F0b0s commented May 17, 2023

We can't reduce the SAEA to a single SAEA. The point of using asynchronous event model is to make sure that the server socket can accept multiple incoming connection asynchronously without blocking.

Ok, i was confused by MS example in the same post.

Wouldn't AcceptAsync() be actually needed to start the socket? It just returns a bool to signal if the connection has been accepted, if false then then connection is still being negotiated (handshake) and we can't use the socket yet.

Yes, but after you call AcceptAsync(saea) you ready to accept new connection and it would be accepted and processed because you have Completed handler. It doesn't wait for next Pull. Take a look:

                case SocketEvent evt:
                    var saea = evt.Args;
                    if (saea.SocketError == SocketError.Success)
                        Context.ActorOf(Props.Create<TcpIncomingConnection>(_tcp, saea.AcceptSocket, _bind.Handler, _bind.Options, _bind.PullMode).WithDeploy(Deploy.Local));
                    
                    saea.AcceptSocket = null;
                    if (!_socket.AcceptAsync(saea))
                        Self.Tell(new SocketEvent(saea));

SocketEvent here means we already got a connection, but we call AcceptAsync(saea) and Self.Tell(new SocketEvent(saea)); in the same handler, we don't wait for Pull. Do we need to change it to real Pull mode?

@Aaronontheweb Aaronontheweb modified the milestones: 1.5.20, 1.5.21 Apr 29, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.21, 1.5.22, 1.5.23 May 28, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.23, 1.5.24, 1.5.25 Jun 6, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.25, 1.5.26 Jun 14, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.26, 1.5.27 Jun 27, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.27, 1.5.28 Jul 25, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.28, 1.5.29 Sep 4, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.29, 1.5.30, 1.5.31 Oct 1, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.31, 1.5.32 Nov 14, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.32, 1.5.33 Dec 4, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.33, 1.5.34 Dec 24, 2024
@Aaronontheweb Aaronontheweb modified the milestones: 1.5.34, 1.5.35 Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

No branches or pull requests

3 participants