Akka.Streams: Memory Leak with circular GraphInterpreter reference #6947
Description
Version Information
Akka.Streams 1.5.13
Describe the bug
GraphInterpreter <> GraphStageLogic
and/or
GraphInterpreter <> Connections <> GraphStageLogic
and/or
GraphInterpreter <> GraphInterpreterShell
generating a circular reference and the full Graph and its stages
are found under gcdump "dead objects".
the result will be an OOM
To Reproduce
I am getting this issue with an grpc service to akka.remote (streamRef)
but it should be already be present with an local system.
The Graph/Stages instance is relatively small, but if used with Source.From(IEnumerable<T>)
this Stage will hold the full IEnumerable<T>
as reference.
If it contains >1MB on data it will lead to an OOM faster.
There are no errors with the following code.
But it leaks the full MyRecord entries in memory until OOM.
//inside grpc service
public override async Task<UpdateResponse> Update(UpdateRequest request, ServerCallContext context)
var records = Enumerable.Empty<MyRecord>(); //over 1MB memory
var factory = _services.GetRequiredService<Akka.Actor.IActorRefFactory>();
//defensive instance, it does not free/fix this issue
using var materializer = ActorMaterializer.Create(factory, factory.Materializer().Settings, "update");
try
{
var offer = await _domain.Ask<MyDomainTenantMessages.UpdateStreamOffer>(
new MyDomainTenantCommands.GetUpdateRecordStream(_domain.Tenant),
context.CancellationToken);
var streamTask = Source.From(records)
.CompletionTimeout(context.Deadline - DateTime.UtcNow)
.WatchTermination(Keep.Right)
.To(offer.SinkRef.Sink)
.Run(materializer);
await streamTask.WaitAsync(context.CancellationToken);
return new UpdateResponse
{
Changes = count //todo calc real change count
};
}
catch (Exception ex)
{
_logger.LogError(ex, "update records failed");
throw new RpcException(new Status(StatusCode.Internal, ex.Message));
}
}
Expected behavior
All instances of the Enumerable and GraphInterpreter and Stages should be collectable by the GC
after the successful or unsuccessful execution of the stream.
Actual behavior
The GraphInterpreter Stages and the full used Enumerable are found in "dead objects"
with circular references and getting never collected.
(sometimes after the OOM and a successful ActorSystem termination).
Environment
dotnet 6.0
ubuntu-jammy
Docker Desktop and k8n
Additional context
Because the GraphInterpreter
set the GraphStageLogic.Interpreter
property
I tried to unset it on TerminateStage
but this breaks a lot of assumptions/tests
that the GraphStageLogic.Interpreter
is always available even after stage termination.
The one liner can be found here:
Zetanova@9842d4d