Skip to content

Akka.Streams: Memory Leak with circular GraphInterpreter reference #6947

Open
@Zetanova

Description

Version Information
Akka.Streams 1.5.13

Describe the bug
GraphInterpreter <> GraphStageLogic
and/or
GraphInterpreter <> Connections <> GraphStageLogic
and/or
GraphInterpreter <> GraphInterpreterShell

generating a circular reference and the full Graph and its stages
are found under gcdump "dead objects".
the result will be an OOM

To Reproduce
I am getting this issue with an grpc service to akka.remote (streamRef)
but it should be already be present with an local system.

The Graph/Stages instance is relatively small, but if used with Source.From(IEnumerable<T>)
this Stage will hold the full IEnumerable<T> as reference.
If it contains >1MB on data it will lead to an OOM faster.

There are no errors with the following code.
But it leaks the full MyRecord entries in memory until OOM.

//inside grpc service
public override async Task<UpdateResponse> Update(UpdateRequest request, ServerCallContext context)

var records = Enumerable.Empty<MyRecord>(); //over 1MB memory

var factory = _services.GetRequiredService<Akka.Actor.IActorRefFactory>();

//defensive instance, it does not free/fix this issue
using var materializer = ActorMaterializer.Create(factory, factory.Materializer().Settings, "update");

try
{
    var offer = await _domain.Ask<MyDomainTenantMessages.UpdateStreamOffer>(
        new MyDomainTenantCommands.GetUpdateRecordStream(_domain.Tenant),
        context.CancellationToken);

    var streamTask = Source.From(records)
        .CompletionTimeout(context.Deadline - DateTime.UtcNow)
        .WatchTermination(Keep.Right)
        .To(offer.SinkRef.Sink)
        .Run(materializer);

    await streamTask.WaitAsync(context.CancellationToken);

    return new UpdateResponse
    {
        Changes = count //todo calc real change count
    };
}
catch (Exception ex)
{
    _logger.LogError(ex, "update records failed");

    throw new RpcException(new Status(StatusCode.Internal, ex.Message));
}
}

Expected behavior
All instances of the Enumerable and GraphInterpreter and Stages should be collectable by the GC
after the successful or unsuccessful execution of the stream.

Actual behavior
The GraphInterpreter Stages and the full used Enumerable are found in "dead objects"
with circular references and getting never collected.
(sometimes after the OOM and a successful ActorSystem termination).

Environment
dotnet 6.0
ubuntu-jammy
Docker Desktop and k8n

Additional context
Because the GraphInterpreter set the GraphStageLogic.Interpreter property
I tried to unset it on TerminateStage but this breaks a lot of assumptions/tests
that the GraphStageLogic.Interpreter is always available even after stage termination.

The one liner can be found here:
Zetanova@9842d4d

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions