Version Information
Version of Akka.NET? v1.5.35 and earlier
Which Akka.NET Modules? Akka.Cluster.Sharding and Akka.Persistence

Describe the performance issue

So this is really an Akka.Persistence issue, but Akka.Cluster.Sharding is impacted by it.

Imagine the following scenario: 1m remembered-entity actors, each one having 3-4 children that might also use Akka.Persistence (so we're looking at a total of 3-4m persistent actors total) spread out across 100 shards on 10 nodes.

1 node goes down, 10 shards with ~30-40k persistent actors apiece need to get rebalanced across the 9 remaining nodes.

The first shard on one of the newly allocated nodes begins recovering its remembered entities, but as soon as a second shard tries to recover it runs into

Uh oh, now there's a long queue with 10s of thousands of entity actors all trying to recover - and so the remember-entities infrastructure hits this timeout

And over and over again, starts experiencing massive recovery delays that eventually result in the infrastructure failing and the sharding system not being able to recover.

The fix for this is to do what Akka.Cluster.Sharding has supported for years: have Sharding use its own Journal and SnapshotStore so it can have a separate recovery queue from everybody else. That was, in theory, how I would have solved this issue.

To my surprise, however, this is not possible due to a puzzling design decision in the internals of Akka.Persistence:

This means that there is a single RecoveryPermitter instance spread out across all journals - so there's no way of addressing this prioritization problem through giving the sharding system its own journal. There is therefore, no way of alleviating this recovery pressure on the sharding system. That's a problem

Expected behavior

Additional journals / snapshot stores should all have their own separate recovery permitter.

Actual behavior

They all share the same recovery permitter.

Environment
Are you running on Linux? Windows? Docker? Which version of .NET?

Additional context

I think the reason why it was designed this way is because a "recovery" might constitute separate journals / snapshot stores, so which piece of infrastructure should get to decide if the recovery permitter gets "scoped" to specific plugins?

My advice: the journal is the clear winner here as it has the heaviest burden in terms of total number of records that need to be retrieved, snapshots do not. So the recovery permitter should be scoped only to the journal - snapshot stores should have no say in the matter.