Hi @fanwgwg, thank you for reporting this issue. I am unable to download the DB via the link provided. Could you please check if there are any abnormal transactions that trying to mutate a same document at the same time, or anything that would caught your eyes?

I wonder if this was caused by your persistence settings + large volume of mutations. But without a reproducible code, it is hard to tell what is the root cause.

@milaGGL The link should be valid as I'm able to download it from Incognito mode (unless you meant that it cannot be virus scanned by Google drive due to file being too big, as shown in the screenshot).

For persistence settings, our app does not override any Firestore's persistence settings, so everything about offline persistence should be the default.

For abnormal transactions, we did not identify any of it. Also, that's the reason why I did a dump of the firestore database when the "stuck" happens, which is to send to Firestore engineers for a further inspection in case there is anything mysterious. But for us, we never reproduced the same issue again so it's difficult for us to tell if there's anything abnormal.

Regarding this:

check if there are any abnormal transactions that trying to mutate a same document at the same time

I'm unable to tell if this has happened, as we never encountered the same issue during our 1 year development time with firestore android, so this must be a rare or race condition. However, I recall that all mutations to firestore android sdk is enqueued sequentially:

var bgExecutor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(4));
var docRef = db.collection("test").document("a");
for (int i = 0; i <1000; i++) {
   bgExecutor.submit(() -> docRef.set(ImmutableMap.of("field_1", "value_1")));
   bgExecutor.submit(() -> docRef.delete());
   bgExecutor.submit(() -> docRef.set(ImmutableMap.of("field_1", "value_2")));
}

Please correct me if I'm not understanding it correctly

@fanwgwg, yes, that is correct. Quick question, have there been any updates to your app recently, like upgrading to a newer version of the SDK?

@milaGGL We've been following the latest version and Firebase BOM releases, typically we always update to the latest version within 1 week of its release. The latest firebase android BOM update was 32.3.1 on September 15, 2023, and we updated to this version on Sep 16th.

When this issue happened (around Sep 20-ish), we were already using the 32.3.1 release.

@milaGGL I've just encountered another case of "stuck" today, this time I've enabled Firestore log and here's log output from starting the app to when firestore got stuck: Google Drive Link

One thing I found is that each time the "stuck" happens, the log ends with these two lines. After these two lines, no matter how long you wait for (no matter hours or days), firestore no longer prints any logs. Any further enqueued Task to Firestore will never complete, neither will them fail, but just in a waiting state.

(24.9.0) [Persistence]: Starting transaction: Collect garbage
(24.9.0) [LruGarbageCollector]: Capping sequence numbers to collect down to the maximum of 1000 from 1059

@milaGGL I think I've found something that might be very close to the cause, there is a infinite loop running in this method: SQLiteLruReferenceDelegate#removeOrphanedDocuments:

I've attached a screenshot of debugging at the breakpoints inside the method, as you can see, the upperBound = 19923, but each time inside the while loop, the docsToRemove = 0, therefore this loop runs indefinitely.

@fanwgwg, this is amazing! I wonder if !isPinned(key) ever returns true and gets inside the if block to populate the docsToRemove.
Would you be able to provide a minimal repro app?

Yes isPinned(key) always returns true because mutationQueuesContainKey(key) always return true.

Unfortunately I do not have a way to consistently reproduce the same issue, except that it only happens when there is a large volume of mutations. Even for large volume of mutations, which we often perform such tests on a daily basis, I've only encountered it twice so far.

In case anyone else is encountering the same issue, our mitigation for the issue is to disable persistence because our app doesn't use the persistence feature, i.e.,

    FirebaseFirestore.getInstance()
        .setFirestoreSettings(
            new FirebaseFirestoreSettings.Builder().setPersistenceEnabled(false).build());

@milaGGL @ehsannas Friendly ping, is there any update on this issue?

While disabling persistence works for now, we don't like this approach as mitigation as it might prevent us from developing features that rely on offline persistence, which is one of the core features of Firestore.

Just thinking out loud based on my superficial understanding of the SDK, it seems like that:

1. Firestore runs a garbage collection logic to remove orphaned documents from its persistent cache if the cache becomes too large
2. When trying to remove the orphaned documents, it looks up documents according to sequence number (maybe trying to look for the oldest documents?) and then try to remove them. However, if the document is in the mutation queue, it will skip.
3. What if there is a case where the cache is small, but mutation queue contains so many documents that all documents in cache also happen to be inside the mutation queue? Does this sound like what's happening here?

@fanwgwg Thanks for the detailed investigation, I think you are right that there is a bug in removeOrphanedDocuments.

Your operation put the SDK's GC under pressure, it try to go through all orphaned documents in batches (with batch size REMOVE_ORPHANED_DOCUMENTS_BATCH_SIZE), and it will stop when it processed a batch of a size different than REMOVE_ORPHANED_DOCUMENTS_BATCH_SIZE, this means it has processed all the orphaned documents with sequence number in scope.

Typically, the number of orphaned documents do not exceed REMOVE_ORPHANED_DOCUMENTS_BATCH_SIZE, thus the loop breaks with one iteration. The operation you did triggered the loop into running multiple iterations.

Unfortunately, from the second iteration onwards, the SDK did not "resume" the query from iteration 1, and instead it issue the same query from iteration 1, thus an infinite loop.

We will get this fixed ASAP.

@wu-hui Thanks for the quick response! Looking forward to the fix!