Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock in +[FPRRemoteConfigFlags sharedInstance] #6287

Closed
mike-kerley opened this issue Aug 18, 2020 · 6 comments
Closed

Deadlock in +[FPRRemoteConfigFlags sharedInstance] #6287

mike-kerley opened this issue Aug 18, 2020 · 6 comments
Assignees
@visumickey @alikn @kim-f @yalunqin
Labels
api: performance
Milestone
Firebase 7 - M82

Comments

@mike-kerley
Copy link

[REQUIRED] Step 1: Describe your environment

  • Xcode version: 11.5
  • Firebase SDK version: 6.30.0
  • Firebase Component: Performance
  • Component version: 6.30.0?
  • Installation method: Carthage

[REQUIRED] Step 2: Describe the problem

I've been experiencing some intermittent crashes when running my app's tests. The crash dump points to:

Crashed Thread:        11  Dispatch queue: com.apple.dt.xctest.xctwaiter

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
abort() called

...

    Thread 11 Crashed:: Dispatch queue: com.apple.dt.xctest.xctwaiter
    0   libsystem_kernel.dylib              0x00007fff51af733a __pthread_kill + 10
    1   libsystem_pthread.dylib             0x00007fff51ba1e60 pthread_kill + 430
    2   libsystem_c.dylib                   0x00007fff51a868f6 __abort + 139
    3   libsystem_c.dylib                   0x00007fff51a8686b abort + 135
    4   com.apple.dt.XCTest                 0x000000011a3c9360 +[XCTWaiter handleStalledWaiter:] + 224

Whenever this happens, I see two other threads stuck in a wait state inside +[FPRRemoteConfigFlags sharedInstance] (or somewhere deeper in the stack), such as:

Thread 0:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib              0x00007fff51af255e __ulock_wait + 10
1   libdispatch.dylib                   0x00007fff5197a0cc _dlock_wait + 44
2   libdispatch.dylib                   0x00007fff5197a04b _dispatch_once_wait + 80
3   redacted                            0x0000000111ab5201 +[FPRRemoteConfigFlags sharedInstance] + 45
4   redacted                            0x0000000111aa49e0 -[FPRConfigurations setupRemoteConfigFlags] + 140
5   redacted                            0x0000000111aa491b -[FPRConfigurations update] + 71
6   com.apple.CoreFoundation            0x00007fff23d68d0c __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 12
7   com.apple.CoreFoundation            0x00007fff23d68185 _CFXRegistrationPost1 + 421
8   com.apple.CoreFoundation            0x00007fff23d67ef1 ___CFXNotificationPost_block_invoke + 193
9   com.apple.CoreFoundation            0x00007fff23e652d3 -[_CFXNotificationRegistrar find:object:observer:enumerator:] + 1795
10  com.apple.CoreFoundation            0x00007fff23d67846 _CFXNotificationPost + 950
11  com.apple.Foundation                0x00007fff2590de2b -[NSNotificationCenter postNotificationName:object:userInfo:] + 59
...

Thread 6:: Dispatch queue: com.google.perf.FPREventsQueue
0   libsystem_kernel.dylib              0x00007fff51af3882 __psynch_cvwait + 10
1   libsystem_pthread.dylib             0x00007fff51ba2425 _pthread_cond_wait + 698
2   com.apple.Foundation                0x00007fff25913646 -[NSOperation waitUntilFinished] + 706
3   com.apple.Foundation                0x00007fff2590d65d -[__NSObserver _doit:] + 232
4   com.apple.CoreFoundation            0x00007fff23d68d0c __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 12
5   com.apple.CoreFoundation            0x00007fff23d68185 _CFXRegistrationPost1 + 421
6   com.apple.CoreFoundation            0x00007fff23d67ef1 ___CFXNotificationPost_block_invoke + 193
7   com.apple.CoreFoundation            0x00007fff23e652d3 -[_CFXNotificationRegistrar find:object:observer:enumerator:] + 1795
8   com.apple.CoreFoundation            0x00007fff23d67846 _CFXNotificationPost + 950
9   com.apple.Foundation                0x00007fff2590de2b -[NSNotificationCenter postNotificationName:object:userInfo:] + 59
10  redacted                            0x0000000111af511c -[RCNUserDefaultsManager resetInstanceUserDefaults] + 210
11  redacted                            0x0000000111af05ee -[RCNConfigSettings initWithDatabaseManager:namespace:firebaseAppName:googleAppID:] + 682
12  redacted                            0x0000000111ae3282 -[FIRRemoteConfig initWithAppName:FIROptions:namespace:DBManager:configContent:analytics:] + 331
13  redacted                            0x0000000111ae677c -[FIRRemoteConfigComponent remoteConfigForNamespace:] + 771
14  redacted                            0x0000000111ae3006 +[FIRRemoteConfig remoteConfigWithFIRNamespace:app:] + 133
15  redacted                            0x0000000111ab5250 __38+[FPRRemoteConfigFlags sharedInstance]_block_invoke + 76
16  libdispatch.dylib                   0x00007fff519798cb _dispatch_client_callout + 8
17  libdispatch.dylib                   0x00007fff5197ab02 _dispatch_once_callout + 20
18  redacted                            0x0000000111ab5201 +[FPRRemoteConfigFlags sharedInstance] + 45
19  redacted                            0x0000000111aa49e0 -[FPRConfigurations setupRemoteConfigFlags] + 140
20  redacted                            0x0000000111aa491b -[FPRConfigurations update] + 71
21  redacted                            0x0000000111aa2e7b __42-[FPRClient startWithConfiguration:error:]_block_invoke + 140
...

Steps to reproduce:

This happens intermittently when running tests in Jenkins CI builds. CI nodes are running Mac OS X 10.15.5 (19F101). I haven't seen it on my local machine, nor when running the app normally.

The failing test targets use the main app as a host so we can take UI snapshots. Unit tests without the host app appear unaffected. It seems to happen much more frequently since I updated from Fabric Crashlytics to Firebase Crashlytics; maybe that's affecting the timing somehow?

I'd appreciate any help/workarounds or a bug fix. Thanks!
@kim-f
Copy link

kim-f commented Aug 20, 2020

Hello @mike-kerley, thank you for reporting this issue and providing us with the snippets and environment details!

While we work on investigating and narrowing down a root cause, it would be helpful to know what iOS version you're seeing this issue occur within. If you're able to provide this information, please do so -- otherwise we'll continue investigating without it.

Thanks again

@mike-kerley
Copy link
Author

Sure - it was with iOS 13.4. Thanks for looking into it!

@kim-f
Copy link

kim-f commented Aug 20, 2020

Great, thank you!

note: this issue is being tracked internally at b/165809929

@yalunqin
Copy link

Just an update, we found the race condition that caused the issue. Will fix it soon.

@paulb777 paulb777 added this to the Firebase 7 - M82 milestone Oct 8, 2020
@visumickey
Copy link
Contributor

This issue is fixed and will be released in the upcoming release.

@paulb777
Copy link
Member

paulb777 commented Oct 9, 2020

Closing. Please watch the Firebase 7 Milestone to track availability.

@paulb777 paulb777 closed this as completed Oct 9, 2020
@firebase firebase locked and limited conversation to collaborators Nov 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@visumickey visumickey

@alikn alikn

@kim-f kim-f

@yalunqin yalunqin

Labels
api: performance
Projects
None yet
Milestone
Firebase 7 - M82
Development

No branches or pull requests
8 participants
@paulb777 @visumickey @morganchen12 @alikn @google-oss-bot @kim-f @mike-kerley @yalunqin