comes from #7773 (comment)

Thank you @malinajirka for your comments there, that made me look closer! 🙇

How will creating a new Thread prevent the application to end up in a deadlock? Won't the only effect be that instead of UI thread and Thread XYZ being in deadlock there will be Thread ABC and Thread XYZ in deadlock? So it will actually fix the ANR but not the really issue?

I think you nailed it, good question @malinajirka! And I realize my description of the problem in that PR didn't do justice to the exact specific problem. TBH I had the same doubts as well. I think the way I approached it is thinking we may fix the ANR even though we may not fix the root cause of the problem, as this fix would help avoid the conditions on which the deadlock is likely to happen, which in the end is already a probably good enough improvement.

Let me explain a bit more how I think things go then - looking into it a bit closer, the dump shows this:

"main" tid=1 Blocked 
[...]
at org.wordpress.android.push.GCMMessageService.removeNotification (GCMMessageService.java)
- waiting to lock <0x0d44704c> (a java.lang.Class<org.wordpress.android.push.GCMMessageService>) held by thread 23
  at org.wordpress.android.ui.notifications.NotificationDismissBroadcastReceiver.onReceive (NotificationDismissBroadcastReceiver.java:20)

Note that thread 23 does not appear in the dump (been going through a lot of other reports and none of them seem to have the dump for the blocking thread).

Basically that broadcast receiver is set to be called as the delete intent for the notification, i.e. when the user either dismisses it (slide off the screen) or taps on it, the notification is dismissed and this intent gets triggered, and received by the NotificationDismissBroadcastReceiver. What it does internally is it calls removeNotification(), which is locked on the class object itself (GCMMessageService.class) as you've seen.
At the same time the Activity gets launched and the onCreate() method is executed. This method also calls a synchronized method within GCMMessageService.
Here comes the important part: both the Activity lifecycle and the BroadcastReceiver are executed on the main thread (see https://developer.android.com/reference/android/content/BroadcastReceiver#onreceive). I'm tempted to say that's where the deadlock happens, but I'm with you in that there's no 2 threads in this situation, it's the same thread (main thread) locking on the same lock object.

After looking into the reports a bit more, I found something interesting.

There are other similar instances happening, all seem to be due to the synchronized methods in GCMMessageService class
1 - this one happens in WPMainActivity.onResume as it calls GCMMessageService.removeAllNotifications

"main" tid=1 Blocked 
[...]
| held mutexes=
[...]
at org.wordpress.android.push.GCMMessageService.removeAllNotifications (GCMMessageService.java)
- waiting to lock <0x0a59d727> (a java.lang.Class<org.wordpress.android.push.GCMMessageService>) held by thread 71
  at org.wordpress.android.ui.main.WPMainActivity.onResume (WPMainActivity.java:491)

2 - WPMainActivity.onCreate calls WPMainActivity.launchWithNoteId which holds on GCMMessageService.getNotificationsCount()

held mutexes=
  at org.wordpress.android.push.GCMMessageService.getNotificationsCount (GCMMessageService.java)
- waiting to lock <0x0b0f597b> (a java.lang.Class<org.wordpress.android.push.GCMMessageService>) held by thread 68
  at org.wordpress.android.ui.main.WPMainActivity.launchWithNoteId (WPMainActivity.java:399)
  at org.wordpress.android.ui.main.WPMainActivity.onCreate (WPMainActivity.java:267)

In #1, the user taps on a notification and thus that's why WPMainActivity.onResume is executed (main pendingIntent for the notification is to wake it up and launch the NotificationsDetailActivity), but then, also the broadcast receiver should be triggered (and is not shown in the dump). Assuming only the second one to arrive gets dumped, thenit would make sense that this is happening sometimes for one (broadcastreceiver) and sometimes for other entry point in the app where the broadcastreceiver "locked" the object first.

There are many more, and I find a pattern that the blocking thread is not in the dump (the blocked thread only is), but when the blocked thread is NotificationDismissBroadcastReceiver.onReceive then the blocking must be elsewhere, and when the blocked thread is something else, the NotificationDismissBroadcastReceiver.onReceive doesn't appear. This also seems to make sense according to the amount of reports, as we have approximately half of reports on NotificationDismissBroadcastReceiver.onReceive being blocked, and the rest of the reports are other threads supposedly being blocked by the other threads. Given odds of execution should be even, I tend to think that this is, in fact, a deadlock problem.

So, knowledge adds up and gives some meaning to all this information:
1 - tapping on any notification will produce both the actual intent (bringing the app to the foreground, then launching NotificationDetailActivity to show the Note detail), and trigger the NotificationDismissBroadcastReceiver receiver.
2 - the amount of reports and variety tend to show it's always something happening between the NotificationDismissBroadcastReceiver and catching the other thread wanting to access a GCMMessageService synchronized method as well, on any of various entry points which match the lifecycle of the notification-tapped intent.
3 - given this, I think we might have a better chance solving the issue by encapsulating the BroadcastReceiver execution in a different thread, rather than doing it separately on each call from WPMainActivity or NotificationsDetailActivity (also this should be more future-proof).

Therefore, this PR moves the logic to the already existing NotificationsProcessingService, adding a specific action to handle notification dismissal processing, and gets rid of the broadcast receiver.

I think it works fine, and hope will resolve the issue!