SSDPServerController.OnExpirationTimerElapsed: Cannot acquire synchronization lock. Maybe a deadlock (3 Viewers)

[mm1352000]

*bump*
@MrTechno
@breese

Did your investigations lead anywhere?
This problem is rearing its head again over here:
https://forum.team-mediaportal.com/posts/1094514/

A solution or even some suggestions would be great.

 

[MrTechno]

Adding IP filtering to MP2 server reduced the lock ups but they still happened on my machine. I haven't found the root cause, I just disabled VPN.

 

[mm1352000]

Well hmmm...

The logs in the other thread seem to only show the message after TVE attempts to call SSDPClientController.Close(), whereas the logs in this thread seem to show the message while MP2 Server should be running and the SSDP controller still active. In short: I suspect the existence of two distinct causes or scenarios. TVE's Close() issue is probably much easier to resolve.

Looking at the Close() function:
https://github.com/MediaPortal/Medi...tructure/CP/SSDP/SSDPClientController.cs#L350

...it seems odd to me is that _isActive = false and the socket = null loop aren't all within the same lock. Surely an atomic/locked shutdown would be cleaner/better? The assumption seems to be that if Close() acquires the lock and is able to set _isActive false then the Receive() handling will "die" without any major problem and we should be okay to close the sockets... but I'm not so sure about that assumption.

Looking at the Receive() functions:
https://github.com/MediaPortal/Medi...tructure/CP/SSDP/SSDPClientController.cs#L111
https://github.com/MediaPortal/Medi...tructure/CP/SSDP/SSDPClientController.cs#L149

Obviously they only acquire the lock for checking _isActive and then release it immediately. Crucially: there is no locking around the socket access that happens later (eg. socket.EndReceiveFrom()).

Consider the following sequence with two threads.
1. Thread A: Receive() invoked, lock acquired, _isActive is true so processing will continue. Lock released.
2. Thread B: Close() invoked, lock acquired, _isActive is set false. Lock released.
3. Thread A: The if (socket == null) check is passed before Thread B sets the shared socket reference to null. Local socket reference acquired (so Thread B setting socket to null will not have immediate effect).
4. Thread B: Lock acquired, local socket reference acquired, shared socket reference set null. Lock released.
5. Thread A: EndReceiveFrom() invoked on local socket reference.
6. Thread B: Close() invoked on local socket reference either directly or within NetworkHelper.DisposeSSDPMulticastSocket().

What happens?
I don't know for sure, but it might not be good.
If EndReceiveFrom() unblocks on Close() and an exception is thrown in Thread A everything would be fine I think (the exception would be caught and Receive() would return).
However, if Close() waits for EndReceiveFrom() to receive and return but EndReceiveFrom() never does actually receive or time out... that could be our lovely deadlock. Close() blocks TVE forever.

It interests me to see that there is an optional timeout parameter on socket.Close() which we currently do not use.

I suspect there could potentially be other issues lurking in the Receive() processing too, simply due to the fact that _isActive is only checked at the start of Receive() and not anywhere within the processing (and Close() does not close atomically within the lock).

Plausible?
Garbage?

 

[breese]

I have too as seeing one thread is MP1 and another thread is MP2, is there common code (outside of TVE) that might produce this issue?

 

[mm1352000]

is there common code (outside of TVE) that might produce this issue?

Not if you exclude TVE. The issue is in MP2's UPnP library (which is used for TVE CableCARD tuner support) - that much is very clear. With TVE, even if you don't have a CableCARD tuner, the SSDP/UPnP detector is always used so there is always the potential for problems.

 

[mm1352000]

@MrTechno
If you're able to do something to trigger or make the problem more likely again, maybe you could try this:
http://blog.scriptico.com/04/debugging-with-windbg-deadlocks-in-applications/

Seems to promise a quite accurate way to locate the threads and lines of code involved in a deadlock. It is a little involved, but surely easier than other deadlock debugging methods I've previously encountered.

 

[pnyberg]

Hello,

I have this with the latest MP2 Summer 2015 release of MP2 server as well. But only when closing/stopping the server.
Everything works okay until, stopping the server, eventually it will stop - but is this known error?

Attaching log.