MP2 - V2.1 MP2-632: ResourceServer deadlock on shutdown (1 Viewer)

[tlhackque]

MP 2.1 pre 2 on Windows 10. Client/server on one desktop box. MP exclusively used for TV, but desktop always has other applications running.

When the desktop is rebooted, I still (almost always) see the failure of the client to connect reported previously. (Red x, no guide, you found a race condition in startup of 2.1 pre, but that fix hasn't resolved this for me.) Some combination of restarting the server service & killing & restarting the client eventually gets them to connect. I don't have a recipe.

Probably related: When I close the client (BTW, it's usually a small window, not full screen), the window disappears as expected. But task manager shows the client running as a background process. If I try to re-open the client, nothing happens. If I kill the background client process(es), re-opening the client works as expected. I've taken to minimizing the client as a work-around, but it's still annoying. It seems to me that tearing down the window isn't causing the client to exit; attempting to activate the client sees the background (former client) process, so removes the duplicate - but a window isn't re-created. In any case, the background process shouldn't stay around consuming memory,

Log attached.

 

[Brownard]

Hi, your logs show these errors.

[2017-03-18 12:45:28,874] [2291 ] [Main ] [WARN ] - ResourceServer: Error starting HTTP server (IPv4)
[2017-03-18 12:45:28,876] [2293 ] [Main ] [WARN ] - ResourceServer: Error starting HTTP server (IPv6)

I've not seen those errors before so not sure about the cause, debug logs might show some more info.
Do you have any security software installed? Have you allowed the client through Windows Firewall or similar?

 

[tlhackque]

I don't believe the HTTP errors relate to the client GUI not starting. I'm mildly curious about that, but it doesn't seem to affect anything .. that I can see.

Anyhow, yes I have Norton Internet security. BUT that can't be the issue, since the client works once (a) the service is killed and restarted after boot and (b) thereafter, as long as I kill the windowless stale client process that hangs around after the client window is closed. NIS logs don't show any activity blocked by the firewall.

I enabled debug logging, closed the client, tried to activate it (and got the usual nothing). Killed the stale client, opened the client successfully, and repeated the process. This part is 100% reproducible.

log file for this attached. I didn't do anything else - the client opened to the "Settings" icon; I didn't navigate anywhere else.

 

[breese]

Seems you have 2 sub-nets on your network.
Can you bind the network for MP2 to the correct IP, stop and restart the MP2-Server process and try again.
Settings . General / Connectivity

 

[tlhackque]

There are more than two subnets.... But note that the client and server are on the same box, so subnet really doesn't matter. Anyhow, per your suggestion I bound to a routable Ipv4 address and to localhost (127.0.0.1 & ::1) No change. It's still the case that closing the client leaves a background process (or sometimes 2 - the second being FFMPEG).

Again, closing the client leaves MP2-Client.exe running (no change in pid), although the window disappears & Task Manger moves it from an "App" to a "Background Process" on the summary display.

If I relaunch the client, task manager (details) shows a new copy of MP2-Client briefly. (I assume it notices the first copy & exits.) The (original) client is not exiting; it has to be killed from Task Manager.

I doubt it's helpful (as I don't have symbols), but I tried attaching to the background MP2-Client. As one might expect, it's waiting on a handle:
> mscorlib.dll!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[] waitHandles, int millisecondsTimeout, bool exitContext) + 0x92 bytes
MediaPortal.Common.dll!MediaPortal.Common.Services.Threading.WorkQueue.Dequeue(int timeoutMilliSeconds, System.Threading.WaitHandle cancelHandle) + 0x154 bytes
MediaPortal.Common.dll!MediaPortal.Common.Services.Threading.ThreadPool.ProcessQueue() + 0x7b bytes
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) + 0x9d bytes
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0xc2 bytes
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x16 bytes
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x41 bytes
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() + 0x44 bytes
[Native to Managed Transition]


I triggered a minidump - perhaps you can get further with it. It's compressed with 7-Zip.

Logs and IPCONFIG attached.

 

[tlhackque]

I updated yesterday's post & attached a minidump of a client process that failed to exit.

Hope this helps.

 

[Brownard]

Thanks, I can see in the logs that most of the shutdown routine is not being executed, so it's blocked/deadlocked somewhere. I'm struggling to see where in the code this could happen, except that it happens as it tries to shutdown the ResourceServer which is suspicious given the errors in starting the ResourceServer. Though even then I can't see why it would block.
Can we try and get the ResourceServer to start and see if it eliminates the problem and work from there. Try explicitly setting a port for the server to a known free port (Settings->General->Connectivity->Http server port) then restart the client and see if it helps.
Is there any other reason that you know of in your setup why trying to bind a socket to the IP address/port would fail?

 

[Brownard]

@morpheus_xx I could reproduce the above issue by setting the port to a non-free port which causes the ResourceServer to fail to start. The issue is that when trying to subsequently stop the server it waits for the shutdownEvent to be set, but that's only ever set in the OnAccept method, which doesn't ever get called if the listener failed to start so it just waits there indefinitely.
Any suggestions for a solution? Should we set it immediately if the listener fails to start or avoid calling stop in the first place or...?

EDIT: Added JIRA

 

[tlhackque]

The http server port was set to 80. (Which I did not expect to be in use; I tracked it down to pid 4 "System NT Kernel & System" & then discovered that IIS was installed for some reason - perhaps it's a default in Windoze 10. I'll remove it at the next reboot.)

Meantime, I changed the port to 62134 & restarted the server service . MP2_Client seems to hang around for a while, but now does eventually exit.

Suggestion: When setting the port, see if it's free or if not, if your server responds. If neither is the case, provide an error & don't take the setting. And if the resource server (whatever that is) fails to start, tell me somewhere (in addition to a log file - I need something in the GUI that tells me what to look for in the log file.

Connecting with a web browser to http://localhost:62134/ now returns "Resource not found: http://localhost:62134/, so it would seem to be your service.

I'll turn off debug logging. I'll also see if this resolves the need to kill/restart the service after reboot -- the next time I reboot.

I am slightly curious about what the "resource server" is - I haven't noticed anything not working, but then I'm hardly an advanced user of MP.

Yes, setting your shutdownEvent if the listener fails to start seems reasonable (without reading your code).

Thanks for the help.

 

[Brownard]

Suggestion: When setting the port, see if it's free or if not, if your server responds. If neither is the case, provide an error & don't take the setting. And if the resource server (whatever that is) fails to start, tell me somewhere (in addition to a log file - I need something in the GUI that tells me what to look for in the log file.

Leaving the port as 0 (the default) is supposed to automatically set it to a free port, was that not the case for you? It's quite a low level component which should 'just work' (I don't think there's ever previously been a report of it not working), which is why there isn't any notification outside of the logs.

I am slightly curious about what the "resource server" is

I believe its used in a 'multi-seat' environment, where there is more than 1 client on the network, it allows the possibility to configure clients to view media on the other client's local machine.