[fixed] [MP2-821] Importer glitches in 2.2 (4 Viewers)

[JSchuricht]

So killing mp2-server.exe to restart the import only worked for a short time. I had a sample of 6 movie folders by SMB with 132, 15, 36, 29, 491 and 659 movies in them. I added one folder and let importer finish before adding another. The first 132 have consistently been imported. The 15 folder missed one movie on my first test but got all 15 after a reinstall of MP2 server. The 36 folder missed 6 the first time and the same 6 after a reinstall of MP2 server. The 29 folder missed one movie on the first test and got all after a reinstall of MP2 server. At this point I tried terminating the mp2-server.exe process and restarting the service. The 36 folder found the 6 missing movies. Then I added the 491 folder which imported everything but required terminating mp2-server.exe multiple times. Adding the 659 folder eventually found 99 movies with multiple terminations of mp2-server.exe and has been stalled at that number for two days.

I think I am fighting multiple issues and I am second guessing myself testing that clip I posted and if the results I got were just the importer getting stuck naturally or because of that clip. I have doubts about the clip behaving the same as the full movie. My initial attempts at making a clip all worked in MP2, the one I posted was actually just splitting the full movie into uncompressed zip files and renaming to try preserving whatever was causing the issue to begin with. I'll retest the clip to make sure it is really displaying the same behavior as the full movie. I also had a small sample size for the local movies, I ordered another HDD for the test server so I can increase the sample size rather than assuming the 700 movies I can test with represent all movies working off local storage vs SMB.

So some of the things that are specific to my setup, what I have ruled out and possible things to test. This is just for the general import failing with all individual movie folders showing up under browse media and the movies within not being there, leaving the clip issue for later. Hoping someone sees something I am overlooking here.

MP 2.1 branch working, issue specific to MP 2.2 branch

Windows 10 1903 and 1909 enterprise English language version used for testing
* Could try a German language version of windows to rule out character set differences and importer favoring different sources
* May also try Windows 7, 8 or a server version

Antivirus, tested with Symantec 14 and Windows Defender both with and without exceptions on programdata\TMP and program files\TMP folders

Firewall disabled on all tests

Have done fresh installs of windows not adding anything but MP2 but issue remained

Have run windows as part of a domain and standalone ie. work group mode with no change

Installed MP2 only attaching a server to the client and adding media sources leaving all default settings but issue remained

Network switches changed out (for unrelated upgrade) but issue remained

Moved MP2 server and client to private VLAN to eliminate other network traffic but issue remained

Tried MP2 server virtualized and on physical hardware with no change
*Commonality is Ivy Bridge E3/E5 CPU's with Spectre/Meltdown microcode patches. Could try an older un-patched system or newer hardware

SMB share with movies is on a Windows 2019 server with SMB 3 and active directory for credentials
*Could try forcing a SMB 2 connection, might tie into a Windows 7 test
*Next test is enabling write on SMB 3 share
*After write access, trying local user account for SMB 3 share access

Tried manually specifying SMB path in MP2 using IP vs URN, no change

Movies on SMB share are on REFS partition
*Need to try local storage under REFS vs NTFS with previous tests

@henso, any other ideas on what to test? We have many variances between our setups so I am trying to find the magic bullet that is giving me issues with MP 2.2 imports but not you.

 

[JSchuricht]

Write access to the SMB share is out. It started off looking good with folder 137, 15, 36 and 29 fully importing then folder with 491 stopped 165 movies short. The folder with 491 movies was too large for the local storage test earlier so now that I have gotten past all the movies that were tested local but are on the SMB share for this test I can invalidate my previous local storage test.

Gotta love testing with inconsistent results. Next up testing a local user account on the SMB 3 share vs domain.

 

[JSchuricht]

Local user account made no difference. Folder with 132 movies 100% import, folder 15 missing one, folder with 36 missing 7. There may be a pattern... when movies don't import, the same titles keep showing up as missing. So potentially the issue is related to certain movie titles intermittently not importing. Making a leap, if this is the same issue as the one movie I tracked down in post 13 I will focus testing on the movies that keep making my import fail list. Will see if I can get imports working by replacing the movies that didn't import with copies of known good movies renamed after the failing ones. I will have to do the imports a bunch of times due to the intermittent nature of the failure. Hoping to find a commonality in a particular type of media that is killing the import. This could also explain the imports working for henso.

 

[JSchuricht]

Might have something showing up. The 36 movie or "Y" folder has failed to fully import about 1/2 of the time. Each time it has failed it is the same movies that fail to import. I think from looking at this folder and others that the first movie to fail is causing some or all movies that come alpahbetically after it to fail during import too. There are three things standing out so far with the first movie in a list to fail import. 1st they are persistently staying open as seen on the file server by the MP2 server. 2nd the details of each movie contain "FileExtension_invalid: m2ts mts ssif" when I look at MediaInfo 19.09 program or MPCHC MediaInfo tab. 3rd they are all BDAV with an AVC video stream.

I have tried closing the open file from the file server which stayed closed until the automatic share refresh kicked in then the files were back to being persistently open but the import status did not change. The screencap below shows an example of a folder failing to import as seen under browse media. The folders for each movie are seen but the movies inside are never found yet while this is going on, the file in the second screencap shows that the MP2 server has the movie open on the SMB share.

I have identified 3 files in addition to the one listed in post 13 that have the "FileExtension_invalid: m2ts mts ssif" flag. I am going to try and find a few more for testing and isolate them.

 

[henso]

@henso, any other ideas on what to test? We have many variances between our setups so I am trying to find the magic bullet that is giving me issues with MP 2.2 imports but not you.

I really have no idea what to look for, but hope we find the magic bullet.

1st they are persistently staying open as seen on the file server by the MP2 server. 2nd the details of each movie contain "FileExtension_invalid: m2ts mts ssif" when I look at MediaInfo 19.09 program or MPCHC MediaInfo tab. 3rd they are all BDAV with an AVC video stream.

It would be nice if I could test with such a file. It might be that our version of MediaInfo gets stuck on such a file.

I think from looking at this folder and others that the first movie to fail is causing some or all movies that come alpahbetically after it to fail during import too

I think something like this could happen if an error occurs which then causes the importer to become suspended. Suspended importers should be visible in the logs. Only way to resume a suspended importer is by restarting the server, so that might be what you are seeing. Suspended importers would also make the import progress appear stuck if I remember correctly.

Thanks for testing!

 

[JSchuricht]

@henso, you might be onto something with the MediaInfo version in MP2. The MediaInfo.dll included with MP 2.2.3 is version 0.7.95.0 dated 5/9/2017. I downloaded MediaInfo_GUI_0.7.95_Windows_i386_WithoutInstaller. It locks up pulling data from the movie I found in post 13 but not on the clip of that movie I sent you. Unfortunately it doesn't lockup with the other 3 suspect movies I have found but those are intermittently importing and testing in the GUI isn't 100% the same as in MP2 so its still a possibility.

The change log on MediaInfo shows fixes for a hang on some AVC files on version 0.7.96. 0.7.96 didn't lock up with the known bad movie when I tested the GUI version. It might break some stuff but I am going to try updating MP2 with the 0.7.96 dll.

 

[JSchuricht]

Things are looking good with the MediaInfo 0.7.96 dll in MP 2.2.3. Using the stalled out import from earlier, I terminated MP2-server.exe, replaced MediaInfo.dll with the 0.7.96 version, restarted MP2 service and re-imported with 100% success rate on 703 movies. Because of the previous inconsistencies I can't proclaim this a fix yet so I am doing another fresh install of MP2 with the MediaInfo 0.7.96 dll and a full import of 8,000+ movies. It may take a week to import 27 folders separately so I can check them but if it works then I think we have a fix.

 

[HTPCSourcer]

I captured this under MP2-821 Upgrade mediainfo.dll to 0.7.96.

 

[henso]

I captured this under MP2-821 Upgrade mediainfo.dll to 0.7.96.

Maybe we should upgrade to an even newer version like 19? I think MP1 just upgraded to version 18.

 

[HTPCSourcer]

V19 is the same as 0.7.96. Download the 19 file and you should get a dll with version info 0.7.96