[solved] Large import not working (1 Viewer)

fredco

Portal Pro
November 13, 2010
87
15
Netherlands Netherlands
Country flag
I'm starting a new thread about an old issue of which I seem to be the only one having trouble with. When I use ANY version of MP beyond 1.16 I am not able to import a large movie collection at once. So I'm still stuck with 1.16 on ALL my MP instances, which are quite a few... I have a collection of over 1.200 movies and when I let Moving Pictures import them at once using any version from 1.17 through 1.23 the Moving Pictures Configuration stops responding and needs to be closed. Only a few movies may be imported then.

What I don't understand is that nobody else is complaining about this. It is a very clear issue: use 1.16 and all is fine, use 1.17 and beyond and it's broken when importing a large library.

I am using the latest Moving Pictures and IMDb+ and Windows 7
 

RoChess

Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,342
    1,824
    Country flag
    @fredco, I've never had to do a fresh import, as I've always just migrated everything over (still on 1.22 though as I'm holding off until that one bug is fixed).

    Do your logs indicate at all where it is failing?

    In my experience it tends to often be your thread count is too high (so lower it in advanced settings), or MediaInfo freaks out (be sure to manually update DLL).

    What I also noticed is that often users will have corrupted media file headers and not realize it. To that extend when I had to rely on a partly successful parity recovery I ended up writing a batch script that relies on ffmpeg to play each file at maximum speed (most modern computers can handle 600fps+, especially if you activate hardware acceleration) to ensure that each media file is not corrupt. The script is available on forum somewhere, but I can locate it if you cannot find it.

    Logs should reveal this to you, but if you have thread count higher than 1, then logs in asynchronous state can be difficult to read.

    Of course importing at 1 thread will take a lot longer, but when I do an IMDb+ refresh (to get updated vote/cert/info) I always run it when I sleep, so I wake up to a fresh collection.
     

    fredco

    Portal Pro
    November 13, 2010
    87
    15
    Netherlands Netherlands
    Country flag
    @RoChess: Thanks for your extensive answer. You should try and do a fresh import yourselves, just to see what happens. You could do it overnight too.

    First and easiest thing I tried is lowering the thread count to 1 (was 5). I didn't help. All 1.200 movies get recognized in the first run (they all have a green dot), but then only three movies get a check mark and then Moving Pictures stops responding.

    I will try your other suggestions later, but I need to delve a little deeper to understand what you mean.

    Still strange though that the error started with MP version 1.17. Before that: no issue at all! Does that give you ANY clue?
     

    fredco

    Portal Pro
    November 13, 2010
    87
    15
    Netherlands Netherlands
    Country flag
    I just did a fresh install with full cleanup of Mediaportal 1.16. I installed Moving Pictures and IMDb+ too. Started the import of the complete library, it took two hours and voilà: all done and up to date without any errors. Just to be sure about my issue...
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,342
    1,824
    Country flag
    @fredco, if it fails at 1 thread, then you should have a perfect log explaining why it failed, either within movingpictures.log or within mediaportal.log

    Did you retain those?

    I'm in the process of moving, so it will be a little while before I can try HTPC tests myself. Reading log files is easy though, so if you can provide those they should give enough insight to figure out what the issue is.

    One thing that it can still be is corrupted mediafile headers that old MediaInfo.DLL accepts, and freaks out new version. You could try downgrading MediaInfo.DLL to the same one in use on 1.16 after you test on 1.22. Cannot use 1.23 because it made massive changes to the way MediaInfo works, and Moving-Pictures has not yet been updated for that.

    Is that the simple explanation, you tried to do import on 1.23?
     

    fredco

    Portal Pro
    November 13, 2010
    87
    15
    Netherlands Netherlands
    Country flag
    @RoChess: Well, I did the following: Completely new and fresh install of Mediaportal 1.22 and Moving Pictures and IMDb+. Started import of 1.200+ movies. First run went fine, all green dots. Second run gave only two check marks: for Aladdin and Alice in Wonderland. Then Moving Pictures stopped responding. Logs are included. I see some errors. but I'm not sure what they mean.

    About the Mediainfo.dll you mention: should this not be resolved by a full cleanup with fresh install?
    I have not used 1.23 as of yet...
     

    Attachments

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,342
    1,824
    Country flag
    The fact things hung is due to NullException (those are the ones that cause software to panic and crash)

    [2019-08-10 00:54:17,394] [Error ] [MPMain ] [ERROR] - OnMessage exception:confused:ystem.NullReferenceException: Object reference not set to an instance of an object.
    at MediaPortal.Plugins.MovingPictures.MainUI.MovingPicturesGUI.OnPageDestroy(Int32 new_windowId)
    at MediaPortal.GUI.Library.GUIWindow.OnMessage(GUIMessage message)​

    However based on MovPic logs, the order is that it indeed finds three movies, and on 2nd one it does:

    10-Aug-2019 03:57:20 Info [ MovieImporter]: Retrieving media information for: Alice in Wonderland (Cartoon)
    10-Aug-2019 03:57:22 Info [ MediaInfoWrapper]: Scanning bluray structure: \\nas\d\disney - classic\alice in wonderland (1951)\bdmv\STREAM\00000.m2ts​

    But on third one it doesn't get that far and only does:

    10-Aug-2019 03:57:26 Info [ MovieImporter]: Retrieving media information for: Atlantis: The Lost Empire​

    Which seems to indicate reading the MediaInfo for that movie causes failure (my corrupted header theory)

    However MovPic shuts the threads down almost 6 hours later at: 10-Aug-2019 09:33:01 Info [ MovieImporter]: Shutting Down Media Scanner Threads...

    Try excluding Atlantis for a test-run, see if it is corrupted, or run that media file integrity check batch file to scan your library (starting with Atlantis)
     

    fredco

    Portal Pro
    November 13, 2010
    87
    15
    Netherlands Netherlands
    Country flag
    I located your script and tried it on some movies of mine. It seems it only scans mkv-files. My complete library consists of Blu Ray Structures which ScanMedia seems to skip.
    It seems my library does not have Mediainfo headers. Only thing I have apart from the BD-structures are poster.jpg and fanart.jpg for use in MP.
    And it now halted on Atlantis, but sometimes it checks a lot more movies and halts after, say, 100 movies.
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,342
    1,824
    Country flag
    Yeah I saw from your logs that you have mainly BR rips, so that complicates things for my script (no idea if ffmpeg can verify integrity on those).

    Moving-Pictures is pretty stand-alone during the import, but the one thing it relies on for MediaPortal during import is the MediaInfo scanning to get resolution/etc.

    MediaPortal by default also likes to generate thumbnails (disable that myself always to avoid issues).

    Seeing as it works under MediaPortal 1.16, but then fails when you do the same step under 1.17+ there is clearly something different about the way MediaPortal handles the MediaInfoWrapper requests that Moving-Pictures makes during the import process.

    The main factor on that is the mediainfo.dll file within the MediaPortal folder, and if you install 1.17+ that is obviously a different version than the one v1.16 installs.

    I've worked with the MediaInfo developers when I had serious issues on a WiFi share with some AVI files, and they constantly fix things, but they also adjust the way things work that might break software that relies on that. Manually replacing the DLL in the MePo root folder would then solve my issues.

    MediaInfo - Download MediaInfo for Microsoft Windows

    Latest version is v19.07 now, and you'll need to use the 32-bit DLL download for MediaPortal, but it would be wise to first check what version is bundled with MePo 1.16 and what is included with 1.17+.

    Downloaded v1.16 ZIP real quick, and it comes with: Mediainfo.dll v0.7.69.0
    MediaPortal v1.17 comes with MediaInfo.dll v0.7.95.0

    So, you can try downgrading MediaPortal v1.17 to Mediainfo.dll v0.7.69, and if collection imports fine then clearly issue is centered around MediaInfo.

    You can also try MediaInfo v19.07 to see if that solves it again (might be a bug they fixed).

    Otherwise it is a new bug that MediaInfo developers need to solve, but perhaps that is what MediaPortal v1.23 is about, which requires a fix to Moving-Pictures.

    Fun stuff :confused:
     

    fredco

    Portal Pro
    November 13, 2010
    87
    15
    Netherlands Netherlands
    Country flag
    You can also try MediaInfo v19.07 to see if that solves it again (might be a bug they fixed).

    You've got it spot-on! I now have freshly installed MP 1.22 which comes with MediaInfo.dll v0.7.95.0 (just like 1.17 and probably all in between). I changed that to v19.7.0.0 (the newest) and voilà: Moving Pictures Configuration works again like it should! Thanks for your input as always!
    A few thoughts about this:

    1. This should happen to everybody! Am I the only one who wants to import a large library at once?
    2. Your IMDb+ issues a warning about an outdated MediaInfo.dll and offers to change it. That happens when I first change it to v19.7.0.0 and later install IMDb+. I skipped the offer, but it should not happen as my version is not outdated.
    3. The MP developers should know about this and add the new MediaInfo.dll to version 1.23. Can you inform them?

    Thanks again very much!
     

    Users Who Are Viewing This Thread (Users: 0, Guests: 1)

    OP Similar threads Forum Replies Date
    D Large xml database in My Movies Management. I remain unclear on use My Films as bridge to MP2.2 General 9
    toricred Using TMM, How to Stop Importer From Going to The Movie Database for Data General 1
    Lichloathe MP2 importing keyart instead of poster General 0
    P Recordings not showing up unless re-imported and not possible to delete General 0
    C Importer works super slow My TVSeries 5
    C Imported media doesn't show up immediately? General 1
    X Recording import hängt Allgemeines Support- und Diskussionsforum 0
    J [fixed] Import issue with alternate movie title matching the name of another movie Submit: Bug Reports 7
    B xmltv - cannot import tvguide.xml australia Installation, configuration support 0
    B [solved] tvguide.xml not importing australian channels Electronic Program Guide 10
    J [fixed] Importer confused on specific movies with the same title, different years and different NFO's Submit: Bug Reports 3
    azzuro [fixed] [WIP MP2.3] - Audio files not imported to MP MediaPortal 2 18
    thepahakurki TVDB import not working My TVSeries 3
    T I cannot get a basic media import-what is the best configuration? General 10
    L TV-Series kein Import Plugins & Erweiterungen 6
    J [fixed] [MP2-833] 2.2.2 server crash during import Submit: Bug Reports 50
    D Ripped movies not importing unless IMDB code is included in folder name General 4
    R [solved] Moving pictures importer wont add files from a removable drive. Is there a work around ? Moving Pictures 8
    MausWiesel LightsOut-Connector für MP2? Allgemeines Support- und Diskussionsforum 9
    M [solved] MP2 Not discovering all movies Newcomers Forum 2
    J [fixed] [MP2-821] Importer glitches in 2.2 Submit: Bug Reports 42
    Golf4 [solved] Bei Import und grabben der Medieninhalte - Client ausschalten? Allgemeines Support- und Diskussionsforum 1
    cbr954rr Possibillity to export and import all client settings? General 0
    Jean-Christophe MP 2.2 - Where can I find Movies NOT imported (and why??) ? General 13
    N [Finished] [MP2-791] Radio recordings do not get imported into Recordings listing MediaPortal 2 25
    Similar threads

























    Top Bottom