[solved] - Large import not working

Discussion in 'Moving Pictures' started by fredco, August 6, 2019.

  1. fredco

    fredco Portal Pro

    Joined:
    November 13, 2010
    Messages:
    81
    Likes Received:
    5
    Gender:
    Male
    Ratings:
    +16 / 0
    Home Country:
    Netherlands Netherlands
    Show System Specs
    I'm starting a new thread about an old issue of which I seem to be the only one having trouble with. When I use ANY version of MP beyond 1.16 I am not able to import a large movie collection at once. So I'm still stuck with 1.16 on ALL my MP instances, which are quite a few... I have a collection of over 1.200 movies and when I let Moving Pictures import them at once using any version from 1.17 through 1.23 the Moving Pictures Configuration stops responding and needs to be closed. Only a few movies may be imported then.

    What I don't understand is that nobody else is complaining about this. It is a very clear issue: use 1.16 and all is fine, use 1.17 and beyond and it's broken when importing a large library.



    I am using the latest Moving Pictures and IMDb+ and Windows 7
     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,172
    Likes Received:
    1,301
    Ratings:
    +1,675 / 2
    @fredco, I've never had to do a fresh import, as I've always just migrated everything over (still on 1.22 though as I'm holding off until that one bug is fixed).

    Do your logs indicate at all where it is failing?

    In my experience it tends to often be your thread count is too high (so lower it in advanced settings), or MediaInfo freaks out (be sure to manually update DLL).

    What I also noticed is that often users will have corrupted media file headers and not realize it. To that extend when I had to rely on a partly successful parity recovery I ended up writing a batch script that relies on ffmpeg to play each file at maximum speed (most modern computers can handle 600fps+, especially if you activate hardware acceleration) to ensure that each media file is not corrupt. The script is available on forum somewhere, but I can locate it if you cannot find it.

    Logs should reveal this to you, but if you have thread count higher than 1, then logs in asynchronous state can be difficult to read.

    Of course importing at 1 thread will take a lot longer, but when I do an IMDb+ refresh (to get updated vote/cert/info) I always run it when I sleep, so I wake up to a fresh collection.
     
  4. fredco

    fredco Portal Pro

    Joined:
    November 13, 2010
    Messages:
    81
    Likes Received:
    5
    Gender:
    Male
    Ratings:
    +16 / 0
    Home Country:
    Netherlands Netherlands
    Show System Specs
    @RoChess: Thanks for your extensive answer. You should try and do a fresh import yourselves, just to see what happens. You could do it overnight too.

    First and easiest thing I tried is lowering the thread count to 1 (was 5). I didn't help. All 1.200 movies get recognized in the first run (they all have a green dot), but then only three movies get a check mark and then Moving Pictures stops responding.

    I will try your other suggestions later, but I need to delve a little deeper to understand what you mean.

    Still strange though that the error started with MP version 1.17. Before that: no issue at all! Does that give you ANY clue?
     
  5. fredco

    fredco Portal Pro

    Joined:
    November 13, 2010
    Messages:
    81
    Likes Received:
    5
    Gender:
    Male
    Ratings:
    +16 / 0
    Home Country:
    Netherlands Netherlands
    Show System Specs
    I just did a fresh install with full cleanup of Mediaportal 1.16. I installed Moving Pictures and IMDb+ too. Started the import of the complete library, it took two hours and voilà: all done and up to date without any errors. Just to be sure about my issue...
     
  6. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,172
    Likes Received:
    1,301
    Ratings:
    +1,675 / 2
    @fredco, if it fails at 1 thread, then you should have a perfect log explaining why it failed, either within movingpictures.log or within mediaportal.log

    Did you retain those?

    I'm in the process of moving, so it will be a little while before I can try HTPC tests myself. Reading log files is easy though, so if you can provide those they should give enough insight to figure out what the issue is.

    One thing that it can still be is corrupted mediafile headers that old MediaInfo.DLL accepts, and freaks out new version. You could try downgrading MediaInfo.DLL to the same one in use on 1.16 after you test on 1.22. Cannot use 1.23 because it made massive changes to the way MediaInfo works, and Moving-Pictures has not yet been updated for that.

    Is that the simple explanation, you tried to do import on 1.23?
     
  7. fredco

    fredco Portal Pro

    Joined:
    November 13, 2010
    Messages:
    81
    Likes Received:
    5
    Gender:
    Male
    Ratings:
    +16 / 0
    Home Country:
    Netherlands Netherlands
    Show System Specs
    @RoChess: Well, I did the following: Completely new and fresh install of Mediaportal 1.22 and Moving Pictures and IMDb+. Started import of 1.200+ movies. First run went fine, all green dots. Second run gave only two check marks: for Aladdin and Alice in Wonderland. Then Moving Pictures stopped responding. Logs are included. I see some errors. but I'm not sure what they mean.

    About the Mediainfo.dll you mention: should this not be resolved by a full cleanup with fresh install?
    I have not used 1.23 as of yet...
     

    Attached Files:

  8. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,172
    Likes Received:
    1,301
    Ratings:
    +1,675 / 2
    The fact things hung is due to NullException (those are the ones that cause software to panic and crash)

    [2019-08-10 00:54:17,394] [Error ] [MPMain ] [ERROR] - OnMessage exception:confused:ystem.NullReferenceException: Object reference not set to an instance of an object.
    at MediaPortal.Plugins.MovingPictures.MainUI.MovingPicturesGUI.OnPageDestroy(Int32 new_windowId)
    at MediaPortal.GUI.Library.GUIWindow.OnMessage(GUIMessage message)​

    However based on MovPic logs, the order is that it indeed finds three movies, and on 2nd one it does:

    10-Aug-2019 03:57:20 Info [ MovieImporter]: Retrieving media information for: Alice in Wonderland (Cartoon)
    10-Aug-2019 03:57:22 Info [ MediaInfoWrapper]: Scanning bluray structure: \\nas\d\disney - classic\alice in wonderland (1951)\bdmv\STREAM\00000.m2ts​

    But on third one it doesn't get that far and only does:

    10-Aug-2019 03:57:26 Info [ MovieImporter]: Retrieving media information for: Atlantis: The Lost Empire​

    Which seems to indicate reading the MediaInfo for that movie causes failure (my corrupted header theory)

    However MovPic shuts the threads down almost 6 hours later at: 10-Aug-2019 09:33:01 Info [ MovieImporter]: Shutting Down Media Scanner Threads...

    Try excluding Atlantis for a test-run, see if it is corrupted, or run that media file integrity check batch file to scan your library (starting with Atlantis)
     
  9. fredco

    fredco Portal Pro

    Joined:
    November 13, 2010
    Messages:
    81
    Likes Received:
    5
    Gender:
    Male
    Ratings:
    +16 / 0
    Home Country:
    Netherlands Netherlands
    Show System Specs
    I located your script and tried it on some movies of mine. It seems it only scans mkv-files. My complete library consists of Blu Ray Structures which ScanMedia seems to skip.
    It seems my library does not have Mediainfo headers. Only thing I have apart from the BD-structures are poster.jpg and fanart.jpg for use in MP.
    And it now halted on Atlantis, but sometimes it checks a lot more movies and halts after, say, 100 movies.
     
  10. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,172
    Likes Received:
    1,301
    Ratings:
    +1,675 / 2
    Yeah I saw from your logs that you have mainly BR rips, so that complicates things for my script (no idea if ffmpeg can verify integrity on those).

    Moving-Pictures is pretty stand-alone during the import, but the one thing it relies on for MediaPortal during import is the MediaInfo scanning to get resolution/etc.

    MediaPortal by default also likes to generate thumbnails (disable that myself always to avoid issues).

    Seeing as it works under MediaPortal 1.16, but then fails when you do the same step under 1.17+ there is clearly something different about the way MediaPortal handles the MediaInfoWrapper requests that Moving-Pictures makes during the import process.

    The main factor on that is the mediainfo.dll file within the MediaPortal folder, and if you install 1.17+ that is obviously a different version than the one v1.16 installs.

    I've worked with the MediaInfo developers when I had serious issues on a WiFi share with some AVI files, and they constantly fix things, but they also adjust the way things work that might break software that relies on that. Manually replacing the DLL in the MePo root folder would then solve my issues.

    MediaInfo - Download MediaInfo for Microsoft Windows

    Latest version is v19.07 now, and you'll need to use the 32-bit DLL download for MediaPortal, but it would be wise to first check what version is bundled with MePo 1.16 and what is included with 1.17+.

    Downloaded v1.16 ZIP real quick, and it comes with: Mediainfo.dll v0.7.69.0
    MediaPortal v1.17 comes with MediaInfo.dll v0.7.95.0

    So, you can try downgrading MediaPortal v1.17 to Mediainfo.dll v0.7.69, and if collection imports fine then clearly issue is centered around MediaInfo.

    You can also try MediaInfo v19.07 to see if that solves it again (might be a bug they fixed).

    Otherwise it is a new bug that MediaInfo developers need to solve, but perhaps that is what MediaPortal v1.23 is about, which requires a fix to Moving-Pictures.

    Fun stuff :confused:
     
  11. fredco

    fredco Portal Pro

    Joined:
    November 13, 2010
    Messages:
    81
    Likes Received:
    5
    Gender:
    Male
    Ratings:
    +16 / 0
    Home Country:
    Netherlands Netherlands
    Show System Specs

    You've got it spot-on! I now have freshly installed MP 1.22 which comes with MediaInfo.dll v0.7.95.0 (just like 1.17 and probably all in between). I changed that to v19.7.0.0 (the newest) and voilà: Moving Pictures Configuration works again like it should! Thanks for your input as always!
    A few thoughts about this:

    1. This should happen to everybody! Am I the only one who wants to import a large library at once?
    2. Your IMDb+ issues a warning about an outdated MediaInfo.dll and offers to change it. That happens when I first change it to v19.7.0.0 and later install IMDb+. I skipped the offer, but it should not happen as my version is not outdated.
    3. The MP developers should know about this and add the new MediaInfo.dll to version 1.23. Can you inform them?

    Thanks again very much!
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!