Suggested Patch to Filter Out Sample Files Faster

Discussion in 'Moving Pictures' started by Bernard, July 9, 2013.

  1. Bernard

    Bernard Portal Pro

    Joined:
    March 5, 2012
    Messages:
    103
    Likes Received:
    20
    Gender:
    Male
    Occupation:
    Software Engineer
    Location:
    Centurion
    Ratings:
    +25 / 0
    Home Country:
    South Africa South Africa
    Show System Specs
    Hi Moving Pictures Devs,

    I have made a small change to Moving Pictures source and would like to post it to the Developers Group on Google, but I can't figure out how to join since it complains that I don't have the correct Permissions to Join.

    Problem: I have a bunch (about 80) sample videos between all my movies on a network drive. Every time the import of Moving Pictures run, it collects this list of samples and tries to process them like "new" video files, only to spit them out just before they are queued for online lookup. Part of this process is to build a Media object in memory and load it's current setting from the db.

    Because my client PC/Laptop is working over a slow wifi connection, with a slow CPU. It takes a while to work through these files every time. And all the progress indicators "act" as if there are about 80 "new" video files to process.

    Moral of the story: A lot of high level processing is involved on the samples to determine if the video files is new and need to be looked up online that can be skipped.

    Suggested Solution: My suggested solution is that the Importer should disregards sample files the same way it disregard any other file type that is not a video file, like .nfo or .txt. In essence a sample video is like an invalid file format: It will never be in the DB, so no point in "just checking" every time.

    In the source I have "copied" the code that spit out the samples higher up in the call stack so that it ignores those files in the same process as it ignores "unsupported" files.

    Implementation:
    MovingPictures\Database\DBImportPath.cs, line 237, add:
    Code (Text):
    1. // exclude samplefiles and ignore them from any processing
    2. if (VideoUtility.isSampleFile(videoFile))
    3. {
    4.     logger.Info("IGNORED: File='{0}', Bytes={1}, Reason='Sample detected'",videoFile.Name, videoFile.Length);
    5.     continue;
    6. }
    That is it. This code comes from MovingPictures\LocalMediaManagement\MovieImporter.cs, line 1090. I would have suggested that the code be removed from ScanFiles(...) but then the importer will not spit out sample files that was in fact imported previously and is not "resent to importer" (this will happen if the setting was off during an initial import run).



    I have attached a "patch" file as well.
     

    Attached Files:

    Last edited: July 12, 2013
    • Like Like x 1
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. spacemanc
    • Premium Supporter

    spacemanc MP Donator

    Joined:
    October 20, 2011
    Messages:
    89
    Likes Received:
    7
    Ratings:
    +8 / 0
    I also waste quite a bit of time with sample files. The other similar problem I have is that a lot of my movie folders have a separate "2 channel audio" file, which also get picked up on a scan. I suppose one way around it is to just manually delete all the unneeded files via windows, but I've never got around to it and I don't scan often enough to make it a big problem!
     
  4. Bernard

    Bernard Portal Pro

    Joined:
    March 5, 2012
    Messages:
    103
    Likes Received:
    20
    Gender:
    Male
    Occupation:
    Software Engineer
    Location:
    Centurion
    Ratings:
    +25 / 0
    Home Country:
    South Africa South Africa
    Show System Specs

    If you would like to try this "alternative code" you can copy this dll to your C:\Program Files\Team MediaPortal\MediaPortal\plugins\Windows folder.

    For my 'slow' laptop this actually help the GUI get faster through the initial import run when I'm waiting for new movie files to show up in the GUI importer, and also when there is no new movies and it is just running in the background.

    Full Disclaimer: Please just backup (rename) your existing MovingPictures.dll file, before using this one, if this somehow break your MePo you can just delete it and rename the old file back. I did not extensively test the code, I don't have any "
     

    Attached Files:

    Last edited: July 16, 2013
  5. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,185
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    You might also be able to modify the 'noise filter' and ignore the samples at a higher level, the same way MP-TVSeries does.
     
    • Like Like x 1
  6. Bernard

    Bernard Portal Pro

    Joined:
    March 5, 2012
    Messages:
    103
    Likes Received:
    20
    Gender:
    Male
    Occupation:
    Software Engineer
    Location:
    Centurion
    Ratings:
    +25 / 0
    Home Country:
    South Africa South Africa
    Show System Specs
    I will see if I can find this then. I did not notice any part in MovingPictures that pointed to a "noise filter" it only does 'n simple:

    Get Files in Watched Folders > Loop and Keep Only Video Extentions

    then at a very later stage (part of the movie online details processing code) it removes entries in this queue if they match the samples RegEx. Important to note that "Send to Importer" starts in this same function, so the code should be here if you want the importer to remove samples even if they have been imported already as movies.
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!