Is there any way to speed this import up?

Discussion in 'My TVSeries' started by Lyfesaver74, March 12, 2016.

  1. Lyfesaver74
    • Team MediaPortal

    Lyfesaver74 Public Relations

    Joined:
    September 25, 2010
    Messages:
    1,544
    Likes Received:
    890
    Location:
    Texas
    Ratings:
    +1,093 / 3
    Home Country:
    United States of America United States of America
    Show System Specs
    My import time is excessively long. As @ltfearme knows, my collection is massive, so I expect it to be slow compared to others. It is so long though that I am suspecting something is wrong.

    I have attached my log and in it you can see the biggest bulk of time is after it parses the files and before it adds anything new. as seen between the time stamps:

    2016-03-12 13:32:01.775 [INFO][07]: Finished Local Filename Parsing
    2016-03-12 13:34:11.655 [INFO][07]: Adding 0 new file(s) to Database

    That's 2 mins and 10 seconds between those two.

    Is there anything I can do (other than mark ended series as "Ignore on scan" as I have done that) to make this quicker? Any advice would be greatly appreciated.


     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. ltfearme
    • Premium Supporter

    ltfearme Community Plugin Dev

    Joined:
    June 10, 2007
    Messages:
    6,457
    Likes Received:
    4,241
    Gender:
    Male
    Occupation:
    Software Test Engineer
    Location:
    Sydney
    Ratings:
    +5,385 / 0
    Home Country:
    Australia Australia
    Maybe but probably not easily. Thats just how long it took your system to work out all the files on your system import paths and find what might be unknown (0 files in this case).

    The only thing I can suggest is ensure that the import folders are isolated to only have files used by tvseries. This is probably something you already do but just in case there might be some optimisations. What we could do is add an option to completely ignore an import folder so you can potentially put all the ended shows there.

    I would say hardware wise that a CPU / HDD will be the limitations, so faster the better, im sure @RoChess will have some suggestions on that :)
     
    • Thank You! Thank You! x 1
  4. Lyfesaver74
    • Team MediaPortal

    Lyfesaver74 Public Relations

    Joined:
    September 25, 2010
    Messages:
    1,544
    Likes Received:
    890
    Location:
    Texas
    Ratings:
    +1,093 / 3
    Home Country:
    United States of America United States of America
    Show System Specs
    Hey, D :) Yeah I do already separate my media by drives, not only by folders. As for your suggestion that would be a really cool feature!!! This is something I also thought would be good for MovPic and mentioned it to someone who will go nameless and was told it was a stupid idea. To have folders would hold media for library, but are full so no longer needed to search for importing.

    Is this something you think you could add to MPTVS in the future?
     
  5. ltfearme
    • Premium Supporter

    ltfearme Community Plugin Dev

    Joined:
    June 10, 2007
    Messages:
    6,457
    Likes Received:
    4,241
    Gender:
    Male
    Occupation:
    Software Test Engineer
    Location:
    Sydney
    Ratings:
    +5,385 / 0
    Home Country:
    Australia Australia
    Yes, it's something I could add in a future version.
     
    • Thank You! Thank You! x 1
  6. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    I already compared my import speed and even through my collection is only 5% smaller, it does the same scan to find 0 new files about 2-3x faster. Would trace logging reveal what is happening in that section of code, or do you have to add extra logging @ltfearme?
     
  7. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    Ohh the only thing I just realized is that I use a master subfolder usually along the lines of "TV Shows" even if the entire drive is used. If @Lyfesaver74 is sharing the X:\ root folder then those hidden Windows system folders might be causing a delay to ignore/skip the content for recycling, system volume, etc.
     
  8. Lyfesaver74
    • Team MediaPortal

    Lyfesaver74 Public Relations

    Joined:
    September 25, 2010
    Messages:
    1,544
    Likes Received:
    890
    Location:
    Texas
    Ratings:
    +1,093 / 3
    Home Country:
    United States of America United States of America
    Show System Specs
    @ltfearme is there any way to see what is going on during that 2 min 10 seconds?
     
  9. ltfearme
    • Premium Supporter

    ltfearme Community Plugin Dev

    Joined:
    June 10, 2007
    Messages:
    6,457
    Likes Received:
    4,241
    Gender:
    Male
    Occupation:
    Software Test Engineer
    Location:
    Sydney
    Ratings:
    +5,385 / 0
    Home Country:
    Australia Australia
    Its doubtful extra logging can be added but will double check when I have time. There could be many reasons why Rochess has a faster scan e.g. Local vs Network Drive, Better CPU/ HDD etc.

    Rochess does have a point regarding the root drive, it would be recommended to have a master sub-folder to filter out any hidden folders.
     
  10. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    @ltfearme, here are the comparison log results between @Lyfesaver74 database of 64MB and the one on my box of 35MB:

    Mine:

    Code (Text):
    1. 2016-03-14 17:19:47.278 [INFO][03]: ***************  Gathering Local Information  ***************
    2. 2016-03-14 17:19:47.282 [INFO][03]: Starting Local Filename Parsing, processing 19324 files
    3. 2016-03-14 17:19:56.659 [INFO][03]: Finished Local Filename Parsing
    4. 2016-03-14 17:20:10.508 [INFO][03]: Adding 0 new file(s) to Database
    5. 2016-03-14 17:20:19.698 [DEBG][03]: Begin Parsing action: MediaInfo
    His (only 9.7% more files):

    Code (Text):
    1. 2016-03-14 16:36:27.234 [INFO][03]: ***************  Gathering Local Information  ***************
    2. 2016-03-14 16:36:27.234 [INFO][03]: Starting Local Filename Parsing, processing 21198 files
    3. 2016-03-14 16:36:38.714 [INFO][03]: Finished Local Filename Parsing
    4. 2016-03-14 16:38:54.994 [INFO][03]: Adding 0 new file(s) to Database
    5. 2016-03-14 16:39:01.194 [DEBG][03]: Begin Parsing action: MediaInfo
    The massive delay is not in the parsing (9s versus 11s), but in the determining if new files were added since last time. Taking roughly 14 seconds for me, but 2m16s on his box.

    From what I understand from the code @ https://github.com/damienhaynes/mpt...ne Parsing Classes/OnlineParse.cs#L3043-L3059 that makes no sense to me :)

    Adding some extra log lines might clarify if there is one specific item causing the delay, such as for example the RemoveFilesInDB() function, your guess is as good as mine.

    PS: I had him do a vacuum and reindex on the database after cleaning up some shows no longer on disk, and it shrunk database to 56MB (did the same on mine and it shrunk to 31MB), but no idea why it is almost 100% more in size in comparison for only 10% additional episodes.

    PPS: System spec wise we are pretty identical, mine is a crappy AMD A4-5300 dual-core APU with 8GB of memory, and an old 40GB Mushkin SSD, and pretty sure his was roughly the same (@Lyfesaver74 correct me if I am wrong).
     
  11. Lyfesaver74
    • Team MediaPortal

    Lyfesaver74 Public Relations

    Joined:
    September 25, 2010
    Messages:
    1,544
    Likes Received:
    890
    Location:
    Texas
    Ratings:
    +1,093 / 3
    Home Country:
    United States of America United States of America
    Show System Specs
    A6-5400K with 4GB RAM and a Samsung 840 Pro SSD
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!