Potential Memory Leak when adding a large collection of movies

Discussion in 'Moving Pictures' started by jasonwc, November 4, 2011.

  1. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    Note: I posted this before on the main thread a while back but I never got a response so I'm reposting in its own thread as it's a pretty serious issue.

    I've had a recurring memory usage issue which I hope someone might be able to shed some light on. I'm not sure if it's caused by Mediaportal or if it's actually a memory leak in Windows. Basically, if I import my entire collection of movies (only about 400 movies) into Moving Pictures on ANY x64 version of Windows, the kernel paged memory usage skyrockets until I run out of memory. If I install an x86 version of Windows on the same system, the issue doesn't occur. The reason I think this may be a Windows issue is that the problem occurs where the movies are stored - not necessarily on the system where Mediaportal is being run. For example, I now have about 4 TB of movies stored on a WHS 2011 storage server, and when I ran a moving pictures import on my HTPC, kernel memory usage climbed on the storage server to over 7 GB from about 150-200 MB. If the movies are stored locally, the memory usage increases on the machine doing the import.

    I've now seen this issue on three different systems w/ Mediaportal 1.2 Beta or 1.2 final and various versions of Moving Pictures including 1.2.3:

    Desktop
    Core i7 860 2.8 Ghz Quad Core CPU
    Asus P7P55D-E PRO 1156 Motherboard
    12 GB DDR3 1600 RAM
    2 x ATI 5850 @ stock
    160 GB Intel X-25 M SSD
    2 TB drive temp storage
    Windows 7 x64 (Issues occurs)/Windows 7 x86 (no issue - same driver versions; all up-to-date)

    HTPC
    Core i3 550 @ 3.2 Ghz Dual Core CPU
    GIGABYTE GA-P55-UD3R 1156 Socket Motherboard
    4 GB DDR 1333 RAM
    Nvidia GT430
    Cruical C300 60 GB SSD
    2 TB of storage (used to be 12 TB but since moved to storage server)
    Windows 7 x64 (issue occurs); Windows 7 x86 (no issue)



    Storage Server:
    Norco RPC-4224 4U Rackmount case with 24 hotswap bays
    Asus P8B WS Workstation Motherboard (Dual Intel 82574L Server NICs)
    Supermicro AOC-SASLP-MV8 SAS Controller
    Core i3-2100 3.2 Ghz 1155 Socket CPU
    8 GB DDR3 1333 RAM
    No video card
    WD Black 1 TB for OS
    14 TB for Storage
    WHS 2011 (based on Win Server 2008 R2, only comes in x64 version - issue occurs as described above)

    I was hoping someone could provide some insight into this issue. If logs would be helpful, tell me what to provide. The memory usage for the Mediaportal process doesn't seem to be the issue. Rather, the problem is that the kernel paged memory skyrockets until there is no more free memory, at which time the system becomes slow and unresponsive, requiring a reboot
     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. fforde

    fforde Community Plugin Dev

    Joined:
    June 7, 2007
    Messages:
    2,666
    Likes Received:
    1,690
    Occupation:
    Software Engineer
    Location:
    Texas
    Ratings:
    +1,696 / 0
    Home Country:
    United States of America United States of America
    Well the only memory issue I am aware of should be fixed. It was not a memory leak it was just an overly aggressive cache. To speed things up the importer would cache retrieved web pages so if the same URL is called twice the second time the cached copy would be used. The problem was this cache was done completely in memory, so this would create problems for people with large collections or people using overly aggressive scripts like IMDB+. The latest version of Moving Pictures corrects this and caches this data on disk so the problem should be resolved with version 1.2.3.

    So how many movies are you importing? And with what scripts? And are you certain you are seeing this problem with Moving Pictures 1.2.3? If you can still reproduce the problem can you please post debug logs?
     
  4. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    The collection isn't that large - around 325 movies. I'm currently using IMDB+ scraper but I first reported the issue using the default IMDB scraper. I've had this issue with many versions of Moving Pictures including 1.2.3.

    I can do an import on my desktop tonight and send you the logs
     
  5. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,239
    Likes Received:
    1,336
    Ratings:
    +1,745 / 2
    There was a memory problem in MovPic before, but fforde did an awesome job at fixing this and I've done 3000+ collection imports on 2GB total memory systems without any problems since then.

    Check your movingpictures.log file otherwise (or the old.log one) and look for: (date) (time) Info [ MovingPicturesCore]: Moving Pictures (1.2.3:1335)
     
  6. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    fforde,

    I just installed Mediaportal 1.2.1 and Moving Pictures 1.2.3 on my Desktop running Windows 7 x64 SP1. I had no unusual memory usage on the desktop while adding movies, but as expected as soon as the import started the kernel paged memory usage spiked on the server (where the movies are stored) to nearly 6 GB from a few hundred MB, resulting in 93% memory utilization up from around 20%. I have attached the debug logs you requested.

    RoChess,

    I see a 1.2.3.1329 - not 1.2.3.1355, and I just downloaded a copy from the Moving Pictures forum.

    04-Nov-2011 18:56:54 Info [ MovingPicturesCore]: Moving Pictures (1.2.3:1329)
     

    Attached Files:

  7. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,239
    Likes Received:
    1,336
    Ratings:
    +1,745 / 2
    The issue in question that solves the problem was r1313 as per: r1313 - Google Project Hosting

    And this was merged into the MediaPortal v1.2.x branch via: r1317 - Google Project Hosting

    So that "should" mean the version you are running has the bug fixed.

    Is perhaps something else going wrong? MediaInfo for example is also processed during import and is known to have some serious issues with an older release that is installed by MediaPortal.

    So you can either disable MediaInfo retrieval in MovPic during import, or download the 32-bit Stand-Alone v0.7.50 DLL from the MediaInfo website and overwrite the older version you currently have.
     
  8. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    No dice. After renaming the old DLL and adding the new 0.7.50 DLL, I cleared my movie database, restarted Mediaportal config, and re-added my Movie directory. As soon as the scan began kernel paged memory shot up to 6 GB on the server as it did yesterday.
     

    Attached Files:

    • before.PNG
      before.PNG
      File size:
      24 KB
      Uploaded:
      November 5, 2011
      Views:
      91
    • after.PNG
      after.PNG
      File size:
      25.9 KB
      Uploaded:
      November 5, 2011
      Views:
      91
  9. fforde

    fforde Community Plugin Dev

    Joined:
    June 7, 2007
    Messages:
    2,666
    Likes Received:
    1,690
    Occupation:
    Software Engineer
    Location:
    Texas
    Ratings:
    +1,696 / 0
    Home Country:
    United States of America United States of America
    Based on the log files you posted an import is not even occuring. Moving Pictures is actually doing very little. There might be more information with debug logging turned on. Would you mind posting debug log files?

    Also, I think I misunderstood but it sounds like you are saying your movies are stored on a separate machine and you are seeing the excessive memory usage on the file server and not your HTPC? Is this correct?
     
  10. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    I am re-running an import with debug logging on and should have new logs shortly.

    You are correct about the memory usage. The kernel paged memory spikes wherever the movies are stored. Less than 2 months ago, I moved my movies from my HTPC to a dedicated file server. Before doing so, the memory spike occurred on the HTPC when doing a large import. Now it occurs on the file server. The spike is very obvious and it's the only application I know of that causes such behavior.

    Based on the release date of Moving Pictures 1.2.3, I believe I was using it on the HTPC before migrating my data to the file server. I use StreamedMP and it updates my plugins regularly.
     
  11. fforde

    fforde Community Plugin Dev

    Joined:
    June 7, 2007
    Messages:
    2,666
    Likes Received:
    1,690
    Occupation:
    Software Engineer
    Location:
    Texas
    Ratings:
    +1,696 / 0
    Home Country:
    United States of America United States of America
    I am not sure I will be able to help you much then. Moving Pictures can only allocate memory on the machine it is running on. It is impossible for it to allocate memory on your file server. Moving Pictures is running a mediainfo scan on each file on import though which is non trivial. Can you try turning off mediainfo retrieval? This is togglable under the gear icon on the importer dialog.

    Setting mediainfo processing aside, all Moving Pictures is doing to the file server is passively browsing the file system...
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!