Potential Memory Leak when adding a large collection of movies | Page 3

Discussion in 'Moving Pictures' started by jasonwc, November 4, 2011.

  1. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,239
    Likes Received:
    1,336
    Ratings:
    +1,745 / 2
    If no AV software is installed, then Microsoft Security Essentials will show up as optional WUG download. So it depends on your settings if the optional/recommended ones are also installed auto alongside the critical ones.

    But it wasn't just your server that relates to, also your client, because it could generate more activity. And the type of activity generated by MovingPictures is not sequential in nature, it is more akin to a DDoS attack due to asynchrone multi-thread import basis.

    Default import thread count is '5' if memory serves me, but perhaps you increased that as well via advanced settings. Each request normally goes lightning fast, but if you are importing say 1500 movies, then you are generating a lot of requests to the server, one to get filename, one to read header via MediaInfo, to check for NFO file, etc.

    There are many tools available however that you can use to get to the source of the problem. A lot of which are build into the Windows OS, but usually for these type of problems there are some better 3rd party tools to use. One of the popular tools I use myself for these problems is procexp.exe from SysInternals.



    You can use the graphical "System Information" charts in that one to monitor what is going on at both client side and server side when you run that import scan. Once you identified at what layer the problem occurs you can then zoom in closer to find the actual cause/source of the process itself.
     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    I just checked the server. Windows Security Essentials isn't installed. I have NOD32 Antivirus installed on my desktop, where the import was run. However, I disabled scans of network shares precisely because of the potential for these issues, so it shouldn't interfere. I can disable it temporarily as well.

    I installed process explorer, but I don't see a way to monitor the kernel paged memory. I see all my running processes but I thought the kernel paged and nonpaged memory was a separate pool.
     
  4. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    So, I think I was able to successfully replicate the issue locally on my Desktop. Here's what I did. I took one of my movie drives from the file server and placed it in an external eSATA enclosure. I then connected the device to my desktop, deleted the Moving Pictures database to start fresh, opened the Moving Pictures configuration and added the Movies folder from the external disk. For this test, I DISABLED my NOD32 Antivirus and Mediainfo scanning.

    Note that the drive only contains 43 movies (350 GB) which is a small fraction of my 4.5 TB collection. The effect was less extreme but clearly noticeable. Immediately after beginning the import, while the files were being scanned for matches, kernel paged memory usage increased from 789 MB to 1493 MB. This occurred in the matter of a few seconds, and the memory usage did not decline after the import was complete, nor did the memory get released when Mediaportal was closed.

    I have attached debug logs. In addition, I also have provided the hardware specs for the Desktop machine on which the test was run.

    OS: Windows 7 Professional x64 SP1
    Software: Mediaportal 1.2.1; Moving Pictures 1.2.3
    Settings: MediaInfo scanning disabled, all else default
    Movies stored on a hard drive in an external enclosure, connected via eSATA to the the eSATA port on the back panel of my motherboard (JMicron controller)

    Hardware specs:
    Core i7 860 2.8 Ghz Quad Core CPU
    Asus P7P55D-E PRO 1156 Motherboard
    12 GB DDR3 1600 RAM
    2 x ATI 5850 @ stock
    160 GB Intel X-25 M SSD
    2 TB drive temp storage
     
  5. tourettes
    • Premium Supporter

    tourettes Retired Team Member

    Joined:
    January 7, 2005
    Messages:
    17,301
    Likes Received:
    4,595
    Ratings:
    +4,810 / 3
    Does the same memory hogging happen if you copy movies from eSATA to internal disk? If not at that point then by importing the movies from the internal HD?

    In any case when you are able to cause some memory increase on server side (orver the network just by reading files) it is 100% sure that the issue is outside MediaPortal (or any other application that reads those files).
     
  6. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    Re-ran the test with another drive that has more movies (~73 movies, 957 GB). As expected, memory usage increased to a greater extent (from ~700 MB kernel paged memory to 2600 in the matter of a few seconds) . This time I also enabled the Mediainfo scan to replicate an actual scan. Antivirus was still disabled.

    Logs attached.

    Copying from an external eSATA disk to an internal disk doesn't cause memory issues. Kernel paged memory usage doesn't increase. During the copy, there is increased memory usage, presumably because Windows is copying data into memory to allow for faster transfers. However, that memory is returned immediately after the copy finishes. Large file copies over the network to/from the file server also don't cause issues.

    However, I am copying 200 GB of movies from the disk I have connected via eSATA to my internal 2 TB disk, and I'll run the test again off the internal disk.

    It's possible that the issue on the file server is different. The machines have completely different hardware (different motherboard/cpu, different chipset (file server has a workstation motherboard with the C206 chipset), different storage controllers. They're running different operating systems (WHS 2011 vs. Windows 7).
     
  7. tourettes
    • Premium Supporter

    tourettes Retired Team Member

    Joined:
    January 7, 2005
    Messages:
    17,301
    Likes Received:
    4,595
    Ratings:
    +4,810 / 3
  8. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    1) I copied 200 GB of movies (28 films) to the internal drive. Interestingly, the kernel paged memory DID increase by about 500 MB over the process of the transfer. Perhaps this is just a Windows issue.

    2) I ran a test importing the movies from my internal drive with Mediainfo enabled (0.75 DLL) and NOD32 disabled. Kernel paged memory usage was 164 MB before the test and 595 MB after.

    2) I then installed the hotfix you linked, restarted, and tested again with the settings as described above. Kernel paged memory usage began at 198 MB before the test and was 646 MB after.
     
  9. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,239
    Likes Received:
    1,336
    Ratings:
    +1,745 / 2
    Do you have any explorer extensions installed for the media types in questions?

    Windows by default will generate thumbnails for media, which on super large and badly indexed container files can lead to excessive memory usage. Some Matroska explorer extensions are known to cause memory leaks like that, and Windows method itself is not free from harm either when it comes to system impact.

    Uninstall any explorer extension otherwise, and disable Windows thumb generation. On Win7 you can't actually disable it easy, you have to deny ACL permissions on the thumbnail folder to do it right.
     
  10. jasonwc

    jasonwc Portal Pro

    Joined:
    July 4, 2009
    Messages:
    83
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United States of America United States of America
    I had the DivX explorer filter installed a while back to get thumbnails working. However, it was horribly unstable and would cause Explorer to crash constantly. So, I uninstalled it.

    My desktop has a fresh install of Windows 7 Professional x64 which I installed a few weeks ago. I haven't installed any Explorer filters for the reasons you describe.
     
  11. MonkeyJug

    MonkeyJug Portal Member

    Joined:
    October 18, 2012
    Messages:
    34
    Likes Received:
    3
    Gender:
    Male
    Ratings:
    +6 / 0
    Home Country:
    United Kingdom United Kingdom
    Google search led me here...

    I have the EXACT same problem that you describe. I have 800 movies and i was pleasantly suprised to see that you have flexraid installed. That sure was interesting!

    I have spent the last 3 days trying to solve why my memory (16gb) gets used up to the maximum and not released. I have managed to track it down to being somehow weirdly linked to creating a shared folder on my flexraid storage pool. For whatever reason, this causes the memory issue and eventually it crashes mepo after a few seconds launch of stuttering video...

    I think i have narrowed it down to a hardware fault, either the motherboard or the ssd. Tomorrow i am trying with a different ssd, but same brand and model. Gonna back up, then restore to the similar drive. If that fails, i'm replacing the motherboard.

    I just want to know how you solved this error, if indeed you ever did?
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!