Potential Memory Leak when adding a large collection of movies (1 Viewer)

RoChess

Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    I don't have any Antivirus software installed on the file server because it doesn't execute any 3rd party software other than Hamachi that provides VPN access. Is Windows Security Essentials downloaded by Windows Update automatically? I have recommended updates including security patches installed automatically.

    If no AV software is installed, then Microsoft Security Essentials will show up as optional WUG download. So it depends on your settings if the optional/recommended ones are also installed auto alongside the critical ones.

    But it wasn't just your server that relates to, also your client, because it could generate more activity. And the type of activity generated by MovingPictures is not sequential in nature, it is more akin to a DDoS attack due to asynchrone multi-thread import basis.

    Default import thread count is '5' if memory serves me, but perhaps you increased that as well via advanced settings. Each request normally goes lightning fast, but if you are importing say 1500 movies, then you are generating a lot of requests to the server, one to get filename, one to read header via MediaInfo, to check for NFO file, etc.

    There are many tools available however that you can use to get to the source of the problem. A lot of which are build into the Windows OS, but usually for these type of problems there are some better 3rd party tools to use. One of the popular tools I use myself for these problems is procexp.exe from SysInternals.

    You can use the graphical "System Information" charts in that one to monitor what is going on at both client side and server side when you run that import scan. Once you identified at what layer the problem occurs you can then zoom in closer to find the actual cause/source of the process itself.
     

    jasonwc

    Portal Pro
    July 4, 2009
    83
    4
    Home Country
    United States of America United States of America
    I just checked the server. Windows Security Essentials isn't installed. I have NOD32 Antivirus installed on my desktop, where the import was run. However, I disabled scans of network shares precisely because of the potential for these issues, so it shouldn't interfere. I can disable it temporarily as well.

    I installed process explorer, but I don't see a way to monitor the kernel paged memory. I see all my running processes but I thought the kernel paged and nonpaged memory was a separate pool.
     

    jasonwc

    Portal Pro
    July 4, 2009
    83
    4
    Home Country
    United States of America United States of America
    So, I think I was able to successfully replicate the issue locally on my Desktop. Here's what I did. I took one of my movie drives from the file server and placed it in an external eSATA enclosure. I then connected the device to my desktop, deleted the Moving Pictures database to start fresh, opened the Moving Pictures configuration and added the Movies folder from the external disk. For this test, I DISABLED my NOD32 Antivirus and Mediainfo scanning.

    Note that the drive only contains 43 movies (350 GB) which is a small fraction of my 4.5 TB collection. The effect was less extreme but clearly noticeable. Immediately after beginning the import, while the files were being scanned for matches, kernel paged memory usage increased from 789 MB to 1493 MB. This occurred in the matter of a few seconds, and the memory usage did not decline after the import was complete, nor did the memory get released when Mediaportal was closed.

    I have attached debug logs. In addition, I also have provided the hardware specs for the Desktop machine on which the test was run.

    OS: Windows 7 Professional x64 SP1
    Software: Mediaportal 1.2.1; Moving Pictures 1.2.3
    Settings: MediaInfo scanning disabled, all else default
    Movies stored on a hard drive in an external enclosure, connected via eSATA to the the eSATA port on the back panel of my motherboard (JMicron controller)

    Hardware specs:
    Core i7 860 2.8 Ghz Quad Core CPU
    Asus P7P55D-E PRO 1156 Motherboard
    12 GB DDR3 1600 RAM
    2 x ATI 5850 @ stock
    160 GB Intel X-25 M SSD
    2 TB drive temp storage
     

    tourettes

    Retired Team Member
  • Premium Supporter
  • January 7, 2005
    17,301
    4,800
    Does the same memory hogging happen if you copy movies from eSATA to internal disk? If not at that point then by importing the movies from the internal HD?

    In any case when you are able to cause some memory increase on server side (orver the network just by reading files) it is 100% sure that the issue is outside MediaPortal (or any other application that reads those files).
     

    jasonwc

    Portal Pro
    July 4, 2009
    83
    4
    Home Country
    United States of America United States of America
    Re-ran the test with another drive that has more movies (~73 movies, 957 GB). As expected, memory usage increased to a greater extent (from ~700 MB kernel paged memory to 2600 in the matter of a few seconds) . This time I also enabled the Mediainfo scan to replicate an actual scan. Antivirus was still disabled.

    Logs attached.

    Does the same memory hogging happen if you copy movies from eSATA to internal disk? If not at that point then by importing the movies from the internal HD?

    In any case when you are able to cause some memory increase on server side (orver the network just by reading files) it is 100% sure that the issue is outside MediaPortal (or any other application that reads those files).

    Copying from an external eSATA disk to an internal disk doesn't cause memory issues. Kernel paged memory usage doesn't increase. During the copy, there is increased memory usage, presumably because Windows is copying data into memory to allow for faster transfers. However, that memory is returned immediately after the copy finishes. Large file copies over the network to/from the file server also don't cause issues.

    However, I am copying 200 GB of movies from the disk I have connected via eSATA to my internal 2 TB disk, and I'll run the test again off the internal disk.

    It's possible that the issue on the file server is different. The machines have completely different hardware (different motherboard/cpu, different chipset (file server has a workstation motherboard with the C206 chipset), different storage controllers. They're running different operating systems (WHS 2011 vs. Windows 7).
     

    jasonwc

    Portal Pro
    July 4, 2009
    83
    4
    Home Country
    United States of America United States of America

    1) I copied 200 GB of movies (28 films) to the internal drive. Interestingly, the kernel paged memory DID increase by about 500 MB over the process of the transfer. Perhaps this is just a Windows issue.

    2) I ran a test importing the movies from my internal drive with Mediainfo enabled (0.75 DLL) and NOD32 disabled. Kernel paged memory usage was 164 MB before the test and 595 MB after.

    2) I then installed the hotfix you linked, restarted, and tested again with the settings as described above. Kernel paged memory usage began at 198 MB before the test and was 646 MB after.
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    Do you have any explorer extensions installed for the media types in questions?

    Windows by default will generate thumbnails for media, which on super large and badly indexed container files can lead to excessive memory usage. Some Matroska explorer extensions are known to cause memory leaks like that, and Windows method itself is not free from harm either when it comes to system impact.

    Uninstall any explorer extension otherwise, and disable Windows thumb generation. On Win7 you can't actually disable it easy, you have to deny ACL permissions on the thumbnail folder to do it right.
     

    jasonwc

    Portal Pro
    July 4, 2009
    83
    4
    Home Country
    United States of America United States of America
    I had the DivX explorer filter installed a while back to get thumbnails working. However, it was horribly unstable and would cause Explorer to crash constantly. So, I uninstalled it.

    My desktop has a fresh install of Windows 7 Professional x64 which I installed a few weeks ago. I haven't installed any Explorer filters for the reasons you describe.
     

    MonkeyJug

    Portal Member
    October 18, 2012
    34
    7
    Home Country
    United Kingdom United Kingdom
    Google search led me here...

    I have the EXACT same problem that you describe. I have 800 movies and i was pleasantly suprised to see that you have flexraid installed. That sure was interesting!

    I have spent the last 3 days trying to solve why my memory (16gb) gets used up to the maximum and not released. I have managed to track it down to being somehow weirdly linked to creating a shared folder on my flexraid storage pool. For whatever reason, this causes the memory issue and eventually it crashes mepo after a few seconds launch of stuttering video...

    I think i have narrowed it down to a hardware fault, either the motherboard or the ssd. Tomorrow i am trying with a different ssd, but same brand and model. Gonna back up, then restore to the similar drive. If that fails, i'm replacing the motherboard.

    I just want to know how you solved this error, if indeed you ever did?
     

    Users who are viewing this thread

    Top Bottom