Potential Memory Leak when adding a large collection of movies (1 Viewer)

[jasonwc]

Note: I posted this before on the main thread a while back but I never got a response so I'm reposting in its own thread as it's a pretty serious issue.

I've had a recurring memory usage issue which I hope someone might be able to shed some light on. I'm not sure if it's caused by Mediaportal or if it's actually a memory leak in Windows. Basically, if I import my entire collection of movies (only about 400 movies) into Moving Pictures on ANY x64 version of Windows, the kernel paged memory usage skyrockets until I run out of memory. If I install an x86 version of Windows on the same system, the issue doesn't occur. The reason I think this may be a Windows issue is that the problem occurs where the movies are stored - not necessarily on the system where Mediaportal is being run. For example, I now have about 4 TB of movies stored on a WHS 2011 storage server, and when I ran a moving pictures import on my HTPC, kernel memory usage climbed on the storage server to over 7 GB from about 150-200 MB. If the movies are stored locally, the memory usage increases on the machine doing the import.

I've now seen this issue on three different systems w/ Mediaportal 1.2 Beta or 1.2 final and various versions of Moving Pictures including 1.2.3:

Desktop
Core i7 860 2.8 Ghz Quad Core CPU
Asus P7P55D-E PRO 1156 Motherboard
12 GB DDR3 1600 RAM
2 x ATI 5850 @ stock
160 GB Intel X-25 M SSD
2 TB drive temp storage
Windows 7 x64 (Issues occurs)/Windows 7 x86 (no issue - same driver versions; all up-to-date)

HTPC
Core i3 550 @ 3.2 Ghz Dual Core CPU
GIGABYTE GA-P55-UD3R 1156 Socket Motherboard
4 GB DDR 1333 RAM
Nvidia GT430
Cruical C300 60 GB SSD
2 TB of storage (used to be 12 TB but since moved to storage server)
Windows 7 x64 (issue occurs); Windows 7 x86 (no issue)

Storage Server:
Norco RPC-4224 4U Rackmount case with 24 hotswap bays
Asus P8B WS Workstation Motherboard (Dual Intel 82574L Server NICs)
Supermicro AOC-SASLP-MV8 SAS Controller
Core i3-2100 3.2 Ghz 1155 Socket CPU
8 GB DDR3 1333 RAM
No video card
WD Black 1 TB for OS
14 TB for Storage
WHS 2011 (based on Win Server 2008 R2, only comes in x64 version - issue occurs as described above)

I was hoping someone could provide some insight into this issue. If logs would be helpful, tell me what to provide. The memory usage for the Mediaportal process doesn't seem to be the issue. Rather, the problem is that the kernel paged memory skyrockets until there is no more free memory, at which time the system becomes slow and unresponsive, requiring a reboot

 

[fforde]

Well the only memory issue I am aware of should be fixed. It was not a memory leak it was just an overly aggressive cache. To speed things up the importer would cache retrieved web pages so if the same URL is called twice the second time the cached copy would be used. The problem was this cache was done completely in memory, so this would create problems for people with large collections or people using overly aggressive scripts like IMDB+. The latest version of Moving Pictures corrects this and caches this data on disk so the problem should be resolved with version 1.2.3.

So how many movies are you importing? And with what scripts? And are you certain you are seeing this problem with Moving Pictures 1.2.3? If you can still reproduce the problem can you please post debug logs?

 

[jasonwc]

The collection isn't that large - around 325 movies. I'm currently using IMDB+ scraper but I first reported the issue using the default IMDB scraper. I've had this issue with many versions of Moving Pictures including 1.2.3.

I can do an import on my desktop tonight and send you the logs

 

[RoChess]

The collection isn't that large - around 325 movies. I'm currently using IMDB+ scraper but I first reported the issue using the default IMDB scraper. I've had this issue with many versions of Moving Pictures including 1.2.3.

I can do an import on my desktop tonight and send you the logs

There was a memory problem in MovPic before, but fforde did an awesome job at fixing this and I've done 3000+ collection imports on 2GB total memory systems without any problems since then.

Check your movingpictures.log file otherwise (or the old.log one) and look for: (date) (time) Info [ MovingPicturesCore]: Moving Pictures (1.2.3:1335)

 

[jasonwc]

fforde,

I just installed Mediaportal 1.2.1 and Moving Pictures 1.2.3 on my Desktop running Windows 7 x64 SP1. I had no unusual memory usage on the desktop while adding movies, but as expected as soon as the import started the kernel paged memory usage spiked on the server (where the movies are stored) to nearly 6 GB from a few hundred MB, resulting in 93% memory utilization up from around 20%. I have attached the debug logs you requested.

RoChess,

I see a 1.2.3.1329 - not 1.2.3.1355, and I just downloaded a copy from the Moving Pictures forum.

04-Nov-2011 18:56:54 Info [ MovingPicturesCore]: Moving Pictures (1.2.3:1329)

 

[RoChess]

The issue in question that solves the problem was r1313 as per: r1313 - Google Project Hosting

And this was merged into the MediaPortal v1.2.x branch via: r1317 - Google Project Hosting

So that "should" mean the version you are running has the bug fixed.

Is perhaps something else going wrong? MediaInfo for example is also processed during import and is known to have some serious issues with an older release that is installed by MediaPortal.

So you can either disable MediaInfo retrieval in MovPic during import, or download the 32-bit Stand-Alone v0.7.50 DLL from the MediaInfo website and overwrite the older version you currently have.

 

[jasonwc]

No dice. After renaming the old DLL and adding the new 0.7.50 DLL, I cleared my movie database, restarted Mediaportal config, and re-added my Movie directory. As soon as the scan began kernel paged memory shot up to 6 GB on the server as it did yesterday.

 

[fforde]

I just installed Mediaportal 1.2.1 and Moving Pictures 1.2.3 on my Desktop running Windows 7 x64 SP1. I had no unusual memory usage on the desktop while adding movies, but as expected as soon as the import started the kernel paged memory usage spiked on the server (where the movies are stored) to nearly 6 GB from a few hundred MB, resulting in 93% memory utilization up from around 20%. I have attached the debug logs you requested.

Based on the log files you posted an import is not even occuring. Moving Pictures is actually doing very little. There might be more information with debug logging turned on. Would you mind posting debug log files?

Also, I think I misunderstood but it sounds like you are saying your movies are stored on a separate machine and you are seeing the excessive memory usage on the file server and not your HTPC? Is this correct?

 

[jasonwc]

I am re-running an import with debug logging on and should have new logs shortly.

You are correct about the memory usage. The kernel paged memory spikes wherever the movies are stored. Less than 2 months ago, I moved my movies from my HTPC to a dedicated file server. Before doing so, the memory spike occurred on the HTPC when doing a large import. Now it occurs on the file server. The spike is very obvious and it's the only application I know of that causes such behavior.

Based on the release date of Moving Pictures 1.2.3, I believe I was using it on the HTPC before migrating my data to the file server. I use StreamedMP and it updates my plugins regularly.

 

[fforde]

I am not sure I will be able to help you much then. Moving Pictures can only allocate memory on the machine it is running on. It is impossible for it to allocate memory on your file server. Moving Pictures is running a mediainfo scan on each file on import though which is non trivial. Can you try turning off mediainfo retrieval? This is togglable under the gear icon on the importer dialog.

Setting mediainfo processing aside, all Moving Pictures is doing to the file server is passively browsing the file system...