[fixed] [MP2-833] 2.2.2 server crash during import (1 Viewer)

[JSchuricht]

This was an interesting test. It could be my imagination but the import seems significantly faster. I pulled in 8268 movies without a crash. I also noticed there is barely any CPU usage during the import. Normally when I start an import the CPU is near 100% on all threads, also there are some large spikes in memory usage that didn't occur this time.

I am wondering if we are hitting a memory limit in the OCVVideoThumbnailer plugin with the server process being 32 bit. I ran across the OutOfMemoryException in the logs an hour before the MP2-Server.exe process crashed. I stopped getting data at the time of the crash but when the SNMP poll found data in the from the OID stream MP2-Server.exe was eating up 2.36GB. Making a bit of a leap but if OCVVideoThumbnailer is making the difference in memory usage during the import, that one plugin could be hitting the 2GB limit.

@morpheus_xx IIRC there are some things not supported in x64 server which is holding up the ability to publish a x64 version. Is it possible to have a x64 MP2 server and keep the tvserver or whatever is still 32 bit at the same time? Basically a hybrid x64 server with some 32bit parts.

Crashed import with OCVVideoThumbnailer.

Import without OCVVideoThumbnailer.

 

[JSchuricht]

Another good thing just came out of this test. I identified one specific media item that has refused to import. With OCVVideoThumbnailer disabled, that media item imported for the first time.

 

[HTPCSourcer]

wondering if we are hitting a memory limit in the OCVVideoThumbnailer plugin with the server process

This seems to be a frequent issue when using Open CV. Yet memory leaks seem to occur in the code that uses the library, not in the library itself. See for example
OpenCV Memory Management

 

[HTPCSourcer]

Captured as MP2-833 Memory leak during thumbnail creation

 

[Lehmden]

Hi.

It could be my imagination but the import seems significantly faster.

I can't believe it's imagination. Generating ten thousands of thumbnails will eat a lot of time, CPU and memory for sure. Imho the import has to be faster a lot without thumbnail generation. If it wouldn't be faster there something really must go wrong...

With OCVVideoThumbnailer disabled, that media item imported for the first time.

This indicates a faulty video for sure. Most videos out there are not 100% clean and perfect. This especially is valid for TV recordings but it happens to nearly any type of video... A lot (most) of those "not that perfect" videos are still playing well but sometimes you can not recode them without failures. And creating thumbnails is a sort of recode. In general MediaInfo is the one that noticed those faulty videos at first. But then it would not import at all so in this case it should be ffmpeg who is not willing to work with this video...

But here the culprit most likely is the fact that this specific "mkv" is no mkv at all. It's a m2ts BluRay stream with a wrong file extension... Try to rename the file to .m2ts or remux it into a real mkv container (with MKVToolnix or Media-Buddy for example, it's as fast as copying the file to a different space) and after that it most likely will be imported without issues...

 

[morpheus_xx]

We could use the former ffmpeg.exe based extractor again. It has the advantage that it spawns extra processes (also 64 bit). And if process leaks memory it has no influence on MP2 server process.

For testing I can post a 64bit server exe. But a mix with 32 bit does not work, so no tv then.

Gesendet von meinem SM-G930F mit Tapatalk

 

[JSchuricht]

@Lehmden You are correct about the video source. I could convert this one file, what concerns me is I have hundreds of videos done the same way and only one has an issue during MP2 imports.

 

[Lehmden]

Hi.

I have hundreds of videos done the same way

It never is a good idea to rename files to a wrong file extension, never ever... You can mass-remux all your wrongly named files into mkv container with only one single mouse click in Media-Buddy and make "real" mkv out of them or you can batch rename all your wrongly named "mkv" to "m2ts" also with very little work in e.g. TotalCommander or other tools that can do batch renaming... Mediaportal 2 can handle m2ts files without issues so no need to name them wrong. Especially as MP2 tries to read Matroska tags out of those fake mkv files...

 

[henso]

We could use the former ffmpeg.exe based extractor again.

I was thinking the same thing, but it will be slower than with OpenCV if I remember correctly. I was also thinking if we should store the thumbnails in the fanart cache instead of in the database? It might lighten the load on the database during import.

 

[HTPCSourcer]

I was thinking the same thing, but it will be slower than with OpenCV

Is the thumbnail creation an asynchronous process? If not, would it make sense to convert it into one? From what morpheus and you write, I conclude that you rather not believe in a memory leak. Could you imagine to add code for testing that logs heap information to confirm or deny this hypothesis?