[fixed] [MP2-833] 2.2.2 server crash during import (4 Viewers)

[henso]

Is the thumbnail creation an asynchronous process?

You could say that, yes.

I conclude that you rather not believe in a memory leak

From the memory graph shown it looks like memory usage is pretty stable except for the spikes occurring at times. Memory leaks should look like increasing memory usage that never decreases again. I could add throttling to avoid too many thumbnails being generated at once which might be the cause of the spikes.

 

[HTPCSourcer]

Memory leaks should look like increasing memory usage that never decreases again.

OK. The theoretical memory usage limit for the 32bit process would be 4 GB, in practice maybe 3.5 GB because of administratve overhead. Or is it even just 2 GB? What is the code supposed to do if the memory requirements grow above such limits, e.g. instead of importing 8000 movies you would look at 20000 with respectively more pictures to load/create, "thumbnail" them and store? If you're correct then throttling would help, yet it would be subjct to the CPU speed, i.e. crash more likely on a fast PC rather than a slow one.

 

[JSchuricht]

A more powerful CPU may increase the chance of this issue happening but I have seen the importer crash on lower powered systems. I was pointing out the memory spike since there is a 2GB limit on most 32 bit processes. I do agree with a memory leak being linear on the graph vs spikes. I have chased down a few leaks on manufacturing tools, the memory isn't released and the application just keeps grabbing more till something fails. What I am seeing with MP2 looks like a sudden demand for memory usage and maybe it is hitting a limit and causing a crash. The issue could still be other things.

I grabbed a few more graphs to show more than just the memory spike.

 

[morpheus_xx]

Yes those spikes could mean that there is a corrupt file, which causes on "decoding" an huge memory allocation once.
I have read about manipulated zip files before that create an unlimited size file on decompression, so called zip bomb. It also allocates all memory until something crashes

Gesendet von meinem SM-G930F mit Tapatalk

 

[henso]

If you're correct then throttling would help, yet it would be subjct to the CPU speed, i.e. crash more likely on a fast PC rather than a slow one.

By throttling I mean how many thumbnails can be extracted at the same time. I will also try to test if we could move the thumbnail extraction to the fanart handling instead. This should increase the import speed and decrease database size/load.

 

[HTPCSourcer]

@JSchuricht , can you repeat your import tests? @henso has added code that should limit the memory usage

 

[JSchuricht]

I will give it a try. Have one other import issue I am tracking down then will work on this.

 

[JSchuricht]

I started the stress test with WIP 168. Should OCVVideoThumbnailer be present in the installation? The import is going fast but with OCVVideoThumbnailer not in the plugin directory it leaves me wondering if it should be there.

 

[HTPCSourcer]

Should OCVVideoThumbnailer be present in the installation?

I have a "OpenCvLib" folder in my setup.

 

[JSchuricht]

I have a "OpenCvLib" folder in my setup.

Didn't realize that was the same thing/replacement.

Anyways, first 34% was really quick but it has slowed down. Up to 41% now with memory <300MB. Should have results in a day but it is looking promising so far.