[fixed] MP2 Importer Exception of type 'System.OutOfMemoryException' was thrown. (1 Viewer)

[Terra]

@morpheus_xx I replaced the files now and started the import

Attached new logs from today night with the changes of @Brownard, this time the logs clearly show an out of memory error again

 

[Terra]

@Terra can you test attached plugin?

Same again, i now have all the modifications active from @Brownard and your new thumbnail generator.
New logs attached.

[2018-12-11 10:29:59,878] [7242638] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item 7dde5e50-34d5-4c07-9bb1-9e8f0b9185a6 (Elizabeth Wilson) of type Thumbnail
[2018-12-11 10:29:59,879] [7242639] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Elizabeth Wilson [7dde5e50-34d5-4c07-9bb1-9e8f0b9185a6]
[2018-12-11 10:30:00,177] [7242937] [92       ] [ERROR] - SettingsManager: Error setting property 'PendingFanArtActions' in settings of type 'FanArtActionSettings'
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StringWriter.ToString()
   at System.Xml.XmlNode.get_InnerXml()
   at System.Xml.XmlElement.get_InnerXml()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.GetValue(String entryName, Type type)
   at MediaPortal.Common.Services.Settings.SettingsManager.LoadSettingsObject(Type settingsType, Boolean useGlobaleScope, Boolean useUserScope)
[...]
[2018-12-11 10:30:00,371] [7243131] [13       ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Mandy Chan [c3900e31-0ed9-42d4-a243-7b7c0a9e08a7]
[2018-12-11 10:30:00,427] [7243187] [139      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item c3900e31-0ed9-42d4-a243-7b7c0a9e08a7 (Mandy Chan) of type Thumbnail
[2018-12-11 10:30:00,428] [7243188] [139      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Mandy Chan [c3900e31-0ed9-42d4-a243-7b7c0a9e08a7]
[2018-12-11 10:30:00,686] [7243446] [115      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Martin Hentschel [f270c730-b8c8-4fb3-bfe7-684280a6a46b]
[2018-12-11 10:30:00,687] [7243447] [92       ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
   at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)
[2018-12-11 10:30:00,690] [7243450] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Katja Riemann [46c12796-5143-412e-949e-7003e86537ac]
[2018-12-11 10:30:00,696] [7243456] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 0 for media item 46c12796-5143-412e-949e-7003e86537ac (Katja Riemann) of type Thumbnail
[2018-12-11 10:30:00,697] [7243457] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Katja Riemann [46c12796-5143-412e-949e-7003e86537ac]
[2018-12-11 10:30:00,738] [7243498] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item f270c730-b8c8-4fb3-bfe7-684280a6a46b (Martin Hentschel) of type Thumbnail
[2018-12-11 10:30:00,739] [7243499] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Martin Hentschel [f270c730-b8c8-4fb3-bfe7-684280a6a46b]
[2018-12-11 10:30:00,841] [7243601] [105      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Wolfgang Lukschy [a197bba0-687e-4aa1-ac8a-aa8e8efb80df]
[2018-12-11 10:30:00,902] [7243662] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item a197bba0-687e-4aa1-ac8a-aa8e8efb80df (Wolfgang Lukschy) of type Thumbnail
[2018-12-11 10:30:00,902] [7243662] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Wolfgang Lukschy [a197bba0-687e-4aa1-ac8a-aa8e8efb80df]
[2018-12-11 10:30:01,141] [7243901] [92       ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
   at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)

 

[morpheus_xx]

@Brownard what makes me wonder is the often repeated OOM exception when saving settings:
[2018-12-11 10:17:27,070] [6489830] [92 ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)

It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.

 

[Terra]

It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.

I can confirm that, when I look in "C:\ProgramData\Team MediaPortal\MP2-Server\Config" during the import the file "MediaPortal.Extensions.UserServices.FanArtService.Settings.FanArtActionSettings.xml" is changed the whole time very often.

 

[Brownard]

It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.

Yes currently working on it. The settings store the fanart downloads that are currently pending so they can be restored if the server is stopped. Currently they are being saved after every new fanart action is added/completed, but it turns our this is very frequent and the list of pending actions grows very big .
I already reduced the amount of stuff it keeps in memory in the previous changes I posted, so the memory footprint shouldn't be that big (only a couple of guids for every item to collect fanart for) but it does try to allocate two very long arrays when saving the settings which probably causes the issues when memory is already tight. One of those allocations is unnecessary so I've removed it and I'll also batch up the settings saving so it does it less frequently.

I think ultimately it's the 15 concurrent imports that cause the initial memory pressure though, so I'll also implement a queue for those.

 

[Terra]

I think ultimately it's the 15 concurrent imports that cause the initial memory pressure though, so I'll also implement a queue for those.

Just to let you know in the last log files I only started manually around 3-5 share imports at the same time.

 

[Brownard]

Just to let you know in the last log files I only started manually around 3-5 share imports at the same time.

Thanks, can you try the attached, same procedure as before.

 

[Terra]

The importer is running now with the V3 changes.

In the meanwhile the MP2 importer has started indexing my pictures folder and it also crashed the server.
Also seems like an OOM issue.
-> The underlying compression routine could not reserve sufficient memory.


Please note this log file is before I installed the new changes from @Brownard (V3):

[2018-12-11 23:21:08,130] [6091840] [AMQ 'FanArtLibraryManager'] [DEBUG] - FanArtManagement: Scheduling fanart collection for 6ed2e267-b8f7-49c4-9a92-94c2f10cff8a.
[2018-12-11 23:21:08,132] [6091842] [74       ] [DEBUG] - MediaAccessor: Importing file 'CIMG0405.JPG'
[2018-12-11 23:21:08,343] [6092054] [74       ] [WARN ] - ImageProcessorThumbnailProvider: Error loading bitmapSource from file data stream
System.OutOfMemoryException: Out of memory.

[2018-12-11 23:21:08,417] [6092127] [91       ] [DEBUG] - Adding media item  with name CIMG0393 (CIMG0393.JPG)
[2018-12-11 23:21:08,428] [6092138] [91       ] [INFO ] - Media item 5c9ff0f7-cd9c-4457-b253-093552b0fa32 with name CIMG0393 (CIMG0393.JPG) imported (10 ms)
[2018-12-11 23:21:08,428] [6092138] [70       ] [ERROR] - Error: UPnPServer: Error handling HTTP request 'http://127.0.0.1:55555/MediaPortal/UPnPServer_4662DE78/upnphost/control/uuid:1d47ac12-d16a-4217-acd4-26c17289b38b/urn:schemas-team-mediaportal-com:service:ServerStateService:1'
[2018-12-11 23:21:08,430] [6092140] [91       ] [DEBUG] - Adding media item  with name CIMG0394 (CIMG0394.JPG)
[2018-12-11 23:21:08,435] [6092145] [70       ] [DEBUG] - Exception: System.IO.Compression.ZLibException: The underlying compression routine could not reserve sufficient memory.
   at System.IO.Compression.DeflaterZLib.DeflateInit(CompressionLevel compressionLevel, Int32 windowBits, Int32 memLevel, CompressionStrategy strategy)
   at System.IO.Compression.DeflaterZLib..ctor(CompressionLevel compressionLevel)
   at System.IO.Compression.DeflateStream.CreateDeflater(Nullable`1 compressionLevel)
   at System.IO.Compression.DeflateStream..ctor(Stream stream, CompressionMode mode, Boolean leaveOpen)
   at UPnP.Infrastructure.Utils.DeflateCompression.CreateCompressionStream(Stream targetStream)
   at UPnP.Infrastructure.Utils.CompressionHelper.Compress(IDeCompressor compressor, Stream inputStream)
   at UPnP.Infrastructure.Utils.CompressionHelper.<WriteCompressedStream>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at UPnP.Infrastructure.Dv.UPnPServer.<HandleHTTPRequest>d__21.MoveNext()
[2018-12-11 23:21:08,436] [6092146] [70       ] [DEBUG] -   Message: The underlying compression routine could not reserve sufficient memory.

Please see logs attached.

 

[Terra]

With all the changes installed so far the importer now went through without any Server Errors!
However the GUI crashed in the morning and I am also missing a lot of covers for recognized tv shows now.

Logs are attached.

 

[ge2301]

With all the changes installed so far the importer now went through without any Server Errors!
However the GUI crashed in the morning and I am also missing a lot of covers for recognized tv shows now.

Logs are attached.

When did you start your last import?
I see the logs full of

[2018-12-12 01:50:42,398] [7823004] [132 ] [WARN ] - ImageProcessorThumbnailProvider: Error loading bitmapSource from file data stream
System.OutOfMemoryException: Out of memory.
at System.Drawing.Image.FromStream(Stream stream, Boolean useEmbeddedColorManagement, Boolean validateImageData)
at MediaPortal.Extensions.MetadataExtractors.GDIThumbnailProvider.GDIThumbnailProvider.GetThumbnail(Stream stream, Int32 width, Int32 height, Boolean cachedOnly, Byte[]& imageData, ImageType& imageType)

Thats probably the reason, why you have no thumbnails. But to be sure that those entries are not from a previous import, please provide date and time of the last import.