[fixed] MP2 Importer Exception of type 'System.OutOfMemoryException' was thrown. (1 Viewer)

Terra

Portal Pro
August 25, 2009
230
57
Home Country
Austria Austria
@morpheus_xx I replaced the files now and started the import

Attached new logs from today night with the changes of @Brownard, this time the logs clearly show an out of memory error again
 

Terra

Portal Pro
August 25, 2009
230
57
Home Country
Austria Austria
@Terra can you test attached plugin?

Same again, i now have all the modifications active from @Brownard and your new thumbnail generator.
New logs attached.


Code:
[2018-12-11 10:29:59,878] [7242638] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item 7dde5e50-34d5-4c07-9bb1-9e8f0b9185a6 (Elizabeth Wilson) of type Thumbnail
[2018-12-11 10:29:59,879] [7242639] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Elizabeth Wilson [7dde5e50-34d5-4c07-9bb1-9e8f0b9185a6]
[2018-12-11 10:30:00,177] [7242937] [92       ] [ERROR] - SettingsManager: Error setting property 'PendingFanArtActions' in settings of type 'FanArtActionSettings'
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StringWriter.ToString()
   at System.Xml.XmlNode.get_InnerXml()
   at System.Xml.XmlElement.get_InnerXml()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.GetValue(String entryName, Type type)
   at MediaPortal.Common.Services.Settings.SettingsManager.LoadSettingsObject(Type settingsType, Boolean useGlobaleScope, Boolean useUserScope)
[...]
[2018-12-11 10:30:00,371] [7243131] [13       ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Mandy Chan [c3900e31-0ed9-42d4-a243-7b7c0a9e08a7]
[2018-12-11 10:30:00,427] [7243187] [139      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item c3900e31-0ed9-42d4-a243-7b7c0a9e08a7 (Mandy Chan) of type Thumbnail
[2018-12-11 10:30:00,428] [7243188] [139      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Mandy Chan [c3900e31-0ed9-42d4-a243-7b7c0a9e08a7]
[2018-12-11 10:30:00,686] [7243446] [115      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Martin Hentschel [f270c730-b8c8-4fb3-bfe7-684280a6a46b]
[2018-12-11 10:30:00,687] [7243447] [92       ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
   at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)
[2018-12-11 10:30:00,690] [7243450] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Katja Riemann [46c12796-5143-412e-949e-7003e86537ac]
[2018-12-11 10:30:00,696] [7243456] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 0 for media item 46c12796-5143-412e-949e-7003e86537ac (Katja Riemann) of type Thumbnail
[2018-12-11 10:30:00,697] [7243457] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Katja Riemann [46c12796-5143-412e-949e-7003e86537ac]
[2018-12-11 10:30:00,738] [7243498] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item f270c730-b8c8-4fb3-bfe7-684280a6a46b (Martin Hentschel) of type Thumbnail
[2018-12-11 10:30:00,739] [7243499] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Martin Hentschel [f270c730-b8c8-4fb3-bfe7-684280a6a46b]
[2018-12-11 10:30:00,841] [7243601] [105      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Wolfgang Lukschy [a197bba0-687e-4aa1-ac8a-aa8e8efb80df]
[2018-12-11 10:30:00,902] [7243662] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item a197bba0-687e-4aa1-ac8a-aa8e8efb80df (Wolfgang Lukschy) of type Thumbnail
[2018-12-11 10:30:00,902] [7243662] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Wolfgang Lukschy [a197bba0-687e-4aa1-ac8a-aa8e8efb80df]
[2018-12-11 10:30:01,141] [7243901] [92       ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
   at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)
 
Last edited:

morpheus_xx

Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    @Brownard what makes me wonder is the often repeated OOM exception when saving settings:
    [2018-12-11 10:17:27,070] [6489830] [92 ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
    at System.Text.StringBuilder.ToString()
    at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
    at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)

    It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.

    I can confirm that, when I look in "C:\ProgramData\Team MediaPortal\MP2-Server\Config" during the import the file "MediaPortal.Extensions.UserServices.FanArtService.Settings.FanArtActionSettings.xml" is changed the whole time very often.
     

    Attachments

    • Bildschirmfoto 2018-12-11 um 22.00.15.png
      Bildschirmfoto 2018-12-11 um 22.00.15.png
      432.5 KB

    Brownard

    Development Group
  • Team MediaPortal
  • March 21, 2007
    2,290
    1,872
    Home Country
    United Kingdom United Kingdom
    It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.
    Yes currently working on it. The settings store the fanart downloads that are currently pending so they can be restored if the server is stopped. Currently they are being saved after every new fanart action is added/completed, but it turns our this is very frequent and the list of pending actions grows very big :whistle:.
    I already reduced the amount of stuff it keeps in memory in the previous changes I posted, so the memory footprint shouldn't be that big (only a couple of guids for every item to collect fanart for) but it does try to allocate two very long arrays when saving the settings which probably causes the issues when memory is already tight. One of those allocations is unnecessary so I've removed it and I'll also batch up the settings saving so it does it less frequently.

    I think ultimately it's the 15 concurrent imports that cause the initial memory pressure though, so I'll also implement a queue for those.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    I think ultimately it's the 15 concurrent imports that cause the initial memory pressure though, so I'll also implement a queue for those.
    Just to let you know in the last log files I only started manually around 3-5 share imports at the same time.
     

    Brownard

    Development Group
  • Team MediaPortal
  • March 21, 2007
    2,290
    1,872
    Home Country
    United Kingdom United Kingdom
    Just to let you know in the last log files I only started manually around 3-5 share imports at the same time.
    Thanks, can you try the attached, same procedure as before.
     

    Attachments

    • Importer_memory_usage_v3.zip
      474.2 KB

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    The importer is running now with the V3 changes.

    In the meanwhile the MP2 importer has started indexing my pictures folder and it also crashed the server.
    Also seems like an OOM issue.
    -> The underlying compression routine could not reserve sufficient memory.


    Please note this log file is before I installed the new changes from @Brownard (V3):

    Code:
    [2018-12-11 23:21:08,130] [6091840] [AMQ 'FanArtLibraryManager'] [DEBUG] - FanArtManagement: Scheduling fanart collection for 6ed2e267-b8f7-49c4-9a92-94c2f10cff8a.
    [2018-12-11 23:21:08,132] [6091842] [74       ] [DEBUG] - MediaAccessor: Importing file 'CIMG0405.JPG'
    [2018-12-11 23:21:08,343] [6092054] [74       ] [WARN ] - ImageProcessorThumbnailProvider: Error loading bitmapSource from file data stream
    System.OutOfMemoryException: Out of memory.

    Code:
    [2018-12-11 23:21:08,417] [6092127] [91       ] [DEBUG] - Adding media item  with name CIMG0393 (CIMG0393.JPG)
    [2018-12-11 23:21:08,428] [6092138] [91       ] [INFO ] - Media item 5c9ff0f7-cd9c-4457-b253-093552b0fa32 with name CIMG0393 (CIMG0393.JPG) imported (10 ms)
    [2018-12-11 23:21:08,428] [6092138] [70       ] [ERROR] - Error: UPnPServer: Error handling HTTP request 'http://127.0.0.1:55555/MediaPortal/UPnPServer_4662DE78/upnphost/control/uuid:1d47ac12-d16a-4217-acd4-26c17289b38b/urn:schemas-team-mediaportal-com:service:ServerStateService:1'
    [2018-12-11 23:21:08,430] [6092140] [91       ] [DEBUG] - Adding media item  with name CIMG0394 (CIMG0394.JPG)
    [2018-12-11 23:21:08,435] [6092145] [70       ] [DEBUG] - Exception: System.IO.Compression.ZLibException: The underlying compression routine could not reserve sufficient memory.
       at System.IO.Compression.DeflaterZLib.DeflateInit(CompressionLevel compressionLevel, Int32 windowBits, Int32 memLevel, CompressionStrategy strategy)
       at System.IO.Compression.DeflaterZLib..ctor(CompressionLevel compressionLevel)
       at System.IO.Compression.DeflateStream.CreateDeflater(Nullable`1 compressionLevel)
       at System.IO.Compression.DeflateStream..ctor(Stream stream, CompressionMode mode, Boolean leaveOpen)
       at UPnP.Infrastructure.Utils.DeflateCompression.CreateCompressionStream(Stream targetStream)
       at UPnP.Infrastructure.Utils.CompressionHelper.Compress(IDeCompressor compressor, Stream inputStream)
       at UPnP.Infrastructure.Utils.CompressionHelper.<WriteCompressedStream>d__7.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
       at UPnP.Infrastructure.Dv.UPnPServer.<HandleHTTPRequest>d__21.MoveNext()
    [2018-12-11 23:21:08,436] [6092146] [70       ] [DEBUG] -   Message: The underlying compression routine could not reserve sufficient memory.


    Please see logs attached.
     
    Last edited:

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    With all the changes installed so far the importer now went through without any Server Errors!
    However the GUI crashed in the morning and I am also missing a lot of covers for recognized tv shows now.

    Logs are attached.
     

    ge2301

    Lead Design MP2
  • Team MediaPortal
  • January 11, 2014
    8,705
    3,491
    Stuttgart
    Home Country
    Germany Germany
    With all the changes installed so far the importer now went through without any Server Errors!
    However the GUI crashed in the morning and I am also missing a lot of covers for recognized tv shows now.

    Logs are attached.
    When did you start your last import?
    I see the logs full of
    [2018-12-12 01:50:42,398] [7823004] [132 ] [WARN ] - ImageProcessorThumbnailProvider: Error loading bitmapSource from file data stream
    System.OutOfMemoryException: Out of memory.
    at System.Drawing.Image.FromStream(Stream stream, Boolean useEmbeddedColorManagement, Boolean validateImageData)
    at MediaPortal.Extensions.MetadataExtractors.GDIThumbnailProvider.GDIThumbnailProvider.GetThumbnail(Stream stream, Int32 width, Int32 height, Boolean cachedOnly, Byte[]& imageData, ImageType& imageType)
    Thats probably the reason, why you have no thumbnails. But to be sure that those entries are not from a previous import, please provide date and time of the last import.
     

    Users who are viewing this thread

    Top Bottom