[fixed] MP2 Importer Exception of type 'System.OutOfMemoryException' was thrown. (1 Viewer)

Terra

Portal Pro
August 25, 2009
230
56
Austria Austria
Country flag
@morpheus_xx I replaced the files now and started the import

Attached new logs from today night with the changes of @Brownard, this time the logs clearly show an out of memory error again
 

Terra

Portal Pro
August 25, 2009
230
56
Austria Austria
Country flag
@Terra can you test attached plugin?
Same again, i now have all the modifications active from @Brownard and your new thumbnail generator.
New logs attached.


Code:
[2018-12-11 10:29:59,878] [7242638] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item 7dde5e50-34d5-4c07-9bb1-9e8f0b9185a6 (Elizabeth Wilson) of type Thumbnail
[2018-12-11 10:29:59,879] [7242639] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Elizabeth Wilson [7dde5e50-34d5-4c07-9bb1-9e8f0b9185a6]
[2018-12-11 10:30:00,177] [7242937] [92       ] [ERROR] - SettingsManager: Error setting property 'PendingFanArtActions' in settings of type 'FanArtActionSettings'
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StringWriter.ToString()
   at System.Xml.XmlNode.get_InnerXml()
   at System.Xml.XmlElement.get_InnerXml()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.GetValue(String entryName, Type type)
   at MediaPortal.Common.Services.Settings.SettingsManager.LoadSettingsObject(Type settingsType, Boolean useGlobaleScope, Boolean useUserScope)
[...]
[2018-12-11 10:30:00,371] [7243131] [13       ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Mandy Chan [c3900e31-0ed9-42d4-a243-7b7c0a9e08a7]
[2018-12-11 10:30:00,427] [7243187] [139      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item c3900e31-0ed9-42d4-a243-7b7c0a9e08a7 (Mandy Chan) of type Thumbnail
[2018-12-11 10:30:00,428] [7243188] [139      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Mandy Chan [c3900e31-0ed9-42d4-a243-7b7c0a9e08a7]
[2018-12-11 10:30:00,686] [7243446] [115      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Martin Hentschel [f270c730-b8c8-4fb3-bfe7-684280a6a46b]
[2018-12-11 10:30:00,687] [7243447] [92       ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
   at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)
[2018-12-11 10:30:00,690] [7243450] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Katja Riemann [46c12796-5143-412e-949e-7003e86537ac]
[2018-12-11 10:30:00,696] [7243456] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 0 for media item 46c12796-5143-412e-949e-7003e86537ac (Katja Riemann) of type Thumbnail
[2018-12-11 10:30:00,697] [7243457] [132      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Katja Riemann [46c12796-5143-412e-949e-7003e86537ac]
[2018-12-11 10:30:00,738] [7243498] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item f270c730-b8c8-4fb3-bfe7-684280a6a46b (Martin Hentschel) of type Thumbnail
[2018-12-11 10:30:00,739] [7243499] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Martin Hentschel [f270c730-b8c8-4fb3-bfe7-684280a6a46b]
[2018-12-11 10:30:00,841] [7243601] [105      ] [DEBUG] - MovieTheMovieDbMatcher Download: Downloading images for Wolfgang Lukschy [a197bba0-687e-4aa1-ac8a-aa8e8efb80df]
[2018-12-11 10:30:00,902] [7243662] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Saved 1 for media item a197bba0-687e-4aa1-ac8a-aa8e8efb80df (Wolfgang Lukschy) of type Thumbnail
[2018-12-11 10:30:00,902] [7243662] [147      ] [DEBUG] - MovieTheMovieDbMatcher Download: Finished saving images for Wolfgang Lukschy [a197bba0-687e-4aa1-ac8a-aa8e8efb80df]
[2018-12-11 10:30:01,141] [7243901] [92       ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
   at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)
 
Last edited:

morpheus_xx

Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,390
    6,966
    Germany Germany
    Country flag
    @Brownard what makes me wonder is the often repeated OOM exception when saving settings:
    [2018-12-11 10:17:27,070] [6489830] [92 ] [ERROR] - SettingsManager: Error writing settings of type 'FanArtActionSettings'... Will clear settings files for this setting
    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
    at System.Text.StringBuilder.ToString()
    at MediaPortal.Common.Services.Settings.SettingsFileHandler.SetValue(String entryName, Object value)
    at MediaPortal.Common.Services.Settings.SettingsManager.SaveSettingsObject(Object settingsObject, Boolean useGlobaleScope, Boolean useUserScope)

    It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    56
    Austria Austria
    Country flag
    It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.
    I can confirm that, when I look in "C:\ProgramData\Team MediaPortal\MP2-Server\Config" during the import the file "MediaPortal.Extensions.UserServices.FanArtService.Settings.FanArtActionSettings.xml" is changed the whole time very often.
     

    Attachments

    Brownard

    Development Group
  • Team MediaPortal
  • March 21, 2007
    1,946
    1,298
    United Kingdom United Kingdom
    Country flag
    It looks like the FanArtActionSettings is very frequently modified, this should be checked if there are unnecessary write operations.
    Yes currently working on it. The settings store the fanart downloads that are currently pending so they can be restored if the server is stopped. Currently they are being saved after every new fanart action is added/completed, but it turns our this is very frequent and the list of pending actions grows very big :whistle:.
    I already reduced the amount of stuff it keeps in memory in the previous changes I posted, so the memory footprint shouldn't be that big (only a couple of guids for every item to collect fanart for) but it does try to allocate two very long arrays when saving the settings which probably causes the issues when memory is already tight. One of those allocations is unnecessary so I've removed it and I'll also batch up the settings saving so it does it less frequently.

    I think ultimately it's the 15 concurrent imports that cause the initial memory pressure though, so I'll also implement a queue for those.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    56
    Austria Austria
    Country flag
    I think ultimately it's the 15 concurrent imports that cause the initial memory pressure though, so I'll also implement a queue for those.
    Just to let you know in the last log files I only started manually around 3-5 share imports at the same time.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    56
    Austria Austria
    Country flag
    The importer is running now with the V3 changes.

    In the meanwhile the MP2 importer has started indexing my pictures folder and it also crashed the server.
    Also seems like an OOM issue.
    -> The underlying compression routine could not reserve sufficient memory.


    Please note this log file is before I installed the new changes from @Brownard (V3):

    Code:
    [2018-12-11 23:21:08,130] [6091840] [AMQ 'FanArtLibraryManager'] [DEBUG] - FanArtManagement: Scheduling fanart collection for 6ed2e267-b8f7-49c4-9a92-94c2f10cff8a.
    [2018-12-11 23:21:08,132] [6091842] [74       ] [DEBUG] - MediaAccessor: Importing file 'CIMG0405.JPG'
    [2018-12-11 23:21:08,343] [6092054] [74       ] [WARN ] - ImageProcessorThumbnailProvider: Error loading bitmapSource from file data stream
    System.OutOfMemoryException: Out of memory.
    Code:
    [2018-12-11 23:21:08,417] [6092127] [91       ] [DEBUG] - Adding media item  with name CIMG0393 (CIMG0393.JPG)
    [2018-12-11 23:21:08,428] [6092138] [91       ] [INFO ] - Media item 5c9ff0f7-cd9c-4457-b253-093552b0fa32 with name CIMG0393 (CIMG0393.JPG) imported (10 ms)
    [2018-12-11 23:21:08,428] [6092138] [70       ] [ERROR] - Error: UPnPServer: Error handling HTTP request 'http://127.0.0.1:55555/MediaPortal/UPnPServer_4662DE78/upnphost/control/uuid:1d47ac12-d16a-4217-acd4-26c17289b38b/urn:schemas-team-mediaportal-com:service:ServerStateService:1'
    [2018-12-11 23:21:08,430] [6092140] [91       ] [DEBUG] - Adding media item  with name CIMG0394 (CIMG0394.JPG)
    [2018-12-11 23:21:08,435] [6092145] [70       ] [DEBUG] - Exception: System.IO.Compression.ZLibException: The underlying compression routine could not reserve sufficient memory.
       at System.IO.Compression.DeflaterZLib.DeflateInit(CompressionLevel compressionLevel, Int32 windowBits, Int32 memLevel, CompressionStrategy strategy)
       at System.IO.Compression.DeflaterZLib..ctor(CompressionLevel compressionLevel)
       at System.IO.Compression.DeflateStream.CreateDeflater(Nullable`1 compressionLevel)
       at System.IO.Compression.DeflateStream..ctor(Stream stream, CompressionMode mode, Boolean leaveOpen)
       at UPnP.Infrastructure.Utils.DeflateCompression.CreateCompressionStream(Stream targetStream)
       at UPnP.Infrastructure.Utils.CompressionHelper.Compress(IDeCompressor compressor, Stream inputStream)
       at UPnP.Infrastructure.Utils.CompressionHelper.<WriteCompressedStream>d__7.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
       at UPnP.Infrastructure.Dv.UPnPServer.<HandleHTTPRequest>d__21.MoveNext()
    [2018-12-11 23:21:08,436] [6092146] [70       ] [DEBUG] -   Message: The underlying compression routine could not reserve sufficient memory.

    Please see logs attached.
     
    Last edited:

    Terra

    Portal Pro
    August 25, 2009
    230
    56
    Austria Austria
    Country flag
    With all the changes installed so far the importer now went through without any Server Errors!
    However the GUI crashed in the morning and I am also missing a lot of covers for recognized tv shows now.

    Logs are attached.
     

    ge2301

    MP2 Design
  • Team MediaPortal
  • January 11, 2014
    6,615
    2,385
    Stuttgart
    Germany Germany
    Country flag
    With all the changes installed so far the importer now went through without any Server Errors!
    However the GUI crashed in the morning and I am also missing a lot of covers for recognized tv shows now.

    Logs are attached.
    When did you start your last import?
    I see the logs full of
    [2018-12-12 01:50:42,398] [7823004] [132 ] [WARN ] - ImageProcessorThumbnailProvider: Error loading bitmapSource from file data stream
    System.OutOfMemoryException: Out of memory.
    at System.Drawing.Image.FromStream(Stream stream, Boolean useEmbeddedColorManagement, Boolean validateImageData)
    at MediaPortal.Extensions.MetadataExtractors.GDIThumbnailProvider.GDIThumbnailProvider.GetThumbnail(Stream stream, Int32 width, Int32 height, Boolean cachedOnly, Byte[]& imageData, ImageType& imageType)
    Thats probably the reason, why you have no thumbnails. But to be sure that those entries are not from a previous import, please provide date and time of the last import.
     

    Users Who Are Viewing This Thread (Users: 0, Guests: 1)

    OP Similar threads Forum Replies Date
    Lichloathe MP2 importing keyart instead of poster General 0
    J [fixed] [MP2-833] 2.2.2 server crash during import Submit: Bug Reports 50
    MausWiesel LightsOut-Connector für MP2? Allgemeines Support- und Diskussionsforum 9
    M [solved] MP2 Not discovering all movies Newcomers Forum 2
    J [fixed] [MP2-821] Importer glitches in 2.2 Submit: Bug Reports 42
    N [Finished] [MP2-791] Radio recordings do not get imported into Recordings listing MediaPortal 2 25
    raffe [solved] Export tool and import support from MP-TVSeries & Moving Pictures to MP2 General 22
    H MP2 database won't keep in synch with recordings General 3
    H [solved] MP2 Wiedergabe von Aufnahmen und Live TV geht nicht Allgemeines Support- und Diskussionsforum 6
    ge2301 [WiP] MP2Web - A platform independent web client for MP2 MediaPortal 2 58
    F Problems with new install of MP2 Newcomers Forum 3
    catalpa Bug MP2-837 Allgemeines Support- und Diskussionsforum 2
    P MP2: US user, OTA - can NOT get SchedulesDirect plugin to work on MP1 or MP2 --> how do I get OTA schedule? General 5
    bigboss97 Does anyone has a SoundGraph imon remote file for MP2 to share? General 0
    ge2301 Android based client for MP2 MediaPortal 2 3
    R MP2 GUI Issues General 7
    P [solved] MP2 new install, 2 x hdhomerun tuners, client on same PC unable to view TV & recordings Newcomers Forum 8
    Lichloathe Trakt for MP2 question General 7
    C Streaming/transcoding MP2 vs MP1? General 1
    P MP 2 - Request to add an additional TV series Filename protocal Feature Requests 0
    C Streaming to clients over Wifi. MP1 or MP2? Newcomers Forum 0
    CyberSimian MP2 synchronising of stoptime MediaPortal 1 Talk 5
    M Initial Setup MP2 (On Wndows 10) General 2
    R [solved] MP2 Client ist extrem langsam Allgemeines Support- und Diskussionsforum 5
    D [solved] EPG Buddy for MP2 - Does it work? General 5
    Similar threads

























    Top Bottom