[fixed] MP2 Importer Exception of type 'System.OutOfMemoryException' was thrown. (2 Viewers)

Terra

Portal Pro
August 25, 2009
230
57
Home Country
Austria Austria
MediaPortal 2 Version: 2.2 Final

Description
Hi guys,

So I tried MP2 again after some time and I have the issue that the MP2 Service crashes while importing my media assets.
I now restarted it at least 10 times over the last 3 days but its really annoying, not sure if it will finish any time...

I have 8GB of RAM and 312GB is free on the SSD.

Version: 2.2.1811.11777
Source: MP2.2_Final-11cc07

Here is the ServerError.log excerpt:

Code:
[2018-12-08 11:06:14,355] [2323606] [130      ] [INFO ] - ImporterWorker: 51 % completed (4 ImportJob(s), in total 11057 of 21613 so far identified resources processed)
[2018-12-08 11:06:18,623] [2327873] [111      ] [ERROR] - Unable to query
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Collections.Generic.List`1..ctor(Int32 capacity)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledFilter.CompileStatementParts(MIA_Management miaManagement, IFilter filter, IFilter subqueryFilter, Namespace ns, BindVarNamespace bvNamespace, ICollection`1 requiredMIATypes, String outerMIIDJoinVariable, ICollection`1 tableJoins, IList`1 resultParts, IList`1 resultBindVars)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledFilter..ctor(MIA_Management miaManagement, IFilter filter, IFilter subqueryFilter, Namespace ns, BindVarNamespace bvNamespace, String outerMIIDJoinVariable, ICollection`1 tableJoins)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.MainQueryBuilder.CreateCompiledFilter(Namespace ns, BindVarNamespace bvNamespace, String outerMIIDJoinVariable, IList`1 tableJoins)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.MainQueryBuilder.GenerateSqlStatement(Boolean groupByValues, IDictionary`2 miamAliases, String& mediaItemIdOrGroupSizeAlias, IDictionary`2& attributeAliases, String& statementStr, IList`1& bindVars)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledMediaItemQuery.AddMultipleMIAResults(ISQLDatabase database, ITransaction transaction, MultipleMediaItemAspectMetadata miam, IDictionary`2 attributes, MainQueryBuilder builder, IDictionary`2 multipleMiaValues)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledMediaItemQuery.AddMultipleMIAs(ISQLDatabase database, ITransaction transaction, IEnumerable`1 selectedMIAs, IList`1 ids, IDictionary`2 multipleMiaValues)
   at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledMediaItemQuery.Query(ISQLDatabase database, ITransaction transaction, Boolean singleMode)
[2018-12-08 11:06:21,324] [2330574] [87       ] [INFO ] - ImporterWorker.ImportJob(#5).DirectorySaveBlock: MediaLibrary disconnected. Requesting suspension...
[2018-12-08 11:06:21,328] [2330578] [145      ] [INFO ] - ImporterWorker.ImportJob(#4): Suspended
[2018-12-08 11:06:21,329] [2330579] [145      ] [INFO ] - ImporterWorker.ImportJob(#5): Suspended
[2018-12-08 11:06:21,330] [2330580] [145      ] [INFO ] - ImporterWorker.ImportJob(#6): Suspended
[2018-12-08 11:06:21,330] [2330580] [145      ] [INFO ] - ImporterWorker.ImportJob(#2): Suspended
[2018-12-08 11:06:21,331] [2330581] [145      ] [INFO ] - ImporterWorker: Suspended (4 ImportJobs pending)
[2018-12-08 11:06:20,933] [2330183] [67       ] [ERROR] - SettingsManager: Error setting property 'PendingFanArtActions' in settings of type 'FanArtActionSettings'
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
   at System.Text.StringBuilder.Append(Char* value, Int32 valueCount)
   at System.Text.StringBuilder.AppendHelper(String value)
   at System.Text.StringBuilder.Append(String value)
   at System.IO.StringWriter.Write(String value)
   at System.Xml.XmlTextWriter.WriteStartElement(String prefix, String localName, String ns)
   at System.Xml.XmlDOMTextWriter.WriteStartElement(String prefix, String localName, String ns)
   at System.Xml.XmlElement.WriteStartElement(XmlWriter w)
   at System.Xml.XmlElement.WriteElementTo(XmlWriter writer, XmlElement e)
   at System.Xml.XmlElement.WriteTo(XmlWriter w)
   at System.Xml.XmlElement.WriteContentTo(XmlWriter w)
   at System.Xml.XmlNode.get_InnerXml()
   at System.Xml.XmlElement.get_InnerXml()
   at MediaPortal.Common.Services.Settings.SettingsFileHandler.GetValue(String entryName, Type type)
   at MediaPortal.Common.Services.Settings.SettingsManager.LoadSettingsObject(Type settingsType, Boolean useGlobaleScope, Boolean useUserScope)
[2018-12-08 11:06:19,014] [2328264] [82       ] [ERROR] - OCVVideoThumbnailer: Exception reading resource '{03dd2da6-4da8-4d3e-9e55-80e3165729a3}:////s*/m*$/m*/R*.mkv' (Text: 'Exception of type 'System.OutOfMemoryException' was thrown.')
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at OpenCvSharp.VectorOfByte.ToArray()
   at OpenCvSharp.Cv2.ImEncode(String ext, InputArray img, Byte[]& buf, Int32[] prms)
   at MediaPortal.Extensions.MetadataExtractors.OCVVideoThumbnailer.OCVVideoThumbnailer.ExtractThumbnailAsync(ILocalFsResourceAccessor lfsra, IDictionary`2 extractedAspectData)
   at MediaPortal.Extensions.MetadataExtractors.OCVVideoThumbnailer.OCVVideoThumbnailer.<TryExtractMetadataAsync>d__8.MoveNext()


thanks

Steps to Reproduce:
1. Start MP2 + Service (or PC)
2. Wait for some minutes
3. See that the Service crashed, MP2 GUI still running
 
Last edited:

Brownard

Development Group
  • Team MediaPortal
  • March 21, 2007
    2,290
    1,872
    Home Country
    United Kingdom United Kingdom
    Here is the ServerError.log excerpt:
    As @HTPCSourcer states, ideally we need a set of debug logs showing the issue. That being said, can you try the attached MediaPortal.Common.dll and see if it fixes it.
    1. Stop the server
    2. Backup the existing MediaportalPortal.Common.dll in C:\Program Files (x86)\Team MediaPortal\MP2-Server\
    3. Extract and replace with the version in the attached zip
    4. Restart the server and try and reproduce
     

    Attachments

    • MediaPortal.Common_OOM.zip
      307.7 KB

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    Attached are new logs with DEBUG enabled and a screenshot of the task manager during the import.
    I noticed that after I enabled verbose logging it took longer for the MP2 service to crash than in the past.

    Bildschirmfoto 2018-12-09 um 10.52.27.png


    As @HTPCSourcer states, ideally we need a set of debug logs showing the issue. That being said, can you try the attached MediaPortal.Common.dll and see if it fixes it.

    I will try this now, the logs attached are with the default 2.2 dll.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    So I tried to restart the importer again with the new DLL you provided @Brownard

    Good news, the import seems to have completed now.
    Bad news MP2 service crashed then later on. I don't know the exact time when it crashed as I left the PC unattended, but when I returned just now it was not running anymore, could not find a hint in the logs myself at what time it crashed.

    In the MP2 Server Error Logs I see a lot of:

    "OCVVideoThumbnailer: Exception reading resource '{03dd2da6-4da8-4d3e-9e55-80e3165729a3}:////*****.mkv' (Text: 'u != 0')
    OpenCvSharp.OpenCVException: u != 0"

    What is that?
     
    Last edited:

    Brownard

    Development Group
  • Team MediaPortal
  • March 21, 2007
    2,290
    1,872
    Home Country
    United Kingdom United Kingdom
    Good news, the import seems to have completed now.
    It looks like some imports were still running when the server stopped, but as you say, no indication as to what stopped it.

    All 15 of your shares are being scanned when the server first starts because automatic share refresh is enabled (Settings->Media Sources->Automatic share refresh). The server tries to import all 15 concurrently, and it looks like there are quite a few new files to process, which leads to the high memory usage. Ideally we need to queue the share refreshes to limit the number that can run concurrently to avoid cases like this.
    "OCVVideoThumbnailer: Exception reading resource '{03dd2da6-4da8-4d3e-9e55-80e3165729a3}:////*****.mkv' (Text: 'u != 0')
    OpenCvSharp.OpenCVException: u != 0"

    What is that?
    It's a library we use to grab a frame from a video file to use as a thumbnail image. It seems to have issues reading some of your videos which may be contributing to the high memory usage.
     

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    Today it crashed again with the new DLL and I see an out of memory exception in the logs attached
     

    Brownard

    Development Group
  • Team MediaPortal
  • March 21, 2007
    2,290
    1,872
    Home Country
    United Kingdom United Kingdom
    Today it crashed again with the new DLL and I see an out of memory exception in the logs attached
    Thanks, I think I've spotted where a lot of the memory usage is. Can you try the attached, you'll need to:
    • Stop the server
    • Backup and replace MediaPortal.Common and MediaPortal.Backend in C:\Program Files (x86)\Team MediaPortal\MP2-Server
    • Backup and replace the FanArtService.* plugins in C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins
    • Restart the server
     

    Attachments

    • Importer_memory_usage_v2.zip
      473 KB

    Terra

    Portal Pro
    August 25, 2009
    230
    57
    Home Country
    Austria Austria
    It ran quite long but then the service crashed again. This time I do not see any out of memory error in the logs, the service was just gone
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    @Terra can you test attached plugin?
    1. Stop MP2-Service
    2. Extract attached archive to to C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins
    3. Remove "OCVVideoThumbnailer" from this folder (move to another folder!)
    I'd like to test if the external thumb creation by ffmpeg.exe works better for your case than the OCV version.
     

    Attachments

    • Plugins.7z
      22.6 MB

    Users who are viewing this thread

    Top Bottom