[fixed] - MP2 Importer Exception of type 'System.OutOfMemoryException' was thrown.

Discussion in '2.2 Archive' started by Terra, December 8, 2018.

  1. Terra

    Terra Portal Pro

    Joined:
    August 25, 2009
    Messages:
    230
    Likes Received:
    36
    Ratings:
    +55 / 1
    Home Country:
    Austria Austria
    Show System Specs
    MediaPortal 2 Version: 2.2 Final

    Description
    Hi guys,

    So I tried MP2 again after some time and I have the issue that the MP2 Service crashes while importing my media assets.
    I now restarted it at least 10 times over the last 3 days but its really annoying, not sure if it will finish any time...

    I have 8GB of RAM and 312GB is free on the SSD.

    Version: 2.2.1811.11777
    Source: MP2.2_Final-11cc07

    Here is the ServerError.log excerpt:

    Code (Text):
    1. [2018-12-08 11:06:14,355] [2323606] [130      ] [INFO ] - ImporterWorker: 51 % completed (4 ImportJob(s), in total 11057 of 21613 so far identified resources processed)
    2. [2018-12-08 11:06:18,623] [2327873] [111      ] [ERROR] - Unable to query
    3. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
    4.    at System.Collections.Generic.List`1..ctor(Int32 capacity)
    5.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledFilter.CompileStatementParts(MIA_Management miaManagement, IFilter filter, IFilter subqueryFilter, Namespace ns, BindVarNamespace bvNamespace, ICollection`1 requiredMIATypes, String outerMIIDJoinVariable, ICollection`1 tableJoins, IList`1 resultParts, IList`1 resultBindVars)
    6.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledFilter..ctor(MIA_Management miaManagement, IFilter filter, IFilter subqueryFilter, Namespace ns, BindVarNamespace bvNamespace, String outerMIIDJoinVariable, ICollection`1 tableJoins)
    7.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.MainQueryBuilder.CreateCompiledFilter(Namespace ns, BindVarNamespace bvNamespace, String outerMIIDJoinVariable, IList`1 tableJoins)
    8.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.MainQueryBuilder.GenerateSqlStatement(Boolean groupByValues, IDictionary`2 miamAliases, String& mediaItemIdOrGroupSizeAlias, IDictionary`2& attributeAliases, String& statementStr, IList`1& bindVars)
    9.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledMediaItemQuery.AddMultipleMIAResults(ISQLDatabase database, ITransaction transaction, MultipleMediaItemAspectMetadata miam, IDictionary`2 attributes, MainQueryBuilder builder, IDictionary`2 multipleMiaValues)
    10.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledMediaItemQuery.AddMultipleMIAs(ISQLDatabase database, ITransaction transaction, IEnumerable`1 selectedMIAs, IList`1 ids, IDictionary`2 multipleMiaValues)
    11.    at MediaPortal.Backend.Services.MediaLibrary.QueryEngine.CompiledMediaItemQuery.Query(ISQLDatabase database, ITransaction transaction, Boolean singleMode)
    12. [2018-12-08 11:06:21,324] [2330574] [87       ] [INFO ] - ImporterWorker.ImportJob(#5).DirectorySaveBlock: MediaLibrary disconnected. Requesting suspension...
    13. [2018-12-08 11:06:21,328] [2330578] [145      ] [INFO ] - ImporterWorker.ImportJob(#4): Suspended
    14. [2018-12-08 11:06:21,329] [2330579] [145      ] [INFO ] - ImporterWorker.ImportJob(#5): Suspended
    15. [2018-12-08 11:06:21,330] [2330580] [145      ] [INFO ] - ImporterWorker.ImportJob(#6): Suspended
    16. [2018-12-08 11:06:21,330] [2330580] [145      ] [INFO ] - ImporterWorker.ImportJob(#2): Suspended
    17. [2018-12-08 11:06:21,331] [2330581] [145      ] [INFO ] - ImporterWorker: Suspended (4 ImportJobs pending)
    18. [2018-12-08 11:06:20,933] [2330183] [67       ] [ERROR] - SettingsManager: Error setting property 'PendingFanArtActions' in settings of type 'FanArtActionSettings'
    19. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
    20.    at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
    21.    at System.Text.StringBuilder.Append(Char* value, Int32 valueCount)
    22.    at System.Text.StringBuilder.AppendHelper(String value)
    23.    at System.Text.StringBuilder.Append(String value)
    24.    at System.IO.StringWriter.Write(String value)
    25.    at System.Xml.XmlTextWriter.WriteStartElement(String prefix, String localName, String ns)
    26.    at System.Xml.XmlDOMTextWriter.WriteStartElement(String prefix, String localName, String ns)
    27.    at System.Xml.XmlElement.WriteStartElement(XmlWriter w)
    28.    at System.Xml.XmlElement.WriteElementTo(XmlWriter writer, XmlElement e)
    29.    at System.Xml.XmlElement.WriteTo(XmlWriter w)
    30.    at System.Xml.XmlElement.WriteContentTo(XmlWriter w)
    31.    at System.Xml.XmlNode.get_InnerXml()
    32.    at System.Xml.XmlElement.get_InnerXml()
    33.    at MediaPortal.Common.Services.Settings.SettingsFileHandler.GetValue(String entryName, Type type)
    34.    at MediaPortal.Common.Services.Settings.SettingsManager.LoadSettingsObject(Type settingsType, Boolean useGlobaleScope, Boolean useUserScope)
    35. [2018-12-08 11:06:19,014] [2328264] [82       ] [ERROR] - OCVVideoThumbnailer: Exception reading resource '{03dd2da6-4da8-4d3e-9e55-80e3165729a3}:////s*/m*$/m*/R*.mkv' (Text: 'Exception of type 'System.OutOfMemoryException' was thrown.')
    36. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
    37.    at OpenCvSharp.VectorOfByte.ToArray()
    38.    at OpenCvSharp.Cv2.ImEncode(String ext, InputArray img, Byte[]& buf, Int32[] prms)
    39.    at MediaPortal.Extensions.MetadataExtractors.OCVVideoThumbnailer.OCVVideoThumbnailer.ExtractThumbnailAsync(ILocalFsResourceAccessor lfsra, IDictionary`2 extractedAspectData)
    40.    at MediaPortal.Extensions.MetadataExtractors.OCVVideoThumbnailer.OCVVideoThumbnailer.<TryExtractMetadataAsync>d__8.MoveNext()
    41.  

    thanks

    Steps to Reproduce:
    1. Start MP2 + Service (or PC)
    2. Wait for some minutes
    3. See that the Service crashed, MP2 GUI still running


     
    Last edited: December 9, 2018
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. HTPCSourcer
    • Team MediaPortal
    • Administrator

    HTPCSourcer MP2 Product Manager

    Joined:
    May 16, 2008
    Messages:
    10,745
    Likes Received:
    1,125
    Gender:
    Male
    Ratings:
    +2,408 / 22
    Home Country:
    Germany Germany
    Show System Specs
    Your logs don't show any errors.;)

    Please set your system to DEBUG level, detailed explanations here:
    Reproduce the issue and then post a new set of logs.
     
  4. Brownard
    • Team MediaPortal

    Brownard Development Group

    Joined:
    March 21, 2007
    Messages:
    1,914
    Likes Received:
    787
    Gender:
    Male
    Ratings:
    +1,320 / 1
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    As @HTPCSourcer states, ideally we need a set of debug logs showing the issue. That being said, can you try the attached MediaPortal.Common.dll and see if it fixes it.
    1. Stop the server
    2. Backup the existing MediaportalPortal.Common.dll in C:\Program Files (x86)\Team MediaPortal\MP2-Server\
    3. Extract and replace with the version in the attached zip
    4. Restart the server and try and reproduce
     

    Attached Files:

  5. Terra

    Terra Portal Pro

    Joined:
    August 25, 2009
    Messages:
    230
    Likes Received:
    36
    Ratings:
    +55 / 1
    Home Country:
    Austria Austria
    Show System Specs
    Attached are new logs with DEBUG enabled and a screenshot of the task manager during the import.
    I noticed that after I enabled verbose logging it took longer for the MP2 service to crash than in the past.

    Bildschirmfoto 2018-12-09 um 10.52.27.png

    I will try this now, the logs attached are with the default 2.2 dll.
     
  6. Terra

    Terra Portal Pro

    Joined:
    August 25, 2009
    Messages:
    230
    Likes Received:
    36
    Ratings:
    +55 / 1
    Home Country:
    Austria Austria
    Show System Specs
    So I tried to restart the importer again with the new DLL you provided @Brownard

    Good news, the import seems to have completed now.
    Bad news MP2 service crashed then later on. I don't know the exact time when it crashed as I left the PC unattended, but when I returned just now it was not running anymore, could not find a hint in the logs myself at what time it crashed.

    In the MP2 Server Error Logs I see a lot of:

    "OCVVideoThumbnailer: Exception reading resource '{03dd2da6-4da8-4d3e-9e55-80e3165729a3}:////*****.mkv' (Text: 'u != 0')
    OpenCvSharp.OpenCVException: u != 0"

    What is that?
     
    Last edited: December 9, 2018
  7. Brownard
    • Team MediaPortal

    Brownard Development Group

    Joined:
    March 21, 2007
    Messages:
    1,914
    Likes Received:
    787
    Gender:
    Male
    Ratings:
    +1,320 / 1
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    It looks like some imports were still running when the server stopped, but as you say, no indication as to what stopped it.

    All 15 of your shares are being scanned when the server first starts because automatic share refresh is enabled (Settings->Media Sources->Automatic share refresh). The server tries to import all 15 concurrently, and it looks like there are quite a few new files to process, which leads to the high memory usage. Ideally we need to queue the share refreshes to limit the number that can run concurrently to avoid cases like this.
    It's a library we use to grab a frame from a video file to use as a thumbnail image. It seems to have issues reading some of your videos which may be contributing to the high memory usage.
     
  8. Terra

    Terra Portal Pro

    Joined:
    August 25, 2009
    Messages:
    230
    Likes Received:
    36
    Ratings:
    +55 / 1
    Home Country:
    Austria Austria
    Show System Specs
    Today it crashed again with the new DLL and I see an out of memory exception in the logs attached
     
  9. Brownard
    • Team MediaPortal

    Brownard Development Group

    Joined:
    March 21, 2007
    Messages:
    1,914
    Likes Received:
    787
    Gender:
    Male
    Ratings:
    +1,320 / 1
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    Thanks, I think I've spotted where a lot of the memory usage is. Can you try the attached, you'll need to:
    • Stop the server
    • Backup and replace MediaPortal.Common and MediaPortal.Backend in C:\Program Files (x86)\Team MediaPortal\MP2-Server
    • Backup and replace the FanArtService.* plugins in C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins
    • Restart the server
     

    Attached Files:

  10. Terra

    Terra Portal Pro

    Joined:
    August 25, 2009
    Messages:
    230
    Likes Received:
    36
    Ratings:
    +55 / 1
    Home Country:
    Austria Austria
    Show System Specs
    It ran quite long but then the service crashed again. This time I do not see any out of memory error in the logs, the service was just gone
     
  11. morpheus_xx
    • Team MediaPortal

    morpheus_xx Lead Dev MP2

    Joined:
    March 24, 2007
    Messages:
    11,065
    Likes Received:
    4,760
    Ratings:
    +6,813 / 11
    Home Country:
    Germany Germany
    Show System Specs
    @Terra can you test attached plugin?
    1. Stop MP2-Service
    2. Extract attached archive to to C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins
    3. Remove "OCVVideoThumbnailer" from this folder (move to another folder!)
    I'd like to test if the external thumb creation by ffmpeg.exe works better for your case than the OCV version.
     

    Attached Files:

    • Plugins.7z
      File size:
      22.6 MB
      Uploaded:
      December 10, 2018
      Views:
      29
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!