[fixed] Endless loop in importer with some zip files (1 Viewer)

morpheus_xx

Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    Hi,

    I found this major issue when I tested MP2 on another PC than my own:

    First issue: the importer worker will run in an endless loop with my bad zip file that is attached here, probably it is somehow invalid?
    Code:
    [2012-01-26 18:14:42,694] [27007  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,721] [27034  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,746] [27059  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,772] [27085  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,818] [27131  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,844] [27157  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,870] [27183  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,896] [27209  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,922] [27235  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)
    [2012-01-26 18:14:42,947] [27260  ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias' (11 more resources pending)

    When I tried to reproduce the folder structure in a new zip file this issue does not occure:
    Code:
    [2012-01-26 18:10:56,249] [154473 ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\'
    [2012-01-26 18:10:56,333] [154557 ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /' (12 more resources pending)
    [2012-01-26 18:10:56,382] [154606 ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias/' (12 more resources pending)
    [2012-01-26 18:10:56,430] [154654 ] [Importer ] [INFO ] - ImporterWorker: Importing 'D:\Coding\MP\MP2\Reflexion\mathias.zip > /mathias/Symbole_Test/' (12 more resources pending)

    The most obvious issue is, that in failing case the last "/" is missing
    fails: mathias.zip > /mathias' (11 more resources pending)
    works: mathias.zip > /mathias/' (12 more resources pending)



    2nd issue:
    The ZipResourceAcessor leaves the files opened, so you can't delete them while the server is running.

    The issue is the "Proxy.IncCounter" is call in CTOR of ZipResourceAccessor, and this is called in:
    Code:
        public IResourceAccessor Clone()
        {
          return new ZipResourceAccessor(_zipProvider, _zipProxy, _pathToDirOrFile);
        }
    which is called in ResourcePath:
    Code:
        public IResourceAccessor CreateLocalResourceAccessor()
        {
          IResourceAccessor resourceAccessor;
          if (TryGetFromCache(this, out resourceAccessor))
            return resourceAccessor.Clone();
    .

    Thats why in my test the counter quickly increased to 50, but got only once decreased.
     

    Attachments

    • mathias.zip
      24.8 KB

    Albert

    MP2 Developer
  • Premium Supporter
  • February 18, 2008
    1,297
    1,130
    45
    Freiburg im Breisgau, Germany
    Home Country
    Germany Germany
    AW: Endless loop in importer with some zip files

    2nd issue: Can you give me the stacktrace? It's by design that open resource accessors hold their underlaying files open. But if the usage counter grows to 50 and never decreases, that means some resource accessors aren't properly disposed.
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    • Thread starter
    • Moderator
    • #3
    AW: Endless loop in importer with some zip files

    there is no single stack trace, there are multiple calls to clone and/or constructor. best way to debug this is to place a break point on "IncCounter()" and see the multiple callers...
     

    Albert

    MP2 Developer
  • Premium Supporter
  • February 18, 2008
    1,297
    1,130
    45
    Freiburg im Breisgau, Germany
    Home Country
    Germany Germany
    AW: Endless loop in importer with some zip files

    The problem is not that IncCounter() is called in general!

    The usage pattern is: IncCounter is called when a resources is being accessed, DecCounter is called when it is released. The release of a resource is done when a resource accessor is disposed. If somewhere the dispose is missing, the resource leak happens. That's why I asked you for a concrete stack trace where the counter is increased but not decreased - here, we will probably find a missing call to Dispose().
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    • Thread starter
    • Moderator
    • #5
    AW: Endless loop in importer with some zip files

    I know the intended behaviour, and like I already wrote, the increment is executed in CTOR and by Clone, which both happens quite often.

    Here are the first 3 distinct stacktraces for you, if you don't find the breakpoint shortcut on keyboard :p

    Code:
    >	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor.ZipResourceAccessor(MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider zipProvider = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider}, MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProxy zipProxy = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProxy}, string pathToDirOrFile = "/") Zeile 63	C#
     	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider.TryChainUp(MediaPortal.Common.ResourceAccess.IResourceAccessor potentialBaseResourceAccessor = {MediaPortal.Common.Services.ResourceAccess.LocalFsResourceProvider.LocalFsResourceAccessor}, string path = "/", out MediaPortal.Common.ResourceAccess.IResourceAccessor resultResourceAccessor = null) Zeile 129 + 0x1e Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.ResourceAccess.FileSystemResourceNavigator.TryUnfold(MediaPortal.Common.ResourceAccess.IResourceAccessor fileAccessor = {MediaPortal.Common.Services.ResourceAccess.LocalFsResourceProvider.LocalFsResourceAccessor}, out MediaPortal.Common.ResourceAccess.IResourceAccessor resultResourceAccessor = null) Zeile 123 + 0x15 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.ResourceAccess.FileSystemResourceNavigator.GetChildDirectories(MediaPortal.Common.ResourceAccess.IResourceAccessor directoryAccessor = {MediaPortal.Common.Services.ResourceAccess.LocalFsResourceProvider.LocalFsResourceAccessor}) Zeile 63 + 0xb Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.Process(MediaPortal.Common.Services.MediaManagement.ImportJob importJob = {MediaPortal.Common.Services.MediaManagement.ImportJob}) Zeile 572 + 0x8 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.ImporterLoop() Zeile 228 + 0xb Bytes	C#
     	[Externer Code]

    Code:
    >	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor.ZipResourceAccessor(MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider zipProvider = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider}, MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProxy zipProxy = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProxy}, string pathToDirOrFile = "/mathias/") Zeile 63	C#
     	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor.GetChildDirectories.AnonymousMethod__a(ICSharpCode.SharpZipLib.Zip.ZipEntry directoryEntry = {ICSharpCode.SharpZipLib.Zip.ZipEntry}) Zeile 304 + 0x48 Bytes	C#
     	[Externer Code]	
     	MediaPortal.Utilities.dll!MediaPortal.Utilities.CollectionUtils.AddAll<MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor,MediaPortal.Common.ResourceAccess.IFileSystemResourceAccessor>(System.Collections.Generic.ICollection<MediaPortal.Common.ResourceAccess.IFileSystemResourceAccessor> target = Count = 0, System.Collections.Generic.IEnumerable<MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor> source = {System.Linq.Enumerable.WhereSelectListIterator<ICSharpCode.SharpZipLib.Zip.ZipEntry,MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor>}) Zeile 73 + 0x11a Bytes	C#
     	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor.GetChildDirectories() Zeile 303 + 0xe8 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.ResourceAccess.FileSystemResourceNavigator.GetChildDirectories(MediaPortal.Common.ResourceAccess.IResourceAccessor directoryAccessor = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor}) Zeile 56 + 0x9 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.Process(MediaPortal.Common.Services.MediaManagement.ImportJob importJob = {MediaPortal.Common.Services.MediaManagement.ImportJob}) Zeile 572 + 0x8 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.ImporterLoop() Zeile 228 + 0xb Bytes	C#
     	[Externer Code]


    Code:
    >	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor.ZipResourceAccessor(MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider zipProvider = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProvider}, MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProxy zipProxy = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceProxy}, string pathToDirOrFile = "/mathias/Symbole_Alwis/symbol1_obj_40.jpg") Zeile 63	C#
     	ZipResourceProvider.dll!MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor.Clone() Zeile 260 + 0x2d Bytes	C#
     	ImageMetadataExtractor.dll!MediaPortal.Extensions.MetadataExtractors.ImageMetadataExtractor.ImageMetadataExtractor.TryExtractMetadata(MediaPortal.Common.ResourceAccess.IResourceAccessor mediaItemAccessor = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor}, System.Collections.Generic.IDictionary<System.Guid,MediaPortal.Common.MediaManagement.MediaItemAspect> extractedAspectData = Count = 4, bool forceQuickMode = false) Zeile 172 + 0x9 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.MediaAccessor.ExtractMetadata(MediaPortal.Common.ResourceAccess.IResourceAccessor mediaItemAccessor = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor}, System.Collections.Generic.IEnumerable<MediaPortal.Common.MediaManagement.IMetadataExtractor> metadataExtractors = Count = 1, bool forceQuickMode = false) Zeile 357 + 0x14 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.ImportResource(MediaPortal.Common.ResourceAccess.IResourceAccessor mediaItemAccessor = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor}, System.Guid parentDirectoryId = {System.Guid}, System.Collections.Generic.ICollection<MediaPortal.Common.MediaManagement.IMetadataExtractor> metadataExtractors = Count = 1, MediaPortal.Common.MediaManagement.IImportResultHandler resultHandler = {MediaPortal.Backend.Services.MediaLibrary.MediaLibrary.ImportResultHandler}, MediaPortal.Common.ResourceAccess.IMediaAccessor mediaAccessor = {MediaPortal.Common.Services.MediaManagement.MediaAccessor}) Zeile 292 + 0x11 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.ImportDirectory(MediaPortal.Common.Services.MediaManagement.ImportJob importJob = {MediaPortal.Common.Services.MediaManagement.ImportJob}, System.Guid parentDirectoryId = {System.Guid}, MediaPortal.Common.ResourceAccess.IFileSystemResourceAccessor directoryAccessor = {MediaPortal.Extensions.ResourceProviders.ZipResourceProvider.ZipResourceAccessor}, System.Collections.Generic.ICollection<MediaPortal.Common.MediaManagement.IMetadataExtractor> metadataExtractors = Count = 1, MediaPortal.Common.MediaManagement.IMediaBrowsing mediaBrowsing = {MediaPortal.Backend.Services.MediaLibrary.MediaLibrary.MediaBrowsingCallback}, MediaPortal.Common.MediaManagement.IImportResultHandler resultHandler = {MediaPortal.Backend.Services.MediaLibrary.MediaLibrary.ImportResultHandler}, MediaPortal.Common.ResourceAccess.IMediaAccessor mediaAccessor = {MediaPortal.Common.Services.MediaManagement.MediaAccessor}) Zeile 428 + 0x2e Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.Process(MediaPortal.Common.Services.MediaManagement.ImportJob importJob = {MediaPortal.Common.Services.MediaManagement.ImportJob}) Zeile 565 + 0x65 Bytes	C#
     	MediaPortal.Common.dll!MediaPortal.Common.Services.MediaManagement.ImporterWorker.ImporterLoop() Zeile 228 + 0xb Bytes	C#
     	[Externer Code]
     

    Albert

    MP2 Developer
  • Premium Supporter
  • February 18, 2008
    1,297
    1,130
    45
    Freiburg im Breisgau, Germany
    Home Country
    Germany Germany
    AW: Endless loop in importer with some zip files

    Ok, so you see in
    MediaPortal.Common.dll!MediaPortal.Common.ResourceAccess.FileSystemResourceNavigator.TryUnfold(MediaPortal.Common.ResourceAccess.IResourceAccessor fileAccessor = {MediaPortal.Common.Services.ResourceAccess.LocalFsResourceProvider.LocalFsResourceAccessor}, out MediaPortal.Common.ResourceAccess.IResourceAccessor resultResourceAccessor = null) Zeile 123
    that the resource accessor isn't correctly disposed. So here is the problem.

    I'll fix it.
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    • Thread starter
    • Moderator
    • #8
    AW: Endless loop in importer with some zip files

    first issue is confirmed to be fixed!

    2nd seems not yet ok, file is still locked. I will check this more
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    • Thread starter
    • Moderator
    • #9
    I made a new test and both issue seem to be fixed now!

    :D again
     

    Users who are viewing this thread

    Top Bottom