MP2 Audio Scan Tests (1 Viewer)

breese

Retired Team Member
  • Premium Supporter
  • July 11, 2011
    3,902
    770
    65
    Arlington Heights, Illinois
    Home Country
    United States of America United States of America
    While I chomp at the bit waiting to get my hands on the new SQLite code (Sorry MJGraf, I could not resist), I did some testing of the scan capabilities of MP2 for Audio.
    Some of the things Morpheus_xx expressed in https://forum.team-mediaportal.com/threads/sqlitedatabase-plugin-for-mp2.118535/page-17 Got me thinking, again….
    Starting with a clean install with SQLite: I configured the client to use the Titanium skin, changed the menu options to allow minimize of MP2 client, and did the initial scan of my music folder (f:\Music)
    Note: I expanded this music folder to include multiple folders of Albums and some of these Albums have multiple CDXY in them. For this test I primarily used f:\music\verious\80’s Night
    This album has 16 CD folders labeled CD01 thru CD16 in it. Total there are 299 items within all the CD’s.
    While the initial scan was running , Resource Manager showed all the directories and files being accessed.
    When the scan was completed (about 3 minutes), I went into MP2 Client and went into Audio.
    The first level of albums was present but, when I opened 80’s Night it listed all 299 songs instead of the CD albums / folders.
    Using Explorer, I went into f:\music\verious and deleted the entire 80’s Night album
    Next I went back into Shares and initiated a ReImport
    This time Resource Manager did not show any folder or file access.
    When the scan was done, the client log showed the scan started but there were 2 errors listed
    Thread23 and Thread87 had “Failed to update currently importing shares”
    The server log had no errors, had a complete listing of all directories and files and did Not show any scan for f:\music\verious\80’s Night
    Going back into MP2 (was in minimize mode), I pulled up Audio and 80’s Night was not listed.
    As a result of this test a few things come to mind.
    1) What are the Thread Issues registered to the client
    2) Why does the scan stop at a level and not allow Subfolders to be used resulting in large media lists
    3) How does the Reimport manage to complete the task without showing any file access within Resource Manager
    Over all, the test results are both good (import works, reimport works to discover missing / deleted folders and files and all has great access times seeing the large amount of data being read from the folder and file structure) and bad (the 3 key issues).
    On another note: Is there a count of folders and or files being kept in the DB or someplace else?
    It might be nice if the Shares Import page process had some type of visual update telling the user / operator what the current status is.
    This might keep someone from Killing process’s or powering off the server / client because they think the system is not functioning.

    The server log zip also includes the SQLite DB files incase someone cares to look thru it. I would be interested to know if the album 80's nights has any residual still in the DB file.

    As always, logs and other documents are included and if you have any questions, please let me know.

    P.s. Is there a location I can create a personal test blog? I would like to keep a list of what I have tested and what tests I am going or designing to do.
     

    Attachments

    • Verious Dir Info.zip
      480 KB

    Lehmden

    Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,552
    3,934
    Lehmden
    Home Country
    Germany Germany
    Hi.
    not allow Subfolders to be used
    Same as with series. No season subfolders possible. Here you only get rid of this limitation by using .mkv with proper tagging. Then most of the time (95-99%) the media items are recognized correct. I don't know exactly how it is with tagging in Music, as I first want to have my series perfectly scanned into DB. That's what I'm using most atm in MP2. So I only had some test audio files scanned before...
     

    breese

    Retired Team Member
  • Premium Supporter
  • July 11, 2011
    3,902
    770
    65
    Arlington Heights, Illinois
    Home Country
    United States of America United States of America
    Thanks Lehmden.
    As I move forward into testing MP2, I will be looking at the series scans. Right now I have been working with the Audio / Musoc section because of the size of my collection and testing the new SQLite functions and speed. So far it is very impressive and I expect it to get even better...
    Thanks for the info on Series and I will be sure to watch for it.
     

    MJGraf

    Retired Team Member
  • Premium Supporter
  • January 13, 2006
    2,478
    1,385
    How does the Reimport manage to complete the task without showing any file access within Resource Manager
    Let's start with the easy one: It compares the last modified dates of the files in the file system with the last import date of such file. If the file in the file system wasn't touched since the last import, it doesn't even open the file - which is why you don't see any file i/o :D

    2) Why does the scan stop at a level and not allow Subfolders to be used resulting in large media lists
    Well, the scan does not stop at a certain level, it obviously went into each subdirectory and imported all the audio files...
    The particular problem you describe is caused by one thing not yet fully implemented: One album may have multiple "media" - in this case CDs. While our AudioAspect already has properties for "DiscID" and "NumDiscs", your AudioMetadataExtractor does not fill these properties, yet (have a look here, if you are interessted in what exactly is imported - I think it should be more or less "understandable" for non-devs although it is c#: https://github.com/MediaPortal/Medi...adataExtractor/AudioMetadataExtractor.cs#L352). If this was implemented (which isn't too complicated) in your case NumDiscs should be 16 for all the audio files and DiscID should be 1, 2, ..., 16 depending on what disc the respective title is.
    But even if we had this imported, it would require our MP2 Client media plugin to group albums by DiscID - at least when NumDiscs is greater than 1.I think this implementation is also missing - and since this involves the MP2 Client side and in particular skinning, I unfortunately can't help with that...

    But before implementing more and more into the AudioMetadataExtractor, maybe this is a good point to start a structured discussion of
    • which sources should finally be used to import data for audio files? Besides directory structure/file name and tags I'm a big fan to get more data from MusicBrainz - but should this go into the AudioAspect? Should there be a separate MusicBrainzAspect?
    • We need an order of precedence regarding the sources
    • And of course we should be clear what properties we import
    And this again brings us back to one problem, which (i think) chefkoch has already mentioned in the past: During an import, some information may be 100% certain (I would assume, info from tags is 100% certain - if not, the user should retag its files). But other information may not be 100% accurate (e.g. title names derived from file names). In this situation we may use online sources to make the info we have more accurate - but this may require user interaction (e.g. we guess an album name from the path name and then try to look it up in MusicBrainz. As a result, we may get three potential hits in MusicBrainz. The user must first choose, which hit is the right one, before we can import additional data. And therefore we need amendments to MP2 Client to make this possible...

    I thought about doing something in this area for quite some time. Maybe I should take this as next project after SQLite imporvements :D But I need help for the client side... And I should make up kind of a chart to show how exactly the import should work...

    One thing we also have to take into account here is configurability. Currently we have e.g. three hard-coded Regular Expressions in the code for guessing titles, etc. from path and file name. We should at least make these settings for now. Of course no "regular" user would touch regular expressions, but this would make it at least possible to later on offer some pre defined reg-exps for certain folder and sorting schemes.

    Is there a count of folders and or files being kept in the DB or someplace else?
    Nope, not really. You could issue a "select count (*) from M_MediaItem" to the database with some kind of SQLite database frontend. This returns you the total number of MediaItems in the library. But this includes everything: videos, movies, series, pictures and audio files and I think it even includes folders (not 100% sure about that...).

    It might be nice if the Shares Import page process had some type of visual update telling the user / operator what the current status is.
    That's already more or less the case for local shares (at least you get a notification when the import has completed), but not (yet) for global shares. But you're right, would be nice to have...

    Is there a location I can create a personal test blog
    Good idea - your personal wiki space is a perfect place for this. Log on to the wiki and you should see "My Page" in the upper left.
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    Browsing the MediaLibrary is already improved a lot in the new branch, it will be merged into dev soon. There it is possible to configure the filters as you like: I.e. Artist-Album-Disc*-Title

    *) not yet available.

    What is not possible yet are custom views depending on result counts. This would allow us to show different screens: I.e. to show album only if there are more than one, otherwise show all titles directly. (Same would be useful for series and optional season grouping)

    Edit: and in this branch you can browse by folder structure :)
     

    chefkoch

    Retired Team Member
  • Premium Supporter
  • October 5, 2004
    3,129
    1,634
    Dresden / Munich / Maastricht
    Home Country
    Germany Germany
    • which sources should finally be used to import data for audio files? Besides directory structure/file name and tags I'm a big fan to get more data from MusicBrainz - but should this go into the AudioAspect? Should there be a separate MusicBrainzAspect?
    • We need an order of precedence regarding the sources
    • And of course we should be clear what properties we import
    And this again brings us back to one problem, which (i think) chefkoch has already mentioned in the past: During an import, some information may be 100% certain (I would assume, info from tags is 100% certain - if not, the user should retag its files). But other information may not be 100% accurate (e.g. title names derived from file names). In this situation we may use online sources to make the info we have more accurate - but this may require user interaction (e.g. we guess an album name from the path name and then try to look it up in MusicBrainz. As a result, we may get three potential hits in MusicBrainz. The user must first choose, which hit is the right one, before we can import additional data. And therefore we need amendments to MP2 Client to make this possible...
    I 100% agree, there are many things that can and should be improved and I think it is not worth to fix two use cases now and then later again when new users tested MP2 discussing and working on their use cases again.
    We should have enough experience about metadata from filenames (file/folder), file tags (ID3, ...), additional files (nfo,...), ids for unique identification (imdbid, acoustid, puid, mbid, tmdbid, tvdbid, ...) etc... and online resources from MP1, xbmc and the current MP2 implementations. We all agree that the import must be split in multiple runs with different objectives, and these steps also should be configurable and some might need additional user interaction.

    http://wiki.team-mediaportal.com/2_MEDIAPORTAL_2/8_Contribute/Development/3_Concepts/Metadata_system and it's subpages, especially
    http://wiki.team-mediaportal.com/2_...3_Concepts/Metadata_system/Planning_Extension already contains an overview. Many of the use cases could be written down in one or two sentences, but imo we need to organize and classify them which part (mentioned above) they belong to.

    This is not only about Music, it's similar for movies and/or series.

    Other things are not related to the import but more the filters and browsing the items: i.e.:
    Same as with series. No season subfolders possible.

    a) Either the season index is read by MP2 from tag or filename/foldername/filepath or not, but this is a importer related issue and topic.

    b) If it is about browsing the series by seriesname and next season number (folder/item) then it is not about the importer (assuming the import of the index is working), it is about the views, filters, groupings, sortings and their default definition and the user experience to switch between different ones.

    c) If it is about browsing the series (or other media) by folder structure like within the windows explorer, we need...
    not sure if we now have one in latest dev, EDIT already answered:
    Browsing the MediaLibrary is already improved a lot in the new branch, it will be merged into dev soon. There it is possible to configure the filters as you like: I.e. Artist-Album-Disc*-Title
    .....
    Edit: and in this branch you can browse by folder structure :)

    ...a "view/filter" which provides the folder structure like it is on the filesystem, means the paths to the media items are split by '\' and each folder is a "directory/folder" item in MP2, but still showing and presenting the information from the database. These could either be infos read only from filenames or filled by metadata from tags or additional online resources. (depends on the user configuration)
    The difference is only that here the media is browsed by path (-elements).


    There have been and there might be again requests to only browse the files without importing anything. I am again this, because the media plugin is not design for this usage. The user should be able to configure (and block) the usage of online resources for some shares or media types, but even the filename of an item should be imported as title if there is nothing more to import.


    If there are still use cases this might be useful kind of an "explorer" or "file manager" plugin might be more sufficient to resolve these ones.
    An potential argument to speed up browsing the media items without an import before, is only valid as long as the importer logic is not optimized and structured/splitted into several runs/stages.


    This is just my opinion. If there are good arguments for other decisions, I am happy to change my mind. (hopefully my points can be understood.) ;)
     
    Last edited:

    Lehmden

    Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,552
    3,934
    Lehmden
    Home Country
    Germany Germany
    Hi.
    The user should be able to configure (and block) the usage of online resources for some shares or media types, but even the filename of an item should be imported as title if there is nothing more to import.
    Think about "temporary" files, as example insert an SD card from a guests camera to quickly watch the latest pictures on it. Later the SD card will be taken away so those files can't ever be accessed again. But they stay in DB until eternity.... And it's a matter of speed. In MP1 or XBMC I plug the SD card and immediately I can watch the pictures. Thumbnails are generated on the fly while browsing and watching the pictures. In MP2 first I need to trigger the import, wait until all thumbnails are generated and finally (after half an hour or so as MP2 generate thumbs for all pictures on the card...) I can start watching the pictures. But in meantime my guests had left, taken their SD card with them so there is no further need for those pictures, database entries and thumbnails at all. All stuff that would stay (for some time at least) should be in DB, but all that is "one time only" is nothing that ever should touch the DB. It's just garbage and is letting DB size explode.

    To be honest, I rather would use MP1 or MP2 in this case. When it comes to pictures I stuck with XBMC. MP1 pictures plugin is terrible, MP2 pictures plugin is incomplete and the need to add all stuff to DB first is another point not to use it... But I really want to replace MP1 and XBMC completely by MP2 some day, at least on all Windows- Devices.
     

    breese

    Retired Team Member
  • Premium Supporter
  • July 11, 2011
    3,902
    770
    65
    Arlington Heights, Illinois
    Home Country
    United States of America United States of America
    It compares the last modified dates of the files in the file system with the last import date of such file.
    I had a feeling that was the case. Nice to know for fact.

    your AudioMetadataExtractor does not fill these properties, yet (have a look here, if you are interessted in what exactly is imported
    Very interseting read... I do understand it and I see a few items I did not even know about... like the added / in the Authors if there is more than one.
    Has actually got me (or my tester brain) working on a few tet ideas... Will have to see what I come up with.... :)

    I would assume, info from tags is 100% certain - if not, the user should retag its files
    I have had this collection for Years. Some years back I used (and continue to use) Tag&Rename. Its good and its bad. I have had issues where my other media server canot import the song titles yet it always picks up the folder.jpg
    Speaking of.... I mentioned just recently I have thumbnails turned off for large area's like this. another reasion is because of MS infor like these...
    Sep 2012
    http://social.technet.microsoft.com...ery-old-problem-still-without?forum=w7itproui
    Discussion goes right thru to June 2012
    http://social.technet.microsoft.com...t-recently-viewed-folder?forum=w7itprogeneral

    That's already more or less the case for local shares (at least you get a notification when the import has completed), but not (yet) for global shares. But you're right, would be nice to have...
    Interseting.... I have yet to see anything but Importing.... Local or Global Shares... If the scan goes into hours, the Importing does not go away on its own, you have to switch to a different screen and back again. I have reported this in one other thread... dont remember when...

    As for the personal blog, thanks. I will be looking into that!!

    EDIT: Wow... In the time it took for me to think this post thru, there were lots of postings going on and I like it. While some have stated they appreasiate my testings, it is also nice to see I can get others thinking about some of the same things that are rattling around in my head.... :)
     
    Last edited:

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    Think about "temporary" files, as example insert an SD card from a guests camera to quickly watch the latest pictures on it. Later the SD card will be taken away so those files can't ever be accessed again. But they stay in DB until eternity....
    I think (but am not 100% sure ;)) this is wrong: removable media is added to "Browse Media" section on the fly, without importing them into database. But there is an issue with the kind of detection, what primary "media type" the disc contains: it scans all files on it, not only a few, to detect if it is primary audio, video or image.

    There is already a long standing issue filed under https://issues.team-mediaportal.com/browse/MP2-158.
     

    Users who are viewing this thread

    Top Bottom