this view could not be built (2 Viewers)

[MJGraf]

please let the Unit tests run for this part (maybe some fail already )

Tested this morning without any modifications.
Series tests all pass.
Movies tests pass except for "Rango". But I can reproduce the "error" manually (and it is no error on our side). If you search for a movie called "Rango" on their site, you get 7 results (same if you use the API - tested as well). So there is no unique match and therefore the test fails.
If we add the year (2011) to our test, Rango passes. So while I'm at it, I would add the year 2011 to our tests...

 

[resa]

Nice discussion tough

For all the developers:

I wan't to add this suggestion, even if it slides a bit offtopic here right now: The ability to read from AND on-demand generating into .nfo files on a per-videofile base (i.e. videofilename.mkv -> videofilename.nfo on same physical location) should be generally included. ALWAYS. And into all software that handles media. Relevant software is supporting this already: MePo1's Moving-Picture (scraper: read from local nfo in addition with external nfo-export plugin), XBMC/Kodi where this "standard" seems to come from, Ember-Media, UniversalMediaServer (ex PlaystationMediaServer) is currently working on this... to name some major ones.

Because this is not only an transparent and the only fail-safe way to backup the scraped data (which in most cases was needed to be manually matched by the user for relevant amounts of items or is manually altered in detail for several reasons), even if the active used dbase of the software fails completely without having backup handy or the user is accidentially over-scraping items with wrong data into the active used dbase, and after all, making it easy to re-import the correct data from the inactive nfo-files in such cases or simply when moving numerous videofiles to different locations etc. too (i.e. instead manually alter the location of any single item -> clear the database from unavailable items and then import data from nfos at new location).

It is also sort of an easy to use universal exchange format for video-metadata, to hassle-free moving media-librarys between different software or even to use it by several tools at the same time, no matter if SQLite, MySQL or whatever is used by an specific project and no matter how often the developers alters table-structures within those active dbases or other things in the softwares at all, what makes it necessary to rewrite all the dbase-importers on all the other projects etc.

It seems that it is not necessary to take care about all the tags possibly used by any and all other projects in those .nfo-xml files, but the basic ones (i.e. title, sorting title, collection, imdb-id, description, watch-count or any other one which makes sense or the users often likes to intentionally altering manually) should at least be supoorted.

To go further: I think, all the developers of media handling tools should work together more closely in order to develop this as "industrial standard" for backup and exchange the metadata

Just my 2 cents.

Feel free to move this to the whish-list or whatever

Thanks for reading and best regards.

 

[JSchuricht]

Thanks for all the help. It has been a busy last few days so I am just now catching up with the changes suggested. Anyways a full import is running now so hopefully I can post results later today.

 

[MJGraf]

Hi resa,

on-demand generating into .nfo files on a per-videofile base (i.e. videofilename.mkv -> videofilename.nfo on same physical location) should be generally included

In general, I tend to agree. But there are a few "buts":

• The most important word in your statement for me is "on-demand". I would never want any Media Application to automatically generate nfo-files in my media collections (or even worse, let it automatically overwrite any nfo-files that were already there before).
• The operation of generating nfo-files should ideally be a one-time task. Once you have generated an nfo-files from locally or online available information, you may want to manually edit the nfo-file. But after you have manually edited the nfo-file, you will probably not want your edits to be overwritten by a newly generated nfo-file. As a result, I don't think the operation of generating nfo-files should be part of a Media Center Application such as MP2, which is supposed to be used with a remote control. Generating nfo-files is closely related to making edits to them and the latter one is definitely a task for a separate application (of which there are already a lot: MP2Edit, TMM, EMM, etc.).
• Nevertheless, I think MP2 should support this - but IMHO not from the MP2-Client application, but via a Web Interface. I'm still dreaming of a web server built into MP2-Server with a web-frontend that you can just open with your browser. On that web-frontend, you should be able to do a lot of things such as Configuring MP2-Server, maybe watch or listen to the media in your MediaLibrary via streaming, but also just browse through your MediaLibrary and make changes to the metadata. Once you are fine with the metadata that was either previously fetched by our MetadataExtractors or manually edited in the MediaLibrary, I would imagine that on right-click on a MediaItem (or for batches on a parent folder), you get the option wo "generate a nfo-file". Only downside is: There is still a long way to go until we are there... But the good news is: I'm willing to go that way - it may just take a while...

To go further: I think, all the developers of media handling tools should work together more closely in order to develop this as "industrial standard" for backup and exchange the metadata

Oh and you can believe me, there is nothing in the respect that I would support more. But I wouldn't hold my breath...
I have no clue who first "created" nfo-files - it might have been Kodi. The issue with nfo-files is that they, as they are now, look like they are based on another standard. But in fact they are not. As an example: Nfo-files look like they are XML-files. But actually they are not. The "standard" for a nfo-file belonging to a video file with multiple episodes of a series is that the nfo-file has multiple <episodedetails> root elements. The xml standard does not allow multiple root elements, which is why any xml-parser out there throws an error when reading such files (and why I had to write our own XmlNfoReader class...).
Another example is what we are dealing with in this thread: Kodi allows nfo-files just with a link to the imdb webpage of a movie in it. This has nothing to do with xml. And even worse, AFAIK, Kodi also allows part of the nfo-file to contain (kind of) xml and at the end have a link to imdb so that you can take metadata from imdb first and "overwrite" part of it via xml.
To cut a long story short: IMHO an nfo-standard should be based on a document standard and follow this standard 100%. If this is not the case (as currently), there will never be a "real" standard because no matter which programming language a media application uses, you cannot use the standard tools of this language to read it and it will always be a lot of manual work (as it was here with MP2). As long as there is manual work involved, there will always be deviations from implementation to implementation.
But even if the programmers of different media applications would agree upon a "new" standard for nfo-files, this would mean that the implementations of all such programs would have to change completely and this I would consider unlikely...

 

[resa]

• The most important word in your statement for me is "on-demand". I would never want any Media Application to automatically generate nfo-files in my media collections (or even worse, let it automatically overwrite any nfo-files that were already there before).

Agreed, and therefore i was using the words on-demand Including my response for some of the following points on your list too, let me go into detail right here... Logically there should be 2 or more different situations for generate the nfo's, but the first one should be enough at least for the experienced useres (and the first situation represents how far the Kodi-Devs has gone until now):

1. Export the whole library into seperate .nfo-files: Will export the whole library into seperate video-filename.nfo where each is written to the the same location of the video-filename.avi/mkv/.... as said already. Before starting this Process, there MUST be the question what to do with items that already have existing nfo's - should the nfo's be overwritten or should export for those items to be skipped? Along with an super-clear instruction for overwriting meaning that the current data stored in the library is used to overwrite and the user have to be absolutely sure, that the data in the library is currently correct for each and every item. This option can be used as globally function when i.e.: Initially saving the nfo's after the first bunch of items was added to the library (answering overwrite or skip). Each time the User has made numerous personal alterations to the data and is sure, that all data for all items in the library is currently correct (answering overwrite). Each time the User has added the next bunch of items to the library and made no alterations to existing items or is unsure if all data for existing items in the library is currently correct anymore (answering skip). Correcting the content of nfo's for already exported items after the items data is updated in library and the existing nfo is deleted (asnwering skip). And so on...

[To give you the complete information: the complete library export in Kodi can be done either to seperated nfo's written to video-filename location as we are talking here about OR into one single xml containing the data for all the items in library together, what is somewhat intransparent and almost counteracts the idea for use as a backup, fallback-backup and exchange format since all the data in library should never be damaged partly or similar mess (for not needing to paste data from different xml's together manually some day). Also having more disadvantages like the need to include the physical file location, hampering to move the files to another location and easily reimport the data using the nfo's and so on and on.]

2. For more convienient usability there should be also the ability to export the nfo for the marked single item or multiple marked items from within list views, after the same MUST BE question along the instruction mentioned above already. And in an optimal world the user also be asked for saving/updating the changed data into nfo-file every time the user has re-scraped the item data or have opened the detail view of an item and altered the content of one or more fields in there (i.e. altering title, sorting title, description etc. like many users love to do for several reasons, as far as i have learned), this can be worked out either with the use of an checkbox "generate/update nfo-file on leaving/approval" or also with an option in the main settings to supress those question for the users who dont want to use the nfo's in general.

Additionally there should be functions to generate nfo's after user/auto-approval of scraped data for new items and maybe to generate all the nfo's only for the specified paths with or without processing it's subfolers... and so on... The more functionalities you have handy within MePo, the more it will be an one-time task without any accidents. Kodi lacks of more flexible options here, alltough the ground idea is somewhat great

Fully equipped with that functions and understoodable communicated to the user within the gui's even the unexperienced users should notice the purpose of the nfo's no matter if placed within an WPF-like configutil as seen on MePo1's Videos/MovingPictures or placed within the frontend. I am currently not really aware how editing the the media-librarys in MePo2's has to be done, i wasen't enough satisfied to go that deep into the details during my tryouts on MePo2 until now, but to add/edit/rescrape a single or some few items was always be mission possible to do with the remote (and on other hand: who the heck is wasting all the full PC abilities and has not at least something like the thirty-bucks Logitech K400 attached and is hiding it by sliding this under his sofa?!?!? )

But i double agree, the idea to do all the configurations on client, server and plugins from smartphone/tablet using an web-interface from the sofa is a charming one also doing all the settings and details within the ir-remotable frontend like realised in Kodi is somewhat terrifying and anoying. MePo1 fully scores here with it's WPF-like configs

Oh and you can believe me, there is nothing in the respect that I would support more.

This is good to hear about and let me assure you that you are not the only dev on an topic related project that have an open ear for this...

I have no clue who first "created" nfo-files - it might have been Kodi. The issue with nfo-files is that they, as they are now, look like they are based on another standard. But in fact they are not. As an example: Nfo-files look like they are XML-files. But actually they are not.

I am not abselutely sure, if this was initiated from XBMC or not, but i think the whole thing is already sort of beginning standard already. As far as i have looked over it, the XML seems to be simple and valid XML and seems also to not beeing changed over an long time. The XMLs generated by an current Kodi-Version is understood by i.e. an scraper delivered with MePo1's MovingPictures called XBMC(Local) wich was published on 23.07.2011 - more than 4 years ago and the data is still well imported, since every plugin on other projects, dedicated to import Kodis SQLite dbase get broke nearly on every version the Kodi devs are releasing dure to altered tables and variables and so on. Furthermore Emby Media Server seems to handle this nfo's but is at least generating nfo's using all the same tags, as i noticed on a first overlook. Conclusion: There are already 4 projects that supports this specific nfo' format ... and that is only what i know so far (but i will check all other topic relevant projects for that too in the future).

I hope this ins't interfere with forum rules if i post a link into Kodis WIKI for an brief description into this spoiler:

Spoiler: Mod: please remove if interfere with forum rules.

http://kodi.wiki/view/NFO_files/movies

The "standard" for a nfo-file belonging to a video file with multiple episodes of a series is that the nfo-file has multiple <episodedetails> root elements. The xml standard does not allow multiple root elements, which is why any xml-parser out there throws an error when reading such files (and why I had to write our own XmlNfoReader class...)

OK, but this class is already there? So this should not be that big deal anymore And: I will not swear on it, but as far as i remember even Wordpress is using XML-Files sturctered in that way for importing/exporting new posts and even edits from external clients and i think for exchanging the pingbacks with other Blog/CMS/Forum-Systems too. Also this structure is used within Kodis movie nfo's for streaming-details (sort of media-info of the videofile) and in the tvshow.nfo too (which is placed into the series mainfolder and contains the infos about the series in general)... One thing to add may will be an additional scrape thetvdb for the season-related infos whilst import the nfo's of episodes from this season.

Much more complex seems to be the handling of dynamic data in the nfo's... the only that comes to my mind here is the view-count... but this value should be synchronized between library and .nfo in an reliable way and should be waive external things like trakt and also be easy to setup (kodi needs an extra plugin as well as some system-settings and on top you need to edit a XML config-file manually to get this working, i think you guys on MePo could perform better on this issue easily ) ... Maybe parted movies (bla-a.avi, bla-b.avi or bla-CD1.avi, bla-CD2.avi, i.e. Kodi responds on -CDx and perform generate nfo onto file CD1 only and also span the parts at playback) will make extra work... or movie-collections/series/sets and things like that. But at all, it wcould be not that big problem in order to make use of these nfo-format

Another example is what we are dealing with in this thread: Kodi allows nfo-files just with a link to the imdb webpage of a movie in it. This has nothing to do with xml. And even worse, AFAIK, Kodi also allows part of the nfo-file to contain (kind of) xml and at the end have a link to imdb so that you can take metadata from imdb first and "overwrite" part of it via xml.

I think we could ignore on this. It seems this was intended to have more flexibility in handing over goal-orientaed informations to the scraper, mainly this was implemented to easily make use of an completely different kind of files that wears the extension .nfo too i think. (Meaning those ascii-files which are mostly attached to video files released by <the-part-of-internet-we-dont-talking-about-in-here> and that are containing not many usefull informations, except an imdb-link or tt-number mostly ). As you said, you drag and drop the folder you have extracted from some .rars onto your video-archive disk, letting kodi pick the imdb-link/tt-number to scrape with and after scraping you replayce this .nfo with kodis .nfo filled with the whole XML data

To cut a long story short: IMHO an nfo-standard should be based on a document standard and follow this standard 100%. If this is not the case (as currently), there will never be a "real" standard because no matter which programming language a media application uses

So we have at least 4 projects working with this standard, and to repeat that, that seems to be not changed since years in general terms. I also cann not see the problem, if there are projects using some own specific tags in addition to the universal used tags, since you not have to process on every tag and can skip unknown tags. Also it is free for any project to skip some universal tags in order to generate these information by itselfs, i.e. media-information or something like that. Important for the User are only the tags like title, sorting-title, plot, outlin and some more. On tags like actor or writer the number of user that is altering content here will be much more smaller already. Importing the nfo also dosen't say, that there should be none additional scraping tasks added to fulfill the needs of the project i.e. for fill up some specific art-work that is used, only to NOT leave content altered by the user untouched is an no-go.

you cannot use the standard tools of this language to read it

I cannot agree here, since structure of the nfo's looks like valid XML to me and is used by other serious applications that way too. It also reflects exactly the example that is shown on wikipedia at all (not that this is an measurement to me, i only say ). Dunno, but we are talkin about VS here, right? I am a bit out of practice. But at least i would be able to fire up free VS-MistressEdition and handle this like simplyfied noobish text "parsing" in the way of searching for the tags and cut the text between them, pushing this into vars in order to write this down to an dbase, if this would not be that totally Mr. Dumbiverse code of the year it would be then

As long as there is manual work involved, there will always be deviations from implementation to implementation.

I can not get your point here, it all counts down on the major tags never to be changed. How this is implemented, processed, sourrounded by additional things and turning out in any project or not - all this is fully decided by an projects dev-crew on their own behalf. Only the tags have to be keeped the same.

But even if the programmers of different media applications would agree upon a "new" standard for nfo-files, this would mean that the implementations of all such programs would have to change completely and this I would consider unlikely...

Seems like standard is already there and working fine. Why not simply using the power of 4 Projects (remember there are maybe already more) to coalesce on this as an exchange format? The only thing to do is fixing this existing standard tags to be sure it remains unchanged during the next years (unchanged means here: never alter an existing tag, only add new tags and let old tags simply dieying, so it is compatible downwards at any time without blocking new developments). But in the end i am sure, that this nfo's will work the next years if there is an coalition or not. This was only an additional improvment that i was suggesting with this. The Important thing is to make it possible, that at least MePo2, better also MePo1 can import these nfo's generally for the users that want to switch from another project to MePo or using multiple projects simultaneousely, that was my major point

But don't think my vision stops with the nfo's ... did you know that Kodi and the 3 other projects can im/export artwork in an format like videofilename-poster.jpg, videofilename-thumb.jpg etc. too?

No no, dont be afraid, now i am done here this time

If any of the devs like further informations i can link to more infos as well as to nfo's with real sample-data for movies and tv shows, please PM me in that case

Thanks for discussion and your interrest, with all the very best regards.

 

[JSchuricht]

Alright new logs attached with the modifications requested by MJGraf.

Definitely some weirdness in the import. I ran import 6 times because the numbers kept changing. Videos seems to grab everything by the second import but movies actually goes backwards, forgetting what it had on some imports.

Number of files after each import
Movies/Videos
1067/1458
2609/3488
2606/3488
2608/3488
2610/3488
2607/3488

 

[MJGraf]

Thanks, JSchuricht,

First of all it seems that you are not using my binaries from above. Can you please check whether you have copied the DLL as described in the linked thread?

But I think there was also a misunderstanding. The log tells me that your settings are now like this:

2015-12-11 17:07:13.350154 [Info.][Main]: -------------------------------------------------------------
2015-12-11 17:07:13.350154 [Info.][Main]: NfoMovieMetadataExtractor v0.10 instantiated
2015-12-11 17:07:13.350154 [Info.][Main]: Setttings:
2015-12-11 17:07:13.350154 [Info.][Main]:  EnableDebugLogging: True
2015-12-11 17:07:13.350154 [Info.][Main]:  WriteRawNfoFileIntoDebugLog: True
2015-12-11 17:07:13.350154 [Info.][Main]:  WriteStubObjectIntoDebugLog: True
2015-12-11 17:07:13.350154 [Info.][Main]:  MovieNfoFileNames: movie
2015-12-11 17:07:13.350154 [Info.][Main]:  NfoFileNameExtensions: .nfo .xml .txt imdb
2015-12-11 17:07:13.350154 [Info.][Main]:  SeparatorCharacters: | , /
2015-12-11 17:07:13.350154 [Info.][Main]:  IgnoreStrings: unknown
2015-12-11 17:07:13.350154 [Info.][Main]: -------------------------------------------------------------

So it seems you have added "imdb" as file name extension (without dot in the beginning). From the logs I can see that the file name of your nfo-files is quite often "imdb.txt". So you have to add "imdb" as additional file name - not as file name extension.

Can you please try again with the two hints above? Thanks!

 

[MJGraf]

I was just about to make the regular expression I use for parsing an IMDB-ID a setting and had some interesting reading about the C# regex class. Although it is probably only interesting for developers, I want to write it down somewhere so that it is not forgotten.

So the question was: How can we serialize a C# Regex object. The Regex class implements ISerializable - but unfortunately not IXmlSerializable, which we need for our settings system. My first thought was that we just inherit from the Regex class and implement IXmlSerializable on our own (something like public class XmlSerializableRegex : Regex, IXmlSerializable). The problem here is that once a Regex object is created, it is immutable - meaning you cannot change its regular expression anymore. But during deserialization, first an object is created and then its fields are set, which doesn't work here. According to the MS reference source, it is not possible that we manually do this, because the classes we need to initialize all the fields of the Regex class are "internal".

The next idea was to just serialize the regular expression as string and create a Regex object from that on demand. The downside of this approach is that we don't serialize the RegexTimeout (which IMHO would be acceptable) but also not the RegexOptions (such as IgnoreCase or InvariantCulture). These would then still be hardcoded which I don't think is a good idea.

The best solution I can currently think of is wrapping a Regex object into our own class that implements IXmlSerializable and besides this only as one (readonly) property: Regex. Using this regex then requires a call like OurSettingsObject.ImdbRegex.Regex.Match(...), which is not extremely beautiful, but IMHO acceptable.

And there is another advantage of this:
We currently create lots of Regex objects on demand as local variables. This is anything but ideal from a performance perspective. The constructor of the Regex class compiles the regular expression, which is why instantiating a Regex object is a relatively time consuming operation (ok, we are talking about milliseconds here, but if we instantiate 10 Regex objects per MediaItem in an import and the import has tens of thousands of MediaItems, we suddenly talk about several minutes...).
For that reason, there is an internal cache for compiled Regex objects in the Regex class. But this cache is according to MSDN only used, when calling the static methods of the Regex class (which we currently don't do). Instantiated Regex objects are not cached. But calling the static methods instead might also not be ideal, as by default, the cache has a maximum size of 15 Regex objects, which for the whole MP2 system might not be enough. We can set a higher size limit, but in the end it is quite difficult to estimate the maximum number of Regex objects in the MP2 system.
Now if we use the last approach above, we basically use our settings cache to cache our own Regex objects (meaning: every (XmlSerializable)Regex we have in a settings file is automatically cached so that we only have one instance of it and compilation only has to happen once). According to MSDN, the whole Regex class is thread safe so that the whole MP2 system can use that single instance of the Regex object without problems.

Conclusion: I'll implement the last approach described above. I'll for now put the class in the OnlineLibraries project as there we will likely need it most. We can later move it to MediaPortal.Utilities.Xml, but this (if I finally understood it correctly) requires a version bump of MediaPortal.Utilities so that it might be better to do that in the end...
If anyone has more experience with the Regex class than me and disagrees with the above, please let me know...

 

[resa]

@MJGraf :

In addition to my post above i have to add this correction, since i was unable to connect your lines with those i.e. S04E01E02 things that occurs sometimes. After reading your post nearly every half hour again to get your point over the last 36hrs now i have got it finally and understand what you meant with 2 times <episodedetails> rootknot, my fault

But for this was no way possible to catch this specific error and restarting with the xml parser on beginning of the second rootknot so you needed an own class only about this? Or have you thinked about to cache these nfo's in this specific situations first to split them and sending seperately to importer? No offending, just asking - i know there can be numerous reasons why not

An last question i have... if i have understand it right MePo2 is using xml-export/import for metadata more frequent way that i have suspected... do you do this under respecting the kodi-used tags already oris this done in an MePo-specific structure usually (sorry to ak this, but due to too heavy problems with MePo2 on installing/reliability i had urgently to schedule any further experiments until the next release of MePo2, since i have a lot of work to do with home/officemedia-setups currently and must concentrate on the live and unattended usuable things)

Thanks again and thanks for diskussion.

best regards.

 

[Lehmden]

Hi.

do you do this under respecting the kodi-used tags already oris this done in an MePo-specific structure usually

We have worked hard to accept as much xml tags as possible, as nearly every application uses some variations of the tags... As example not even the (extremely important) IMDB-ID has the same tag in all .nfo files out there. Sometimes the tag is <id>, sometimes it is <imdbid> sometimes <imdb-td> and so on. MP2 is supporting as much of them as possible. And yes, this includes the tags known from Kodi...
If you have an export of your Kodi database in the "one nfo per movie" format then you directly can import them into MP2. This is valid not only for movies but for series too..
@MJGraf has done all the great programming work but some other people (incl. myself) have worked with him to get the best possible solution. So I also can answer your question this time..

By the way, my own program MKV-Buddy also generates .nfo. This .nfo are compatible with Kodi too even if my main focus is on MP2 (as you may imagine)... Main difference to "original" Kodi .nfo is, .nfo from MKV-Buddy are encoded in UTF16 (normally UTF-8 is used) for better support of locally used special characters (like Ä Ö Ü in German language) in the name and description fields. The encoding nevertheless is compatible with Kodi so no problem at all... Only the file size of the .nfo is a little bit bigger this way.