Importing user ratings from thetvdb taking 30+ minutes for 400 series (1 Viewer)

[dir]

Tonight I noticed that an episode wasn't appearing in the system. "Touch (2012)" Season 02, Episode 01.

Episode 02 was there, just no episode 1. So I manually ran an import via MP-TV Series Configuration to see what it would say about it. It found the new episode but failed to retrieve info from thetvdb api. I tried manually browsing to it, and it worked, so I'd assume it was a time-out situation.

I re-ran the import and this time it worked, but then when MP-TV Series Configuration went to one of the next screens (the one that shows progress during all the stages of importing, ie. Local File Parsing, MetaData, Artwork, and Misc), I noticed something odd: It was able to get through all the stages of importing and scanning all my 24,000 episodes / 433 series, but when it gets to the last stage ("Retrieving User Ratings"), it went really slowly.

It took from 9:40 until 10:33 to do this stage (almost an hour). The big delay seems to be with a very frequent timeout when connecting to thetvdb.com:

Can't connect to http://thetvdb.com/api/GetRatingsForUser.php?apikey=<blah>&accountid=blah&seriesid=blah :  The operation timed out

This occurred every minute or so. It was able to get through perhaps 10 to 15 series, then would have a timeout occur. The timeouts added a few additional seconds, but overall, even if they didn't time out, the act of doing 433 of these web api calls took at least 30 minutes.

Clearly thetvdb.com is overloaded, so there's not much that can be done at the importer side of things, but it got me wondering:

Does the MP-TV Series importer do this scan of all of my tv shows every time it does an import, i.e. every time it detects a new file? I never use user ratings and don't care about them at all, so I'd prefer to disable this stage completely. It also doesn't make a lot of sense to constantly check for user ratings on shows that are old - I'd estimate that out of my 433 series, there are less than 2% that would have updated user ratings anyway. It makes little sense to constantly check the other series for updated data >every time a new file/episode< is imported.

Normally, the importer is happening in the background and I'd never notice this, but if this stage was optional, it would not only reduce the activity at my end, it would reduce the load on thetvdb's servers as well.

 

[RoChess]

Inside MP-TVSeries config on the details tab with all your series, you can right click a show and select to "Ignore on Scan" (useful for series that are "Ended").

This will speed things up for you then.

Further improvements to not use TheTVdB as much, and use local database more often when possible (adding new episodes to an existing show for example during first stages in import), are also in the works.

 

[dir]

good tip - thanks for that. Does setting a series to ignore mean that it won't detect a new episode that gets added, in addition to not checking for user ratings and other things?
I'm happy to mark many of my old series as 'ignore', but some still have a season or episode or two missing, even though they are 1960's shows. Occasionally those episodes/seasons become available, so I'd still want new episodes added.
For those series that are complete and old, I can definitely mark them as 'ignored'.

[edit: after going through the first 200 series, it's clear that what I'm really doing is following the logic of, "If the series has ended, set it to ignore". This would be a good option to have in the settings rather manually setting each series one by one.

 

[RoChess]

I actually already passed your logic to developer, as I'm using it myself often. But I can't find it on ticket list, so must have done so via IRC chat. Did find another ticket I submitted: http://code.google.com/p/mptvseries/issues/detail?id=599 but I now use a different method by not showing all episodes anymore and shows dissapear once I deleted last episode from disk (and auto re-appear on new episodes when seasons resume or start anew).

As for your other question, the episodes do get added to a show set to ignore on scan. They just do not get updated with TheTVdB info, but they are added like "Episode 1" for episode title with no summary/etc.

I'm sure you got a record/history on when the rare occasion happens that those missing episodes are 'found', so you can then unignore, run import and set it back to ignore.

 

[dir]

Ok, that clears it up a bit. I suppose the more 'granular' opportunity would be to be able to not have the importer perform some of the more lengthy activities, such as the user ratings, as an option for those that don't want/care about user ratings.

For those of us with rather (ahem) large collections, (and I think we're all moving towards that as time goes on), importers will need to use a selective approach to updates rather than "en masse". I'm sure MPTVSeries already has some clever decision code as to what it should do, so now it comes down to the user making trade-off decisions of information vs. efficiency.

 

[RoChess]

A large part also has to do with what TheTVdB provides and I believe they have pretty much of an "en masse" approach on their end, zip versions of entire show with all data. Everybody is waiting forever on their v2.0 of site+API which is supposed to make it all run much better and I think they are all still applying ductape to what they got now (then again could be totally wrong, because I'm not exactly looking at the raw results myself)

 

[ltfearme]

You can disable episode ratings import in the Online Data tab of configuration. It will still do it for series but that should be very fast, its episodes which are the killer as it needs a seperate request for each series.

Another alternative is to remove the Account ID from settings, if that is empty then it will no longer import user ratings or favourites.