TMDB import problems (1 Viewer)

[ponchick]

MP cannot import some titles using TMDB. Network sniffer shows correct API answers, all necessary data received, but movie importer doesn't even show possible matches from this scrapper.

Spoiler

GET http://api.themoviedb.org/3/movie/tt0395479?api_key=cc25933c4094ca50635f94574491f320&language=ru HTTP/1.1
User-Agent: MovingPictures/1.5.1.1487
Accept: application/json
Host: api.themoviedb.org
Proxy-Connection: Keep-Alive

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public, max-age=86402
Content-Type: application/json;charset=utf-8
Date: Thu, 09 May 2013 14:55:45 GMT
ETag: "8a300dd149e787e3b0326855c56f2156"
Server: nginx
Status: 200 OK
Via: 1.1 varnish
X-Cache: MISS
X-Memc: HIT
X-Memc-Age: 50495
X-Memc-Expires: 35907
X-Memc-Key: 1847c7f43230a607a02d0da26b510de7
X-Varnish: 2249901869
X-Varnish-Sid: tm-v-p-1200
Content-Length: 1017
X-Cache: MISS from atom.home
X-Cache-Lookup: HIT from atom.home:3128
Connection: keep-alive

{"adult":false,"backdrop_path":"/wFfHC9aQBDMc0a44XHTpld1VdjX.jpg","belongs_to_collection":{"id":189257,"name":"Короткометражные фильмы Pixar","poster_path":"/1WHtPF7LXQGl6PkqPxRby2488SN.jpg","backdrop_path":null},"budget":0,"genres":[{"id":16,"name":"мультфильм"},{"id":10751,"name":"семейный"},{"id":null,"name":"короткометражка"}],"homepage":"","id":13062,"imdb_id":"tt0395479","original_title":"Boundin'","overview":"Веселый мультфильм о приключениях, беззаботного горного барашка.","popularity":3.62820868,"poster_path":"/35jJseJi3BiQ2ucCxKpnUoJhal3.jpg","production_companies":[{"name":"Pixar Animation Studios","id":3}],"production_countries":[{"iso_3166_1":"US","name":"United States of America"}],"release_date":"2003-11-04","revenue":0,"runtime":0,"spoken_languages":[{"iso_639_1":"en","name":"English"}],"status":"Released","tagline":"","title":"Барашек","vote_average":8.0,"vote_count":8}

GET http://api.themoviedb.org/3/search/...e=1&language=ru&include_adult=False&year=2003 HTTP/1.1
User-Agent: MovingPictures/1.5.1.1487
Accept: application/json
Host: api.themoviedb.org

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: public, max-age=3600
Content-Type: application/json;charset=utf-8
Date: Thu, 09 May 2013 14:43:17 GMT
ETag: "3ac11112417c74072905982f2d5b4cb3"
Server: nginx
Status: 200 OK
Via: 1.1 varnish
X-Cache: MISS
X-Memc: MISS, STORE
X-Memc-Age: 0
X-Memc-Expires: 3600
X-Memc-Key: 2e115fee0228de21bbfd17358a2e577b
X-Varnish: 2249464322
X-Varnish-Sid: tm-v-p-1200
Content-Length: 323
Age: 748
X-Cache: HIT from atom.home
X-Cache-Lookup: HIT from atom.home:3128
Connection: keep-alive

{"page":1,"results":[{"adult":false,"backdrop_path":"/kcHvzJuHXsMNXwR9bjjsf7RX1lQ.jpg","id":13062,"original_title":"Boundin'","release_date":"2003-11-04","poster_path":"/35jJseJi3BiQ2ucCxKpnUoJhal3.jpg","popularity":3.288462604,"title":"Барашек","vote_average":8.0,"vote_count":8}],"total_pages":1,"total_results":1}

GET http://api.themoviedb.org/3/movie/13062?api_key=cc25933c4094ca50635f94574491f320&language=ru HTTP/1.1
User-Agent: MovingPictures/1.5.1.1487
Accept: application/json
Host: api.themoviedb.org

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public, max-age=86402
Content-Type: application/json;charset=utf-8
Date: Thu, 09 May 2013 14:55:45 GMT
ETag: "8a300dd149e787e3b0326855c56f2156"
Server: nginx
Status: 200 OK
Via: 1.1 varnish
X-Cache: MISS
X-Memc: HIT
X-Memc-Age: 50495
X-Memc-Expires: 35907
X-Memc-Key: 1847c7f43230a607a02d0da26b510de7
X-Varnish: 2249902018
X-Varnish-Sid: tm-v-p-1200
Content-Length: 1017
X-Cache: MISS from atom.home
X-Cache-Lookup: HIT from atom.home:3128
Connection: keep-alive

{"adult":false,"backdrop_path":"/wFfHC9aQBDMc0a44XHTpld1VdjX.jpg","belongs_to_collection":{"id":189257,"name":"Короткометражные фильмы Pixar","poster_path":"/1WHtPF7LXQGl6PkqPxRby2488SN.jpg","backdrop_path":null},"budget":0,"genres":[{"id":16,"name":"мультфильм"},{"id":10751,"name":"семейный"},{"id":null,"name":"короткометражка"}],"homepage":"","id":13062,"imdb_id":"tt0395479","original_title":"Boundin'","overview":"Веселый мультфильм о приключениях, беззаботного горного барашка.","popularity":3.62820868,"poster_path":"/35jJseJi3BiQ2ucCxKpnUoJhal3.jpg","production_companies":[{"name":"Pixar Animation Studios","id":3}],"production_countries":[{"iso_3166_1":"US","name":"United States of America"}],"release_date":"2003-11-04","revenue":0,"runtime":0,"spoken_languages":[{"iso_639_1":"en","name":"English"}],"status":"Released","tagline":"","title":"Барашек","vote_average":8.0,"vote_count":8}

GET http://api.themoviedb.org/3/search/...&page=1&language=ru&include_adult=False&year= HTTP/1.1
User-Agent: MovingPictures/1.5.1.1487
Accept: application/json
Host: api.themoviedb.org

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: public, max-age=3600
Content-Type: application/json;charset=utf-8
Date: Thu, 09 May 2013 14:43:17 GMT
ETag: "3ac11112417c74072905982f2d5b4cb3"
Server: nginx
Status: 200 OK
Via: 1.1 varnish
X-Cache: MISS
X-Memc: MISS, STORE
X-Memc-Age: 0
X-Memc-Expires: 3600
X-Memc-Key: 4ccab2cf3e7a1e1d5b5b40ae66e910a8
X-Varnish: 2249464456
X-Varnish-Sid: tm-v-p-1200
Content-Length: 323
Age: 749
X-Cache: HIT from atom.home
X-Cache-Lookup: HIT from atom.home:3128
Connection: keep-alive

{"page":1,"results":[{"adult":false,"backdrop_path":"/kcHvzJuHXsMNXwR9bjjsf7RX1lQ.jpg","id":13062,"original_title":"Boundin'","release_date":"2003-11-04","poster_path":"/35jJseJi3BiQ2ucCxKpnUoJhal3.jpg","popularity":3.288462604,"title":"Барашек","vote_average":8.0,"vote_count":8}],"total_pages":1,"total_results":1}

GET http://api.themoviedb.org/3/movie/13062?api_key=cc25933c4094ca50635f94574491f320&language=ru HTTP/1.1
User-Agent: MovingPictures/1.5.1.1487
Accept: application/json
Host: api.themoviedb.org

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: public, max-age=86402
Content-Type: application/json;charset=utf-8
Date: Thu, 09 May 2013 14:55:45 GMT
ETag: "8a300dd149e787e3b0326855c56f2156"
Server: nginx
Status: 200 OK
Via: 1.1 varnish
X-Cache: MISS
X-Memc: HIT
X-Memc-Age: 50495
X-Memc-Expires: 35907
X-Memc-Key: 1847c7f43230a607a02d0da26b510de7
X-Varnish: 2249902018
X-Varnish-Sid: tm-v-p-1200
Content-Length: 1017
Age: 2
X-Cache: HIT from atom.home
X-Cache-Lookup: HIT from atom.home:3128
Connection: keep-alive

{"adult":false,"backdrop_path":"/wFfHC9aQBDMc0a44XHTpld1VdjX.jpg","belongs_to_collection":{"id":189257,"name":"Короткометражные фильмы Pixar","poster_path":"/1WHtPF7LXQGl6PkqPxRby2488SN.jpg","backdrop_path":null},"budget":0,"genres":[{"id":16,"name":"мультфильм"},{"id":10751,"name":"семейный"},{"id":null,"name":"короткометражка"}],"homepage":"","id":13062,"imdb_id":"tt0395479","original_title":"Boundin'","overview":"Веселый мультфильм о приключениях, беззаботного горного барашка.","popularity":3.62820868,"poster_path":"/35jJseJi3BiQ2ucCxKpnUoJhal3.jpg","production_companies":[{"name":"Pixar Animation Studios","id":3}],"production_countries":[{"iso_3166_1":"US","name":"United States of America"}],"release_date":"2003-11-04","revenue":0,"runtime":0,"spoken_languages":[{"iso_639_1":"en","name":"English"}],"status":"Released","tagline":"","title":"Барашек","vote_average":8.0,"vote_count":8}

Debug mode doesn't help much:

09-May-2013 22:43:12  Info [	  MovieImporter]: User reprocessing Барашек (Boundin') (2003) [tt0395479].mkv
09-May-2013 22:43:20  Info [	  MovieImporter]: No exact match for Барашек (Boundin') (2003) [tt0395479].mkv

Here some broken movies (not complete list):
https://www.themoviedb.org/movie/13062-boundin
https://www.themoviedb.org/movie/158-knockin-on-heaven-s-door
https://www.themoviedb.org/movie/13928-knick-knack
https://www.themoviedb.org/movie/17963-gone-nutty
https://www.themoviedb.org/movie/53985-mutant-pumpkins-from-outer-space
https://www.themoviedb.org/movie/13926-red-s-dream

 

[RoChess]

For these situations, MovPic has what is called scraper-script debug mode. If you enable that, it will allow you to see exactly why the movie failed as you can see every step a scraper-script goes through. Well this works for external scraper-scripts such as IMDb+ anyway, never really tested it much for internal scraper-scripts such as TMDb, but should work the same.

Steps on how to enable scraper-script debugging = http://code.google.com/p/imdbplus/wiki/WikiInstallScraper

And be sure to include *full* MovPic log file, not just the parts you think are needed. And you will have to of course repeat the steps so that the log file contains the info needed. Easy way would be to send one of the failing movies back to the importer. Due to asynchrone nature and large log files in scraper-script debug mode, keep it down to just one movie that fails.

Just tested and all those movies import fine with IMDb+ for me, so might be something specific to your setup or indeed something wrong with TMDb.

 

[ponchick]

As I mentioned before debug mode is on. Log are attached.

 

[RoChess]

As I mentioned before debug mode is on. Log are attached.

If you had enabled scraper-script debug mode, alongside MediaPortal debug verbosity, then the log file would have shown lines containing the following:

Debug [ ScraperNode]:

This lacks from your logs, so there is nothing really in them that is of any help to properly diagnose the issue.

Based on previous experiences the most likely answer is that TMDb has trouble with the Russian filename you use, and your MovPic noise filter ends up removing the English title or confuses TMDb even more in the search. If TMDb scraper is able to make use of the IMDb tt-ID, then more then likely this tt-ID is missing at TMDb website for these movies, causing it to fail when searching.

IMDb+ would solve this for you, as it supports multiple methods to ensure IMDb tt-ID lookups result in an instant match. Or you can user contribute the missing data to TMDb, or adjust your noise filter to create a workable title, or adjust your filenames.

If you wish to keep TMDb then adding the missing data would be the best way, as it will fix it not just for your collection, but the next user as well that might run into this problem.

 

[ponchick]

I forgot change MediaPortal log verbosity to debug. Updated log is attached but it does not explain the problem. Sniffer shows same network activity, scrapper get all data about that movie 3 times.
I've 300+ movies and only 20 have problems with this scrapper. All folders named by pattern, all of them contain translated title and themoviedb.org scrapper works fine with them. Again, scrapper fetch all data about movie but cannot parse it.
IMDB is not a option for russian - a lot of movie data still untranslated.

 

[Edalex]

Нет практически ни слова про themoviedb в логе.
-----
No word of tmbd in log

 

[ponchick]

themoviedb.org scrapper written in c#, not xml. Seems it has no logging capabilities at all.

 

[RoChess]

Guess it has to be step-traced inside Visual Studio then.

@Edalex perfect job for you since you know C# and Russian.

 

[Edalex]

Fine, @RoChess
I found the problem
There was json deserialization error
Looks like, int& is needed somewhere instead of int, @ltfearme
Failing here https://code.google.com/p/moving-pi...ingPictures/DataProviders/TheMovieDbAPI.cs#49
So something in MovieInfo is wrong

Spoiler

System.Runtime.Serialization.SerializationException перехвачено
Message=ValueType "System.Int32" не может иметь значение null.
Source=System.ServiceModel.Web
StackTrace:
в ReadMovieInfo.GenreFromJson(XmlReaderDelegator , XmlObjectSerializerReadContextComplexJson , XmlDictionaryString , XmlDictionaryString[] )
в System.Runtime.Serialization.Json.JsonClassDataContract.ReadJsonValueCore(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
в System.Runtime.Serialization.Json.JsonDataContract.ReadJsonValue(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
в System.Runtime.Serialization.Json.XmlObjectSerializerReadContextComplexJson.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
в System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, DataContract& dataContract)
в System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
в System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
в ReadArrayOfMovieInfo.GenreFromJson(XmlReaderDelegator , XmlObjectSerializerReadContextComplexJson , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )
в System.Runtime.Serialization.Json.JsonCollectionDataContract.ReadJsonValueCore(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
в System.Runtime.Serialization.Json.JsonDataContract.ReadJsonValue(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
в System.Runtime.Serialization.Json.XmlObjectSerializerReadContextComplexJson.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
в System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, DataContract& dataContract)
в System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
в System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 declaredTypeID, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
в ReadMovieInfoFromJson(XmlReaderDelegator , XmlObjectSerializerReadContextComplexJson , XmlDictionaryString , XmlDictionaryString[] )
в System.Runtime.Serialization.Json.JsonClassDataContract.ReadJsonValueCore(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
в System.Runtime.Serialization.Json.JsonDataContract.ReadJsonValue(XmlReaderDelegator jsonReader, XmlObjectSerializerReadContextComplexJson context)
в System.Runtime.Serialization.Json.XmlObjectSerializerReadContextComplexJson.ReadDataContractValue(DataContract dataContract, XmlReaderDelegator reader)
в System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, DataContract& dataContract)
в System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
в System.Runtime.Serialization.XmlObjectSerializerReadContextComplex.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
в System.Runtime.Serialization.Json.DataContractJsonSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName)
в System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName)
в System.Runtime.Serialization.Json.DataContractJsonSerializer.ReadObject(XmlDictionaryReader reader)
в System.Runtime.Serialization.Json.DataContractJsonSerializer.ReadObject(Stream stream)
в Cornerstone.Extensions.JsonExtensions.FromJson[T](String json) в D:\Documents and Settings\Дмитрий\Мои документы\Visual Studio 2010\Projects\moving-pictures\Cornerstone\Extensions\JsonExtensions.cs:строка 45
InnerException:

 

[Edalex]

So it is themoviedb api bug because of null genre id. Tell them about it, @ponchick.

{"id":null,"name":"короткометражка"}