Dutch language scraping support (1 Viewer)

[HomeY]

Quick question: Is it possible with this scraper to take all movie info from IMDB, but to use another language (moviemeter scraper for example for Dutch) for the summary?

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

Quick question: Is it possible with this scraper to take all movie info from IMDB, but to use another language (moviemeter scraper for example for Dutch) for the summary?

Yes, this is possible, if the movie site supports a direct lookup based on imdb.com tt-ID. I know moviemeter has a very complete database of IMDb tt-IDs, but I'm unable to find out how to search on it.

For example RottenTomatoes allows me to use:

http://www.rottentomatoes.com/alias?type=imdbid&s=0499549

And it will retrieve the RottenTomatoes details page for the Avatar movie for me, without any fuzzy search logic mistakes. Tell me how to do the same with moviemeter.nl and I can add anything that sites gives me into the IMDb+ scraper, be it summary, tagline, etc.

Thinking about it, I'll expand the scraper to support German, French, Italian, Spanish and Portugese, because IMDb has a localized website in those languages, so I can use any information from them easy based on IMDb tt-ID. I'll also scan all the existing scraper scripts, to see if any of them support a direct IMDb tt-ID search method, so I can include support for those languages as well.

 

[HomeY]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

I'll go see if i can figure out how that search works! I'd love to have that option

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

I'll go see if i can figure out how that search works! I'd love to have that option

Let me know, I just went through all the scrapers, and I'll be able to add support for Swedish and Icelandic, as their respective movie websites (filmtipset.se and kvikmyndir.is) support direct IMDb tt-ID references.

So I will be able to add support for 7 languages, but if you want Dutch, find me a way to do the same on moviemeter.nl website. I even tried to an advanced google "site:moviemeter.nl link:/tt0499549" search, but it looks like the webmaster of moviemeter blocked that ability in their robot definitions.

 

[HomeY]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

I'll go see if i can figure out how that search works! I'd love to have that option

Let me know, I just went through all the scrapers, and I'll be able to add support for Swedish and Icelandic, as their respective movie websites (filmtipset.se and kvikmyndir.is) support direct IMDb tt-ID references.

API - MovieMeterWiki

is that something you can use??
I asked for the search option like the 1 (you showed above) for Rotten Tomaties, and the admin said the API could do that.

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

I'll go see if i can figure out how that search works! I'd love to have that option

Let me know, I just went through all the scrapers, and I'll be able to add support for Swedish and Icelandic, as their respective movie websites (filmtipset.se and kvikmyndir.is) support direct IMDb tt-ID references.

API - MovieMeterWiki

is that something you can use??
I asked the admin for the search option like the 1 (you showed above) for Rotten Tomaties, and he said the API could do that.

Eventhough I could use the API output without a problem, it is not possible for me to interact with the API the way the scraper scripts work. This is why the existing MovieMeter scraper that MovingPictures uses is not an XML based scraper like IMDb+, but a C# based one with more options.

If there was someway for me to initiate the XML output via http://www.moviemeter.nl/ws?api_key=blablabla&imdb=0499549 then it would be no problem. Just like it is no problem with RottenTomatoes or as shown in the following Avatar movie examples with filmtipset.se and kvikmyndir.is:

Swedish lookup of Avatar via IMDb tt-ID -- This has HTTP output that I can scrape from
Icelandic lookup of Avatar via IMDb tt-ID -- This one has XML output, simelair to what I expect MovieMeter provides

So there would be three solutions to the problem (in order of preference).

1). MovieMeter developers explain how I can use a GET method to obtain the XML results, as per crude example: http://www.moviemeter.nl/ws?api_key=blablabla&imdb=0499549
2). With MovieMeter developers permission I setup a PHP proxy on a seperate server to do the above, which should not be a problem considering they show a PHP example on their wiki.
3). I talk with MovPic developer who created MovieMeter C# scraper, to see if there is a way I can use that one indirectly or via a modified version. This will however mean modifications to MovPic, so more then likely will not get approved.

So option 1 would be best/easiest, but I could do option 2 if needed. Check otherwise if the MovieMeter developers have instant messaging contact info (you can private message me their MSN/Live/Yahoo/AIM/ICQ info), or can hop on Freenode IRC #moving-pictures channel. There would be a lot of Q&A, so e-mail is not very effective then.

 

[HomeY]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

I've sent a PM to their tech (Jordy), explaining what you're trying to accomplis, and refrerred to these posts. Let's hope he'll jump in to give you some support so we can get the Dutch language in there also!

In the meantime i've installed the script and did a clean dB lookup and got a question about it, since renaming isn't working and i've noticed some strange scores.
My Scraper options are set to single scores (because of the comma thingy).
Settings:

<imdbplus>
	<set id="01" name="global_options_original_title"	value="false" />
	<set id="02" name="global_options_foreign_title"	value="true" />
	<set id="03" name="global_options_foreign_first"	value="true" />
	<set id="04" name="global_options_uk_rating"		value="false" />
	<set id="05" name="global_options_imdb_score"		value="false" />
	<set id="06" name="global_options_imdb_metascore"	value="false" />
	<set id="07" name="global_options_long_summary"		value="false" />
	<set id="08" name="global_options_rotten_meter"		value="false" />
	<set id="09" name="global_options_rotten_average"	value="false" />
	<set id="10" name="global_options_rotten_top_critics"	value="false" />
	<set id="11" name="global_options_special_edition"	value="true" />
	<set id="12" name="global_options_rename_titles"	value="true" />
	<set id="13" name="global_options_single_score"		value="true" />
	<set id="14" name="global_options_min_imdb_votes"	value="false" />
	<set id="15" name="global_options_refresh_all_fields"	value="false" />

	<set id="98" name="global_options_country_filter"	value="us|ca|gb|ie|au|nz" />
	<set id="99" name="global_options_language_filter"	value="en" />
</imdbplus>

'Options IMDb+ Scraper' & 'Rename dBase IMDb+ Scraper' are in root of "C:\"
Getting strange scores (with Artificial Intelligence: AI (2001) for example) and in logs it shows that there still is a rotten tomatoes lookup. AI seems to be divided by 2 (since the score i'm getting is a 3,5)

** EDIT **

LoL, after rescanning i'm getting a 5,8 (that's already closer to the 7 on imdb)
But now it the scanner starts to hang an manual approval for some reason. Time to start over with a clean dB.

 

[Merlyn]

AW: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

the moviemeter.nl api has a function to search by imdb id: film.retrieveByImdb

Not quite sure though, what the exact syntax is... The wiki looks like you have to send a xml file to the api to then get the results back... pretty confusing... might be google translater... Oh and you need to get an API key which again you need to keep private...

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

the moviemeter.nl api has a function to search by imdb id: film.retrieveByImdb

Not quite sure though, what the exact syntax is... The wiki looks like you have to send a xml file to the api to then get the results back... pretty confusing... might be google translater... Oh and you need to get an API key which again you need to keep private...

Yes, I would need to convert the XML based IMDb+ scraper into a C# one, just like what was done to the actual MovieMeter scaper. However then it requires a new release of MovingPictures when changes are made to the scraper and my scraper updates too many times to risk that.

LoL, after rescanning i'm getting a 5,8 (that's already closer to the 7 on imdb)
But now it the scanner starts to hang an manual approval for some reason. Time to start over with a clean dB.

You indeed enabled the single_score option, but you did not use:

<set id="05" name="global_options_imdb_score" value="true" />

So that means the individual RottenTomatoes scores are used. Some of those scores are in the 5-star range, so I have to do math to them and multiply by 2 to get the 10-star rating that MovPic works with.

But adjust the above setting (if you have upgraded to MediaPortal v1.2, you can install the IMDb+ plugin to change this very easy) and you will be good to go.

 

[Merlyn]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

Yes, I would need to convert the XML based IMDb+ scraper into a C# one, just like what was done to the actual MovieMeter scaper. However then it requires a new release of MovingPictures when changes are made to the scraper and my scraper updates too many times to risk that.

Indeed. I had to find a solution for that, too with my FilmInfo+ as themoviedb.org works with api key aswell. I solved that issue by creating a seperate php file on my own webspace, that takes the requests from the scraper and forwards it to the api and then hands back the results. One solution, maybe not the most elegant one...