Dutch language scraping support (1 Viewer)

HomeY

Test Group
  • Team MediaPortal
  • February 23, 2008
    6,475
    4,645
    49
    ::1
    Home Country
    Netherlands Netherlands
    Quick question: Is it possible with this scraper to take all movie info from IMDB, but to use another language (moviemeter scraper for example for Dutch) for the summary?
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    Quick question: Is it possible with this scraper to take all movie info from IMDB, but to use another language (moviemeter scraper for example for Dutch) for the summary?

    Yes, this is possible, if the movie site supports a direct lookup based on imdb.com tt-ID. I know moviemeter has a very complete database of IMDb tt-IDs, but I'm unable to find out how to search on it.

    For example RottenTomatoes allows me to use:

    Code:
    http://www.rottentomatoes.com/alias?type=imdbid&s=0499549

    And it will retrieve the RottenTomatoes details page for the Avatar movie for me, without any fuzzy search logic mistakes. Tell me how to do the same with moviemeter.nl and I can add anything that sites gives me into the IMDb+ scraper, be it summary, tagline, etc.

    Thinking about it, I'll expand the scraper to support German, French, Italian, Spanish and Portugese, because IMDb has a localized website in those languages, so I can use any information from them easy based on IMDb tt-ID. I'll also scan all the existing scraper scripts, to see if any of them support a direct IMDb tt-ID search method, so I can include support for those languages as well.
     

    HomeY

    Test Group
  • Team MediaPortal
  • February 23, 2008
    6,475
    4,645
    49
    ::1
    Home Country
    Netherlands Netherlands
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    I'll go see if i can figure out how that search works! I'd love to have that option ;)
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    I'll go see if i can figure out how that search works! I'd love to have that option ;)

    Let me know, I just went through all the scrapers, and I'll be able to add support for Swedish and Icelandic, as their respective movie websites (filmtipset.se and kvikmyndir.is) support direct IMDb tt-ID references.

    So I will be able to add support for 7 languages, but if you want Dutch, find me a way to do the same on moviemeter.nl website. I even tried to an advanced google "site:moviemeter.nl link:/tt0499549" search, but it looks like the webmaster of moviemeter blocked that ability in their robot definitions.
     

    HomeY

    Test Group
  • Team MediaPortal
  • February 23, 2008
    6,475
    4,645
    49
    ::1
    Home Country
    Netherlands Netherlands
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    I'll go see if i can figure out how that search works! I'd love to have that option ;)

    Let me know, I just went through all the scrapers, and I'll be able to add support for Swedish and Icelandic, as their respective movie websites (filmtipset.se and kvikmyndir.is) support direct IMDb tt-ID references.

    API - MovieMeterWiki

    is that something you can use??
    I asked for the search option like the 1 (you showed above) for Rotten Tomaties, and the admin said the API could do that.
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    I'll go see if i can figure out how that search works! I'd love to have that option ;)

    Let me know, I just went through all the scrapers, and I'll be able to add support for Swedish and Icelandic, as their respective movie websites (filmtipset.se and kvikmyndir.is) support direct IMDb tt-ID references.

    API - MovieMeterWiki

    is that something you can use??
    I asked the admin for the search option like the 1 (you showed above) for Rotten Tomaties, and he said the API could do that.

    Eventhough I could use the API output without a problem, it is not possible for me to interact with the API the way the scraper scripts work. This is why the existing MovieMeter scraper that MovingPictures uses is not an XML based scraper like IMDb+, but a C# based one with more options.

    If there was someway for me to initiate the XML output via http://www.moviemeter.nl/ws?api_key=blablabla&imdb=0499549 then it would be no problem. Just like it is no problem with RottenTomatoes or as shown in the following Avatar movie examples with filmtipset.se and kvikmyndir.is:

    Swedish lookup of Avatar via IMDb tt-ID -- This has HTTP output that I can scrape from
    Icelandic lookup of Avatar via IMDb tt-ID -- This one has XML output, simelair to what I expect MovieMeter provides

    So there would be three solutions to the problem (in order of preference).

    1). MovieMeter developers explain how I can use a GET method to obtain the XML results, as per crude example: http://www.moviemeter.nl/ws?api_key=blablabla&imdb=0499549
    2). With MovieMeter developers permission I setup a PHP proxy on a seperate server to do the above, which should not be a problem considering they show a PHP example on their wiki.
    3). I talk with MovPic developer who created MovieMeter C# scraper, to see if there is a way I can use that one indirectly or via a modified version. This will however mean modifications to MovPic, so more then likely will not get approved.

    So option 1 would be best/easiest, but I could do option 2 if needed. Check otherwise if the MovieMeter developers have instant messaging contact info (you can private message me their MSN/Live/Yahoo/AIM/ICQ info), or can hop on Freenode IRC #moving-pictures channel. There would be a lot of Q&A, so e-mail is not very effective then.
     

    HomeY

    Test Group
  • Team MediaPortal
  • February 23, 2008
    6,475
    4,645
    49
    ::1
    Home Country
    Netherlands Netherlands
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    I've sent a PM to their tech (Jordy), explaining what you're trying to accomplis, and refrerred to these posts. Let's hope he'll jump in to give you some support so we can get the Dutch language in there also!

    In the meantime i've installed the script and did a clean dB lookup and got a question about it, since renaming isn't working and i've noticed some strange scores.
    My Scraper options are set to single scores (because of the comma thingy).
    Settings:
    Code:
    <imdbplus>
    	<set id="01" name="global_options_original_title"	value="false" />
    	<set id="02" name="global_options_foreign_title"	value="true" />
    	<set id="03" name="global_options_foreign_first"	value="true" />
    	<set id="04" name="global_options_uk_rating"		value="false" />
    	<set id="05" name="global_options_imdb_score"		value="false" />
    	<set id="06" name="global_options_imdb_metascore"	value="false" />
    	<set id="07" name="global_options_long_summary"		value="false" />
    	<set id="08" name="global_options_rotten_meter"		value="false" />
    	<set id="09" name="global_options_rotten_average"	value="false" />
    	<set id="10" name="global_options_rotten_top_critics"	value="false" />
    	<set id="11" name="global_options_special_edition"	value="true" />
    	<set id="12" name="global_options_rename_titles"	value="true" />
    	<set id="13" name="global_options_single_score"		value="true" />
    	<set id="14" name="global_options_min_imdb_votes"	value="false" />
    	<set id="15" name="global_options_refresh_all_fields"	value="false" />
    
    	<set id="98" name="global_options_country_filter"	value="us|ca|gb|ie|au|nz" />
    	<set id="99" name="global_options_language_filter"	value="en" />
    </imdbplus>

    'Options IMDb+ Scraper' & 'Rename dBase IMDb+ Scraper' are in root of "C:\"
    Getting strange scores (with Artificial Intelligence: AI (2001) for example) and in logs it shows that there still is a rotten tomatoes lookup. AI seems to be divided by 2 (since the score i'm getting is a 3,5)

    ** EDIT **

    LoL, after rescanning i'm getting a 5,8 (that's already closer to the 7 on imdb) :D
    But now it the scanner starts to hang an manual approval for some reason. Time to start over with a clean dB.
     

    Merlyn

    Portal Pro
    July 8, 2011
    250
    322
    Home Country
    Germany Germany
    AW: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    the moviemeter.nl api has a function to search by imdb id: film.retrieveByImdb

    Not quite sure though, what the exact syntax is... The wiki looks like you have to send a xml file to the api to then get the results back... pretty confusing... might be google translater... Oh and you need to get an API key which again you need to keep private...
     

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    the moviemeter.nl api has a function to search by imdb id: film.retrieveByImdb

    Not quite sure though, what the exact syntax is... The wiki looks like you have to send a xml file to the api to then get the results back... pretty confusing... might be google translater... Oh and you need to get an API key which again you need to keep private...

    Yes, I would need to convert the XML based IMDb+ scraper into a C# one, just like what was done to the actual MovieMeter scaper. However then it requires a new release of MovingPictures when changes are made to the scraper and my scraper updates too many times to risk that.

    LoL, after rescanning i'm getting a 5,8 (that's already closer to the 7 on imdb) :D
    But now it the scanner starts to hang an manual approval for some reason. Time to start over with a clean dB.

    You indeed enabled the single_score option, but you did not use:

    <set id="05" name="global_options_imdb_score" value="true" />

    So that means the individual RottenTomatoes scores are used. Some of those scores are in the 5-star range, so I have to do math to them and multiply by 2 to get the 10-star rating that MovPic works with.

    But adjust the above setting (if you have upgraded to MediaPortal v1.2, you can install the IMDb+ plugin to change this very easy) and you will be good to go.
     

    Merlyn

    Portal Pro
    July 8, 2011
    250
    322
    Home Country
    Germany Germany
    Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.4

    Yes, I would need to convert the XML based IMDb+ scraper into a C# one, just like what was done to the actual MovieMeter scaper. However then it requires a new release of MovingPictures when changes are made to the scraper and my scraper updates too many times to risk that.

    Indeed. I had to find a solution for that, too with my FilmInfo+ as themoviedb.org works with api key aswell. I solved that issue by creating a seperate php file on my own webspace, that takes the requests from the scraper and forwards it to the api and then hands back the results. One solution, maybe not the most elegant one...
     

    Users who are viewing this thread

    Similar threads

    Check this web page for the informations about he plugin. To fix the issue, I recommend to uninstal and install the plugin again (if you need it).
    Check this web page for the informations about he plugin. To fix the issue, I recommend to uninstal and install the plugin again...
    I've just installed MP 1.31 with Streamed skin, IMDB+ plugin, Moving Pictures and MP-TV Series. First time starting up MP to...
    Replies
    5
    Views
    471
    Fresh install of MP with Moving Pictures and IMDB+ (newest versions) on a new Windows 10 machine. Started scraping part of my library. I'm used to a relatively smooth and fast process. But now it was EXTREMELY slow. Also many titles that always gets identified by IMDB+ now were not. Themoviedb.org instead provided details. At first...
    Fresh install of MP with Moving Pictures and IMDB+ (newest versions) on a new Windows 10 machine. Started scraping part of my...
    Fresh install of MP with Moving Pictures and IMDB+ (newest versions) on a new Windows 10 machine. Started scraping part of my...
    Replies
    0
    Views
    451
    restarting is supported but depends on the reason why it failed. As an external tool try JDownloader.org - Official Homepage
    restarting is supported but depends on the reason why it failed. As an external tool try JDownloader.org - Official Homepage
    Earlier this year I changed my internet service provider from a landline (where I was consistently overcharged), to a mains-powered...
    Replies
    1
    Views
    584
    • Sticky
    Your log clearly reports: [2024-03-03 18:26:58,258] [Log ] [MPMain ] [DEBUG] - Main: Wait for TV service requested [2024-03-03 18:26:58,262] [Log ] [MPMain ] [DEBUG] - Main: TV service found. Checking status... [2024-03-03 18:26:58,263] [Log ] [MPMain ] [INFO ] - Main: TV service is in status Running - proceeding...
    Your log clearly reports: [2024-03-03 18:26:58,258] [Log ] [MPMain ] [DEBUG] - Main: Wait for TV service requested...
    MediaPortal 1.33 Release Written by Team-MediaPortal. We have just released MediaPortal 1.33 - Springtime / 20th Anniversary...
    Replies
    7
    Views
    761
    I have all of my media on a NAS. I guess it might just be a network issue, then.
    I have all of my media on a NAS. I guess it might just be a network issue, then.
    Whenever I go into the back end for Moving Pictures, it almost immediately hangs on the Movie Importer tab. If I want to go into...
    Replies
    4
    Views
    364
    Top Bottom