IMDb Scraper with RottenTomatoes rating (check end of thread for final versions)

Discussion in 'Moving Pictures' started by RoChess, January 3, 2010.

  1. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,153
    Likes Received:
    1,294
    Ratings:
    +1,659 / 2
    • Like Like x 7
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. Surferosa

    Surferosa Portal Pro

    Joined:
    September 2, 2009
    Messages:
    55
    Likes Received:
    5
    Ratings:
    +5 / 0
    Home Country:
    England England
    Re: IMDb Scraper with RottenTomatoes rating

    Going to give this a pop now- many thanks RoChess.

    :D

    OK- I may be doing something incredibly stupid- but I'm getting the error 'The Script is malformed or not a Moving Pictures script'. Get this error trying to load either file after selecting it from my download location.

    Using latest MP Beta (1.0.1.1001) with MediaPortal 1.1 (though it says 1.0.4.23491 in the 'About' part of the config)
     
  4. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,153
    Likes Received:
    1,294
    Ratings:
    +1,659 / 2
    Re: IMDb Scraper with RottenTomatoes rating

    Figures, I screwed up even the most basic code :oops:

    The & char used in the URL should have been & and I never noticed it when writing the code.

    All fixed now, and this time I made sure to test it, both scripts correctly imported the RottenTomatoes rating on a test movie.

    I used "Sherlock Holmes [2009].avi" and instead of the 7.7 IMDb.com rating, it perfectly took the 6.9 and 6.2 scores for the TomatoMeter and Average rating scripts.

    To make it easier, I've also added the scripts to the first post for download.

    :sorry: for the messup and :D for reporting it.
     
  5. Surferosa

    Surferosa Portal Pro

    Joined:
    September 2, 2009
    Messages:
    55
    Likes Received:
    5
    Ratings:
    +5 / 0
    Home Country:
    England England
    Re: IMDb Scraper with RottenTomatoes rating

    Testing the average rating version and it seems to be working a treat.

    Currently now in the process of testing it against 500+ movies- Ill let you know how I get on.

    Many thanks for this- its really useful and makes the scoring MUCH more meaningful. In case you / anyone else picks this up- if there was anyway of grabbing a 'review' (as per the original thread) that would be ace- though Im guessing this is db config + scraper? Either way- really pleased with this.

    Thanks again!
     
  6. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,153
    Likes Received:
    1,294
    Ratings:
    +1,659 / 2
    Re: IMDb Scraper with RottenTomatoes rating

    So you want the IMDB summary replaced with a RottenTomatoes review?

    Because I could simply modify the average rating scraper I've made to do just that and make a special version for you.

    If you are thinking about an extra option, then that would mean that MovingPictures plugin would need to be modified to support reviews. However there are some unused TEXT fields right now. For example the 'Tagline' TEXT field isn't used by many skins, so it would be possible to put the review text into this one and then adapt your skin to show this under a 'RT Review' option. But then you will first have to find a skin expert to do this for you, as my skin knowledge is limited right now (working on it though).

    We are heading towards a very custom solution then, but it would work and wouldn't be much work on the scraper side, so I don't mind doing this for you (as there is bound to be somebody else that might find it useful). All I ask is that you find some expert to also modify the Blue3Wide skin then, as I wouldn't mind this functionality myself as well :D

    In the meantime you could add a Google SVN issue to request an enhancement to allow for reviews.
     
  7. Surferosa

    Surferosa Portal Pro

    Joined:
    September 2, 2009
    Messages:
    55
    Likes Received:
    5
    Ratings:
    +5 / 0
    Home Country:
    England England
    Re: IMDb Scraper with RottenTomatoes rating

    Sure- no problem adding a request.

    When I'm deciding what to watch (I have around 150+ unwatched movies at present), we tend to go through a process of looking through all the unwatched and reading the descriptions. I'll then pull up the RT page- if the score looks alright I may read some of the critics from the saved 'My Critics' tab. When I wrote the original request I had in mind MP emulating this behaviour. Now the score is fixed, if it could pull up a critics review from your saved 'My Critics' (in RT) then great- of not, maybe the 1st one it finds.

    I had thought that this would work as an additional information panel accessible from the 'i' menu- which as you say would mean config both in MedPortal and the skin. However, your solution seems much more achievable (and if you need a tester- happy to run with it)- though no expectations of osmething specific designed for me: Im sure there would be plenty of interest in this from a few people.

    As an aside, your new scraper that I ran in batch mode worked for around 90% of the titles for me. Some titles seem to have got skipped though (21, 300, Aliens). Of these, some updated fine when I reran manually (21)- others are simply refusing to update (300, Aliens). Can't work out way- I did wonder whether the scraper was using the imdb search link within the imdb page? (as per comment 2 by bgmeiner in Issue 570). If so, I can't fathom why it wouldn't work. If there some sort of log file that you want posting let me know.

    Cheers.
     
  8. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,153
    Likes Received:
    1,294
    Ratings:
    +1,659 / 2
    Re: IMDb Scraper with RottenTomatoes rating

    That's exactly the procedure.

    So let's take '300' as an example.

    • The existing imdb.com scraper will locate: h**p://www.imdb.com/title/tt0416449/ and get all the information from it. So please verify when you go into your Movie Details, that '300' has tt0416449 in the imdb_id field.
    • At the very end of the imdb scraper, my little bit of code kicks in and uses that number, strips it of the 'tt', so it ends up with 0416449.
    • Then it loads up: h**p://www.rottentomatoes.com/alias?type=imdbid&s=0416449
    • Which then in turn ends up giving: h**p://www.rottentomatoes.com/m/300/
    • And from this page, I extract the Average rating (or the TomatoMeter one with the other version).

    Since the entire HTML page from "h**p://www.rottentomatoes.com/m/300/" is obtained, it would not be difficult at all to use any other information as well. For example you might prefer the "Synopsis" from RottenTomatoes (click on [More] first) compared to the 'Summary' that IMDb provides. Or use the "Consensus" for the 'Tagline'.

    There is one big problem with the reviews, and that is that the full reviews are on a seperate page, adding even more URL requests to be scraped, so I prefer to avoid that (would also make code more complex). However if you scroll down to the bottom, there is a wide range of short review burps. It would be possible to filter out only the real critic reviews (the one who have a photo) and join them together in a text format (don't think MovingPictures supports images inside the 'Summary').

    Infact it would even be possible to join/merge these RottenTomatoes reviews into the IMDb summary.

    So you get for example, something like this:

    Show Spoiler
    Code (Text):
    1.  
    2. RottenTomatoes Reviews (124 total with 106 fresh and 18 rotten):
    3. ----------------------------------------------------------------------------
    4. Tony Macklin = ROTTEN (02/02/09): 300 is a fun-fest of blood, mayhem, and absurdity. At its best, it is entertaining; when it's not at its best, it's pretty dumb. It's pretty dumb much of the time.
    5. ******************
    6. Roger Ebert = ROTTEN (08/08/08): 300 has one-dimensional caricatures who talk like professional wrestlers plugging their next feud.
    7. ******************
    8. Tricia Olszewski = ROTTEN (03/04/08): All this bellowing and testosterone gets old fast -- especially since there's not much of a plot outside of the combat scenes, and the not-much-of-a-plot scenes are laden with dialogue worthy of Anakin and Padmé.
    9. ******************
    10. Jeff Bayer = FRESH (03/03/08): The look and feel of this historic battle is perfect for the virtual backgrounds and obscene violence. Again, trust me on the violence.
    11. ******************
    12. Brandon Fibbs = FRESH (02/28/08): 300 is an orgy of gore, a blood-letting on a titanic scale, a ballet of butchery in which half-naked men and the torrents of blood they elicit move in perfect, slow-motion choreography to a thunderous soundtrack. And I loved every minute of it.
    13. ******************
    14. Luke Y. Thompson = FRESH (01/03/08): I still wish David Wenham weren't the narrator, but everything else about the movie is a brutal kind of hypnotic that keeps me coming back.
    15. ******************
    16. Brian Webster = ROTTEN (08/05/07): While far from sophisticated in its 'I have filled my heart with hate' messaging, it resonates with the 'us versus them' worldview that's wildly popular in some circles.
    17. ******************
    18. John J. Puccio = FRESH (07/17/07 ): ...relentless in its attempt to recreate the graphic novel's vision of the carnage at Thermopylae.
    19. ******************
    20.  
    21. IMDb Summary:
    22. -------------------
    23. (this is where existing summary of IMDb could go, but can also be placed first)
    24.  


    PS: If images are possible inside Summary field, then I could replace FRESH and ROTTEN with the same images that RottenTomatoes.com uses.

    Adding it to the existing Summary like the above example, is something I could make work without any skin modifications, but not sure if you like that idea. And do you prefer IMDb summary, or the RottenTomatoes synopsis. Also what you see is what you get, I've taken '300' as an example, but that's a lousy sampling rate to see if they all have photo reviews. So there is the risk of ending up with an empty addition because no positive match was found. I'm not going to waste much time making this fool proof for every possible combination (at least not at first), so you might want to manually check on RT to see if it would work.

    And you have to be expecting added delays, because all this extra scraper processing will end up costing CPU cycles.

    Let me know.
     
  9. kiwijunglist
    • Team MediaPortal

    kiwijunglist Super Moderator

    Joined:
    June 10, 2008
    Messages:
    6,741
    Likes Received:
    1,416
    Occupation:
    Doctor
    Location:
    New Zealand
    Ratings:
    +1,845 / 9
    Home Country:
    New Zealand New Zealand
    Show System Specs
    Re: IMDb Scraper with RottenTomatoes rating

    I guess what is needed is for moving pictures to support a review link on the movie details page. Then either the scraper downloads all the reviews when you import your movies, or alternatively there is a scraper that imports the movie review as a sort of live function when you click on the review link.

    This would go well when they introduce trailers, actor details etc.

    On movie details page you could have the following links

    Play Movie
    Watch Alternative Version (when moving pics add supports for directors cut, extended edition, making of etc)
    Watch Trailer (either local or online)
    Read Reviews (with reviews grabed from rotten tomatoes)
    List Actor(s) (Then click on actor to load either actor bio/pic or a category view with all movies done by that actor in your collection)
     
  10. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,153
    Likes Received:
    1,294
    Ratings:
    +1,659 / 2
    Re: IMDb Scraper with RottenTomatoes rating

    Yup, I've been able to get close to this by muddling away with the existing options, and try to help others who wish to get the same. But it's far from ideal. What I've done on my own setup is modify the noise filter to make the scraper ignore all the "(Director's Cut)" and "(Extended Edition)" parts on my filenames. The movies are then added with the same title, and via custom Genres I can then filter on filename to put those movies in their own group. But unless I manually edit the movies in the database, I'm stuck with double entries when I go to "All Movies", so I'll be looking forward to a default integration.

    Everytime I think my setup is ideal, somebody posts a request with something that I want as well :D
     
  11. kiwijunglist
    • Team MediaPortal

    kiwijunglist Super Moderator

    Joined:
    June 10, 2008
    Messages:
    6,741
    Likes Received:
    1,416
    Occupation:
    Doctor
    Location:
    New Zealand
    Ratings:
    +1,845 / 9
    Home Country:
    New Zealand New Zealand
    Show System Specs
    Re: IMDb Scraper with RottenTomatoes rating

    My post was not talking about category filters, it was about modifing the details page/moving pictures plugin.

    For what you have written, you can just add a filter for All Movies category to ignore any filenames that have "(Director's Cut)" in them, this will eliminate the double entries.

    Filename Does Not Contain - "(Director's Cut)"
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!