- March 10, 2006
- 4,434
- 1,897
- Moderator
- #1
The IMDb+ scraper-script and plugin combination for Moving-Pictures is the first scraper to support custom options, allowing you to change the way this scraper works exactly the way you prefer. The options can be changed from within MediaPortal for full couch control of your movie collection.
NEW: There is now multi-language support for Title, Summary, Language, Genres and Certification for; Dutch (moviemeter.nl),German (imdb.de), French (imd.fr), Italian (imdb.it), Spanish (imdb.es), Portugese (imdb.pt), Icelandic (kvikmyndir.is) and Swedish (filmtipset.se). The title can also be forced in English via an additional option for those foreign users who prefer this. -- Amazon in their wisdom decided to kill all the localized versions of imdb.com website, and integrate this into their main website. Unfortunatly they have only done so for the title, and without IMDb tt-ID alternatives for those languages, my hands are still tied. If you absolutely want localized language, and keep the grouping feature of IMDb+, then import your collection with TMDb first, allow IMDb+ to take control and refresh your collection for just the title.
The most favorite feature of this scraper is not only to force English titles, but grouping series together. This is best shown visually, so on the left is the default imdb.com scraper result and on the right is the IMDb+ result with the rename option enabled:
» » » » » »
If you upgrade to MediaPortal v1.2, then you can install the IMDb+ plugin to adjust all the options in real-time. The plugin takes care of installing the IMDb+ scraper script into MovingPictures and will keep it up-to-date as well. For an easy way to get all this goodness, please visit the wiki Install Guide which uses pictures to explain everything as easy as possible.
The scraper forces English title on foreign movies. For example the movie Chin gei bin (2003) will be imported as 'Vampire Effect'.
To install the plugin:
================================================================================
================================================================================
Changelog:
Enjoy.
As the IMDb+ system has taken up a lot of my time already and will keep consuming a lot of my free time in the foreseeable future, I'm hoping that some of the more fortunate users amongst you are not opposed to a small donation:
NEW: There is now multi-language support for Title, Summary, Language, Genres and Certification for; Dutch (moviemeter.nl),
The most favorite feature of this scraper is not only to force English titles, but grouping series together. This is best shown visually, so on the left is the default imdb.com scraper result and on the right is the IMDb+ result with the rename option enabled:
NEW: Scraper version 3.3.0+ has a temporary fix included that does a math node test now to see if things are working correct. If it fails, it will disable any average or scoring system that requires math to be done. You can check the movingpictures.log file for information on what was disabled.
The new score stystem calculates an average score based on the IMDb score, Metacritics Metascore, RottenTomatoes All Critics, RottenTomatoes Top Critics and RottenTomatoes Audience score.
If you wish to return back to the old method for scores, then enable the 'Single Score' setting via the IMDb+ plugin.
The users who have their Windows system configured to use a 'comma' as floating point seperator, are getting errors due to the math part to calculate the average. Please use the 'Single Score' option for the time being on the IMDb or RottenTomatoes average critics scores, as no math is done on those.
The new score stystem calculates an average score based on the IMDb score, Metacritics Metascore, RottenTomatoes All Critics, RottenTomatoes Top Critics and RottenTomatoes Audience score.
If you wish to return back to the old method for scores, then enable the 'Single Score' setting via the IMDb+ plugin.
The users who have their Windows system configured to use a 'comma' as floating point seperator, are getting errors due to the math part to calculate the average. Please use the 'Single Score' option for the time being on the IMDb or RottenTomatoes average critics scores, as no math is done on those.
If you upgrade to MediaPortal v1.2, then you can install the IMDb+ plugin to adjust all the options in real-time. The plugin takes care of installing the IMDb+ scraper script into MovingPictures and will keep it up-to-date as well. For an easy way to get all this goodness, please visit the wiki Install Guide which uses pictures to explain everything as easy as possible.
The scraper forces English title on foreign movies. For example the movie Chin gei bin (2003) will be imported as 'Vampire Effect'.
- Enable 'Original Title' option via the IMDb+ plugin to import the same movie as "Chin gei bin".
- Enable 'Add Foreign Title' option for "Vampire Effect (Chin gei bin)".
- Enable 'Foreign Title First' option for "Chin gei bin (Vampire Effect)".
- Enable 'Add Foreign Title' option for "Vampire Effect (Chin gei bin)".
- Enable 'Foreign Title First' option for "Chin gei bin (Vampire Effect)".
Use the IMDb+ Plugin to enable the following options:
- Single Score
- Long Summary
- IMDb Score
If you enable 'IMDb Score', but not enable the 'Single Score' option then an average between imdb.com score and metacritics score will be used. The rename system to group movie series together by title is also very nice, so check out the benefits to enable it. And if anything else is not self-explanatory or explained in this first post, then please let me know, so I can improve the experience for the next user.
The IMDb+ plugin will detect if you have an existing movie collection with movies that got imported via the default IMDb scraper. If it detects that you have those in your collection, you can use the hidden menu on the left side to find the "Force IMDb+..." option.
This new option allows you to switch those movies over to the IMDb+ scraper, so that you can enjoy all the new benefits. Please see the wiki install guide for more information.
This new option allows you to switch those movies over to the IMDb+ scraper, so that you can enjoy all the new benefits. Please see the wiki install guide for more information.
To install the plugin:
- Open the "MediaPortal Extension Installer" and look for the "IMDb+" plugin under "Known extensions" or download the IMDb+ plugin MPEI package manually.
- Install the MPEI package
- Launch MediaPortal and go to the 'plugins' menu, so that you can adjust all the options to your liking.
- ALERT: If you do not find the IMDb+ plugin in the list, then I'm sorry, but you have then encountered a small bug in MediaPortal. It is being worked on to get it fixed, but in the meantime; simply open MediaPortal configuration and then press 'OK' to properly close the configuration. This should then correctly update the mediaportal.xml file to show the IMDb+ plugin. Please retry the previous step to confirm and verify if all the options are to your liking before proceeding.
- Then close MediaPortal, and open MovingPictures configuration to setup an import path and import movies, or refresh existing movies via the IMDb+ scraper.
================================================================================
Technical details on scraper:
- Gets high resolution covers from imdb.com.
- Solves non-English title problem for users outside of the US.
- Supports made-for-TV movies, mini-series and straight-to-video movies.
- Configurable options to change the behaviour of the scraper. Use the IMDb+ Plugin to adjust (MediaPortal v1.2).
- Enabling UK ratings uses British-English movie titles.
- Auto-rename support based on static XML file for default and custom entries
- If you are bothered by the tiny delay versus the default imdb.com scraper, then please switch to IMDb rating. This removes the extra step to get info from the RottenTomatoes website. You do miss out then on all the extras, such as the much nicer RottenTomatoes rating or the ability of this scraper to use the RottenTomatoes synopsis and/or runtime info when this information is not available on the imdb website.
- The title found during search node will not match the final title from the details node if a forced-English title conversion is taking place. This is only a nucance inside the "Movie Importer" tab during import of a new movie, and might lead to unexpected results where the "Possible Matches" end result after auto (or manual) approval does not match the "File(s)" input title. A solution for this problem caused severe delays to the import process (takes 3-5x longer). Please voice your opinion as a reply to this thread if you feel the delay is worth it or not.
- If you experience unexplained delays during import; try again with your anti-virus solution temporary disabled. If you then sent the same movies back to the importer and there is no delay, modify your anti-virus solution to ignore/white-list MediaPortal.
Changelog:
- v3.0.0 - Public release, the previous revisions were used internal.
- v3.0.1 - Fixed non-English title problem properly (no IP tricks anymore) as well as summary issue on some movies.
- v3.0.2 - Forgot to disable non-English title conversion for UK, Canada, Australia and New Zealand as they already show proper English titles (please let me know if I overlooked one).
- v3.0.3 - Added option for metascore from imdb.com, this requires "global_options_imdb_score" be set to "true" as well.
- v3.0.4 - Corrected 2-letter language code for Britain into GB, and added Norway (NO) to the blacklist.
- v3.0.5 - Rewrote the entire English title system. Hopefully this will solve any problems now.
- v3.0.6 - Fixed a few minor issues and enabled British-English movie titles when UK ratings are enabled.
- v3.0.7 - Per request of zicoz, added ability to retain the original title on movies created in certain languages. This allows a Dutch user to import "Black Book (2006)" as "Zwartboek" (meaning all Dutch spoken movies will use the Dutch movie title), while all other foreign-language spoken movies get imported with an English title.
- v3.0.8 - Fixed SortBy method to keep article removal intact. This way "Kinpeibai (1969)" gets imported as "The Concubines", but with a SortBy field of "concubines the" as per the default article removal settings.
- v3.0.9 - Per request of vpupkin, added "English (Foreign)" title support.
- v3.1.0 - Fixed a few bugs on rare titles, also added country filter to improve detection of foreign movies. Movie that would fail was for example Arthur and the Revenge of Maltazard (Arthur et la vengeance de Maltazard), which is a French title movie in English language. Filtering on country being USA wasn't enough, because then movies such as The Machine Girl (Kataude mashin gâru) would fail as being released in USA first, but having Japanese language. Unfortunatly this new method adds a small delay, but had to be done to prevent mistakes.
- v3.1.1 - Fixed rare AKA bug and improved speed in search node, Improved speed in cover node, and RottenTomatoes synopsis is now used when summary is missing from imdb.com with RT ratings enabled (default setting).
- v3.1.2 - Increased detection of English titles, added method to use RottenTomatoes runtime if missing from imdb.com and included an extra check to see if a new USA title was issued for an English movie. That way a movie like The Tomb (2009) imports correctly as "The Tomb" and not as the original "Ligeia" title.
- v3.1.3 - Added support for title manipulation to indicate special editions (3D, Unrated, Extended, etc). To make this work, your filenames have to contain this text between brackets as well as the IMDb tt-ID number (or in NFO). Auto-rename support is now included to retain any manual title changes after a refresh or re-import of your collection.
- v3.1.4 - Per suggestion of 'drealit', it is now possible to adjust the sortby title during the rename process, either by itself or alongside a title rename. You have to edit the rename XML file and add any sortby="..." values to the movies you wish to do this with. Or you can do a mass-replace on 'title=' into 'sortby=', which will leave the movie title intact as used by IMDb site, but will sort them together as a group. This will cause weird results in some cases, for example 'Casino Royale' will be sorted under the 'J' for "James Bond 21". This is why you can also rename both title and sortby title.
- v3.1.5 - Upon request of 'ninjatobbe' added the option to use "Foreign title first (English title)", and included improvement to get English title on Canadian released movies and some Italian released movies with English language tracks.
- v3.1.6 - Fixed imdb.com rating, also added support for "(Alternate Ending)" special editions and improved English language title detection.
- v3.1.7 - Fixed RottenTomatoes rating for users in foreign countries who would get localized RT page with different HTML code, as well as rounded average ratings on some movies where 3.0/5 would show as 3/5 and fail to get any.
- v3.2.0 - Internal version to test new improvements.
- v3.2.1 - Internal version to test new improvements.
- v3.2.2 - Improved English title detection. Correctly uses RottenTomatoes runtime info if it is missing from imdb.com details. Average score system for a nice balanced result. Option to have a minimum imdb vote amount of 20 to filter out bad ratings from movie staff voters. Added support for custom rename entries, so that it is easier to upgrade the default one. Option to only refresh empty + some fields that is enabled by default. And finally the global_options can now be adjusted on the fly via XML file in "C:\" root folder, to make it easier to try out different settings and retain your custom profile on a script update.
- v3.2.3 - Fixed: Adjusted code to compensate for the fact that over at imdb.com they decided that the movie titles needed to be prefixed with an "IMDb - " string.
- v3.2.4 - Fixed: Language due to change at imdb.com and a rare situation in which some short summaries would get skipped. Also made scraper ready for integration with new plugin for auto-update feature and optimized the regular expression code on all the movie fields.
- v3.2.5 - Fixed: RottenTomatoes critics scores, and XPATH errors in log files when not all XML files were present in 'C:\'. The special edition system is also modified and will work on imports without IMDb tt-ID as well now. Just remember that the search process itself will get confused by adding more text to the title, so be sure to modify your Advanced Settings -> Noise Filter to compensate. For example: [\(\[\{](?!(?:\d{4}|tt\d{7})).+?[\}\]\)] would filter out anything between brackets, except a 4-digit year or IMDb tt-ID. I'll eleborate more on this after I've ran more tests first.
- v3.3.0 - Fixed: Added loop limits to avoid delays on bad data. Modified rename system to make it much faster and use less resources. Added: Math node test to avoid errors. New option to limit writers and directors to just one name, to avoid horizontal scrolling in some skins. Simplified logging to make it easier to spot problems without needing scraper-debug mode.
Added: Multi-Language support for summary, spoken-language, genres, and certifications (German, French, Icelandic, Italian, Portugese, Spanish, and Swedish) with fallback option to use English summary when non-English one is missing. - v3.3.1 - Fixed: Tagline and English title detection.
- v3.3.2 - Fixed: In preperation for new IMDb+ plugin feature the rename title system is now always used to update a title even with update_all_fields disabled. This is because you will soon be able to mass-update not only your entire collection, but also refresh any movie that has an entry in the rename databases.
- v3.3.3 - Fixed: New faster Rename system failed on HTML encoded titles with characters such as '&'.
- v3.3.4 - Fixed: When foreign language is selected, the foreign title is now used. To refresh existing collection, keep in mind to enable the "Refresh all of the fields" setting as well, and disable the "Rename titles...." setting which would otherwise overwrite the foreign title again.
- v3.3.5 - Fix RottenTomatoes rare case, Fix imdb.com "TV Series" during search node and cleanup titles with single quotes.
- v3.3.6 - Added Dutch (moviemeter.nl) support, and prepared scraper-script for future updates.
- v3.3.7 - Improved English title detection system on foreign/asian movies.
- v3.3.8 - Workaround for MovPic ImdbBuilder not using IMDb tt-ID from filename when 'prefer foldername' setting is enabled and only the filename has the ID we want to use, and improved UK title detection.
- v3.3.9 - Fixed snafu in UK title detection that messed up some US titles.
- v4.0.0 - Skipped version due to bug in MovingPictures.
- v4.0.1 - Added full foreign title support, as well as ability to use English titles with foreign details. The Roman numerals for first movie in a series can now also be removed, as well as using user-reviews when there is no summary.
- v4.0.2 - Fixed Spanish title problem by adding new blacklisted entries.
- v4.0.3 - Added 'fake working title' to the blacklist of English title detection.
- v4.0.4 - Added a bunch more English title detection postfixes to the blacklist.
- v4.0.5 - Fixed Roman numeral strip system when Part or Volume was involved, and added more English title detection postfixes to the blacklist.
- v4.1.0 - Skipped due to bug in MovingPictures.
- v4.1.1 - Added initial support for other English rating systems (AU/CA/NZ) and fixed bad bug in previous code (darn XML encoding).
- v4.1.2 - Fixed problem with new option when it does not yet exist inside options XML file.
- v4.1.3 - Fixed problem with XML encoding of '&' char for "Director's Cut" blacklist title detection.
- v4.1.4 - Added 'alternative spelling' in a double way to the blacklist title detection.
- v4.1.5 - Allow for 6-digit IMDb tt-IDs in filenames.
- v4.1.6 - Fix Writers, Directors, Actors, Genre, Year and Language due to changes from imdb.com website.
- v4.1.7 - Fix English title detection system due to changes from imdb.com website.
- v4.1.8 - Fixed English title detection system proper this time (crossing fingers), also added in safety checks on other RegExps.
- v4.1.9 - Fixed Studio due to changes at IMDb website.
- v4.2.0 - Skipped due to bug in MovingPictures.
- v4.2.1 - Added custom 'Special Edition' tagging system via ([...]).
- v4.3.1 - Adjusted the way foreign titles are handled in default rename database so that they respect the options for it.
- v4.3.2 - Speed improvement on the rename database lookups.
- v4.3.3 - Adjusted language declaration to 'various'.
- v4.3.4 - Fixed short summaries issues when it had a direct link to an actor.
- v4.3.5 - Remove 'See Full Summary' reference introduced by previous fix.
- v4.3.6 - Fixed 'Genre' problem for Swedish secondary language due to changes at Filmtipset.se website.
- v4.3.7 - Fixed user review alternative for missing summary not getting HTML tags stripped.
- v4.3.8 - Blacklisted "(promotional abbreviation)" for English title detection.
- v4.3.9 - Fixed search node. (and RottenTomatoes improvements to supplement missing info (summary/runtime/etc)
- v4.4.0 - Weird bug in IMDb new search system were "title+(year)" can fail, added second search with just "title".
- v4.4.1 - IMDb search uses UTF-8 encoding now
- v4.4.2 - Fixed Icelandic language by forcing correct encoding
- v4.4.3 - Adjusted regular expression to match new IMDb results and ensured the no-year search works correct
- v4.4.4 - Fixed foreign language issues for sources that have unsafe headers
- v4.4.5 - Fixed Icelandic genres
- v4.4.6 - Fixed mini-series support
- v4.4.7 - Fixed Geo-IP search results and added additional logging
- v4.4.8 - Improved AKA auto approval results at the cost of an additional scrape
- v4.4.9 - Compensating for Moving-Pictures weird new auto-approval behavior
- v4.5.1 - IMDb added link tracking for Directors/Writers/Actors, had to fix RegExp to compensate
- v4.5.2 - Increased timeout for foreign details from default 5s to 10 seconds to mainly compensate for Dutch MovieMeter site
- v4.5.3 - Fixed a lot of other things that related to the new IMDb tracking system as well
- v4.5.4 - Small tweaks to avoid HTTP 302 redirects
- v4.5.5 - Fixed bug introduced by previous tweak and discovered user reviews summary fallback was also broken
- v4.5.6 - Fixed small bug related to actors, and added temporary workaround for foreign users not getting English titles
- v4.7.1 - Fixed foreign users not getting English titles for real this time (crossing fingers)
- v4.7.2 - IMDb broke search node with HTML source changes, all fixed again
- v4.7.3 - The fun with IMDb changing their HTML code never stops
- v4.7.4 - IMDb uses different HTML code for actors for European users, so made both methods work
- v4.7.5 - Temporary disable of DE/ES/IT/FR/PT, as IMDb has disabled their respective localized sites
- v4.7.6 - Genre translation for DE+FR added and fixed XML bug. ES/IT/PT languages will follow soon
- v4.7.7 - Genre translation for ES+IT added, only PT language left now
- v4.7.8 - Fixed Spanish titles getting used as English ones
- v4.7.9 - *FacePalm* typo on width= when it should be with=, that is what I get for doing HTML all day and then making adjustments to IMDb+
- v4.8.1 - Fixed 'Music' genre getting string replaced correct for DE/ES/IT/FR/PT system
- v4.8.2 - Fixed RegExp character conflict as well as tweaked code on arrays and foreign IMDb summary bug
- v4.8.3 - The UTF-8 format on the scraper-script got lost somewhere
- v4.8.4 - Adjusted RegExp to fix long summary (and repair versioning mishap on previous commit)
- v4.8.5 - Added support for Plot Keywords that will be added to next MovPic release
- v4.8.6 - Fixed cover node and added extra logging
- v4.8.7 - Fixed summaries due to new "synopsis" term used by IMDb
- v4.8.8 - Added support for new Moving-Pictures Collections system
- v4.8.9 - Forgot to enclose StringList values to stay consistent
- v4.9.1 - Fixed collections conflict with optional stripping of Roman numerals for 1st movie in a series
- v4.9.2 - Fixed studios, runtime and score/votes by compensating for new IMDb HTML code
- v4.9.3 - Optimized collections to merge remakes/reboots/etc
- v4.9.4 - Added initial support for new Moving-Pictures 'release date' feature
- v4.9.5 - Fixed bug for custom rename collections and use new 'date' function for release_date
- v4.9.6 - Double enclosed quotes got me again
- v4.9.9 - Auto collection system for "IMDb Top 250"
- v4.9.10 - Taking out :date function, as it is not working correct
- v4.9.11 - Took out one character too much when taking out :date function
- v4.9.12 - Fixed regexp to deal with collections for "Kill Bill" and "The Godfather"
- v4.9.13 - Added support for collections derived from decimal numbered series
- v4.9.14 - Fixed single digit IMDb+ scores
- v4.9.15 - Mass RegExp updates required after Amazon changed the HTML code on AKA page
- v4.9.16 - Tweaked RegExp for AKA results
- v4.9.17 - Added new English title detection method for non-USA users
- v4.9.18 - Re-issue to see if it will fix auto-update for some users
- v4.9.19 - Quick fix to deal with swap of title/country for foreign users code
- v4.9.20 - Updated all the RegExps for English title detection to match the new Amazon/IMDb style
- v4.9.21 - New Amazon/IMDb style forces me to add check for empty title situations
- v4.9.22 - Fixed AKA regexp to swap country/title and added "(promotional title)" to the blacklist
- v4.9.23 - Added 'informal title' to the new blacklist
- v4.9.24 - Prevent runaway parsing expression if options XML file becomes corrupted
- v4.9.25 - Fixed collections XML/HTML encoding and made options XML system more robust
- v4.9.26 - Fixed special edition detection
- v4.9.27 - Fixed special edition detection (the proper way)
- v4.9.28 - Added Portugese genres and additional black-list entries for English title detection.
- v4.9.29 - Fixed cover node, and 'Short' translation for some languages
- v4.9.30 - Added new blacklist entries to improve English title detection
- v4.9.33 - Temporary fix for Amazon/IMDb stupidity in adding a localized/translated title as another "(original title)" reference.
- v4.9.34 - Test to Force English language
- v4.9.35 - Reverted changes due to failing features (needs more investigation)
- v4.9.36 - Fixed IMDb Top 250 custom category
- v4.9.37 - Dirty hack to fix special tag rename bug caused by MovPic code change
- v4.9.38 - Proper fix for special tag rename bug
- v4.9.39 - Fixed IMDb Top 250 custom category
- v4.9.40 - Add additional exception for empty plots
- v4.9.41 - Fix Swedish imports
- v4.9.42/43/44 - RottenTomatoes JSON conversion
- v4.9.45 - Temporary fix for non-USA imports
- v4.9.46 - Fix AKA
- v4.9.47 - Fix AKA array
- v4.9.48 - Fix foreign AKA
- v4.9.49 - Fix RottenTomatoes RegExp
- v4.9.50 - Fix IMDb+ Options XML parsing check
- v4.9.51 - Accept votes >= 1 million
Enjoy.
Normally the first movie does not need to be renamed, except in the case of article removal movies, this is because the group ordering is done on SortBy title. So "The Matrix" would end up with sortby title of "matrix the", which would order it below the 2nd and 3rd movie in the group. You can now also edit the sortby field seperate from the title if you wish.
Note: If you are going to rename/add entries to the custom XML file, please remember to use proper XML format. For example the '&' character needs to be written out as '&'. You can verify that the correct XML syntax was used by copy and pasting your file contents into the tester at: XML DOM - Validate XML
The list is now available via the Google Code project SVN @ rename XML database view with only showing the changes. You can use it to quickly see all the new additions and changes to exiting entries. Near the top in the middle you can switch the drop down selection from 'Single-column' to 'Side by side' to see all entries. Near the top at the right side you can adjust the revision as well to look at older versions, but also newer ones incase I forgot to update the link on here
Note: If you are going to rename/add entries to the custom XML file, please remember to use proper XML format. For example the '&' character needs to be written out as '&'. You can verify that the correct XML syntax was used by copy and pasting your file contents into the tester at: XML DOM - Validate XML
The list is now available via the Google Code project SVN @ rename XML database view with only showing the changes. You can use it to quickly see all the new additions and changes to exiting entries. Near the top in the middle you can switch the drop down selection from 'Single-column' to 'Side by side' to see all entries. Near the top at the right side you can adjust the revision as well to look at older versions, but also newer ones incase I forgot to update the link on here
As the IMDb+ system has taken up a lot of my time already and will keep consuming a lot of my free time in the foreseeable future, I'm hoping that some of the more fortunate users amongst you are not opposed to a small donation:
If you do not have the means and still want to show your appreciation for this scraper, use the 'Thanks' button below
New download location @ IMDb+ plugin for MediaPortal's MovingPictures plugin
New download location @ IMDb+ plugin for MediaPortal's MovingPictures plugin
Last edited:
United States of America
