MP-4212 My Videos internal scrapers rework

Discussion in 'Area 51 - Testing Area' started by Deda, September 27, 2013.

  1. Deda
    • Team MediaPortal

    Deda Lead Dev MP1 Videos

    Joined:
    March 18, 2009
    Messages:
    2,423
    Likes Received:
    2,098
    Gender:
    Male
    Occupation:
    IT Consultant
    Location:
    Zagreb
    Ratings:
    +2,385 / 1
    Home Country:
    Croatia Croatia
    Show System Specs
    Description of Change
    Currently all internal scrapes which are used by MyVideos are hardcoded and hard to maintain if some changes occur on the Internet scraper source. With this rework they are all moved to a csscript files which can be updated as normal movie scrapers and maintained outside of MP project source. Their public methods are not removed or changed in MP core, so they will work as before with this change.

    Source code
    GIT link
    GIT branch commits

    JIRA Issue
    JIRA link

    Installer/Patch
    Installer - not needed, patched files are given for Mp 1.5.0 Final release

    Patched files - link for MP 1.5.0 release is in 2nd post (simple copy-paste-replace method with usual warning to backup original files first), link always contain newest version

    Testing
    1st step
    Install final MP 1.5 version and replace files provided from link in post 2 (backup originals).
    Update internal grabbers from MP web site via MP MyVideos database configuration or later in MyVideos Database GUI Settings because they can be changed independently of MP code source.

    2nd step
    Try covers (all 3 sources one by one), fanart and actor details scrapers from configuration on already existing movie in the database

    3rd step
    Try 2nd step in MP MyVideos plugin (scan one movie which is not in the database). Try to change covers with image up/down button.
    Click on cast button and on one actor to see actor details (for this I prefer DefaultWide skin because it contains all properties for actor info screen)

    4th step
    Manually delete internal grabbers from windows ( default location in Win7 -> C:\ProgramData\Team MediaPortal\MediaPortal\scripts)
    ScreenShot006.png
    and try to update them in GUI from MyVideos or MyVideos database GUI settings and test again 3rd step

    5th step
    Try again from step 1 :)

    Note
    For testing fanart you need to install FanartHandler plugin and enable fanart scraping
    or
    you can fool the system (to enable fanart scraping) and manually create fanart folder in
    C:\ProgramData\Team MediaPortal\MediaPortal\Thumbs\Skin FanArt\Scraper\movies
    and check if fanart files are there after movie scan

    Changelog

    Version 1.
    Source code based on current Master (1.5.0 release).

    Version 1.1
    Reworked code in cover change method for Movie Info screen (when used spin up/down button for cover change), changed dll file -> WindowPlugins.dll

    Version 1.2
    Added additional debug log, re-added folder.jpg change on cover change (only for DVD/DB folders)

    Version 1.3
    Internal scripts are now loaded and compiled only once per session (although updating scripts will reload and recompiled them again - tech limitation), changed files -> all except scripts



    Version 1.4
    Radically reduced all scripts compiling. Due to change, all scripts need to include new class inheritance so all working movie scripts need slight change as:
    Code (Text):
    1.  
    2. internal class Grabber : MarshalByRefObject, IIMDBScriptGrabber
    3. {
    4.  
    Movie scripts for MP 1.6+ are now downloaded from new MP Web location for easier maintenance, I copied working ones from old location to a new with change written above and to use new patched files either update scripts from web or copy ones provided in patch archive.
    All files are changed so you need to copy them all to appropriate place.
     
    Last edited: October 10, 2013
    • Thank You! Thank You! x 8
    • Like Like x 1
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. Deda
    • Team MediaPortal

    Deda Lead Dev MP1 Videos

    Joined:
    March 18, 2009
    Messages:
    2,423
    Likes Received:
    2,098
    Gender:
    Male
    Occupation:
    IT Consultant
    Location:
    Zagreb
    Ratings:
    +2,385 / 1
    Home Country:
    Croatia Croatia
    Show System Specs
    Patched files for Mp 1.5.0 Release

    MediaPortal folder content copy to Media Portal program folder (usually for Win7 in C:\Program Files\Team MediaPortal\MediaPortal). Backup original files first to some safe place.

    Scripts folder content copy to MP scripts folder (usually for Win7 in C:\ProgramData\Team MediaPortal\MediaPortal\scripts) or just update them from GUI or Configuration (new files are uploaded to the MP web site).
     
    Last edited: October 1, 2013
    • Thank You! Thank You! x 1
  4. Wo0zy
    • Team MediaPortal

    Wo0zy Retired Team Member

    Joined:
    April 30, 2008
    Messages:
    394
    Likes Received:
    128
    Gender:
    Male
    Ratings:
    +148 / 0
    Home Country:
    United Kingdom United Kingdom
    Hi Deda,

    Just tried applying the 1.5 patch files and when firing up MP Configuration I get the error "TVPlugin and Mediaportal don't have the same version....".

    I guess we need a patched TvPlugin.dll before we can test using this method.

    No biggie as I guess you'll be publishing binaries anyway now that 1.5 is released :):notworthy:.

    All the best.

    Wo0zy
     

    Attached Files:

    • Error.JPG
      Error.JPG
      File size:
      22.7 KB
      Uploaded:
      October 1, 2013
      Views:
      251
  5. Deda
    • Team MediaPortal

    Deda Lead Dev MP1 Videos

    Joined:
    March 18, 2009
    Messages:
    2,423
    Likes Received:
    2,098
    Gender:
    Male
    Occupation:
    IT Consultant
    Location:
    Zagreb
    Ratings:
    +2,385 / 1
    Home Country:
    Croatia Croatia
    Show System Specs
    Should be ok now, bad compiled file versions :)
     
    • Like Like x 1
    • Thank You! Thank You! x 1
  6. rogerleifert
    • Team MediaPortal

    rogerleifert Retired Team Member

    Joined:
    April 22, 2009
    Messages:
    239
    Likes Received:
    57
    Gender:
    Male
    Location:
    Aachen
    Ratings:
    +78 / 1
    Home Country:
    Germany Germany
    Show System Specs
    Hi Deda,

    I manually had to stop MPTray to copy all files to the Media Portal program folder.
    Apparently MPTray was using some of the files.
    After that everything works OK.

    THX

    Roger
     
    • Thank You! Thank You! x 1
  7. Wo0zy
    • Team MediaPortal

    Wo0zy Retired Team Member

    Joined:
    April 30, 2008
    Messages:
    394
    Likes Received:
    128
    Gender:
    Male
    Ratings:
    +148 / 0
    Home Country:
    United Kingdom United Kingdom
    Hi Deda,

    Tested as per instructions and everything seems to be working well.

    I did have one instance of "stuck" coverart (similar to what Catavolt reported elsewhere). This is where the movie details page showed that there were multiple covers (15 IIRC) but selecting a different one didn't trigger the image change.

    Fixed by deleting all images relating to that movie from the thumbs folder (C:\ProgramData\Team MediaPortal\MediaPortal\Thumbs\Videos\Title), going back into the movie details page in MP and clicking the up/down arrows until I found one I liked.

    Thanks again for the fix!.

    All the best,

    Wo0zy
     
  8. Deda
    • Team MediaPortal

    Deda Lead Dev MP1 Videos

    Joined:
    March 18, 2009
    Messages:
    2,423
    Likes Received:
    2,098
    Gender:
    Male
    Occupation:
    IT Consultant
    Location:
    Zagreb
    Ratings:
    +2,385 / 1
    Home Country:
    Croatia Croatia
    Show System Specs
    Well, plz provide debug log in case like that :). I put extra debug lines for that just to see what is the problem.
    In log, if cover change issuccessful, you should see something like this:

    Code (Text):
    1. [2013-10-01 10:25:59,849] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo: Refresh image: New Image url -> http://www.impawards.com/2013/posters/iron_man_three_ver12.jpg
    2. [2013-10-01 10:25:59,852] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo Refresh image: Downloading image, new Image url -> http://www.impawards.com/2013/posters/iron_man_three_ver12.jpg
    3. [2013-10-01 10:25:59,880] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo Refresh image: Creating new image -> C:\ProgramData\Team MediaPortal\MediaPortal\Thumbs\Videos\Title\Iron Man 3{9}.jpg
    4. [2013-10-01 10:26:00,557] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo Refresh image: Creating new image -> C:\ProgramData\Team MediaPortal\MediaPortal\Thumbs\Videos\Title\Iron Man 3{9}L.jpg
    5.  
    IMHO, I think that issue could be in image rendering (something stuck in GUI MP part or image is not released and redraw there)
     
    Last edited: October 2, 2013
    • Agree Agree x 1
  9. Wo0zy
    • Team MediaPortal

    Wo0zy Retired Team Member

    Joined:
    April 30, 2008
    Messages:
    394
    Likes Received:
    128
    Gender:
    Male
    Ratings:
    +148 / 0
    Home Country:
    United Kingdom United Kingdom
    It was on a work PC. Will remote in later and see if the logs are there. I hadn't set debug mode though so I guess I'll have to try and reproduce the issue again before the logs will be helpful :(.

    Cheers,

    Wo0zy.
     
  10. GaryMartynUK
    • Premium Supporter

    GaryMartynUK MP Donator

    Joined:
    June 12, 2012
    Messages:
    89
    Likes Received:
    48
    Gender:
    Male
    Location:
    Portsmouth
    Ratings:
    +77 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    Deda,

    This may be the same thing Wo0zy spotted.
    1. I 'scan' to add a movie. The scrappers do their job perfectly and grab artist, backdrop and cover images.
    2. When I access the movie's info page the screen updates to show there are multiple covers available.
    3. Then I scroll through the list of covers the on screen image for the cover does not change.
    4. If I exit MP and restart then access the movie's info page the cover images now update as I scroll through the list.
    The scrappers seem to do their job in the background as if I exit the movie info page on a different cover image number, when I exit MP and restart the new cover is shown for the movie. So it seems it is just a problem for a movie that was added while MP was running, as soon as I exit and restart the on screen image updates just fine. So I guess it is actually something in Movies plugin itself that is not updated on a newly added movie.

    Anyway I've included debug logs below in case they help.

    Actions for Log 1
    • Start MP
    • Enter Videos
    • run SCAN
    • access the movie info and scroll through the covers list then exit on a different cover (all the time the screen shows just the one image)
    • use backup button to exit from Videos to basic home screen then re-enter Videos and scroll through list again (again cover image does not update)

    Actions for Log 2
    • Start MP
    • Enter Videos (this time the updated cover image selected in Log 1 shows up)
    • access the movie info and scroll through the covers list (this time the cover image updates each time)

    I should probably add that I did this in the Titan skin (I guess it could be an element of the skin, I've not tried this in any other skin)

    But this tiny issue is certainly one I can live with.... so happy to have covers, backdrops and artists back. Thanks for the great work!
     
  11. Deda
    • Team MediaPortal

    Deda Lead Dev MP1 Videos

    Joined:
    March 18, 2009
    Messages:
    2,423
    Likes Received:
    2,098
    Gender:
    Male
    Occupation:
    IT Consultant
    Location:
    Zagreb
    Ratings:
    +2,385 / 1
    Home Country:
    Croatia Croatia
    Show System Specs
    Thx for logs, in log1 everything is ok on cover change:

    Code (Text):
    1.  
    2. // URL link is ok
    3. [2013-10-05 12:01:33,972] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo: Refresh image: New Image url -> http://cf2.imgobject.com/t/p/w500/4AKwDYbXoZlYZC4Z8UAUzFNHrjU.jpg
    4.  
    5. // Image is downloaded ok
    6. [2013-10-05 12:01:33,974] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo Refresh image: Downloading image, new Image url -> http://cf2.imgobject.com/t/p/w500/4AKwDYbXoZlYZC4Z8UAUzFNHrjU.jpg
    7.  
    8. // You will not see this if downloaded image do not exists, so all is ok
    9. [2013-10-05 12:01:33,977] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo Refresh image: Creating new image -> C:\ProgramData\Team MediaPortal\MediaPortal\Thumbs\Videos\Title\Keep the Lights On{902}.jpg
    10. [2013-10-05 12:01:34,067] [Log    ] [MPMain  ] [DEBUG] - GUIVideoInfo Refresh image: Creating new image -> C:\ProgramData\Team MediaPortal\MediaPortal\Thumbs\Videos\Title\Keep the Lights On{902}L.jpg
    So there is only few possibilities which can happen outside of MyVideos plugin:

    1. Downloaded temp file (as image) is the same as original taken from scraper (this could really be a mystery but can happen with a chance to win a lottery with 8 random numbers)
    2. Previous covers in thumb directory are not deleted before but then we should see error like ("Util: FileDelete {0} error: {1}", strFile , ex.Message);
    3. Something weird is going on in function:
    public static bool CreateThumbnail(string thumbnailImageSource, string thumbnailImageDest, int aThumbWidth,
    int aThumbHeight, int iRotate, bool aFastMode, bool autocreateLargeThumbs, bool fallBack)
    in MP core
    4. This do not work as intended sometimes when property parameter is the same as previous but parameter content is different
    GUIPropertyManager.SetProperty("#thumb", strThumb);

    I put my bet on 3 and 4 but I can't resolve that mystery because it's MP global problem so I need to catch this myself to find the cause. I know that there was many changes in code for 3 and 4 in recent MP releases so maybe something is not good there.
     
    Last edited: October 5, 2013
    • Agree Agree x 1
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!