MovingPictures: Hang at retrieving details (connection fails) ? (4 Viewers)

[RoChess]

Thanks for the hint, I believe my example was not good though. Sorry for that.

One thing I am noticing from your info logs is that you have some movies probably assigned incorrect, like TMDb does not have artwork for: Planet Earth (TV Mini-Series 2006) - IMDb

But that is a TV-Series that fits much better to add to MP-TVSeries and TheTVDb does have a good listing for that one.

The other options is to re-assign each episode to their individual IMDb tt-ID listing.

 

[dangerzone]

Is MediaPortal set to log in debug verbosity?
And verify that the green little icon shows in MovPic scraper-settings to indicate scraper-debug is active.

Yes mepo is in debug mode and the scraper i set the enable debug mode on.
I used this guide:

Issues Debugging

IMDb+ scraper for MediaPortal's MovingPictures plugin - RoChess/IMDbPlus

github.com

 

[dangerzone]

I checken again and mepo is in debug and the scraper has advanced data provider debugging active.
I did refresh info from imdb+ on a movie and got the following small log..
If you can not do anything with it, its to bad

 

[RoChess]

Yes mepo is in debug mode and the scraper i set the enable debug mode on.

Sorry, my bad, you weren't debugging IMDb+ (I'm used to the HTML source-codes in that one), but IMDb default one no longer does that and new update shortens the IF node output now (look at EDIT section below).

Found your issue:

22-Jun-2020 19:13:03 Debug [ ScraperNode]: Retrieving URL: http://akas.imdb.com/find?s=tt;q=Artemis Fowl+(2020)
22-Jun-2020 19:13:45 Warn [ WebGrabber]: Connection failed: Reached retry limit of 5. URL=

So, that took 42 seconds (the answer to everything!!) to finally "fail", which probably means a time-out of around 8 seconds per attempt with a little overhead-delay.

It is still using non-SSL "http" which gets redirected to "https" by IMDb.com, but more importantly akas.imdb.com sub-domain for international movie title searches has been killed-off by Amazon, which is why everything is failing.

Could have swore that @ltfearme updated the default scraper, but I am looking at master and that appears to not be the case @ damienhaynes/moving-pictures

I'll try to help out if I can to fix the default scraper-script, and put in a pull-request, as more than just the URL changed if memory serves me, but if you need a fix now, then IMDb+ was already fixed (Jan 8th by looking at GitHub history), and it would also provide you with more options/features.

EDIT: Helps if I look at entire log file, because I notice you are using IMDb+, but either your database got corrupted and IMDb+ plugin stopped updating the scraper-script, or you are on the initial bundled scraper-script still and the auto-update hasn't been run yet, or is failing. You're on version 4.3.8 of IMDb+ scraper-script from April 3rd, 2012, but you should have been automatically upgraded to the latest v4.10.5 from a few months ago.

Check your "C:\ProgramData\Team MediaPortal\MediaPortal\log\IMDb+.log" file please.

It should contain something like this near the end:

6/23/2020 01:29:33[Check for Updates][19][Info] Checking for scraper update
6/23/2020 01:29:33[Check for Updates][19][Debug] Downloading file from: https://raw.githubusercontent.com/RoChess/IMDbPlus/master/Scraper/IMDb%2B.Scraper.SVN.xml
6/23/2020 01:29:34[Check for Updates][19][Info] Updating paths in scraper script
6/23/2020 01:29:34[Check for Updates][19][Info] Finished updating paths in scraper script

Really need to publish a new MPEI update that does it automatic, because it comes bundled with an old version, and no update check is done during installation. Instead it does an update-check on the next launch of MediaPortal. I'm used to installing new plugins via the MPEI shortcut to desktop, and then afterwards you would launch MediaPortal and it would immediately update itself. If your skin has support it will even show a little dialog during the initial launch of MediaPortal.

This is explained to some degree in the installation manual @ RoChess/IMDbPlus

But who reads manuals right

 

[dangerzone]

EDIT: Helps if I look at entire log file, because I notice you are using IMDb+, but either your database got corrupted and IMDb+ plugin stopped updating the scraper-script, or you are on the initial bundled scraper-script still and the auto-update hasn't been run yet, or is failing. You're on version 4.3.8 of IMDb+ scraper-script from April 3rd, 2012, but you should have been automatically upgraded to the latest v4.10.5 from a few months ago.

Check your "C:\ProgramData\Team MediaPortal\MediaPortal\log\IMDb+.log" file please.

It should contain something like this near the end:

6/23/2020 01:29:33[Check for Updates][19][Info] Checking for scraper update
6/23/2020 01:29:33[Check for Updates][19][Debug] Downloading file from: https://raw.githubusercontent.com/RoChess/IMDbPlus/master/Scraper/IMDb%2B.Scraper.SVN.xml
6/23/2020 01:29:34[Check for Updates][19][Info] Updating paths in scraper script
6/23/2020 01:29:34[Check for Updates][19][Info] Finished updating paths in scraper script

Really need to publish a new MPEI update that does it automatic, because it comes bundled with an old version, and no update check is done during installation. Instead it does an update-check on the next launch of MediaPortal. I'm used to installing new plugins via the MPEI shortcut to desktop, and then afterwards you would launch MediaPortal and it would immediately update itself. If your skin has support it will even show a little dialog during the initial launch of MediaPortal.

This is explained to some degree in the installation manual @ RoChess/IMDbPlus

But who reads manuals right

Thanks for the input.

I realized the scraper is from 2012, But there where no updates in extension manager therefore its updates true scripts was my idea
Looked into that and it was updated this month.
I'm using titan, and if the scraper script is updated i got an pop-up: imdb+ scraper updated to version 2.X.x or something
And manuals.? Install it, if it does not work search the forum

Also the log looks good in my opinion

 

[RoChess]

Also the log looks good in my opinion

It does indeed:

24-6-2020 18:25:34[Check for Updates][21][Warning] Load Script Warning: Scraper debug-mode enabled, so existing script was replaced.

If you now go into MediaPortal GUI, and goto Plugins -> IMDb+, then if you open the fly-out menu (left-arrow for me on Titan), it should allow you to view "IMDb+ Info", and you should be able to confirm that the scraper-script is on version 4.10.5

There is an irritating random bug where Moving-Pictures corrupts the database and stops updating, so initially I thought that was happening for you.

All you have to do now is refresh "Artemis Fowl" (via F9 inside Moving-Pictures or whatever remote control button you have configured to get the popup), or use the desktop shortcut.

The IMDb+ plugin also has an option to refresh your entire library, but you first might need to force IMDb+ to be used on all your existing movies (forgot what the option is called, but "Force IMDb+...." or something and it is in the wiki somewhere). Keep in mind if you ever made any custom edits then those will be lost, but you do get benefit of updated IMDb info, which is especially handy on score/votes as those change when time passes and more people have watched the movie. Often summaries/taglines get improved as well. You can still use MediaPortal fully when IMDb+ is updating if your system is fast enough to do it in the background (has to be a really old/slow system if it is unable to do so), which is nice as the process can take a long time on a large collection when you're looking at say 3 seconds per movie on details+artwork, and you got ~1200 movies then it takes a full hour.

Personally I often just pick a night once a quarter/year where I go to bed, and leave the HTPC running while it does an update and then I power it down the next morning.

PS: Once you are happy, don't forget to disable the scraper-debug option in Moving-Pictures, and MediaPortal debug verbosity logging. Writing out those many megabytes of logs does slow the process down. Of course that does mean if you ever run into a future issue you have to turn that back on again and try to reproduce the bug you ran into, but without the ability to reproduce it would be extremely hard for me to fix it, so that works out and only takes a few extra steps. For normal usage you want to enjoy a fast system, unless you are gluten for punishment like myself

 

[dangerzone]

If you now go into MediaPortal GUI, and goto Plugins -> IMDb+, then if you open the fly-out menu (left-arrow for me on Titan), it should allow you to view "IMDb+ Info", and you should be able to confirm that the scraper-script is on version 4.10.5

Hmmmm.... it says v4.3.8 ?

The IMDb+ plugin also has an option to refresh your entire library, but you first might need to force IMDb+ to be used on all your existing movies.

Busy with that but 1 movie takes about 3 minutes so something is wrong

 

[RoChess]

Yeah, the quickness only works on latest v4.10.5 version, so I guess you do have corruption preventing script from updating.

You installed IMDb+ recently right?

 

[dangerzone]

No.... only did a reinstall of imdb+ to be sure.....

 

[RoChess]

It has only happened a few times for me in the ~10 years of using IMDb+, but the way I solve it, is to close MediaPortal, and then open the movingpictures.db3 database file (after you first copy it for a backup obviously) in an SQLite editor such as "SQLite Expert Personal".

Then you select the "scripts" table, and this is where it gets tricky. On the right side you switch to the "Data" tab. At the bottom it will show you how many scraper-scripts you have installed (including old versions). On mine it shows "1 out of 56" as I haven't cleaned things up lately.

Jump to the last record via the ">>|" button, and it should show:

<ScriptableScraper>
    <details>
        <!-- These details are for display purposes only. -->
        <name>IMDb+</name>
        <author>Ron (RoChess)</author>
        <description>Combination scraper, using the best data from IMDb and RottenTomatoes.</description>
        <id>314159265</id>
        <version major="4" minor="10" point="5" />
        <published month="02" day="24" year="2020" />
        <type>MovieDetailsFetcher|MovieCoverFetcher</type>
        <language>various</language>
    </details>
[...and much more...]

If you see another scraper-script other than IMDb+ v4.10.5, then use the "<" button to go back one record at a time until you find it.

Then right click on that row containing the IMDb+ v4.10.5 scraper-script XML code and select "Record Editor".

Click inside the "contents" textbox, and use CTRL+A to select the entire XML script. Then use CTRL+C to copy it to clipboard, and click CANCEL to close the record editor window.

Now keep using the "<" button until you locate the IMDb+ v4.3.8 entry, right click on that cell, and select "Record Editor" again.

This time select all the content again inside the "contents" textbox via CTRL+A, but then use CTRL+V to replace that one with the v4.10.5 content.

Then use the SAVE button to make the change, and use that 3rd icon in the toolbar to close the database. It will then ask you if you wish to save changes to the DB3 file, so say YES, and then close SQLite program.

There might be a much easier way, such as using Moving-Pictures configuration to go to manual scraper-script control and "delete" the v4.3.8 script, but I've tried so many different things when this random bug was brought to my attention that I never could figure out exactly what solved it or how to hunt down the bug in MovPic source-code, but the SQLite database method is guaranteed to fix it and that is what I use myself.

PS: I've marked this bug on my ToDo list again to look at this weekend, but I still don't know if it is a bug in Moving-Pictures or IMDb+ Plugin. Do you remember the order in which you installed things? Did you perhaps install IMDb+, and then maybe also applied an update to Moving-Pictures before you launched MediaPortal GUI? That's one scenario that might cause things to freak out as new MovPic installs updated scraper-scripts on the internal ones, which might not always respect the primary order the IMDb+ plugin assigns itself before MediaPortal GUI has launched. Just writing this out as a reminder note for when I can find time to test.

PPS: Feel free to provide me with your movingpictures.db3 database, and I can do the fix for you if the above technical steps look scary.