IMDB+ issue (2 Viewers)

RoChess · May 24, 2018

@fredco, I'm still confused by your logs. Your last one did not contain a fresh re-import of Red Sparrow, so not much for me to look at. The original one didn't appear to contain any failure. Can you verify that IMDb+ is linked to that movie as the primary scraper script, and not say the TMDB one? This is visible inside MovPic config, on the movie details tab, and after you select the movie look for primary scraper reference.

This can be fixed via IMDb+ plugin otherwise with the "Force IMDb+" option.

fredco · May 24, 2018

@RoChess: Hi, you're talking about Red Sparrow, but I don't even have that movie, so clearly there's no import for that one. Never mentioned that movie I think...
I've made a new import and included a screenshot and a log. I've re-imported one movie: Daredevil. As you can see, all info in Moving Pictures is in English EXCEPT the summary. You can also see IMDb+ is the primary source, which is always the case. IMDb+ is always the primary scraper script. ALL my movies are scraped by your plugin.

RoChess · May 25, 2018

@fredco, mixing logs up then. What your logs are missing though is DEBUG mode from MediaPortal, and please be sure IMDb+ hidden option is configured to include HTML sources.

A single movie import (done fresh like you did Daredevil), should be like 1MB for movingpictures.log, and not 24kB (feel free to ZIP it up)

Open file in Notepad, and search for <html> which should be prefixed by ########## and then a ton of HTML source from IMDb and end with #######.

That way I can properly verify the RegExp results manually in Expresso and debug issue.

fredco · May 25, 2018

@RoChess: I think I understand what you're saying, but I have "Log Verbosity" ALWAYS set to DEBUG. But I changed another setting: in Options IMDb+ Scraper.xml I set the "global_options_debug_imdb_sources" value to TRUE (was FALSE). Now I get a MUCH larger movingpictures.log. I did a fresh import of my Pixar Collection. All movies were scraped by IMDb+ and ALL summaries are in Dutch. Does this help?

RoChess · May 26, 2018

Hi @fredco

The log is impossible to read right now, because it reads:

movie 1
movie 2
movie 3
movie 1
movie 4
movie 5
movie 2
movie 1
(155,000+ other lines like that)

That makes it extremely difficult to fix it.

Please allow MovPic to settle for a few minutes, delete the movingpictures.log file, ensure it doesn't get created again (background tasks can take a while)

Then send 1, as in ONE movie back to the importer (easy to do from the "Movie Details" tab via that drop down gear icon menu)

And then give me that log file.

You created the right logs, but that would prevent me from going down a rabbit hole looking at the code for movie #1, and running expression results from movie #2, etc.

fredco · May 26, 2018

Hi RoChess, I think I did what you wanted: deleted the log and resent "Coco" back to the importer. Log is now 400 kB, still a lot of text...

RoChess · May 28, 2018

@fredco, weird how your log did not contain any actual scraper debug log entries on the processing of the IMDb+ script. But I was able to obtain enough information from it to realize that short/long summaries came up empty.

Moving-Pictures on missing data will then resort to any fallback scraper-script to compliment missing data.

Your configuration must have TMDb or some other script configured/active, which was then used to supplement the missing summary and resort to a Dutch one for that.

Updated the IMDb+ regular expressions for v4.9.79, so you should be good to go again.

That is... until the next time IMDb.com decides to change things again

RoChess · May 28, 2018

To update you can use IMDb+ plugin to mass-refresh all movies (can take a while, but has benefit of refreshing any `Top 250` entries, update votes/ratings, etc.) or you go to an individual movie inside Moving-Pictures GUI and use F9 -> 1 -> 1 to update online info. Another way is to his the refresh icon within MovPic config on Movie Details tab.

fredco · May 29, 2018

Well RoChess. you did it! All summaries are back to English, like they should! THANKS! I still have some thoughts:

1. Strange that my log does not contain what you expect, as it's a clean install of MP, Moving Pictures and Windows. Log verbosity to DEBUG and global_options_debug_imdb_sources to TRUE. Don't know what more to do?
2. I DO have other fall-back scaper-scripts active for the rare cases IMDb+ doesn't find a match, like IMDb.com and TMDb. That should be ok, but I don't understand why they resort to Dutch?
3. I waited quite some time for the scraper-script to automatically update to v4.9.79 as I forgot how to force the update?
4. I've got 1.200 movies in my library and 4 instances of MP, so scraping them all takes some time, but I've done it more than once. And like you said, it refreshes all data.
5. Lately however it seems Moving Pictures stops responding when scanning a large number of movies. And it does so quite fast, after scanning say 50 movies. I don't think it's IMDb+ related, but I don't read about this issue at all just yet. Have you experienced this?

RoChess · May 30, 2018

fredco said:
Well RoChess. you did it! All summaries are back to English, like they should! THANKS!

Glad it works again for you as well. It is a constant battle to keep up with IMDb.com changes, but until AI works, it is a never ending cat-and-mouse game.

fredco said:
1. Strange that my log does not contain what you expect, as it's a clean install of MP, Moving Pictures and Windows. Log verbosity to DEBUG and global_options_debug_imdb_sources to TRUE. Don't know what more to do?

MediaPortal -- in debug mode (under General somewhere)
Moving-Pictures -- in scraper-debug mode (shows little green bug icon inside scraper-script section)
IMDb+ -- turn on hidden XML option to include HTML sources

I think you did 1+3, but forgot/overlooked #2, no biggie, just remember it for the future, and be sure to change it back once done as creating those extreme log files has a serious impact on performance.

fredco said:
2. I DO have other fall-back scaper-scripts active for the rare cases IMDb+ doesn't find a match, like IMDb.com and TMDb. That should be ok, but I don't understand why they resort to Dutch?

That depends on their settings, so you should allow one movie to import using them and verify why they are acting up. Localized detection is a pain in the ass for this, which is the main reason why I need to completely rewrite IMDb+, but trouble with finding the time to do so.

fredco said:
3. I waited quite some time for the scraper-script to automatically update to v4.9.79 as I forgot how to force the update?

Default time is 24 hours to not overload the GitHub server, but simply adjust the IMDb+ XML settings file to force a check upon launch.

fredco said:
4. I've got 1.200 movies in my library and 4 instances of MP, so scraping them all takes some time, but I've done it more than once. And like you said, it refreshes all data.

Yes, even at 3 seconds per movie, that is a full hour, but you want to be sure that all the debug stuff is disabled so it doesn't take 12 seconds per movie and you are stuck waiting for 4 hours

Of course I usually launch it, go to bed, and enjoy results when I wake up.

fredco said:
5. Lately however it seems Moving Pictures stops responding when scanning a large number of movies. And it does so quite fast, after scanning say 50 movies. I don't think it's IMDb+ related, but I don't read about this issue at all just yet. Have you experienced this?

That I'm not sure about, but the key about solving any problem is the ability to consistently reproduce an issue, because then whatever developer needs to fix it (be it MediaPortal, Moving-Pictures, or IMDb+) can step through their code until they hit the offending code-section and examine/fix the issue.

The problem with these random or hard to define issues is that this becomes a painstaking process, but keep in mind if it is difficult for you to reproduce, it is practically impossible for a developer to fix it. Good logs are a lifesaver though, and you might be lucky enough that somebody encountered the same thing.

One suggestion I have is that you might have an anti-virus solution interfering, which I had myself, and was forced to exclude the MediaPortal process, MediaPortal program folder, and my import folders from the default Windows Defender AV solution that ships with Windows.

Weird thing on that was that I was hitting 100% CPU usage, but the AV solution wasn't showing in task manager. Third party tools such as ProcMonitor from SysInternals revealed what was really going on. Microsoft system resource tool probably would have worked as well, but you get the idea.

IMDB+ issue (2 Viewers)

RoChess

Extension Developer

fredco

Portal Pro

Attachments

RoChess

Extension Developer

fredco

Portal Pro

Attachments

RoChess

Extension Developer

fredco

Portal Pro

Attachments

RoChess

Extension Developer

RoChess

Extension Developer

fredco

Portal Pro

RoChess

Extension Developer

Users who are viewing this thread