Ratings wrong (Math node problem when (comma) is used as decimal seperator -- FIXED) (1 Viewer)

[Hell_Angel]

Hi,

I'm just trying out this new version of your scraper. I was on 3.1.7. before.
After I installed it and went on with default options, I got really weird ratings.

Movie | Rating
11th Hour = 1,3
13 = 3,9
17 Again = 9,6
5 Days of War = 3,3
etc...

But you can clearly see, that these are far from correct. I didn't change any options before I tested it, so do you have any ideas what might be going on? I have to mention, that with 3.1.7. everything was okay and ratings just fine.

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

Hi,

I'm just trying out this new version of your scraper. I was on 3.1.7. before.
After I installed it and went on with default options, I got really weird ratings.

Movie | Rating
11th Hour = 1,3
13 = 3,9
17 Again = 9,6
5 Days of War = 3,3
etc...

But you can clearly see, that these are far from correct. I didn't change any options before I tested it, so do you have any ideas what might be going on? I have to mention, that with 3.1.7. everything was okay and ratings just fine.

I thought I had tested it enough that the results were consistent enough to be used. New default setting is to use an average rating that takes the IMDb score, Metacritics Metascore, RottenTomatoes Top Critics, RottenTomatoes All Critics and Audience score and then averages all 5 of those into the actual score.

Set the 'global_options_single_score' option to "true" which will re-enable the old system of using a single rating, which will match what you had with v3.1.7 as I do more testing into what went wrong based on the movies you provided.

PS: Ok, I tested "The 11th Hour" as in: http://www.imdb.com/title/tt0492931/

And results are:

imdb = 7.0
metascore = 6.3
rottentomatoes all = 6.5
rottentomatoes top = 6.1
rottentomatoes audience = 7.2

7.0 + 6.3 + 6.5 + 6.1 + 7.2 = 33.1 / 5 = average score of 6.6 (This perfectly matches the results I get importing this movie, and the scraper-debug-enabled log file shows me the match went perfect)

As for the other movies, I get:

13 = 6.3
17 Again = 5.8
5 Days of War = 5.7

Which is totally different then your results, so now I'm curious as to why.

Can you repeat the wrong results each time? If you can, then please enable scraper debug mode, refresh the movie details on one of those movies and provide me with your movingpictures.log file.

 

[Hell_Angel]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

Hi,

I'm just trying out this new version of your scraper. I was on 3.1.7. before.
After I installed it and went on with default options, I got really weird ratings.

Movie | Rating
11th Hour = 1,3
13 = 3,9
17 Again = 9,6
5 Days of War = 3,3
etc...

But you can clearly see, that these are far from correct. I didn't change any options before I tested it, so do you have any ideas what might be going on? I have to mention, that with 3.1.7. everything was okay and ratings just fine.

I thought I had tested it enough that the results were consistent enough to be used. New default setting is to use an average rating that takes the IMDb score, Metacritics Metascore, RottenTomatoes Top Critics, RottenTomatoes All Critics and Audience score and then averages all 5 of those into the actual score.

Set the 'global_options_single_score' option to "true" which will re-enable the old system of using a single rating, which will match what you had with v3.1.7 as I do more testing into what went wrong based on the movies you provided.

PS: Ok, I tested "The 11th Hour" as in: 11th Hour - 5 vor 12 (2007) - IMDb

And results are:

imdb = 7.0
metascore = 6.3
rottentomatoes all = 6.5
rottentomatoes top = 6.1
rottentomatoes audience = 7.2

7.0 + 6.3 + 6.5 + 6.1 + 7.2 = 33.1 / 5 = average score of 6.6 (This perfectly matches the results I get importing this movie, and the scraper-debug-enabled log file shows me the match went perfect)

As for the other movies, I get:

13 = 6.3
17 Again = 5.8
5 Days of War = 5.7

Which is totally different then your results, so now I'm curious as to why.

Can you repeat the wrong results each time? If you can, then please enable scraper debug mode, refresh the movie details on one of those movies and provide me with your movingpictures.log file.

Hi again and thanks for the lightning fast answer.
Those rates you gave looks like the way they should, but somehow it doesn't work (at least for me).
But like I said, nothing has been changed in my configuration after upgrading from 3.1.7. to 3.2.2.
I enabled scraper's debug mode and will post log file as attachment. I updated Adjustment Bureau's rating and result was 1,3, if I remember correctly. Thanks in advance for your help!

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

Hi again and thanks for the lightning fast answer.
Those rates you gave looks like the way they should, but somehow it doesn't work (at least for me).

You are in luck.

The problem relates to the fact that my scraper script uses a lot of math to calculate the new features. It seems that for the users who have their Windows setup configured to use ',' for decimal seperators freak out on the floating point calculations.

This is why your error log shows "Error parsing numbers" when it is doing the calculations to get average score.

This however has recently been fixed in: Issue 608 - moving-pictures - Math Nodes Sometimes Fail to Parse Floating Point Numbers - A movies plugin for the MediaPortal media center application. - Google Project Hosting

And there is a new public release that contains this fix. So please download and install latest version (v1.1.5 or v1.2.3), which looking at your log would mean v1.2.3 for you.

Downloads - moving-pictures - A movies plugin for the MediaPortal media center application. - Google Project Hosting

Let me know if that fixes things.

 

[Hell_Angel]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

Hi again and thanks for the lightning fast answer.
Those rates you gave looks like the way they should, but somehow it doesn't work (at least for me).

You are in luck.

The problem relates to the fact that my scraper script uses a lot of math to calculate the new features. It seems that for the users who have their Windows setup configured to use ',' for decimal seperators freak out on the floating point calculations.

This is why your error log shows "Error parsing numbers" when it is doing the calculations to get average score.

This however has recently been fixed in: Issue 608 - moving-pictures - Math Nodes Sometimes Fail to Parse Floating Point Numbers - A movies plugin for the MediaPortal media center application. - Google Project Hosting

And there is a new public release that contains this fix. So please download and install latest version (v1.1.5 or v1.2.3), which looking at your log would mean v1.2.3 for you.

Downloads - moving-pictures - A movies plugin for the MediaPortal media center application. - Google Project Hosting

Let me know if that fixes things.

I tried it that way yesterday and it did change at least something, because ratings are changing.
Some of them might even be correct, but most of them are still messed up. I changed back to 3.1.7. and everything is back to normal. I haven't tried that 3.2.3. version, you have just published. I like your scraper much more than original, so keep up the good work! I'm also having that IMDB - prefix bug, but you already mentioned, that it's fixed in 3.2.3.

EDIT: Just tried version 3.2.3., but the problem still exists. I posted new log file, if there's something new information. I also noticed from log file, that at least it gets right ratings from IMDb and Rotten Tomatoes, but something is going wrong in calculation part or somewhere else. I also found that "Error parsing numbers" phrase, but couldn't make much more out of it.

I also have little question/feature request. Is it/Will it be possible to use Original/english title and finnish title. I mean like this: 11th Hour - Yhdestoista hetki or 11th Hour (Yhdestoista hetki)? Now it works that way only with original and english titles, am I right?

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

I tried it that way yesterday and it did change at least something, because ratings are changing.

It looks like there is still a problem with MovingPictures. I clearly see it takes the value of '48/100' for Metascore, then it divides that by 10, the result should be 4.8, but it becomes 4,8 which then freaks out the other calculations. I'll communicate with fforde on fixing that.

In the meantime, just turn v3.2.3 into the same methods used as v3.1.7, and that is by adjusting the global_options_single_score one into "true". This will then use a single value without math, so it will not cause any problems.

I also have little question/feature request. Is it/Will it be possible to use Original/english title and finnish title. I mean like this: 11th Hour - Yhdestoista hetki or 11th Hour (Yhdestoista hetki)? Now it works that way only with original and english titles, am I right?

I looked at your settings and it seems you made some adjustments that might have disabled the proper detection of English titles. You removed all the English language speaking countries, when you should have kept them and added Finland to the list.

So use "us|ca|gb|ie|au|nz|fi" for country and "en|fi" for language.

I put in a lot of fall-back methods into my scraper, so your result was most likely still English titles for the non-Finnish movies, but it could lead to unexpected results.

As for your request, it is very hard for me to implement a feature, let alone test/debug it, when I can not recreate what you see on my USA based computer. So please re-import "The 11th Hour" by itself, so I can 'see' what results you get in Finland. Be sure to re-import and not refresh, unless you adjust the 'global_option_refresh_all_fields' setting to "true".

 

[drealit]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.3

Hmm I very briefly tested out the latest script right before bed last night. It looks like some scores pulled in fine while others... I honestly don't know where the score came from.

I have my script set to pull all critic scores and % from RT and for some reason titles such as Assault on Precinct 13 (1976) and Conan the Barbarian pulled in what appears to be very random scoring. Precinct 13 should have been a 97% but it pulled in a 7.x value (can't remember off top of my head) and Conan should have been a 77% but pulled in a 5.8. Other titles pulled in fine though so it doesn't exactly make sense. The only reason I noticed these 2 is because I recently added them.

I'll play around with it more when I get home today and pop up some logs for you with just those 2 titles along with a good one.

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.3

Hmm I very briefly tested out the latest script right before bed last night. It looks like some scores pulled in fine while others... I honestly don't know where the score came from.

I have my script set to pull all critic scores and % from RT and for some reason titles such as Assault on Precinct 13 (1976) and Conan the Barbarian pulled in what appears to be very random scoring. Precinct 13 should have been a 97% but it pulled in a 7.x value (can't remember off top of my head) and Conan should have been a 77% but pulled in a 5.8. Other titles pulled in fine though so it doesn't exactly make sense. The only reason I noticed these 2 is because I recently added them.

I'll play around with it more when I get home today and pop up some logs for you with just those 2 titles along with a good one.

Adjust the 'global_options_single_score' setting to "true" and the v3.1.x method of scoring system will be back. I have a ton of movies with freaky scores. There are some movies (hard to locate them now on the spot) that have for example an IMDb score of 9.7, because only 10 people have voted, which most likely are people who worked on the movie.

For this reason I added a new option that allows you to filter out any IMDb score that has less then 20 votes. Eventually scores settle out to a much more appropriated rating that the movie deserves, but I wanted to create a method that would get to that point much faster.

So that is when I decided to average the scores of IMDb, Metacritics, RottenTomatoes All Critics, RottenTomatoes Top Critics, and RottenTomatoes Audiences. On a 200 movie sample set on my own box on old, popular, new and unreleased movies, this resulted in a much better system then any single score was capable of.

The problem I find out now is that for users who are living in a country where (comma) is used for floating point seperator, as in "8,5" instead of "8.5" are getting very bad results due to a bug in MovingPictures. This was fixed in the new public release, but as Hell_Angel found out, there are still some issues with it. I've reported the problems to fforde, who hopefully will have a solution for the next release of MovingPictures.

In the meantime adjust 'global_options_single_score' setting to "true", so that the old scoring system is back in use for you. However if the math is done right for you locale, then you might want to reconsider, as I ended up with very accurate scores myself. But scoring is an extremly sensitive subject, what one person considers a crappy movie, somebody else might call their favorite movie of all times

 

[Hell_Angel]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

It looks like there is still a problem with MovingPictures. I clearly see it takes the value of '48/100' for Metascore, then it divides that by 10, the result should be 4.8, but it becomes 4,8 which then freaks out the other calculations. I'll communicate with fforde on fixing that.

In the meantime, just turn v3.2.3 into the same methods used as v3.1.7, and that is by adjusting the global_options_single_score one into "true". This will then use a single value without math, so it will not cause any problems.

Ok, hopefully it will be fixed someday. I will use that single score setting in meantime.

I looked at your settings and it seems you made some adjustments that might have disabled the proper detection of English titles. You removed all the English language speaking countries, when you should have kept them and added Finland to the list.

So use "us|ca|gb|ie|au|nz|fi" for country and "en|fi" for language.

I put in a lot of fall-back methods into my scraper, so your result was most likely still English titles for the non-Finnish movies, but it could lead to unexpected results.

This was just a test, I tried different variations. I changed them back to normal already and you're right about titles still being in english.

As for your request, it is very hard for me to implement a feature, let alone test/debug it, when I can not recreate what you see on my USA based computer. So please re-import "The 11th Hour" by itself, so I can 'see' what results you get in Finland. Be sure to re-import and not refresh, unless you adjust the 'global_option_refresh_all_fields' setting to "true".

It was just a thought. I can live without it, but here's my log file for "The 11th Hour" with 'refresh_all_fields' set to 'true'. Take a look, if you can make something out of it.

 

[RoChess]

Re: IMDb+ Scraper (Fix English, Rename+Group, RottenTomatoes, and more) v3.2.2

This was just a test, I tried different variations. I changed them back to normal already and you're right about titles still being in english.

The whole method of being able to adjust the country and language filtering was so that you could decide to not translate certain original titles.

So you might want any Finland+Finnish movie left along, per example:

Harjunpää ja pahan pappi (2010) - IMDb

Would then show up as "Harjunpää ja pahan pappi" and not forced into the English title of "Priest of Evil". But at the same token:

Jûsan-nin no shikaku (2010) - IMDb

Would then still be forced into English title of "13 Assassins" and not the original Japanese title of "Jûsan-nin no shikaku".

Of course then there are users who want to see both titles, so you get the results of "Priest of Evil (Harjunpää ja pahan pappi)" and "13 Assassins (Jûsan-nin no shikaku)" on the two examples. This is actually the setting I use myself, as I have an extensive foreign-title collection and sometimes only remember the original title, and sometimes the English title. By using a new title that consists of both, I can still use SMS filtering to find it quick.

The problem is that the IMDb website is making it very hard to find our which title is which. If I view the Harjunpää ja pahan pappi (2010) - IMDb page from an American computer, then I get the following results:

Priest of Evil (2010) 
Harjunpää ja pahan pappi (original title)

Which would allow me to use that easy. However you being in Finland do not get those results, and the American 'akas' page is also different. This is where all the problems started, and what lead to this IMDb+ scraper being on steroids

Your log file should give me another insight into how IMDb shows their same imdb.com website different to users around the world, and hopefully I can figure out a way to make it all work.