Spanish Scraper FilmAffinity.com with IMDb.es bonus to get fanarts -- v2.1.0 (1 Viewer)

[RoChess]

@peque, otherwise perhaps there is a day/time to do like a screenshare, unless you can give me RDP (or TeamViewer) access to a system that has the issue. Without a means to reproduce it is nearly impossible to narrow down the issue in order to even begin working on a solution.

 

[peque]

Ok. I've spent some time with Wireshark... I attach two captures: one done from Moving Pictures, and one done with Chrome. Same user-agent, same system obviously. Url is Búsqueda avanzada - FilmAffinity

What we see is that Filmaffinity responds with a 307 redirect http packet, which I assume Moving Pictures doesn't handle correctly, while Chrome does... Then Moving Pictures does a second request with different url (I don't know why, url is http://www.filmaffinity.com/es/filmStar Wars-Los ultimos Jedi.html), with same 307 redirect anwswer from Filmaffinity. If I test this second url with Chrome, I get "Page not found".

 

[RoChess]

@peque, so the 307 is done to convert HTTP to HTTPS, and somehow MovPic is dropping the & from the redirect (might be .NET doing it). Easy fix is to just make calls HTTPS to begin with -- please try v2.1.7 from first post.

@ltfearme, that does mean MovPic has a bug on the follow-redirect code which I couldn't locate easy, but hoping you know what to look for better. I'll give it a more thorough look this weekend otherwise. I believe there are some extra parameters on the <retrieve> scraper-node to configure it more, so maybe I'm omitting configuration settings that already handle it, but thought you would want to know.

 

[peque]

2.1.7 is not solving the problem. Log shows same problem...

Watching at wireshark captures, it seems there's some problem with TLS protocol versión. Chrome is using TLS 1.2, which is accepted by Filmaffinity, while Moving Pictures is using TLS 1.0, which is rejected.

Attached both captures.

 

[RoChess]

@peque, thanks for the diligent debugging to narrow it down.

@ltfearme, no idea what point exactly, but it sounds like you need to add the code I'm using myself in a different project (SSL/Poodle flashback!!):

// Add support for TLS 1.2 on .NET 4.5
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;

// Add support for TLS 1.2 on .NET 4.0
//ServicePointManager.SecurityProtocol |= (SecurityProtocolType)3072;​

Using |= will append it to the existing protocols, so you can still support TLS 1.0/1.1 as the industry rule to only allow TLS 1.2 is not enforced yet (though highly recommended, but you would need to test all scraper-script sources to confirm if forcing it via '=', but not like scraper-scripts pose a PCI DDS security risk). This will then cover those servers that only accept TLS 1.2, and therefore will not allow a TLS 1.0/1.1 forward-request to take place.

TLS 1.3 is still in draft, so don't have to worry about it for a while (appears to be set for another update in September 2018).

More information is available @ TLS 1.2 and .NET Support: How to Avoid Connection Errors - Microsoft

The Moving-Pictures code making the connections appears to be @ damienhaynes/moving-pictures

But I'm not familiar with the `WebGrabber` as I rely on `HttpWebRequest` myself, but the ServicePointManager should still be applicable.

Microsoft made TLS 1.2 support default in .NET 4.6, but I still have that code in place on 4.6.2 so not sure if I couldn't be bothered to remove it, or it was still needed. I'm using the Tls12 entry, but the 3072 should work just as well for you if you are on an older framework.

As you can see via SSLLabs under Configuration -> Protocols, FilmAffinity only supports TLS 1.2 -- SSL Server Test: filmaffinity.com (Powered by Qualys SSL Labs)

 

[peque]

I would say there's some time now without plugin updates... Am I right? Should we expect a fix for these soon?

 

[RoChess]

@peque, expect a patch to try shortly that @ltfearme will probably attach to this thread.

 

[ltfearme]

Hey Guys,

I made the advised changes (thanks @RoChess), can you please test if it solves the problem. You only need the patched Cornerstone.dll library, so overwrite your existing one.

Note: I believe using the enum 3072 will only work if you have .Net 4.5 or above installed, so I would be interested in what happens if you don't...I suspect requests will not work. .NET 4.5 was released back in 2012 so unlikely to be a problem. I had to use this method as MovPics / MP still using .NET 4

Cheers
Damien

 

[peque]

Hey Guys,

I made the advised changes (thanks @RoChess), can you please test if it solves the problem. You only need the patched Cornerstone.dll library, so overwrite your existing one.

Note: I believe using the enum 3072 will only work if you have .Net 4.5 or above installed, so I would be interested in what happens if you don't...I suspect requests will not work. .NET 4.5 was released back in 2012 so unlikely to be a problem. I had to use this method as MovPics / MP still using .NET 4

Cheers
Damien

Will test ASAP and come back to tell you.

Thanks a lot!

Enviado desde mi Pixel mediante Tapatalk

 

[peque]

@ltfearme, fix is working like a charm. Thanks a lot.

@RoChess, I've just realized that "release date" is being filled with importation time, not the real film release date. Can you check it? Date added is being filled with file last modification time, which is Ok for me.

Thanks both!!!