Spanish Scraper FilmAffinity.com with IMDb.es bonus to get fanarts -- v2.1.0 (2 Viewers)

RoChess

Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    @peque, otherwise perhaps there is a day/time to do like a screenshare, unless you can give me RDP (or TeamViewer) access to a system that has the issue. Without a means to reproduce it is nearly impossible to narrow down the issue in order to even begin working on a solution.
     

    peque

    Moderator - Spanish Forums
  • Premium Supporter
  • August 4, 2007
    861
    99
    Home Country
    Spain Spain
    Ok. I've spent some time with Wireshark... I attach two captures: one done from Moving Pictures, and one done with Chrome. Same user-agent, same system obviously. Url is Búsqueda avanzada - FilmAffinity

    What we see is that Filmaffinity responds with a 307 redirect http packet, which I assume Moving Pictures doesn't handle correctly, while Chrome does... Then Moving Pictures does a second request with different url (I don't know why, url is http://www.filmaffinity.com/es/filmStar Wars-Los ultimos Jedi.html), with same 307 redirect anwswer from Filmaffinity. If I test this second url with Chrome, I get "Page not found".
     

    Attachments

    • Captures.rar
      2.2 KB

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    @peque, so the 307 is done to convert HTTP to HTTPS, and somehow MovPic is dropping the & from the redirect (might be .NET doing it). Easy fix is to just make calls HTTPS to begin with -- please try v2.1.7 from first post.

    @ltfearme, that does mean MovPic has a bug on the follow-redirect code which I couldn't locate easy, but hoping you know what to look for better. I'll give it a more thorough look this weekend otherwise. I believe there are some extra parameters on the <retrieve> scraper-node to configure it more, so maybe I'm omitting configuration settings that already handle it, but thought you would want to know.
     

    peque

    Moderator - Spanish Forums
  • Premium Supporter
  • August 4, 2007
    861
    99
    Home Country
    Spain Spain
    2.1.7 is not solving the problem. Log shows same problem...

    Watching at wireshark captures, it seems there's some problem with TLS protocol versión. Chrome is using TLS 1.2, which is accepted by Filmaffinity, while Moving Pictures is using TLS 1.0, which is rejected.

    Attached both captures.
     

    Attachments

    • capture.rar
      17 KB

    RoChess

    Extension Developer
  • Premium Supporter
  • March 10, 2006
    4,434
    1,897
    @peque, thanks for the diligent debugging to narrow it down.

    @ltfearme, no idea what point exactly, but it sounds like you need to add the code I'm using myself in a different project (SSL/Poodle flashback!!):

    // Add support for TLS 1.2 on .NET 4.5
    ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;

    // Add support for TLS 1.2 on .NET 4.0
    //ServicePointManager.SecurityProtocol |= (SecurityProtocolType)3072;​

    Using |= will append it to the existing protocols, so you can still support TLS 1.0/1.1 as the industry rule to only allow TLS 1.2 is not enforced yet (though highly recommended, but you would need to test all scraper-script sources to confirm if forcing it via '=', but not like scraper-scripts pose a PCI DDS security risk). This will then cover those servers that only accept TLS 1.2, and therefore will not allow a TLS 1.0/1.1 forward-request to take place.

    TLS 1.3 is still in draft, so don't have to worry about it for a while (appears to be set for another update in September 2018).

    More information is available @ TLS 1.2 and .NET Support: How to Avoid Connection Errors - Microsoft

    The Moving-Pictures code making the connections appears to be @ damienhaynes/moving-pictures

    But I'm not familiar with the `WebGrabber` as I rely on `HttpWebRequest` myself, but the ServicePointManager should still be applicable.

    Microsoft made TLS 1.2 support default in .NET 4.6, but I still have that code in place on 4.6.2 so not sure if I couldn't be bothered to remove it, or it was still needed. I'm using the Tls12 entry, but the 3072 should work just as well for you if you are on an older framework.

    As you can see via SSLLabs under Configuration -> Protocols, FilmAffinity only supports TLS 1.2 -- SSL Server Test: filmaffinity.com (Powered by Qualys SSL Labs)
     

    peque

    Moderator - Spanish Forums
  • Premium Supporter
  • August 4, 2007
    861
    99
    Home Country
    Spain Spain
    I would say there's some time now without plugin updates... Am I right? Should we expect a fix for these soon?
     

    ltfearme

    Community Plugin Dev
  • Premium Supporter
  • June 10, 2007
    6,751
    7,196
    Sydney
    Home Country
    Australia Australia
    Hey Guys,

    I made the advised changes (thanks @RoChess), can you please test if it solves the problem. You only need the patched Cornerstone.dll library, so overwrite your existing one.

    Note: I believe using the enum 3072 will only work if you have .Net 4.5 or above installed, so I would be interested in what happens if you don't...I suspect requests will not work. .NET 4.5 was released back in 2012 so unlikely to be a problem. I had to use this method as MovPics / MP still using .NET 4 :(

    Cheers
    Damien
     

    Attachments

    • Cornerstone.zip
      288.3 KB

    peque

    Moderator - Spanish Forums
  • Premium Supporter
  • August 4, 2007
    861
    99
    Home Country
    Spain Spain
    Hey Guys,

    I made the advised changes (thanks @RoChess), can you please test if it solves the problem. You only need the patched Cornerstone.dll library, so overwrite your existing one.

    Note: I believe using the enum 3072 will only work if you have .Net 4.5 or above installed, so I would be interested in what happens if you don't...I suspect requests will not work. .NET 4.5 was released back in 2012 so unlikely to be a problem. I had to use this method as MovPics / MP still using .NET 4 :(

    Cheers
    Damien
    Will test ASAP and come back to tell you.

    Thanks a lot!

    Enviado desde mi Pixel mediante Tapatalk
     

    peque

    Moderator - Spanish Forums
  • Premium Supporter
  • August 4, 2007
    861
    99
    Home Country
    Spain Spain
    @ltfearme, fix is working like a charm. :) Thanks a lot.

    @RoChess, I've just realized that "release date" is being filled with importation time, not the real film release date. Can you check it? Date added is being filled with file last modification time, which is Ok for me. :)

    Thanks both!!!
     

    Users who are viewing this thread

    Top Bottom