Spanish Scraper FilmAffinity.com with IMDb.es bonus to get fanarts -- v2.1.0 | Page 31

Discussion in 'Moving Pictures' started by RoChess, December 28, 2009.

  1. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    @peque, otherwise perhaps there is a day/time to do like a screenshare, unless you can give me RDP (or TeamViewer) access to a system that has the issue. Without a means to reproduce it is nearly impossible to narrow down the issue in order to even begin working on a solution.


     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. peque
    • Premium Supporter

    peque Moderator - Spanish Forums

    Joined:
    August 4, 2007
    Messages:
    855
    Likes Received:
    88
    Ratings:
    +111 / 1
    Home Country:
    Spain Spain
    Show System Specs
    Ok. I've spent some time with Wireshark... I attach two captures: one done from Moving Pictures, and one done with Chrome. Same user-agent, same system obviously. Url is Búsqueda avanzada - FilmAffinity

    What we see is that Filmaffinity responds with a 307 redirect http packet, which I assume Moving Pictures doesn't handle correctly, while Chrome does... Then Moving Pictures does a second request with different url (I don't know why, url is http://www.filmaffinity.com/es/filmStar Wars-Los ultimos Jedi.html), with same 307 redirect anwswer from Filmaffinity. If I test this second url with Chrome, I get "Page not found".
     

    Attached Files:

    • Captures.rar
      File size:
      2.2 KB
      Uploaded:
      April 5, 2018
      Views:
      32
  4. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    @peque, so the 307 is done to convert HTTP to HTTPS, and somehow MovPic is dropping the & from the redirect (might be .NET doing it). Easy fix is to just make calls HTTPS to begin with -- please try v2.1.7 from first post.

    @ltfearme, that does mean MovPic has a bug on the follow-redirect code which I couldn't locate easy, but hoping you know what to look for better. I'll give it a more thorough look this weekend otherwise. I believe there are some extra parameters on the <retrieve> scraper-node to configure it more, so maybe I'm omitting configuration settings that already handle it, but thought you would want to know.
     
  5. peque
    • Premium Supporter

    peque Moderator - Spanish Forums

    Joined:
    August 4, 2007
    Messages:
    855
    Likes Received:
    88
    Ratings:
    +111 / 1
    Home Country:
    Spain Spain
    Show System Specs
    2.1.7 is not solving the problem. Log shows same problem...

    Watching at wireshark captures, it seems there's some problem with TLS protocol versión. Chrome is using TLS 1.2, which is accepted by Filmaffinity, while Moving Pictures is using TLS 1.0, which is rejected.

    Attached both captures.
     

    Attached Files:

    • capture.rar
      File size:
      17 KB
      Uploaded:
      April 6, 2018
      Views:
      31
  6. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    @peque, thanks for the diligent debugging to narrow it down.

    @ltfearme, no idea what point exactly, but it sounds like you need to add the code I'm using myself in a different project (SSL/Poodle flashback!!):

    // Add support for TLS 1.2 on .NET 4.5
    ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12;

    // Add support for TLS 1.2 on .NET 4.0
    //ServicePointManager.SecurityProtocol |= (SecurityProtocolType)3072;​

    Using |= will append it to the existing protocols, so you can still support TLS 1.0/1.1 as the industry rule to only allow TLS 1.2 is not enforced yet (though highly recommended, but you would need to test all scraper-script sources to confirm if forcing it via '=', but not like scraper-scripts pose a PCI DDS security risk). This will then cover those servers that only accept TLS 1.2, and therefore will not allow a TLS 1.0/1.1 forward-request to take place.

    TLS 1.3 is still in draft, so don't have to worry about it for a while (appears to be set for another update in September 2018).

    More information is available @ TLS 1.2 and .NET Support: How to Avoid Connection Errors - Microsoft

    The Moving-Pictures code making the connections appears to be @ damienhaynes/moving-pictures

    But I'm not familiar with the `WebGrabber` as I rely on `HttpWebRequest` myself, but the ServicePointManager should still be applicable.

    Microsoft made TLS 1.2 support default in .NET 4.6, but I still have that code in place on 4.6.2 so not sure if I couldn't be bothered to remove it, or it was still needed. I'm using the Tls12 entry, but the 3072 should work just as well for you if you are on an older framework.

    As you can see via SSLLabs under Configuration -> Protocols, FilmAffinity only supports TLS 1.2 -- SSL Server Test: filmaffinity.com (Powered by Qualys SSL Labs)
     
    • Thank You! Thank You! x 2
  7. peque
    • Premium Supporter

    peque Moderator - Spanish Forums

    Joined:
    August 4, 2007
    Messages:
    855
    Likes Received:
    88
    Ratings:
    +111 / 1
    Home Country:
    Spain Spain
    Show System Specs
    I would say there's some time now without plugin updates... Am I right? Should we expect a fix for these soon?
     
  8. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    @peque, expect a patch to try shortly that @ltfearme will probably attach to this thread.
     
    • Like Like x 1
    • Agree Agree x 1
  9. ltfearme
    • Premium Supporter

    ltfearme Community Plugin Dev

    Joined:
    June 10, 2007
    Messages:
    6,457
    Likes Received:
    4,241
    Gender:
    Male
    Occupation:
    Software Test Engineer
    Location:
    Sydney
    Ratings:
    +5,385 / 0
    Home Country:
    Australia Australia
    Hey Guys,

    I made the advised changes (thanks @RoChess), can you please test if it solves the problem. You only need the patched Cornerstone.dll library, so overwrite your existing one.

    Note: I believe using the enum 3072 will only work if you have .Net 4.5 or above installed, so I would be interested in what happens if you don't...I suspect requests will not work. .NET 4.5 was released back in 2012 so unlikely to be a problem. I had to use this method as MovPics / MP still using .NET 4 :(

    Cheers
    Damien
     

    Attached Files:

    • Like Like x 1
    • Thank You! Thank You! x 1
  10. peque
    • Premium Supporter

    peque Moderator - Spanish Forums

    Joined:
    August 4, 2007
    Messages:
    855
    Likes Received:
    88
    Ratings:
    +111 / 1
    Home Country:
    Spain Spain
    Show System Specs
    Will test ASAP and come back to tell you.

    Thanks a lot!

    Enviado desde mi Pixel mediante Tapatalk
     
    • Thank You! Thank You! x 1
  11. peque
    • Premium Supporter

    peque Moderator - Spanish Forums

    Joined:
    August 4, 2007
    Messages:
    855
    Likes Received:
    88
    Ratings:
    +111 / 1
    Home Country:
    Spain Spain
    Show System Specs
    @ltfearme, fix is working like a charm. :) Thanks a lot.

    @RoChess, I've just realized that "release date" is being filled with importation time, not the real film release date. Can you check it? Date added is being filled with file last modification time, which is Ok for me. :)

    Thanks both!!!
     
    • Thank You! Thank You! x 1
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!