Adult movie scraper (details+covers) -- v2.0.7 | Page 8

Discussion in 'Moving Pictures' started by RoChess, December 17, 2009.

  1. no1bossman

    no1bossman Portal Member

    Joined:
    April 30, 2009
    Messages:
    24
    Likes Received:
    0
    Ratings:
    +0 / 0
    Home Country:
    RoChess, I used your suggestion a few years back regarding using the parental filtering option. Long story short works fine, but I never see any fanart for these movies. Is this by design, or is there something you suggest I should check first? Cheers.


     
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    CDUniverse offers no fanarts/backdrops. TMDb has limited support for 'adult' movies, and requires the IMDb tt-ID reference to work.

    So the only way possible would be to manually add IMDb tt-ID to the filename or NFO file and then it will work (if you reimport).

    Example: http://www.imdb.com/title/tt1733302/
    TMDb = http://www.themoviedb.org/movie/50019-this-ain-t-avatar-xxx

    You can also use TMDb scraper obviously, but they have a very limited 'adult' collection, which is user-managed so you can always add missing ones.
     
    • Funny Funny x 1
  4. no1bossman

    no1bossman Portal Member

    Joined:
    April 30, 2009
    Messages:
    24
    Likes Received:
    0
    Ratings:
    +0 / 0
    Home Country:
    Cheers, that's what I assumed.
     
  5. Matt Kirby

    Matt Kirby Portal Member

    Joined:
    June 14, 2009
    Messages:
    43
    Likes Received:
    8
    Ratings:
    +8 / 0
    Home Country:
    United Kingdom United Kingdom
    RoChess,

    Firstly, thanks for you work on this script and the IMDB+ one - great work!

    However, I think that the scraping detection at CD Universe has been tightened to stop your script from working.

    I noticed that I didn't get any results during an import, and subsequently when visiting CD Universe I get a block warning:



    So, maybe I hit it with too many requests in one go - fair enough.

    However, I have access to a couple of VPNs, so I can easily change my IP address.

    To test if it's an automatic block here's what I do:
    • I set all files in the importer to be ignored, and restart the importer for good measure to ensure that nothing is pending
    • I connect a VPN and check that my external IP address has actually changed (icanhazip.com - very useful!)
    • I go to the CD Universe website to check I'm not weirdly blocked already - website comes up
    • I release 1 file from the ignored list and restart the importer - no results
    • I revisit the CD Universe website and get the block message above
    So, I suspect that they are now blocking any requests that look like they are not from a web browser - whereas previously it appeared to let things through up to a certain point and then block.

    Not sure if there's actually much you can do - if CD Universe are blocking it outright there's probably not a work-around to continue using them without becoming an "approved partner" - and I can't see that happening! Using another site could be an option, but I know that will need a rewrite and could end up with a similar situation down the line. Just thought I'd post this in case others are having issues, and to let you know.
     
  6. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    MovPic allows for adjustment of the user-agent string sent to the site, so I could play with that.

    It could also be that they caught on to the way in which I auto-bypass the 18+ validation screen.

    Haven't imported any new adult media in a while, so I never noticed. Will have to run some tests, to see what I can do.
     
  7. Matt Kirby

    Matt Kirby Portal Member

    Joined:
    June 14, 2009
    Messages:
    43
    Likes Received:
    8
    Ratings:
    +8 / 0
    Home Country:
    United Kingdom United Kingdom
    Hi RoChess,

    Have you had a chance to investigate to see if there's anything that can be done?
     
  8. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    Not in great detail, at first view it seems they have a pretty airtight solution. Have a few more ideas, and if those all fail, will consider looking into a full rewrite with another source. First I want to get my new HTPC build, my old one is on its last legs and I've been sitting on these 'new' parts long enough that it is time, and then I'll be better equiped to test new scenarios as well.
     
  9. Matt Kirby

    Matt Kirby Portal Member

    Joined:
    June 14, 2009
    Messages:
    43
    Likes Received:
    8
    Ratings:
    +8 / 0
    Home Country:
    United Kingdom United Kingdom
    A rewrite sounds like a big job! Good luck with your new build.
     
  10. ltfearme
    • Premium Supporter

    ltfearme Community Plugin Dev

    Joined:
    June 10, 2007
    Messages:
    6,457
    Likes Received:
    4,241
    Gender:
    Male
    Occupation:
    Software Test Engineer
    Location:
    Sydney
    Ratings:
    +5,385 / 0
    Home Country:
    Australia Australia
    I added support at the API level but I did not enable it by default in MovingPictures. The reason for that is I didnt want adult movies being accidentally chosen or seen in list of possible matches.

    I think what might be best is to add an advanced setting which allows the inclusion of TMDb Adult titles in search / match results.
     
  11. RoChess
    • Premium Supporter

    RoChess Extension Developer

    Joined:
    March 10, 2006
    Messages:
    4,182
    Likes Received:
    1,304
    Ratings:
    +1,683 / 2
    Yeah, pretty much every blockbuster movie has an xXx variant on the title, so that would complicate matches. But if a correct IMDb tt-ID entry is added to the filename/NFO and imported as such then the TMDb backdrop scraper does work. Maybe I just need to work on adding a fuzzy IMDb match system to it, the same way I did for Spanish FilmInfo scraper. Just have to work out the conversion between CDUniverse title format and the way IMDb does it.

    At the same time there are other things I need to look at. Just haven't had much time, and getting sick with flu again didn't help this weekend.
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!