Adult movie scraper (details+covers) -- v2.0.7 (4 Viewers)

[no1bossman]

RoChess, I used your suggestion a few years back regarding using the parental filtering option. Long story short works fine, but I never see any fanart for these movies. Is this by design, or is there something you suggest I should check first? Cheers.

 

[RoChess]

CDUniverse offers no fanarts/backdrops. TMDb has limited support for 'adult' movies, and requires the IMDb tt-ID reference to work.

So the only way possible would be to manually add IMDb tt-ID to the filename or NFO file and then it will work (if you reimport).

Example: http://www.imdb.com/title/tt1733302/
TMDb = http://www.themoviedb.org/movie/50019-this-ain-t-avatar-xxx

You can also use TMDb scraper obviously, but they have a very limited 'adult' collection, which is user-managed so you can always add missing ones.

 

[no1bossman]

Cheers, that's what I assumed.

 

[Matt Kirby]

RoChess,

Firstly, thanks for you work on this script and the IMDB+ one - great work!

However, I think that the scraping detection at CD Universe has been tightened to stop your script from working.

I noticed that I didn't get any results during an import, and subsequently when visiting CD Universe I get a block warning:

Almost there!

CDU Error Code: DS for IP <redacted>

We're sorry but we have blocked your page request due to possible unusual activity on your computer.

Please click here to continue to our website.

If you are getting this message in error, please contact us at:
Email: abuse@cduniverse.com
CD Universe
101 N Plains Ind Rd
Wallingford, CT 06492
USA
Phone: 203-294-1648
Electronic data feeds containing our product information can be made available to approved partners.
Specialized pages for real time checking on pricing and availability are also available.
For more information, contact partners@cduniverse.com.

So, maybe I hit it with too many requests in one go - fair enough.

However, I have access to a couple of VPNs, so I can easily change my IP address.

To test if it's an automatic block here's what I do:

• I set all files in the importer to be ignored, and restart the importer for good measure to ensure that nothing is pending
• I connect a VPN and check that my external IP address has actually changed (icanhazip.com - very useful!)
• I go to the CD Universe website to check I'm not weirdly blocked already - website comes up
• I release 1 file from the ignored list and restart the importer - no results
• I revisit the CD Universe website and get the block message above

So, I suspect that they are now blocking any requests that look like they are not from a web browser - whereas previously it appeared to let things through up to a certain point and then block.

Not sure if there's actually much you can do - if CD Universe are blocking it outright there's probably not a work-around to continue using them without becoming an "approved partner" - and I can't see that happening! Using another site could be an option, but I know that will need a rewrite and could end up with a similar situation down the line. Just thought I'd post this in case others are having issues, and to let you know.

 

[RoChess]

MovPic allows for adjustment of the user-agent string sent to the site, so I could play with that.

It could also be that they caught on to the way in which I auto-bypass the 18+ validation screen.

Haven't imported any new adult media in a while, so I never noticed. Will have to run some tests, to see what I can do.

 

[Matt Kirby]

Hi RoChess,

Have you had a chance to investigate to see if there's anything that can be done?

 

[RoChess]

Hi RoChess,
Have you had a chance to investigate to see if there's anything that can be done?

Not in great detail, at first view it seems they have a pretty airtight solution. Have a few more ideas, and if those all fail, will consider looking into a full rewrite with another source. First I want to get my new HTPC build, my old one is on its last legs and I've been sitting on these 'new' parts long enough that it is time, and then I'll be better equiped to test new scenarios as well.

 

[Matt Kirby]

A rewrite sounds like a big job! Good luck with your new build.

 

[ltfearme]

You can also use TMDb scraper obviously, but they have a very limited 'adult' collection, which is user-managed so you can always add missing ones.

I added support at the API level but I did not enable it by default in MovingPictures. The reason for that is I didnt want adult movies being accidentally chosen or seen in list of possible matches.

I think what might be best is to add an advanced setting which allows the inclusion of TMDb Adult titles in search / match results.

 

[RoChess]

You can also use TMDb scraper obviously, but they have a very limited 'adult' collection, which is user-managed so you can always add missing ones.

I added support at the API level but I did not enable it by default in MovingPictures. The reason for that is I didnt want adult movies being accidentally chosen or seen in list of possible matches.

I think what might be best is to add an advanced setting which allows the inclusion of TMDb Adult titles in search / match results.

Yeah, pretty much every blockbuster movie has an xXx variant on the title, so that would complicate matches. But if a correct IMDb tt-ID entry is added to the filename/NFO and imported as such then the TMDb backdrop scraper does work. Maybe I just need to work on adding a fuzzy IMDb match system to it, the same way I did for Spanish FilmInfo scraper. Just have to work out the conversion between CDUniverse title format and the way IMDb does it.

At the same time there are other things I need to look at. Just haven't had much time, and getting sick with flu again didn't help this weekend.