Suggestion to use XBMC's XML scrapers for HTTP scraping

[Gamester17]

I would like to suggest that MediaPortal should start using XBMC's XML scrapers for HTTP media information scraping.

XBMC today has a very nice generic API for letting anyone create and/or maintain XML and Regular expression (RegEx) based parsing scrapers without prior programming knowledge, scrapers that scrape HTTP websites for metadata (sites such as IMDb.com, TheTVDB.com, TV.com, and many more), metadata for movies, TV-series and music-videos and parse that metadata before entering it into the database library. I think that it would be great if MediaPortal could integrate that same parser API so that it could use XBMC scraper XML files as is, and vice versa, so cross-compatible with both applications, (later even other open source media center, such as maybe MeediOS, will catch on so that they could all share this library/interface and use the same RegEx XML scraper files).

This scraper API should in theory only require that you first integrate basic XML parser and RegEx parser into MediaPortal (which I assume already exist and if so only a hook should be needed), and then you will have to convert XBMC's ScraperParser.cpp from C++ to C# code in order for MediaPortal to use it naively.

You can download the XBMC source code from the SVN, instructions can be found here:
SourceForge.net - The XBMC (Xbox Media Center) Project

The existing scapers can be found in the SVN under "/trunk/XBMC/system/scrapers/"
(Please note that IMDb.xml is the best scraper of these to use as a reference).
Find the C++ source code for XBMC parser in "/trunk/XBMC/xbmc/utils/ScraperParser.cpp"

If you do not have a SVN tool then download vi the web-interface here:
SourceForge.net XBMC SVN Repository - /trunk/XBMC/system/scrapers/video
SourceForge.net XBMC SVN Repository - /trunk/XBMC/xbmc/utils/ScraperParser.cpp

More about the scraper function can be found in the XBMC wiki:
Category of wiki articles tagged as "Scraper" related
Scraper.xml structure
How To Write Media Info Scrapers for XBMC
Scrap (Scrap.exe for testing of scrapers under Windows)
How To use Scrapers
TV Shows handling in XBMC
Music Videos handling in XBMC


What do you Team-MediaPortal developers think about this idea?

[FlipGer]

Hi,

thanks alot for the hint. The developers will take a look into it.

Flip.

[Gamester17]

FYI, I posted the same idea on MeediOS forum and got a discussion started:
MeediOS :: View topic - Suggestion to use XBMC's XML scrapers for HTTP scraping

They might have a few ideas on improvements or changes to keep track of

[Gamester17]

Just a heads-up; a couple of XBMC developer are now currently in the process of finalizing a new scraper which uses the slightly updated scraper API in XBMC, so if anyone plans on integrating this into MediaPortal today then maybe you should wait for just a couple weeks first, (otherwise your work will have to be redone again later).

The new API will be PCRE (Perl Compatible Regular Expressions) compatible, that will allow PCRE RegEx to be used in the XML files which should make for a faster, simpler and more user-friendly RegEx by those working on XML scrapers. So if you like to be one step ahead then you might want to implement a PCRE parser library, and/or PCRE support in MediaPortal's existing RegEx parser.

Perl Compatible Regular Expressions - Wikipedia, the free encyclopedia
PCRE - Perl Compatible Regular Expressions

PS! Know that some existing scrapers that are available in XBMC's SVN are not currently working, this is simply because the website that they scape have been changed in a way so that someone will need to update those XML scrapers for them to work again, (which none one have done yet if they are broken at any given time). However if MediaPortal (and possible MeediOS as well) starts using the same scraper API then we should all together be able to make a better job of keeping all the available scrapers up-to-date.

[Gamester17]

As I posted FYI information on the MeediOS forum though I should post it here too:

Team-XBMC have plans (in the not so soon future) to also implement the same scraper API and similar XML-files for music (and other audio files) metadata scraping of the internet, and later maybe even when other type of metadata scraping is needed (like weather-forecasts, and XMLTV EPG TV-Guide scraping, etc.), in order to if possible make it a unified scraper API throughout a media center application like XBMC. Again, that is not on the 'soon future' roadmap as many other things have a higher priority and there are only so many hours in a day, ...so far only the concept theory have been written down on paper then we have put it aside for now.

By the way, I think that the optimal solution would be if our projects someday could come to a compromise to make each individual XML scraper 100% compatible so they could be used in each media center application without modifications, that way maybe in the future we could start a new common project (like on sourceforge.net or code.google.com) where we could host and maintain these "HTPC XML scrapers", ...it should then be simple to make each media center application automatically check and download updated scrapers from that common project, which IMHO would be very user-friendly.

[Gamester17]

FYI; XBMC now uses its same generic scrapers (importers) API that I initially described in the first post for not only Movies (and Porn), but also for TV-Shows, Music Videos, and Music. They automatically download Posters, Album Cover Art, Banners, Screenshots, and Fan Art from multiple sites within on scraper (importer), and it is has multi-lingual support.

These two new HOT-TO guides are recommended read for this:
HOW-TO Write Media Info Scrapers (introduction): http://xbmc.org/wiki/?title=HOW-TO_W...ntroduction%29
HOW-TO Write Media Info Scrapers (the complete dummies guide): http://xbmc.org/wiki/?title=HOW-TO_W...mmies_guide%29

Again, I think should consider reusing this same API in MediaPortal so that we should share the scraper XML files 8)

[panic]

is this still considered to get added to MP?