WebGrab+Plus a new xmltv grabber

Discussion in 'xmlTV' started by WG++Maker, October 25, 2010.

  1. WG++Maker

    WG++Maker Portal Pro

    Joined:
    October 25, 2010
    Messages:
    130
    Likes Received:
    56
    Gender:
    Male
    Occupation:
    retired microchip engineer
    Location:
    La Gomera, Canary Islands
    Ratings:
    +58 / 0
    Home Country:
    Spain Spain
    I have made an advanced /multisite/incremental EPG grabber and like to share it with MediaPortal users

    Some background:
    * I live in a rural area on the Canary Islands where the internet connections are just dialup speed (no ADSL or 3G). On occasion I can use a satellite downlink . Lets say the speed varies between 2-50kB/s.
    * Under these conditions I have been using several epg grabbers, but it always took 'hours' to just the daily update.
    * The above was the reason I decided to make my own epg grabber which limits downloading to an absolute minimum without loosing richness of data. With it, the daily download for 25 channels in full detail takes about 10 minutes (sat downlink) to 1 hour (in dial-up)

    Some features:
    * Incremental grabbing. It only grabs what is new or what needs updated because it is changed or missing.
    * Can grab from multiple tvguide sites in one run. Each site is programmable through a SiteIni file (I have made a few Dutch, Belgium, Spanish and English siteini files so far). These siteinifiles can be made by the user with the help of debugging modes and the documentation. Of course I am prepared to help or make one if requested
    * The output follows the xmltv standard.
    * Most xmltv elements are supported, like start and stop times, title (+ original title), sub-title, category, rating, star-rating, credits (director, actor, presenter), (production)date and episode. All of which are just a choice of the user and programmable per channel.
    * All kinds of other options, like, days to grab, one-show-update-only and site-index-page-only (less detail but super fast) and xmltv-element-value filtering and modification.
    * Written in C# , executable less that 1MB. Tested for XP, Vista and W7. It runs in command mode, no GUI , no drivers to install. FreeWare, Absolutely non commercial, no malware or virus (I wouldn't even know how to get them in it !?)

    In function it comes close to tvxb but tests show it is about 2 to 3 times faster while providing a more detailed epg.

    The program is in its first beta and available for download on
    WebGrab+Plus, an advanced XMLTV grabber - ServerCare home
    Further there is an active user group in Spain on
    WebGrab+Plus Nuevo Grabber para generar una gran guia EPG - TodoHTPC - TodoHTPC

    Jan


     
    • Like Like x 13
    • Thank You! Thank You! x 1
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. richy759
    • Premium Supporter

    richy759 MP Donator

    Joined:
    July 16, 2007
    Messages:
    130
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United Kingdom United Kingdom
    Hi Jan,
    Having a play with your webgrab++ program, I must say it looks good. There doesn't seem to be a radiotimes.com.ini but there is refernence to radiotimes in the WebGrab++.config.xml file. Is this missing from the latest release? Do you have a file you could attach that I can use to get this going?

    Thanks,
    Rich
     
  4. WG++Maker

    WG++Maker Portal Pro

    Joined:
    October 25, 2010
    Messages:
    130
    Likes Received:
    56
    Gender:
    Male
    Occupation:
    retired microchip engineer
    Location:
    La Gomera, Canary Islands
    Ratings:
    +58 / 0
    Home Country:
    Spain Spain
    Rich,

    I have removed the support for radiotimes because the results were very dissapointing and unreliable.
    For now the prefered tvguide site for the UK is tvguide.co.uk, which comes into two flavours: The standard full detail and the ultra fast (a fraction of a second/program) index-only. This site is rather unique in that it provides a near complete epg in this super fast mode.

    Jan
     
  5. richy759
    • Premium Supporter

    richy759 MP Donator

    Joined:
    July 16, 2007
    Messages:
    130
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United Kingdom United Kingdom
    Jan,
    I have added all the channels I can from tvguide.co.uk but I can't seem to see details for any of the BBC HD channels (BBCHD and BBC ONE HD). You can add them to the guide but they never appear. I was using hover over to get the channel ID, is there another way that you know of?

    managed to find BBC HD channels, just missing 301 and BBC news now...

    Cheers,
    Rich
     
  6. WG++Maker

    WG++Maker Portal Pro

    Joined:
    October 25, 2010
    Messages:
    130
    Likes Received:
    56
    Gender:
    Male
    Occupation:
    retired microchip engineer
    Location:
    La Gomera, Canary Islands
    Ratings:
    +58 / 0
    Home Country:
    Spain Spain
    Rich ,
    see personal message about a complete channel list for tvguide.co.uk.
    BBC news = 66, 301 doesn't exist
     
    • Like Like x 1
  7. doornjoostje

    doornjoostje Portal Pro

    Joined:
    December 6, 2007
    Messages:
    432
    Likes Received:
    14
    Ratings:
    +14 / 0
    Home Country:
    Netherlands Netherlands
    Wouldn't it be nice to incorpocate the faster grabbing into the webepg of mp 1.2.0?
     
  8. WG++Maker

    WG++Maker Portal Pro

    Joined:
    October 25, 2010
    Messages:
    130
    Likes Received:
    56
    Gender:
    Male
    Occupation:
    retired microchip engineer
    Location:
    La Gomera, Canary Islands
    Ratings:
    +58 / 0
    Home Country:
    Spain Spain
    Latest version 1.0.2 went public today @
    WebGrab+Plus, an advanced XMLTV grabber - ServerCare home

    Supported tvguide sites:

    Belgium: tv.be, mijn-tv-gids.be
    Brazil: hagah.com.br
    Netherlands: tvgids.nl, gids.publiekeomroep.nl, tvgids.upc.nl tvtv.nl
    Spain: plus.es, laguiatv.com, teleprogramma.tv
    Sweden: dagenstv.com, svt.se
    UK: tvguide.co.uk, radiotimes.com, tvtv.co.uk
     
    • Like Like x 2
  9. WG++Maker

    WG++Maker Portal Pro

    Joined:
    October 25, 2010
    Messages:
    130
    Likes Received:
    56
    Gender:
    Male
    Occupation:
    retired microchip engineer
    Location:
    La Gomera, Canary Islands
    Ratings:
    +58 / 0
    Home Country:
    Spain Spain
    There is a new version of the program 1.0.4 that went public a couple of days ago.
    WebGrab+Plus, an advanced XMLTV grabber - ServerCare home

    Three new supported sites are added:

    For Denmark : tvguide.dk
    For USA and N/S America's : directv.com
    Belgium : yelo.be

    And of course the sites from the list on the previous post

    There is also a special script that facilitates the use of this grabber in Media Portal and solves a few issues with the Media Portal xmltv importer.
     
  10. richy759
    • Premium Supporter

    richy759 MP Donator

    Joined:
    July 16, 2007
    Messages:
    130
    Likes Received:
    4
    Ratings:
    +4 / 0
    Home Country:
    United Kingdom United Kingdom
    I have noticed that I get a very repeatable "Error downloading page: the remote server returned an error: (500) internal server error" on some channels when grabbing from tvguide.co.uk. It retries but eventually goes onto the next item. This problem doesn't see like the web page isn't there (I'm not familiar with the error codes) but perhaps there is some problem with the request. Any ideas?
    Regards,
    Richard
     
  11. WG++Maker

    WG++Maker Portal Pro

    Joined:
    October 25, 2010
    Messages:
    130
    Likes Received:
    56
    Gender:
    Male
    Occupation:
    retired microchip engineer
    Location:
    La Gomera, Canary Islands
    Ratings:
    +58 / 0
    Home Country:
    Spain Spain
    Richy,

    As you can read below , this problem is created by the server (tvguide.co.uk).

    Fixing 500 errors - general
    This error can only be resolved by fixes to the Web server software. It is not a client-side problem. It is up to the operators of the Web server site to locate and analyse the logs which should give further information about the error.


    These thing seem to happen now and then and (hopefully) dissappear soon.
    Although it shouldn't make much difference and assuming you run WG++ V1.0.3 or higher you can try to add mode w in the configfile. It activates the alternative Grabengine Wget.

    WG++Maker
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!