WebEPG trouble creating grabber for US radio stations (1 Viewer)

benicehavefun

Portal Member
September 12, 2014
42
10
Home Country
United States of America United States of America
Country flag
I'm attempting to create a WebEPG grabber for US radio stations using DAR.fm as a source. I don't appear to be getting any errors during the grabbing process, but no programs are found for my test station. I suspect the problem is with my template -- but trial and error isn't getting me anywhere, so I'm hoping for some input from someone whose been down this road creating a new grabber.

Here's my US\www_dar_fm_com.xml file:

XML:
<?xml version="1.0" encoding="utf-8"?>
<Grabber>
  <Info language="en" availableDays="7" timezone="Pacific Standard Time" version="2.0" />
  <Channels>
    <Channel id="KNOW-FM" siteId="3740" />
  </Channels>
  <Listing type="Html">
    <Site url="http://www.dar.fm/user_search.php?search_type=stations&amp;station_id=[ID]&amp;list_type=station_detail" post="" external="false" encoding="" />
    <Html>
      <Template name="default" start="&apos;s Schedule" end="Schedule a recording">
        <SectionTemplate tags="TIA">
          <TemplateText>  &lt;tr&gt;&lt;td&gt;&lt;img&gt;&lt;/td&gt;&lt;td&gt;&lt;a&gt;&lt;img&gt;&lt;/a&gt;&lt;span&gt;&lt;a&gt;#TITLE&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;&lt;td&gt;#TIME&lt;/td&gt;&lt;/tr&gt;
          </TemplateText>
        </SectionTemplate>
      </Template>
    </Html>
  </Listing>
</Grabber>

Here's a section of HTML source from DAR.fm that contains the program data I'm trying to extract:

CSS:
<tr>
                                                <td width="10%" style="text-align:center;">
                                                <img src='http://s.mp3tunes.com/images/l/p129.gif' style='min-height:30px;max-height:70px;max-width:70px;margin:0 auto;'/>                                                </td><td width="60%">
                                                                                                        <a href="#" title="Record The World Today" class="new_user_talk"  format_id="101" showinfo_id="6428" show_title="The World Today" station_id="3740"  start_time="2017-05-22 23:00:00"><img src="images/record.png" alt="" width="44" height="22" border="0" class="record" /></a><span>
                                                                                                                <a href="#" title="Details" search_t="The World Today" showinfo_id="6428" class="get_details nodec">
                                                                                                                The World Today                                                        </a>                                                        </span>
                                                                                                </td>
                                                <td width='20%'>11:00pm Pacific</td>
<td width='10%'>60 min</td>


                                        </tr>

I'm attempting to extract "The World Today" and "11:00pm" from the above snippet.

Here's the relevant section of my SetupTV.log file:

Code:
[2017-05-22 13:44:09,906] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Channels
[2017-05-22 13:44:09,909] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Existing channels.xml
[2017-05-22 13:44:10,781] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Grabbers
[2017-05-22 13:44:10,927] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Existing WebEPG.xml
[2017-05-22 13:44:29,686] [Log    ] [WebEPGImporter] [INFO ] - plugin:webepg importing
[2017-05-22 13:44:29,689] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Using directory C:\ProgramData\Team MediaPortal\MediaPortal TV Server
[2017-05-22 13:44:29,693] [Log    ] [WebEPGImporter] [INFO ] - Writing to tvguide.xml in C:\ProgramData\Team MediaPortal\MediaPortal TV Server\WebEPG
[2017-05-22 13:44:29,697] [Log    ] [WebEPGImporter] [INFO ] - Assembly versions:
[2017-05-22 13:44:29,701] [Log    ] [WebEPGImporter] [INFO ] - WebEPG 1.15.0.0
[2017-05-22 13:44:29,704] [Log    ] [WebEPGImporter] [INFO ] - TvLibrary.Interfaces 1.15.0.0
[2017-05-22 13:44:29,734] [Log    ] [WebEPGImporter] [INFO ] - Loading Config File: C:\ProgramData\Team MediaPortal\MediaPortal TV Server\WebEPG\WebEPG.xml
[2017-05-22 13:44:29,739] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Loading Channel Config
[2017-05-22 13:44:29,747] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Getting Channel ID: KNOW-FM
[2017-05-22 13:44:29,751] [Log    ] [WebEPGImporter] [INFO ] -         [1 of 1]
[2017-05-22 13:44:29,756] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Opening US\www_dar_fm_com.xml
[2017-05-22 13:44:29,760] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Version: 2.0
[2017-05-22 13:44:29,763] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: TimeZone, Local: Central Standard Time
[2017-05-22 13:44:29,767] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: TimeZone, Site : Pacific Standard Time
[2017-05-22 13:44:29,790] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: ChannelId: KNOW-FM
[2017-05-22 13:44:30,064] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Reading http://www.dar.fm/user_search.php?search_type=stations&station_id=3740&list_type=station_detail POST:  - Delay: 500ms
[2017-05-22 13:45:11,575] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: No Listings Found
[2017-05-22 13:45:11,579] [Log    ] [WebEPGImporter] [ERROR] - WebEPG: ChannelId: KNOW-FM grabber error - stopping
[2017-05-22 13:45:11,599] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Writing Channel: KNOW-FM
[2017-05-22 13:45:11,611] [Log    ] [WebEPGImporter] [INFO ] - HTTP Statistics: Site www.dar.fm : Pages 1 : Bytes 2392391 : Total Time 00:00:41.4713721 : Av. Rate 57.68777 KBps
[2017-05-22 13:45:11,620] [Log    ] [WebEPGImporter] [INFO ] - plugin:webepg import done

As you can see in the log there's one line that says "grabber error - stopping", but that's after "No Listings Found". It appears to be scrapping the page based on the "HTTP Statistics".

Any input would be appreciated!

Scott
 

RonD

Test Group
  • Team MediaPortal
  • December 20, 2011
    910
    278
    SillyValley CA
    Home Country
    United States of America United States of America
    Country flag
    Have you looked at Localize Mediaportal info for WebEPG in the following link. I'd suggest reading through the pages to see what is needed. Getting started with WebEPG has a steep learning curve takes some time and lots of patience.
    Web-EPG - MediaPortal 1 - MediaPortal Wiki

    I notice the download WebEPG Designer file WebEPG-Designer-v5.zip does not link to the zip, I was able to find a copy I downloaded back in 2012. The WebEPG-Designer tool helps you "experiment" with how your grabber decodes/parses the web page data.

    A different option is use WebGrab+Plus to make a tvguide.xml file and use the xmltv plugin to import the file. A different learning curve.
    WebGrab+Plus
     

    Attachments

    • WebEPG-Designer-v5.zip
      37.6 KB · Views: 237

    benicehavefun

    Portal Member
    September 12, 2014
    42
    10
    Home Country
    United States of America United States of America
    Country flag
    RonD -

    You're the man! I'd given up hope of finding the WebEPG Designer -- since the link was dead, and I could find few mentions of it. But that did the trick, I was able to load the page source and move much more quickly experimenting with possible templates. I've hit on one that works for loading one day's schedule for one radio station. Next I'll set my sights on adding more stations and importing the full 7 days that appears to be available.

    For anyone that's interested, here's the grabber for the current day, one station:

    XML:
    <?xml version="1.0" encoding="utf-8"?>
    <Grabber>
      <Info language="en" availableDays="7" timezone="Pacific Standard Time" version="0.1" />
      <Channels>
        <Channel id="KNOW-FM" siteId="3740" />
      </Channels>
      <Listing type="Html">
        <Site url="http://www.dar.fm/user_search.php?search_type=stations&amp;station_id=[ID]&amp;list_type=station_detail" post="" external="false" encoding="" />
        <Html>
          <Template name="default" start="" end="">
            <SectionTemplate tags="AT">
              <TemplateText>
                &lt;tr&gt;&lt;td&gt;&lt;img&gt;&lt;/td&gt;&lt;td&gt;&lt;a&gt;&lt;img&gt;&lt;/a&gt;&lt;a&gt;&lt;#TITLE&gt;&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;&lt;td&gt;&lt;#START&gt; Pacific&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;/tr&gt;
              </TemplateText>
            </SectionTemplate>
          </Template>
        </Html>
      </Listing>
    </Grabber>

    Understanding how the "template tags" combine with the "template text" during parsing was where WebEPG Designer really made a difference for me. Plus, being able to work with the template text in "normal" form rather than "xml" was sweet.

    Cheers,

    Scott
     

    RonD

    Test Group
  • Team MediaPortal
  • December 20, 2011
    910
    278
    SillyValley CA
    Home Country
    United States of America United States of America
    Country flag
    glad you are making progress. 2 more comments

    1. you can set WebEPG to write a tvguide.xml file to a test folder then use a web browser or xml friendly editor to see what WebEPG is doing. When you are done, you can then write to the epg database, but for debug it helps if you can quickly see WebEPG output

    2. if you are writing to a tvguide.xml file, you can use other tags like #ACTORS etc, as scratch locations to see what your grabber is trying to do.
     
    Last edited:

    benicehavefun

    Portal Member
    September 12, 2014
    42
    10
    Home Country
    United States of America United States of America
    Country flag
    So, for anyone that would like EPG data for radio stations in the US, here's my finished grabber. You'll need to customize it much like what's required for the IMDb US TV grabber. Go to DAR.fm to figure out the "siteId" (a 4 digit number) for each station you'd like guide data for, and modify the www_dar_fm_com.xml file according to this Wiki: WebEPG - TVGuide - MediaPortal 1 - MediaPortal Wiki (though there's no zip code requirement). The modified file needs to be placed in the C:\ProgramData\Team MediaPortal\MediaPortal TV Server\WebEPG\grabbers\US on your MePo TV Server. You should find the www_imdb_com.xml file there already.

    Once you've scanned your stations on your tuner card and placed the xml file in the the above directory, you can map your scanned channels to the stations listed in the grabber file using the WebEPG plugin mapping feature for radio. DAR.fm seems to have data for most stations that have actual shows (talk, news, etc) -- not so much for stations that are mostly music. You can add stations and scheduling data to their site though, with a free account, if there's something you want that's missing. With a paid account (a tad expensive), you can can record in the cloud and download the recordings like podcasts. Obviously, you'll be able to do much the same on your HTPC with MediaPortal, a tuner card that does FM radio, and the EPG data you'll get with this grabber!
     

    Attachments

    • www_dar_fm_com.xml
      934 bytes · Views: 83

    Users who are viewing this thread

    Similar threads

    Hi, I have come to the same conclusion, and there is also a problem that SVT seems to believe that the date is changed at 05:00 in the morning and not at 00:00. That is quite sad when SVT got program guides for four weeks, and not just two as at dagenstv.com (perhaps we should contact him and ask to increase SVT channels to include...
    Hi, I have come to the same conclusion, and there is also a problem that SVT seems to believe that the date is changed at 05:00 in...
    Hi, Swedish SVT have changed their site so that TV-Tablå gets redirected to TV-Tablå so no channels gets imported. SVT have better...
    Replies
    5
    Views
    2K
    MP1 MP2 Webepg not working DE
    Bingo! Exactly as I guessed. The error comes when trying to import the merged channel SVTB/SVT24: [2017-03-22 01:04:38,870] [Log ] [WebEPGImporter] [DEBUG] - Loading Merged Channel SVTB/SVT24 This can only be seen when looking at the full log. ;)
    Bingo! Exactly as I guessed. The error comes when trying to import the merged channel SVTB/SVT24: [2017-03-22 01:04:38,870] [Log...
    I reinstalled my mediabox and now I want WebEPGto grab the information. I have mapped the channels to grabbers and set up a...
    Replies
    9
    Views
    1K
    First double check your WebEPG GrabDays setting in the General Page of the plugin. I just tried 5 days, and downloaded EPG up to Sat afternoon. Make sure you press "save" after making changes. If you still have problems upload your SetupTv.log file after you do a "GrabNow" so we can see what is not working.
    First double check your WebEPG GrabDays setting in the General Page of the plugin. I just tried 5 days, and downloaded EPG up to...
    Hello community! I just recently set up the WebEPG Schedule in accordance to these directions...
    Replies
    1
    Views
    2K
    I am not sure what I was doing wrong but your instructions fixed my problem. Guide data is now populated. Thank you so much for your help.
    I am not sure what I was doing wrong but your instructions fixed my problem. Guide data is now populated. Thank you so much for...
    I believe I have configured the www_imdb_com_46803.xml file correctly but I cannot seem to grab data. I am no programmer but I not...
    Replies
    5
    Views
    2K
    Bekanntes Problem, das am Zusammenspiel von TV Today und WebEPG liegt: https://forum.team-mediaportal.com/threads/web-epg-kein-import-von-tv-today.130469/ Alternativ vielleicht Clickfinder für 12 EUR pro Jahr einsetzen?
    Bekanntes Problem, das am Zusammenspiel von TV Today und WebEPG liegt...
    Hallo Leute, ohne EPG macht MediaPortal keinen Spaß. Ich habe mein Mediaportal vor ein paar Wochen auf 1.8.0.0 aktualisiert...
    Replies
    3
    Views
    2K
    Top Bottom